Distributed reinforcement learning (DRL) has been studied as an approach to learn control policies thorough interactions between distributed agents and environments. The main emphasis of DRL has been put on the way to learn sub-optimal policies with the least or limited sharing of agents' perceptions. In this study, we introduce a new concept, privacy-preservation, into DRL. In our setting, agents' perceptions, such as states, rewards, and actions, are not only distributed but also are desired to be kept private. This can occur when agents' perceptions include private or confidential information. Conventional DRL algorithms could be applied to such problems, but do not theoretically guarantee privacy preservation. We design solutions that achieve optimal policies in standard reinforcement leering settings without requiring the agents to share their private information by means of well-known cryptographic primitive, secure function evaluation.