The development of intelligent agents has seen significant progress in the lastdecade, showing impressive capabilities in various tasks, such as video games orrobot navigation. These advances were made possible by the advent of deep reinforcementlearning (RL), which allows to train neural network-based policies,through interaction of the agent with its environment. However, in practice, theimplementation of such agents requires significant human intervention and priorknowledge on the task at hand, which can be seen as forms of supervision. In thisthesis, we tackle three different aspects of supervision in RL, and propose methodsto reduce the amount of human intervention required to train agents.We first investigate the impact of supervision on the choice of observations seen bythe agent. In robot navigation for example, the modalities of the environment observedby the agent are an important design choice that can have a significant impacton the difficulty of the task. To tackle this question, we focus on image-goal navigationin photo-realistic environments, and propose a method for learning to navigatefrom raw visual inputs, i.e., without relying on depth or position information.Second, we target the problem of reward supervision in RL. Standard RL algorithmsrely on the availability of a well-shaped reward function to solve a specific task.However, the design of such functions is often a difficult and time-consuming process,which requires prior knowledge on the task and environment. This limits thescalability and generalization capabilities of the designed approaches. To addressthis issue, we tackle the problem of learning state-reaching policies without rewardsupervision, and design methods that leverage intrinsic reward functions to learnsuch policies.Finally, we study the problem of learning agents offline, from pre-collected demonstrations,and question the availability of such data. Collecting expert trajectories isoften a difficult and time-consuming process, which can be more difficult than thedownstream task itself. Offline algorithms should therefore rely on existing data,and we propose a method for learning goal-conditioned agents from tutorial videos,which contains expert demonstrations aligned with natural language captions.