With the advent of the Internet of Things (IoT), connected devices are becoming ubiquitous and progressively introduced into domestic networks. These heterogeneous constrained objects, are potential targets for a wide spectrum of anomalies, e.g., software and hardware dysfunctions and malicious attacks. Orchestrating such fleet of devices, ensuring their nominal operation, and diagnosing anomalous activities are becoming a primordial asset for Telecommunication operators and service providers.The present thesis explores the problem of unsupervised and robust anomaly detection based on network traffic analysis, and applied to Internet of Things (IoT) device monitoring. The principle is to firstly model the nominal behavior of a connected device using a set of metadata extracted from its network traffic. Then, any atypical behavior, i.e., any significant deviation from the nominal behavior expected by the model, is considered as an anomaly. In particular, we explore the topic of representation learning with artificial neural networks, and specifically the autoencoder architecture. To model the norm, classical anomaly detection approaches train an autoencoder to reconstruct the nominal data, minimizing an error criterion between the initial data and the output of the network. As the new anomalous observations are structurally different, their reconstruction is accompanied by a significant loss of information, with a large reconstruction error. However, the constitution of a training dataset devoid of non-nominal data remains costly, time-consuming, and most of the time infeasible, as experts cannot updatedomain knowledge with new anomalies exactly as they arise. Thus, we strove to develop robust autoencoders, which are able to model the norm, even if the training dataset is contaminated by an unknown ratio anomalies.In particular, we propose three contributions. The first two contributions focus on the robust detection of punctual anomalies and the third one extends the scope of our study to contextual and collective anomaly detection in time series.First, we propose to analyze the distribution of reconstruction scores in order to perform a self-supervision. We dynamically estimate thresholds from the reconstruction histogram to identify potential anomalies in the training set. Then, we leverage them to enhance the discriminative potential of the model. This contribution has been realized with RADON (Robust Autoencoder with Dynamic Outlier filteriNg).In a second step, we propose to harness the power of variational autoencoders and normalizing flows to improve the anomaly estimation process. The thresholding criterion on the reconstruction error histogram is replaced by a statistical modeling thanks to the Extreme Value Theory (EVT). The combination of these contributions leads to our model GRAnD (Generative Robust autoencoder for unsupervised Anomaly Detection).Finally, we propose a third contribution, RESIST (Robust transformEr developed for unsupervised tIme Series anomaly deTection), which relies on sequence-to-sequence models, and in particular Transformers, to model the temporal dependencies between tokens network flow sequence and detect any contextual and collective deviation. The impact of contaminants during learning is significantly mitigated thanks to a Siamese architecture and a robust objective function.Indeed, infrequent and atypical observations have less weight than common data and have less influence on the optimization of our model parameters.We demonstrate the advantage of our methods over classical approaches of the literature, and complete the performance characterization with ablation and sensitivity studies. These experimental analyses are based on 4 public datasets, NSL-KDD, MedBIoT, CICIDS17, and TON-IoT, including network traffic metadata from real IoT devices.