The thesis is divided into two main parts: i) Nonparametric statistics on high-dimensional and functional spaces, and ii) Nonparametric statistics on Riemannian manifolds. In this part, we will summarize the major contributions of the thesis. Nonparametric statistics on high-dimensional and functional spacesIn statistical learning, we introduce a new notion entitled: scalable Gaussian process classifier. The proposal is more general than the usual Gaussian process classifier for representing and classifying data lying on high-dimensional spaces. It is more advantageous for learning the hyper-parameters of the mapping (embedding) that maps initial data into a low-dimensional (feature) space and those of the Gaussian process classifier through its covariance function, jointly, with different optimization methods. The modified covariance function, depending on the embedding and operating on the feature space, is more expressive since the Euclidean metric is more informative in low-dimensional spaces. To summarize, our formulation takes care of non-linearity/high-correlation of data and increases the separability between them thanks to the Representer Theorem. In order to estimate the model's hyper-parameters we usually maximize the marginal likelihood. Unlike regression, the computation of the exact marginal likelihood remains difficult and even impossible in the classification case due to the discrete likelihoods. Thus, we introduce two methods to approximate the non-Gaussian posterior distribution by a Gaussian one in order to improve the efficiency and the scalability of the Gaussian process.For functional and even high-dimensional data, we also introduce the notion of Gaussian processes indexed by probability density functions. We will show how Gaussian processes can be defined into functional spaces, in particular that of density functions endowed with the Fisher-Rao metric. More precisely, we will extend the traditional methods of nonparametric statistics based on Gaussian processes from finite vectors in Euclidean spaces to constrained functions with Riemannian metrics. Our motivation is that several categories of observations can be represented by their density functions with more advantages than initial vector or functional inputs. This choice is very crucial for many reasons. First of all, density functions make the problem formulation more understood when identifying the initial vector inputs or functional data, which are hard to interpret, by their occurrences or their corresponding probabilities. Second, density functions improve the visualization of local data distributions. Finally, when dealing with high dimensional datasets (set of repetitive features), we can visualize them using density functions which would be very helpful to explore the skewness of initial data.Applications: Image classification (breast cancer/metallic boxes/growth charts /maize leaves/animal temperature) and video classification (violence detection).Nonparametric statistics on Riemannian manifoldsIn statistical learning on Riemannian manifold of curves, one of major problems is that of registration. For curve registration, we have to find the optimal deformation in terms of the best reparametrization function (or local distribution) between two curves. The space of reparametrization functions is a group of diffeomorphisms for the composition operation, which makes the optimization task quite complicated due to the structure of the group. In fact, there is no intuitive direction nor an underlying metric (or structure) in this group. (...)