Centralizing data is impractical or undesirable in many scenarios, especially when sensitive information is involved. In such cases, the need for alternative methods becomes evident. As large datasets are known to facilitate the learning of efficient models, distributed methods have emerged as a powerful tool to overcome the challenges posed by centralized data. Consequently, this thesis introduces innovative approaches to tackle large-scale Bayesian inference and uncertainty quantification, aiming to provide effective solutions in the context of distributed data environments. The federated Monte Carlo (MC) approaches allow multiple agents/nodes to conduct computations locally and securely, with a central server combining the results to obtain samples from the global posterior distribution. Bayesian posterior sampling techniques benefit from the incorporation of prior knowledge, leading to improved results. Additionally, the uncertainty associated with the parameters and the predictions are naturally quantified, which is crucial for decision-making. Especially with limited or noisy data, the ability to quantify uncertainty becomes even more essential.The first part of this manuscript focuses on MC via Markov chains (MCMC) methods. In particular, we introduce two procedures, named DG-LMC and FALD, designed to target a global posterior distribution while ensuring scalability. Local agents are associated with a central server that aggregates information from each agent to generate samples from the posterior distribution. This approach minimizes the need to transmit large amounts of data across participating agents, making it especially advantageous in federated environments with limited bandwidth or low computational power. Considering the distributed nature of today's datasets, concerns about trust and confidence arise when transferring information to a central server. The proposed methods not only address practical applications but also extend existing learning algorithms to Bayesian inference problems. The proposed approach contributes to the development of more robust and efficient machine learning algorithms, and holds potential applications in various domains, including epidemiology and finance, where large-scale inference and data privacy are significant concerns. To demonstrate the effectiveness of the approach, real-world datasets are employed, and the results show the performance of federated MCMC simulation.The second part of the thesis focuses on uncertainty management. Initially, we present the Bayesian approach, which involves defining a prior and a likelihood. To address bandwidth bottlenecks while efficiently generating samples, our proposed approach leverages compression operators. In the final part of this thesis, we introduce a novel frequentist FL method based on conformal predictions. Unlike other methods, our model-agnostic approach does not rely on specific model assumptions and can be applied to any underlying prediction model. Referred to as DP-FedCP, this method leverages quantile regression techniques to generate personalized prediction sets while maintaining robustness to outliers. The label shift between agents is addressed by determining quantiles based on importance weights. One crucial aspect of our approach is the preservation of differential privacy, it allows users to assess the confidence level of predictions and make informed decisions based on the associated level of uncertainty. By incorporating this privacy measure, we ensure safeguarding the user' sensitive information.