This work focus on the iterative solution of large linear systems with multiple right-hand sidesthat appear in various scientific applications. When multiple right-hand sides have to be solved, blockvariants of Krylov subspace methods are the methods of choice. Because the block algorithm implementation enables the use of efficient BLAS-3 like computational kernels, the time for solution is expected to reduce. Unfortunately these potential advantages come at the price of numerical difficulties induced by the possible different convergence rate of the right-hand sides or linear combination of some of them. This numerical feature is referred to as partial convergence. For sequences of unsymmetric linear systems in Chapter 2, we develop novel block minimum norm residual approach that combines two main ingredients. The first component exploits ideas from GCRO-DR, enabling us to recycle spectral information from one linear system to the next. The second component is the numerical mechanism for managing partial convergence of the right-hand sides, referred to as the inexact breakdown detecting mechanism in IB-BGMRES, that enables the monitoring of the rank deficiency in the residual space basis expanded blockwise. Next, for the class of block minimum norm residual approaches, that relies on a block Arnoldi-like equality between the search space and residual space, we introduce new search space expansion policies defined on novel criteria to detect partial convergence. These novel detection criteria are tuned to the targeted stopping criterion and convergence threshold. This enables us to monitor the computational effort while ensuring the final accuracy of each individual solution. Because the partial convergence issue appears in any type of block Krylov solvers, in Chapter 3, we extend the partial convergence detecting mechanism to the block variant of the conjugate residual method, a minimum residual norm approach with short term recurrence scheme, for symmetric but not necessary positive definite linear systems. Then, inspired by the idea of reusing generated information to approximate spectral information for accelerating convergence, in Chapter 4, we devise a corresponding subspace recycling strategy. More precisely, we apply the thick-restart ideas introduced in the Lanczos method for eigenpair calculation in the block conjugate gradient algorithm (BCG) for the symmetric positive definite linear systems. We also investigate the possibility to use the partial convergence detection to the BCG algorithm as a heuristic. Finally, these partial convergence detection and subspace recycling strategies can be efficiently combined to design a deflated block conjugate gradient algorithm for sequences of symmetric positive definite linear systems. An alternative path to the above mentioned traditional numerical linear algebra approaches is to consider using the scientific machine learning techniques. In Chapter 5, we presents some ways of hybridizing the newly emerging deep learningsolvers and the more traditional numerical linear algebra techniques to let them benefit from each other. Inthe context of solving a heterogeneous Helmholtz equation, we first focus on introducing some mathematicalingredients from classical iterative solver into the training phase of a recently proposed deep neural networksolver. The main benefit is a significant improvement in the training phase that is more robust and faster,which turns out to be applicable to the testing process as well. Furthermore, once the network solvers havebeen properly trained, their inferences can be applied as a nonlinear preconditioner in the traditional flexibleGMRES method. This part demonstrates that these hybrid variants have clear advantages over both thenewly emerging deep neural network approach and the classical iterative Krylov solver in terms of bothcomputational cost and accuracy of the computed solution.