Generalized minimal residual method explained

In mathematics, the generalized minimal residual method (GMRES) is an iterative method for the numerical solution of an indefinite nonsymmetric system of linear equations. The method approximates the solution by the vector in a Krylov subspace with minimal residual. The Arnoldi iteration is used to find this vector.

The GMRES method was developed by Yousef Saad and Martin H. Schultz in 1986.^[1] It is a generalization and improvement of the MINRES method due to Paige and Saunders in 1975.^{[2] [3]} The MINRES method requires that the matrix is symmetric, but has the advantage that it only requires handling of three vectors. GMRES is a special case of the DIIS method developed by Peter Pulay in 1980. DIIS is applicable to non-linear systems.

The method

Denote the Euclidean norm of any vector v by

\|v\| . Denote the (square) system of linear equations to be solved by $$Ax\; =\; b.$$The matrix A is assumed to be invertible of size m-by-m. Furthermore, it is assumed that b is normalized, i.e., that \|b\| = 1 .

The n-th Krylov subspace for this problem is $$K\_n\; =\; K\_n(A,r\_0)\; =\; \backslash operatorname\; \backslash ,\; \backslash .\; \backslash ,$$where

r _{0} = b - A x _{0} is the initial error given an initial guess x _{0} \ne 0 . Clearly r _{0} = b if x _{0} = 0 .

GMRES approximates the exact solution of

Ax = b by the vector x _{n} \in x _{0} + K _{n} that minimizes the Euclidean norm of the residual r _{n=} b - Ax _{n} .

The vectors

r _{0,Ar} _{0,\ldots} A ^{n-1} r _{0} might be close to linearly dependent, so instead of this basis, the Arnoldi iteration is used to find orthonormal vectors q _{1,} q _{2,} \ldots, q _{n} which form a basis for K _{n} . In particular, q _{1=\|r} _{0\|} ^{-1} ^{2} r _{0} .

Therefore, the vector

x _{n} \in x _{0} + K _{n} can be written as x _{n} = x _{0} + Q _{n} y _{n} with y _{n} \in R ^{n} , where Q _{n} is the m-by-n matrix formed by q _{1,\ldots,q} _{n} . In other words, finding the n-th approximation of the solution (i.e., x _{n} ) is reduced to finding the vector y _{n} , which is determined via minimizing the residue as described below.

The Arnoldi process also constructs

\tilde{H} _{n} , an (n +1 )-by-n upper Hessenberg matrix which satisfies $$AQ\_n\; =\; Q\_\; \backslash tilde\_n\; \backslash ,$$an equality which is used to simplify the calculation of y _{n} (see). Note that, for symmetric matrices, a symmetric tri-diagonal matrix is actually achieved, resulting in the MINRES method.

Because columns of

Q _{n} are orthonormal, we havewhere $$e\_1\; =\; (1,0,0,\backslash ldots,0)^T\; \backslash ,$$ is the first vector in the standard basis of R ^{n+1} , and$$\backslash beta\; =\; \backslash |r\_0\backslash |\; \backslash ,,$$ r _{0} being the first trial residual vector (usually b ). Hence, x _{n} can be found by minimizing the Euclidean norm of the residual$$r\_n\; =\; \backslash tilde\_n\; y\_n\; -\; \backslash beta\; e\_1.$$This is a linear least squares problem of size n.

This yields the GMRES method. On the

n -th iteration:

1. calculate

q _{n} with the Arnoldi method;

1. find the

y _{n} which minimizes \|r _{n\|} ;

1. compute

x _{n} = x _{0} + Q _{n} y _{n} ;

1. repeat if the residual is not yet small enough.

At every iteration, a matrix-vector product

A q _{n} must be computed. This costs about 2 m ^{2} floating-point operations for general dense matrices of size m , but the cost can decrease to O(m) for sparse matrices. In addition to the matrix-vector product, O(nm) floating-point operations must be computed at the n -th iteration.

Convergence

The nth iterate minimizes the residual in the Krylov subspace

K _{n} . Since every subspace is contained in the next subspace, the residual does not increase. After m iterations, where m is the size of the matrix A, the Krylov space K_m is the whole of R^m and hence the GMRES method arrives at the exact solution. However, the idea is that after a small number of iterations (relative to m), the vector x_n is already a good approximation to the exact solution.

This does not happen in general. Indeed, a theorem of Greenbaum, Pták and Strakoš states that for every nonincreasing sequence a₁, ..., a_m-1, a_m = 0, one can find a matrix A such that the = a_n for all n, where r_n is the residual defined above. In particular, it is possible to find a matrix for which the residual stays constant for m - 1 iterations, and only drops to zero at the last iteration.

In practice, though, GMRES often performs well. This can be proven in specific situations. If the symmetric part of A, that is

(A ^{T} + A)/ 2 , is positive definite, then$$\backslash |r\_n\backslash |\; \backslash leq\; \backslash left(1-\backslash frac\; \backslash right)^\; \backslash |r\_0\backslash |,$$where λ _{min} (M) and λ _{max} (M) denote the smallest and largest eigenvalue of the matrix M , respectively.^[4]

If A is symmetric and positive definite, then we even have$$\backslash |r\_n\backslash |\; \backslash leq\; \backslash left(\backslash frac\; \backslash right)^\; \backslash |r\_0\backslash |.$$where

\kappa _{2(A)} denotes the condition number of A in the Euclidean norm.

In the general case, where A is not positive definite, we have$$\backslash frac\; \backslash le\; \backslash inf\_\; \backslash |p(A)\backslash |\; \backslash le\; \backslash kappa\_2(V)\; \backslash inf\_\; \backslash max\_\; |p(\backslash lambda)|,\; \backslash ,$$where P_n denotes the set of polynomials of degree at most n with p(0) = 1, V is the matrix appearing in the spectral decomposition of A, and σ(A) is the spectrum of A. Roughly speaking, this says that fast convergence occurs when the eigenvalues of A are clustered away from the origin and A is not too far from normality.^[5]

All these inequalities bound only the residuals instead of the actual error, that is, the distance between the current iterate x_n and the exact solution.

Extensions of the method

Like other iterative methods, GMRES is usually combined with a preconditioning method in order to speed up convergence.

The cost of the iterations grow as O(n²), where n is the iteration number. Therefore, the method is sometimes restarted after a number, say k, of iterations, with x_k as initial guess. The resulting method is called GMRES(k) or Restarted GMRES. For non-positive definite matrices, this method may suffer from stagnation in convergence as the restarted subspace is often close to the earlier subspace.

The shortcomings of GMRES and restarted GMRES are addressed by the recycling of Krylov subspace in the GCRO type methods such as GCROT and GCRODR.^[6] Recycling of Krylov subspaces in GMRES can also speed up convergence when sequences of linear systems need to be solved.^[7]

Comparison with other solvers

The Arnoldi iteration reduces to the Lanczos iteration for symmetric matrices. The corresponding Krylov subspace method is the minimal residual method (MinRes) of Paige and Saunders. Unlike the unsymmetric case, the MinRes method is given by a three-term recurrence relation. It can be shown that there is no Krylov subspace method for general matrices, which is given by a short recurrence relation and yet minimizes the norms of the residuals, as GMRES does.

Another class of methods builds on the unsymmetric Lanczos iteration, in particular the BiCG method. These use a three-term recurrence relation, but they do not attain the minimum residual, and hence the residual does not decrease monotonically for these methods. Convergence is not even guaranteed.

The third class is formed by methods like CGS and BiCGSTAB. These also work with a three-term recurrence relation (hence, without optimality) and they can even terminate prematurely without achieving convergence. The idea behind these methods is to choose the generating polynomials of the iteration sequence suitably.

None of these three classes is the best for all matrices; there are always examples in which one class outperforms the other. Therefore, multiple solvers are tried in practice to see which one is the best for a given problem.

Solving the least squares problem

One part of the GMRES method is to find the vector

y _{n} which minimizes$$\backslash left\backslash |\; \backslash tilde\_n\; y\_n\; -\; \backslash beta\; e\_1\; \backslash right\backslash |.$$Note that \tilde{H} _{n} is an (n + 1)-by-n matrix, hence it gives an over-constrained linear system of n+1 equations for n unknowns.\tilde{R} _{n} such that$$\backslash Omega\_n\; \backslash tilde\_n\; =\; \backslash tilde\_n.$$The triangular matrix has one more row than it has columns, so its bottom row consists of zero. Hence, it can be decomposed as$$\backslash tilde\_n\; =\; \backslash begin\; R\_n\; \backslash \backslash \; 0\; \backslash end,$$where R _{n} is an n-by-n (thus square) triangular matrix.

The QR decomposition can be updated cheaply from one iteration to the next, because the Hessenberg matrices differ only by a row of zeros and a column: where h_n+1 = (h_1,n+1, ..., h_n+1,n+1)^T. This implies that premultiplying the Hessenberg matrix with Ω_n, augmented with zeroes and a row with multiplicative identity, yields almost a triangular matrix:This would be triangular if σ is zero. To remedy this, one needs the Givens rotationwhere$$c\_n\; =\; \backslash frac\; \backslash quad\backslash text\backslash quad\; s\_n\; =\; \backslash frac.$$With this Givens rotation, we formIndeed, is a triangular matrix with $r\_\; =\; \backslash sqrt$.

Given the QR decomposition, the minimization problem is easily solved by noting thatDenoting the vector

\beta\Omega _{ne} _{1} by$$\backslash tilde\_n\; =\; \backslash begin\; g\_n\; \backslash \backslash \; \backslash gamma\_n\; \backslash end$$with g_n ∈ Rⁿ and γ_n ∈ R, this isThe vector y that minimizes this expression is given by $$y\_n\; =\; R\_n^\; g\_n.$$Again, the vectors g _{n} are easy to update.^[8]

Example code

Regular GMRES (MATLAB / GNU Octave)

function [x, e] = gmres(A, b, x, max_iterations, threshold) n = length(A); m = max_iterations;

% use x as the initial vector r = b - A * x;

b_norm = norm(b); error = norm(r) / b_norm;

% initialize the 1D vectors sn = zeros(m, 1); cs = zeros(m, 1); %e1 = zeros(n, 1); e1 = zeros(m+1, 1); e1(1) = 1; e = [error]; r_norm = norm(r); Q(:,1) = r / r_norm; % Note: this is not the beta scalar in section "The method" above but % the beta scalar multiplied by e1 beta = r_norm * e1; for k = 1:m

% run arnoldi [H(1:k+1, k), Q(:, k+1)] = arnoldi(A, Q, k); % eliminate the last element in H ith row and update the rotation matrix [H(1:k+1, k), cs(k), sn(k)] = apply_givens_rotation(H(1:k+1,k), cs, sn, k); % update the residual vector beta(k + 1) = -sn(k) * beta(k); beta(k) = cs(k) * beta(k); error = abs(beta(k + 1)) / b_norm;

% save the error e = [e; error];

if (error <= threshold) break; end end % if threshold is not reached, k = m at this point (and not m+1) % calculate the result y = H(1:k, 1:k) \ beta(1:k); x = x + Q(:, 1:k) * y;end

%----------------------------------------------------%% Arnoldi Function %%----------------------------------------------------%function [h, q] = arnoldi(A, Q, k) q = A*Q(:,k); % Krylov Vector for i = 1:k % Modified Gram-Schmidt, keeping the Hessenberg matrix h(i) = q' * Q(:, i); q = q - h(i) * Q(:, i); end h(k + 1) = norm(q); q = q / h(k + 1);end

%---------------------------------------------------------------------%% Applying Givens Rotation to H col %%---------------------------------------------------------------------%function [h, cs_k, sn_k] = apply_givens_rotation(h, cs, sn, k) % apply for ith column for i = 1:k-1 temp = cs(i) * h(i) + sn(i) * h(i + 1); h(i+1) = -sn(i) * h(i) + cs(i) * h(i + 1); h(i) = temp; end

% update the next sin cos values for rotation [cs_k, sn_k] = givens_rotation(h(k), h(k + 1));

% eliminate H(i + 1, i) h(k) = cs_k * h(k) + sn_k * h(k + 1); h(k + 1) = 0.0;end

%%----Calculate the Givens rotation matrix----%%function [cs, sn] = givens_rotation(v1, v2)% if (v1

See also

• Biconjugate gradient method

References

• Book: Meister . Andreas . Numerik linearer Gleichungssysteme . Vömel . Christof . 2005 . Vieweg . 978-3-528-13135-7 . Wiesbaden.
• Book: Saad, Y. . Iterative Methods for Sparse Linear Systems . 2003 . SIAM . 978-0-89871-534-7 . 2nd . Philadelphia.
• Eisenstat . Stanley C. . Elman . Howard C. . Schultz . Martin H. . 1983 . Variational Iterative Methods for Nonsymmetric Systems of Linear Equations . SIAM Journal on Numerical Analysis . 20 . 2 . 345–357 . 10.1137/0720023 . 0036-1429.
• Dongarra et al., Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, 2nd Edition, SIAM, Philadelphia, 1994
• Imankulov . Timur . Lebedev . Danil . Matkerim . Bazargul . Daribayev . Beimbet . Kassymbek . Nurislam . 2021-10-08 . Numerical Simulation of Multiphase Multicomponent Flow in Porous Media: Efficiency Analysis of Newton-Based Method . Fluids . en . 6 . 10 . 355 . 10.3390/fluids6100355 . free . 2311-5521.

Notes and References

1. Saad . Youcef . Schultz . Martin H. . 1986 . GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems . SIAM Journal on Scientific and Statistical Computing . en . 7 . 3 . 856–869 . 10.1137/0907058 . 0196-5204.
2. Paige and Saunders, "Solution of Sparse Indefinite Systems of Linear Equations", SIAM J. Numer. Anal., vol 12, page 617 (1975) https://doi.org/10.1137/0712047
3. Naoufal . Nifa . Solveurs performants pour l'optimisation sous contraintes en identification de paramètres . Efficient solvers for constrained optimization in parameter identification problems . fr . 2017 .
4. . NB all results for GCR also hold for GMRES, cf.
5. Book: Trefethen . Lloyd N. . Bau . David, III. . Numerical Linear Algebra . 1997 . Society for Industrial and Applied Mathematics . 978-0-89871-361-9 . Philadelphia . Theorem 35.2 .
6. 10.1016/j.jcp.2015.09.040. Recycling Krylov subspaces for CFD applications and a new hybrid recycling solver. 2015. Amritkar. Amit. de Sturler. Eric. Świrydowicz. Katarzyna . Tafti. Danesh . Ahuja. Kapil . Journal of Computational Physics . 303 . 222 . 1501.03358 . 2015JCoPh.303..222A . 2933274 .
7. Ph.D. . 10.14279/depositonce-4147 . Recycling Krylov subspace methods for sequences of linear systems . 2014 . Gaul . André . TU Berlin.
8. Book: Stoer . Josef . Bulirsch . Roland . Introduction to Numerical Analysis . 2002 . 3rd . Springer . 978-0-387-95452-3 . Texts in Applied Mathematics . New York . §8.7.2 .