Geometrical and Spectral Properties of the Orthogonal Projections of the Identity

Abstract

We analyze the best approximation 𝐴𝑁 (in the Frobenius sense) to the identity matrix in an arbitrary matrix subspace 𝐴𝑆 (𝐴∈ℝ^𝑛×𝑛 nonsingular, 𝑆 being any fixed subspace of ℝ^𝑛×𝑛). Some new geometrical and spectral properties of the orthogonal projection 𝐴𝑁 are derived. In particular, new inequalities for the trace and for the eigenvalues of matrix 𝐴𝑁 are presented for the special case that 𝐴𝑁 is symmetric and positive definite.

1. Introduction

The set of all 𝑛×𝑛 real matrices is denoted by ℝ^𝑛×𝑛, and 𝐼 denotes the identity matrix of order 𝑛. In the following, 𝐴^𝑇 and tr(𝐴) denote, as usual, the transpose and the trace of matrix 𝐴∈ℝ^𝑛×𝑛. The notations ⟨⋅,⋅⟩_𝐹 and ‖⋅‖_𝐹 stand for the Frobenius inner product and matrix norm, defined on the matrix space ℝ^𝑛×𝑛. Throughout this paper, the terms orthogonality, angle, and cosine will be used in the sense of the Frobenius inner product.

Our starting point is the linear system 𝐴𝑥=𝑏,𝐴∈ℝ^𝑛×𝑛,𝑥,𝑏∈ℝ^𝑛, (1) where 𝐴 is a large, nonsingular, and sparse matrix. The resolution of this system is usually performed by iterative methods based on Krylov subspaces (see, e.g., [1, 2]). The coefficient matrix 𝐴 of the system (1) is often extremely ill-conditioned and highly indefinite, so that in this case, Krylov subspace methods are not competitive without a good preconditioner (see, e.g., [2, 3]). Then, to improve the convergence of these Krylov methods, the system (1) can be preconditioned with an adequate nonsingular preconditioning matrix 𝑁, transforming it into any of the equivalent systems 𝑁𝐴𝑥=𝑁𝑏,𝐴𝑁𝑦=𝑏,𝑥=𝑁𝑦, (2) the so-called left and right preconditioned systems, respectively. In this paper, we address only the case of the right-hand side preconditioned matrices 𝐴𝑁, but analogous results can be obtained for the left-hand side preconditioned matrices 𝑁𝐴.

The preconditioning of the system (1) is often performed in order to get a preconditioned matrix 𝐴𝑁 as close as possible to the identity in some sense, and the preconditioner 𝑁 is called an approximate inverse of 𝐴. The closeness of 𝐴𝑁 to 𝐼 may be measured by using a suitable matrix norm like, for instance, the Frobenius norm [4]. In this way, the problem of obtaining the best preconditioner 𝑁 (with respect to the Frobenius norm) of the system (1) in an arbitrary subspace 𝑆 of ℝ^𝑛×𝑛 is equivalent to the minimization problem; see, for example, [5] min_𝑀∈𝑆‖𝐴𝑀−𝐼‖_𝐹=‖𝐴𝑁−𝐼‖_𝐹. (3)

The solution 𝑁 to the problem (3) will be referred to as the “optimal” or the “best” approximate inverse of matrix 𝐴 in the subspace 𝑆. Since matrix 𝐴𝑁 is the best approximation to the identity in subspace 𝐴𝑆, it will be also referred to as the orthogonal projection of the identity matrix onto the subspace 𝐴𝑆. Although many of the results presented in this paper are also valid for the case that matrix 𝑁 is singular, from now on, we assume that the optimal approximate inverse 𝑁 (and thus also the orthogonal projection 𝐴𝑁) is a nonsingular matrix. The solution 𝑁 to the problem (3) has been studied as a natural generalization of the classical Moore-Penrose inverse in [6], where it has been referred to as the 𝑆-Moore-Penrose inverse of matrix 𝐴.

The main goal of this paper is to derive new geometrical and spectral properties of the best approximations 𝐴𝑁 (in the sense of formula (3)) to the identity matrix. Such properties could be used to analyze the quality and theoretical effectiveness of the optimal approximate inverse 𝑁 as preconditioner of the system (1). However, it is important to highlight that the purpose of this paper is purely theoretical, and we are not looking for immediate numerical or computational approaches (although our theoretical results could be potentially applied to the preconditioning problem). In particular, the term “optimal (or best) approximate inverse” is used in the sense of formula (3) and not in any other sense of this expression.

Among the many different works dealing with practical algorithms that can be used to compute approximate inverses, we refer the reader to for example, [4, 7–9] and to the references therein. In [4], the author presents an exhaustive survey of preconditioning techniques and, in particular, describes several algorithms for computing sparse approximate inverses based on Frobenius norm minimization like, for instance, the well-known SPAI and FSAI algorithms. A different approach (which is also focused on approximate inverses based on minimizing ‖𝐴𝑀−𝐼‖_𝐹) can be found in [7], where an iterative descent-type method is used to approximate each column of the inverse, and the iteration is done with “sparse matrix by sparse vector” operations. When the system matrix is expressed in block-partitioned form, some preconditioning options are explored in [8]. In [9], the idea of “target” matrix is introduced, in the context of sparse approximate inverse preconditioners, and the generalized Frobenius norms ‖𝐵‖²_𝐹,𝐻=tr(𝐵𝐻𝐵^𝑇) (𝐻 symmetric positive definite) are used, for minimization purposes, as an alternative to the classical Frobenius norm.

The last results of our work are devoted to the special case that matrix 𝐴𝑁 is symmetric and positive definite. In this sense, let us recall that the cone of symmetric and positive definite matrices has a rich geometrical structure and, in this context, the angle that any symmetric and positive definite matrix forms with the identity plays a very important role [10]. In this paper, the authors extend this geometrical point of view and analyze the geometrical structure of the subspace of symmetric matrices of order 𝑛, including the location of all orthogonal matrices not only the identity matrix.

This paper has been organized as follows. In Section 2, we present some preliminary results required to make the paper self-contained. Sections 3 and 4 are devoted to obtain new geometrical and spectral relations, respectively, for the orthogonal projections 𝐴𝑁 of the identity matrix. Finally, Section 5 closes the paper with its main conclusions.

2. Some Preliminaries

Now, we present some preliminary results concerning the orthogonal projection 𝐴𝑁 of the identity onto the matrix subspace 𝐴𝑆⊂ℝ^𝑛×𝑛. For more details about these results and for their proofs, we refer the reader to [5, 6, 11].

Taking advantage of the prehilbertian character of the matrix Frobenius norm, the solution 𝑁 to the problem (3) can be obtained using the orthogonal projection theorem. More precisely, the matrix product 𝐴𝑁 is the orthogonal projection of the identity onto the subspace 𝐴𝑆, and it satisfies the conditions stated by the following lemmas; see [5, 11].

Lemma 1. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, 0≤‖𝐴𝑁‖²_𝐹()‖‖=tr𝐴𝑁≤𝑛,0≤𝐴𝑁−𝐼²_𝐹=𝑛−tr(𝐴𝑁)≤𝑛. (4)(5)

An explicit formula for matrix 𝑁 can be obtained by expressing the orthogonal projection 𝐴𝑁 of the identity matrix onto the subspace 𝐴𝑆 by its expansion with respect to an orthonormal basis of 𝐴𝑆 [5]. This is the idea of the following lemma.

Lemma 2. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular. Let 𝑆 be a linear subspace of  ℝ^𝑛×𝑛 of dimension 𝑑 and {𝑀₁,…,𝑀_𝑑} a basis of  𝑆 such that {𝐴𝑀₁,…,𝐴𝑀_𝑑} is an orthogonal basis of 𝐴𝑆. Then, the solution 𝑁 to the problem (3) is 𝑁=^𝑑_𝑖=1tr𝐴𝑀_𝑖𝐴𝑀_𝑖²_𝐹𝑀_𝑖, (6) and the minimum (residual) Frobenius norm is ‖𝐴𝑁−𝐼‖²_𝐹=𝑛−^𝑑_𝑖=1tr𝐴𝑀_𝑖²𝐴𝑀_𝑖²_𝐹. (7)

Let us mention two possible options, both taken from [5], for choosing in practice the subspace 𝑆 and its corresponding basis {𝑀_𝑖}^𝑑_𝑖=1. The first example consists of considering the subspace 𝑆 of 𝑛×𝑛 matrices with a prescribed sparsity pattern, that is, 𝑆=𝑀∈ℝ^𝑛×𝑛∶𝑚_𝑖𝑗,{}×{}.=0∀𝑖,𝑗∉𝐾𝐾⊂1,2,…,𝑛1,2,…,𝑛 (8) Then, denoting by 𝑀_𝑖,𝑗, the 𝑛×𝑛 matrix whose only nonzero entry is 𝑚_𝑖𝑗=1, a basis of subspace 𝑆 is clearly {𝑀_𝑖,𝑗∶(𝑖,𝑗)∈𝐾}, and then {𝐴𝑀_𝑖,𝑗∶(𝑖,𝑗)∈𝐾} will be a basis of subspace 𝐴𝑆 (since we have assumed that matrix 𝐴 is nonsingular). In general, this basis of 𝐴𝑆 is not orthogonal, so that we only need to use the Gram-Schmidt procedure to obtain an orthogonal basis of 𝐴𝑆, in order to apply the orthogonal expansion (6).

For the second example, consider a linearly independent set of 𝑛×𝑛 real symmetric matrices {𝑃₁,…,𝑃_𝑑} and the corresponding subspace 𝑆^′=span𝑃₁𝐴^𝑇,…,𝑃_𝑑𝐴^𝑇, (9) which clearly satisfies 𝑆^′⊆𝑀=𝑃𝐴^𝑇∶𝑃∈ℝ^𝑛×𝑛𝑃^𝑇==𝑃𝑀∈ℝ^𝑛×𝑛∶(𝐴𝑀)^𝑇=𝐴𝑀. (10)

Hence, we can explicitly obtain the solution 𝑁 to the problem (3) for subspace 𝑆^′, from its basis {𝑃₁𝐴^𝑇,…,𝑃_𝑑𝐴^𝑇}, as follows. If {𝐴𝑃₁𝐴^𝑇,…,𝐴𝑃_𝑑𝐴^𝑇} is an orthogonal basis of subspace 𝐴𝑆^′, then we just use the orthogonal expansion (6) for obtaining 𝑁. Otherwise, we use again the Gram-Schmidt procedure to obtain an orthogonal basis of subspace 𝐴𝑆^′, and then we apply formula (6). The interest of this second example stands in the possibility of using the conjugate gradient method for solving the preconditioned linear system, when the symmetric matrix 𝐴𝑁 is positive definite. For a more detailed exposition of the computational aspects related to these two examples, we refer the reader to [5].

Now, we present some spectral properties of the orthogonal projection 𝐴𝑁. From now on, we denote by {𝜆_𝑖}^𝑛_𝑖=1 and {𝜎_𝑖}^𝑛_𝑖=1 the sets of eigenvalues and singular values, respectively, of matrix 𝐴𝑁 arranged, as usual, in nonincreasing order, that is, 𝜆₁≥𝜆₂≥⋯≥𝜆_𝑛𝜎>0,₁≥𝜎₂≥⋯≥𝜎_𝑛>0. (11)

The following lemma [11] provides some inequalities involving the eigenvalues and singular values of the preconditioned matrix 𝐴𝑁.

Lemma 3. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Then, ^𝑛_𝑖=1𝜆²_𝑖≤^𝑛_𝑖=1𝜆_𝑖²≤^𝑛_𝑖=1𝜎²_𝑖=‖𝐴𝑁‖²_𝐹=tr(𝐴𝑁)=^𝑛_𝑖=1𝜆_𝑖≤^𝑛_𝑖=1𝜆_𝑖≤^𝑛_𝑖=1𝜎_𝑖. (12)

The following fact [11] is a direct consequence of Lemma 3.

Lemma 4. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Then, the smallest singular value and the smallest eigenvalue’s modulus of the orthogonal projection 𝐴𝑁 of the identity onto the subspace 𝐴𝑆 are never greater than 1. That is, 0<𝜎_𝑛≤𝜆_𝑛≤1. (13)

The following theorem [11] establishes a tight connection between the closeness of matrix 𝐴𝑁 to the identity matrix and the closeness of 𝜎_𝑛(|𝜆_𝑛|) to the unity.

Theorem 5. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, 1−𝜆_𝑛²≤1−𝜎_𝑛²≤‖𝐴𝑁−𝐼‖²_𝐹≤𝑛1−𝜆_𝑛²≤𝑛1−𝜎²_𝑛. (14)

Remark 6. Theorem 5 states that the closer the smallest singular value 𝜎_𝑛 of matrix 𝐴𝑁 is to the unity, the closer matrix 𝐴𝑁 will be to the identity, that is, the smaller ‖𝐴𝑁−𝐼‖_𝐹 will be, and conversely. The same happens with the smallest eigenvalue’s modulus |𝜆_𝑛| of matrix 𝐴𝑁. In other words, we get a good approximate inverse 𝑁 of 𝐴 when 𝜎_𝑛(|𝜆_𝑛|) is sufficiently close to 1.

To finish this section, let us mention that, recently, lower and upper bounds on the normalized Frobenius condition number of the orthogonal projection 𝐴𝑁 of the identity onto the subspace 𝐴𝑆 have been derived in [12]. In addition, this work proposes a natural generalization (related to an arbitrary matrix subspace 𝑆 of ℝ^𝑛×𝑛) of the normalized Frobenius condition number of the nonsingular matrix 𝐴.

3. Geometrical Properties

In this section, we present some new geometrical properties for matrix 𝐴𝑁, 𝑁 being the optimal approximate inverse of matrix 𝐴, defined by (3). Our first lemma states some basic properties involving the cosine of the angle between matrix 𝐴𝑁 and the identity, that is, ()=⟨⟩cos𝐴𝑁,𝐼𝐴𝑁,𝐼_𝐹‖𝐴𝑁‖_𝐹‖𝐼‖_𝐹=tr(𝐴𝑁)‖𝐴𝑁‖_𝐹√𝑛. (15)

Lemma 7. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, ()=()cos𝐴𝑁,𝐼tr𝐴𝑁‖𝐴𝑁‖_𝐹√𝑛=‖𝐴𝑁‖_𝐹√𝑛=tr(𝐴𝑁)√𝑛,()‖‖0≤cos𝐴𝑁,𝐼≤1,𝐴𝑁−𝐼²_𝐹=𝑛1−cos²(𝐴𝑁,𝐼). (16)(17)(18)

Proof. First, using (15) and (4) we immediately obtain (16). As a direct consequence of (16), we derive that cos(𝐴𝑁,𝐼) is always nonnegative. Finally, using (5) and (16), we get ‖𝐴𝑁−𝐼‖²_𝐹=𝑛−tr(𝐴𝑁)=𝑛1−cos²(𝐴𝑁,𝐼) (19) and the proof is concluded.

Remark 8. In [13], the authors consider an arbitrary approximate inverse 𝑄 of matrix 𝐴 and derive the following equality: ‖𝐴𝑄−𝐼‖²_𝐹=‖𝐴𝑄‖_𝐹−‖𝐼‖_𝐹²+2(1−cos(𝐴𝑄,𝐼))‖𝐴𝑄‖_𝐹‖𝐼‖_𝐹, (20) that is, the typical decomposition (valid in any inner product space) of the strong convergence into the convergence of the norms (‖𝐴𝑄‖_𝐹−‖𝐼‖_𝐹)² and the weak convergence (1−cos(𝐴𝑄,𝐼))‖𝐴𝑄‖_𝐹‖𝐼‖_𝐹. Note that for the special case that 𝑄 is the optimal approximate inverse 𝑁 defined by (3), formula (18) has stated that the strong convergence is reduced just to the weak convergence and, indeed, just to the cosine cos(𝐴𝑁,𝐼).

Remark 9. More precisely, formula (18) states that the closer cos(𝐴𝑁,𝐼) is to the unity (i.e., the smaller the angle ∠(𝐴𝑁,𝐼) is), the smaller ‖𝐴𝑁−𝐼‖_𝐹 will be, and conversely. This gives us a new measure of the quality (in the Frobenius sense) of the approximate inverse 𝑁 of matrix 𝐴, by comparing the minimum residual norm ‖𝐴𝑁−𝐼‖_𝐹 with the cosine of the angle between 𝐴𝑁 and the identity, instead of with tr(𝐴𝑁), ‖𝐴𝑁‖_𝐹 (Lemma 1), or 𝜎_𝑛,  |𝜆_𝑛| (Theorem 5). So for a fixed nonsingular matrix 𝐴∈ℝ^𝑛×𝑛 and for different subspaces 𝑆⊂ℝ^𝑛×𝑛, we have tr(𝐴𝑁)↗𝑛⟺‖𝐴𝑁‖_𝐹↗√𝑛⟺𝜎_𝑛↗1⟺𝜆_𝑛()‖‖↗1⟺cos𝐴𝑁,𝐼↗1⟺𝐴𝑁−𝐼_𝐹↘0. (21) Obviously, the optimal theoretical situation corresponds to the case tr(𝐴𝑁)=𝑛⟺‖𝐴𝑁‖_𝐹=√𝑛⟺𝜎_𝑛=1⟺𝜆_𝑛()‖‖=1⟺cos𝐴𝑁,𝐼=1⟺𝐴𝑁−𝐼_𝐹=0⟺𝑁=𝐴⁻¹⟺𝐴⁻¹∈𝑆. (22)

Remark 10. Note that the ratio between cos(𝐴𝑁,𝐼) and cos(𝐴,𝐼) is independent of the order 𝑛 of matrix 𝐴. Indeed, assuming that tr(𝐴)≠0 and using (16), we immediately obtain cos(𝐴𝑁,𝐼)()=‖‖cos𝐴,𝐼𝐴𝑁_𝐹∶√𝑛tr(𝐴)∶‖𝐴‖_𝐹√𝑛=‖𝐴𝑁‖_𝐹‖𝐴‖_𝐹tr(𝐴). (23)

The following lemma compares the trace and the Frobenius norm of the orthogonal projection 𝐴𝑁.

Lemma 11. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of  ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, tr(𝐴𝑁)≤‖𝐴𝑁‖_𝐹⟺‖𝐴𝑁‖_𝐹()()≤1≤1⟺tr𝐴𝑁≤1⟺cos𝐴𝑁,𝐼√𝑛,tr(𝐴𝑁)≥‖𝐴𝑁‖_𝐹⟺‖𝐴𝑁‖_𝐹()()≥1≥1⟺tr𝐴𝑁≥1⟺cos𝐴𝑁,𝐼√𝑛. (24)(25)

Proof. Using (4), we immediately obtain the four leftmost equivalences. Using (16), we immediately obtain the two rightmost equivalences.

The next lemma provides us with a relationship between the Frobenius norms of the inverses of matrices 𝐴 and its best approximate inverse 𝑁 in subspace 𝑆.

Lemma 12. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, 𝐴⁻¹_𝐹𝑁⁻¹_𝐹≥1. (26)

Proof. Using (4), we get ‖𝐴𝑁‖_𝐹≤√𝑛=‖𝐼‖_𝐹=(𝐴𝑁)(𝐴𝑁)⁻¹_𝐹≤‖𝐴𝑁‖_𝐹(𝐴𝑁)⁻¹_𝐹⟹(𝐴𝑁)⁻¹_𝐹≥1, (27) and hence 𝐴⁻¹_𝐹𝑁⁻¹_𝐹≥𝑁⁻¹𝐴⁻¹_𝐹=(𝐴𝑁)⁻¹_𝐹≥1, (28) and the proof is concluded.

The following lemma compares the minimum residual norm ‖𝐴𝑁−𝐼‖_𝐹 with the distance (with respect to the Frobenius norm) ‖𝐴⁻¹−𝑁‖_𝐹 between the inverse of 𝐴 and the optimal approximate inverse 𝑁 of 𝐴 in any subspace 𝑆⊂ℝ^𝑛×𝑛. First, note that for any two matrices 𝐴,𝐵∈ℝ^𝑛×𝑛 (𝐴 nonsingular), from the submultiplicative property of the Frobenius norm, we immediately get ‖𝐴𝐵−𝐼‖²_𝐹=𝐴𝐵−𝐴⁻¹²_𝐹≤‖𝐴‖²_𝐹𝐵−𝐴⁻¹²_𝐹. (29)

However, for the special case that 𝐵=𝑁 (the solution to the problem (3)), we also get the following inequality.

Lemma 13. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, ‖𝐴𝑁−𝐼‖²_𝐹≤‖𝐴‖_𝐹𝑁−𝐴⁻¹_𝐹. (30)

Proof. Using the Cauchy-Schwarz inequality and (5), we get 𝐴⁻¹−𝑁,𝐴^𝑇_𝐹≤𝐴⁻¹−𝑁_𝐹𝐴^𝑇_𝐹⟹tr𝐴⁻¹−𝑁𝐴≤𝐴⁻¹−𝑁_𝐹𝐴^𝑇_𝐹⟹tr𝐴𝐴⁻¹−𝑁≤𝑁−A⁻¹_𝐹‖𝐴‖_𝐹⟹|tr(𝐼−𝐴𝑁)|≤𝑁−𝐴⁻¹_𝐹‖𝐴‖_𝐹⟹𝑛−tr(𝐴𝑁)≤𝑁−𝐴⁻¹_𝐹‖𝐴‖_𝐹⟹‖𝐴𝑁−𝐼‖²_𝐹≤‖𝐴‖_𝐹𝑁−𝐴⁻¹_𝐹. (31)

The following extension of the Cauchy-Schwarz inequality, in a real or complex inner product space (𝐻,⟨⋅,⋅⟩), was obtained by Buzano [14]. For all 𝑎,𝑥,𝑏∈𝐻, we have ⟩⋅⟨≤1|⟨𝑎,𝑥𝑥,𝑏⟩|2(‖𝑎‖‖𝑏‖+|⟨𝑎,𝑏⟩|)‖𝑥‖². (32)

The next lemma provides us with lower and upper bounds on the inner product ⟨𝐴𝑁,𝐵⟩_𝐹, for any 𝑛×𝑛 real matrix 𝐵.

Lemma 14. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, for every 𝐵∈ℝ^𝑛×𝑛, we have ⟨𝐴𝑁,𝐵⟩_𝐹≤12√𝑛‖𝐵‖_𝐹+|tr(𝐵)|. (33)

Proof. Using (32) for 𝑎=𝐼,  𝑥=𝐴𝑁,  𝑏=𝐵, and (4), we get ⟨𝐼,𝐴𝑁⟩_𝐹⋅⟨𝐴𝑁,𝐵⟩_𝐹≤12‖𝐼‖_𝐹‖𝐵‖_𝐹+⟨𝐼,𝐵⟩_𝐹‖𝐴𝑁‖²_𝐹⟹tr(𝐴𝑁)⋅⟨𝐴𝑁,𝐵⟩_𝐹≤12√𝑛‖𝐵‖_𝐹+|tr(𝐵)|‖𝐴𝑁‖²_𝐹⟹⟨𝐴𝑁,𝐵⟩_𝐹≤12√𝑛‖𝐵‖_𝐹+|tr(𝐵)|. (34)

The next lemma provides an upper bound on the arithmetic mean of the squares of the 𝑛² terms in the orthogonal projection 𝐴𝑁. By the way, it also provides us with an upper bound on the arithmetic mean of the 𝑛 diagonal terms in the orthogonal projection 𝐴𝑁. These upper bounds (valid for any matrix subspace 𝑆) are independent of the optimal approximate inverse 𝑁, and thus they are independent of the subspace 𝑆 and only depend on matrix 𝐴.

Lemma 15. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular with tr(𝐴)≠0 and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then, ‖𝐴𝑁‖²_𝐹𝑛²≤‖𝐴‖²_𝐹[tr(𝐴)]²,tr(𝐴𝑁)𝑛≤𝑛‖𝐴‖²_𝐹[tr(𝐴)]². (35)

Proof. Using (32) for 𝑎=𝐴^𝑇,  𝑥=𝐼,  and 𝑏=𝐴𝑁, and the Cauchy-Schwarz inequality for ⟨𝐴^𝑇,𝐴𝑁⟩_𝐹 and (4), we get 𝐴^𝑇,𝐼_𝐹⋅⟨𝐼,𝐴𝑁⟩_𝐹≤12𝐴^𝑇_𝐹‖𝐴𝑁‖_𝐹+𝐴^𝑇,𝐴𝑁_𝐹‖𝐼‖²_𝐹⟹|(𝐴)(≤𝑛tr⋅tr𝐴𝑁)|2‖𝐴‖_𝐹‖𝐴𝑁‖_𝐹+𝐴^𝑇,𝐴𝑁_𝐹≤𝑛2‖𝐴‖_𝐹‖𝐴𝑁‖_𝐹+𝐴^𝑇_𝐹‖𝐴𝑁‖_𝐹=𝑛‖𝐴‖_𝐹‖𝐴𝑁‖_𝐹⟹|tr(𝐴)|‖𝐴𝑁‖²_𝐹≤𝑛‖𝐴‖_𝐹‖𝐴𝑁‖_𝐹⟹‖𝐴𝑁‖_𝐹𝑛≤‖𝐴‖_𝐹|(𝐴⟹‖‖tr)|𝐴𝑁²_𝐹𝑛²≤‖𝐴‖²_𝐹[tr(𝐴)]²⟹tr(𝐴𝑁)𝑛≤𝑛‖𝐴‖²_𝐹[tr(𝐴)]². (36)

Remark 16. Lemma 15 has the following interpretation in terms of the quality of the optimal approximate inverse 𝑁 of matrix 𝐴 in subspace 𝑆. The closer the ratio 𝑛‖𝐴‖_𝐹/|tr(𝐴)| is to zero, the smaller tr(𝐴𝑁) will be, and thus, due to (5), the larger ‖𝐴𝑁−𝐼‖_𝐹 will be, and this happens for any matrix subspace 𝑆.

Remark 17. By the way, from Lemma 15, we obtain the following inequality for any nonsingular matrix 𝐴∈ℝ^𝑛×𝑛. Consider any matrix subspace 𝑆 s.t. 𝐴⁻¹∈𝑆. Then, 𝑁=𝐴⁻¹, and using Lemma 15, we get ‖𝐴𝑁‖²_𝐹𝑛²=‖𝐼‖²_𝐹𝑛²=1𝑛≤‖𝐴‖²_𝐹[tr(𝐴)]²⟹|tr(𝐴)|≤√𝑛‖𝐴‖_𝐹. (37)

4. Spectral Properties

In this section, we present some new spectral properties for matrix 𝐴𝑁, 𝑁 being the optimal approximate inverse of matrix 𝐴, defined by (3). Mainly, we focus on the case that matrix 𝐴𝑁 is symmetric and positive definite. This has been motivated by the following reason. When solving a large nonsymmetric linear system (1) by using Krylov methods, a possible strategy consists of searching for an adequate optimal preconditioner 𝑁 such that the preconditioned matrix 𝐴𝑁 is symmetric positive definite [5]. This enables one to use the conjugate gradient method (CG-method), which is, in general, a computationally efficient method for solving the new preconditioned system [2, 15].

Our starting point is Lemma 3, which has established that the sets of eigenvalues and singular values of any orthogonal projection 𝐴𝑁 satisfy ^𝑛_𝑖=1𝜆²_𝑖≤^𝑛_𝑖=1𝜆_𝑖²≤^𝑛_𝑖=1𝜎²_𝑖=^𝑛_𝑖=1𝜆_𝑖≤^𝑛_𝑖=1𝜆_𝑖≤^𝑛_𝑖=1𝜎_𝑖. (38)

Let us particularize (38) for some special cases.

First, note that if 𝐴𝑁 is normal (i.e., for all 1≤𝑖≤𝑛: 𝜎_𝑖=|𝜆_𝑖| [16]), then (38) becomes ^𝑛_𝑖=1𝜆²_𝑖≤^𝑛_𝑖=1𝜆_𝑖²=^𝑛_𝑖=1𝜎²_𝑖=^𝑛_𝑖=1𝜆_𝑖≤^𝑛_𝑖=1𝜆_𝑖=^𝑛_𝑖=1𝜎_𝑖. (39) In particular, if 𝐴𝑁 is symmetric (𝜎_𝑖=|𝜆_𝑖|=±𝜆_𝑖∈ℝ), then (38) becomes ^𝑛_𝑖=1𝜆²_𝑖=^𝑛_𝑖=1𝜆_𝑖²=^𝑛_𝑖=1𝜎²_𝑖=^𝑛_𝑖=1𝜆_𝑖≤^𝑛_𝑖=1𝜆_𝑖=^𝑛_𝑖=1𝜎_𝑖. (40) In particular, if 𝐴𝑁 is symmetric and positive definite (𝜎_𝑖=|𝜆_𝑖|=𝜆_𝑖∈ℝ⁺), then the equality holds in all (38), that is, ^𝑛_𝑖=1𝜆²_𝑖=^𝑛_𝑖=1𝜆_𝑖²=^𝑛_𝑖=1𝜎²_𝑖=^𝑛_𝑖=1𝜆_𝑖=^𝑛_𝑖=1𝜆_𝑖=^𝑛_𝑖=1𝜎_𝑖. (41)

The next lemma compares the traces of matrices 𝐴𝑁 and (𝐴𝑁)².

Lemma 18. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of  ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Then,(i) for any orthogonal projection 𝐴𝑁tr(𝐴𝑁)²≤‖𝐴𝑁‖²_𝐹=tr(𝐴𝑁), (42)(ii) for any symmetric orthogonal projection 𝐴𝑁(𝐴𝑁)²_𝐹≤tr(𝐴𝑁)²=‖𝐴𝑁‖²_𝐹=tr(𝐴𝑁), (43)(iii)for any symmetric positive definite orthogonal projection 𝐴𝑁(𝐴𝑁)²_𝐹≤tr(𝐴𝑁)²=‖𝐴𝑁‖²_𝐹=tr(𝐴𝑁)≤[tr(𝐴𝑁)]². (44)

Proof. (i) Using (38), we get ^𝑛_𝑖=1𝜆²_𝑖≤^𝑛_𝑖=1𝜎²_𝑖=^𝑛_𝑖=1𝜆_𝑖. (45)
(ii) It suffices to use the obvious fact that ‖(𝐴𝑁)²‖_𝐹≤‖𝐴𝑁‖²_𝐹 and the following equalities taken from (40): ^𝑛_𝑖=1𝜆²_𝑖=^𝑛_𝑖=1𝜎²_𝑖=^𝑛_𝑖=1𝜆_𝑖. (46)
(iii) It suffices to use (43) and the fact that (see, e.g., [17, 18]) if 𝑃 and 𝑄 are symmetric positive definite matrices then tr(𝑃𝑄)≤tr(𝑃)tr(𝑄) for 𝑃=𝑄=𝐴𝑁.

The rest of the paper is devoted to obtain new properties about the eigenvalues of the orthogonal projection 𝐴𝑁 for the special case that this matrix is symmetric positive definite.

First, let us recall that the smallest singular value and the smallest eigenvalue’s modulus of the orthogonal projection 𝐴𝑁 are never greater than 1 (see Lemma 4). The following theorem establishes the dual result for the largest eigenvalue of matrix 𝐴𝑁 (symmetric positive definite).

Theorem 19. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Suppose that matrix 𝐴𝑁 is symmetric and positive definite. Then, the largest eigenvalue of the orthogonal projection 𝐴𝑁 of the identity onto the subspace 𝐴𝑆 is never less than 1. That is, 𝜎₁=𝜆₁≥1. (47)

Proof. Using (41), we get ^𝑛_𝑖=1𝜆²_𝑖=^𝑛_𝑖=1𝜆_𝑖⟹𝜆²_𝑛−𝜆_𝑛=^𝑛−1_𝑖=1𝜆_𝑖−𝜆²_𝑖. (48) Now, since 𝜆_𝑛≤1 (Lemma 4), then 𝜆²_𝑛−𝜆_𝑛≤0. This implies that at least one summand in the rightmost sum in (48) must be less than or equal to zero. Suppose that such summand is the 𝑘th one (1≤𝑘≤𝑛−1). Since 𝐴𝑁 is positive definite, then 𝜆_𝑘>0, and thus 𝜆_𝑘−𝜆²_𝑘≤0⟹𝜆_𝑘≤𝜆²_𝑘⟹𝜆_𝑘≥1⟹𝜆₁≥1 (49) and the proof is concluded.

In Theorem 19, the assumption that matrix 𝐴𝑁 is positive definite is essential for assuring that |𝜆₁|≥1, as the following simple counterexample shows. Moreover, from Lemma 4 and Theorem 19, respectively, we have that the smallest and largest eigenvalues of 𝐴𝑁 (symmetric positive definite) satisfy 𝜆_𝑛≤1 and 𝜆₁≥1, respectively. Nothing can be asserted about the remaining eigenvalues of the symmetric positive definite matrix 𝐴𝑁, which can be greater than, equal to, or less than the unity, as the same counterexample also shows.

Example 20. For 𝑛=3, let 𝐴_𝑘=3000𝑘0001,𝑘∈ℝ, (50) let 𝐼₃ be identity matrix of order 3, and let 𝑆 be the subspace of all 3×3 scalar matrices; that is, 𝑆=span{𝐼₃}. Then the solution 𝑁_𝑘 to the problem (3) for subspace 𝑆 can be immediately obtained by using formula (6) as follows: 𝑁_𝑘=tr𝐴_𝑘𝐴_𝑘²_𝐹𝐼₃=𝑘+4𝑘²𝐼+10³ (51) and then we get 𝐴_𝑘𝑁_𝑘=𝑘+4𝑘²𝐴+10^𝑘=𝑘+4𝑘²+103000𝑘0001. (52)

Let us arrange the eigenvalues and singular values of matrix 𝐴_𝑘𝑁_𝑘, as usual, in nonincreasing order (as shown in (11)).

On one hand, for 𝑘=−2, we have 𝐴₋₂𝑁₋₂=173000−20001, (53) and then 𝜎₁=𝜆₁=37≥𝜎₂=𝜆₂=27≥𝜎₃=𝜆₃=17. (54) Hence, 𝐴₋₂𝑁₋₂ is indefinite and 𝜎₁=|𝜆₁|=3/7<1.

On the other hand, for 1<𝑘<3, we have (see matrix (52)) 𝜎₁=𝜆₁=3𝑘+4𝑘²+10≥𝜎₂=𝜆₂=𝑘𝑘+4𝑘²+10≥𝜎₃=𝜆₃=𝑘+4𝑘²,+10 (55) and then 𝑘=2:𝜎₁=𝜆₁=97𝜎>1,₂=𝜆₂=67𝜎<1,₃=𝜆₃=375<1,𝑘=2:𝜎₁=𝜆₁=65𝜎>1,₂=𝜆₂𝜎=1,₃=𝜆₃=258<1,𝑘=3:𝜎₁=𝜆₁=90𝜎77>1,₂=𝜆₂=80𝜎77>1,₃=𝜆₃=3077<1. (56) Hence, for 𝐴_𝑘𝑁_𝑘 positive definite, we have (depending on 𝑘) 𝜆₂<1,  𝜆₂=1, or 𝜆₂>1.

The following corollary improves the lower bound zero on both tr(𝐴𝑁), given in (4), and cos(𝐴𝑁,𝐼), given in (17).

Corollary 21. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of  ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Suppose that matrix 𝐴𝑁 is symmetric and positive definite. Then, 1≤‖𝐴𝑁‖_𝐹≤tr(𝐴𝑁)=‖𝐴𝑁‖²_𝐹()≥1≤𝑛,cos𝐴𝑁,𝐼√𝑛. (57)(58)

Proof. Denote by ‖⋅‖₂ the spectral norm. Using the well-known inequality ‖⋅‖₂≤‖⋅‖_𝐹 [19], Theorem 19, and (4), we get ‖𝐴𝑁‖_𝐹≥‖𝐴𝑁‖₂=𝜎₁=𝜆₁‖‖≥1⟹1≤𝐴𝑁_𝐹≤tr(𝐴𝑁)=‖𝐴𝑁‖²_𝐹≤𝑛. (59) Finally, (58) follows immediately from (57) and (25).

Let us mention that an upper bound on all the eigenvalues moduli and on all singular values of any orthogonal projection 𝐴𝑁 can be immediately obtained from (38) and (4) as follows: ^𝑛_𝑖=1𝜆_𝑖²≤^𝑛_𝑖=1𝜎²_𝑖=‖𝐴𝑁‖²_𝐹⟹𝜆≤𝑛_𝑖,𝜎_𝑖≤√𝑛,∀𝑖=1,2,…,𝑛. (60)

Our last theorem improves the upper bound given in (60) for the special case that the orthogonal projection 𝐴𝑁 is symmetric positive definite.

Theorem 22. Let 𝐴∈ℝ^𝑛×𝑛 be nonsingular and let 𝑆 be a linear subspace of  ℝ^𝑛×𝑛. Let 𝑁 be the solution to the problem (3). Suppose that matrix 𝐴𝑁 is symmetric and positive definite. Then, all the eigenvalues of matrix 𝐴𝑁 satisfy 𝜎_𝑖=𝜆_𝑖≤1+√𝑛2∀𝑖=1,2,…,𝑛. (61)

Proof. First, note that the assertion is obvious for the smallest singular value since |𝜆_𝑛|≤1 for any orthogonal projection 𝐴𝑁 (Lemma 4). For any eigenvalue of 𝐴𝑁, we use the fact that 𝑥−𝑥²≤1/4 for all 𝑥>0. Then from (41), we get ^𝑛_𝑖=1𝜆²_𝑖=^𝑛_𝑖=1𝜆_𝑖⟹𝜆²₁−𝜆₁=^𝑛_𝑖=2𝜆_𝑖−𝜆²_𝑖≤𝑛−14⟹𝜆₁≤1+√𝑛2⟹𝜆_𝑖≤1+√𝑛2∀𝑖=1,2,…,𝑛. (62)

5. Conclusion

In this paper, we have considered the orthogonal projection 𝐴𝑁 (in the Frobenius sense) of the identity matrix onto an arbitrary matrix subspace 𝐴𝑆 (𝐴∈ℝ^𝑛×𝑛 nonsingular, 𝑆⊂ℝ^𝑛×𝑛). Among other geometrical properties of matrix 𝐴𝑁, we have established a strong relation between the quality of the approximation 𝐴𝑁≈𝐼 and the cosine of the angle ∠(𝐴𝑁,𝐼). Also, the distance between 𝐴𝑁 and the identity has been related to the ratio 𝑛‖𝐴‖_𝐹/|tr(𝐴)| (which is independent of the subspace 𝑆). The spectral analysis has provided lower and upper bounds on the largest eigenvalue of the symmetric positive definite orthogonal projections of the identity.

Acknowledgments

The authors are grateful to the anonymous referee for valuable comments and suggestions, which have improved the earlier version of this paper. This work was partially supported by the “Ministerio de Economía y Competitividad” (Spanish Government), and FEDER, through Grant Contract CGL2011-29396-C03-01.