A Representation of Nonhomogeneous Quadratic Forms with Application to the Least Squares Solution

Abstract

The least squares problem appears, among others, in linear models, and it refers to inconsistent system of linear equations. A crucial question is how to reduce the least squares solution in such a system to the usual solution in a consistent one. Traditionally, this is reached by differential calculus. We present a purely algebraic approach to this problem based on some identities for nonhomogeneous quadratic forms.

1. Introduction and Notation

The least squares problem appears, among others, in linear models, and it refers to inconsistent system 𝐀𝐱=𝐛 of linear equations. Formally, it reduces to minimizing the nonhomogeneous quadratic form 𝑓(𝐱)=(𝐀𝐱−𝐛)^𝑇(𝐀𝐱−𝐛). (1.1) Classical approach to the problem, presented in such known books as Scheffé ([1, Chapter 1]), Rao ([2, pages 222 and 223]), Rao and Toutenburg ([3, pages 20–23]), uses differential calculus and leads to the so called normal equation 𝐀^𝑇𝐀𝐱=𝐀^𝑇𝐛, which is consistent. The aim of this note is to present some useful algebraic identities for nonhomogeneous quadratic forms leading directly to normal equation.

Traditional vector-matrix notation will be used. Among others, if 𝐌 is a matrix then 𝐌^𝑇, ℛ(𝐌), and 𝑟(𝐌) stand for its transposition, range (column space), and rank. Moreover, by 𝑅^𝑛 will be denoted the 𝑛-dimensional euclidean space represented by column vectors.

2. Background

Any system of linear equations may be presented in the vector-matrix form as𝐀𝐱=𝐛, (2.1) where 𝐀 is a given 𝑛×𝑝 matrix, 𝐛 is a given vector in 𝑅^𝑛, while 𝐱∈𝑅^𝑝 is unknown vector. It is well known that (2.1) is consistent, if and only if, 𝐛 belongs to the range ℛ(𝐀).

If (2.1) is inconsistent, one can seek for a vector 𝐱 minimizing the norm ‖𝐀𝐱−𝐛‖ or, equivalently, its square (𝐀𝐱−𝐛)^𝑇(𝐀𝐱−𝐛). The Least Squares Solution (LSS) of (2.1) is defined as a vector 𝐱₀∈𝑅^𝑝 such that𝐀𝐱₀−𝐛^𝑇𝐀𝐱₀−𝐛≤(𝐀𝐱−𝐛)^𝑇(𝐀𝐱−𝐛)∀𝐱∈𝑅^𝑝. (2.2)

A crucial problem is how to reduce the LSS of the inconsistent equation (2.1) to the usual solution of a consistent one. Formally, the least squares problem deals with minimizing the nonhomogeneous quadratic form 𝑓(𝐱)=(𝐀𝐱−𝐛)^𝑇(𝐀𝐱−𝐛). Traditionally, this problem is solved by differential calculus and leads to the normal equation 𝐀^𝑇𝐀𝐱=𝐀^𝑇𝐛.

In the next section, we will present some useful algebraic identities for nonhomogeneous quadratic forms. They yield directly the inequality (2.2).

3. Identities and Inequalities for Nonhomogeneous Quadratic Forms

The usual, that is homogeneous quadratic form is a real function 𝑓_𝑀(𝐱)=𝐱^𝑇𝐌𝐱 defined on 𝑅^𝑝. In this note, we shall consider also nonhomogeneous quadratic forms of type𝑓_𝑀,𝑎(𝐱)=(𝐱−𝐚)^𝑇𝐌(𝐱−𝐚), (3.1) where 𝐌 is a symmetric 𝑝×𝑝 matrix and 𝐚 is a vector in 𝑅^𝑝.

Some inequalities for nonhomogeneous quadratic forms may be found in Stępniak [4]. Let us recall one of these results, which is very useful in the nonhomogeneous linear estimation.

Lemma 3.1. For any symmetric nonnegative definite matrices 𝐌₁ and 𝐌₂ of order 𝑝, the condition 𝐱−𝐱₁^𝑇𝐌₁𝐱−𝐱₁+𝑐₁≥𝐱−𝐱₂^𝑇𝐌₂𝐱−𝐱₂+𝑐₂ (3.2) for some 𝑐₁,𝑐₂∈𝑅, 𝐱₁,𝐱₂∈𝑅^𝑝 and all 𝐱∈𝑅^𝑝 implies that 𝐌₁−𝐌₂ is nonnegative definite and 𝑐₁−𝑐₂≥0.

Now we will present some identity which may serve as a convenient tool in the LSS of (2.1). For convenience, we will start from the case 𝑟(𝐀)=𝑝, leaving the singular case 𝑟(𝐀)<𝑝 to Section 5.

Proposition 3.2. For arbitrary 𝑛×𝑝 matrix 𝐀 of rank 𝑝 and arbitrary vector𝐛∈𝑅^𝑛(𝐀𝐱−𝐛)^𝑇(𝐀𝐱−𝐛)=𝐀^𝑇𝐀𝐱−𝐀^𝑇𝐛^𝑇𝐀^𝑇𝐀⁻¹𝐀^𝑇𝐀𝐱−𝐀^𝑇𝐛+𝐛^𝑇𝐈_𝑛−𝐀𝐀^𝑇𝐀⁻¹𝐀^𝑇𝐛. (3.3)

Proof. Let us start from the trivial identity 𝐈_𝑛=𝐀𝐀^𝑇𝐀⁻¹𝐀^𝑇+𝐈_𝑛−𝐀𝐀^𝑇𝐀⁻¹𝐀. (3.4)
We only need to premultiply this identity by (𝐀𝐱−𝐛)^𝑇, postmultiply it by (𝐀𝐱−𝐛), and then collect the terms to get (3.3).

4. Least Squares and Usual Solutions: Nonsingular Case

As above, we consider an inconsistent equation 𝐀𝐱=𝐛, where 𝐀 is an 𝑛×𝑝 matrix of rank 𝑝. We are interested in the LSS of this equation.

Theorem 4.1. Vector 𝐱∈𝑅^𝑝 is a Least Squares Solution of the inconsistent equation 𝐀𝐱=𝐛, if and only if, it is usual solution of the consistent equation 𝐀^𝑇𝐀𝐱=𝐀^𝑇𝐛. (4.1)

Proof. Consistency of (4.1) follows from the fact that ℛ(𝐀^𝑇𝐀)=ℛ(𝐀^𝑇).
We note that the second component in the right side of the identity (3.3) does not depend on 𝐱. Thus, we only need to minimize the first one. Since 𝐀^𝑇𝐀, and in consequence (𝐀^𝑇𝐀)⁻¹, is positive definite, this component is minimal, if and only if, 𝐀^𝑇𝐀𝐱−𝐀^𝑡𝐛=𝟎. This completes the proof.

5. General Case

If 𝑟(𝐀)<𝑝 then the matrix 𝐀^𝑇𝐀 is singular, and, therefore, the identity (3.3) is not applicable. However, as we will show, it remains true if we replace (𝐀^𝑇𝐀)⁻¹ by arbitrary generalized inverse (𝐀^𝑇𝐀)⁻.

There are many papers on generalized inverses and several books; among others Bapat [5] Ben-Israel and Greville [6], Campbell and Meyer [7], Pringle and Rayner [8], and Rao and Mitra [9]. A recent paper by Stępniak [10] may serve as a brief and self-contained introduction to this field.

Let us recall that a given 𝑛×𝑝 matrix 𝐀, its generalized inverse 𝐀⁻ is defined as an arbitrary 𝑝×𝑛 matrix 𝐆 satisfying the condition 𝐀𝐆𝐀=𝐀.

A key result in this section is stated as follows.

Proposition 5.1. For arbitrary 𝑛×𝑝 matrix 𝐀 and arbitrary vector 𝐛∈𝑅^𝑛(𝐀𝐱−𝐛)^𝑇(𝐀𝐱−𝐛)=𝐀^𝑇𝐀𝐱−𝐀^𝑇𝐛^𝑇𝐀^𝑇𝐀⁻𝐀^𝑇𝐀𝐱−𝐀^𝑇𝐛+𝐛^𝑇𝐈_𝑛−𝐀𝐀^𝑇𝐀⁻𝐀^𝑇𝐛, (5.1) where − means arbitrary generalized inverse.

Proof. The idea of the proof is the same as in Proposition 3.2. Since ℛ(𝐀^𝑇𝐀)=ℛ(𝐀^𝑇) one can replace the vector 𝐀^𝑇𝐛 by 𝐀^𝑇𝐀𝐜 for some 𝐜.

Now we will apply the identity (5.1) to the least squares problem.

Theorem 5.2. For arbitrary 𝑛×𝑝 matrix 𝐀 and arbitrary vector 𝐛∈𝑅^𝑛,(i)a vector 𝐱∈𝑅^𝑝 is a Least Squares Solution of the equation 𝐀𝐱=𝐛, if and only if, it is the usual solution of the (consistent) equation 𝐀^𝑇𝐀𝐱=𝐀^𝑇𝐛;(ii)the lower bound of (𝐀𝐱−𝐛)^𝑇(𝐀𝐱−𝐛) is equal to 𝐛^𝑇[𝐈_𝑛−𝐀(𝐀^𝑇𝐀)⁻𝐀^𝑇]𝐛, and it does not depend on choice of generalized inverse (𝐀^𝑇𝐀)⁻.

Proof. By setting 𝐀^𝑇𝐛=𝐀^𝑇𝐀𝐜, the first component in the right side of (5.1) reduces to (𝐱−𝐜)^𝑇(𝐀^𝑇𝐀)(𝐱−𝐜) which is nonnegative and takes zero, if and only if, (𝐀^𝑇𝐀)(𝐱−𝐜)=𝟎 or, equivalently, if 𝐀^𝑇𝐀𝐱=𝐀^𝑇𝐛. Since the second component does not depend on 𝐱, this is just total minimum. The same setting shows that the lower bound does not depend on the choice of generalized inverse.

Acknowledgment

The author thanks a referee for his (or her) useful suggestions concerning presentation of this paper.