Comparison Results on Preconditioned GAOR Methods for Weighted Linear Least Squares Problems

Abstract

We present preconditioned generalized accelerated overrelaxation methods for solving weighted linear least square problems. We compare the spectral radii of the iteration matrices of the preconditioned and the original methods. The comparison results show that the preconditioned GAOR methods converge faster than the GAOR method whenever the GAOR method is convergent. Finally, we give a numerical example to confirm our theoretical results.

1. Introduction

Consider the weighted linear least squares problem min_{𝑥∈𝑅𝑛}(𝐴𝑥−𝑏)^𝑇𝑊⁻¹(𝐴𝑥−𝑏), (1.1) where 𝑊 is the variance-covariance matrix. The problem has many scientific applications. A typical source is parameter estimation in mathematical modeling.

This problem has been discussed in many books and articles. In order to solve it, one has to solve a nonsingular linear system as 𝐻𝑦=𝑓, (1.2) where 𝐻=𝐼−𝐵₁𝑈𝐶𝐼−𝐵₂ (1.3) is an invertible matrix with 𝐵₁=𝑏_{𝑖𝑗𝑝×𝑝},𝐵₂=𝑏_{𝑖𝑗(𝑛−𝑝)×(𝑛−𝑝)},𝐶=𝑐_{𝑖𝑗(𝑛−𝑝)×𝑝},𝑈=𝑢_{𝑖𝑗𝑝×(𝑛−𝑝)}. (1.4)

Yuan proposed a generalized SOR (GSOR) method to solve linear system (1) in [1]; afterwards, Yuan and Jin [2] established a generalized AOR (GAOR) method to solve linear system (1). In [3, 4], authors studied the convergence of the GAOR method for solving the linear system 𝐻𝑦=𝑓. In [3], authors studied the convergence of the GAOR method when the coefficient matrices are consistently ordered matrices and gave the regions of convergence. In [4], authors studied the convergence of the GAOR method for diagonally dominant coefficient matrices and gave the regions of convergence.

In order to solve the linear system (1.2) using the GAOR method, we split 𝐻 as −𝐵𝐻=𝐼−00−𝐶0¹−𝑈0𝐵₂. (1.5)

Then, for 𝜔≠0, one GAOR method can be defined by 𝑦^(𝑘+1)=𝐿_𝑟,𝜔𝑦^(𝑘)+𝜔𝑔,𝑘=0,1,2,…, (1.6) where 𝐿_𝑟,𝜔=𝐼0𝑟𝐶𝐼⁻¹()()𝐵1−𝜔𝐼+𝜔−𝑟00−𝐶0+𝜔¹−𝑈0𝐵₂=(1−𝜔)𝐼+𝜔𝐵₁𝜔()−𝜔𝑈𝑟−1𝐶−𝜔𝑟𝐶𝐵₁(1−𝜔)𝐼+𝜔𝐵₂+𝜔𝑟𝐶𝑈 (1.7) is the iteration matrix and 𝑔=𝐼0−𝑟𝐶𝐼𝑓. (1.8)

In order to decrease the spectral radius of 𝐿_𝑟,𝜔, an effective method is to precondition the linear system (1.2), namely, 𝑃𝐻=𝐼−𝐵^∗₁𝑈^∗𝐶^∗𝐼−𝐵^∗₂, (1.9) then the preconditioned GAOR method can be defined by 𝑦^(𝑘+1)=𝐿^∗_𝑟,𝜔𝑦^(𝑘)+𝜔𝑔^∗,𝑘=0,1,2,…, (1.10) where 𝐿^∗_𝑟,𝜔=(1−𝜔)𝐼+𝜔𝐵^∗₁−𝜔𝑈^∗𝜔(1−𝑟)𝐶^∗−𝜔𝑟𝐶^∗𝐵^∗₁(1−𝜔)𝐼+𝜔𝐵^∗₂+𝜔𝑟𝐶^∗𝑈^∗,𝑔^∗=𝐼0−𝑟𝐶^∗𝐼𝑃𝑓. (1.11)

In [5], authors presented three kinds of preconditioners for preconditioned modified accelerated overrelaxation method to solve systems of linear equations. They showed that the convergence rate of the preconditioned modified accelerated overrelaxation method is better than that of the original method, whenever the original method is convergent.

This paper is organized as follows. In Section 2, we give some important definition and the known results as the preliminaries of the paper. In Section 3, we propose three preconditioners and give the comparison theorems between the preconditioned and original methods. These results show that the preconditioned GAOR methods converge faster than the GAOR method whenever the GAOR method is convergent. In Section 4, we give an example to confirm our theoretical results.

2. Preliminaries

We need the following definition and results.

Definition 2.1. Let 𝐴=(𝑎_𝑖𝑗)_𝑛×𝑛 and 𝐵=(𝑏_𝑖𝑗)_𝑛×𝑛. We say 𝐴≥𝐵 if 𝑎_𝑖𝑗≥𝑏_𝑖𝑗 for all 𝑖,𝑗=1,2,…,𝑛.
This definition can be carried over to vectors by identifying them with 𝑛×1 matrices.
In this paper, 𝜌(⋅) denotes the spectral radius of a matrix.

Lemma 2.2 (see [6]). Let 𝐴∈𝑅^𝑛×𝑛 be nonnegative and irreducible. Then(i)𝐴has a positive real eigenvalue equal to its spectral radius 𝜌(𝐴);(ii)for 𝜌(𝐴), there corresponds an eigenvector 𝑥>0.

Lemma 2.3 (see [7]). Let 𝐴∈𝑅^𝑛×𝑛 be nonnegative and irreducible. If 0≠𝛼𝑥≤𝐴𝑥≤𝛽𝑥,𝛼𝑥≠𝐴𝑥,𝐴𝑥≠𝛽𝑥, (2.1) for some nonnegative vector 𝑥, then 𝛼<𝜌(𝐴)<𝛽 and 𝑥 is a positive vector.

3. Comparison Results

We consider the preconditioned linear system 𝐻𝑦=𝑓, (3.1) where 𝐻=(𝐼+𝑆)𝐻 and 𝑓=(𝐼+𝑆)𝑓 with 𝑆=𝑆000, (3.2)𝑆 is a 𝑝×𝑝 matrix with 1<𝑝<𝑛.

We take 𝑆 as follows: 𝑆₁=0𝛼₂𝑏₁₂𝛽⋯00₂𝑏₂₁0⋱00⋮⋱⋮⋱⋮00⋱0𝛼_𝑝𝑏_{𝑝−1,𝑝}00⋯𝛽_𝑝𝑏_{𝑝,𝑝−1}0,𝑆₂=0𝛼₂𝑏₁₂⋯𝛼_𝑝𝑏_1𝑝𝛽₂𝑏₂₁𝛽0⋯0⋮⋮⋱⋮_𝑝𝑏_𝑝10⋯0. (3.3)

Now, we obtain two preconditioned linear systems with coefficient matrices 𝐻_𝑖=𝐼−𝐵₁−𝑆_𝑖𝐼−𝐵₁𝐼+𝑆_𝑖𝑈𝐶𝐼−𝐵₂,for𝑖=1,2, (3.4) where 𝐵₁−𝑆₁𝐼−𝐵₁=𝑏₁₁+𝛼₂𝑏₂₁𝑏₁₂⋯𝑏_1𝑝+𝛼₂𝑏_2𝑝𝑏₁₂𝑏₂₁+𝛼₃𝑏₂₃𝑏₃₁−𝛽₂𝑏₂₁1−𝑏₁₁⋯𝑏_2𝑝+𝛽₂𝑏_1𝑝𝑏₂₁+𝛼₃𝑏_3𝑝𝑏₂₃𝑏⋮⋮⋮_𝑝−1,1+𝛽_𝑝−1𝑏_{𝑝−1,𝑝−2}𝑏_𝑝−2,1+𝛼_𝑝𝑏_{𝑝−1,𝑝}𝑏_𝑝1⋯𝑏_{𝑝−1,𝑝}+𝛽_𝑝−1𝑏_{𝑝−1,𝑝−2}𝑏_{𝑝−2,𝑝}+𝛼_𝑝𝑏_{𝑝−2,𝑝}𝑏_{𝑝−1,𝑝−2}𝑏_𝑝1+𝛽_𝑝𝑏_𝑝−1,1𝑏_{𝑝,𝑝−1}⋯𝑏_𝑝𝑝+𝛽_𝑝𝑏_{𝑝,𝑝−1}𝑏_{𝑝−1,𝑝},𝐵₁−𝑆₂𝐼−𝐵₁=𝑏₁₁+𝛼₂𝑏₁₂𝑏₂₁+⋯+𝛼_𝑝𝑏_1𝑝𝑏_𝑝1⋯𝑏_1𝑝+𝛼₂𝑏₁₂𝑏_2𝑝+⋯+𝛼_𝑝𝑏_1𝑝1−𝑏_𝑝𝑝𝑏₂₁−𝛽₂𝑏₂₁1−𝑏₁₁⋯𝑏_2𝑝+𝛽₂𝑏₂₁𝑏_1𝑝𝑏⋮⋱⋮_𝑝1−𝛽_𝑝𝑏_𝑝11−𝑏₁₁⋯𝑏_𝑝𝑝+𝛽_𝑝𝑏_𝑝1𝑏_1𝑝. (3.5)

We split 𝐻_𝑖 (𝑖=1,2) as 𝐻_𝑖−𝐵=𝐼−00−𝐶0¹−𝑆_𝑖𝐼−𝐵₁−𝐼+𝑆_𝑖𝑈0𝐵₂,for𝑖=1,2, (3.6) then the preconditioned GAOR methods for solving (3.1) are defined as follows 𝑦^(𝑘+1)=𝐿^(𝑖)_𝑟,𝜔𝑦^(𝑘)+𝜔𝑔,𝑘=0,1,2,…, (3.7) where 𝐿^(𝑖)_𝑟,𝜔=(1−𝜔)𝐼+𝜔𝐵₁−𝑆_𝑖𝐼−𝐵₁−𝜔𝐼+𝑆_𝑖𝑈𝜔(𝑟−1)𝐶−𝜔𝑟𝐶𝐵₁−𝑆_𝑖𝐼−𝐵₁(1−𝜔)𝐼+𝜔𝐵₂+𝜔𝑟𝐶𝐼+𝑆_𝑖𝑈 (3.8) are iteration matrices and 𝑔=𝐼0−𝑟𝐶𝐼𝑓. (3.9)

Now, we give comparison results between the preconditioned GAOR methods defined by (3.7) and the corresponding GAOR method defined by (1.6).

Theorem 3.1. Let 𝐿_𝑟,𝜔,𝐿⁽¹⁾_𝑟,𝜔 be the iteration matrices associated with the GAOR and preconditioned GAOR methods, respectively. If the matrix 𝐻 in (1.2) is irreducible with 𝐶≤0, 𝑈≤0, 𝐵₁≥0, 𝐵₂≥0, 0<𝜔≤1, 0≤𝑟<1, 𝑏_𝑖,𝑖+1>0, 𝑏_𝑖+1,𝑖>0 for some 𝑖∈{2,…,𝑝}, when 0≤𝑏_𝑖𝑖<1(𝑖∈{2,…,𝑝}), 0<𝛼_𝑖<𝑏_{𝑖−1,𝑖−2}𝑏_{𝑖−2,𝑖}+𝑏_{𝑖−1,𝑖}1−𝑏_{𝑖−2,𝑖−2}𝑏_{𝑖−1,𝑖−2}1−𝑏_𝑖𝑖1−𝑏_{𝑖−2,𝑖−2}−𝑏_{𝑖,𝑖−2}𝑏_{𝑖−2,𝑖}𝑓𝑜𝑟𝑖∈3,…,𝑝,𝛼₂<11−𝑏₂₂,0<𝛽_𝑖<𝑏_{𝑖,𝑖−1}1−𝑏_{𝑖+1,𝑖+1}+𝑏_{𝑖−1,𝑖}𝑏_{𝑖+1,𝑖−1}𝑏_{𝑖,𝑖−1}1−𝑏_{𝑖−1,𝑖−1}1−𝑏_{𝑖+1,𝑖+1}−𝑏_{𝑖−1,𝑖+1}𝑏_{𝑖+1,𝑖−1}𝑓𝑜𝑟𝑖∈2,…,𝑝−1,𝛽_𝑝<11−𝑏_𝑝𝑝, (3.10) or when 𝑏_𝑖𝑖≥1, 𝛼_𝑖>0, 𝛽_𝑖>0(𝑖∈{2,…,𝑝}), then either 𝜌𝐿⁽¹⁾_𝑟,𝜔<𝜌𝐿_𝑟,𝜔<1 (3.11) or 𝜌𝐿⁽¹⁾_𝑟,𝜔>𝜌𝐿_𝑟,𝜔>1. (3.12)

Proof. By direct operation, we have 𝐿_𝑟,𝜔=(1−𝜔)𝐼+𝜔𝐵₁()𝐶()−𝜔𝑈−𝜔1−𝑟1−𝜔𝐼+𝜔𝐵₂+𝜔𝑟00−𝐶𝐵₁𝐶𝑈. (3.13)
Since 0<𝜔≤1, 0≤𝑟<1, 𝐶≤0, 𝑈≤0, 𝐵₁≥0, 𝐵₂≥0, then 00−𝐶𝐵₁𝐶𝑈≥0 (3.14) and 𝐿_𝑟,𝜔 is nonnegative. Since 𝐻 is irreducible, from (3.13), it is easy to see that the matrix 𝐿_𝑟,𝜔 is nonnegative and irreducible.
Similarly, we can prove that the matrix 𝐿⁽¹⁾_𝑟,𝜔 is a nonnegative and irreducible matrix.
By Lemma 2.2, there is a positive vector 𝑥 such that 𝐿_𝑟,𝜔𝑥=𝜆𝑥, (3.15) where 𝜆=𝜌(𝐿_𝑟,𝜔). Since the matrix 𝐻 is nonsingular, 𝜆≠1. Hence, we get either 𝜆>1 or 𝜆<1.
Now, from (3.15) and by the definitions of 𝐿_𝑟,𝜔 and 𝐿⁽¹⁾_𝑟,𝜔, we have 𝐿⁽¹⁾_𝑟,𝜔𝑥−𝜆𝑥=𝐿⁽¹⁾_𝑟,𝜔−𝐿_𝑟,𝜔𝑥=−𝜔𝑆₁𝐼−𝐵₁−𝜔𝑆₁𝑈𝜔𝑟𝐶𝑆₁𝐼−𝐵₁𝜔𝑟𝐶𝑆₁𝑈𝑥=𝑆₁0−𝑟𝐶𝑆₁0−𝜔𝐼−𝐵₁𝑥=𝑆−𝜔𝑈00₁0−𝑟𝐶𝑆₁0−𝜔𝐼−𝐵₁𝜔()−𝜔𝑈𝑟−1𝐶−𝜔𝑟𝐶𝐵₁−𝜔𝐼+𝜔𝐵₂𝑥=𝑆+𝜔𝑟𝐶𝑈₁0−𝑟𝐶𝑆₁0𝐿_𝑟,𝜔(𝑆−𝐼𝑥=𝜆−1)¹0−𝑟𝐶𝑆₁0𝑥. (3.16)
Since 𝑏_𝑖,𝑖+1>0, 𝑏_𝑖+1,𝑖>0 for some 𝑖∈{2,…,𝑝}, when 0≤𝑏_𝑖𝑖<1 (𝑖∈{2,…,𝑝}), 0<𝛼_𝑖<𝑏_{𝑖−1,𝑖−2}𝑏_{𝑖−2,𝑖}+𝑏_{𝑖−1,𝑖}1−𝑏_{𝑖−2,𝑖−2}𝑏_{𝑖−1,𝑖−2}1−𝑏_𝑖𝑖1−𝑏_{𝑖−2,𝑖−2}−𝑏_{𝑖,𝑖−2}𝑏_{𝑖−2,𝑖}for𝑖∈3,…,𝑝,𝛼₂<11−𝑏₂₂,0<𝛽_𝑖<𝑏_{𝑖,𝑖−1}1−𝑏_{𝑖+1,𝑖+1}+𝑏_{𝑖−1,𝑖}𝑏_{𝑖+1,𝑖−1}𝑏_{𝑖,𝑖−1}1−𝑏_{𝑖−1,𝑖−1}1−𝑏_{𝑖+1,𝑖+1}−𝑏_{𝑖−1,𝑖+1}𝑏_{𝑖+1,𝑖−1}for𝑖∈2,…,𝑝−1,𝛽_𝑝<11−𝑏_𝑝𝑝, (3.17) or when 𝑏_𝑖𝑖≥1, 𝛼_𝑖>0, 𝛽_𝑖>0 (𝑖∈{2,…,𝑝}), then 𝑆₁≥0 and 𝑆₁≠0. So we have ^𝑆10_{−𝑟𝐶𝑆10}𝑥≥0, ^𝑆10_{−𝑟𝐶𝑆10}𝑥≠0.
If 𝜆<1, then 𝐿⁽¹⁾_𝑟,𝜔𝑥−𝜆𝑥≤0, 𝐿⁽¹⁾_𝑟,𝜔𝑥−𝜆𝑥≠0.
By Lemma 2.3, the inequality (3.11) is proved.
If 𝜆>1, then 𝐿⁽¹⁾_𝑟,𝜔𝑥−𝜆𝑥≥0, 𝐿⁽¹⁾_𝑟,𝜔𝑥−𝜆𝑥≠0.
By Lemma 2.3, the inequality (3.12) is proved.

Theorem 3.2. Let 𝐿_𝑟,𝜔,𝐿⁽²⁾_𝑟,𝜔 be the iteration matrices associated with the GAOR and preconditioned GAOR methods, respectively. If the matrix 𝐻 in (1.2) is irreducible with 𝐶≤0, 𝑈≤0, 𝐵₁≥0, 𝐵₂≥0, 0<𝜔≤1, 0≤𝑟<1, 𝑏_𝑖,1>0, 𝑏_1,𝑖>0 for some 𝑖∈{2,3,…,𝑝}, when 0≤𝑏₁₁<1, 0<𝛽_𝑖<1/(1−𝑏₁₁), 0<𝛼_𝑖<𝑏_1𝑖+𝛼₂𝑏₁₂𝑏_2𝑖+⋯𝛼_𝑖−1𝑏_1,𝑖−1𝑏_{𝑖−1,𝑖}+𝛼_𝑖+1𝑏_1,𝑖+1𝑏_𝑖+1,𝑖+⋯+𝛼_𝑝𝑏_1𝑝𝑏_𝑝𝑖𝑏_1𝑖1−𝑏_𝑖𝑖,𝑖∈2,3,…,𝑝 (3.18) or when 𝑏₁₁≥1, 𝛼_𝑖>0, 𝛽_𝑖>0, 𝑖∈{2,3,…,𝑝}, then either 𝜌𝐿⁽²⁾_𝑟,𝜔<𝜌𝐿_𝑟,𝜔<1 (3.19) or 𝜌𝐿⁽²⁾_𝑟,𝜔>𝜌𝐿_𝑟,𝜔>1. (3.20)
By the analogous proof of Theorem 3.1, we can prove Theorem 3.2.

4. Numerical Example

Now, we present an example to illustrate our theoretical results.

Example 4.1. The coefficient matrix 𝐻 in (1.2) is given by 𝐻=𝐼−𝐵₁𝑈𝐶𝐼−𝐵₂, (4.1) where 𝐵₁=(𝑏⁽¹⁾_𝑖𝑗)_𝑝×𝑝, 𝐵₂=(𝑏⁽²⁾_𝑖𝑗)_{(𝑛−𝑝)×(𝑛−𝑝)}, 𝐶=(𝑐_𝑖𝑗)_{(𝑛−𝑝)×𝑝}, and 𝑈=(𝑢_𝑖𝑗)_{𝑝×(𝑛−𝑝)} with 𝑏⁽¹⁾_𝑖𝑖=1()𝑏10×𝑖+1,𝑖=1,2,…,𝑝,_(1)𝑖𝑗=1−130𝑏30×𝑗+𝑖,𝑖<𝑗,𝑖=1,2,…,𝑝−1,𝑗=2,…,𝑝,_(1)𝑖𝑗=1−130𝑏30×𝑖−𝑗+1+𝑖,𝑖>𝑗,𝑖=2,…,𝑝,𝑗=1,2,…,𝑝−1,_(2)𝑖𝑖=1𝑏10×𝑝+𝑖+1,𝑖=1,2,…,𝑛−𝑝,_(2)𝑖𝑗=1−130𝑏30×𝑝+𝑗+𝑝+𝑖,𝑖<𝑗,𝑖=1,2,…,𝑛−𝑝+1,𝑗=2,…,𝑛−𝑝,_(2)𝑖𝑗=1−130𝑐30×𝑖−𝑗+1+𝑝+𝑖,𝑖>𝑗,𝑖=,2,…,𝑛−𝑝,𝑗=1,2,…,𝑛−𝑝−1,_𝑖𝑗=1−130×𝑝+𝑖−𝑗+1+𝑝+𝑖𝑢30,𝑖=1,2,…,𝑛−𝑝,𝑗=1,2,…,𝑝,_𝑖𝑗=1−130×𝑝+𝑗+𝑖30,𝑖=1,2,…,𝑝,𝑗=1,2,…,𝑛−𝑝. (4.2)
Table 1 displays the spectral radii of the corresponding iteration matrices with some randomly chosen parameters 𝑟, 𝜔, 𝑝. The randomly chosen parameters 𝛼_𝑖 and 𝛽_𝑖 satisfy the conditions of two theorems.
From Table 1, we see that these results accord with Theorems 3.1-3.2.

Table 1 

The spectral radii of the GAOR and preconditioned GAOR iteration matrices.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grant no. 11001144) and the Science and Technology Program of Shandong Universities of China (J10LA06).