The Fréchet Derivative of an Analytic Function of a Bounded Operator with Some Applications

Abstract

The main result in this paper is the determination of the Fréchet derivative of an analytic function of a bounded operator, tangentially to the space of all bounded operators. Some applied problems from statistics and numerical analysis are included as a motivation for this study. The perturbation operator (increment) is not of any special form and is not supposed to commute with the operator at which the derivative is evaluated. This generality is important for the applications. In the Hermitian case, moreover, some results on perturbation of an isolated eigenvalue, its eigenprojection, and its eigenvector if the eigenvalue is simple, are also included. Although these results are known in principle, they are not in general formulated in terms of arbitrary perturbations as required for the applications. Moreover, these results are presented as corollaries to the main theorem, so that this paper also provides a short, essentially self-contained review of these aspects of perturbation theory.

1. Introduction

Motivated by certain applications in numerical analysis and, in particular, statistics, this paper deals with the Fréchet derivative of an analytic function 𝜑 of a bounded linear operator 𝑇 on a separable Hilbert space ℍ (in the sense of the usual functional calculus), tangentially to the Banach space ℒ of all bounded linear operators mapping ℍ into itself. More precisely, a first order approximation to the difference 𝜑(𝑇^{0𝑥002𝑑𝑐})−𝜑(𝑇),𝑇^{0𝑥002𝑑𝑐}=𝑇+Π,Π∈ℒ, (1.1)is obtained, including the order of magnitude of the remainder. An example of such a function 𝜑 is a generalized or regularized inverse of the square root𝜑(𝑇^{0𝑥002𝑑𝑐})=(𝛼𝐼+𝑇^{0𝑥002𝑑𝑐})^−1/2,𝛼>0, (1.2)where 𝐼 is the identity operator. Once the Fréchet derivative has been established (Section 2), it yields the asymptotic distribution of functions of certain random operators via an ensuing delta method: a well-known statistical technique (see Section 4).

Clearly ^0𝑥0𝑇^02𝑑𝑐 can be regarded as a perturbed version of 𝑇, and it is not surprising that perturbation methods are employed to obtain the desired result. The authors are aware of the possibility that the rather straightforward result on the Fréchet derivative might be hidden somewhere in the rich literature on perturbation theory [1–3]. Yet they have not been successful in identifying a reference that states the result in its present form, tailored to the applications they have in mind. Some remarks are particularly in order.

(a)The perturbations Π are typically of small norm but otherwise arbitrary bounded or Hermitian. In literature, they are often of the formΠ=𝜖𝑇₁+𝜖²𝑇₂+⋯ (1.3)for operators 𝑇₁,𝑇₂,…, and a small number 𝜖. In statistics, there is no point in representing the perturbation in such a form.(b)The perturbation Π and the operator 𝑇 are not assumed to commute, because in our applications such an assumption would not in general be fulfilled. If the operators do commute, however, the Fréchet derivative would reduce to 𝜑^′(𝑇), in the sense of functional calculus with 𝜑^′ the derivative of 𝜑. In the case considered here, the actual Fréchet derivative and 𝜑^′(𝑇) may differ considerably.(c)A central theme in perturbation theory concerns the perturbation of an isolated eigenvalue and corresponding eigenprojection (see, e.g, the references mentioned before). Some of the results are included, because they can be easily derived from the main result on the Fréchet derivative by choosing a special function 𝜑 (Section 3). In this way, the paper presents a concise and essentially self-contained review of some basic results in this area. They are again presented in terms of a general (Hermitian) perturbation Π, as being required for statistical application, in the same vein as, but somewhat more general than, Dauxois et al. [4].

As has already been mentioned in the beginning, ℍ will be a separable Hilbert space and ℒ the Banach space of all bounded linear operators mapping ℍ into itself. The inner product on ℍ will be denoted by ⟨⋅,⋅⟩ and the norm by ‖⋅‖. The norm on ℒ will be written ‖⋅‖_ℒ, and the notation ℒ_𝐻 and 𝒞_𝐻 will be used to denote the subspace of all Hermitian and all compact Hermitian operators, respectively.

We will exclusively deal with infinite dimensional Hilbert spaces and will not attempt to include the simpler finite dimensional case in our formulation. The Fréchet derivative for arbitrary perturbations is well known in the finite dimensional matrix case. This result and further references can be found in the recent monograph by Bhatia [5]. In the finite dimensional case, this derivative is also implicitly present in Theorem 2.1 of Ruymgaart and Yang [6] to obtain the asymptotic distribution of a function of a random matrix.

2. The Fréchet Derivative

Let us fix an arbitrary 𝑇∈ℒ with spectrum 𝜎(𝑇) and a bounded open region Ω⊂ℂ in the complex plane with smooth boundary Γ=𝜕Ω, such that𝜎(𝑇)⊂Ω,𝛿_Γ=dist(Γ,𝜎(𝑇))>0. (2.1)Furthermore, let us consider functions of type𝜑∶𝐷→ℂ,analytic, (2.2)where 𝐷⊃Ω is an open neighborhood of Ω. Let us write𝑀_Γ=max_𝑧∈Γ|𝜑(𝑧)|<∞,𝐿_Γ=lengthofΓ<∞. (2.3)

The resolvent𝑅(𝑧)=(𝑧𝐼−𝑇)⁻¹,𝑧∈𝜌(𝑇), (2.4)is analytic on the resolvent set 𝜌(𝑇)=[𝜎(𝑇)]^𝑐, and the operator1𝜑(𝑇)=2𝜋𝑖_Γ𝜑(𝑧)R(𝑧)𝑑𝑧 (2.5)is well defined. This relation establishes an algebra homomorphism [7, Section 17.2] which implies in particular that𝜑(𝑇)𝜓(𝑇)=(𝜑𝜓)(𝑇), (2.6)if 𝜓∶𝐷→ℂ is also analytic. In particular, we have1𝑇=2𝜋𝑖_Γ𝑧𝑅(𝑧)𝑑𝑧. (2.7)

The operators 𝜑^′_𝑇1Π=2𝜋𝑖_Γ𝑆𝜑(𝑧)𝑅(𝑧)Π𝑅(𝑧)𝑑𝑧,_𝜑,Π=12𝜋𝑖_Γ𝜑(𝑧)𝑅(𝑧)(Π𝑅(𝑧))²(𝐼−Π𝑅(𝑧))⁻¹𝑑𝑧 (2.8)(2.9) are well defined for every Π∈ℒ sufficiently small. Note that according to Dunford and Schwartz [8, Lemma VII.6.11], there is a constant 0<𝐾<∞, such that‖𝑅(𝑧)‖_ℒ≤𝐾𝛿_Γ,∀𝑧∈Ω^𝑐. (2.10)

Theorem 2.1 (Fréchet Derivative). Let 𝑇∈ℒ and suppose that 𝜑 satisfies (2.2). Then 𝜑 maps the neighborhood {𝑇^{0𝑥002𝑑𝑐}=𝑇+Π∶Π∈ℒ,‖Π‖_ℒ≤𝑐𝛿_Γ/𝐾,forsome0<𝑐<1} into ℒ, when defined in the usual way of functional calculus. This mapping is Fréchet differentiable at 𝑇, tangentially to ℒ, with bounded derivative 𝜑^′_𝑇∶ℒ→ℒ, as defined by (2.8). More specifically, we have𝜑(𝑇+Π)=𝜑(𝑇)+𝜑^′_𝑇Π+𝑆_𝜑,Π, (2.11)where 𝑆_𝜑,Π is defined in (2.9) and ‖𝜑^′_𝑇Π‖_ℒ≤1𝑀2𝜋^Γ𝐿_Γ(𝐾𝛿_Γ)²‖Π‖_ℒ,‖𝑆_𝜑,Π‖_ℒ≤1𝑀2(1−𝑐)𝜋^Γ𝐿_Γ(𝐾𝛿_Γ)³‖Π‖²_ℒ. (2.12)(2.13)

Proof. For 𝜑 to be well defined on the neighborhood, let us first show that𝜎(𝑇^{0𝑥002𝑑𝑐})⊂Ω,∀Π∈ℒwith‖Π‖_ℒ𝛿≤𝑐^Γ𝐾. (2.14)To verify this, note that by (2.10) we have ‖Π𝑅(𝑧)‖_ℒ<𝑐 for such Π. Consequently, the operator𝑅(𝑧)(𝐼−Π𝑅(𝑧))⁻¹=𝑅(𝑧){(𝑅(𝑧)⁻¹−Π)𝑅(𝑧)}⁻¹=(𝑧𝐼−𝑇−Π)⁻¹=𝑅^{0𝑥002𝑑𝑐}(𝑧) (2.15)is bounded for each 𝑧∈Ω^𝑐, which entails (2.14). Hence, 𝜑(𝑇^{0𝑥002𝑑𝑐}1)=2𝜋𝑖_Γ𝜑(𝑧)𝑅^{0𝑥002𝑑𝑐}(𝑧)𝑑𝑧 (2.16)is well defined for Π with ‖Π‖_ℒ≤𝑐𝛿_Γ/𝐾.
Applying a Neumann series expansion [9, Section 5.2] to the inverse on the left in (2.15), we obtain^0𝑥0𝑅^02𝑑𝑐(𝑧)=𝑅(𝑧){𝐼+Π𝑅(𝑧)+(Π𝑅(𝑧))²+⋯}=𝑅(𝑧)+𝑅(𝑧)Π𝑅(𝑧)+𝑅(𝑧)(Π𝑅(𝑧))²(𝐼−Π𝑅(𝑧))⁻¹, (2.17)just as in Watson [10] for matrices. Term-wise integration yields (2.11).
The upper bounds in (2.12) and (2.13) are immediate from (2.8) and (2.9), respectively, by exploiting (2.3) and (2.15). The boundedness of 𝜑^′_𝑇 as a linear operator mapping ℒ into itself follows at once from (2.12).

Remark 2.2. It will be seen in Section 4 that for the applications we have in mind it is important that we do not require that 𝑇 and Π commute. If they do, however, it is clear that the Fréchet derivative in (2.8) reduces to𝜑^′_𝑇1Π={2𝜋𝑖_Γ𝜑(𝑧)𝑅²(𝑧)𝑑𝑧}Π. (2.18)It has been shown in Dunford and Schwartz [8, proof of Theorem VII.6.10] that12𝜋𝑖_Γ𝜑(𝑧)𝑅²1(𝑧)𝑑𝑧=2𝜋𝑖_Γ𝜑^′(𝑧)𝑅(𝑧)𝑑𝑧. (2.19)Combination of (2.11) with (2.18) and (2.19) yields𝜑(𝑇+Π)=𝜑(𝑇)+𝜑^′(𝑇)Π+𝒪(‖Π‖²_ℒ), (2.20)writing, for any 𝑟>0,𝒪(‖Π‖^𝑟_ℒ),as‖Π‖_ℒ→0, (2.21)to indicate any quantity (operator, vector, number) whose norm or absolute value is of the given order. Note that in (2.20) the operator, 𝜑^′(𝑇) is to be understood in the sense of the usual functional calculus as in (2.5) with 𝜑 replaced by its derivative 𝜑^′.
In this situation of commuting operators, Dunford and Schwartz [8] obtain the Taylor expansion𝜑(𝑇+Π)=^∞_𝑛=0𝜑^(𝑛)(𝑇)Π𝑛!^𝑛, (2.22)which implies, of course, (2.20).
Keeping the perturbation as before, we now restrict 𝑇 to the class 𝒞_𝐻 of compact Hermitian operators. The bounded and countable spectrum consists of the number 0, whether an eigenvalue or not, and all the nonzero eigenvalues 𝜆₁,𝜆₂,…∈ℝ. In this work, we avoid technical issues related to 𝜆=0 being an eigenvalue, and assume that 𝑇 is one-to-one, that is, 𝑇𝑓=0 implies that 𝑓=0. It is well known [7] that such a 𝑇 can be represented as𝑇=^∞_𝑗=1𝜆_𝑗𝑃_𝑗, (2.23)where the 𝑃_𝑗 are the corresponding orthogonal eigenprojections onto the mutually orthogonal finite dimensional eigenspaces. These projections provide a resolution of the identity in ℍ, that is,𝐼_ℍ=^∞_𝑗=1𝑃_𝑗. (2.24)The resolvent has the expansion𝑅(𝑧)=^∞_𝑗=11𝑧−𝜆_𝑗𝑃_𝑗,𝑧∈𝜌(𝑇). (2.25)

Corollary 2.3. Let the conditions of Theorem 2.1 be fulfilled for 𝑇∈𝒞_𝐻 with expansion (2.23). In this case the Fréchet derivative 𝜑^′_𝑇∶ℒ→ℒ is given by𝜑^′_𝑇Π=_𝑗𝜑^′(𝜆_𝑗)𝑃_𝑗Π𝑃_𝑗+_𝑗≠𝑘𝜑(𝜆_𝑘)−𝜑(𝜆_𝑗)𝜆_𝑘−𝜆_𝑗𝑃_𝑗Π𝑃_𝑘,Π∈ℒ. (2.26)

Proof. Let us substitute the expansion (2.25) for 𝑅(𝑧) into the expression for 𝜑^′_𝑇Π in (2.8). Application of the partial fraction method yields𝜑^′_𝑇Π=_𝑗𝑘{12𝜋𝑖_Γ𝜑(𝑧)(𝑧−𝜆_𝑗)(𝑧−𝜆_𝑘)𝑑𝑧}𝑃_𝑗Π𝑃_𝑘=_𝑗{12𝜋𝑖_Γ𝜑(𝑧)(𝑧−𝜆_𝑗)²𝑑𝑧}𝑃_𝑗Π𝑃_𝑗+_𝑗≠𝑘[1𝜆_𝑘−𝜆_𝑗12𝜋𝑖_Γ{𝜑(𝑧)(𝑧−𝜆_𝑘)−𝜑(𝑧)(𝑧−𝜆_𝑗)}𝑑𝑧]𝑃_𝑗Π𝑃_𝑘. (2.27)The right-hand side of (2.27) reduces at once to the expression on the right in (2.26) by an application of Cauchy's integral formula.

Example 2.4. The function 𝜑(𝑧)=𝑧, 𝑧∈ℂ, is analytic on the entire complex plane so that Corollary 2.3 applies. The Fréchet derivative in (2.26) now reduces to𝜑^′_𝑇Π=_𝑗𝑘𝑃_𝑗Π𝑃_𝑘=Π, (2.28)Π∈ℒ. Of course this result is immediate because in this simple case 𝜑(𝑇+Π)=𝑇+Π=𝜑(𝑇)+Π.

Example 2.5. Next let us, for 𝑝>0, consider the function𝜑_𝛼(𝑧)=(𝛼+𝑧)^−𝑝,𝛼>𝛿_Γ>0, (2.29)for 𝑧≠−𝛼. Note that the choice of 𝛼 ensures that the pole at 𝑧=−𝛼 remains outside the contour Γ. Clearly there exists an open region Ω of the type required, such that 𝜑 is analytic on some open neighborhood 𝐷 of Ω. Hence Corollary 2.3 applies again. The operator 𝜑_𝛼(𝑇)=(𝛼𝐼+𝑇)^−𝑝 represents a regularized or generalized inverse of Tikhonov type, according to whether 𝑇 is injective or not. The Fréchet derivative in (2.26) now equals𝜑^′_𝛼,𝑇Π=−𝑝_𝑗1(𝛼+𝜆_𝑗)^𝑝+1𝑃_𝑗Π𝑃_𝑗+_𝑗≠𝑘(𝛼+𝜆_𝑗)^𝑝−(𝛼+𝜆_𝑘)^𝑝(𝜆_𝑘−𝜆_𝑗)(𝛼+𝜆_𝑗)^𝑝(𝛼+𝜆_𝑘)^𝑝𝑃_𝑗Π𝑃_𝑘, (2.30)for Π∈ℒ.

Remark 2.6. For 𝑇∈𝒞_𝐻, 𝑇 and Π commuting the double sum on the right in (2.26) cancels and we obtain𝜑^′_𝑇Π=_𝑗𝜑^′(𝜆_𝑗)𝑃_𝑗Π, (2.31)in accordance with (2.20). Apparently, the double sum is a correction term needed when 𝑇 and Π do not commute.

3. Perturbation of Eigenvalues and Eigenvectors

Throughout this section, both 𝑇 and Π are assumed to be Hermitian, so that also 𝑇+Π∈ℒ_𝐻. In addition to this, we assume that𝑇∈ℒ_𝐻hasanisolatedeigenvalue𝜆₁, (3.1)with one-dimensional eigenspace. Consequently, the eigenprojection can be written𝑃₁=𝑝₁⊗𝑝₁,forsome𝑝₁∈ℍwith‖𝑝₁‖=1, (3.2)where for 𝑎,𝑏∈ℍ the operator 𝑎⊗𝑏 is defined by (𝑎⊗𝑏)𝑥=⟨𝑥,𝑏⟩𝑎,𝑥∈ℍ.

The region Ω⊃𝜎(𝑇) will now be chosen in such a way that it has a connected component Ω₁ with the propertiesΩ₁∩𝜎(𝑇)=𝜆₁,dist(Ω₁,Ω⧵Ω₁)>0. (3.3)A special analytic function 𝜑₁∶𝐷→ℂ such that𝜑₁(𝑧)=1,𝑧∈Ω₁,𝜑₁(𝑧)=0,𝑧∈Ω₀=Ω⧵Ω₁, (3.4)will play an important role in the sequel. Note, for instance, that𝜑₁(𝑇)=𝑃₁. (3.5)

For the Fréchet derivative of 𝜑₁(𝑇^{0𝑥002𝑑𝑐}) at 𝑇, a special expression can be obtained. Let us write𝑇=𝜆₁𝑃₁+𝑇₀, (3.6)where 𝑇₀ is Hermitian with spectrum 𝜎(𝑇₀)⊂Ω₀. According to the spectral theorem, there exists a resolution of the identity 𝐸(𝜆), 𝜆∈𝜎(𝑇₀), such that𝑇₀=_𝜎(𝑇0)𝜆𝑑𝐸(𝜆). (3.7)It should be noted that𝑃₁𝐸(𝜆)=𝐸(𝜆)𝑃₁=𝑂,∀𝜆∈𝜎(𝑇₀), (3.8)where 𝑂 is the zero operator, and that1𝑅(𝑧)=𝑧−𝜆₁𝑃₁+_𝜎(𝑇0)1𝑧−𝜆𝑑𝐸(𝜆). (3.9)Let us define𝑄₁=_𝜎(𝑇0)1𝜆₁−𝜆𝑑𝐸(𝜆). (3.10)

Lemma 3.1. The Fréchet derivative of 𝜑₁(𝑇^{0𝑥002𝑑𝑐}) at 𝑇 is given by𝜑^′_1,𝑇Π=𝑃₁Π𝑄₁+𝑄₁Π𝑃₁,Π∈ℒ_𝐻. (3.11)

Proof. This follows by substitution of (3.9) in the expression on the right in𝜑^′_1,𝑇1Π=2𝜋𝑖_Γ1𝑅(𝑧)Π𝑅(𝑧)𝑑𝑧,Γ₁=𝜕Ω₁, (3.12)for this derivative; see also (2.8). We thus obtain𝜑^′_1,𝑇1Π=2𝜋𝑖_Γ11(𝑧−𝜆₁)²𝑃₁Π𝑃₁+1𝑑𝑧2𝜋𝑖_Γ11𝑧−𝜆₁𝑃₁Π{_𝜎(𝑇0)1+1𝑧−𝜆𝑑𝐸(𝜆)}𝑑𝑧2𝜋𝑖_Γ1{_𝜎(𝑇0)11𝑧−𝜆𝑑𝐸(𝜆)}Π𝑧−𝜆₁𝑃₁+1𝑑𝑧2𝜋𝑖_Γ1{_𝜎(𝑇0)1𝑧−𝜆𝑑𝐸(𝜆)}Π{_𝜎(𝑇0)1𝑧−𝜇𝑑𝐸(𝜇)}. (3.13)
By Cauchy's integral formula12𝜋𝑖_Γ11(𝑧−𝜆₁)²𝑑𝑧=𝜑^′₁(𝜆₁)=0, (3.14)so that the first term on the right side in (3.13) is the zero operator. Regarding the second, note that12𝜋𝑖_Γ11(𝑧−𝜆₁1)(𝑧−𝜆)𝑑𝑧=𝜆₁(1−𝜆2𝜋𝑖_Γ11𝑧−𝜆₁1𝑑𝑧−2𝜋𝑖_Γ111𝑧−𝜆𝑑𝑧)=𝜆₁,−𝜆 (3.15)because each 𝜆∈𝜎(𝑇₀) lies outside the contour Γ₁. Consequently, the second term equals𝑃₁1Π{2𝜋𝑖_𝜎(𝑇0)1𝜆₁−𝜆𝑑𝐸(𝜆)}=𝑃₁Π𝑄₁. (3.16)Similarly, the third term equals 𝑄₁Π𝑃₁. The last term cancels, because12𝜋𝑖_Γ11(𝑧−𝜆)(𝑧−𝜇)𝑑𝑧=0, (3.17)since both 𝜆 and 𝜇 lie outside Γ₁.

Some results about the perturbation of 𝜆₁ and 𝑃₁ in a given direction as in (1.3) that are well known in literature [1, 2] can be partly recovered for perturbations in some neighborhood, in an essentially self-contained manner, as simple consequences of the results in Section 2.

Corollary 3.2. Under the assumptions (3.1), (3.2), and for Π∈ℒ_𝐻 sufficiently small, the operator ^0𝑥0𝑇^02𝑑𝑐 has an isolated eigenvalue ^0𝑥0𝜆^02𝑑𝑐₁ with eigenprojection ^0𝑥0𝑃^02𝑑𝑐₁=𝑝^{0𝑥002𝑑𝑐}₁⊗𝑝^{0𝑥002𝑑𝑐}₁ for some unit vector ^0𝑥0𝑝^02𝑑𝑐₁∈ℍ, satisfying^0𝑥0𝑃^02𝑑𝑐₁=𝑃₁+𝑃₁Π𝑄₁+𝑄₁Π𝑃₁+𝒪(‖Π‖²_ℒ), (3.18)where 𝑄₁ is defined in (3.10).

Proof. In view of (3.5) and (3.11), application of (2.11) with 𝜑=𝜑₁ yields 𝜑₁(𝑇^{0𝑥002𝑑𝑐})=𝑃₁+𝑃₁Π𝑄₁+𝑄₁Π𝑃₁+𝒪(‖Π‖²_ℒ). Clearly 𝜑₁(𝑇^{0𝑥002𝑑𝑐}) is Hermitian, and because (𝜑₁(𝑇^{0𝑥002𝑑𝑐}))²=𝜑²₁(𝑇^{0𝑥002𝑑𝑐})=𝜑₁(𝑇^{0𝑥002𝑑𝑐}) by (2.6), it is also idempotent so that it is in fact some projection ^0𝑥0𝑃^02𝑑𝑐₁, for example, it follows that ‖𝑃₁−𝑃^{0𝑥002𝑑𝑐}₁‖_ℒ<1 for all Π sufficiently small, and hence the range of ^0𝑥0𝑃^02𝑑𝑐₁ must also have dimension 1 [11] so that ^0𝑥0𝑃^02𝑑𝑐₁=𝑝^{0𝑥002𝑑𝑐}₁⊗𝑝^{0𝑥002𝑑𝑐}₁ for some ^0𝑥0𝑝^02𝑑𝑐₁∈ℍ with ‖𝑝^{0𝑥002𝑑𝑐}₁‖=1.
Next, let 𝜒(𝑧)=𝑧, 𝑧∈ℂ, be the identity function. By (2.6), again, on the one hand we have (𝜒𝜑₁)(𝑇^{0𝑥002𝑑𝑐})𝑝^{0𝑥002𝑑𝑐}₁=𝑇^{0𝑥002𝑑𝑐}𝑃^{0𝑥002𝑑𝑐}𝑝₁^{0𝑥002𝑑𝑐}₁=𝑇^{0𝑥002𝑑𝑐}𝑝^{0𝑥002𝑑𝑐}₁, and on the other (𝜑₁𝜒)(𝑇^{0𝑥002𝑑𝑐})𝑝^{0𝑥002𝑑𝑐}₁=𝑃^{0𝑥002𝑑𝑐}𝑇₁^{0𝑥002𝑑𝑐}𝑝^{0𝑥002𝑑𝑐}₁=(𝑝^{0𝑥002𝑑𝑐}₁⊗𝑝^{0𝑥002𝑑𝑐}₁)𝑇^{0𝑥002𝑑𝑐}𝑝^{0𝑥002𝑑𝑐}₁=⟨𝑇^{0𝑥002𝑑𝑐}_{0𝑥002𝑑𝑐}𝑝₁,𝑝^{0𝑥002𝑑𝑐}₁⟩𝑝^{0𝑥002𝑑𝑐}₁=𝜆^{0𝑥002𝑑𝑐}𝑝₁^{0𝑥002𝑑𝑐}₁. Hence ^0𝑥0𝑝^02𝑑𝑐₁ is an eigenvector of ^0𝑥0𝑇^02𝑑𝑐 with eigenvalue ^0𝑥0𝜆^02𝑑𝑐₁.

Corollary 3.3. Under the assumptions of Corollary 3.2, we have^0𝑥0𝑝^02𝑑𝑐₁=𝑝₁+𝑄₁Π𝑝₁+𝒪(‖Π‖²_ℒ). (3.19)

Proof. Let us first observe that 𝑃₁Π𝑄₁𝑝₁=0 because of (3.8). Hence (3.18) yields ^0𝑥0𝑝^02𝑑𝑐₁−𝑝₁=(𝑃^{0𝑥002𝑑𝑐}₁−𝑃₁)𝑝₁+𝑟₁+𝑟₂=𝑄₁Π𝑝₁+𝑟₁+𝑟₂+𝒪(‖Π‖²_ℒ), where𝑟₁=𝑃₁(𝑝^{0𝑥002𝑑𝑐}₁−𝑝₁),𝑟₂=(𝑃^{0𝑥002𝑑𝑐}₁−𝑃₁)(𝑝^{0𝑥002𝑑𝑐}₁−𝑝₁). (3.20)It sufficies to show that ‖𝑟_𝑗‖=𝒪(‖Π‖²_ℒ) for 𝑗=1,2. The idea of the proof can be found in Dauxois et al. [4].
Regarding 𝑟₁, note that 1−⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩²=‖⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩𝑝^{0𝑥002𝑑𝑐}₁−𝑝₁‖²≤‖𝑃^{0𝑥002𝑑𝑐}₁−𝑃₁‖²_ℒ=𝒪(‖Π‖²_ℒ), once more using (3.18). Hence ⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩→1, as ‖Π‖_ℒ→0, and therefore 2≥1+⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩≥1 for ‖Π‖_ℒ sufficiently small. This entails‖𝑟₁‖=‖⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩𝑝^{0𝑥002𝑑𝑐}₁−𝑝₁‖=|1−⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁=⟩||1−⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩²||1+⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩|=𝒪(‖Π‖²_ℒ). (3.21)
For 𝑟₂ we have‖𝑟₂‖≤‖𝑃^{0𝑥002𝑑𝑐}₁−𝑃₁‖_ℒ‖𝑝^{0𝑥002𝑑𝑐}₁−𝑝₁‖=𝒪(‖Π‖_ℒ)2(1−⟨𝑝^{0𝑥002𝑑𝑐}₁,𝑝₁⟩)=𝒪(‖Π‖_ℒ)2‖𝑟₁‖=𝒪(‖Π‖²_ℒ), (3.22)as can be seen from (3.21).

Corollary 3.4. Under the assumptions of Corollary 3.2, we have^0𝑥0𝜆^02𝑑𝑐₁=𝜆₁+⟨Π𝑝₁,𝑝₁⟩+𝒪(‖Π‖²_ℒ). (3.23)

Proof. With the help of (3.19), we see that ^0𝑥0𝜆^02𝑑𝑐₁=⟨𝑇^{0𝑥002𝑑𝑐}𝑝^{0𝑥002𝑑𝑐}₁,𝑝^{0𝑥002𝑑𝑐}₁⟩=⟨(𝑇+Π)(𝑝₁+𝑄₁Π𝑝₁),𝑝₁+𝑄₁Π𝑝₁⟩+𝒪(‖Π‖²_ℒ). The result follows from a routine calculation combined with the equalities ⟨𝑇𝑝₁,𝑝₁⟩=𝜆₁, ⟨𝑇𝑝₁,𝑄₁Π𝑝₁⟩=𝜆₁⟨𝑝₁,𝑄₁Π𝑝₁⟩=𝜆₁⟨𝑄₁𝑝₁,Π𝑝₁⟩=0, and ⟨𝑇𝑄₁Π𝑝₁,𝑝₁⟩=⟨𝑄₁Π𝑝₁,𝑇𝑝₁⟩=𝜆₁⟨𝑄₁Π𝑝₁,𝑝₁⟩=𝜆₁⟨Π𝑝₁,𝑄₁𝑝₁⟩=0. For the last two equalities we assume that 𝑇 and 𝑄₁ are Hermitian and 𝑄₁𝑝₁=0 by (3.8).

Corollary 3.5. Let 𝑇∈𝒞_𝐻 be given by (2.23) and satisfy (3.2). Then (3.18) and (3.19) remain true with𝑄₁=^∞_𝑗=21𝜆₁−𝜆_𝑗𝑃_𝑗. (3.24)

Proof. All nonzero eignvalues of 𝑇 are isolated, in particular 𝜆₁. It is immediate from (2.23) that 𝑇₀=∑^∞_𝑗=2𝜆_𝑗𝑃_𝑗, and this leads to the special expression for 𝑄₁ in (3.24).

Remark 3.6. The assumption that Π be Hermitian is in fact not necessary. Of course, if we just require Π to be bounded, the perturbed operator ^0𝑥0𝑇^02𝑑𝑐 is not in general Hermitian anymore. In particular, a suitably modified version of Corollary 3.3 will now claim the existence of a pair of eigenvectors, ^0𝑥0𝑝^02𝑑𝑐₁ for ^0𝑥0𝑇^02𝑑𝑐 and ^0𝑥0𝑝^{02𝑑𝑐∗}₁ for ^0𝑥0𝑇^{02𝑑𝑐∗}, with expansions^0𝑥0𝑝^02𝑑𝑐₁=𝑝₁+𝑄₁Π𝑝₁+𝒪(‖Π‖²_ℒ),𝑝^{0𝑥002𝑑𝑐∗}₁=𝑝₁+𝑄₁Π^∗𝑝₁+𝒪(‖Π‖²_ℒ), (3.25)as ‖Π‖→0.

4. Applications

In this section, we will sketch three applications: two in statistics and one in numerical analysis.

4.1. Noisy Integral Equations

Let 𝐾∶𝐿²(0,1)→𝐿²(0,1) be a compact injective integral operator, with measurable real kernel denoted by the same symbol without confusion. More specifically, input 𝑓∈𝐿²(0,1) and output 𝑔∈𝐿²(0,1) are related according to𝑔(𝑠)=¹₀𝐾(𝑠,𝑡)𝑓(𝑡)𝑑𝑡. (4.1)In practice, only finitely many data regarding the output are available, usually blurred by random measurement error. If the data are collected according to a random design, we may think of the data set as of 𝑛 independent copies (𝑋₁,𝑌₁),…,(𝑋_𝑛,𝑌_𝑛) of a pair (𝑋,𝑌) of random variables, where𝑌=𝑔(𝑋)+𝜖=(𝐾𝑓)(𝑋)+𝜖, (4.2)the design variable 𝑋 has a Uniform (0,1) distribution, the error variable 𝜖 has finite variance and zero mean, and where 𝑋 and 𝜖 are stochastically independent.

It is the purpose to recover 𝑓 from these data. It is expedient to “precondition” with the adjoint operator 𝐾^∗ and recover 𝑓 from the equation𝑞=𝐾^∗𝑔=𝐾^∗𝐾𝑓=𝑅𝑓, (4.3)where 𝑅 is compact, Hermitian, and strictly positive. Under suitable conditions, 1𝑞(𝑡)=𝑛^𝑛_𝑖=1𝑌_𝑖𝐾^∗(𝑡,𝑋_𝑖1)=𝑛^𝑛_𝑖=1𝑌_𝑖𝐾(𝑋_𝑖,𝑡),𝑡∈[0,1], (4.4)is an unbiased and √𝑛-consistent estimator of 𝑞; see, for instance, van Rooij and Ruymgaart [12]. Since 𝑅⁻¹ is unbounded, an estimator of the input 𝑓 is obtained by applying a regularized inverse of 𝑅 to 𝑞. Here we will use the Tikhonov type inverse(𝛼𝐼+𝑅)⁻¹=𝜑_𝑎(𝑅),𝛼>0, (4.5)where 𝜑_𝛼(𝑧)=(𝛼+𝑧)⁻¹; see also (2.29). This yields the input estimator𝑓_𝛼=𝜑_𝛼(𝑅)𝑞,𝛼>0. (4.6)To assess the quality of the estimator, one considers the mean integrated squared error (MISE)𝔼‖𝑓_𝛼−𝑓‖². (4.7)The behavior of the MISE is well studied in literature.

Recently, there is an interest in certain econometric models where the operator 𝐾 (or 𝑅) is unknown but can be estimated from the data. Let 𝑅 denote an estimator of 𝑅 and assume that 𝑅 is also compact, Hermitian, and nonnegative. In this case, the input estimator𝑓_𝛼=(𝛼𝐼+𝑅)⁻¹𝑞=𝜑_𝛼(𝑅)𝑞,𝛼>0, (4.8)will be employed. One expects that estimation of 𝑅 will increase the MISE, and naturally the question arises how much bigger the MISE of 𝑓_𝛼 will be than that of 𝑓_𝛼.

An upper bound for this increase of the MISE can be easily found from the results in Section 2. For large sample size 𝑛, 𝑅 will be close to 𝑅, and Π=𝑅−𝑅 can be considered as a small random perturbation of 𝑅. Writing 𝜑^′_𝛼,𝑅 for the Fréchet derivative at 𝑅, we see from Theorem 2.1 that𝑓_𝛼−𝑓={𝜑_𝛼(𝑅)−𝜑_𝛼(𝑅)}𝑞+𝜑_𝛼(𝑅)𝑞−𝑓=(𝜑^′_𝛼,𝑅Π)𝑞+𝑓_𝛼−𝑓+𝒪(‖Π‖²). (4.9)Apparently, (𝜑^′_𝛼,𝑅Π)𝑞 is an extra error term due to the estimation of 𝑅.

To find an upper bound for its MISE, let us first observe that (2.30) simplifies for 𝑝=1 and yields𝜑^′_𝛼,𝑅Π=−_𝑗𝑘1(𝛼+𝜆_𝑗)(𝛼+𝜆_𝑘)𝑃_𝑗Π𝑃_𝑘, (4.10)where now the 𝜆_𝑗 and the 𝑃_𝑗 are the spectral characteristics of 𝑅. Let us write, for brevity,ℎ=_𝑘1𝛼+𝜆_𝑘𝑃_𝑘𝑞, (4.11)and note that‖ℎ‖²≤1𝛼²‖𝑞‖². (4.12)We thus arrive at𝔼‖(𝜑^′_𝛼,𝑅Π)𝑞‖²=𝔼‖_𝑗1𝛼+𝜆_𝑗𝑝_𝑗Πℎ‖²≤1𝛼²𝔼‖Π‖²_ℒ‖ℎ‖²≤1𝛼⁴𝔼‖Π‖²_ℒ‖𝑞‖². (4.13)

Hence, under suitable assumptions, estimation of the kernel yields an extra term in the MISE of the input estimator which is of order 𝛼⁻⁴. In the Russian literature, sharper bounds can be found; see in particular Bakushinsky and Kokurin [13,Section 2.2]. For results of this type in the statistical literature, obtained in a different manner, see, for instance, Hall and Horowitz [14] and Florens [15].

4.2. Some Asymptotics for Functional Canonical Correlations

Let 𝑋 be a real random element in the Hilbert space 𝐿²(0,1) and assume that 𝔼‖𝑋‖⁴<∞. Its mean 𝜇∈𝐿²(0,1) and covariance operator 𝑆∶𝐿²(0,1)→𝐿²(0,1) are well defined by the relations 𝔼⟨𝑓,𝑋⟩=⟨𝑓,𝜇⟩, 𝔼⟨𝑓,𝑋−𝜇⟩⟨𝑋−𝜇,𝑔⟩=⟨𝑓,𝑆𝑔⟩ for all 𝑓,𝑔∈𝐿²(0,1). The operator 𝑆 is known to be of finite trace and hence Hilbert-Schmidt and compact. It is also nonnegative Hermitian. Without real loss of generality, we will assume 𝑆 to be injective, so that it will be strictly positive.

Next suppose that we are given a random sample 𝑋₁,…,𝑋_𝑛 of independent copies of 𝑋. The usual estimators of 𝜇 and 𝑆 are 𝑋=(1/𝑛)∑^𝑛_𝑖=1𝑋_𝑖 and 𝑆=(1/𝑛)∑^𝑛_𝑖=1(𝑋_𝑖−𝑋)⊗(𝑋_𝑖−𝑋), respectively, where 𝑆 shares all the properties of 𝑆, except that it cannot be injective because it has a finite dimensional kernel whose range has dimension at most 𝑛−1.

Because 𝑆 cannot be injective, the finite dimensional definition of sample canonical correlation has to be modified, and some kind of smoothing or regularization is recommended in literature [16]. Regularization might even be useful when the population is considered, although 𝑆 is injective [17]. This regularization yields Tikhonov type inverses in an expression for the canonical correlation .

For a precise definition, let ℍ₁ and ℍ₂ be two closed subspaces of 𝐿²(0,1) and 𝐼_𝑗 the orthogonal projection onto ℍ_𝑗 (𝑗=1,2). Let us write 𝑆_𝑗𝑘=𝐼_𝑗𝑆𝐼_𝑘, and note that 𝐼_𝑗(𝛼𝐼+𝑆)𝐼_𝑘=𝑆_𝑗𝑘 for 𝑗≠𝑘. Similar notation will be used for 𝑆. The regularized squared principal canonical correlation for the population is now defined as𝜌²=sup_{𝑓1,𝑓2≠0}⟨𝑓₁,𝑆₁₂𝑓₂⟩²⟨𝑓₁,(𝛼𝐼₁+𝑆₁₁)𝑓₁⟩⟨𝑓₂,(𝛼𝐼₂+𝑆₂₂)𝑓₂⟩. (4.14)Its sample analogue 𝜌² is obtained by replacing the 𝑆_𝑗𝑘 with 𝑆_𝑗𝑘 in (4.14). The supremum is actually a maximum, and pairs of maximizers will be denoted by 𝑓^∗₁, 𝑓^∗₂, and 𝑓^∗₁, 𝑓^∗₂, respectively. The corresponding canonical variates then are⟨𝑋,𝑓^∗_𝑗⟩,⟨𝑋,𝑓^∗_𝑗⟩,𝑗=1,2. (4.15)

For an alternative description of these canonical correlations, let us introduce the operator𝑅₁=(𝛼𝐼₁+𝑆₁₁)^−1/2𝑆₁₂(𝛼𝐼₂+𝑆₂₂)⁻¹𝑆₂₁(𝛼𝐼₁+𝑆₁₁)^−1/2. (4.16)Interchanging the indices 1 and 2 yields 𝑅₂, and replacing 𝑆_𝑗𝑘 with 𝑆_𝑗𝑘 yields 𝑅₁ and 𝑅₂. It can be seen that all these operators are Hilbert-Schmidt and strictly positive Hermitian. It will be assumed that 𝑅_𝑗 has the largest eigenvalue with one-dimensional eigenspace generated by 𝑓^∗_𝑗 with ‖𝑓^∗_𝑗‖=1. Under this condition, it has been shown in Cupidon et al. [18] that𝜌²=largesteigenvalueof𝑅_𝑗=⟨𝑓^∗_𝑗,𝑅_𝑗𝑓^∗_𝑗⟩, (4.17)for 𝑗=1,2. A similar result holds true for 𝜌².

It is well known that the asymptotic distribution of the eigenvalues and eigenfunctions of a random operator can be derived from the asymptotic distribution of this random operator itself (see [10] for Euclidean spaces and [4] for Hilbert spaces). This technique is based on the results of Section 3. In the present situation, this means that we have to show the convergence in distribution of the suitably standardized 𝑅_𝑗. Because all operators are Hilbert-Schmidt, it can be shown thatℛ_𝑗=√𝑛(R_𝑗−R_𝑗)→−^𝑑𝒢,asn→∞,inℒ. (4.18)

Result (4.18) follows easily if convergence in distribution can be established for each of the factors defining 𝑅_𝑗, for instance,(𝛼𝐼+𝑆_𝑗𝑗)^−1/2=𝜑_𝛼(𝑆_𝑗𝑗), (4.19)where this time 𝜑_𝛼(𝑧)=(𝛼+𝑧)^−1/2, compare (2.29).It is known [4] that √𝑛{𝑆_𝑗𝑗−𝑆_𝑗𝑗}→−^𝑑𝒢_𝑗𝑗,asn→∞,inℒ_HS, (4.20)for some Gaussian random element 𝒢_𝑗𝑗, where ℒ_HS is the Hilbert space of all Hilbert-Schmidt operators mapping ℍ into itself. Writing 𝜑^′_𝛼,𝑗 for the Fréchet derivative evaluated at 𝑆_𝑗𝑗 (Section 2) and exploiting the fact that the imbedding of ℒ_HS is ℒ are continuous, we obtain via a kind of delta-method [18, 19]√𝑛{𝜑_𝛼(𝑆_𝑗𝑗)−𝜑_𝛼(𝑆_𝑗𝑗)}→−^𝑑𝜑^′_𝛼,𝑗𝒢_𝑗𝑗,as𝑛→∞,inℒ (4.21)the desired result. A combination of results like this for each of the factors of 𝑅_𝑗 yields (4.18).

4.3. Solution of a Nonlinear Operator Equation

In Bakushinsky and Kokurin [13], the following problem is considered. Let ℍ₁ and ℍ₂ be Hilbert spaces and 𝐹∶ℍ₁→ℍ₂ an operator, not necessarily linear. The (nonlinear) equation𝐹(𝑥)=0,𝑥∈ℍ, (4.22)is studied. Let 𝑥^∗ be a solution of (4.22) and introduce a set Ω={𝑥∈ℍ₁∶‖𝑥−𝑥^∗‖₁≤𝑟}, for some 𝑟>0. It is assumed that 𝐹 is Fréchet differentiable on Ω. If 𝐹^′_𝑥 is the derivative at 𝑥∈Ω it is, moreover, assumed that‖𝐹^′_𝑥−𝐹^′_𝑦‖_{ℒ(ℍ1,ℍ2)}≤𝐿‖𝑥−𝑦‖₁,𝑥,𝑦∈Ω, (4.23)where 0<𝐿<∞ is a given number. Given an initial point 𝑥₀∈Ω and a sequence {𝛼_𝑛}, 𝛼_𝑛>0, of regularization parameters, these authors show that, under some further conditions, the generalized Gauss-Newton method generates a sequence of points {𝑥_𝑛} such that‖𝑥_𝑛−𝑥^∗‖=𝒪(𝛼^𝑝_𝑛1),forsome𝑝≥2. (4.24)

In their proof of this result, the authors need a crucial upper bound. Under some additional assumptions, we want to derive this upper bound as an immediate consequence of Theorem 2.1. In order to relate the present problem to the setup of our paper, let us assume that ℍ₁=ℍ₂=ℍ, and note that(𝐹^′_𝑥∗)^∗(𝐹^′_𝑥∗)=𝑇∈ℒ_𝐻. (4.25)For 𝑥_𝑛, let(𝐹^′_𝑥𝑛)^∗(𝐹^′_𝑥𝑛)=𝑇^{0𝑥002𝑑𝑐}∈ℒ_𝐻, (4.26)and set^0𝑥0𝑇^02𝑑𝑐=𝑇+𝑇^{0𝑥002𝑑𝑐}−𝑇=𝑇+Π, (4.27)where obviously Π∈ℒ_𝐻. It is not hard to see that (4.23) entails‖Π‖_ℒ≤𝑎‖𝑥^∗−𝑥_𝑛‖, (4.28)for some 0<𝑎<∞. Let Γ be the contour in (2.1) and 𝐷 the corresponding domain. As in Bakushinsky and Kokurin [13], a function Θ(𝑧,𝛼), 𝑧∈𝐷, is employed in the iteration scheme, which is analytic on 𝐷.

Narrowing down the generality in Bakushinsky and Kokurin [13] somewhat further, so that the current conditions are satisfied, their proof of the convergence of the iterations requires an upper bound for the expression (in our notation)12𝜋𝑖_Γ{1−Θ(𝑧,𝛼_𝑛)}{𝑅(𝑧)−𝑅^{0𝑥002𝑑𝑐}(𝑧)}𝑑𝑧=1−Θ(𝑇,𝛼_𝑛)−{1−Θ(𝑇^{0𝑥002𝑑𝑐},𝛼_𝑛)}=Θ(𝑇^{0𝑥002𝑑𝑐},𝛼_𝑛)−Θ(𝑇,𝛼_𝑛). (4.29)Keeping 𝑛 fixed, let us briefly write this last expression as Θ(0𝑥002𝑑𝑐𝑇)−Θ(𝑇). Now Theorem 2.1 applies with 𝜑=Θ, and application yields at once‖Θ(𝑇^{0𝑥002𝑑𝑐})−Θ(𝑇)‖_ℒ≤‖Θ^′_𝑇(𝑇^{0𝑥002𝑑𝑐}−𝑇)‖_ℒ+𝒪(‖𝑇^{0𝑥002𝑑𝑐}−𝑇‖²_ℒ)≤𝑏‖𝑇^{0𝑥002𝑑𝑐}−𝑇‖_ℒ+𝒪(‖𝑇^{0𝑥002𝑑𝑐}−𝑇‖²_ℒ)≤𝑎𝑏‖𝑥^∗−𝑥_𝑛‖+𝒪(‖𝑥^∗−𝑥_𝑛‖²), (4.30)for some 0<𝑏<∞, by (4.28).

Acknowledgments

The authors are grateful to the referee for some useful comments. For this research, D. S. Gilliam was supported by AFOSR Grant no. FA9550-04-1027 and F. H. Ruymgaart by NSF Grant no. DMS-0605167.