A Smoothing Interval Neural Network

Abstract

In many applications, it is natural to use interval data to describe various kinds of uncertainties. This paper is concerned with an interval neural network with a hidden layer. For the original interval neural network, it might cause oscillation in the learning procedure as indicated in our numerical experiments. In this paper, a smoothing interval neural network is proposed to prevent the weights oscillation during the learning procedure. Here, by smoothing we mean that, in a neighborhood of the origin, we replace the absolute values of the weights by a smooth function of the weights in the hidden layer and output layer. The convergence of a gradient algorithm for training the smoothing interval neural network is proved. Supporting numerical experiments are provided.

1. Introduction

In the last two decades artificial neural networks have been successfully applied to various domains, including pattern recognition [1], forecasting [2, 3], and data mining [4, 5]. One of the most widely used neural networks is the feedforward neural network with the well-known error backpropagation learning algorithm. But in most neural network architectures, input variables and the predicted results are represented in the form of single point value, not in the form of intervals. However, in real-life situations, available information is often uncertain, imprecise, and incomplete, which can be represented by fuzzy data, a generalization of interval data. So in many applications it is more natural to treat the input variables and the predicted results in the form of intervals than a set of single-point value.

Since multilayer feedforward neural networks have high capability as a universal approximator of nonlinear mappings [6–8], some methods via neural networks for handling interval data have been proposed. For instance, in [9], the BP algorithm [10, 11] was extended to the case of interval input vectors. In [12], the author proposed a new extension of backpropagation by using interval arithmetic which called Interval Arithmetic Back-propagation (IABP). This new algorithm permits the use of training samples and targets which can be indistinctly points and intervals. In [13], the author proposed a new model of multilayer perceptron based on interval arithmetic that facilitates handling input and output interval data, where weights and biases are single valued and not interval valued.

However, weights oscillation phenomena during the learning procedure were observed in our numerical experiments for these interval neural networks models. In order to prevent the weights oscillation, a smoothing interval neuron is proposed in this paper. Here, by smoothing we mean that, in the activation function and in a neighborhood of the origin, we replace the absolute values of the weights by a smooth function of the weights. Gradient algorithms [14–17] are applied to train the smoothing interval neural network. The weak and strong convergence theorems of the algorithms are proved. Supporting numerical results are provided.

The remainder of this paper is organized as follows. Some basic notations of interval analysis are described in Section 2. The traditional interval neural network is introduced in Section 3. Section 4 is devoted to our smoothing interval neural network and the gradient algorithm. The convergence results of the gradient learning algorithm are shown in Section 5. Supporting numerical experiments are provided in Section 6. The appendix is devoted to the proof of the theorem.

2. Interval Arithmetic

Interval arithmetic as a tool appeared in numerical computing in late 1950s. Then the interval mathematic is a theory introduced by Moore [18] and Sunaga [19] in order to give control of errors in numeric computations. Fundamentals used in this paper are described below.

Let us denote the intervals by uppercase letters such as 𝐴 and the real numbers by lowercase letters such as 𝑎. An interval can be represented by its lower bounds 𝐿 and upper bounds 𝑈 as 𝐴=[𝑎^𝐿,𝑎^𝑈], or equivalently by its midpoint 𝐶 and radius 𝑅 as 𝐴=⟨𝑎^𝐶,𝑎^𝑅⟩, where 𝑎^𝐶=𝑎^𝐿+𝑎^𝑈2,𝑎^𝑅=𝑎^𝑈−𝑎^𝐿2. (2.1) For intervals 𝐴=[𝑎^𝐿,𝑎^𝑈] and 𝐵=[𝑏^𝐿,𝑏^𝑈], the basic interval operations are defined by 𝐴+𝐵=𝑎^𝐿+𝑏^𝐿,𝑎^𝑈+𝑏^𝑈,𝐴−𝐵=𝑎^𝐿−𝑏^𝑈,𝑎^𝑈−𝑏^𝐿,𝑘⋅𝐴=𝑘⋅𝑎^𝐿,𝑘⋅𝑎^𝑈,𝑘>0,𝑘⋅𝑎^𝑈,𝑘⋅𝑎^𝐿,𝑘<0, (2.2) where 𝑘 is a constant.

If 𝑓 is an increasing function, then the interval output is given by 𝑓(𝐴)=𝑓𝑎^𝐿,𝑓𝑎^𝑈. (2.3) In this paper, we use the following weighted Euclidean distance for a pair of intervals 𝐴 and 𝐵𝑑(𝐴,𝐵)=𝛽𝑎^𝐶−𝑏^𝐶2+1−𝛽𝑎^𝑅−𝑏^𝑅2,𝛽∈(0,1). (2.4) The parameter 𝛽∈[0,1] facilitates giving more importance to the prediction of the output centres or to the prediction of the radii. For 𝛽=1 learning concentrates on the prediction of the output interval centre and no importance is given to the prediction of its radius. For 𝛽=0.5 both predictions (centres and radii) have the same weights in the objective function. For our purpose, we assume 𝛽∈(0,1).

3. Interval Neural Network

In this paper, we consider an interval neural network with three layers, where the input and output are interval value, the weights are real value. The numbers of neurons for the input, hidden and output layers are 𝑁,𝑀,1, respectively. Let 𝑊_𝑚=(𝑤_𝑚1,𝑤_𝑚2,…,𝑤_𝑚𝑁)^𝑇∈ℝ^𝑁, 𝑚=1,2,…,𝑀 be the weight matrix connecting the input and the hidden layers. The weight vector connecting the hidden and the output layers is denoted by 𝑊₀=(𝑤_0,1,𝑤_0,2,…,𝑤_0,𝑀)^𝑇∈ℝ^𝑀. To simplify the presentation, we write 𝑊=(𝑊^𝑇₀,𝑊^𝑇₁,…,𝑊^𝑇_𝑀)^𝑇∈ℝ^{𝑁𝑀+𝑀}. In the interval neural network, a nonlinear activation function 𝑓(𝑥) is used in the hidden layer, and a linear activation function in the output layer.

For an arbitrary interval-valued input 𝐗=(𝑋₁,𝑋₂,…,𝑋_𝑁), where 𝑋_𝑖=⟨𝑥^𝐶_𝑖,𝑥^𝑅_𝑖⟩, 𝑖=1,2,…,𝑁, as the weights of the proposed structure are real value, this linear combination results in a interval given by 𝑆_𝑚=^𝑁_𝑖=1𝑤_𝑚𝑖𝑋_𝑖=𝑠^𝐶_𝑚,𝑠^𝑅_𝑚=^𝑁_𝑖=1𝑤_𝑚𝑖𝑥^𝐶_𝑖,^𝑁_𝑖=1𝑤_𝑚𝑖𝑥^𝑅_𝑖. (3.1) Then the output of the interval neuron in the hidden layer is given by 𝐻_𝑚=𝑓𝑆_𝑚=𝑓𝑠^𝐶_𝑚−𝑠^𝑅_𝑚,𝑓𝑠^𝐶_𝑚+𝑠^𝑅_𝑚=ℎ^𝐶_𝑚,ℎ^𝑅_𝑚=𝑓𝑠^𝐶−𝑠^𝑅+𝑓𝑠^𝐶+𝑠^𝑅2,𝑓𝑠^𝐶+𝑠^𝑅−𝑓𝑠^𝐶−𝑠^𝑅2. (3.2) Finally, the output of the interval neuron in the output layer is given by 𝑌=𝑦^𝐶,𝑦^𝑅,𝑦^𝐶=^𝑀_𝑚=1𝑤_0𝑚ℎ^𝐶_𝑚,𝑦^𝑅=^𝑀_𝑚=1𝑤_0𝑚ℎ^𝑅_𝑚. (3.3)(3.4)(3.5)

4. Smoothing Interval Neural Network

4.1. Smoothing Interval Neural Network Structure

As revealed in the numerical experiment below in this paper, there appear weights oscillation phenomena during the learning procedure for the original interval neural network presented in the last section. In order to prevent the weights oscillation, we propose a smoothing interval neural network by replacing |𝑤_𝑚𝑖| and |𝑤_0𝑚| with a smooth function 𝜑(𝑤_𝑚𝑖) and 𝜑(𝑤_0𝑚) in (3.1) and (3.5). Then, the output of the smoothing interval neuron in the hidden layer is defined as 𝑆_𝑚=^𝑁_𝑖=1𝑤_𝑚𝑖𝑋_𝑖=^𝑁_𝑖=1𝑤_𝑚𝑖𝑥^𝐶_𝑖,^𝑁_𝑖=1𝜑𝑤_𝑚𝑖𝑥^𝑅_𝑖,𝐻_𝑚=𝑓𝑆_𝑚=𝑓𝑠^𝐶_𝑚−𝑠^𝑅_𝑚,𝑓𝑠^𝐶_𝑚+𝑠^𝑅_𝑚=ℎ^𝐶_𝑚,ℎ^𝑅_𝑚=𝑓𝑠^𝐶_𝑚−𝑠^𝑅_𝑚+𝑓𝑠^𝐶_𝑚+𝑠^𝑅_𝑚2,𝑓𝑠^𝐶_𝑚+𝑠^𝑅_𝑚−𝑓𝑠^𝐶_𝑚−𝑠^𝑅_𝑚2. (4.1)(4.2) The output of the smoothing interval neuron in the output layer is given by 𝑦^𝐶=^𝑀_𝑚=1𝑤_0𝑚ℎ^𝐶_𝑚,𝑦^𝑅=^𝑀_𝑚=1𝜑𝑤_0𝑚ℎ^𝑅_𝑚. (4.3) For our purpose, 𝜑(𝑥) can be chosen as any smooth function that approximates |𝑥| near the origin. For definiteness and simplicity, we choose 𝜑(𝑥) as a polynomial function: 𝜑(𝑥)=𝜑(𝑥)−𝑥,𝑥≤−𝜇,,−𝜇<𝑥<𝜇,𝑥,𝑥≥𝜇, (4.4) where 𝜇>0 is a small constant and 𝜑(𝑥)1=−8𝜇³𝑥⁴+3𝑥4𝜇²+38𝜇. (4.5) We observe that the above defined 𝜑(𝑥) is a convex function in 𝐶²(𝑅), and it is identical to the absolute value function |𝑥| outside the zero neighborhood (−𝜇,𝜇).

4.2. Gradient Algorithm of the Smoothing Interval Neural Network

Suppose that we are supplied with a training sample set {𝑋_𝑗,𝑂_𝑗}^𝐽_𝑗=1, where 𝑋_𝑗’s and 𝑂_𝑗’s are input and ideal output samples, respectively, as follows: 𝑋_𝑗=(𝑋_1𝑗,𝑋_2𝑗,…,𝑋_𝑁𝑗)^𝑇, 𝑋_𝑖𝑗=[𝑥^𝐿_𝑖𝑗,𝑥^𝑈_𝑖𝑗]=⟨𝑥^𝐶_𝑖𝑗,𝑥^𝑅_𝑖𝑗⟩, 𝑖=1,2,…,𝑁, 𝑂_𝑗=[𝑜^𝐿_𝑗,𝑜^𝑈_𝑗]=⟨𝑜^𝐶_𝑗,𝑜^𝑅_𝑗⟩. Our task is to find the weights 𝑊=(𝑊^𝑇₀,𝑊^𝑇₁,…,𝑊^𝑇_𝑀)^𝑇 such that 𝑂_𝑗=𝑌𝑋_𝑗,𝑗=1,2,…,𝐽. (4.6) But usually, the weight 𝑊=(𝑊^𝑇₀,𝑊^𝑇₁,…,𝑊^𝑇_𝑀)^𝑇 satisfying (4.6) does not exit and, instead, the aim of the network learning is to choose the weight 𝑊 to minimize an error function of the smoothing interval neural network. By (2.4), a simple and typical error function is the quadratic error function: 𝐸(𝑊)=12^𝐽_𝑗=1𝛽𝑜^𝐶_𝑗−𝑦^𝐶_𝑗²+1−𝛽𝑜^𝑅_𝑗−𝑦^𝑅_𝑗². (4.7) Let us denote 𝑓^𝐶_𝑗(𝑡)=(1/2)(𝑜^𝐶_𝑗−𝑡^𝐶_𝑗)², 𝑓^𝑅_𝑗(𝑡)=(1/2)(𝑜^𝑅_𝑗−𝑡^𝑅_𝑗)², 𝑗=1,2,…,𝐽, 𝑡∈ℝ, then the error function (4.7) is rewritten as 𝐸(𝑊)=^𝐽_𝑗=1𝛽𝑓^𝐶_𝑗𝑦+1−𝛽𝑓^𝑅_𝑗𝑦. (4.8) Now, we introduce the gradient algorithm [15, 16] for the smoothing interval neural network. The gradient of the error function 𝐸(𝑊) with respect to 𝑊₀ is given by 𝜕𝐸(𝑊)𝜕𝑊₀=^𝐽_𝑗=1𝛽𝑦^𝐶_𝑗−𝑜^𝐶_𝑗𝜕𝑦^𝐶_𝑗𝜕𝑊₀+1−𝛽𝑦^𝑅_𝑗−𝑜^𝑅_𝑗𝜕𝑦^𝑅_𝑗𝜕𝑊₀=^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗𝑦𝜕𝑦^𝐶_𝑗𝜕𝑊₀+1−𝛽𝑓^′𝑅_𝑗𝑦𝜕𝑦^𝑅_𝑗𝜕𝑊₀, (4.9) where 𝜕𝑦^𝐶_𝑗𝜕𝑊₀=ℎ^𝐶_𝑗,𝜕𝑦^𝑅_𝑗𝜕𝑊₀=𝜑^′𝑊₀ℎ^𝑅_𝑗. (4.10) The gradient of the error function 𝐸(𝑊) with respect to 𝑊_𝑚,𝑚=1,2,…,𝑀 is given by 𝜕𝐸(𝑊)𝜕𝑊_𝑚=^𝐽_𝑗=1𝛽𝑦^𝐶_𝑗−𝑜^𝐶_𝑗𝜕𝑦^𝐶_𝑗𝜕ℎ^𝐶_𝑗𝑚𝜕ℎ^𝐶_𝑗𝑚𝜕𝑊_𝑚+1−𝛽𝑦^𝑅_𝑗−𝑜^𝑅_𝑗𝜕𝑦^𝑅_𝑗𝜕ℎ^𝑅_𝑗𝑚𝜕ℎ^𝑅_𝑗𝑚𝜕𝑊_𝑚=^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗𝑦𝜕𝑦^𝐶_𝑗𝜕ℎ^𝐶_𝑗𝑚𝜕ℎ^𝐶_𝑗𝑚𝜕𝑊_𝑚+1−𝛽𝑓^′𝑅_𝑗𝑦𝜕𝑦^𝑅_𝑗𝜕ℎ^𝑅_𝑗𝑚𝜕ℎ^𝑅_𝑗𝑚𝜕𝑊_𝑚, (4.11) where 𝜕𝑦^𝐶_𝑗𝜕ℎ^𝐶_𝑗𝑚=𝑤_0𝑚,𝜕𝑦^𝑅_𝑗𝜕ℎ^𝑅_𝑗𝑚=𝜑𝑤_0𝑚,𝜕ℎ^𝐶_𝑗𝑚𝜕𝑊_𝑚=𝑓^′𝑠^𝐶_𝑗𝑚−𝑠^𝑅_𝑗𝑚𝑥^𝐶_𝑗−𝜑^′𝑊_𝑚𝑥^𝑅_𝑗2+𝑓^′𝑠^𝐶_𝑗𝑚+𝑠^𝑅_𝑗𝑚𝑥^𝐶_𝑗+𝜑^′𝑊_𝑚𝑥^𝑅_𝑗2,𝜕ℎ^𝑅_𝑗𝑚𝜕𝑊_𝑚=𝑓^′𝑠^𝐶_𝑗𝑚+𝑠^𝑅_𝑗𝑚𝑥^𝐶_𝑗+𝜑^′𝑊_𝑚𝑥^𝑅_𝑗2−𝑓^′𝑠^𝐶_𝑗𝑚−𝑠^𝑅_𝑗𝑚𝑥^𝐶_𝑗−𝜑^′𝑊_𝑚𝑥^𝑅_𝑗2. (4.12) In the learning procedure, the weights 𝑊 are iteratively refined as follows: 𝑊^𝑘+1=𝑊^𝑘+Δ𝑊^𝑘, (4.13) where Δ𝑊^𝑘𝑊=−𝜂𝜕𝐸^𝑘,𝜕𝑊 (4.14) where 𝜂>0 a constant learning rate and 𝑘=1,2,.…

5. Convergence Theorem for SINN

For any 𝐱∈ℝ^𝑛, its Euclidean norm is ‖𝐱‖=∑^𝑛_𝑖=1𝑥²_𝑖. Let Ω₀={𝑊∈Ω∶𝐸_𝑊(𝑊)=0} be the stationary point set of the error function 𝐸(𝑊), where Ω⊂ℝ^{𝑁𝑀+𝑀} is a bounded region satisfying (𝐴2) below. Let Ω_0,𝑠⊂ℝ be the projection of Ω₀ onto the 𝑠th coordinate axis, that is, Ω_0,𝑠=𝑤_𝑠∈ℝ∶𝑊=𝑤₁,…,𝑤_𝑠,…,𝑤_{𝑁𝑀+𝑀}^𝑇∈Ω₀, (5.1) for 𝑠=1,2,…,𝑁𝑀+𝑀. To analyze the convergence of the algorithm, we need the following assumptions. (𝐴1)|𝑓(𝑡)|, |𝑓^′(𝑡)|, |𝑓^′′(𝑡)| are uniformly bounded for 𝑡∈𝑅. (𝐴2) There exists a bounded region Ω⊂ℝ^𝑛 such that {𝑊^𝑘}⊂Ω(𝑘∈ℕ). (𝐴3) The learning rate 𝜂 is small enough such that (A.10) below is valid. (𝐴4)Ω_0,𝑠 does not contain any interior point for every 𝑠=1,2,…,𝑁𝑀+𝑀.

Now we are ready to present one convergence theorem of the learning algorithms. Its proof is given in the appendix later on.

Theorem 5.1. Let the error function 𝐸(𝑊) be defined by (4.7), and the weight sequence {𝑊^𝑘} be generated by the learning procedure (4.13) and (4.14) for smoothing interval neuron with 𝑊⁰ being an arbitrary initial guess. If Assumptions (𝐴1), (𝐴2), and (𝐴3) are valid, then we have 𝐸𝑊^𝑘+1≤𝐸𝑊^𝑘,lim_𝑘→∞𝐸_𝑊𝑊^𝑘=0. (5.2)(5.3) Furthermore, if Assumption (𝐴4) also holds, there exists a point 𝑊^∗∈Ω₀ such that lim_𝑘→∞𝑊^𝑘=𝑊^∗. (5.4)

6. Numerical Experiment

We compare the performances of the interval neural network and the smoothing interval neural network by approximating a simple interval function 𝑌=0.01×(𝑋+11)². (6.1) In this example, the training set contains five training samples. Their midpoints are all 0 and their radii are (0.85522.62488.01010.29229.2885), respectively. The corresponding outputs of the samples are 𝑌=⟨𝑦^𝐶,𝑦^𝑅⟩=0.01×(𝑋+11)².

For the above two interval neural networks, the error function 𝐸(𝑊) is defined as in (4.7). But in order to see the error more clearly in the figures, we will also use the error 𝐷 defined by 1𝐷=ln𝐸=ln2^𝐽_𝑗=1𝛽𝑜^𝐶_𝑗−𝑦^𝐶_𝑗²+1−𝛽𝑜^𝑅_𝑗−𝑦^𝑅_𝑗². (6.2)

The number of training iterations is 2000, the initial midpoint of weight vector is selected randomly from [−0.01,0.01], and two neurons are selected in the hidden layer. The fix learning rate is 𝜂=0.2, 𝛽=0.5, and 𝜇=0.5.

In the learning procedure for the interval neural network, we clearly see from Figure 1(a) that the gradient norm is not convergent. Figure 2(a) shows that the error function 𝐷 is oscillating and not convergent. On the contrary, we see from Figure 1(b) that the gradient norm of the smoothing interval neural network is convergent. Figure 2(b) shows that the error function 𝐷, as well as 𝐸, is monotone decreasing and convergent.

(a) Interval neural network

(b) Smoothing interval neural network

Figure 1 

Norm of gradient of the interval neural network and the smoothing interval neuron in the training.

(a) Value of 𝐷 for interval neural network

(b) Value of 𝐷 for smoothing interval neural network

Figure 2 

Values of the error function 𝐷 for the interval neural network and the smoothing neural network.

From this numerical experiment, we can see that the proposed smoothing neural network can efficiently avoid the oscillation during the training process.

Appendix

First, we give Lemmas A.1 and A.2. Then, we use them to prove Theorem 5.1.

Lemma A.1. Let {𝑏_𝑚} be a bounded sequence satisfying lim_𝑚→∞(𝑏_𝑚+1−𝑏_𝑚)=0. Write 𝛾₁=lim_𝑛→∞inf_𝑚>𝑛𝑏_𝑚, 𝛾₂=lim_𝑛→∞sup_𝑚>𝑛𝑏_𝑚, and 𝑆={𝑎∈ℝ∶ There exists a subsequence {𝑏_𝑖𝑘} of {𝑏_𝑚} such that 𝑏_𝑖𝑘→𝑎 as 𝑘→∞}. Then we have 𝑆=𝛾₁,𝛾₂. (A.1)

Proof. It is obvious that 𝛾₁≤𝛾₂ and 𝑆⊂[𝛾₁,𝛾₂]. If 𝛾₁=𝛾₂, then (A.1) follows simply from lim_𝑚→∞𝑏_𝑚=𝛾₁=𝛾₂. Let us consider the case 𝛾₁<𝛾₂ and proceed to prove that 𝑆⊃[𝛾₁,𝛾₂].
For any 𝑎∈(𝛾₁,𝛾₂), there exists 𝜀>0 such that (𝑎−𝜀,𝑎+𝜀)⊂(𝛾₁,𝛾₂). Noting lim_𝑚→∞(𝑏_𝑚+1−𝑏_𝑚)=0, we observe that 𝑏_𝑚 travels between 𝛾₁ and 𝛾₂ with very small pace for all large enough 𝑚. Hence, there must be infinite number of points of the sequence {𝑏_𝑚} falling into (𝑎−𝜀,𝑎+𝜀). This implies 𝑎∈𝑆 and thus (𝛾₁,𝛾₂)⊂𝑆. Furthermore, (𝛾₁,𝛾₂)⊂𝑆 immediately leads to [𝛾₁,𝛾₂]⊂𝑆. This completes the proof.
For any 𝑘=0,1,2,…,  1≤𝑗≤𝐽, we define the following notations. Φ^𝐶_0,𝑘,𝑗=𝑊^𝑘₀⋅ℎ^𝐶_𝑘,𝑗,Φ^𝑅_0,𝑘,𝑗=𝜑𝑊^𝑘₀⋅ℎ^𝑅_𝑘,𝑗,Ψ^𝐶_𝑘,𝑗=ℎ^𝐶_𝑘+1,𝑗−ℎ^𝐶_𝑘,𝑗,Ψ^𝑅_𝑘,𝑗=ℎ^𝑅_𝑘+1,𝑗−ℎ^𝑅_𝑘,𝑗. (A.2)

Lemma A.2. Suppose Assumption (𝐴2),(𝐴3) holds, for any 𝑘=0,1,2,… and 1≤𝑗≤𝐽, then we have max𝑥^𝐶_𝑗,𝑥^𝑅_𝑗,𝑜^𝐶_𝑗,𝑜^𝑅_𝑗𝑊^𝑘₀,Φ^𝐶_0,𝑘,𝑗,Φ^𝑅_0,𝑘,𝑗≤𝑀₀,_𝐽𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗ℎ^𝐶_𝑘,𝑗Δ𝑊^𝑘₀+1−𝛽𝑓^′𝑅_𝑗Φ^𝑅_0,𝑘,𝑗ℎ^𝑅_𝑘,𝑗𝜑^′𝑊^𝑘₀Δ𝑊^𝑘₀𝑊=−𝜂𝜕𝐸^𝑘𝜕𝑊₀²_𝐽𝑗=1,𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗≤𝑀₁𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊²_𝐽𝑗=1,𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗+^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗𝜑𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗≤−𝜂+𝑀₂𝜂^2𝑀_𝑚=1𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚²,12^𝐽_𝑗=1𝛽𝑓_𝑗^′′𝐶𝜉^𝐶_0,𝑘,𝑗Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗²≤𝑀₃𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊²_𝐽𝑗=1,1−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗𝜑^′𝜁^𝑘₁Δ𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗≤𝑀₄𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊²,12^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗𝜑^′′𝜁^𝑘₂Δ𝑊^𝑘₀²⋅ℎ^𝑅_𝑘,𝑗≤𝑀₅𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊₀²,12^𝐽_𝑗=11−𝛽𝑓_𝑗^′′𝑅𝜉^𝑅_0,𝑘,𝑗Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗²≤𝑀₆𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊², (A.3)(A.4)(A.5)(A.6)(A.7)(A.8)(A.9)(A.10) where 𝑀_𝑖(𝑖=0,1,2,3,4,5,6) is independent of 𝑘 and 𝑗, 𝜉^𝐶_0,𝑘,𝑗 lies on the segment between Φ^𝐶_{0,𝑘+1,𝑗} and Φ^𝐶_0,𝑘,𝑗, 𝜉^𝑅_0,𝑘,𝑗 lies on the segment between Φ^𝑅_{0,𝑘+1,𝑗} and Φ^𝑅_0,𝑘,𝑗, 𝜁^𝑘₁,𝜁^𝑘₂ both lie on the segment between 𝑊₀^𝑘+1 and 𝑊^𝑘₀.

Proof. The proof of (A.3) in Lemma A.2: For the given training sample set, by Assumption (𝐴2), (4.2), and (4.4), it is easy to known that (A.3) is valid.
The proof of (A.4) in Lemma A.2: by (4.9) and (4.14), we have ^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗ℎ^𝐶_𝑘,𝑗Δ𝑊^𝑘₀+1−𝛽𝑓^′𝑅_𝑗Φ^𝑅_0,𝑘,𝑗ℎ^𝑅_𝑘,𝑗𝜑^′𝑊^𝑘₀Δ𝑊^𝑘₀=𝜕𝐸𝑊^𝑘𝜕𝑊₀⋅𝑊−𝜂𝜕𝐸^𝑘𝜕𝑊₀𝑊=−𝜂𝜕𝐸^𝑘𝜕𝑊₀². (A.11) This proves (A.4).
The proof of (A.5) in Lemma A.2: using the Mean Value Theorem, for any 1≤𝑚≤𝑀, 1≤𝑗≤𝐽, and 𝑘=0,1,2,…, we have Ψ^𝐶_{𝑘,𝑗,𝑚}=ℎ^𝐶_{𝑘+1,𝑗,𝑚}−ℎ^𝐶_{𝑘,𝑗,𝑚}=12𝑓𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑓𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}+𝑓𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑓𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}=12𝑓^′𝑡¹_{𝑘,𝑗,𝑚}𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}+𝑓^′𝑡²_{𝑘,𝑗,𝑚}𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}, (A.12) where 𝑡¹_{𝑘,𝑗,𝑚} is on the segment between 𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚} and 𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}, 𝑡²_{𝑘,𝑗,𝑚} is on the segment between 𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚} and 𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}. By (A.3), we have Ψ^𝐶_{𝑘,𝑗,𝑚}≤𝑀₀2𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}+𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}≤𝑀₀2𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}+𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}=𝑀₀𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}=𝑀₀Δ𝑊^𝑘_𝑚𝑥^𝐶_𝑗+𝜑𝑊_𝑚^𝑘+1−𝜑𝑊^𝑘_𝑚𝑥^𝑅_𝑗≤𝑀²₀Δ𝑊^𝑘_𝑚+𝜑^′𝜏^𝑘_1,𝑚Δ𝑊^𝑘_𝑚, (A.13) where 𝜏^𝑘_1,𝑚 is on the segment between 𝑊_𝑚^𝑘+1 and 𝑊^𝑘_𝑚. Since 𝜑(𝑥)=𝜑(𝑥)−𝑥,if𝑥≤−𝜇,,if−𝜇<𝑥<𝜇,𝑥,if𝑥≥𝜇, (A.14) if 𝑥≤−𝜇 and 𝑥≥𝜇, |𝜑^′(𝑥)|=1, |𝜑^′′(𝑥)|=0.
If −𝜇<𝑥<𝜇, we have 𝜑^′(𝑥)1=−2𝜇³𝑥³+3(),𝜑2𝜇𝑥∈−1,1_′′(𝑥)3=−2𝜇³𝑥²+3∈32𝜇0,,2𝜇 (A.15) so if 𝑥∈ℝ, we have 𝜑^′(𝑥)≤1,𝜑^′′(𝑥≤3).2𝜇 (A.16) According to (A.16) and (A.13), we can obtain that Ψ^𝐶_{𝑘,𝑗,𝑚}≤2𝑀²₀Δ𝑊^𝑘_𝑚. (A.17) By (A.17), for any 1≤𝑗≤𝐽 and 𝑘=0,1,2,…, we have Ψ^𝐶_𝑘,𝑗²=ℎ^𝐶_{𝑘+1,𝑗,1}−ℎ^𝐶_𝑘,𝑗,1ℎ^𝐶_{𝑘+1,𝑗,2}−ℎ^𝐶_𝑘,𝑗,2⋮ℎ^𝐶_{𝑘+1,𝑗,𝑀}−ℎ^𝐶_{𝑘,𝑗,𝑀}²≤4𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚². (A.18) According to the definition of 𝑓^𝐶_𝑗(𝑡), we get that 𝑓^′𝐶_𝑗(𝑡)=𝑡^𝐶_𝑗−𝑜^𝐶_𝑗, combining with (A.3), we deduce that |𝑓^′𝐶_𝑗(Φ^𝐶_0,𝑘,𝑗)|≤2𝑀₀. By (A.18), we have ^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗≤2𝛽𝑀₀^𝐽_𝑗=1Δ𝑊^𝑘₀Ψ^𝐶_𝑘,𝑗≤𝛽𝑀₀^𝐽_𝑗=1Δ𝑊^𝑘₀²+Ψ^𝐶_𝑘,𝑗²≤𝛽𝐽𝑀₀Δ𝑊^𝑘₀²+4𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²≤𝑀₁^𝑀_𝑚=0Δ𝑊^𝑘_𝑚²=𝑀₁𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊², (A.19) where 𝑀₁=𝛽𝐽𝑀₀max{1,4𝑀⁴₀}. This proves (A.5).
The proof of (A.6) in Lemma A.2: using the Taylor expansion, we get that Ψ^𝐶_{𝑘,𝑗,𝑚}=ℎ^𝐶_{𝑘+1,𝑗,𝑚}−ℎ^𝐶_{𝑘,𝑗,𝑚}=12𝑓𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑓𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}+𝑓𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑓𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}=12𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}+𝑓^′′𝑡³_{𝑘,𝑗,𝑚}𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}×𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}+𝑓^′′𝑡⁴_{𝑘,𝑗,𝑚}𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}², (A.20) where 𝑡³_{𝑘,𝑗,𝑚} is on the segment between 𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚} and 𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}, 𝑡⁴_{𝑘,𝑗,𝑚} is on the segment between 𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚} and 𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}. By (A.3), (A.16), we deduce that 𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}=Δ𝑊^𝑘_𝑚𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚Δ𝑊^𝑘_𝑚+𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗=𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚−𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗,𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}²=Δ𝑊^𝑘_𝑚𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚Δ𝑊^𝑘_𝑚𝑥^𝑅_𝑗²=𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚², (A.21) where 𝜏^𝑘_2,𝑚,𝜏^𝑘_3,𝑚 both lie on the segment between 𝑊_𝑚^𝑘+1 and 𝑊^𝑘_𝑚. Similarly, we can deduce that 𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}=𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+𝜙^′′𝜏^𝑘_4,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗,𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚}−𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}²=𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_5,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚², (A.22) where 𝜏^𝑘_4,𝑚,𝜏^𝑘_5,𝑚 both lie on the segment between 𝑊_𝑚^𝑘+1 and 𝑊^𝑘_𝑚. Combining with (A.20), we have Ψ^𝐶_{𝑘,𝑗,𝑚}=ℎ^𝐶_{𝑘+1,𝑗,𝑚}−ℎ^𝐶_{𝑘,𝑗,𝑚}=12𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚−𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡³_{𝑘,𝑗,𝑚}×𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}×𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+𝜙^′′𝜏^𝑘_4,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁴_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_5,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²=12𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡³_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_4,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁴_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_5,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚². (A.23) By (A.23), we get that 𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗=12^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡³_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_4,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁴_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_5,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²=Δ₁+Δ₂, (A.24) where Δ₁=12^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚,Δ₂=12^𝑀_𝑚=1𝑤^𝑘_0,𝑚−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡³_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_4,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁴_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_5,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚². (A.25)(A.26) This together with (A.25) leads to ^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ₁=12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚. (A.27) This together with (A.26) leads to ^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ₂=12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡³_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_4,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁴_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_5,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚². (A.28) By (A.3), (A.16) and |𝑓^′𝐶_𝑗(Φ^𝐶_0,𝑘,𝑗)|≤2𝑀₀, we have 12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_2,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗≤12𝛽^𝐽𝑀_𝑗=1𝑚=1𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗⋅𝑤^𝑘_0,𝑚⋅𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}⋅𝜙^′′𝜏^𝑘_2,𝑚⋅Δ𝑊^𝑘_𝑚²⋅𝑥^𝑅_𝑗≤12𝛽^𝐽𝑀_𝑗=1𝑚=12𝑀₀⋅𝑀₀⋅𝑀₀⋅32𝜇⋅𝑀₀⋅Δ𝑊^𝑘_𝑚²=32𝜇𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚². (A.29) Similarly, we can obtain that 12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′′𝑡³_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_3,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²≤4𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²,12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_4,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗≤32𝜇𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²,12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′′𝑡⁴_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_5,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²≤4𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚². (A.30) So by (A.28), (A.29), and (A.30), we have ^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ₂≤32𝜇𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²+4𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²+32𝜇𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²+4𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²=3𝜇+8𝑀₀𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚², (A.31) with (A.23), similarly, we get that Ψ^𝑅_{𝑘,𝑗,𝑚}=ℎ^𝑅_{𝑘+1,𝑗,𝑚}−ℎ^𝑅_{𝑘,𝑗,𝑚}=12𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+𝜙^′′𝜏^𝑘_6,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁵_{𝑘,𝑗,𝑚}×𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_7,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}×𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚−𝜙^′′𝜏^𝑘_8,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗−𝑓^′′𝑡⁶_{𝑘,𝑗,𝑚}𝑥^𝑐_𝑗−𝜙^′𝜏^𝑘_9,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²=12𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_6,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁵_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_7,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_8,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗−𝑓^′′𝑡⁶_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_9,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚², (A.32) where 𝜏^𝑘_6,𝑚,𝜏^𝑘_7,𝑚,𝜏^𝑘_8,𝑚,𝜏^𝑘_9,𝑚 lie on the segment between 𝑊_𝑚^𝑘+1 and 𝑊^𝑘_𝑚, 𝑡⁵_{𝑘,𝑗,𝑚} lies on the segment between 𝑠^𝐶_{𝑘+1,𝑗,𝑚}+𝑠^𝑅_{𝑘+1,𝑗,𝑚} and 𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}, 𝑡⁶_{𝑘,𝑗,𝑚} lies on the segment between 𝑠^𝐶_{𝑘+1,𝑗,𝑚}−𝑠^𝑅_{𝑘+1,𝑗,𝑚} and 𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}. By (A.32), we have 𝜑𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗=12^𝑀_𝑚=1𝜑𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_6,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁵_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_7,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_8,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗−𝑓^′′𝑡⁶_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_9,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²=Δ₃+Δ₄, (A.33) where Δ₃=12^𝑀_𝑚=1𝜑𝑤^𝑘_0,𝑚×𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚,Δ₄=12^𝑀_𝑚=1𝜑𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_6,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗+𝑓^′′𝑡⁵_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝜏^𝑘_7,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚²+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝜙^′′𝜏^𝑘_8,𝑚Δ𝑊^𝑘_𝑚²𝑥^𝑅_𝑗−𝑓^′′𝑡⁶_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝜏^𝑘_9,𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚². (A.34)(A.35) By (A.34), we have 1−𝛽𝑓^𝐽_𝑗=1^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗Δ₃=12^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗^𝑀_𝑚=1𝜑𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚, (A.36) with (A.31), similarly, this together with (A.35) leads to 1−𝛽𝑓^𝐽_𝑗=1^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗Δ₄≤32𝜇1−𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²+41−𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²+32𝜇1−𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²+41−𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²=3𝜇+8𝑀₀1−𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚². (A.37) By (A.27), (A.31), (A.36) and (A.37), we obtain that ^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗+^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗𝜑𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗=^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ₁+Δ₂+^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗Δ₃+Δ₄≤12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+12^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗^𝑀_𝑚=1𝜑𝑤^𝑘_0,𝑚×𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+3𝜇+8𝑀₀𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²+3𝜇+8𝑀₀1−𝛽𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²=12^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗^𝑀_𝑚=1𝑤^𝑘_0,𝑚𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗+𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗×Δ𝑊^𝑘_𝑚+12^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗^𝑀_𝑚=1𝜑𝑤^𝑘_0,𝑚×𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}+𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗+𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗−𝑓^′𝑠^𝐶_{𝑘,𝑗,𝑚}−𝑠^𝑅_{𝑘,𝑗,𝑚}𝑥^𝐶_𝑗−𝜙^′𝑊^𝑘_𝑚𝑥^𝑅_𝑗Δ𝑊^𝑘_𝑚+3𝜇+8𝑀₀𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚². (A.38) Combining with (4.11), (4.12), and (4.14), we get that ^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗+^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗𝜑𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗≤^𝑀_𝑚=1𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚⋅Δ𝑊^𝑘_𝑚+3𝜇+8𝑀₀𝐽𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²=−𝜂^𝑀_𝑚=1𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚²+𝑀₂𝜂^2𝑀_𝑚=1𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚²=−𝜂+𝑀₂𝜂^2𝑀_𝑚=1𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚², (A.39) where 𝑀₂=((3/𝜇)+8𝑀₀)𝐽𝑀⁴₀. This proves (A.6).
The proof of (A.7) in Lemma A.2: According to the definition of 𝑓^𝐶_𝑗(𝑡), we get that 𝑓_𝑗^′′𝐶(𝑡)=1, combining with (A.3), (A.18), we have 12^𝐽_𝑗=1𝛽𝑓_𝑗^′′𝐶𝜉^𝐶_0,𝑘,𝑗Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗²=12𝛽^𝐽_𝑗=1Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗²=12𝛽^𝐽_𝑗=1𝑊₀^𝑘+1⋅ℎ^𝐶_𝑘+1,𝑗−𝑊^𝑘₀⋅ℎ^𝐶_𝑘,𝑗²=12𝛽^𝐽_𝑗=1𝑊₀^𝑘+1−𝑊^𝑘₀⋅ℎ^𝐶_𝑘+1,𝑗+𝑊^𝑘₀⋅ℎ^𝐶_𝑘+1,𝑗−ℎ^𝐶_𝑘,𝑗²≤𝛽^𝐽_𝑗=1𝑀²₀Δ𝑊^𝑘₀²+𝑀²₀Ψ^𝐶_𝑘,𝑗²≤𝛽𝐽𝑀²₀Δ𝑊^𝑘₀²+4𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²≤𝑀₃^𝑀_𝑚=0Δ𝑊^𝑘_𝑚²=𝑀₃𝜂^2𝑀_𝑚=0𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚²=𝑀₃𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊², (A.40) where 𝑀₃=𝛽𝐽𝑀²₀max{1,4𝑀⁴₀}. This proves (A.7).
The proof of (A.8) in Lemma A.2: With (A.17), similarly, for any 1≤𝑗≤𝐽 and 𝑘=0,1,2,…, we can get that Ψ^𝑅_𝑘,𝑗²≤4𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚². (A.41) According to the definition of 𝑓^𝑅_𝑗(𝑡), we get that 𝑓^′𝑅_𝑗(𝑡)=𝑡^𝑅_𝑗−𝑜^𝑅_𝑗, combining with (A.3), we can obtain that |𝑓^′𝑅_𝑗(Φ^𝑅_0,𝑘,𝑗)|≤2𝑀₀. By (A.16) and (A.41), we deduce that 1−𝛽𝑓^𝐽_𝑗=1^′𝑅_𝑗Φ^𝑅_0,𝑘,𝑗𝜑^′𝜁^𝑘₁Δ𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗≤21−𝛽𝑀₀^𝐽_𝑗=1Δ𝑊^𝑘₀Ψ^𝑅_𝑘,𝑗≤1−𝛽𝑀₀^𝐽_𝑗=1Δ𝑊^𝑘₀²+Ψ^𝑅_𝑘,𝑗²≤1−𝛽𝐽𝑀₀Δ𝑊^𝑘₀²+41−𝛽𝐽𝑀⁵₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²≤𝑀₄^𝑀_𝑚=0Δ𝑊^𝑘_𝑚²=𝑀₄𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊², (A.42) where 𝑀₄=(1−𝛽)𝐽𝑀₀max{1,4𝑀⁴₀}. This proves (A.8).
The proof of (A.9) in lemma A.2: By |𝑓^′𝑅_𝑗(Φ^𝑅_0,𝑘,𝑗)|≤2𝑀₀, (A.3) and (A.16), we get that 12^𝐽_𝑗=11−𝛽𝑓^′𝑅_𝑗𝜙^𝑅_0,𝑘,𝑗𝜑^′′𝜁^𝑘₂Δ𝑊^𝑘₀²⋅ℎ^𝑅_𝑘,𝑗≤1−𝛽𝑀₀⋅32𝜇^𝐽_𝑗=1Δ𝑊^𝑘₀²⋅ℎ^𝑅_𝑘,𝑗≤32𝜇1−𝛽𝐽𝑀²₀Δ𝑊^𝑘₀²≤𝑀₅𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊₀², (A.43) where 𝑀₅=(3/2𝜇)(1−𝛽)𝐽𝑀²₀. This proves (A.9).
The proof of (A.10) in Lemma A.2: According to the definition of 𝑓^𝑅_𝑗(𝑡), we get that 𝑓_𝑗^′′𝑅(𝑡)=1, combining with (A.3) and (A.41), we have 12^𝐽_𝑗=11−𝛽𝑓_𝑗^′′𝑅𝜉^𝑅_0,𝑘,𝑗Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗²=121−𝛽^𝐽_𝑗=1Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗²=121−𝛽^𝐽_𝑗=1𝜑𝑊₀^𝑘+1⋅ℎ^𝑅_𝑘+1,𝑗−𝜑𝑊^𝑘₀⋅ℎ^𝑅_𝑘,𝑗²=121−𝛽^𝐽_𝑗=1𝜑𝑊₀^𝑘+1−𝜑𝑊^𝑘₀⋅ℎ^𝑅_𝑘+1,𝑗+𝜑𝑊^𝑘₀⋅ℎ^𝑅_𝑘+1,𝑗−ℎ^𝑅_𝑘,𝑗²≤1−𝛽^𝐽_𝑗=1𝑀²₀Δ𝑊^𝑘₀²+𝑀²₀Ψ^𝑅_𝑘,𝑗²≤1−𝛽𝐽𝑀²₀Δ𝑊^𝑘₀²+4𝑀⁴₀^𝑀_𝑚=1Δ𝑊^𝑘_𝑚²≤𝑀₆^𝑀_𝑚=0Δ𝑊^𝑘_𝑚²=𝑀₆𝜂^2𝑀_𝑚=0𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚²=𝑀₆𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊², (A.44) where 𝑀₆=(1−𝛽)𝐽𝑀²₀max{1,4𝑀⁴₀}. This proves (A.10). Thus this completes the proof of Lemma A.2.

Now we are ready to prove Theorem 5.1.

Proof. Using the Taylor expansion and Lemma A.2, for any 𝑘=0,1,2,…, we have 𝐸𝑊^𝑘+1−𝐸𝑊^𝑘=^𝐽_𝑗=1𝛽𝑓^𝐶_𝑗Φ^𝐶_{0,𝑘+1,𝑗}+1−𝛽𝑓^𝑅_𝑗Φ^𝑅_{0,𝑘+1,𝑗}−𝛽𝑓^𝐶_𝑗Φ^𝐶_0,𝑘,𝑗−1−𝛽𝑓^𝑅_𝑗Φ^𝑅_0,𝑘,𝑗=^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗+12𝛽𝑓_𝑗^′′𝐶𝜉^𝐶_0,𝑘,𝑗Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗²+1−𝛽𝑓^′𝑅_𝑗Φ^𝑅_0,𝑘,𝑗Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗+121−𝛽𝑓_𝑗^′′𝑅𝜉^𝑅_0,𝑘,𝑗Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗²=^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗𝑊₀^𝑘+1ℎ^𝐶_𝑘+1,𝑗−𝑊^𝑘₀ℎ^𝐶_𝑘,𝑗+12𝛽𝑓_𝑗^′′𝐶𝜉^𝐶_0,𝑘,𝑗Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗²+1−𝛽𝑓^′𝑅_𝑗Φ^𝑅_0,𝑘,𝑗𝜑𝑊₀^𝑘+1ℎ^𝑅_𝑘+1,𝑗−𝜑𝑊^𝑘₀ℎ^𝑅_𝑘,𝑗+121−𝛽𝑓_𝑗^′′𝑅𝜉^𝑅_0,𝑘,𝑗Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗²=^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ𝑊^𝑘₀⋅ℎ^𝐶_𝑘,𝑗+𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗+Δ𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗+12𝛽𝑓_𝑗^′′𝐶𝜉^𝐶_0,𝑘,𝑗Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗²+1−𝛽𝑓^′𝑅_𝑗Φ^𝑅_0,𝑘,𝑗×𝜑𝑊₀^𝑘+1−𝜑𝑊^𝑘₀⋅ℎ^𝑅_𝑘,𝑗+𝜑𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗+𝜑𝑊₀^𝑘+1−𝜑𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗+121−𝛽𝑓_𝑗^′′𝑅𝜉^𝑅_0,𝑘,𝑗Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗²=^𝐽_𝑗=1𝛽𝑓^′𝐶_𝑗Φ^𝐶_0,𝑘,𝑗Δ𝑊^𝑘₀⋅ℎ^𝐶_𝑘,𝑗+𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗+Δ𝑊^𝑘₀⋅Ψ^𝐶_𝑘,𝑗+12𝛽𝑓_𝑗^′′𝐶𝜉^𝐶_0,𝑘,𝑗Φ^𝐶_{0,𝑘+1,𝑗}−Φ^𝐶_0,𝑘,𝑗²+1−𝛽𝑓^′𝑅_𝑗Φ^𝑅_0,𝑘,𝑗×𝜑^′𝑊^𝑘₀Δ𝑊^𝑘₀+𝜑^′′𝜁^𝑘₂Δ𝑊^𝑘₀²⋅ℎ^𝑅_𝑘,𝑗+𝜑𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗+𝜑^′𝜁^𝑘₁Δ𝑊^𝑘₀⋅Ψ^𝑅_𝑘,𝑗+121−𝛽𝑓_𝑗^′′𝑅𝜉^𝑅_0,𝑘,𝑗Φ^𝑅_{0,𝑘+1,𝑗}−Φ^𝑅_0,𝑘,𝑗²𝑊≤−𝜂𝜕𝐸^𝑘𝜕𝑊₀²+−𝜂+𝑀₂𝜂^2𝑀_𝑚=1𝜕𝐸𝑊^𝑘𝜕𝑊_𝑚²+𝑀₁𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊²+𝑀₄𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊²+𝑀₅𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊₀²+𝑀₃𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊²+𝑀₆𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊²≤−𝜂−𝑀₇𝜂²𝜕𝐸𝑊^𝑘𝜕𝑊², (A.45) where 𝑀₇=𝑀₁+𝑀₂+𝑀₃+𝑀₄+𝑀₅+𝑀₆, 𝜉^𝐶_0,𝑘,𝑗 lies on the segment between Φ^𝐶_{0,𝑘+1,𝑗} and Φ^𝐶_0,𝑘,𝑗, 𝜉^𝑅_0,𝑘,𝑗 lies on the segment between Φ^𝑅_{0,𝑘+1,𝑗} and Φ^𝑅_0,𝑘,𝑗, 𝜁^𝑘₁,𝜁^𝑘₂ both lie on the segment between 𝑊₀^𝑘+1 and 𝑊^𝑘₀. Let 𝛾=𝜂−𝑀₇𝜂², then 𝐸𝑊^𝑘+1≤𝐸𝑊^𝑘𝑊−𝛾𝜕𝐸^𝑘𝜕𝑊². (A.46) Obviously, we require the learning rate 𝜂 to satisfy 10<𝜂<𝑀₇. (A.47) Thus, we can obtain that 𝐸𝑊^𝑘+1≤𝐸𝑊^𝑘,𝑘=0,1,2,…. (A.48) This together with (A.46) leads to 𝐸𝑊^𝑘+1≤𝐸𝑊^𝑘𝑊−𝛾𝜕𝐸^𝑘𝜕𝑊²≤⋯≤𝐸𝑊⁰−𝛾^𝑘_𝑡=0𝜕𝐸𝑊^𝑡𝜕𝑊². (A.49) Since 𝐸(𝑊^𝑘+1)≥0, we have 𝛾^𝑘_𝑡=0≤𝐸𝑊⁰. (A.50) Letting 𝑘→∞ results in 𝜕𝐸𝑊^∞_𝑡=0^𝑡𝜕𝑊²≤1𝛾𝐸𝑊⁰<∞. (A.51) So this immediately gives lim_𝑘→∞𝜕𝐸𝑊^𝑘𝜕𝑊²=0. (A.52) According to (4.14) and (A.52), we get that lim_𝑘→∞Δ𝑊^𝑘=0. (A.53)
According to (𝐴1), the sequence {𝐰^𝑚}(𝑚∈ℕ) has a subsequence {𝐰^𝑚𝑘}(𝑘∈ℕ) that is convergent to, say, 𝐰^∗∈Ω₀. It follows from (5.3) and the continuity of 𝐸_𝐰(𝐰) that 𝐸_𝐰𝐰^∗=lim_𝑘→∞𝐸_𝐰𝐰^𝑚𝑘=lim_𝑚→∞𝐸_𝐰𝐰^𝑚=0. (A.54) This implies that 𝐰^∗ is a stationary point of 𝐸(𝐰). Hence, {𝐰^𝑚} has at least one accumulation point and every accumulation point must be a stationary point.
Next, by reduction to absurdity, we prove that {𝐰^𝑚} has precisely one accumulation point. Let us assume the contrary that {𝐰^𝑚} has at least two accumulation points 𝐰≠𝐰^∼. We write 𝐰^𝑚=(𝑤^𝑚₁,𝑤^𝑚₂,…,𝑤^𝑚_𝑛(𝑝+1))^𝑇. It is easy to see from (4.13) and (4.14) that lim_𝑚→∞‖𝐰^𝑚+1−𝐰^𝑚‖=0, or equivalently, lim_𝑚→∞|𝑤_𝑖^𝑚+1−𝑤^𝑚_𝑖|=0 for 𝑖=1,2,…,𝑛(𝑝+1). Without loss of generality, we assume that the first components of 𝐰 and 𝐰^∼ do not equal to each other, that is, 𝑤₁≠𝑤₁. For any real number 𝜆∈(0,1), let 𝑤^𝜆₁=𝜆𝑤₁+(1−𝜆)𝑤₁. By Lemma A.1, there exists a subsequence {𝑤^𝑚𝑘1₁} of {𝑤^𝑚₁} converging to 𝑤^𝜆₁ as 𝑘₁→∞. Due to the boundedness of {𝑤^𝑚𝑘1₂}, there is a convergent subsequence {𝑤^𝑚𝑘2₂}⊂{𝑤^𝑚𝑘1₂}. We define 𝑤^𝜆₂=lim_𝑘2→∞𝑤^𝑚𝑘2₂. Repeating this procedure, we end up with decreasing subsequences {𝑚_𝑘1}⊃{𝑚_𝑘2}⊃⋯⊃{𝑚_{𝑘𝑛(𝑝+1)}} with 𝑤^𝜆_𝑖=lim_{𝑘𝑖→∞}𝑤^𝑚𝑘𝑖_𝑖 for each 𝑖=1,2,…,𝑛(𝑝+1). Write 𝐰^𝜆=(𝑤^𝜆₁,𝑤^𝜆₂,…,𝑤^𝜆_𝑛(𝑝+1))^𝑇. Then, we see that 𝐰^𝜆 is an accumulation point of {𝐰^𝑚} for any 𝜆∈(0,1). But this means that Ω_0,1 has interior points, which contradicts (𝐴4). Thus, 𝐰^∗ must be a unique accumulation point of {𝐰^𝑚}^∞_𝑚=0. This proves (5.4). Thus this completes the proof of Theorem 5.1.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (11171367) and the Fundamental Research Funds for the Central Universities of China.