Estimation of Approximating Rate for Neural Network in 𝐿^𝑝𝑤 Spaces

Abstract

A class of Soblove type multivariate function is approximated by feedforward network with one hidden layer of sigmoidal units and a linear output. By adopting a set of orthogonal polynomial basis and under certain assumptions for the governing activation functions of the neural network, the upper bound on the degree of approximation can be obtained for the class of Soblove functions. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.

1. Introduction

Artificial neural networks have been extensively applied in various fields of science and engineering. Why is so mainly because the feedforward neural networks (FNNs) have the universal approximation capability [1–13]. A typical example of such universal approximation assertions states that, for any given continuous function defined on a compact set K of ℛ^𝑑, there exists a three-layer of FNN so that it can approximate the function arbitrarily well. A three-layer of FNN with one hidden layer, 𝑑 inputs and one output can be mathematically expressed as𝒩(𝑥)=^𝑚_𝑖=1𝑐_𝑖𝜎^𝑑_𝑗=1𝑤_𝑖𝑗𝑥_𝑗+𝜃_𝑖,𝑥∈ℛ^𝑑,𝑑≥1,(1.1) where 1≤𝑖≤𝑚,  𝜃_𝑖∈ℛ are the thresholds, 𝑤_𝑖=(𝑤_𝑖1,𝑤_𝑖2,…,𝑤_𝑖𝑑)^𝑇∈ℛ^𝑑 are connection weights of neuron 𝑖 in the hidden layer with the input neurons, 𝑐_𝑖∈ℛ are the connection strength of neuron 𝑖 with the output neuron, and 𝜎 is the activation function used in the network. The activation function is normally taken as sigmoid type; that is, it satisfies 𝜎(𝑡)→1 as 𝑡→+∞ and 𝜎(𝑡)→0 as 𝑡→−∞. Equation (1.1) can be further expressed in vector form as𝒩(𝑥)=^𝑚_𝑖=1𝑐_𝑖𝜎w_𝑖⋅x+𝜃_𝑖,𝑥∈𝑅^𝑑.(1.2)

Universal approximation capabilities for a broad range of neural network topologies have been established by researchers like Cybenko [1], Ito [5], and T. P. Chen and H. Chen [6]. Their work concentrated on the question of denseness. But from the point of application, we are concerned about the degree of approximation by neural networks.

For any approximation problem, the establishment of performance bounds is an inevitable but very difficult issues. As we know, feedforward neural networks (FNNS) have been shown to be capable of approximating general class of functions, including continuous and integrable ones. Recently, several researchers have been derived approximation error bounds for various functional classes (see, e.g., [7–13]) approximated by neural networks. While many open issues remain concerning approximation degree, we stress in this paper on the issue of approximation of functions defined over [−1,1]^𝑑 by FNNS. In [10], the researcher took some basics tools from the theory of weighted polynomial of functions (The weight function is 𝜔(𝑥)=exp(−𝑄(𝑥))), under certain assumptions on the smoothness of functions being approximated and on the activation functions in the neural network, the authors present upper bounds on the degree of approximation achieved over the domain 𝑅^𝑑.

In this paper, using the Chebyshev Orthogonal series from the approximation theory and moduli of continuity, we obtain upper bounds on the degree of approximation in [−1,1]^𝑑. We take advantage of the properties of the Chebyshev polynomial and the methods of paper [10], we yield the desired results, which can be easily extended to the space 𝑅^𝑑.

2. Multivariate Chebyshev Polynomial Approximation

Before introducing the main results, we firstly introduce some basic results on Chebyshev polynomials from the approximation theory. For convenience, we introduce a weighted norm of a function 𝑓 [14] given by‖𝑓‖_𝑝,𝜔=_[]−1,1𝑑𝜔(𝑥)|𝑓(𝑥)|^𝑝𝑑𝑥^1/𝑝,(2.1) where 1≤𝑝<∞, 𝜔(𝑥)=∏^𝑑𝑖=1𝜔(𝑥_𝑖) is multivariate weighted function, 𝜔(𝑥_𝑖)=(1−𝑥^2𝑖)^−1/2, 𝑥=(𝑥₁,𝑥₂,…,𝑥_𝑑)∈𝑅^𝑑, 𝑚=(𝑚₁,𝑚₂,…,𝑚_𝑑)∈𝑍^𝑑, 𝑑𝑥=𝑑𝑥₁𝑑𝑥₂…𝑑𝑥_𝑑. We denote the class of functions for which ‖𝑓‖_𝑝,𝜔 is finite by 𝐿_𝑝,𝜔.

For function 𝑓∶𝑅^𝑑→𝑅, the class of functions we wish to approximate in this work is defined as follows:Ψ^{𝑟,𝑑𝑝,𝜔}=𝑓∶‖‖𝑓^(𝜆)‖‖_𝑝,𝜔≤𝑀,||𝜆||≤𝑟,(2.2) where 𝜆=(𝜆₁,𝜆₂,…,𝜆_𝑑), |𝜆|=𝜆₁+𝜆₂+⋯+𝜆_𝑑, 𝑓^(𝜆)=𝜕^|𝜆|𝑓/𝜕_𝑥1^𝜆1…𝜕_𝑥𝑑^𝜆𝑑, 𝑟 is a natural number, and 𝑀<∞.

2.1. A Chebyshev Polynomial Approximation of Multivariate Functions

As we know, Chebyshev polynomial of a single real variable is a very important polynomial in approximation theory. Using the above notation, we introduce multivariate Chebyshev polynomials: 𝑇₀(𝑥)=1/√𝜋, 𝑇_𝑛(𝑥)=∏^𝑑𝑖=1𝑇_𝑛𝑖(𝑥_𝑖), 𝑇_𝑛𝑖(𝑥_𝑖)=2/√𝜋cos(𝑛_𝑖arccos𝑥_𝑖). Evidently, for any 𝑚, 𝑙∈𝑍^𝑑, we have_[]−1,1𝑑𝑇_𝑚(𝑥)𝑇_𝑙(𝑥)𝜔(𝑥)𝑑𝑥=10𝑚𝑚≠=𝑙𝑙.,(2.3)

For 𝑓∈𝐿_𝑝,𝜔, 𝑚∈𝑍^𝑘, let 𝑓(𝑚)=∫_[−1,1]𝑑𝑓(𝑥)𝑇_𝑚(𝑥)𝜔(𝑥)𝑑𝑥, then we have the orthogonal expansion 𝑓(𝑥)∼∑^∞𝑚=0𝑓(𝑚)𝑇_𝑚(𝑥),𝑥∈[−1,1]^𝑑.

For one-dimension degree of approximation of a function 𝑔 by polynomials of degree 𝑚, one has the following:𝐸_𝑚𝑔,𝑃_𝑚,𝐿_𝑝,𝜔=inf_{𝑃∈𝑃𝑚}‖𝑔−𝑃‖_𝑝,𝜔,(2.4) where 𝑃_𝑚 stands for the class of degree-m algebraic polynomials. From [15], we have a simple relationship which we will be used in the following. Let 𝑔 be differentiable, then we have𝐸𝑔,𝑃_𝑚,𝐸𝐿𝑔_𝑝,𝜔,𝑃≤_𝑚𝑀₁𝑚⁻¹𝐸𝑔^,𝑃_𝑚,𝐿_𝑝,𝜔,,𝐿_𝑝,𝜔≤‖𝑔‖_𝑝,𝜔.(2.5)

Let 𝑆_𝑛(𝑓,𝑡)=∑^{𝑛−1𝑘=1}𝑓(𝑘)𝑇_𝑘(𝑥), and the de la Valle Poussin Operators is defined, that is,𝑉_𝑛(𝑓,𝑡)=1𝑛+1^𝑛+1_{𝑚=𝑛+3/2}𝑆_𝑛(𝑓,𝑡).(2.6)

Furthermore, we can simplify 𝑉_𝑛(𝑓,𝑡) as follows:𝑉_𝑛(𝑓,𝑡)=^𝑛_𝑘=1𝜉_𝑘𝑓(𝑘)𝑇_𝑘(𝑡),(2.7) where𝜉_𝑘=⎧⎪⎨⎪⎩2(𝑚𝑚−+11),if0≤𝑘≤𝑚+3𝑚−𝑘2,𝑚+1,if𝑚+32≤𝑘≤𝑚.(2.8)

A basic result concerning Valle Poussin Operators 𝑉_𝑚(𝑓,𝑡) is𝐸_2𝑚𝑓,𝑃_2𝑚,𝐿_𝑝,𝜔≤‖‖𝑓−𝑉_𝑚𝑓‖‖_𝑝,𝜔≤𝐸_𝑚𝑓,𝑃_𝑚,𝐿_𝑝,𝜔.(2.9)

Now we consider a class of multivariate polynomials defined as follows:𝑃_𝑚=𝑃∶𝑃(𝑥)=_{0≤|𝑖|≤|𝑚|}𝑏_{𝑖1,𝑖2,…,𝑖𝑑}𝑥^𝑖1₁⋯𝑥^𝑖𝑑_𝑑,𝑏_{𝑖1,𝑖2,…,𝑖𝑑}∈𝑅,∀𝑖₁,…,𝑖_𝑑.(2.10)

Hence, we have the following theorem.

Theorem 2.1. For 1≤𝑝≤∞, let 𝑓∈Ψ^{𝑟,𝑑𝑝,𝜔}. Then for any 𝑚=(𝑚₁,𝑚₂,…,𝑚_𝑑), 𝑚_𝑖≤𝑚, we have inf_{𝑃∈𝑃𝑚}‖𝑓−𝑃‖_𝑝,𝜔≤𝐶𝑚^−𝑟.(2.11)

Proof. We consider the Chebyshev orthogonal polynomials 𝑇_𝑚(𝑥), and obtain the following equality from (2.7): 𝑉_{𝑖,𝑚𝑖,}(𝑓)=^𝑚𝑖_𝑠=1𝜉_𝑠𝑓_𝑠,𝑖𝑇_𝑠𝑥_𝑖,(2.12) where 𝑓_𝑠,𝑖=∫_[−1,1]𝑑𝑓(𝑥)𝑇_𝑠(𝑥_𝑖)𝜔(𝑥_𝑖)𝑑𝑥_𝑖. Hence, we define the following operators: 𝑉(𝑓)==𝑉_1,𝑚1𝑚1𝑉_2,𝑚2⋯𝑉_{𝑑,𝑚𝑑}𝑓_𝑠₁₌₁⋯^𝑚𝑑_𝑠_𝑑=1𝜉_𝑠1⋯𝜉_𝑠𝑑𝑓_{𝑠1,…,𝑠𝑑}𝑇_𝑠1𝑥₁⋯𝑇_𝑠𝑑𝑥_𝑑,(2.13) where 𝑓_{𝑠1,…,𝑠𝑑}=∫_[−1,1]𝑑(∏^𝑑𝑖=1𝜔(𝑥_𝑖)𝑇_𝑠𝑖(𝑥_𝑖))𝑓(𝑥)𝑑𝑥. Then we have ‖𝑓−𝑉(𝑓)‖_𝑝,𝜔=‖‖+𝑓𝑉−𝑉_{1,𝑚11,𝑚1}(𝑓)+𝑉_1,𝑚1(𝑓)−𝑉_1,𝑚1𝑉_2,𝑚2(𝑓)𝑉_2,𝑚2(𝑓)−⋯−𝑉(𝑓)‖‖_𝑝,𝜔≤^𝑑_𝑖=1‖‖𝑉₀⋯𝑉_{𝑖−1,𝑚𝑖−1}𝑓−𝑉₀⋯𝑉_{𝑖,𝑚𝑖}𝑓‖‖_𝑝,𝜔,(2.14) where 𝑉₀ is the identity operator. Let 𝑔=𝑉₀⋯𝑉_{𝑖−1,𝑚𝑖−1}𝑓, then 𝑉_{𝑖,𝑚𝑖}𝑔=𝑉₀⋯𝑉_{𝑖,𝑚𝑖}𝑓, 𝑔^𝑟𝑖(𝑥)=𝑉₀⋯𝑉_{𝑖−1,𝑚𝑖−1}𝐷^𝑟𝑖𝑓(𝑥). We view 𝑉_{𝑖,𝑚𝑖}𝑔 as a one-dimensional function 𝑥_𝑖. Using (2.4), (2.5), and (2.6), we have ‖‖𝑔−𝑉_{𝑖,𝑚𝑖}𝑔‖‖_𝑝,𝜔≤≤𝐶𝐶₁₁𝐸_𝑚𝑖𝑔,𝑃_𝑚𝑖,𝐿_𝑝,𝜔𝑀^𝑟𝑖₁𝑚1_𝑖⋯𝑚_𝑖−1𝑟_𝑖+1𝐸^{𝑚𝑖−𝑟𝑖}𝑔^𝑟𝑖≤𝐶₁,𝑃_{𝑚𝑖−𝑟𝑖},𝐿_𝑝,𝜔𝑀^𝑟𝑖₁𝑚1_𝑖⋯𝑚_𝑖−1𝑟_𝑖+1‖𝑔^𝑟𝑖‖_𝑝,𝜔=𝐶_𝑟𝑖𝑚1_𝑖⋯𝑚_𝑖−1𝑟_𝑖+1‖‖𝑉⁰⋯𝑉_{𝑖−1,𝑚𝑖−1}𝐷^𝑟𝑖𝑓‖‖_𝑝,𝜔.(2.15) Letting 𝑟_𝑖=𝑟,  𝑚_𝑖=𝑚,  𝑖=1,…,𝑑, if 𝑚>𝑟(𝑟−1), we get from (2.15), (2.13), (2.14) and the inequality ∏^𝑛𝑖=1(1+𝑎_𝑖)≥1+∑^𝑛𝑖=1𝑎_𝑖,(𝑎_𝑖≥−1), ‖𝑓−𝑉(𝑓)‖_𝑝,𝜔≤𝐶_𝑟𝑑_𝑖=1𝑚1⋯𝑚−1𝑟+1‖𝐷^𝑟𝑓‖_𝑝,𝜔≤𝐶_𝑟𝑑𝑀1=𝐶_𝑟𝑚⋯𝑚−1𝑟+1𝑑𝑀𝑚^−𝑟1−1𝑚⁻¹1−𝑚2⁻¹⋯1−𝑟−1≤𝐶_𝑟𝑚𝑑𝑀𝑚^−𝑟1−𝑟(𝑟2𝑚−1)⁻¹≤2𝑑𝐶_𝑟𝑀𝑚^−𝑟.(2.16) In order to obtain a bound valid for all 𝑚, for 𝑚≤𝑟(𝑟−1), we always have the trivial bound ‖𝑓−𝑉(𝑓)‖_𝑝,𝜔≤𝑀₂ since ‖𝑓‖_𝑝,𝜔≤𝑀₂. Letting 𝐶=max{2𝑑𝐶_𝑟𝑑𝑀,2⁻¹𝑀₂(𝑟(𝑟−1))^𝑟}, we conclude an inequality of the desired type for every 𝑚.

This theorem reveals two things: (i) for any multivariate functions 𝑓∈Ψ^{𝑟,𝑑𝑝,𝜔}, there is a polynomial 𝑃∈𝑃_𝑚 that approximates 𝑓 arbitrarily well in 𝐿^𝑝𝜔, (ii) quantitatively, the approximation accuracy of a polynomial 𝑃∈𝑃_𝑚 can attain the order of ○(𝑚^−𝑟), where 𝑚 is the dimension of multivariate polynomial, and 𝑟 is the smoothness of the function to be approximated.

3. Approximation by Feedforward Neural Networks

We consider the approximation of functions by feedforward neural networks with a ridge functions. We define the approximating function class composed of a single hidden layer feedforward neural network with 𝑛 hidden units. The class of function is𝐹_𝑛=𝑓∶𝑓(𝑥)=^𝑛_𝑘=1𝑑_𝑘𝜙𝑎_𝑘⋅𝑥+𝑏_𝑘;𝑎_𝑘∈𝑅^𝑑,𝑏_𝑘,𝑑_𝑘∈𝑅,𝑘=1,2,…,𝑛,(3.1) where 𝜙(𝑥) satisfy the following assumptions.(1)There is a constant 𝐶_𝜙 such that |𝜙^(𝑘)(𝑥)|≥𝐶_𝜙>0,  𝑘=0,1,…(2)For each finite 𝑘, there is a finite constant 𝑙_𝑘 such that  |𝜙^(𝑘)(𝑥)|≤𝑙_𝑘.

We define the distance from 𝐹 to 𝐺 asdist𝐹,𝐺,𝐿_𝑝,𝜔=sup_𝑓∈𝐹inf_𝑔∈𝐺‖𝑓−𝑔‖_𝑝,𝜔,(3.2) where 𝐹, 𝐺 are two sets in 𝐿^𝑝𝜔. We have the following results.

Theorem 3.1. Let condition (1) and (2) hold for the activation function 𝜙(𝑥). Then for every 0<𝐿<∞, 𝑚=(𝑚₁,𝑚₂,…,𝑚_𝑑)∈𝑍^𝑑+, 𝑚_𝑖≤𝑚,𝜖>0 and 𝑛>(𝑚+1)^𝑑, we have dist𝐵𝑃_𝑚(𝐿),𝐹_𝑛,𝐿_𝑝,𝜔≤𝜖,(3.3) where 𝐵𝑃_𝑚(𝐿)=𝑃∶𝑝(𝑥)=_{0≤𝑠≤𝑚}𝑎_𝑠𝑥^𝑠;max_{0≤𝑠≤𝑚}||𝑎_𝑠||≤𝐿.(3.4)

Proof. Firstly, we consider the partial derivative 𝜙^(𝑠)(𝑤⋅𝑥+𝑏)=𝜕^(|𝑠|)𝜕^𝑠1_𝑤1⋯𝜕^𝑠𝑑_𝑤𝑑(𝜙(𝑤⋅𝑥+𝑏))=𝐱^𝐬𝜙^|𝑠|(𝑤⋅𝑥+𝑏),(3.5) where |𝑠|=𝑠₁+⋯+𝑠_𝑑, and x^𝐬=∏^𝑑𝑖=1𝑥^𝑠𝑖_𝑖. Thus 𝜙^(𝑠)(𝑏)=x^s𝜙^|𝑠|(𝑏).
For any fixed 𝑏 and |𝑥|<∞ (here |𝑥|=∑^𝑑𝑖=1𝑥_𝑖), we consider a finite difference of orders ▵^𝑠ℎ,𝑥𝜙(𝑏)==𝐱_{0≤𝑙≤𝑠𝑠}(−1)^|𝑙|𝐶^𝑙𝑠𝜙(ℎ𝑙⋅𝐱+𝑏)^ℎ0+𝑎⋯^ℎ0𝜙^(|𝑠|)𝑎₁+⋯+𝑎_𝑠1𝑥₁+⋯_{|𝑠|−𝑠𝑑+1}+⋯+𝑎_|𝑠|𝑥_𝑑+𝑏𝑑𝑎₁⋯𝑑𝑎_|𝑠|≐𝐱^𝑠𝐴^ℎ|𝑠|𝜙(𝐱),(3.6) where 𝐶^𝑙𝑠=∏^𝑑𝑖=1𝐶^𝑙𝐢_𝑠𝐢,  ▵^𝑠ℎ,𝑥𝜙(𝑏)∈𝐹_𝑛 with 𝑛=∏^𝑑𝑖=1(1+𝑠_𝑖), So ||𝜙^𝑠(𝑏)−ℎ^−|𝑠|▵^𝑠ℎ,𝑥𝜙(𝑏)||==||||x^𝑠𝜙^|𝑠|(𝑏)−ℎ^−|𝑠|𝐴^𝑠ℎ𝜙^|𝑠|(x)||≤𝐶x^𝑠_𝑠𝜙^|𝑠|(𝑏)−𝜙^|𝑠|(𝑏+𝜂)||𝜔𝜙^|𝑠|,ℎ,(3.7) where we derive (3.7) by using (3.6), the mean value theorem of integral, (i.e., there is a 𝜂∈[0,ℎ|𝑠⋅𝐱|], such that 𝐴^𝑠ℎ𝜙^|𝑠|(𝐱)=ℎ^|𝑠|𝜙^|𝑠|(𝐛+𝜂)) and the moduli of continuity 𝜔(𝑔,ℎ)=sup_|𝑡|≤ℎ|𝑓(𝑥+𝑡)−𝑓(𝑥)|.
From the definition of  dist(𝐹,𝐺,𝐿_𝑝,𝜔) and (3.7), we have dist𝐵𝑃_𝑚(𝐿),𝐹_𝑛,𝐿_𝑝,𝜔^𝑝≤‖‖‖‖_{0≤𝑠≤𝑚}𝑎_𝑠𝑥^𝑠−_{0≤𝑠≤𝑚}𝑎_𝑠▵^𝑠ℎ,𝑥𝜙(𝑏)ℎ^𝑠𝜙^|𝑠|(𝑏)‖‖‖‖^{𝑝𝑝,𝜔}≤(𝑚+1)^𝑑max_{0≤𝑠≤𝑚}||𝑎^𝑠||‖‖‖𝑥^𝑠−▵^𝑠ℎ,𝑥𝜙(𝑏)≤(𝑚+1)_𝑑ℎ^𝑠𝜙^|𝑠|(𝑏)‖‖‖^{𝑝𝑝,𝜔}≤(𝑚+1)_𝑑𝐿max_{0≤𝑠≤𝑚}𝜙^|𝑠|(𝑏)^−𝑝𝜔𝜙^|𝑠|,ℎ𝐿𝐶^𝑝𝜙𝜔𝜙^|𝑠|,ℎ<𝜖.(3.8) The last step 𝜔(𝜙^|𝑠|,ℎ) can be made arbitrarily small by letting ℎ→0.
Using the Theorems 2.1 and 3.1, we can easily establish our final result.

Theorem 3.2. For 1≤𝑝≤∞, we have distΨ^{𝑟,𝑑𝑝,𝜔},𝐹_𝑛,𝐿_𝑝,𝜔≤𝐶𝑛^−𝑟/𝑑.(3.9)

This theorem reveals two things: (i) for any multivariate functions 𝑓∈Ψ^{𝑟,𝑑𝑝,𝜔}, there is a single hidden layer feedforward neural network 𝑁∈𝐹_𝑛 with 𝑛 hidden units that approximates 𝑓 arbitrarily well in 𝐿^𝑝𝜔. That is, the feedforward neural networks can be used as the universal approximator of functions in Ψ^{𝑟,𝑑𝑝,𝜔}; (ii) quantitatively, the approximation accuracy of a mixture network of the form (3.1) can attain the order of ○(𝑛^−𝑟/𝑑), where 𝑑 is the dimension of input space, and 𝑟 is the smoothness of the function to be approximated.

4. Conclusion

In this work, the approximation order of feedforward neural networks with the form (3.1) has been studied. In terms of smoothness of a function, an upper bound estimation on approximation precision and speed of the neural networks is developed. Our research reveals that the approximation precision and speed of the neural networks depend not only on the number of hidden neurons used, but also on the smoothness of the functions to be approximated. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.

Acknowledgments

This research is supported by Natural Science Foundation of China (no. 11001227), Natural Science Foundation Project of CQ CSTC (no. CSTC,2009BB2306), and the Fundamental Research Funds for the Central Universities (no. XDJK2010B005).