Abstract
A class of Soblove type multivariate function is approximated by feedforward network with one hidden layer of sigmoidal units and a linear output. By adopting a set of orthogonal polynomial basis and under certain assumptions for the governing activation functions of the neural network, the upper bound on the degree of approximation can be obtained for the class of Soblove functions. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.
1. Introduction
Artificial neural networks have been extensively applied in various fields of science and engineering. Why is so mainly because the feedforward neural networks (FNNs) have the universal approximation capability [1β13]. A typical example of such universal approximation assertions states that, for any given continuous function defined on a compact set of , there exists a three-layer of FNN so that it can approximate the function arbitrarily well. A three-layer of FNN with one hidden layer, inputs and one output can be mathematically expressed as where ,ββ are the thresholds, are connection weights of neuron in the hidden layer with the input neurons, are the connection strength of neuron with the output neuron, and is the activation function used in the network. The activation function is normally taken as sigmoid type; that is, it satisfies as and as . Equation (1.1) can be further expressed in vector form as
Universal approximation capabilities for a broad range of neural network topologies have been established by researchers like Cybenko [1], Ito [5], and T. P. Chen and H. Chen [6]. Their work concentrated on the question of denseness. But from the point of application, we are concerned about the degree of approximation by neural networks.
For any approximation problem, the establishment of performance bounds is an inevitable but very difficult issues. As we know, feedforward neural networks (FNNS) have been shown to be capable of approximating general class of functions, including continuous and integrable ones. Recently, several researchers have been derived approximation error bounds for various functional classes (see, e.g., [7β13]) approximated by neural networks. While many open issues remain concerning approximation degree, we stress in this paper on the issue of approximation of functions defined over by FNNS. In [10], the researcher took some basics tools from the theory of weighted polynomial of functions (The weight function is ), under certain assumptions on the smoothness of functions being approximated and on the activation functions in the neural network, the authors present upper bounds on the degree of approximation achieved over the domain .
In this paper, using the Chebyshev Orthogonal series from the approximation theory and moduli of continuity, we obtain upper bounds on the degree of approximation in . We take advantage of the properties of the Chebyshev polynomial and the methods of paper [10], we yield the desired results, which can be easily extended to the space .
2. Multivariate Chebyshev Polynomial Approximation
Before introducing the main results, we firstly introduce some basic results on Chebyshev polynomials from the approximation theory. For convenience, we introduce a weighted norm of a function [14] given by where , is multivariate weighted function, , , , . We denote the class of functions for which is finite by .
For function , the class of functions we wish to approximate in this work is defined as follows: where , , , is a natural number, and .
2.1. A Chebyshev Polynomial Approximation of Multivariate Functions
As we know, Chebyshev polynomial of a single real variable is a very important polynomial in approximation theory. Using the above notation, we introduce multivariate Chebyshev polynomials: , , . Evidently, for any , , we have
For , , let , then we have the orthogonal expansion .
For one-dimension degree of approximation of a function by polynomials of degree , one has the following: where stands for the class of degree-m algebraic polynomials. From [15], we have a simple relationship which we will be used in the following. Let be differentiable, then we have
Let , and the de la Valle Poussin Operators is defined, that is,
Furthermore, we can simplify as follows: where
A basic result concerning Valle Poussin Operators is
Now we consider a class of multivariate polynomials defined as follows:
Hence, we have the following theorem.
Theorem 2.1. For , let . Then for any , , we have
Proof. We consider the Chebyshev orthogonal polynomials , and obtain the following equality from (2.7): where . Hence, we define the following operators: where . Then we have where is the identity operator. Let , then , . We view as a one-dimensional function . Using (2.4), (2.5), and (2.6), we have Letting ,ββ,ββ, if , we get from (2.15), (2.13), (2.14) and the inequality , In order to obtain a bound valid for all , for , we always have the trivial bound since . Letting , we conclude an inequality of the desired type for every .
This theorem reveals two things: (i) for any multivariate functions , there is a polynomial that approximates arbitrarily well in , (ii) quantitatively, the approximation accuracy of a polynomial can attain the order of , where is the dimension of multivariate polynomial, and is the smoothness of the function to be approximated.
3. Approximation by Feedforward Neural Networks
We consider the approximation of functions by feedforward neural networks with a ridge functions. We define the approximating function class composed of a single hidden layer feedforward neural network with hidden units. The class of function is where satisfy the following assumptions.(1)There is a constant such that ,ββ(2)For each finite , there is a finite constant such thatββ.
We define the distance from to as where , are two sets in . We have the following results.
Theorem 3.1. Let condition (1) and (2) hold for the activation function . Then for every , , , and , we have where
Proof. Firstly, we consider the partial derivative
where , and . Thus .
For any fixed and (here ), we consider a finite difference of orders
where ,ββ with , So
where we derive (3.7) by using (3.6), the mean value theorem of integral, (i.e., there is a , such that ) and the moduli of continuity .
From the definition ofββ and (3.7), we have
The last step can be made arbitrarily small by letting .
Using the Theorems 2.1 and 3.1, we can easily establish our final result.
Theorem 3.2. For , we have
This theorem reveals two things: (i) for any multivariate functions , there is a single hidden layer feedforward neural network with hidden units that approximates arbitrarily well in . That is, the feedforward neural networks can be used as the universal approximator of functions in ; (ii) quantitatively, the approximation accuracy of a mixture network of the form (3.1) can attain the order of , where is the dimension of input space, and is the smoothness of the function to be approximated.
4. Conclusion
In this work, the approximation order of feedforward neural networks with the form (3.1) has been studied. In terms of smoothness of a function, an upper bound estimation on approximation precision and speed of the neural networks is developed. Our research reveals that the approximation precision and speed of the neural networks depend not only on the number of hidden neurons used, but also on the smoothness of the functions to be approximated. The results obtained are helpful in understanding the approximation capability and topology construction of the sigmoidal neural networks.
Acknowledgments
This research is supported by Natural Science Foundation of China (no. 11001227), Natural Science Foundation Project of CQ CSTC (no. CSTC,2009BB2306), and the Fundamental Research Funds for the Central Universities (no. XDJK2010B005).