Research Article  Open Access
Tahir Farooq, Aziz Guergachi, Sridhar Krishnan, "KnowledgeBased Green's Kernel for Support Vector Regression", Mathematical Problems in Engineering, vol. 2010, Article ID 378652, 16 pages, 2010. https://doi.org/10.1155/2010/378652
KnowledgeBased Green's Kernel for Support Vector Regression
Abstract
This paper presents a novel prior knowledgebased Green's kernel for support vector regression (SVR). After reviewing the correspondence between support vector kernels used in support vector machines (SVMs) and regularization operators used in regularization networks and the use of Green's function of their corresponding regularization operators to construct support vector kernels, a mathematical framework is presented to obtain the domain knowledge about magnitude of the Fourier transform of the function to be predicted and design a prior knowledgebased Green's kernel that exhibits optimal regularization properties by using the concept of matched filters. The matched filter behavior of the proposed kernel function makes it suitable for signals corrupted with noise that includes many real world systems. We conduct several experiments mostly using benchmark datasets to compare the performance of our proposed technique with the results already published in literature for other existing support vector kernel over a variety of settings including different noise levels, noise models, loss functions, and SVM variations. Experimental results indicate that knowledgebased Green's kernel could be seen as a good choice among the other candidate kernel functions.
1. Introduction
Over the last decade support vector machines (SVMs) have been reported by several studies [1–4] to perform equal or better than other learning machines such as neural networks for the problem of learning from finite dataset and approximating a given function from sparse data. Vapnik [1, 2, 5, 6] has laid down the theoretical foundations of the structural risk minimization (SRM) principle to comprehend the problem of learning from a finite set of data in the context of regularization theory given by Tikhonov [7, 8]. SRM principle provides a connection between capacity of the hypothesis space that contains the learning models to approximate the given function and the size of the training set. Generally, the smaller the size of the training set is, the lower the capacity of the hypothesis space should be to avoid overfitting [1, 2, 6, 9]. This motivates one to understand SVM in the context of regularization theory and find a linear solution in the kernel space that minimizes a certain loss function while keeping capacity of the hypothesis space as small as possible. The kernel function in SVM provides a nonlinear mapping from input space to a higherdimensional feature space. Research studies [10, 11] have affirmed that SVM's regularization properties are associated with the choice of kernel function used for mapping. In the literature, Babu et al. [12] proposed local kernel based color modeling for visual tracking. Maclin et al. [13] presented a method for incorporating and refining domain knowledge for support vector machines via successive linear programming. Yen et al. [14] used kernelbased clustering methods for detecting clusters in weighted, undirected graphs. Toma [15] proposed nonlinear differential equations capable of generating continuous functions similar to pulse sequence for modeling time series. M. Li and J.Y. Li [16] introduced a generalized meansquare error (MSE) to address the predictability of longrange dependent (LRD) series. Bakhoum and Toma [17] presented an extension to the Fourier/Laplace transform for the analysis of signals that are represented by traveling wave equations and offered a mathematical technique for the simulation of the behavior of large systems of optical oscillators. Liu [18] gave analysis of chaotic, dynamic time series events. Poggio and Girosi [19, 20] described a general learning approach using regularization theory. Girosi et al. [21, 22] have provided a unified framework for regularization networks and learning machines. Evgeniou et al. [9], Smola and Schölkopf [23], and Williamson et al. [24] demonstrated a correspondence between regularization networks (RNs) and support vector machines (SVMs). Smola et al. [11] and Scholkopf and Smola [10] have shown a connection between regularization operators used in regularization networks and support vector kernels and presented a method of using Green's functions of their corresponding regularization operators to construct support vector kernels with equivalent regularization properties. However, the problem of choosing the optimal regularization operator to construct the corresponding SV kernel for a given training set still remains unanswered. The work presented herein is focused on using prior knowledge about the magnitude spectrum of the function to be predicted to design the support vector kernels from Green's functions having suitable regularization properties by utilizing the concept of matched filters, an idea inspired by Scholkopf and Smola [10]. The intuition of matching Green's kernel comes from the fact that most real world systems are inevitably contaminated with noise in addition to their intrinsic dynamics [25, 26] and matched filters are known to be the optimal choice to recover signals in the presence of additive white noise [27, 28]. However, no mathematical justification is given in the literature for the use of matched filter theorem to obtain the matching Green's kernel. No experimental results are so far available in the literature to compare the performance of knowledgebased Green's kernel with existing support vector kernels. In this paper, we provide a mathematical framework for utilizing the matched filter theorem to design knowledgebased Green's kernel and conduct experiments on different datasets (mostly benchmarks) with different levels and models (Gaussian and Uniform) of additive white noise to evaluate the performance of our proposed kernel function. Although the assumption of additive white noise will not exactly hold in all real world cases, we keep up with the timehonored tradition [2–4, 25, 26, 29, 30] of using benchmark datasets with additive white noise assumption to evaluate the performance of our proposed method. The focus is on support vector regression (SVR). The rest of the paper is organized as follows. Section 2 reviews the theory of support vector regression, SV kernels, regularization networks, and the connection between support vector method and the theory of regularization networks. Section 3 provides the theory of Green's functions and how they can be used to construct SV kernels. Section 4 describes the theory of matched filters and lays down the mathematical foundation for building knowledgebased Green's kernel. Section 5 presents the experimental results and Section 6 concludes the paper.
2. Support Vector Machines and Regularization Networks
Support vector machines introduced by Vapnik and coworkers for pattern recognition and regression estimation tasks have been reported to be an effective method during the last decade [31–33]. Initially developed for classification problems, a generalization of support vector (SV) algorithm known as insensitive SV regression [1, 33] was derived to solve the problems where the function to be estimated belongs to the set of real numbers.
2.1. Insensitive Support Vector Regression
Suppose that we have as the training set with and , where are the training targets. The problem of calculating an estimate of for training data can be formulated as The goal of insensitive SV algorithm is to calculate an estimate of by selecting the optimal hyperplane and bias such that is at the most distance from while keeping the norm of the hyperplane minimum. The corresponding quadratic optimization problem can be written in terms of regularized risk functional as described by [10, 11], that is, to minimize where is the regularized risk functional, is the regularization constant such that , and the second term on the righthand side of (2.2) is the empirical risk functional with Vapnik's insensitive loss function [1, 2, 34]. By introducing the slack variables, in the sense of [1, 2, 34] and rewriting the problem in (2.2), we get, that is, to minimize subject to In order to obtain the SV expansion of the function , we use the standard [1, 2, 34, 35] Lagrangian technique to form the objective function and the corresponding constraints. The well known formulation of the quadratic optimization problem can be reached by taking the partial derivatives of the objective function, putting them equal to zero for optimal solution and substituting the values obtained into the objective function. We follow the lines of [11, 23] and write the quadratic optimization problem as subject to This leads to the wellknown formulation of SV regression, that is, Comparing (2.7) and (2.1), it shows that the training examples that lie inside the tube contribute to a sparse expansion of because the corresponding Lagrange multipliers are zero [1, 2, 10, 34, 35].
The expression given by (2.7) corresponds to linear SV regression. In order to obtain nonlinearity, SV algorithms can be quipped with nonlinear operators mapping from input space into a highdimensional feature space, as described in [36, 37]. The kernel function is defined as According to Mercer theorem, the kernel is any continuous and symmetric function that satisfies the positivity condition given by (2.9). Such a function defines a dot product in the feature space given by (2.8) [36].
Hence, by making use of (2.8), (2.7) can be written as
2.2. Regularization Networks
The idea of regularization method was first given by Tikhonov [8] and Tikhonov and Arsenin [7] for the solution of illposed problems. Assume that we have a finite dataset , independently and identically drawn from a probability distribution in the presence of noise. Assume that the probability distribution is unknown. One way of approaching the problem is to estimate the function by minimizing a certain empirical risk functional: The problem of approaching the solution through minimizing (2.11) is illposed because the solution is unstable [2]. Hence, the solution is to utilize the idea proposed by [7, 8] and add a capacity control or stabilizer [22] term to (2.11) and minimize a regularized risk functional: where is a linear, positive semidefinite regularization operator. The first term in (2.12) corresponds to finding a function that is as close to the data examples as possible in terms of Vapniks insensitive loss function whereas the second term is the smoothness functional and its purpose is to restrict the size of the functional space and to reduce the complexity of the solution [23]. The filter properties of are given by , where represents the complex conjugate. Following the lines of [11, 23], the problem of minimizing the regularized risk functional given by (2.12) can be transformed into constrained optimization problem by utilizing the standard Lagrange multipliers technique and a formulation similar to (2.5) can be obtained where a direct relationship between SV technique and the RN method can be observed. In other words, training an SVM with a kernel function obtained from the regularization operator is equivalent to implementing RN to minimize the regularized risk functional given by (2.12) with as the regularization operator. We refer the reader to [2, 11, 23] for the detailed discussion on the relationship between the two methods, that is, SV and RN.
3. Green's Functions and Support Vector Kernels
The idea of Green's functions was introduced in the context of solving inhomogeneous differential equations with boundary conditions. However, Green's functions of their corresponding regularization operators can be used to design kernel functions that exhibit the regularization properties given by their corresponding regularization operators, satisfy the Mercer condition, and qualify to be SV kernels [11, 23, 35]. Green's kernel of a discrete regularization operator can be written as [38] where are the basis of the orthonormal eigenvectors of corresponding to nonzero eigenvalues such that confers the spectrum of . The expression given by (3.1) assumes onedimensional case. A generalization to multidimension is straight forward and will be discussed later. From [38], it can be easily shown that satisfies the condition of positive definiteness and the series converges absolutely uniformly since all the eigenvalues of are positive. As , it also satisfies the symmetry property and Mercer theorem can be applied to prove that is an admissible support vector kernel [39] and it can be written as a dot product in feature space, that is, At this point, we refer the reader to literature [10, 11, 23, 35] for useful discussion on regularization properties of commonly used SV kernels. We can also utilize (3.1) to obtain periodic kernels for given regularization operators. For example [10], by taking as eigenvalues of the given discrete regularization operator and Fourier basis as corresponding eigenvectors, we get Green's kernel:
Capacity control can be achieved by restricting the summation to different eigensubspaces with different values of . Excluding the eigenfunctions that correspond to high frequencies would result in increased smoothness thereby decreasing the system capacity and vice versa. A general lowpass smoothing functional is a good choice if there is no prior information available about the frequency distribution of the signal to be predicted. However, (3.1) can be seen as a reasonable choice for building kernels if there is some prior information available about the magnitude spectrum of the signal that we would like to approximate by utilizing the concept of matched filters [10].
4. Matched Filter and KnowledgeBased Matched Green's Kernel
Matched filter [40, 41] is the optimum time invariant filter among all linear or nonlinear filters to recover a known signal from additive white noise [42]. Assume the input signal in the presence of additive white noise passing through the matched filter with impulse response . The output of the filter is given by where denotes the convolution operation. Our aim is to obtain the conditions for which signaltonoise ratio (SNR) at the filter output takes its maximum value since it is understandable that the probability of recovering a signal from noise is high when SNR is maximum [27]. From [27, 42, 43] the impulse response of the matched filter is given by where is the filter gain constant and is the point at which the output power of the filter takes its maximum value. The maximum SNR is given by where is the noise power density. We refer the reader to original literature [27, 42, 43] for details and the proof of matched filter theorem. For simplicity we will assume unity gain, that is, . It is noteworthy in (4.2) that the filter impulse response is independent of noise power density with the prior assumption of white noise. Secondly, maximum SNR (4.3) is a function of signal energy and is independent of signal shape [42].
In order to design a matching kernel based on prior knowledge, it is sufficient to have an estimate of the magnitude spectrum of the signal to be predicted as prior knowledge about the signal as opposed to the theory of matched filters where complete knowledge of the signal is required to recover the signal from noise. From (4.2) it can be seen that the impulse response of the optimum filter is time reversed signal with delay. Nevertheless, in order to obtain matching kernel we are only interested in magnitude spectrum of matched filter which can be obtained by taking the Fourier transform of in (4.2) and multiplying it with its complex conjugate: where is the frequency response of matched filter, is the magnitude response and is the magnitude spectrum of the matched filter and the signal , respectively. An important note at this point is that (4.6) does not depend on delay whereas in the case of matched filters it is necessary to have a delay to make the impulse response realizable. Hence the matching kernel can be obtained by simply calculating the magnitude spectrum of and utilizing (3.1). As is the signal to be predicted, we assume that its magnitude spectrum does not significantly change from the training targets in (2.2) and this is the prior knowledge that we acquire from about to obtain Green's kernel. This is a weak condition since many signals with completely different characterization in time domain share the similar magnitude spectrum.
Figure 1 shows the time and the frequency domain representation of two different signals. Signal in Figure 1(a) is a sinusoid whereas signal in Figure 1(b) is the modified Morlet wavelet. Despite their completely different time domain characterization they share similar frequency localization given by Figures 1(c) and 1(d), respectively.
(a)
(b)
(c)
(d)
In order to be capable of using (3.1) to obtain our desired kernel function we need its eigenvalues and we will use Fourier basis as the corresponding eigenfunctions since complex exponentials are the eigenfunctions of any linear timeinvariant (LTI) system that includes matched filters and sinusoids can be expressed as linear combination of complex exponentials using Eulers formula [44]. The eigenvalues of and LTI system are given by frequency response which is a complexvalued quantity [45]. Frequency response can however be written as namely as a product of magnitude response and phase response [3]. Since we are only interested in smoothness properties of the kernel and not the phase response, it is adequate to take the magnitude response as eigenvalues of the system. Another reason for this is that in order to have positive definite Green's kernel function the eigenvalues need to be strictly positive [38]. Hence, matching Green's kernel function can be obtained by using (3.1): where is the discrete time counterpart of continuous frequency variable , such that , that is, normalized to have a range of . By making use of (4.6) and ignoring the constant eigenfunction with , we get which is a positive definite SV kernel that exhibits matched filter regularization properties given by . From the algorithmic point of view, we only need to compute magnitude of the discrete Fourier transform of the training targets with the assumption that the function to be predicted takes a similar magnitude spectrum with additive noise. To control the model complexity of the system we introduce two variables to restrict the summation calculation to desired eigensubspaces and write (4.9) as where and are the kernel parameters for Green's kernel similar to the kernel parameters of other SV kernel such as kernel width in the case of Gaussian RBF kernel or degree of the kernel in the case of polynomial kernel. Similar to other SV kernels an optimal value for and is required to achieve the best results.
Analogous to the conventional Gaussian kernel that exhibits Gaussian lowpass filter behavior, that is, [10, 11] (recall that the Fourier transform of a Gaussian function is also a Gaussian function) the knowledgebased Green's kernel obtained from the eigenvalues of the matched filter exhibits the matched filter properties. This property makes the knowledgebased Green's kernel an optimal choice for noise regime since matched filters are the optimal filters for noisecorrupted data regardless of the signal shape and the noise level. Since most of the real world systems are unavoidably contaminated with noise in addition to their intrinsic dynamics [25, 26, 30], we keep up with the longestablished tradition [2–4, 25, 26, 29, 30] of using benchmark datasets with additive white noise to evaluate the performance of the proposed techniques and conduct several experiments on mostly benchmark datasets ranging from simple regression models to chaotic and nonlinear time series with additive white noise in order to compare the performance of our technique with that of existing support vector (SV) kernels. Nevertheless, the advantage of knowledgebased Green's kernel comes at the cost of slightly increased computational complexity. However for most of the practical signals only a small portion of the whole eigensubspace turns out to be nonzero thereby lessening the computational load. Another way to overcome this problem is the efficient algorithmic implementation.
A generalization of the kernel function given by (4.10) to dimensions can be easily made by (see [2] for proof of the theorem). Alternatively, can also be used [10].
5. Experimental Results
5.1. Model Complexity Control
The purpose of this experiment is to examine the ability of Green's kernel to control the complexity of an SV model trained with Green's kernel. Sinc function is used as training and testing data. The training data is approximated with different models built only by reducing the size of eigensubspace in kernel matrix computation, that is, by reducing the value of kernel parameter while keeping the SV regularization parameter , and kernel parameter constant throughout the experiment. In other words, the complexity of the model is reduced by reducing the number of nonzero eigenvalues, that is, reducing the value of thereby removing the high capacity eigenfunctions to obtain a smoother approximation. The value of was used for all the models.
Figure 2 shows the regression results obtained for different values of . It is evident from the figure that reducing the size of eigensubspace produces smoother approximations which highlights the ability of Green's kernel as a regularizer. No GOF criteria are used in this experiment since the point of interest is to produce a smoother approximation not necessarily a good approximation.
5.2. Regression on Sinc Function
Sinc function given by (5.1) has become a benchmark to validate the results of SV regression [2, 3, 10, 34, 35, 46]: The training data is 27 points with zero mean, 0.2 variance additive Gaussian white noise. Mean square error was used as the figure of merit. Figure 3 shows the regression results obtained by Green's kernel and other commonly used SV kernels. Although the results obtained by Gaussian RBF and Bspline kernel are very similar, we prefer to use Gaussian RBF because only Bspline of odd order is admissible support vector kernels [10] and this restricts the model complexity control.
(a)
(b)
(c)
(d)
(e)
(f)
Figure 4 shows the magnitude spectrum of the training signal and the actual sinc function. Magnitude spectrum of the training signal is used as the prior knowledge about the actual signal, that is, the signal to be predicted and used to construct the matching Green's kernel. Table 1 shows the regression results obtained with different kernel functions. Results indicate that Green's kernel achieved better performance than any other support vector kernel for the given function. The CPU time is the kernel matrix computation time in seconds on an Intel (R) 2.8 GHz, 2 GB Memory system using Matlab 7. The CPU time for other kernel functions was computed using [47]. The lesser computational time of knowledgebased Green's kernel is owed to efficient algorithmic implementation which only includes nonzero eigenvalues in kernel matrix computation. The (near) optimal values of SVM hyperparameters for each kernel function were selected after several hundred trials.

(a)
(b)
5.3. Regression on Modified Morlet Wavelet Function
Modified Morlet wavelet function is described by This function was selected because of its complex model. A signal of 101 data points with zero mean, 0.3 variance white noise was used the training set. The (near) optimal values of SVM hyperparameters for each kernel function were selected after several hundred trials. Figure 5 shows the performance of the different SV kernels for modified Morlet wavelet function and the magnitude spectrum of training and actual signals. Although the training function is heavily corrupted with noise, there is still some similarity between the magnitude spectrum of two functions and this similarity is used as the prior knowledge about the problem. As shown in Table 2, again, Green's kernel performed better than any other kernel for heavily noise corrupted data.

(a)
(b)
(c)
(d)
(e)
The purpose of next two experiments is to evaluate the performance of the proposed kernel function against the conventional Gaussian kernel in a broader perspective, that is, across different noise models, noise levels, prediction steps (shortterm and longterm prediction for time series), and different variations of SVM that use different loss functions and optimization schemes. To perform a faithful comparison, we use the results already published in literature as our reference point and use the same datasets, noise model, noise level, and loss function as suggested by the corresponding authors. For the next two experiments, longterm and shortterm prediction of chaotic time series is considered as a special case of regression. We use MackeyGlass, a highdimensional chaotic benchmark time series, originally introduced as a model of blood cell regulation [48]. Mackey Glass is generated by the following delay differential equation [3]: with , and .
5.4. Chaotic Time Series Prediction Using SVM and LSSVM
For comparison purposes, we use Muller et al. [3] that employs SVM with Gaussian RBF kernel and Zhu et al. [4] that utilizes LSSVM with Gaussian RBF kernel for shortterm (1 step) and longterm (100 step) prediction of MackeyGlass system for different noise models and noise levels. Table 3 shows the mean square error obtained by Green's kernel using SVM and LSSVM over different noise settings in comparison to the results reported by [3, 4]. 1S and 100S denote the 1 step and 100 step prediction of time series. We use the same definition of SNR as used by the corresponding authors, that is, ratio between the standard deviation of the respective noise and the underlying time series. Experimental results indicate that knowledgebased Green's kernel should be considered as a good kernel choice for noisecorrupted data.

6. Conclusion
This paper provides a mathematical framework for using Green's functions to construct problem specific admissible support vector kernel functions based on the prior knowledge about smoothness properties of the function to be predicted. Matched filter theorem is used to incorporate domain knowledge of the magnitude spectrum of the signal to be predicted into support vector kernels to achieve desired regularization properties. It has been shown that the knowledgebased matching Green's kernel is a positive definite SV kernel that exhibits matched filter behavior. Since matched filters are known to be the optimal choice for noise corrupted data, the key contribution of the proposed technique is its noise robustness (see Figure 5) which makes it suitable for many real world system. Experimental results show that the knowledgebased Green's kernel has the ability to control the model complexity (see Figure 2) of the system and shows good generalization performance compared to other existing support vector kernels (see Tables 1, 2, and 3). Future research would include implementation of Green's kernel on real world problems such as speech synthesis and ultra sound image analysis.
References
 V. N. Vapnik, The Nature of Statistical Learning Theory, Springer, New York, NY, USA, 1995. View at: Zentralblatt MATH  MathSciNet
 V. N. Vapnik, Statistical Learning Theory, John Wiley & Sons, New York, NY, USA, 1998. View at: Zentralblatt MATH  MathSciNet
 K. R. Mller, A. J. Smola, G. Rtsch, B. Schlkopf, J. Kohlmorgen, and V. Vapnik, “Using support vector machines for time series prediction,” in Advances in Kernel Methods: Support Vector Machine, B. Scholkopf, J. Burges, and A. Smola, Eds., MIT Press, Cambridge, Mass, USA, 1999. View at: Google Scholar
 J.Y. Zhu, B. Ren, H.X. Zhang, and Z.T. Deng, “Time series prediction via new support vector machines,” in Proceedings of International Conference on Machine Learning and Cybernetics (ICMLC '02), pp. 364–366, November 2002. View at: Google Scholar
 V. N. Vapnik, “An overview of statistical learning theory,” IEEE Transactions on Neural Networks, vol. 10, no. 5, pp. 988–999, 1999. View at: Google Scholar
 V. Vapnik, Estimation of Dependences Based on Empirical Data, Nauka, Moscow, Russia, 1979, English translation: Springer, New York, NY, USA, 1982.
 A. N. Tikhonov and V. Y. Arsenin, Solutions of IllPosed Problems, John Wiley & Sons, New York, NY, USA, 1977. View at: MathSciNet
 A. N. Tikhonov, “Solution of incorrectly formulated problems and the regularization method,” Soviet Mathematics Doklady, vol. 4, pp. 1035–1038, 1963. View at: Google Scholar
 T. Evgeniou, M. Pontil, and T. Poggio, “Regularization networks and support vector machines,” Advances in Computational Mathematics, vol. 13, no. 1, pp. 1–50, 2000. View at: Publisher Site  Google Scholar  Zentralblatt MATH  MathSciNet
 B. Schlkopf and A. J. Smola, Learning with Kernels, MIT Press, Cambridge, Mass, USA, 2001.
 A. J. Smola, B. Schölkopf, and K.R. Müller, “The connection between regularization operators and support vector kernels,” Neural Networks, vol. 11, no. 4, pp. 637–649, 1998. View at: Publisher Site  Google Scholar
 R. V. Babu, P. Pérez, and P. Bouthemy, “Robust tracking with motion estimation and local Kernelbased color modeling,” Image and Vision Computing, vol. 25, no. 8, pp. 1205–1216, 2007. View at: Publisher Site  Google Scholar
 R. Maclin, E. Wild, J. Shavlik, L. Torrey, and T. Walker, “Refining rules incorporated into knowledgebased support vector learners via successive linear programming,” in Proceedings of the 22nd National Conference on Artificial Intelligence, vol. 1, pp. 584–589, July 2007. View at: Google Scholar
 L. Yen, F. Fouss, C. Decaestecker, P. Francq, and M. Saerens, “Graph nodes clustering with the sigmoid commutetime kernel: a comparative study,” Data and Knowledge Engineering, vol. 68, no. 3, pp. 338–361, 2009. View at: Publisher Site  Google Scholar
 G. Toma, “Specific differential equations for generating pulse sequences,” Mathematical Problems in Engineering, vol. 2010, Article ID 324818, 11 pages, 2010. View at: Publisher Site  Google Scholar  MathSciNet
 M. Li and J.Y. Li, “On the predictability of longrange dependent series,” Mathematical Problems in Engineering, vol. 2010, Article ID 397454, 9 pages, 2010. View at: Publisher Site  Google Scholar
 E. G. Bakhoum and C. Toma, “Mathematical transform of travelingwave equations and phase aspects of quantum interaction,” Mathematical Problems in Engineering, vol. 2010, Article ID 695208, 15 pages, 2010. View at: Publisher Site  Google Scholar  MathSciNet
 Z. Liu, “Chaotic time series analysis,” Mathematical Problems in Engineering, vol. 2010, Article ID 720190, 31 pages, 2010. View at: Publisher Site  Google Scholar
 T. Poggio and F. Girosi, A Theory on Networks for Approximation and Learning, MIT Press, Cambridge, Mass, USA, 1989.
 T. Poggio and F. Girosi, “Networks for approximation and learning,” Proceedings of the IEEE, vol. 78, no. 9, pp. 1481–1497, 1990. View at: Publisher Site  Google Scholar
 F. Girosi, M. Jones, and T. Poggio, “Regularization theory and neural networks architectures,” Neural Computation, vol. 7, pp. 219–269, 1995. View at: Google Scholar
 F. Girosi, M. Jones, and T. Poggio, “Priors, stabilizers and basis functions: from regularization to radial, tensor and additive splines,” AI Memo no. 1430, MIT AI Lab, 1993. View at: Google Scholar
 A. J. Smola and B. Schölkopf, “On a kernelbased method for pattern recognition, regression, approximation and operator inversion,” Algorithmica, vol. 22, no. 12, pp. 211–231, 1998. View at: Publisher Site  Google Scholar  Zentralblatt MATH  MathSciNet
 R. C. Williamson, A. J. Smola, and B. Schölkopf, “Generalization performance of regularization networks and support vector machines via entropy numbers of compact operators,” IEEE Transactions on Information Theory, vol. 47, no. 6, pp. 2516–2532, 2001. View at: Publisher Site  Google Scholar  Zentralblatt MATH  MathSciNet
 R. Metzler, Y. BarYam, and M. Kardar, “Information flowsystem chaotic channel: prediction and ostdiction at finite resolution,” Physical Review E, vol. 70, Article ID 026205, 2004. View at: Google Scholar
 G. Nolte, A. Ziehe, and K.R. Müller, “Noise robust estimates of correlation dimension and ${K}_{2}$ entropy,” Physical Review E, vol. 64, no. 1, Article ID 016112, 10 pages, 2001. View at: Google Scholar
 V. Olshevsky and L. Sakhnovich, “Matched filtering for generalized stationary processes,” IEEE Transactions on Information Theory, vol. 51, no. 9, pp. 3308–3313, 2005. View at: Publisher Site  Google Scholar  MathSciNet
 G. L. Turin, “An introduction to matched filters,” IRE Transactions on Information Theory, vol. 6, pp. 311–329, 1960. View at: Google Scholar  MathSciNet
 E. S. Chng, S. Chen, and B. Mulgrew, “Gradient radial basis function networks for nonlinear and nonstationary time series prediction,” IEEE Transactions on Neural Networks, vol. 7, no. 1, pp. 190–194, 1996. View at: Google Scholar
 D. Lowe and A. R. Webb, “Time series prediction by adaptive networks. A dynamical systems perspective,” IEE Proceedings F, vol. 138, no. 1, pp. 17–24, 1991. View at: Google Scholar
 B. E. Boser, I. M. Guyon, and V. N. Vapnik, “Training algorithm for optimal margin classifiers,” in Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory, D. Haussler, Ed., pp. 144–152, July 1992. View at: Google Scholar
 I. Guyon, B. Boser, and V. Vapnik, “Automatic capacity tuning of very large VCdimension classifiers,” in Advances in Neural Information Processing Systems, S. J. Hanson, J. D. Cowan, and C. L. Giles, Eds., vol. 5, pp. 147–155, Morgan Kaufmann, San Mateo, Calif, USA, 1993. View at: Google Scholar
 C. Cortes and V. Vapnik, “Supportvector networks,” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995. View at: Publisher Site  Google Scholar
 V. Vapnik, S. Golowich, and A. Smola, “Support vector method for function approximation, regression estimation, and signal processing,” in Advances in Neural Information Processing Systems, M. Mozer, M. Jordan, and T. Petsche, Eds., vol. 9, MIT Press, Cambridge, Mass, USA, 1997. View at: Google Scholar
 A. J. Smola and B. Schlkopf, “A tutorial on support vector regression,” Neuro COLT Tech. Rep. TR1998030, Royal Holloway College, 1998. View at: Google Scholar
 M. A. Aĭzerman, È. M. Braverman, and L. I. Rozonoèr, “Theoretical foundations of the potential function method in pattern recognition learning,” Automation and Remote Control, vol. 25, pp. 821–837, 1964. View at: Google Scholar
 N. J. Nilsson, Learning Machines: Foundations of Trainable Pattern Classifying Systems, McGrawHill, New York, NY, USA, 1965.
 G. F. Roach, Green's Functions Introductory Theory with Applications, Van Nostrand Reinhold, London, UK, 1970. View at: MathSciNet
 J. Mercer, “Functions of positive and negative type and their connection with the theory of integral equation,” Philosophical Transactions of the Royal Society A, vol. 209, pp. 415–446, 1909. View at: Google Scholar
 D. O. North, “An analysis of the factors which determine signal/noise discrimination in pulsedcarrier systems,” Tech. Rep. PTR6c, RCA Lab, June 1943. View at: Google Scholar
 D. O. North, “An analysis of the factors which determine signal/noise discrimination in pulsedcarrier systems,” Proceedings of the IEEE, vol. 51, no. 1, pp. 1016–1027, 1963. View at: Google Scholar
 H. J. Blinchikoff and A. I. Zverev, Filtering in the Time and Frequency Domains, John Wiley & Sons, New York, NY, USA, 1976.
 B. V. Kumar, A. Mahalanobis, and R. D. Juday, Correlation Pattern Recognition, Cambridge University Press, New York, NY, USA, 2005.
 J. O. Smith, “Introduction to digital filters with audio applications,” August 2006, https://ccrma.stanford.edu/~jos/filters/. View at: Google Scholar
 O. Akay, “Linear fractional shift invariant (LFSI) systems,” in Proceedings of the 7th International Symposium on Signal Processing and Its Applications, pp. 585–588, 2003. View at: Google Scholar
 J. A. K. Suykens, T. V. Gestel, J. D. Brabanter, B. D. Moor, and J. Vandewalle, Least Squares Support Vector Machines, World Scientific, Singapore, 2002.
 S. Gunn, “Support Vector Machines for Classification and Regression,” Tech. Rep., Image Speech and Intelligent Systems Research Group, University of Southampton, 1997. View at: Google Scholar
 M. C. Mackey and L. Glass, “Oscillation and chaos in physiological control systems,” Science, vol. 197, no. 4300, pp. 287–289, 1977. View at: Google Scholar
Copyright
Copyright © 2010 Tahir Farooq et al. This is an open access article distributed under the