Research Article  Open Access
Mohamed Bouzbida, Lassad Hassine, Abdelkader Chaari, "Robust Kernel Clustering Algorithm for Nonlinear System Identification", Mathematical Problems in Engineering, vol. 2017, Article ID 2427309, 11 pages, 2017. https://doi.org/10.1155/2017/2427309
Robust Kernel Clustering Algorithm for Nonlinear System Identification
Abstract
In engineering field, it is necessary to know the model of the real nonlinear systems to ensure its control and supervision; in this context, fuzzy modeling and especially the TakagiSugeno fuzzy model has drawn the attention of several researchers in recent decades owing to their potential to approximate nonlinear behavior. To identify the parameters of TakagiSugeno fuzzy model several clustering algorithms are developed such as the Fuzzy Means (FCM) algorithm, Possibilistic Means (PCM) algorithm, and Possibilistic Fuzzy Means (PFCM) algorithm. This paper presents a new clustering algorithm for TakagiSugeno fuzzy model identification. Our proposed algorithm called Robust Kernel Possibilistic Fuzzy Means (RKPFCM) algorithm is an extension of the PFCM algorithm based on kernel method, where the Euclidean distance used the robust hyper tangent kernel function. The proposed algorithm can solve the nonlinear separable problems found by FCM, PCM, and PFCM algorithms. Then an optimization method using the Particle Swarm Optimization (PSO) method combined with the RKPFCM algorithm is presented to overcome the convergence to a local minimum of the objective function. Finally, validation results of examples are given to demonstrate the effectiveness, practicality, and robustness of our proposed algorithm in stochastic environment.
1. Introduction
Modeling and identification are significant steps in the design of the control system. Typical applications of these models are the simulation, the prediction, or the control system design. Generally, the modeling process consists of obtaining a parametric model with the same dynamic behavior of the real process. However, when the process is nonlinear and complex, it is very difficult to define the different mathematical or physical laws which describe its behavior [1, 2]. In this context, the modeling of nonlinear systems by the conventional methods is very difficult and occasionally ineffective [3, 4]. So, other nonconventional techniques based on fuzzy logic are used more often in modeling this kind of process due to excellent ability of describing its behavior [4–6].
Among the best fuzzy modeling approaches developed in literature we mention the TakagiSugeno fuzzy model. In effect, this model is described by ifthen rules. Each rule includes a fuzzy set antecedent and mathematical functions as consequent representing the process behavior in each region [2, 3, 7]. The identification problem consists of estimating the model parameters. In this context, to identify the parameters of TakagiSugeno fuzzy model, many techniques were developed such as the Adaptive schemes, heuristic approaches, nearest neighbor clustering, and support vector learning mechanisms. Besides fuzzy clustering algorithms are widely used in fuzzy modeling. Fuzzy Means (FCM), GustafsonKessel (GK), GathGeva (GG), Possibilistic Means (PCM), Fuzzy Regression Model (FCRM), Enhanced Fuzzy Regression Model (EFCRM), and Possibilistic Fuzzy Means (PFCM) are popular clustering algorithms used in structure identification part and LS, WRLS, and orthogonal least square (OLS) technique were applied in consequent parameter estimation. Among the clustering algorithms, Fuzzy Means (FCM) developed by Bezdek is wellknown but this algorithm is sensitive to noise or outliers and susceptible to local minima [8, 9]. However, noise in the data sets can make the situation worse by creating many inauthentic minima. These are able to distort the global minimum solution found by FCM algorithm. This flaw has stimulated the researchers to overcome this inconvenience. To fight against the effects of outlying data, various approaches are considered such as the possibilistic clustering (PCM) proposed by Krishnapuram and Keller [10] and fuzzy noise clustering approach of Dave [11]. The possibilistic approach executed a possibilistic partition, in which a membership relation calculates the absolute degree of typicality of a point in a cluster. Although the PCM algorithm is robust against the noise points and allows identifying these outliers, it is very responsive to initializations and occasionally generates coincident clusters. To solve this deficiency of identical clusters, a Possibilistic Fuzzy Means (PFCM) algorithm was suggested by Wu and Zhou in 2006. Nonetheless, these algorithms are not efficient for unequal dimension clusters and cannot separate clusters that are nonlinearly separable in input space and their limits between two clusters are linear. In our work to overcome this shortcoming, kernel methods [12] are regarded as the way of dealing with this problem. We propose a new clustering algorithm called Robust Kernel Possibilistic Fuzzy Means (RKPFCM), which adopts a kernel induced metric in the data space to replace the original Euclidean norm metric. By changing the inner product with an appropriate hyper tangent kernel function, one can implicitly affect a nonlinear mapping to a high dimensional feature space where the data is more clearly separable. However, our proposed algorithm RKPFCM has an iterative nature that makes it sensitive to initialization and sensitive to converge to a local minimum. To overcome these problems, several solutions have been proposed in the literature. Among them is combining the clustering algorithm with a heuristic optimization technique. In this context, many researches have proposed the evolutionary computation technique based on Particle Swarm Optimization (PSO). They have been successfully applied to solve various optimization problems [13]. Thus, we introduce PSO into the RKPFCM algorithm to achieve global optimization. The efficacy of our algorithm compared to many algorithms is tested on noisy nonlinear systems defined by recurrent equations and an application to Box Jenkins system. This paper will be presented as follows. The second part of work is reserved for introducing the TakagiSugeno fuzzy model. The third part will be devoted to identifying the premise parameters of this model where we used the proposed RKPFCM algorithm and PSO algorithm is introduced. In the fourth part, we will focus on identification of consequent parameters. The simulations results and the model validity of RKPFCM and RKPFCMPSO are presented in part five. Finally, we conclude this paper.
2. TakagiSugeno Fuzzy Model
TakagiSugeno fuzzy model (TS) is one of the best techniques used for modeling a nonlinear system represented by the recurrent equation . TS model is constructed by a rulebased type “if … then” in which the consequent uses numeric variables rather than linguistic variables (case of Mamdani). The consequent can be expressed by a constant, a polynomial, or differential equation depending on the antecedent variables. The TS fuzzy model allows approximating the nonlinear system into several locally linear subsystems [4, 17].
In general, a TakagiSugeno fuzzy model is based on if then rules of the formThe “if” rule function defines the premise part and “then” rule function constitutes the consequent part of the TS fuzzy model. : represents the th rule; : is the parameters vector, such as ; : is a scalar; : is observations vector; : represents the fuzzy subsets,where .
Here, the fuzzy sets are represented by the following membership function [7] where are centers and is the width of the membership function.
The estimated output model is defined by the following equation [4]:Asso,
3. Identification Algorithm for Premise Parameters
To identify the premise parameters of a TakagiSugeno fuzzy model described by equation (1), we used the Possibilistic Fuzzy Means (PFCM) algorithm and our proposed algorithms (RKPFCM and RKPFCMPSO).
3.1. Possibilistic Fuzzy Means (PFCM) Algorithm
The Possibilistic Fuzzy Means (PFCM) algorithm, which uses Euclidean distance, finds the partition of the collection of measures, specified by dimensional vectors , into fuzzy subsets by minimizing the following objective function [18]:where : the number of clusters; : the membership of in cluster satisfying : the typicality of in classes ; : the set of cluster centers (); : the suitable positive numbers described by Typically, is chosen to be 1. are the terminal membership values of FCM. is a weighting degree; this parameter has a significant impact on the form of clusters in data space.To minimize equation (6), we take its partial derivative of variables, , and , equal to zero and obtain the following equations:
PFCM Algorithm Steps. Given a set of observations .
Initialization () Set the number of clusters . Set the level of weighting : . Set the parameters , . Set the stopping criterion ε: . Execute a FCM clustering algorithm to find initial fuzzy partition matrix and cluster centers . Initialize the typicality matrix randomly. Compute by (9).
Repeat for .
Step 1. Compute the cluster centers by (12).
Step 2. Compute the membership matrix by (10).
Step 3. Compute the typicality matrix by (11).
Until ; then stop. Otherwise, set and return to Step 1.
3.2. Proposed Robust Kernel Possibilistic Fuzzy Means (RKPFCM) Algorithm
The PFCM can deal with noisy data better than FCM and PCM; nevertheless, these conventional clustering algorithms become more effective when applied on linearly separable data or with a reasonable quantity of errors. In reality, the linearly separable data are rare. Therefore, FCM, PCM, and PFCM share the same negative point in that they are unable to get good separation of data that are nonlinearly separable in input space. To correct the imperfections found in PFCM particularly the nonlinear separable problem, kernel [7] methods are regarded as the way of dealing with this problem. In this context, we proposed a new extension of Possibilistic Fuzzy Means algorithm based on kernel method (RKPFCM). The present work proposes a way of increasing the accuracy of the PFCM algorithm by exploiting hyper tangent kernel function to calculate the distance used in its objective function.
The kernel function is defined as a generalization of the distance metric that measures the distance between two data points mapped into a future space in which the data are more clearly separable [12, 19–21].
Define a nonlinear map as , where is the transformed feature space with higher or even infinite dimension. denotes the data space mapped into [20–22].
The RKPFCM algorithm minimizes the following objective function:Then is mapped into space [22]:where is an inner product kernel function.
If we adopt the hyper tangent kernel function, that is, then . Thus (15) can be written asConsidering (17), the objective function (13) is transformed as follows: The derivation of the objective function (18) according to , , and , defines the relationship update of cluster centers and membership coefficients.
(i) Derivative of ) with respect to .So,Equating (20) to zero leads toThen,
(ii) Derivative of ) with respect to . In this part we used the Lagrange multiplierFrom expression (25), we can write in this form:Substituting expression (26) in expression (24): It is alsoThe two expressions (26) and (28) give the following expression:Therefore the updating relationship is
(iii) Derivative of ) with respect to .Therefore, the updating typicality matrix isSimilarly (9) is rewritten by
RKPFCM Algorithm Steps. Given a set of observations .
Initialization ( = 0) Set the number of clusters . Set the level of weighting : . Set the parameters , , and . Set the stopping criterion ε: . Execute a FCM clustering algorithm to find initial fuzzy partition matrix and cluster centers . Initialize the typicality matrix randomly. Compute by (34).
Repeat for .
Step 1. Compute the cluster centers by (22).
Step 2. Compute the membership matrix by (30).
Step 3. Compute the typicality matrix by (33).
Until ; then stop. Otherwise, set and return to Step 1.
3.3. Robust Kernel Possibilistic Fuzzy Means Algorithm Based on PSO (RKPFCMPSO)
3.3.1. PSO
The Particle Swarm Optimization is a heuristic search method proposed by Kennedy and Eberhart (1995). This technique uses random population solution particles to find an optimal solution to problems. Each particle moves in the search space with a dynamically adjusted position and velocity for the best solution. The particle is characterized by data structure that contains the coordinates of the current position in the search space, the best solution point visited so far, and the subset of other agents that are seen as neighbors. These adjustments are based on the historical behaviors of itself and other agents in the swarm. The change of speed (acceleration) and the position of each particle in the optimization landscape (search space) are iteratively [6, 12, 23]where : size of particles; : size of the landscape (search space); : the speed of particle; : the position of particle; : the best previous position of particle; : index represents the best particle among all particles in the group; : the constriction factor described by the following relationship: where and are two positive constants satisfying the following relationship: and : random variables defined as follows: where and are two random variables between 0 and 1; : the weight of inertia according to this equation: where and are the initial and final weight, is the maximum iterations, and iter is the current iteration number.
3.3.2. Fitness Function
The fitness function defines our optimization problem described by the following expression:where is a positive constant. represents the objective function of the RKPFCM algorithm.
RKPFCMPSO Algorithm. Given a set of observations , the RKPFCMPSO algorithm is described by the following steps.
Initialization () Select the number of clusters , fuzzy degree , the parameters a and b, the population size NP, the constants and , the random variables and , the weight of inertia and , the size of the search space , the constant , and the stopping criterion . Set the 1st particle generation clusters centers. Initialize the fitness function and speed of each particle. Compute by (34).
Repeat = + 1
Step 1. Compute the fuzzy partition matrix by (30).
Step 2. Compute the typicality matrix by (33).
Step 3. Calculate the new value of fitness for each particle using (40).
Step 4. Compare the fitness of each particle with best, if the value is better than best and then set the best value.
Step 5. Compare the fitness value of best with the following: if the value is better than best, best then is set equal to this value.
Step 6. Update position and speed of each particle by (35).
So this algorithm is converged when ; that is to say, stop iteration and find the best solution in the last generation. If not, go back to Step1.
4. Identification for Consequent Parameters
The defuzzification method, used in the TakagiSugeno fuzzy model, is linear with the consequent parameters which can be obtained as a solution of a weighted least squares problem according to the following equation:where represents an extension of regression matrix; ; is the output vector; is a diagonal matrix of dimension ( × ) containing the coefficients of fuzzy memberships.The RKPFCM and RKPFCMPSO clustering algorithm are used to find width of the membership functions by the following equation [24]:
5. Simulation Results and Validation Model
5.1. Identical Data with Noise
In this example we have used 12 data which are composed of 10 models and two noises; this data set (12) is presented in [25]. The FCM, PCM, PFCM, KPFCM, and our algorithms RKPFCM and RKPFCMPSO were used in clustering the data set in tow groups ().
In this example the parameters settings are , , , , , , = 0.9, = 0.4, , , = 20, , and = 10^{−9}.
Figure 1 shows the clustering results for our proposed method RKPFCMPSO. The Ideal (true) centroids areTable 1 shows the results of center clusters using the six algorithms. The error between the results prototypes and ideal center clusters is calculated by the next expression:where is the FCM, PCM, PFCM, KPFCM, RKPFCM, and RKPFCMPSO.
According to Table 1, our proposed algorithm RKPFCM PSO gives the best prototypes of centers.
Figure 1 shows the effectiveness of our approach as well.
5.2. Identification of TS Fuzzy Model
After applying the identification algorithm, it is necessary to validate the TakagiSugeno fuzzy model. Several validation tests of the model are used. Among them, we cited the Mean Square Error (MSE) test, Root Mean Square Error (RMSE), and the Variance Accounting For (VAF) test. where “” is the real output and “” is the estimated output.
5.2.1. Example 1
Consider a nonlinear system described by the following difference equation [15]:where and are the output and the input of the system, respectively.
is a noise. = 0.01.
200 samples were generated by simulation and were used, where the selected input variables are chosen , , , .
The complete data set has been used to train the model. The noise influence is analyzed with different SNR levels (SNR = 10 dB and SNR = 5 dB).
In this part, we have applied various algorithms and our proposed clustering RKPFCM and RKPFCMPSO which approximate the nonlinear model (46).
The used parameters are , , , , NP = 30, , , , , , σ = 20, , and ε = 10^{−9}.
The shape of the excitation signal used for identification is illustrated in Figure 2.
The simulation result given by the RKPFCMPSO algorithm is illustrated in Figure 3.
Table 2 shows the various modeling performance results without noise obtained by different algorithms; this comparison results demonstrate that the best MSE and best VAF are obtained by the proposed methods (RKPFCM and RKPFCMPSO).
Tables 3 and 4 present the various modeling performance results with noise influence (SNR = 5 dB and 10 dB) obtained by the different algorithms. However, our proposed algorithm RKPFCMPSO retained the best performance with a higher level of noise.


The local linear models identified are given as follows:
5.2.2. Example 2
Consider a highly complex modified nonlinear system described by the following difference equation [16]:where is the model output and is the model input which is bounded between [−1+1]. The is a noise.
The following input signal is expressed as1500 samples were generated by simulation in which 1000 samples were used to train the model. Fuzzy model parameters have been identified once, testing of model was done by the remaining 500 samples, and , , , are chosen as input variables. In this example the parameters settings are , , , , NP = 30, , = 0.9, = 0.4, , , σ = 20, , and = 10^{−9}.
The noise influence is analyzed with different SNR levels (SNR = 20 dB, SNR = 10 dB, SNR =5 dB, and SNR = 1 dB).
The obtained identification results by RKPFCMPSO are, respectively, shown in Figures 4 and 5.
The evaluation performance index (RMSEtrn and RMSEtest) stands for training and testing data, respectively.
Tables 5–8 show the comparative performance of RKPFCM and RKPFCMPSO with different existing algorithms such as FCM, GK, Fuzzy Model Identification (FMI), FCRM, and MFCRMNC. It is clearly seen from the results that our algorithm RKPFCMPSO gives the best performance in noisy environments. The local linear models identified by RKPFCMPSO are given as follows:
5.2.3. Example 3
We consider the Box Jenkins gas furnace data set which is used as a standard test for identification techniques. The data set is composed of 296 pairs of inputoutput measurements. The input “” is the gas flow rate into a furnace and the output “” is the CO_{2} concentration in the outlet gases. In order to take all the abovementioned issues into account, we simulated the following experimental case [16]: all the 296 data pairs are used as training data and , are selected as input variables to various algorithms, while we use two rules (). In this example the used parameters are , , , , NP = 30, , = 0.9, = 0.4, , = ,σ = 10, , and = 10^{−9}.The simulation result given by the RKPFCMPSO algorithm is illustrated in Figure 6.
Based on the comparison presented in Table 9, it is clear that the proposed algorithm RKPFCMPSO is more robust to noise than the other algorithms found in literature.

When we use our algorithm RKPFCMPSO the local linear models identified are given as follows:
6. Conclusion
In literature, various clustering algorithms have been proposed for nonlinear systems identification. In this work, we developed a new clustering algorithm called RKPFCMPSO for the nonlinear systems identification. Our algorithm is an improvement of the Possibilistic Fuzzy Means Clustering (PFCM) where we used a hyper tangent kernel function to calculate the distance of data point from the cluster centers and a heuristic search algorithm PSO to reach the global minimum of the objective function. The proposed algorithm provides better results of fuzzy modeling of unknown nonlinear systems. The robustness and the quality of this proposed method are demonstrated by simulation results of noisy nonlinear systems described by recurrent equations and application to a Box Jenkins gas furnace system. Thus, the proposed methods show favorable results in noisy environments compared with the techniques mentioned in the literature.
In the future, we will integrate other optimization methods such as the gravitational search algorithm to optimize our hybrid method and we will apply this algorithm for identification of some complex nonlinear real systems as the robotic or the mechatronic systems.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
References
 E. Kim, M. Park, S. Kim, and M. Park, “A transformed inputdomain approach to fuzzy modeling,” IEEE Transactions on Fuzzy Systems, vol. 6, no. 4, pp. 596–604, 1998. View at: Google Scholar
 M. Sugeno and G. T. Kang, “Fuzzy modelling and control of multilayer incinerator,” Fuzzy Sets and Systems, vol. 18, no. 3, pp. 329–345, 1986. View at: Publisher Site  Google Scholar
 J. C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum Press, New York, NY, USA, 1981. View at: Publisher Site
 M. Sugeno and T. Yasukawa, “A fuzzylogicbased approach to qualitative modeling,” IEEE Transactions on Fuzzy Systems, vol. 1, no. 1, pp. 7–31, 1993. View at: Publisher Site  Google Scholar
 J.Q. Chen, Y.G. Xi, and Z.J. Zhang, “A clustering algorithm for fuzzy model identification,” Fuzzy Sets and Systems, vol. 98, no. 3, pp. 319–329, 1998. View at: Publisher Site  Google Scholar  MathSciNet
 T. Ahmed, H. Lassad, and C. Abdelkader, “Nonlinear system identification using clustering algorithm and particle swarm optimization,” Scientific Research and Essays, vol. 7, no. 13, pp. 1415–1431, 2012. View at: Google Scholar
 J. Abonyi, R. Babuska, and F. Szeifert, “Modified GathGeva fuzzy clustering for identification of TakagiSugeno fuzzy models,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 32, no. 5, pp. 612–621, 2002. View at: Publisher Site  Google Scholar
 H. Frigui and Krishnapuram, “A robust competitive clustering algorithm with applications in computer vision,” IEEE Transactions on Pattern Analysis and Machine Intelligence Computation, vol. 21, no. 5, pp. 450–465, 1999. View at: Publisher Site  Google Scholar
 K. C. Ying, S. W. Lin, Z. J. Lee, and I. L. Lee, “A novel function approximation based on robust fuzzy regression algorithm model and particle swarm optimization,” Applied Soft Computing, vol. 38, no. 2, pp. 1820–1826, 2011. View at: Publisher Site  Google Scholar
 R. Krishnapuram and J. Keller, “A possibilistic approach to clustering,” IEEE Transactions on Fuzzy Systems, vol. 1, no. 2, pp. 98–110, 1993. View at: Publisher Site  Google Scholar
 R. Dave, “Characterization and detection of noise in clustering,” Pattern Recognition Letters, vol. 12, no. 11, pp. 657–664, 2011. View at: Publisher Site  Google Scholar
 J. Zhang and L. Shen, “An improved fuzzy cmeans clustering algorithm based on shadowed sets and PSO,” Computational Intelligence and Neuroscience, vol. 2014, Article ID 368628, 10 pages, 2014. View at: Publisher Site  Google Scholar
 T. Niknam and B. Amiri, “An efficient hybrid approach based on PSO, ACO and kmeans for cluster analysis,” Applied Soft Computing, vol. 10, pp. 183–197, 2010. View at: Publisher Site  Google Scholar
 M. Tushir and S. Srivastava, “A new Kernelized hybrid cmean clustering model with optimized parameters,” Applied Soft Computing Journal, vol. 10, no. 2, pp. 381–389, 2010. View at: Publisher Site  Google Scholar
 S. Bidyadhar and J. Debashisha, “A difierential evolution based neural network approach to nonlinear identification,” Applied Soft Computing, vol. 11, no. 1, pp. 861–871, 2011. View at: Publisher Site  Google Scholar
 T. Dam and A. Kanti Deb, “Block sparse representations in modified fuzzy cregression model clustering algorithm for ts fuzzy model identification,” in Proceedings of the IEEE Symposium Series on Computational Intelligence, pp. 1687–1694, IEEE, Cape Town, South Africa, December 2015. View at: Publisher Site  Google Scholar
 T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications for modeling and control,” IEEE Transactions on Systems, Man and Cybernetics, vol. 15, no. 1, pp. 116–132, 1985. View at: Google Scholar
 X. H. Wu and Zhou, “A possibilistic cmeans clustering algorithm based on kernel methods,” in Computer Design and Applications (ICCDA '10) International Conference, pp. 2062–2066, IEEE, Guilin, China, June 2006. View at: Google Scholar
 S. R. Kannan, S. Ramathilagam, R. Deviand, and A. Sathya, “Robust kernel FCM in segmentation of breast medical images,” Expert Systems with Applications, vol. 38, pp. 4382–4389, 2011. View at: Publisher Site  Google Scholar
 P. Kaur, I. M. S. Lamba, and A. Gosai, “Novel kernelized type2 fuzzy cmeans clustering algorithm in segmentation of noisy medical images,” in Proceedings of the Recent Advances in Intelligent Computational Systems (RAICS '11), IEEE, Trivandrum, India, November 2011. View at: Publisher Site  Google Scholar
 B. Mohamed, T. Ahmed, H. Lassad, and C. Abdelkader, “A new extension of fuzzy cmeans algorithm using non euclidean distance and kernel methods,” in Proceedings of the Control, Decision and Information Technologies (CoDIT '13), International Conference, IEEE, Hammamet, Tunisia, May 2013. View at: Publisher Site  Google Scholar
 X. Zhao and J. Zhou, “Improved kernel possibilistic fuzzy clustering algorithm based on invasive weed optimization,” Journal of Shanghai Jiaotong University, vol. 20, no. 2, pp. 164–170, 2015. View at: Publisher Site  Google Scholar
 R. C. Eberhart and Y. Shi, “Comparing inertia weights and constriction factors in particle swarm optimization,” in Proceedings of the Congress on Evolutionary Computation, pp. 84–88, La Jolla, CA, USA,, July 2000. View at: Publisher Site  Google Scholar
 R. Babuka and H. Verbruggen, “Neurofuzzy methods for nonlinear system identification,” Annual Reviews in Control, vol. 27, pp. 73–85, 2003. View at: Publisher Site  Google Scholar
 N. R. Pal, K. Pal, J. M. Keller, and J. C. Bezdek, “A possibilistic fuzzy cmeans clustering algorithm,” IEEE Transactions on Fuzzy Systems, vol. 13, no. 4, pp. 517–530, 2005. View at: Publisher Site  Google Scholar
Copyright
Copyright © 2017 Mohamed Bouzbida et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.