Research Article | Open Access
Robust Kernel Clustering Algorithm for Nonlinear System Identification
In engineering field, it is necessary to know the model of the real nonlinear systems to ensure its control and supervision; in this context, fuzzy modeling and especially the Takagi-Sugeno fuzzy model has drawn the attention of several researchers in recent decades owing to their potential to approximate nonlinear behavior. To identify the parameters of Takagi-Sugeno fuzzy model several clustering algorithms are developed such as the Fuzzy -Means (FCM) algorithm, Possibilistic -Means (PCM) algorithm, and Possibilistic Fuzzy -Means (PFCM) algorithm. This paper presents a new clustering algorithm for Takagi-Sugeno fuzzy model identification. Our proposed algorithm called Robust Kernel Possibilistic Fuzzy -Means (RKPFCM) algorithm is an extension of the PFCM algorithm based on kernel method, where the Euclidean distance used the robust hyper tangent kernel function. The proposed algorithm can solve the nonlinear separable problems found by FCM, PCM, and PFCM algorithms. Then an optimization method using the Particle Swarm Optimization (PSO) method combined with the RKPFCM algorithm is presented to overcome the convergence to a local minimum of the objective function. Finally, validation results of examples are given to demonstrate the effectiveness, practicality, and robustness of our proposed algorithm in stochastic environment.
Modeling and identification are significant steps in the design of the control system. Typical applications of these models are the simulation, the prediction, or the control system design. Generally, the modeling process consists of obtaining a parametric model with the same dynamic behavior of the real process. However, when the process is nonlinear and complex, it is very difficult to define the different mathematical or physical laws which describe its behavior [1, 2]. In this context, the modeling of nonlinear systems by the conventional methods is very difficult and occasionally ineffective [3, 4]. So, other nonconventional techniques based on fuzzy logic are used more often in modeling this kind of process due to excellent ability of describing its behavior [4–6].
Among the best fuzzy modeling approaches developed in literature we mention the Takagi-Sugeno fuzzy model. In effect, this model is described by if-then rules. Each rule includes a fuzzy set antecedent and mathematical functions as consequent representing the process behavior in each region [2, 3, 7]. The identification problem consists of estimating the model parameters. In this context, to identify the parameters of Takagi-Sugeno fuzzy model, many techniques were developed such as the Adaptive schemes, heuristic approaches, nearest neighbor clustering, and support vector learning mechanisms. Besides fuzzy clustering algorithms are widely used in fuzzy modeling. Fuzzy -Means (FCM), Gustafson-Kessel (G-K), Gath-Geva (G-G), Possibilistic -Means (PCM), Fuzzy -Regression Model (FCRM), Enhanced Fuzzy -Regression Model (EFCRM), and Possibilistic Fuzzy -Means (PFCM) are popular clustering algorithms used in structure identification part and LS, WRLS, and orthogonal least square (OLS) technique were applied in consequent parameter estimation. Among the clustering algorithms, Fuzzy -Means (FCM) developed by Bezdek is well-known but this algorithm is sensitive to noise or outliers and susceptible to local minima [8, 9]. However, noise in the data sets can make the situation worse by creating many inauthentic minima. These are able to distort the global minimum solution found by FCM algorithm. This flaw has stimulated the researchers to overcome this inconvenience. To fight against the effects of outlying data, various approaches are considered such as the possibilistic clustering (PCM) proposed by Krishnapuram and Keller  and fuzzy noise clustering approach of Dave . The possibilistic approach executed a possibilistic partition, in which a membership relation calculates the absolute degree of typicality of a point in a cluster. Although the PCM algorithm is robust against the noise points and allows identifying these outliers, it is very responsive to initializations and occasionally generates coincident clusters. To solve this deficiency of identical clusters, a Possibilistic Fuzzy -Means (PFCM) algorithm was suggested by Wu and Zhou in 2006. Nonetheless, these algorithms are not efficient for unequal dimension clusters and cannot separate clusters that are nonlinearly separable in input space and their limits between two clusters are linear. In our work to overcome this shortcoming, kernel methods  are regarded as the way of dealing with this problem. We propose a new clustering algorithm called Robust Kernel Possibilistic Fuzzy -Means (RKPFCM), which adopts a kernel induced metric in the data space to replace the original Euclidean norm metric. By changing the inner product with an appropriate hyper tangent kernel function, one can implicitly affect a nonlinear mapping to a high dimensional feature space where the data is more clearly separable. However, our proposed algorithm RKPFCM has an iterative nature that makes it sensitive to initialization and sensitive to converge to a local minimum. To overcome these problems, several solutions have been proposed in the literature. Among them is combining the clustering algorithm with a heuristic optimization technique. In this context, many researches have proposed the evolutionary computation technique based on Particle Swarm Optimization (PSO). They have been successfully applied to solve various optimization problems . Thus, we introduce PSO into the RKPFCM algorithm to achieve global optimization. The efficacy of our algorithm compared to many algorithms is tested on noisy nonlinear systems defined by recurrent equations and an application to Box Jenkins system. This paper will be presented as follows. The second part of work is reserved for introducing the Takagi-Sugeno fuzzy model. The third part will be devoted to identifying the premise parameters of this model where we used the proposed RKPFCM algorithm and PSO algorithm is introduced. In the fourth part, we will focus on identification of consequent parameters. The simulations results and the model validity of RKPFCM and RKPFCM-PSO are presented in part five. Finally, we conclude this paper.
2. Takagi-Sugeno Fuzzy Model
Takagi-Sugeno fuzzy model (T-S) is one of the best techniques used for modeling a nonlinear system represented by the recurrent equation . T-S model is constructed by a rule-based type “if … then” in which the consequent uses numeric variables rather than linguistic variables (case of Mamdani). The consequent can be expressed by a constant, a polynomial, or differential equation depending on the antecedent variables. The T-S fuzzy model allows approximating the nonlinear system into several locally linear subsystems [4, 17].
In general, a Takagi-Sugeno fuzzy model is based on if then rules of the formThe “if” rule function defines the premise part and “then” rule function constitutes the consequent part of the T-S fuzzy model. : represents the th rule; : is the parameters vector, such as ; : is a scalar; : is observations vector; : represents the fuzzy subsets,where .
Here, the fuzzy sets are represented by the following membership function  where are centers and is the width of the membership function.
The estimated output model is defined by the following equation :Asso,
3. Identification Algorithm for Premise Parameters
To identify the premise parameters of a Takagi-Sugeno fuzzy model described by equation (1), we used the Possibilistic Fuzzy -Means (PFCM) algorithm and our proposed algorithms (RKPFCM and RKPFCM-PSO).
3.1. Possibilistic Fuzzy -Means (PFCM) Algorithm
The Possibilistic Fuzzy -Means (PFCM) algorithm, which uses Euclidean distance, finds the partition of the collection of measures, specified by -dimensional vectors , into fuzzy subsets by minimizing the following objective function :where : the number of clusters; : the membership of in cluster satisfying : the typicality of in classes ; : the set of cluster centers (); : the suitable positive numbers described by Typically, is chosen to be 1. are the terminal membership values of FCM. is a weighting degree; this parameter has a significant impact on the form of clusters in data space.To minimize equation (6), we take its partial derivative of variables, , and , equal to zero and obtain the following equations:
PFCM Algorithm Steps. Given a set of observations .
Initialization () Set the number of clusters . Set the level of weighting : . Set the parameters , . Set the stopping criterion ε: . Execute a FCM clustering algorithm to find initial fuzzy partition matrix and cluster centers . Initialize the typicality matrix randomly. Compute by (9).
Repeat for .
Step 1. Compute the cluster centers by (12).
Step 2. Compute the membership matrix by (10).
Step 3. Compute the typicality matrix by (11).
Until ; then stop. Otherwise, set and return to Step 1.
3.2. Proposed Robust Kernel Possibilistic Fuzzy -Means (RKPFCM) Algorithm
The PFCM can deal with noisy data better than FCM and PCM; nevertheless, these conventional clustering algorithms become more effective when applied on linearly separable data or with a reasonable quantity of errors. In reality, the linearly separable data are rare. Therefore, FCM, PCM, and PFCM share the same negative point in that they are unable to get good separation of data that are nonlinearly separable in input space. To correct the imperfections found in PFCM particularly the nonlinear separable problem, kernel  methods are regarded as the way of dealing with this problem. In this context, we proposed a new extension of Possibilistic Fuzzy -Means algorithm based on kernel method (RKPFCM). The present work proposes a way of increasing the accuracy of the PFCM algorithm by exploiting hyper tangent kernel function to calculate the distance used in its objective function.
The kernel function is defined as a generalization of the distance metric that measures the distance between two data points mapped into a future space in which the data are more clearly separable [12, 19–21].
The RKPFCM algorithm minimizes the following objective function:Then is mapped into space :where is an inner product kernel function.
If we adopt the hyper tangent kernel function, that is, then . Thus (15) can be written asConsidering (17), the objective function (13) is transformed as follows: The derivation of the objective function (18) according to , , and , defines the relationship update of cluster centers and membership coefficients.
(i) Derivative of ) with respect to .So,Equating (20) to zero leads toThen,
(ii) Derivative of ) with respect to . In this part we used the Lagrange multiplierFrom expression (25), we can write in this form:Substituting expression (26) in expression (24): It is alsoThe two expressions (26) and (28) give the following expression:Therefore the updating relationship is
(iii) Derivative of ) with respect to .Therefore, the updating typicality matrix isSimilarly (9) is rewritten by
RKPFCM Algorithm Steps. Given a set of observations .
Initialization ( = 0) Set the number of clusters . Set the level of weighting : . Set the parameters , , and . Set the stopping criterion ε: . Execute a FCM clustering algorithm to find initial fuzzy partition matrix and cluster centers . Initialize the typicality matrix randomly. Compute by (34).
Repeat for .
Step 1. Compute the cluster centers by (22).
Step 2. Compute the membership matrix by (30).
Step 3. Compute the typicality matrix by (33).
Until ; then stop. Otherwise, set and return to Step 1.
3.3. Robust Kernel Possibilistic Fuzzy -Means Algorithm Based on PSO (RKPFCM-PSO)
The Particle Swarm Optimization is a heuristic search method proposed by Kennedy and Eberhart (1995). This technique uses random population solution particles to find an optimal solution to problems. Each particle moves in the search space with a dynamically adjusted position and velocity for the best solution. The particle is characterized by data structure that contains the coordinates of the current position in the search space, the best solution point visited so far, and the subset of other agents that are seen as neighbors. These adjustments are based on the historical behaviors of itself and other agents in the swarm. The change of speed (acceleration) and the position of each particle in the optimization landscape (search space) are iteratively [6, 12, 23]where : size of particles; : size of the landscape (search space); : the speed of particle; : the position of particle; : the best previous position of particle; : index represents the best particle among all particles in the group; : the constriction factor described by the following relationship: where and are two positive constants satisfying the following relationship: and : random variables defined as follows: where and are two random variables between 0 and 1; : the weight of inertia according to this equation: where and are the initial and final weight, is the maximum iterations, and iter is the current iteration number.
3.3.2. Fitness Function
The fitness function defines our optimization problem described by the following expression:where is a positive constant. represents the objective function of the RKPFCM algorithm.
RKPFCM-PSO Algorithm. Given a set of observations , the RKPFCM-PSO algorithm is described by the following steps.
Initialization () Select the number of clusters , fuzzy degree , the parameters a and b, the population size NP, the constants and , the random variables and , the weight of inertia and , the size of the search space , the constant , and the stopping criterion . Set the 1st particle generation clusters centers. Initialize the fitness function and speed of each particle. Compute by (34).
Repeat = + 1
Step 1. Compute the fuzzy partition matrix by (30).
Step 2. Compute the typicality matrix by (33).
Step 3. Calculate the new value of fitness for each particle using (40).
Step 4. Compare the fitness of each particle with best, if the value is better than best and then set the best value.
Step 5. Compare the fitness value of best with the following: if the value is better than best, best then is set equal to this value.
Step 6. Update position and speed of each particle by (35).
So this algorithm is converged when ; that is to say, stop iteration and find the best solution in the last generation. If not, go back to Step1.
4. Identification for Consequent Parameters
The defuzzification method, used in the Takagi-Sugeno fuzzy model, is linear with the consequent parameters which can be obtained as a solution of a weighted least squares problem according to the following equation:where represents an extension of regression matrix; ; is the output vector; is a diagonal matrix of dimension ( × ) containing the coefficients of fuzzy memberships.The RKPFCM and RKPFCM-PSO clustering algorithm are used to find width of the membership functions by the following equation :
5. Simulation Results and Validation Model
5.1. Identical Data with Noise
In this example we have used 12 data which are composed of 10 models and two noises; this data set (12) is presented in . The FCM, PCM, PFCM, KPFCM, and our algorithms RKPFCM and RKPFCM-PSO were used in clustering the data set in tow groups ().
In this example the parameters settings are , , , , , , = 0.9, = 0.4, , , = 20, , and = 10−9.
Figure 1 shows the clustering results for our proposed method RKPFCM-PSO. The Ideal (true) centroids areTable 1 shows the results of center clusters using the six algorithms. The error between the results prototypes and ideal center clusters is calculated by the next expression:where is the FCM, PCM, PFCM, KPFCM, RKPFCM, and RKPFCM-PSO.
According to Table 1, our proposed algorithm RKPFCM -PSO gives the best prototypes of centers.
Figure 1 shows the effectiveness of our approach as well.
5.2. Identification of T-S Fuzzy Model
After applying the identification algorithm, it is necessary to validate the Takagi-Sugeno fuzzy model. Several validation tests of the model are used. Among them, we cited the Mean Square Error (MSE) test, Root Mean Square Error (RMSE), and the Variance Accounting For (VAF) test. where “” is the real output and “” is the estimated output.
5.2.1. Example 1
Consider a nonlinear system described by the following difference equation :where and are the output and the input of the system, respectively.
is a noise. = 0.01.
200 samples were generated by simulation and were used, where the selected input variables are chosen , , , .
The complete data set has been used to train the model. The noise influence is analyzed with different SNR levels (SNR = 10 dB and SNR = 5 dB).
In this part, we have applied various algorithms and our proposed clustering RKPFCM and RKPFCM-PSO which approximate the nonlinear model (46).
The used parameters are , , , , NP = 30, , , , , , σ = 20, , and ε = 10−9.
The shape of the excitation signal used for identification is illustrated in Figure 2.
The simulation result given by the RKPFCM-PSO algorithm is illustrated in Figure 3.
Table 2 shows the various modeling performance results without noise obtained by different algorithms; this comparison results demonstrate that the best MSE and best VAF are obtained by the proposed methods (RKPFCM and RKPFCM-PSO).
Tables 3 and 4 present the various modeling performance results with noise influence (SNR = 5 dB and 10 dB) obtained by the different algorithms. However, our proposed algorithm RKPFCM-PSO retained the best performance with a higher level of noise.
The local linear models identified are given as follows:
5.2.2. Example 2
Consider a highly complex modified nonlinear system described by the following difference equation :where is the model output and is the model input which is bounded between [−1+1]. The is a noise.
The following input signal is expressed as1500 samples were generated by simulation in which 1000 samples were used to train the model. Fuzzy model parameters have been identified once, testing of model was done by the remaining 500 samples, and , , , are chosen as input variables. In this example the parameters settings are , , , , NP = 30, , = 0.9, = 0.4, , , σ = 20, , and = 10−9.
The noise influence is analyzed with different SNR levels (SNR = 20 dB, SNR = 10 dB, SNR =5 dB, and SNR = 1 dB).
The evaluation performance index (RMSE-trn and RMSE-test) stands for training and testing data, respectively.
Tables 5–8 show the comparative performance of RKPFCM and RKPFCM-PSO with different existing algorithms such as FCM, G-K, Fuzzy Model Identification (FMI), FCRM, and MFCRM-NC. It is clearly seen from the results that our algorithm RKPFCM-PSO gives the best performance in noisy environments. The local linear models identified by RKPFCM-PSO are given as follows:
5.2.3. Example 3
We consider the Box Jenkins gas furnace data set which is used as a standard test for identification techniques. The data set is composed of 296 pairs of input-output measurements. The input “” is the gas flow rate into a furnace and the output “” is the CO2 concentration in the outlet gases. In order to take all the above-mentioned issues into account, we simulated the following experimental case : all the 296 data pairs are used as training data and , are selected as input variables to various algorithms, while we use two rules (). In this example the used parameters are , , , , NP = 30, , = 0.9, = 0.4, , = ,σ = 10, , and = 10−9.The simulation result given by the RKPFCM-PSO algorithm is illustrated in Figure 6.
Based on the comparison presented in Table 9, it is clear that the proposed algorithm RKPFCM-PSO is more robust to noise than the other algorithms found in literature.
When we use our algorithm RKPFCM-PSO the local linear models identified are given as follows:
In literature, various clustering algorithms have been proposed for nonlinear systems identification. In this work, we developed a new clustering algorithm called RKPFCM-PSO for the nonlinear systems identification. Our algorithm is an improvement of the Possibilistic Fuzzy -Means Clustering (PFCM) where we used a hyper tangent kernel function to calculate the distance of data point from the cluster centers and a heuristic search algorithm PSO to reach the global minimum of the objective function. The proposed algorithm provides better results of fuzzy modeling of unknown nonlinear systems. The robustness and the quality of this proposed method are demonstrated by simulation results of noisy nonlinear systems described by recurrent equations and application to a Box Jenkins gas furnace system. Thus, the proposed methods show favorable results in noisy environments compared with the techniques mentioned in the literature.
In the future, we will integrate other optimization methods such as the gravitational search algorithm to optimize our hybrid method and we will apply this algorithm for identification of some complex nonlinear real systems as the robotic or the mechatronic systems.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
- E. Kim, M. Park, S. Kim, and M. Park, “A transformed input-domain approach to fuzzy modeling,” IEEE Transactions on Fuzzy Systems, vol. 6, no. 4, pp. 596–604, 1998.
- M. Sugeno and G. T. Kang, “Fuzzy modelling and control of multilayer incinerator,” Fuzzy Sets and Systems, vol. 18, no. 3, pp. 329–345, 1986.
- J. C. Bezdek, Pattern Recognition with Fuzzy Objective Function Algorithms, Plenum Press, New York, NY, USA, 1981.
- M. Sugeno and T. Yasukawa, “A fuzzy-logic-based approach to qualitative modeling,” IEEE Transactions on Fuzzy Systems, vol. 1, no. 1, pp. 7–31, 1993.
- J.-Q. Chen, Y.-G. Xi, and Z.-J. Zhang, “A clustering algorithm for fuzzy model identification,” Fuzzy Sets and Systems, vol. 98, no. 3, pp. 319–329, 1998.
- T. Ahmed, H. Lassad, and C. Abdelkader, “Nonlinear system identification using clustering algorithm and particle swarm optimization,” Scientific Research and Essays, vol. 7, no. 13, pp. 1415–1431, 2012.
- J. Abonyi, R. Babuska, and F. Szeifert, “Modified Gath-Geva fuzzy clustering for identification of Takagi-Sugeno fuzzy models,” IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 32, no. 5, pp. 612–621, 2002.
- H. Frigui and Krishnapuram, “A robust competitive clustering algorithm with applications in computer vision,” IEEE Transactions on Pattern Analysis and Machine Intelligence Computation, vol. 21, no. 5, pp. 450–465, 1999.
- K. C. Ying, S. W. Lin, Z. J. Lee, and I. L. Lee, “A novel function approximation based on robust fuzzy regression algorithm model and particle swarm optimization,” Applied Soft Computing, vol. 38, no. 2, pp. 1820–1826, 2011.
- R. Krishnapuram and J. Keller, “A possibilistic approach to clustering,” IEEE Transactions on Fuzzy Systems, vol. 1, no. 2, pp. 98–110, 1993.
- R. Dave, “Characterization and detection of noise in clustering,” Pattern Recognition Letters, vol. 12, no. 11, pp. 657–664, 2011.
- J. Zhang and L. Shen, “An improved fuzzy c-means clustering algorithm based on shadowed sets and PSO,” Computational Intelligence and Neuroscience, vol. 2014, Article ID 368628, 10 pages, 2014.
- T. Niknam and B. Amiri, “An efficient hybrid approach based on PSO, ACO and k-means for cluster analysis,” Applied Soft Computing, vol. 10, pp. 183–197, 2010.
- M. Tushir and S. Srivastava, “A new Kernelized hybrid c-mean clustering model with optimized parameters,” Applied Soft Computing Journal, vol. 10, no. 2, pp. 381–389, 2010.
- S. Bidyadhar and J. Debashisha, “A difierential evolution based neural network approach to nonlinear identification,” Applied Soft Computing, vol. 11, no. 1, pp. 861–871, 2011.
- T. Dam and A. Kanti Deb, “Block sparse representations in modified fuzzy c-regression model clustering algorithm for ts fuzzy model identification,” in Proceedings of the IEEE Symposium Series on Computational Intelligence, pp. 1687–1694, IEEE, Cape Town, South Africa, December 2015.
- T. Takagi and M. Sugeno, “Fuzzy identification of systems and its applications for modeling and control,” IEEE Transactions on Systems, Man and Cybernetics, vol. 15, no. 1, pp. 116–132, 1985.
- X. H. Wu and Zhou, “A possibilistic c-means clustering algorithm based on kernel methods,” in Computer Design and Applications (ICCDA '10) International Conference, pp. 2062–2066, IEEE, Guilin, China, June 2006.
- S. R. Kannan, S. Ramathilagam, R. Deviand, and A. Sathya, “Robust kernel FCM in segmentation of breast medical images,” Expert Systems with Applications, vol. 38, pp. 4382–4389, 2011.
- P. Kaur, I. M. S. Lamba, and A. Gosai, “Novel kernelized type-2 fuzzy c-means clustering algorithm in segmentation of noisy medical images,” in Proceedings of the Recent Advances in Intelligent Computational Systems (RAICS '11), IEEE, Trivandrum, India, November 2011.
- B. Mohamed, T. Ahmed, H. Lassad, and C. Abdelkader, “A new extension of fuzzy c-means algorithm using non euclidean distance and kernel methods,” in Proceedings of the Control, Decision and Information Technologies (CoDIT '13), International Conference, IEEE, Hammamet, Tunisia, May 2013.
- X. Zhao and J. Zhou, “Improved kernel possibilistic fuzzy clustering algorithm based on invasive weed optimization,” Journal of Shanghai Jiaotong University, vol. 20, no. 2, pp. 164–170, 2015.
- R. C. Eberhart and Y. Shi, “Comparing inertia weights and constriction factors in particle swarm optimization,” in Proceedings of the Congress on Evolutionary Computation, pp. 84–88, La Jolla, CA, USA,, July 2000.
- R. Babuka and H. Verbruggen, “Neuro-fuzzy methods for nonlinear system identification,” Annual Reviews in Control, vol. 27, pp. 73–85, 2003.
- N. R. Pal, K. Pal, J. M. Keller, and J. C. Bezdek, “A possibilistic fuzzy c-means clustering algorithm,” IEEE Transactions on Fuzzy Systems, vol. 13, no. 4, pp. 517–530, 2005.
Copyright © 2017 Mohamed Bouzbida et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.