Advanced Stochastic Control Systems with Engineering Applications
View this Special IssueResearch Article  Open Access
Fuzzy Pruning Based LSSVM Modeling Development for a Fermentation Process
Abstract
Due to the complexity and uncertainty of microbial fermentation processes, data coming from the plants often contain some outliers. However, these data may be treated as the normal support vectors, which always deteriorate the performance of soft sensor modeling. Since the outliers also contaminate the correlation structure of the least square support vector machine (LSSVM), the fuzzy pruning method is provided to deal with the problem. Furthermore, by assigning different fuzzy membership scores to data samples, the sensitivity of the model to the outliers can be reduced greatly. The effectiveness and efficiency of the proposed approach are demonstrated through two numerical examples as well as a simulator case of penicillin fermentation process.
1. Introduction
For the limitation of advanced measurement techniques, some important process variables in biochemical industrial processes, such as product composition, product concentration, and biomass concentration, are difficult or impossible to measure online. However, these variables are very important for the products quality and the result of the whole reaction process. A soft sensor model is always needed to construct between variables which are easy to measure online and one which is difficult to measure. Then a value of an objective variable can be inferred by this model. The approaches and corresponding applications of soft sensors have been discussed in some literature [1–4]. For example, partial least squares (PLS) and principal component analysis (PCA) [5, 6] are the most popular projection based soft sensor modeling methods for modeling and prediction. However, a drawback of these models is their linear nature. If it is known that the relation between the easytomeasure and the difficulttomeasure variables is nonlinear, then a nonlinear modeling method should be used. In last decades, databased soft sensor modeling approaches have been intensively studied, such as nonlinear partial least squares (NPLS), nonlinear principal component analysis (NPCA), artificial neural networks (ANNs), and support vector machine (SVM) [7–10]. Although the NPCA is a wellestablished and powerful algorithm, it has several drawbacks. One of them is that the principal components describe very well the input space but do not reflect the relation between the input and the output data space. A solution to this drawback is given by the NPLS method. NPLS models are appropriate to study the behavior of the process. Unfortunately, sometimes the algorithm of NPLS is available only for specific nonlinear relationships. To break through the limitation of NPLS, ANN is adopted to solve the complexity and highly nonlinear problem in the case of the sample data tending to infinity. The disadvantage of ANNs is that during their learning they are prone to get stuck in local minima, which can result in suboptimal performance. Meanwhile, SVM has been demonstrated to work very well for a wide spectrum of applications under the limited training data samples, so it is not surprising that it has also been successfully applied as soft sensor.
Support vector machine (SVM) proposed by Vapnik [11, 12], which is based on statistical learning theory, obtains the optimal classification of the sample data through a quadratic programming. So it can balance the risk of learning algorithm and promotion of the extension ability. As a sophisticated soft sensor modeling method, SVM has a lot of advantages in solving small sample data and nonlinear and high dimensional pattern recognition and has been applied to the fermentation process successfully [13, 14]. Least squares support vector machine (LSSVM) proposed by Suykens and Vandewalle [15] is an extension of the standard SVM. It can solve linear equations with faster solution speed and figure out the robustness, sparseness, and largescale computing problems. However, all training data are treated as the normal support vector which loses the sparseness of SVM [16–19]. In this paper, the effective work addressed in Section 3 could improve the performance of the standard LSSVM effectively.
Penicillin fermentation process is a typical biochemical reaction process with the features of nonlinearity and dynamic, which is caused by the factors such as genetic variation of somatic cell, microbial sensitivity to environment changing, and instability of raw material and seed quality that bring about serious nonlinearity and uncertainty [20]. For this process, key variables are concentration of the biomass, product, and substrate which are difficult to measure directly. However, some other auxiliary variables are easy to measure. So we choose aeration rate, dissolved oxygen concentration, agitator power, and others as auxiliary variables and the concentration of penicillin as the quality variable in this process. The next step is to construct the inferred model between the auxiliary variables and the quality variable. Outliers are commonly encountered in penicillin fermentation process which may be treated as the normal support vector and always has a bad influence on the precision of the soft sensor model. So applying the idea of fuzzy pruning for LSSVM algorithm to cut off these outliers and reduce the number of support vectors will improve the sparseness and precision of the original LSSVM model. Also assigning different fuzzy membership scores to sample data, the sensitivity to the outliers is reduced and the accuracy of the model is further improved as well. Finally, the LSSVM and fuzzy pruning based LSSVM soft sensor models for the penicillin fermentation process are constructed based on the optimal parameters obtained by using particle swarm optimization algorithm [21, 22]. Thus a soft sensor model with higher prediction precision and better generalization capability for penicillin fermentation process is completed.
The remainder of this paper is organized as follows. Section 2 begins with the revisit of LSSVM algorithm and lays out the mathematical formulations. Detailed descriptions of improved LSSVM based on fuzzy pruning algorithm are provided in Section 3. Two numerical simulation examples are illustrated in Section 4 which aims to demonstrate the effectiveness of the proposed method in developing soft sensors. Thereafter, a soft sensor application for the penicillin fermentation process using the proposed approach is presented in Section 5. Section 6 draws conclusions based on the results obtained in this paper.
2. The LSSVM Revisit
Given the training data , and denote the input patterns and onedimension output data, respectively. Similar to the standard SVM, LSSVM nonlinear regression is mapping the data to a higher dimension space by using a nonlinear function and constructing an optimal linear regression function in the higher dimension space: Here is the weight value and is the threshold.
The main difference between LSSVM and SVM is that LSSVM adopts the equality constraints instead of inequality constraints, and empirical risk is the deviation of the quadratic rather than one square deviation. By introducing the Kernel function and the penalty factor , one considers the following optimization problem:
To solve the optimization problem, the constrained optimization problem should be converted to unconstrained optimization problem first. By introducing Lagrange multiplier , we obtain the following Lagrange function as follows:
Then according to the Mercer condition, the specific form of the nonlinear mapping does not need to be known a priori. Suppose the kernel function takes the form ; this optimization problem could be changed into several linear equations. Based on the conditions of KarushKuhnTucker, calculating the partial derivative of with respect to , , , and , respectively, and setting to zero yield To simplify the equations, we can get a compressed matrix equation: where , , , , denotes the penalty factor, and denotes the identity matrix. Solving the matrix equation (5), eventually the function of least squares vector machines is estimated as
3. Improved LSSVM with Fuzzy Pruning Algorithm
3.1. The Idea of Fuzzy Pruning Algorithm
Compared with SVM, the computational load of LSSVM is reduced greatly. However, LSSVM loses its sparseness because all training data are treated as support vectors even the outliers which always have a bad influence on the precision of the soft sensor model. In this paper, aiming to minimize effects of the outliers as well as the antidisturbance ability of sampling data [23, 24], fuzzy pruning approach is employed to handle the problem. The number of the support vectors is reduced which improves the sparseness of LSSVM and model accuracy as well. Furthermore, the sensitivity to outliers of the proposed algorithm can be reduced through the fuzzy membership score assigned to the data samples.
The absolute value of Lagrange multiplier determines the importance of data in the training process which means the higher the absolute value, the greater the influence degree. The absolute value of Lagrange multiplier of outliers is often higher than that of the normal data. Based on this situation, the data which have the higher absolute value of Lagrange multiplier will be cut off according to certain proportion (e.g., 5%). When these data are cut off, the impact of outlier data is minimized, and the model sparseness and accuracy are improved simultaneously.
Since Lagrange multiplier plays an important role in constructing model, a fuzzy membership score is introduced to adjust the weight of data for modeling. Fuzzy membership value is defined as where is the fuzzy membership score and is the Lagrange multiplier of the th sample data. Meanwhile, need to be given an appropriate value between 0 and 1.
It is noticed that the fuzzy membership score is near to zero when Lagrange multiplier is very small. So the corresponding sampling data may play no role in modeling, which means a part of sample data can be cut off according to the absolute value of Lagrange multiplier that is very small. As a result, the sparseness of the proposed LSSVM algorithm is further improved.
3.2. Description of Fuzzy Pruning Based LSSVM Algorithm
Adding fuzzy membership score to error , the new quadratic programming problem is expressed as follows:
Since the direct optimization is not tractable, Lagrange method is introduced to convert it to become an unconstrained optimization problem. Therefore, the Lagrange function can be obtained as
The optimization requires the computation of the derivative of with respect to , , , and , respectively. Thereafter, a set of linear equations are obtained and can be simplified as where , , , , , and denotes the penalty factor.
Eventually, the fuzzy pruning based LSSVM function takes the form as follows:
3.3. The Modeling Steps Based on Fuzzy Pruning LSSVM
The proposed LSSVM algorithm based on fuzzy pruning technique can be summarized as follows.(1)Based on the training data set , we can calculate the Lagrange multiplier .(2)Choose a suitable ; the fuzzy membership scores of training data are obtained from (7).(3)Build a new data set , and train the new data set under the scheme of fuzzy pruning LSSVM algorithm again; then we can get the new .(4)Sort the Lagrange multiplier , and cut off the data taking larger Lagrange multiplier according to certain proportion (e.g., 5%).(5)Then the fuzzy pruning based LSSVM algorithm is applied to train the current data set. If the fitting performance degrades, the training procedure is done. Otherwise, switch to (4).
4. Two Numerical Simulations
4.1. OneDimension Function
The effectiveness and efficiency of handing the outliers through the proposed approach are evaluated through two numerical functions. All the simulation experiments are run on a 2.8 GH CPU with 1024 MB RAM PC using Matlab 7.11.
Consider onedimension function defined as follows: 100 data are generated in randomly as the training data set. To test the performance of detecting outliers, 30% disturbance is added to the 20th, 40th, 60th, 80th, and 100th data sample, respectively. And another 100 data are collected for evaluation.
It can be seen from Figure 1 that the outliers have the higher value of Lagrange multiplier as mentioned above. Using PSO algorithm ( keeps linear decline from 1.2 to 0.4, population size is 20, and maximum number of iterations of the population is 200) to optimize kernel parameter and the penalty factor , then the LSSVM and fuzzy pruning LSSVM models are constructed to predict and compare (Figures 2 and 3). Figure 3 is the 45degree line comparison between different measurements. If two measurements agree with the true outputs, then all data points will fall into the black 45degree line. The blue circles denote the LSSVM measurements and the pink asterisks denote the model predictions of fuzzy pruning LSSVM. We can see that the estimation with the fuzzy pruning LSSVM fits the black line better and thus provides a superior performance compared to the LSSVM observation.
The detailed results such as the maximum absolute error (Max EE), the mean absolute error (Mean EE), and root mean square error (RMSE) are calculated and listed in Table 1. The RMSE decreased from 1.21% to 0.052%, which indicates the fuzzy pruning LSSVM has higher prediction performance and better antidisturbance.

4.2. TwoDimension Function
A twodimension function is described as
100 data are generated randomly in the range of , which makes up a training data set. Then the 20th, 40th, 60th, 80th, and 100th data points are added with 30% disturbance separately and the performance is tested by using another different 100 data. As is shown in Figure 4, Lagrange multiplier value of data points that corrupted by some disturbance always has the higher value. Compared results are shown in Figure 5. From Table 2, prediction accuracy of fuzzy pruning LSSVM is much higher than LSSVM, which indicates the five outliers have been detected and cut off effectively using the proposed method.

(a)
(b)
5. An Experiment Simulation
The Pensim simulator provides a simulation of a fedbatch fermentation process for penicillin production. The main component of the process is a fermenter, where the biological reaction takes place. It fully considers the most factors influencing the penicillin fermentation process, such as PH, aeration rate, substrate feed rate, carbon dioxide, and penicillin production. The practicability and validity of the platform have been fully verified [25–27] and it has been a benchmark problem for modeling and diagnosis detection.
In this paper Pensim simulation platform is used to generate the original 100 training data. Then 30% disturbance is added to the 20th, 30th, 40th, 60th, and 85th, respectively, and another 100 data are used as test data to verify the constructed model. The simulation results are shown in Figures 7 and 8.
To further exhibit the difference of the two methods, the indexes of Max EE, Mean EE, and RMSE of each method are also calculated and listed in Table 3.

Compared to LSSVM, the proposed approach makes RMSE decrease from 2.44% to 0.97%, which indicates the fuzzy pruning LSSVM has better prediction performance.
Lagrange multiplier values according to each data point are shown in Figure 6, and we can easily find out the outliers obviously have much bigger Lagrange multiplier. Figure 8 is the 45degree line comparison between two different soft sensors. Clearly, the fuzzy pruning based LSSVM exhibits the better capability of approximating the true process. It has effectively handled the disturbance of the outliers so that their impact on modeling is minimized to lowest.
6. Conclusions
A novel LSSVM method based on fuzzy pruning technique is investigated in this paper. Pruning algorithm is applied to cut off the outliers. Therefore the number of support vectors is reduced which improves the sparseness and accuracy of LSSVM algorithm. On the other hand, assigning different fuzzy membership score to each of the sample data makes those sample data that play a small role in soft sensor modeling not participate in the construction of the model. Furthermore, the sensitivity to the outliers of the proposed algorithm can be reduced through the fuzzy membership score. The simulation examples demonstrated that the proposed method can effectively handle the outliers and achieved satisfied performance of modeling and prediction.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
The authors thank the financial support by the National Natural Science Foundation of China (nos. 21206053, 21276111, and 61273131) and partial support by the 111 Project (B12018) and the Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD).
References
 S. Yin, H. Luo, and S. Ding, “. Realtime implementation of faulttolerant control systems with performance optimization,” IEEE Transactions on Industrial Electronics, vol. 64, no. 5, pp. 2402–2411, 2014. View at: Google Scholar
 Q.D. Yang, F.L. Wang, and Y.Q. Chang, “Soft sensor of biomass based on improved BP neural network,” Control and Decision, vol. 23, no. 8, pp. 869–878, 2008. View at: Google Scholar
 G. Liu, D. Zhou, H. Xu, and C. Mei, “Soft sensor modeling using SVM in fermentation process,” Chinese Journal of Scientific Instrument, vol. 30, no. 6, pp. 1228–1232, 2009. View at: Google Scholar
 L. Huang, Y. Sun, X. Ji, Y. Huang, and B. Wang, “Soft sensor of lysine fermentation based on tPSOBPNN,” Chinese Journal of Scientific Instrument, vol. 31, no. 10, pp. 2317–2321, 2010. View at: Google Scholar
 S. J. Qin, “Recursive PLS algorithms for adaptive data modeling,” Computers and Chemical Engineering, vol. 22, no. 45, pp. 503–514, 1998. View at: Google Scholar
 W. Li, H. H. Yue, S. ValleCervantes, and S. J. Qin, “Recursive PCA for adaptive process monitoring,” Journal of Process Control, vol. 10, no. 5, pp. 471–486, 2000. View at: Publisher Site  Google Scholar
 S. J. Qin and T. J. McAvoy, “Nonlinear PLS modeling using neural networks,” Computers and Chemical Engineering, vol. 16, no. 4, pp. 379–391, 1992. View at: Google Scholar
 D. Dong and T. J. Mcavoy, “Nonlinear principal component analysis: based on principal curves and neural networks,” Computers and Chemical Engineering, vol. 20, no. 1, pp. 65–78, 1996. View at: Google Scholar
 J. C. B. Gonzaga, L. A. C. Meleiro, C. Kiang, and R. Maciel Filho, “ANNbased softsensor for realtime process monitoring and control of an industrial polymerization process,” Computers and Chemical Engineering, vol. 33, no. 1, pp. 43–49, 2009. View at: Publisher Site  Google Scholar
 S. Yin, G. Wang, and H. Karimi, “Datadriven design of robust fault detection system for wind turbines,” Mechatronics, 2013. View at: Publisher Site  Google Scholar
 V. N. Vapnik, The Nature Statistical Learning Theory, Springer, New York, NY, USA, 1999.
 V. N. Vapnik, “An overview of statistical learning theory,” IEEE Transactions on Neural Networks, vol. 10, no. 5, pp. 988–999, 1999. View at: Publisher Site  Google Scholar
 G. Liu, D. Zhou, H. Xu, and C. Mei, “Microbial fermentation process soft sensors modeling research based on the SVM,” Chinese Journal of Scientific Instrument, vol. 30, no. 6, pp. 1228–1232, 2009. View at: Google Scholar
 X.J. Gao, P. Wang, C.Z. Sun, J.Q. Yi, Y.T. Zhang, and H.Q. Zhang, “Modeling for Penicillin fermentation process based on support vector machine,” Journal of System Simulation, vol. 18, no. 7, pp. 2052–2055, 2006. View at: Google Scholar
 J. A. K. Suykens and J. Vandewalle, “Least squares support vector machine classifiers,” Neural Processing Letters, vol. 9, no. 3, pp. 293–300, 1999. View at: Google Scholar
 X. Wang, J. Chen, C. Liu, and F. Pan, “Hybrid modeling of penicillin fermentation process based on least square support vector machine,” Chemical Engineering Research and Design, vol. 88, no. 4, pp. 415–420, 2010. View at: Publisher Site  Google Scholar
 L. LI, H. SU, and J. CHU, “Modeling of isomerization of C8 aromatics by online least squares support vector machine,” Chinese Journal of Chemical Engineering, vol. 17, no. 3, pp. 437–444, 2009. View at: Publisher Site  Google Scholar
 L. Li, K. Song, and Y. Zhao, “Modeling of ARA fermentation based on affinity propagation clustering,” CIESC Journal, vol. 62, no. 8, pp. 2116–2121, 2011. View at: Publisher Site  Google Scholar
 J. W. Cao and H. W. Ma, Microbial Engineering, Science press, Beijing, China, 2002.
 G. Guo, S. Z. Li, and K. L. Chan, “Support vector machines for face recognition,” Image and Vision Computing, vol. 19, no. 910, pp. 631–638, 2001. View at: Publisher Site  Google Scholar
 R.Q. Chen and J.S. Yu, “Soft sensor modeling based on particle swarm optimization and least squares support vector machines,” Journal of System Simulation, vol. 19, no. 22, pp. 5307–5310, 2007. View at: Google Scholar
 L. Huang, Y. Sun, X. Ji, Y. Huang, and T. Du, “Soft sensor modeling of fermentation process based on the combination of CPSO and LSSVM,” Chinese Journal of Scientific Instrument, vol. 32, no. 9, pp. 2066–2070, 2011. View at: Google Scholar
 X. Zhang, “Using classcenter vectors to build support vector machines,” in Proceedings of the 9th IEEE Workshop on Neural Networks for Signal Processing (NNSP '99), pp. 3–11, August 1999. View at: Google Scholar
 C.F. Lin and S.D. Wang, “Fuzzy support vector machines,” IEEE Transactions on Neural Networks, vol. 13, no. 2, pp. 464–471, 2002. View at: Publisher Site  Google Scholar
 S. Yin, S. Ding, A. Haghani, and H. Hao, “Datadriven monitoring for stochastic systems and its application on batch process,” International Journal of Systems Science, vol. 44, no. 7, pp. 1366–1376, 2013. View at: Google Scholar
 Y. Liu and H.Q. Wang, “Pensim simulator and its application in penicillin fermentation process,” Journal of System Simulation, vol. 18, no. 12, pp. 3524–3527, 2006. View at: Google Scholar
 S. Yin, S. Ding, A. Haghani, H. Hao, and P. Zhang, “A comparison study of basic data driven fault diagnosis and process monitoring methods on the benchmark Tennessee Eastman process,” Journal of Process Control, vol. 22, no. 9, pp. 1567–1581, 2012. View at: Google Scholar
Copyright
Copyright © 2014 Weili Xiong et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.