Abstract
In order to overcome the long training time caused by searching optimal basic functions based on greedy strategy from a redundant basis function dictionary for the intuitionistic fuzzy kernel matching pursuit (IFKMP), the particle swarm optimization algorithm with powerful ability of global search and quick convergence rate is applied to speed up searching optimal basic function data in function dictionary. The approach of intuitionistic fuzzy kernel matching pursuit based on particle swarm optimization algorithm, namely, PS-IFKMP, is proposed. This algorithm is applied to the aerospace target recognition, which requires real-time ability. Simulation results show that, compared with the conventional approaches, the proposed algorithm can decrease training time and improve calculation efficiency obviously with almost unchanged classification accuracy, while the model has better sparsity and generalization. It is also demonstrated that this approach is suitable for the application requiring both accuracy and efficiency.
1. Introduction
The kernel matching pursuit (KMP) Classifier is a new classification technique proposed by Vincent and Bengio in 2002 [1]. The basic idea of KMP originates from matching pursuit (MP) method of sign processing theory, a greedy constructive algorithm that approximates a given function by a linear combination of basis functions chosen from a redundant basis function dictionary, and can be seen as a form of boosting. When training a KMP Classifier, there are mainly three steps. First, it maps training data into a redundant basis dictionary by a given kernel function and then uses greedy algorithm to find a linear expansion of basis functions that are selected from a redundant dictionary of functions to minimize the loss function and, at last, adjusts the corresponding coefficients by backfitting algorithm. The linear expansion of basis functions is the target function that we seek. The performance of the KMP Classifier is comparable to that of support vector machine (SVM) for classification, while typically requiring far fewer support points [2, 3]. Compared with the other Kernel methods, KMP is very flexible and has no constraint on the form of kernels. It can be allowed to mix several shapes of kernels in one instance. Up to now, the theory of KMP has been successfully applied to a variety of fields, such as pattern classification [4, 5], image recognition [6], and intrusion detection [7].
However, there is a special case in practical application; the importance degree (or threat degree) of one type is higher than the other targets, so we need to classify the appointed important target much more precisely and reduce the recognition accuracy of the other unimportant targets. For example, in missile defense battle, the requirement of recognition accuracy of the true warhead is higher than that of bait, debris, and so forth. The traditional kernel matching pursuit algorithm deals with all samples equally and the decision function gives a full scale consideration for all samples in order to minimize total recognition error.
Unfortunately, such algorithm cannot perform well in special cases. To circumvent the disadvantage, Li et al. [8] proposed a fuzzy kernel matching pursuit (FKMP) machine, which can classify the appointed important samples much more precisely according to the predefined importance of the data. However, this method sets the fuzzy factor on the basis of artificial experience, which may bring risks to the training process. Lei et al. [9] proposed intuitionistic fuzzy kernel matching pursuit (IFKMP) machine and expanded KMP algorithm into intuitionistic fuzzy field to solve the problem that some important data cannot be classified precisely by assigning the intuitionistic fuzzy parameters to different samples efficiently. Nevertheless, the IFKMP machine essentially uses MP greedy algorithm to find a linear combination of basis functions that are selected from a redundant dictionary of functions, so the problem of greatly computer burden has not been solved, and the computing time can be increased greatly with searching spaces increasing, which may be more serious when the dictionary has two or few functions [10]. To overcome the above drawback, the optimization methods have been introduced into MP algorithm. An improved genetic algorithm (GA) is proposed to simplify the calculation of MP algorithm in [11]; however, due to GA’s prematurity, the stable capability of learning machine is not good. In [12], a quantum genetic algorithm is introduced for modifying MP process; however, the computing time cannot reduce effectively due to the slow search speed of quantum genetic algorithm. In [13], a novel technique for KMP based on intuitionistic fuzzy -means (IFCM) was proposed, but IFCM is still a local optimization algorithm and reduces the recognition accuracy of KMP. Particle swarm optimization (PSO) proposed by Kennedy and Eberhart [14] is a global optimization algorithm. Since its appearance, for its powerful ability of global search and quick convergence rate, PSO has received more and more attention [15, 16]. Based on analysis above, this paper put forward a novel technique for intuitionistic fuzzy kernel matching pursuit based on particle swarm optimization (PS-IFKMP), where the greedy algorithm is replaced by PSO algorithm. In order to test the effect and validity of PS-IFKMP algorithm, three different datasets are used for simulation. The results are compared with those based on KMP and IFKMP. Experiments results validate the preference and efficiency of PS-IFKMP.
2. Intuitionistic Fuzzy Kernel Matching Pursuit
2.1. Matching Pursuit
Matching pursuit was introduced in the signal-processing community as an algorithm that decomposes any signal into a linear expansion of waveforms that are selected from a redundant dictionary of functions.
Given noisy observations at and a finite dictionary of functions in Hilbert space , we aim to find sparse approximations of that are expansion of the form , which minimize the squared norm of the residue , where and .
The process of MP can be described as follows: it starts at stage 0 with and and recursively appends functions to an initially empty basis. Given , we build , by searching for and that minimize the squared norm of the residue ,Moreover,From above, we can see that MP actually adopts greedy algorithm. In each iterative process, MP finds the most correlative basis function in with the current residue . As the basis is appended continuously, can approach observations more and more accurately. However, algorithm will generally stop when the norm of the residue goes below a predefined threshold or the number of the bases reaches the maximum.
2.2. Kernel Matching Pursuit
Kernel matching pursuit is simply the idea of applying the matching pursuit family of algorithms to problem in machine learning, using a kernel-based dictionary [5]. Given a kernel function , we construct the basis dictionary of KMP on the training data: .
Kernel method is enlightened in great part to the success of the SVM, but there are some differences between them. The kernel used in SVM must satisfy the Mercer condition, while kernel in KMP has no such restriction. Simultaneously, one can use more than one kernel when constructing the function dictionary of KMP. There are some kernels in common use:(1)polynomial kernel: ,(2)radial basis function (RBF) kernel: (3)Sigmoid kernel: .
2.3. Intuitionistic Fuzzy Kernel Matching Pursuit
By assigning the intuitionistic fuzzy parameters to different samples efficiently, IFKMP algorithm can classify the target samples of diverse importance much more precisely according to the predefined importance of samples.
Definition 1 ( operation). Let and be two vectors; then we define the operation as Moreover,Given noisy samples , where represents the sample feature, represents the sample observation, and is intuitionistic fuzzy parameter and given a kernel function , we construct the basis dictionary on the training data: .
Then the residue iswhere represents the estimated value of the th sample; then the squared norm of the residue can be written asAccording to MP algorithm, we can getThen,For any , that minimizes the squared norm of the residue is given byFor this optimal value of , we haveSo that minimizes expression (9) is the one that minimizes (11); we can getand the corresponding coefficient isAt every step in this algorithm, not only the set of basis functions but also their coefficients are obtained. So, the approximate form of observations in th iteration isHowever, when appending , the linear combination may not be optimal. So a backfitting method is also done by reoptimizing all coefficients to minimize the target cost:Finally, we can get the approximation functions in regression: or decision function in classification:where is the support vector set obtained by IFKMP algorithm.
2.4. The Selection of Intuitionistic Fuzzy Parameter
In practice, the recognition accuracy of different targets is different according to the threat degree. Generally, higher threat degree of the targets needs higher recognition accuracy. Aiming at this fact, the selection algorithm of intuitionistic fuzzy parameter can be listed as follows.
Algorithm 1 (the selection algorithm of intuitionistic fuzzy parameter).
Input. Input is the type of sample .
Output. Output is the intuitionistic fuzzy parameter .
Step 1. Determine as appointed important target or nonappointed target.
Step 2. Calculate the membership degree and nonmembership degree ; this paper uses Gaussian function as membership function, so where and denote width and center point, respectively, is denoted as nonhesitancy degree, and denote the uncertainty degree which must be predefined on the basis of practice use.
Step 3. Calculate the intuitionistic fuzzy parameter by the following formula:
The intuitionistic fuzzy parameter can distinguish the appointed important target and nonappointed important target specifically and classify the appointed important samples much more precisely. Moreover, the total misclassification rate will still be lower even with a higher misclassification rate of the nonappointed important target.
3. IFKMP Based on Particle Swarm Optimization
Theoretically IFKMP algorithm solves the problem that some important data cannot be classified precisely according to the predefined importance of the data efficiently, but its implement is a greedy algorithm. Greedy algorithm requires that every step of searching process be global optimal searching in the redundant dictionary of function in order to select best matching signal structure, from which the large amounts of computing time have often not suffered. Particle swarm optimization algorithm with powerful ability of global search and quick convergence rate solves the problem, by replacing greedy algorithm to find a linear expansion of basis functions.
3.1. Theoretical Analysis
We notice that a bottleneck of the IFKMP algorithm is represented by the search of the current best element from the redundant dictionary to be added in the function expansion. Usually, this requires a full search over the whole dictionary and may necessitate a large number of floating point operations. An alternative is provided by the so-called Weak Greedy Algorithm (WGA) [17] which provides an approximation of the MP and related greedy algorithms. WGA and its different formulations have been analyzed in [17, 18] and proofs of convergence of the algorithm exist under various conditions. Unlike classical greedy algorithm, WGA generates an approximate sequence as follows:where , and, clearly, for , WGA retrieves the original algorithm. The sequence is named weakness sequence and it must obey some constraints for the algorithm to converge. While different conditions on result in different guaranteed convergence rates, we will simply require that such that , , which ensures the convergence [18]. These modifications imply that we are no longer forced to produce the global maximum at each iteration of the IFKMP algorithm but just an approximate optimal value. So we can use PSO algorithm to search the next element from the redundant dictionary instead of greedy algorithm. Compared with greedy algorithm, the PSO algorithm may only find an approximate optimal value; we still can ensure the convergence of PS-IFKMP algorithm according to the theory of WGA. As a consequence, the performance of the decision function we get from PS-IFKMP algorithm is similar to the decision function from IFKMP algorithm under the same termination condition. Based on this point, we believe that PS-IFKMP algorithm can decrease computational complexity while classification accuracy remains statistically the same by using PSO algorithm to find a linear expansion of basis functions.
3.2. Particle Swarm Optimization
Particle swarm optimization is an evolutionary optimization technique based on metaphors for social interaction and communication such as flocks of birds and schools of fish. This stochastic, population-based approach has been proven effective for solving both continuous and discrete optimization problems. Each particle in a swarm, which is analogous to a bird in a flock or a fish in a school, moves around in dimensional search space. Based on its own experience and that of the swarm, it moves toward the best position in the search space [19].
The position and velocity of particle at iteration are represented by and , which can be defined as and , respectively, and the personal best (best) of particle is represented by , which denotes the position of particle with the best fitness value found so far and is defined as . The global best (best) of all particles at iteration is represented by , which denotes the best position of the particle with the best fitness value in the swarm found so far and is defined as . The new velocity and position of particle can be obtained by (20) and (21), respectively:where represents velocity of particle at iteration with respect to the th dimension and is new velocity at iteration , is position of particle at iteration with respect to the th dimension, and is new position at iteration . is position value of the th best at iteration with respect to the th dimension and is position value of the best at iteration . is current iteration. , are acceleration coefficients. , are uniform random numbers between 0 and 1. is the inertia weight. Moreover, in order to make the speed of the particle not too high, a speed threshold should be set. If , then , and if , then .
3.3. Target Recognition Algorithm of IFKMP Based on PSO
IFKMP algorithm uses greedy algorithm to find a linear combination of basis functions, that is, the decision function in classification from a redundant dictionary of functions to minimize the objective function . The PS-IFKMP algorithm first maps training data into a redundant basis dictionary by a given kernel function and then sets the population size according to the scale of the training dataset. At last, the linear expansion of basis functions is found by using PSO algorithm. Because the larger the value of linear expansion of basis functions, the more effective the Classifier, formula (16) is selected as the fitness function. The detailed steps of PS-IFKMP algorithm are listed as follows.
Algorithm 2 (intuitionistic fuzzy kernel matching pursuit based on particle swarm optimization).
Input. Input is the training dataset , the intuitionistic fuzzy parameters , and the kernel parameter .
Output. Output is the decision function .
Step 1 (initialization). Set the maximum iterations and iterative threshold of IFKMP, set the population size and the maximum iterations of PSO, and suppose generation , .
Step 2. Construct the basis dictionary on the training data: .
Step 3. Calculate coefficients based on the rule ; then are selected as the initial population of particles randomly.
Step 4. Measure the fitness of each particle in the population by .
Step 5. Update particles according to formulas (21) and (22); then update the best and best .
Step 6. If , stop the iteration; otherwise , and go to Step 4.
Step 7. Let , and if and , go to Step 3 by using the updated and solve the next basic function and weight coefficient .
Step 8. Recompute the optimal set of coefficients by backfitting algorithm; then output the linear combination of the best weight coefficients and the corresponding basic functions, which is also the decision function .
The flowchart of Algorithm 2 is shown in Figure 1.
3.4. Parameters Setting of PS-IFKMP
The inertia weight can be a positive constant or even a positive linear or nonlinear function of time. When , the velocity item becomes the main item in the search direction of the particle. It extends the search area and finds the global optimum. When is between 0.8 and 1.2, three factors, velocity, best, and best, affect the velocity calculation for both local search and global search. When , only the best and best affect the new velocity calculation, which converges to the local optimum. Therefore, the value of inertia weight is a tradeoff between the global search and the local search. In this paper, the inertia weight starts with a high value and nonlinearly decreases to at the maximal number of iterations [20]:where and are the initial and final values of the inertia weight, respectively, denotes the current number of iterations, and denotes the maximum number of iterations. The value is a constant coefficient.
The acceleration factors and are positive constants controlling the relative impact of the personal (local) and common (global) knowledge on the movement of each particle, and we improve the selection of acceleration factors by introducing inheritance mechanism:where is the fitness value of particle at iteration and the convergence rate of PS-IFKMP speeds up efficiently by introducing inheritance mechanism.
In order to keep the diversity of the population, the Gaussian distribution is introduced. If the has not been improved in a few steps, a certain proportion of particles should be randomly selected for mutation:where is a Gaussian random number with mean of 0 and standard deviation of 1.
3.5. Complexity Analysis
The performance of the proposed PS-IFKMP algorithm is analyzed in theory in this section. The IFKMP is a novel algorithm which was proposed in 2011, and no relevant improved algorithms have been proposed at present. Therefore we mainly compared theoretical complexity of the treatment of the proposed PS-IFKMP algorithm with IFKMP algorithm. IFKMP algorithm maps training data into a high-dimensional Hilbert space by a given kernel function to construct the basis function dictionary and then searches the best basis functions and the corresponding weight in each iteration. So the computation complexity can be described by the number of multiplications in one matching process in this paper.
Assuming that the size of dictionary is and the iteration number is , IFKMP algorithm requires single-precision multiplications in one matching process, and the complexity can be described as based on the computational analyses of Montgomery reduction algorithm. In PSO algorithm, position, velocity, and fitness of each particle need to be calculated in every iteration, so the algorithm requires single-precision multiplications in one matching process, where is the population size and the complexity of PS-IFKMP can be described as . As the population size is far less than the dictionary size , the computational cost of PS-IFKMP is far smaller than that of IFKMP when is quite big, while the costs are familiar when is small.
4. Experiment Results and Analysis
This section evaluates the performance of the proposed PS-IFKMP algorithm by comparing its classification results with those obtained for the same cases by the KMP in [1] and IFKMP in [9] where PSO is encoded by real number and RBF kernel is used for both algorithms. To avoid the weak problem, each experiment has been performed at 50 independent runs. The average results and standard deviations (Std. in short) are given. All experiments were carried on a Pentium (R) Dual-Core CPU E5500 @2.8 GHz with 2 GB RAM using MATLAB 7.6 compiler.
4.1. Test on UCI Data
In this experiment, five UCI datasets of Musk, Waveform, German, Diabetes, and Breast Cancer Wisconsin (Breast in short) are used to test our methods, where Waveform contains three classes, and two of them (class 0 and class 2) are selected as experimental dataset. The characteristics of the UCI datasets are shown in Table 1, which shows a wide range of attributes sizes and class sizes.
The selection of different kernel function parameters has great influence on algorithm performance, so this paper validates the kernel parameter by a special training dataset. The intuitionistic fuzzy parameters are set as , , , , and and the kernel function parameter is taken in by equal interval sampling. The influence of on the accuracy is shown in Figure 2.
(a) Accuracy varying with based on Musk data
(b) Accuracy varying with based on Diabetes data
(c) Accuracy varying with based on Waveform
(d) Accuracy varying with based on German data
(e) Accuracy varying with based on Breast data
From Figure 2, we can notice that the optimal kernel function parameters based on different datasets are different. Validated by experiments, , 3, and 6 for Musk, Waveform, and German, respectively. There is no special requirement on the importance of samples; we can set ; then IFKMP algorithm is equivalent to KMP algorithm, so we only compare PS-IFKMP with the KMP in this section; the results are listed in Tables 2–4.
From the above experiments, the average accuracy of KMP and PS-IFKMP is 85.37% and 86.34%, respectively, so the performance of the PS-IFKMP is competitive to that of KMP for classification. However, the average training time in one matching process of KMP and PS-IFKMP is 0.16 and 0.048, respectively; compared with KMP, the average training time of PS-IFKMP is decreased by 70%. So the proposed PS-IFKMP algorithm can decrease obviously training time with almost unchanged classification accuracy, especially for the large size datasets.
Diabetes contains 500 negative samples and 268 positive samples, and Breast contains 241 negative samples and 458 positive samples. As we know, the positive samples describe the pathological features of diseases, so the positive samples should be classified much more precisely in terms of medical diagnosis. Validated by experiments, and 3 for Diabetes and Breast, respectively. The intuitionistic fuzzy parameters are selected by Algorithm 1, of appointed positive samples is 1.6, and of nonappointed negative samples is 0.3. The results are listed in Table 5.
From Table 5, facing imbalanced training set, standard KMP cannot classify the appointed important samples efficiently. IFKMP solves the aforementioned problem by assigning the intuitionistic fuzzy parameters to different samples and studying the weak samples sufficiently. Compared with IFKMP, the proposed PS-IFKMP algorithm can decrease training time obviously, and the classification accuracy is almost unchanged simultaneously.
4.2. Test on Artificial Data
Learning to tell two spirals apart is important both for purely academic reasons and for industrial application. In the research of the pattern recognition, it is a well-known problem for its difficulty. The parametric equation of the two spirals can be presented as follows:where , , , and are parameters. In our experiment, we choose , , and . We generate 12000 samples randomly, and the samples distribution is shown in Figure 3(a). Moreover, a set of concentric circles samples also is selected to test our methods, and the parametric equation of the concentric circles data can be shown as Parameter of the first class is of continuous uniform distribution and the second class . 16000 samples are generated randomly, and the samples distribution is shown in Figure 3(b). Setting , , , , and , the intuitionistic fuzzy parameters are selected by Algorithm 1; of appointed important samples (shown by “” in Figure 3) is 1.5 and of nonappointed samples (shown by “×” in Figure 3) is 0.3. Before the training, noise is added to training data, randomly choosing 10% samples and changing its class attributes. The results are listed in Table 6.
(a) The distribution map of two-spirals data
(b) The distribution map of concentric circles
In Table 6, it is easily found that KMP cannot classify the appointed important samples efficiently when training Classifier with a subset of the data. IFKMP can classify the appointed important samples much more precisely by assigning the intuitionistic fuzzy parameters to different samples and studying the weak samples sufficiently. The proposed PS-IFKMP with advantage of both PSO and IFKMP reduces the training time enormously with the classification accuracy almost unchanged.
4.3. Test on Aerospace Target Data
In order to test the performance of the proposed PS-IFKMP algorithm in practical application, the aerospace targets data are selected for experiment. Generally, the threat degree of tactical ballistic missile (TBM), cruise missile (CM), and stealth aircraft (SA) is higher than the other targets, so we set these three targets as the appointed important targets. The radar cross section (RCS) values of the target are simulated by FEKO software, and the maximum, minimum, mean, and variance of RCS sequence are selected as sample features. And the RCS sequence of TBM within 30 seconds is showed in Figure 4.
Beside the appointed important targets, a large number of debris, baits, and other aircrafts, and so forth, are selected as negative samples. Setting , , , , and , the of TBM equals 1.5, the of CM equals 1.3, and the of CM equals 1.6 by Algorithm 1. The results are listed in Table 7.
From Table 7, by providing equal treatment on all training samples, the classical KMP algorithm cannot classify the weak important data efficiently. The IFKMP circumvents this problem by studying the important samples sufficiently and the less important samples roughly; however, the complexity problem still exists by adopting the greedy algorithm. By introducing particle swarm optimization algorithm, PS-IFKMP decreases the time complexity and reduces the training time with almost unchanged classification accuracy. It is also demonstrated that this approach is suitable to the application requiring both accuracy and timeliness.
5. Conclusion
Theoretically IFKMP algorithm is an excellent method but its implement is based on a greedy algorithm. Therefore, the defect of great computer time has not been solved when dealing with huge volume of data. In order to overcome the limitation, we propose the PS-IFKMP algorithm which optimizes searching process based on particle swarm optimization and reduces computational complexity of IFKMP algorithm. Simulation results show that, compared with the conventional approaches, the proposed algorithm can decrease training time and improve calculation efficiency obviously with the classification accuracy almost unchanged, especially for large size datasets. However, further improvements are needed by this algorithm, and the future research should focus on problems such as how to provide better multiclass recognition performance and how to select the parameters of the kernel function.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgment
The work was supported by the National Natural Science Foundation of China (nos. 61272011 and 61309022).