Research Article | Open Access
Zhe Zhang, Xiyu Liu, Lin Wang, "Spectral Clustering Algorithm Based on Improved Gaussian Kernel Function and Beetle Antennae Search with Damping Factor", Computational Intelligence and Neuroscience, vol. 2020, Article ID 1648573, 9 pages, 2020. https://doi.org/10.1155/2020/1648573
Spectral Clustering Algorithm Based on Improved Gaussian Kernel Function and Beetle Antennae Search with Damping Factor
There are two problems in the traditional spectral clustering algorithm. Firstly, when it uses Gaussian kernel function to construct the similarity matrix, different scale parameters in Gaussian kernel function will lead to different results of the algorithm. Secondly, K-means algorithm is often used in the clustering stage of the spectral clustering algorithm. It needs to initialize the cluster center randomly, which will result in the instability of the results. In this paper, an improved spectral clustering algorithm is proposed to solve these two problems. In constructing a similarity matrix, we proposed an improved Gaussian kernel function, which is based on the distance information of some nearest neighbors and can adaptively select scale parameters. In the clustering stage, beetle antennae search algorithm with damping factor is proposed to complete the clustering to overcome the problem of instability of the clustering results. In the experiment, we use four artificial data sets and seven UCI data sets to verify the performance of our algorithm. In addition, four images in BSDS500 image data sets are segmented in this paper, and the results show that our algorithm is better than other comparison algorithms in image segmentation.
Clustering analysis is an important research problem in the field of data mining. The purpose of clustering is to divide the data set into different clusters according to the intrinsic structure and relationship between the data so that the similarity between data points within the same cluster is higher, and the similarity between data points in different clusters is lower. The main clustering methods include partitioning-based clustering, hierarchical clustering, density-based clustering, grid-based clustering, and graph theory-based clustering. Different clustering algorithms are also applied to different fields, such as image segmentation [1–4], text clustering [5, 6], and community division [7–9].
Spectral clustering is a kind of clustering algorithm based on graph theory. By spectral graph partition theory , the clustering problem of the data set is transformed into the graph partition problem. In spectral clustering, each data point is regarded as the vertex of the graph, and the similarity between data points is regarded as the weight of the edge. By dividing the graph, the sum of the weight of the edge in the subgraph is as high as possible, and the sum of the weight of the edge between different subgraphs is as low as possible.
In 1973, Donath and Hoffman  first proposed the concept of graph partition based on the adjacency matrix, marking the formal birth of spectral clustering. In the same year, Fiedler  found that the two-way partition of the undirected graph is closely related to the eigenvector corresponding to the second small eigenvalue of the corresponding Laplacian matrix, which provides a new way to solve the problem of graph partition. In 2000, Shi and Malik  put forward the standard cut objective function, also known as the N-cut criterion, based on the spectral theory. In 2001, Ding et al.  put forward the minimum and maximum cut-set criterion based on N-cut, which balances the two requirements of minimum division loss and maximum vertex number of subgraphs, making division more inclined to balance the cut set and avoiding segmentation of smaller subgraphs with only a few vertices. In 2002, Jordan, Weiss, and Ng  proposed NJW algorithm, which is different from two-way division. The algorithm is based on k-way division, and it is also the most widely used spectral clustering algorithm so far. Despite the good development of spectral clustering, there are still some problems with the algorithm itself, such as how to select the scale parameters in the Gaussian kernel function. In 2004, scholars  have proved that the selection of scale parameters will affect the clustering results. To solve this problem, Zhang et al.  proposed a construction method of the similarity matrix based on local density. Nataliani and Yang  proposed an energy Gaussian kernel function to solve this problem.
Beetle antennae search algorithm (BAS) is an optimization algorithm inspired by the beetle’s foraging principle proposed by Jiang and Li  in 2017. By simulating the detection function of beetle’s tentacles and the mechanism of beetle’s random walking, an optimization mechanism similar to beetle’s foraging process is realized. According to the smell of food, the moving direction of the beetle is determined. When the smell of the left tentacle is strong, it will move to the left; otherwise, it will move to the right. Through the random orientation mechanism and variable step size mechanism, a beetle can search in the global scope. Compared with other intelligent algorithms, the algorithm does not need to know the specific form of gradient information and function and has the advantages of fast convergence speed and low requirements for parameters. So, it has been applied in some fields. Wang and Liu  combined the reverse neural network with the BAS algorithm to predict the loss of storm disaster. Chen et al.  used the particle swarm optimization algorithm based on the BAS algorithm to solve the portfolio model. Wang and Chen  proposed a kind of bee swarm antenna search algorithm (BSAS).
The main contributions of this paper are as follows: (1) A construction method of the similarity matrix is proposed, which uses the distance information of some nearest neighbors to define the scale parameter σ to overcome the influence of artificial designated scale parameter σ on the results. (2) In the clustering stage, we use the proposed beetle antennae search algorithm with damping factor (DBAS) to complete the clustering. Through such an intelligent optimization algorithm, we can overcome the impact of random initialization of the cluster center on the results when K-means is used in the traditional spectrum clustering. And the damping factor overcomes the oscillation in the iterative process and improves the stability of the algorithm.
The content of this paper is organized as follows. In Section 3, an improved spectral clustering algorithm based on the distance information of some nearest neighbors and beetle antennae search algorithm with damping factor is proposed. Section 4 shows the performance of the algorithm through experimental analysis. The conclusion will be presented in Section 5.
2. Spectral Clustering and Beetle Antennae Search Algorithm
2.1. Spectral Clustering
The spectral clustering algorithm uses the eigenvectors of the Laplacian matrix corresponding to the data set to cluster. In the spectral clustering algorithm, firstly, an undirected graph is constructed according to the data points. Each vertex on the graph corresponds to a data point, and the weight on the edge is the similarity between the data points. In general, we use Gaussian kernel function to construct the similar matrix. Then, we can get a degree matrix , whose main diagonal element is equal to the sum of the row elements corresponding to the similar matrix. There are usually three ways to construct the Laplacian matrix : (1) denormalized Laplacian matrix, (2) normalized symmetric Laplacian matrix , and (3) normalized asymmetric Laplacian matrix . The eigenvector corresponding to the first k eigenvalues of the Laplace matrix can be calculated and set . Then, a new feature matrix is obtained by normalizing . Each row in the feature matrix is regarded as a sample, which is clustered to obtain a group of clusters . NJW algorithm  is the most commonly used spectral clustering algorithm. The basic step of the NJW algorithm is shown in Algorithm 1.
2.2. Beetle Antennae Search Algorithm (BAS)
Based on the principle of beetle’s foraging, three optimization strategies can be simplified: (1) The left and right antennae of the beetle are located on both sides of the individual. (2) The ratio of the step length of each action to the distance between two antennae is a fixed constant. (3) After a move, the direction of its head is random. Then, we can build an optimization model (the beetle is simplified as an individual):(1)For an optimization problem in the n-dimensional space, is used to represent the coordinates of the left antennae of an individual, represent the coordinates of the right antennae of an individual, and is the centroid coordinate. is the distance between two antennae. Since the orientation of the individual is random after each movement, the direction of the vector that the right of the individual points to the left is also random. It can be expressed by a normalized random vector . There is .(2)For the minimization objective function , and . If is less than , then the individual travels in the direction of the left antennae step, otherwise, the distance step of the individual toward the right antennae direction.(3)Repeat step 1 and step 2 until the maximum number of iterations is reached or the individual does not change in M iterations.
3. Improved Spectral Clustering Algorithm
In this section, we improve Gaussian kernel function and BAS algorithm, respectively. After using the new Gaussian kernel function to construct the similarity matrix, we use the spectral clustering algorithm to get a new feature matrix, and then, we use the improved BAS algorithm to cluster.
3.1. An Improved Gaussian Kernel Function
In the traditional spectral clustering, the similarity matrix is usually constructed according to the Gaussian kernel function in the formula of Algorithm 1, where is the scale parameter; in general, the scale parameter is selected artificially. In 2004, scholars  had proved that the selection of scale parameters will affect the clustering results. In order to solve this problem, this paper proposes a method of constructing a similarity matrix based on the distance information of some nearest neighbors:where , which is the mean distance of the nearest points from point . is the ratio of the total number of samples to the square of the number of clusters. , where is the total number of samples and is the number of clusters.
3.2. Beetle Antennae Search Algorithm with Damping Factor (DBAS)
As mentioned in Section 2.2, the direction of the individual is random in each iteration. This results in more oscillations in the process of algorithm iteration. It is possible that the result of the M + 1 iteration is worse than that of the M iteration many times. We proposed to add a damping factor to the formula of the position update of the individual, which updates the position information by using the results of this iteration and the last iteration. The formula is described aswhere indicates the position in the t − 1th iteration, .
We use the algorithm with damping factor and the algorithm without damping factor to experiment on the Iris data set. Figure 1 shows that adding damping factor to the algorithm can effectively overcome the oscillation problem in the iterative process.
3.3. SC-DBAS Algorithm
Firstly, we use the Gaussian kernel function based on the distance information of some nearest neighbors (formula 2) to construct the similarity matrix and then calculate the corresponding degree matrix and Laplace matrix. We select the eigenvectors corresponding to the first k minimum eigenvalues of the Laplace matrix to construct an eigenmatrix and then normalize it to get a new eigenmatrix. Each row of the matrix is regarded as a sample point. For such a new data set, we randomly initialize a group of cluster centers as an individual and then use DBAS algorithm to cluster. SC-DBAS algorithm flow is given in Algorithm 2.
3.4. Computational Complexity
The computational complexity of the proposed algorithm can be calculated as follows: the SC-DBAS algorithm is divided into three parts: (1) constructing a similar graph, which needs , (2) eigenvalue decomposition, which needs , and (3) clustering by using DBAS algorithm, which needs , where k is the number of cluster centers and l is the number of iterations. According to the notation of big O, the computational complexity of the proposed algorithm is .
4. Experimental Results and Analysis
4.1. Experimental Setting
All the experiments are conducted on the computer with Intel core i5-3230M CPU, 8 GB RAM. The experiment environment is Matlab 2016b. In the experiment, we compare the proposed algorithm with the K-means, NJW , MPSC algorithm , PGSC algorithm , and SC-NP algorithm  on four artificial data sets and seven UCI data sets. The proposed algorithm will also use the image in the BSDS500 data set for image segmentation. In the experimental part of image segmentation, the comparison algorithm is K-means, NJW , PGSC algorithm , and SC-NP algorithm .
In the experiment, the parameters are set as follows: step = 0.1; step adjustment factor eta = 0.95; the ratio between step and is 5; the number of iterations n = 100; and damp = 0.5. The information of data sets is shown in Table 1.
4.2. Evaluation Indicators
In the experiment, we use four indicators to evaluate the clustering results: accuracy, ARI, F1 score, and time (s).(1)Accuracy rate: the accuracy rate represents the proportion of the number of correct clustering samples to the total number of samples, where V is the division label and U is the real label:(2)ARI: there are four cases by comparing the calculation results V with the real label U. SS contains sample pairs that belong to the same cluster in V and the same cluster in U. SD contains sample pairs that belong to the same cluster in V but not the same cluster in U. DS contains sample pairs that do not belong to the same cluster in V but belong to the same cluster in U. DD contains sample pairs that do not belong to the same cluster in V and do not belong to the same cluster in U. Set ; there are
The larger the value of ARI means that the clustering results are more consistent with the real situation.(3)F1 score: F1 score is one of the commonly used evaluation criteria in information retrieval. It is a weighted harmonic mean value based on precision and recall. Its definition is as follows, where a, b, and c have been defined in the above content:(4)Time: in this paper, we use the average time of each algorithm running 100 times as the evaluation index.
4.3. Data Set Experiment Result Analysis
4.3.1. Experimental Results of Artificial Data Sets
Table 2 shows the experimental results of the six algorithms on the four artificial data sets. From Figure 2, we can see that our proposed algorithm can well divide the data sets of various structures.
4.3.2. Experimental Results of UCI Data Sets
Table 3 and Figure 3 show the experimental results of the six algorithms on seven UCI data sets. By comparing the results, we can see that the algorithm proposed in this paper performs better than the other five algorithms and has a shorter running time.
4.4. Application of the SC-DBAS Algorithm to Image Segmentation
Clustering-based image segmentation is based on the similarity between image pixels; through some clustering algorithms, the pixels are divided into different clusters so as to complete the segmentation of the original image.
In this section, we segment some images of the BSDS500 data set. For a 481∗321 pixels image, if we treat each pixel as a data point, there will be 154,401 data points. Therefore, in order to reduce the scale of data points, we first use SLIC algorithm  to perform presegmentation (superpixel segmentation) on the image. Each superpixel is an oversegmented region and is considered as a data point. Then, the proposed algorithm is used to segment the image. In the experiment, the number of superpixels of each image is 200. The comparison algorithm used in the experiment is K-means, NJW , PGSC algorithm , and SC-NP algorithm . Then, we can get the results which are given in Figure 4.
From the experimental results, we can see that our algorithm can segment the object and the background better, while the other four comparison algorithms will have the wrong segmentation area. The segmentation accuracy results are shown in Table 4.
In this paper, an improved spectral clustering algorithm combined with the improved BAS algorithm is proposed. The proposed algorithm first improves the construction of the similarity matrix, which uses the distance information of some nearest neighbors of each point to calculate the corresponding scale parameters. In the stage of clustering, we proposed BAS algorithm with damping factor to cluster, which can overcome the problem that the original algorithm oscillates many times in the iterative process. The experimental results show that our algorithm is better than other algorithms in UCI data sets, artificial data sets, and image segmentation. However, in the application of image segmentation, our results will be affected by the effect of superpixel segmentation. The future work is to improve our algorithm so that it does not need to preprocess in image segmentation and can directly segment the image, and we will use more real images and medical images to verify our algorithm.
The four artificial data sets that were manually generated can be obtained by contacting the author. The seven UCI data sets are often used in the existing literature which are from the UCI Machine Learning Repository available at http://archive.ics.uci.edu/ml/datasets.php. The four tested images are from the Berkeley computer vision group, Berkeley segmentation data set, and benchmark 500 (BSDS500), which are available at https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html.
Conflicts of Interest
The authors declare that they have no conflicts of interest.
This research project was supported by the National Natural Science Foundation of China (61876101, 61802234, and 61806114), Social Science Fund Project of Shandong Province, China (16BGLJ06 and 11CGLJ22), Natural Science Fund Project of Shandong Province, China (ZR2019QF007), Postdoctoral Project, China (2017M612339 and 2018M642695), Humanities and Social Sciences Youth Fund of the Ministry of Education, China (19YJCZH244), and Postdoctoral Special Funding Project, China (2019T120607).
- A. W.-C. Liew, H. Hong Yan, and N. F. Law, “Image segmentation based on adaptive cluster prototype estimation,” IEEE Transactions on Fuzzy Systems, vol. 13, no. 4, pp. 444–453, 2005.
- F. Tung, A. Wong, and D. A. Clausi, “Enabling scalable spectral clustering for image segmentation,” Pattern Recognition, vol. 43, no. 12, pp. 4069–4076, 2010.
- W. Yan, S. Shi, L. Pan, G. Zhang, and L. Wang, “Unsupervised change detection in SAR images based on frequency difference and a modified fuzzy c-means clustering,” International Journal of Remote Sensing, vol. 39, no. 10, pp. 3055–3075, 2018.
- H. Ali, L. Rada, and N. Badshah, “Image segmentation for intensity inhomogeneity in presence of high noise,” IEEE Transactions on Image Processing, vol. 27, no. 99, 2018.
- X. Cui, T. E. Potok, and P. Palathingal, “Document clustering using particle swarm optimization,” in Proceedings 2005 IEEE Swarm Intelligence Symposium, Pasadena, CA, USA, June 2005.
- R. Janani and S. Vijayarani, “Text document clustering using spectral clustering algorithm with particle swarm optimization,” Expert Systems with Applications, vol. 134, pp. 192–200, 2019.
- H. Sun, J. Huang, J. Han, H. Deng, P. Zhao, and B. Feng, “Density-based network clustering via structure-connected tree division or agglomeration,” in Proceedings of the IEEE International Conference on Data Mining, Sydney, Australia, December 2010.
- P. G. Sun, L. Gao, and S. Shan Han, “Identification of overlapping and non-overlapping community structure by fuzzy clustering in complex networks,” Information Sciences, vol. 181, no. 6, pp. 1060–1071, 2011.
- Y. Xu, Z. Zhuang, W. Li, and X. Zhou, “Effective community division based on improved spectral clustering,” Neurocomputing, vol. 279, pp. 54–62, 2018.
- W. E. Donath and A. J. Hoffman, “Lower bounds for the partitioning of graphs,” IBM Journal of Research and Development, vol. 17, no. 5, pp. 420–425, 1973.
- M. Fiedler, “Algebraic connectivity of graphs,” Czechoslovak Mathematical Journal, vol. 23, no. 23, pp. 298–305, 1973.
- J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 22, no. 8, pp. 888–905, 2000.
- C. Ding, X. He, H. Zha et al., Spectral Min-Max Cut for Graph Partitioning and Data Clustering, Lawrence Berkeley National Lab, Berkeley, CA, USA, 2001.
- A. Y. Ng, M. I. Jordan, and Y. Weiss, “On spectral clustering: analysis and an algorithm,” in Proceedings of the International Conference on Neural Information Processing Systems: Natural and Synthetic, pp. 849–856, MIT Press, Vancouver, Canada, December 2001.
- L. Zelnik Manor and P. Perona, “Self-tuning spectral clustering,” Advances in Neural Information Processing Systems, vol. 17, pp. 1601–1608, 2005.
- X. Zhang, J. Li, and H. Yu, “Local density adaptive similarity measurement for spectral clustering,” Pattern Recognition Letters, vol. 32, no. 2, pp. 352–358, 2011.
- Y. Nataliani and M.-S. Yang, “Powered Gaussian kernel spectral clustering,” Neural Computing and Applications, vol. 31, no. S1, pp. 557–572, 2019.
- X. Jiang and S. Li, “BAS: beetle antennae search algorithm for optimization problems,” International Journal of Robotics and Control, vol. 1, no. 1, pp. 1–5, 2018.
- T. Wang and Q. Liu, “The assessment of storm surge disaster loss based on BAS-BP model,” Marine Environmental Science, vol. 37, no. 170, pp. 140–146, 2018.
- T. Chen, H. Yin, H. Jiang et al., “Particle swarm optimization algorithm based on bee antenna search for solving portfolio problem,” Computer Systems & Applications, vol. 28, no. 2, pp. 171–176, 2019.
- J. Wang and H. Chen, “Bee swarm antenna search algorithm for optimization problems,” International Journal of Robotics and Control, vol. 1, no. 1, p. 1, 2018.
- L. Wang, S. Ding, and H. Jia, “An improvement of spectral clustering via message passing and density sensitive similarity,” IEEE Access, vol. 7, pp. 101054–101062, 2019.
- X.-Y. Li and L.-J. Guo, “Constructing affinity matrix in spectral clustering based on neighbor propagation,” Neurocomputing, vol. 97, pp. 125–130, 2012.
- R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Süsstrunk, “SLIC superpixels compared to state-of-the-art superpixel methods,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 34, no. 11, pp. 2274–2282, 2012.
Copyright © 2020 Zhe Zhang et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.