Integrating the Supervised Information into Unsupervised Learning

Abstract

This paper presents an assembling unsupervised learning framework that adopts the information coming from the supervised learning process and gives the corresponding implementation algorithm. The algorithm consists of two phases: extracting and clustering data representatives (DRs) firstly to obtain labeled training data and then classifying non-DRs based on labeled DRs. The implementation algorithm is called SDSN since it employs the tuning-scaled Support vector domain description to collect DRs, uses spectrum-based method to cluster DRs, and adopts the nearest neighbor classifier to label non-DRs. The validation of the clustering procedure of the first-phase is analyzed theoretically. A new metric is defined data dependently in the second phase to allow the nearest neighbor classifier to work with the informed information. A fast training approach for DRs’ extraction is provided to bring more efficiency. Experimental results on synthetic and real datasets verify that the proposed idea is of correctness and performance and SDSN exhibits higher popularity in practice over the traditional pure clustering procedure.

1. Introduction

Among diverse issues of data mining, clustering is a fundamental one owing to its ability to provide the data group information before fulfilling other further mining tasks. Clustering pursues data partition that holds maximum cohesion and minimum coupling [1]. It is an intensively researched field, and a number of branches of clustering methods appeared, such as partition-based clustering, hierarchical clustering, density-based clustering, grid-based clustering, boundary-detecting clustering, and model-based clustering [1, 2]. These existing approaches are purely unsupervised learning procedures, which usually draw clustering models from the whole unlabeled dataset with statistics or geometric mechanisms. In other words, the data are probed only once to learn cluster structures. Based on that conventional clustering way, we try to learn discriminative information from data twice; with intention to obtain more accurate clusters. In this paper, we summarize that idea as an assembling clustering idea that consists of two phases. The first phase fulfills the first exploration on the whole data by extracting data representatives (DRs). The rough group information is revealed by DRs. Then, in the second phase, the discriminative information is explored through clustering DRs. We train a classifier based on the labeled DRs, and the discriminative information learned in the second phase is carried by the classifier. If viewing the label information obtained in the second phase as the supervised knowledge, the previous idea actually conducts a procedure that integrates the supervised knowledge into the unsupervised learning process.

Our main idea is implemented by a concrete algorithm, SDSN, which involves support vector technique [3], Spectrum analysis [4–8], and the nearest neighbor classifier. The extraction of DRs of SDSN is achieved by the support vector domain description (SVDD) that is equipped with a tuned scale parameter of Gaussian Kernel. DRs clustering task is fulfilled by a spectrum-analysis-based method. Nearest neighbor classifier (NN) accomplishes label assignment under the guidance of a new metric.

The contributions of SDSN are as follows. (1) The scale parameter of Kernel function used by SVDD is self-tuned, which helps SVDD to generate the semisupport vectors that can act as DRs. (2) In feature space spanned by Gaussian Kernel function, geometric properties of the semisupport vectors and the hyperspheres are explored to guarantee the validation of the first-phase clustering method. (3) The definition of new metric is derived from the cluster structures of DRs, so it makes NN classifier ready to address various-shaped datasets. (4) A fast training approach of SDSN is given to speed up the extraction process of DRs. The fast training approach is based on Schrödinger equation of quantum mechanism [9] to yield an effective learning list of data batches that are friendly to adjusting model incrementally.

Experiments are conducted to compare SDSN with some state-of-the-art clustering methods, say the purely unsupervised clustering methods. Empirical results show that SDSN is of the better or competitive behaviors than its peers. That verifies the validation and the performance of our two-phase clustering idea. Besides, seen in another opinion, SDSN can be thought of as a type of support vector clustering algorithm. So, experiments are conducted to compare SDSN with the popular clustering methods based on support vector technique. Empirical evidence indicates the improvement of SDSN over its counterparts in efficiency and performance.

In the following description, Section 2 illustrates the two-phase clustering idea. Sections 3 and 4 give the details of SDSN’s first phase, that is, extracting DRs and clustering them. Therein the geometric properties of hyper spheres in the feature space spanned by Gaussian Kernel are proposed and proofed to guarantee the validation of the clustering method of the first phase. Section 5 describes the label assignment method of the second phase, where a cluster-structure-dependent metric is defined to facilitate the classifier’s job. The fast training approach is proposed in Section 6. The experimental analysis is given in Section 7. Section 8 is a conclusion.

2. Two-Step Clustering Idea

The essential of two-step clustering idea is to learn discriminative information from data twice by integrating supervised information into the unsupervised learning process. In details, it discovers DRs, learns the label information from DRs, and trains the classifier based on labelled DRs. Then the remaining unlabelled data are classified by the classifier.

Compared with purely unsupervised clustering process that detects clusters from all unlabelled data only once, the proposed idea learns discriminative information of clusters twice. The first learning is during the process of DRs collection. Cluster distribution information is investigated through finding the DRs that are located on the boundaries of dense regions. Secondly, when DRs are clustered, grouping information is probed once again, and this information is carried by the classifier trained on the labelled DRs. That, in turn, benefits the resulted classifier to hold the strong discriminative ability to label other data.

3. Extracting DRs of the First Phase

In the first phase, DRs that hold the ability to describe the dataset sketch are collected. Among existing data reduction approaches, there are two main types. The first type specifies DRs through random sampling; the second type finds DRs based on probability distribution information. Clearly the former incurs randomness and consequently leads to the unexpected results. The latter often produces unpleasant results since the exact probability distribution of real dataset is usually unknown.

Focused on previous difficulties, we exploit SVDD to produce support vectors, which serve as DRs. The reason of choosing SVDD to search DRs is that it is of boundary-detecting mechanism, which fosters SVDD the ability to handle arbitrary-shaped clusters. Furthermore the employment of Kernel function of SVDD enhances its generalization ability. However, the support vectors produced by SVDD only describe the cluster boundaries, without providing the information of inner regions of clusters. Therefore, we propose the tuning-scaled SVDD to yield the support vectors that can describe the dataset sketch. Besides, we give a fast training method for tuning-scaled SVDD to facilitate the efficiency. Now classical SVDD is introduced in brief.

3.1. SVDD

Given the 𝑛-dimensional dataset 𝑋, 𝑋={𝑥₁,…,𝑥_𝑁}. SVDD process is expressed by the following optimization problem: min_𝑅,𝜉𝑅²+𝐶_𝑖𝜉_𝑖,s.t.Φ𝑥_𝑖−𝑎²≤𝑅²+𝜉_𝑖,𝜉_𝑖≥0. (1)Φ is the nonlinear map from the input space to the feature space, 𝜉_𝑖 is the slack variable, 𝑎 is the center of hypersphere, 𝑅 is the radius of the hyper sphere, and 𝐶 is the penalty parameter to tradeoff 𝑅 and 𝜉_𝑖. Transfer it to the Lagrange function, then to the Wolfe dual. Introduce Kernel function 𝐾(𝑥_𝑖,𝑥_𝑗)=⟨Φ(𝑥_𝑖),Φ(𝑥_𝑗)⟩, and we have min_{𝛽𝑖,𝑗}𝛽_𝑖𝛽_𝑗𝐾𝑥_𝑖,𝑥_𝑗−_𝑖𝛽_𝑖𝐾𝑥_𝑖,𝑥_𝑖,s.t.0≤𝛽_𝑖≤𝐶,_𝑖𝛽_𝑖=1. (2)

Points with 𝛽_𝑖=0 are inner-boundary points. Points with 0≤𝛽_𝑖≤𝐶 are SVs, and they actually formulate cluster contours. 𝐾 is often set as the Gaussian Kernel function 𝐾(𝑥,𝑦)=exp(−𝑞‖𝑥−𝑦‖²), where 𝑞 is the scale parameter.

3.2. Tuning-Scaled SVDD

The original SVs are only located on cluster boundaries. However, in this paper, for SVDD, we tune the scale parameter of Gaussian Kernel data adaptively, which enhances the difference among data points, and consequently more SVs are produced to describe both cluster contours and contours of small inner-cluster regions with high density. So, we name these SVs produced by tuning-scaled SVDD as the semisupport vectors (semi-SVs). Obviously, these semi-SVs develop a sketch of dataset, and they are qualified to work as DRs.

In details, for 𝑥, define its scale factor as 𝜎_𝑥=‖𝑥−𝑥_𝑟‖. 𝑥_𝑟 is the 𝑟th furthest point to 𝑥. 𝜎_𝑥 reflects the local distribution density information of 𝑥’s neighborhood. If ‖𝑥−𝑥_𝑟‖≤‖𝑦−𝑦_𝑟‖, it means that the neighborhood of 𝑥 is denser than that of 𝑦. To measure Kernel affinity between 𝑥 and 𝑦, their scale factors are combined in the way that 𝑞_𝑥𝑦=1/𝜎_𝑥𝜎_𝑦, which will serve as the scale parameter. Then it results in the modified Gaussian Kernel 𝐾𝑥,𝑦=exp−𝑞_𝑥𝑦𝑥−𝑦²=exp−𝑥−𝑦²𝜎_𝑥⋅𝜎_𝑦=exp−𝑥−𝑦²𝑥−𝑥_𝑟⋅𝑦−𝑦_𝑟. (3) Therein, 𝑟 is specified as the max gap in the list of distances from 𝑥 to other points. Say 𝑟 is the index of the point with the max gap with 𝑥. Steps of the specification are as follows: (1) compute gap(𝑖)=max_𝑗{[‖𝑥_𝑖−𝑥_𝑗‖−‖𝑥_𝑖−𝑥_𝑗−1‖]/‖𝑥_𝑖−𝑥_𝑗‖}, where the list {‖𝑥_𝑖−𝑥_𝑗‖} (𝑗=1⋯𝑁) is sorted in the ascending order; (2) consider the average of gap(𝑖) as 𝑟 value: 𝑟=ave{gap(𝑖)}.

Two figures in Figure 1 show the SVs and the semi-SVs produced by SVDD and tuning-scaled SVDD. Obviously, SVs produced by classical SVDD only describe cluster boundaries. But semi-SVs produced by tuning-scaled SVDD are located on both cluster boundaries and important inner-cluster positions where sharp changes of density happen. Therefore, these semi-SVs give a good sketch description of dataset and can be regarded as qualified DRs.

(a)

(b)

Figure 1 

SVDD with 𝑞=1.86 (a) and tuning-scaled SVDD (b).

4. Clustering DRs of the First Phase

4.1. Clustering Method

In the first phase, after extracting DRs, we cluster DRs with a spectrum-analysis-based method. The clustering method has the following steps. (1) Compute affinity matrix 𝐾 of DRs according to Gaussian Kernel and denote the entries of 𝐾 as 𝐾_𝑖𝑗: 𝐾_𝑖𝑗=𝐾(𝑥_𝑖,𝑥_𝑗). (2) Normalize 𝐾 into 𝐾^′: 𝐾^′=Λ^−1/2𝐾Λ^−1/2, where Λ=diag(Λ_𝑖)=diag(∑_𝑗𝐾_𝑖𝑗). (3) Take top 𝑔 eigenvectors as columns to form spectrum matrix. (4) Perform 𝐾-means on rows of spectrum matrix, with the cluster number being initialized as 𝑔. (5) Label 𝑥_𝑖 as the 𝑖th row’s cluster membership. Therein, 𝑔 is specified by the number of eigenvalues that are larger than 1 [10]. The previous clustering idea is rooted from spectrum analysis [4]. The validation of this algorithm is supported by the geometric properties of Gaussian Kernel feature space, which is discussed next.

4.2. Geometric Properties of Hyperspheres in Gaussian Kernel Feature Space

In feature space, after the tuning-scaled SVDD is conducted, data are mapped into the hyper sphere. Suppose this sphere being 𝑆, with the center being 𝑎. Since 𝐾(𝑥,𝑥)=⟨Φ(𝑥),Φ(𝑥)⟩=exp(−𝑞‖𝑥−𝑥‖²)=1, it means that all data are located on the surface of the unit ball of the feature space. Denote the unit ball as 𝐵, with the center as the original 𝑂 of the feature space. Then data actually spread on the cap-like surface intersected by 𝑆 and 𝐵. For semi-SVs, they are located on the surface of 𝑆 and 𝐵 simultaneously; therefore, they appear on the rim of the cap. Denote the center of the cap as 𝑎^′. This section presents that the geometric property of semi-SVs and two properties of hyper spheres in the feature space. Before giving the theoretical proof, some existing statements of Gaussian Kernel feature space [11] are mentioned first.

Denote 𝑉 as the set of SVs. In [11], for any 𝑣_𝑖,𝑣_𝑗∈𝑉, in feature space, they have the same angle with 𝑂𝑎^′; that is ∠(Φ(𝑣_𝑖)𝑂𝑎^′)=∠(Φ(𝑣_𝑗)𝑂𝑎^′). For any 𝑣∈𝑉, it generates a cone with Φ(𝑣) as the vertex, 𝑂Φ(𝑣) vector as axis, and ∠(Φ(𝑣)𝑂𝑎^′) as the basic angle. That cone corresponds to a small sphere with 𝑣 as the center and ‖𝑣−Φ⁻¹(𝑎^′)‖ as the radius. Whether a point within 𝑣’s cone decides whether it has the same label with 𝑣. Those statements are summarized as the following lemma. See literature [11] to find more details.

Lemma 1. For any 𝑥∈𝑋 and 𝑣∈𝑉, let 𝜃=∠(Φ(𝑣)𝑂𝑎^′); there is: ∠(Φ(𝑣)𝑂Φ(𝑥))<𝜃⇔‖𝑣−𝑥‖<‖𝑣−Φ⁻¹(𝑎^′)‖⇔Φ(𝑥) is within Φ(𝑣)’s cone ⇔𝑥 is within 𝑣’s small sphere ⇔𝑥 and 𝑣 have the same cluster label.

Although we name those points with nonzero support values as semi-SVs, they are also of the identical properties as SVs. Therefore, based on the previous lemma, we have the following geometric property of semi-SVs in feature space [12].

Corollary 2. In feature space of Gaussian Kernel, semi-SVs are collected in terms of clusters on the intersection hyperline of 𝑆 and 𝐵.

In this study, we give two geometric properties about the hyper spheres produced by SVDD. Those two properties are expected to extend the proposed algorithm to the multi-classification issue in the future work.

Proposition 3. If SVDD procedure is conducted on each class with the same Kernel scale, it ensures that multiple hyper spheres exist in the same feature space.

Proof. For kernel-based algorithms, the kernel function defines a non-linear map Φ by 𝐾(𝑥,𝑦)=⟨Φ(𝑥),Φ(𝑦)⟩. For Gaussian kernels, the scale is the only parameter to determine in the kernel functions and consequently the only factor to determine the feature space. As long as the scale remains the same, the spanned feature space is also the same. With the same scale, SVDD procedures conducted on each class will yield multiple hyper spheres in the same feature space.

Proposition 4. The maps of the same data created by different kernel-based methods are the same, provided that the Gaussian kernels have the same scale parameter.

Proof. According to Proposition 3, the kernel scale is the only factor to determine the feature space where kernel works. As long as the kernel scale keeps the same, the nonlinear map Φ does not change; therefore, the image of the given data 𝑥 mapped by Φ does not change. Of course, different kernel-based methods have different optimization objectives, so their optimization results are different. Take SVM and SVDD as examples. For a given 𝑥, with the same Gaussian kernel scale, SVM and SVDD will employ the same image, Φ(𝑥), in their computation. As they have different optimization objectives, different solutions will be obtained. SVDD tries to find data located around the boundaries and SVM lets those data located around the margin area be equipped with non-zero support values to formulate the discriminative plane.

Based on the previous two propositions, the next job is to probe the further geometric properties of SVs and try to investigate the relationship between geometric positions of SVs and the hyper spheres, with intention to develop the wise discrimination approach. That formulates the future work direction to address multiclassification issue by conducting SVDD procedures on each class and building the new labeling approach.

4.3. Validation of the Clustering Method

The validation of the clustering method is interpreted as follows.

Firstly, SVs are grouped in terms of clusters on the intersection circle, which means that their distribution in feature space has regular directions. Then angle information is a good criterion to detect their clusters. Angle distance is used in the way of Cosine value. Because SVs are on the surface of the unit ball, Cosine value is used in the way of inner product: ⟨Φ(𝑥_𝑖),Φ(𝑥_𝑗)⟩. In feature space, 𝐾(𝑥_𝑖,𝑥_𝑗)=⟨Φ(𝑥_𝑖),Φ(𝑥_𝑗)⟩, so the first-phase method employs Kernel matrix as the affinity matrix.

Secondly, according to spectrum analysis idea, if we eigendecompose the affinity matrix of one dataset, the inherent distribution information of the dataset can be revealed. This type of distribution information is described by eigenvectors because these vectors describe the distribution directions. For our algorithm, the eigendecomposition on the kernel matrix yields the eigenvectors that describe the distribution directions based on angle information. The subsequent clustering operations are well expected to produce correct clusters.

5. Second Phase: Label Assignment

With labeled DRs in hand, we label data using NN classifier. Like any a distance-based classifier, NN is faced with two scenarios of clusters: the first is convex-shaped clusters, where spatial information can reflect cluster structures; the other is nonconvex clusters. In the later scenario, the Euclidean metric often fails and the cluster structures make importance in clusters detection. That can be illustrated in Figure 2.

(a)

(b)

(c)

Figure 2 

Dataset1 (a), the shortest path of Euclidean (b), and the shortest paths of ‖⋅‖∗ (c).

In Figure 2(a), cluster2 and cluster3 are convex shaped, so Euclidean distance meaning is consistent with cluster membership. However, the cluster1 is string shaped, and Euclidean metric cannot provide accurate clusters. So in Figure 2(b), for 𝑥, 𝑦 of cluster1, for 𝑧 of the Cluster2, according to cluster structure, ideal members of the same cluster should have less distance than that of members of different clusters; that is, there should be ‖𝑥−𝑦‖<‖𝑥−𝑧‖. But, with Euclidean metric, there is ‖𝑥−𝑦‖=0.5356, ‖𝑥−𝑧‖=0.2239; and there is ‖𝑥−𝑦‖>‖𝑥−𝑧‖. To solve this problem, cluster shapes should be taken into consideration when the metric is defined. The next section touches that topic.

5.1. New Metric Definition

The key of new metric is to investigate multireachable paths between two points including the path of straight line segment that directly connects two points, and the indirect paths that consist of multiline segments. The final distance is determined by the shortest path with minima path length. In computing length of indirect paths, neighboring information is exploited, and the neighboring information spreads along paths so that the message of global cluster structures is accumulated gradually and integrated into metric formulation.

Given dataset 𝑋={𝑥₁,𝑥₂,…𝑥_𝑁}, we view it as a complete undirected graph, with graph node as data point and edge weight as distance of a pair of points. Inspired by [13], we firstly define edge weight that directly connects 𝑥 and 𝑦 as follows: 𝑊𝑥,𝑦=𝜂_𝑥𝑦exp𝑥−𝑦²−1. (4)

Here, Euclidean distance is rescaled by the exponential mechanism with intention to reduce the distance between similar points while enlarging distance between distinct points. That actually strengthens cluster boundaries. 𝜂_𝑥𝑦 is the relative density ratio, defined as 𝜂_𝑥𝑦(𝑥)𝑦=2−minAD,ADmaxAD(𝑥),AD𝑦, (5) with AD(𝑥)=ave_{𝑢∈Nei(𝑥)}{‖𝑥−𝑢‖}.

Nei(𝑥) represents 𝑥’s neighborhood. AD(𝑥) is the average distance of Nei(𝑥), which reveals the density information of Nei(𝑥). If AD(𝑥) is close to AD(𝑦), then 𝜂_𝑥𝑦 tends to 1, and 𝑊(𝑥,𝑦) tends to exp(‖𝑥−𝑦‖²)−1. If AD(𝑥) is sharply different from AD(𝑦), then 𝜂_𝑥𝑦 tends to 2, and 𝑊(𝑥,𝑦) tends to 2[exp(‖𝑥−𝑦‖²)−1]. The employment of 𝜂_𝑥𝑦 avoids the distorted distance value caused by the unbalance of local distributions between two points. Therein, Nei(𝑥) consists of some top nearest points to 𝑥. Nei(𝑥) size is set as 5% of the whole dataset size.

Then the path length is defined. For a path 𝑝 outgoing from 𝑥 and ingoing to 𝑦, 𝑝={𝑥=𝑧₁,𝑧₂,…𝑧_𝑚=𝑦}, its length is defined as PL𝑝=^𝑚−1_𝑖=1𝑊𝑧_𝑖,𝑧_𝑖+1. (6) Then for 𝑥, 𝑦, their cluster-structure-dependent distance is the length of the shortest path. Consider 𝑥−𝑦_∗=min_{𝑝∈path𝑥,𝑦}PL𝑝. (7)

Therein, path[𝑥,𝑦] is the set of all paths connecting two points. The new metric searches the shortest path among multipaths, so it breaks the limit of Euclidean metric, which takes the straight path that connects two points directly as the shortest path. Since the distance between similar points is small, it is absolutely possible that an indirect path that crosses over a series similar points is shorter than a direct path that crosses distinct points. Still, look at the example of Figure 2(b). With the new metric, there is ‖𝑥−𝑧‖_∗=0.0135, ‖𝑥−𝑦‖_∗=0.0126; consequently, ‖𝑥−𝑧‖_∗>‖𝑥−𝑦‖_∗. Their shortest paths are illustrated in Figure 2(c). Those distance values are consistent with cluster membership. It shows that ‖⋅‖_∗ does address the influence brought by nonconvex distribution of clusters.

5.2. Shortest Distance Computation

We now give the computation steps of the new metric. Different from classical Dijstra algorithm [13], in this paper, we care for the paths between pairs of points and compute the shortest distance values concerned with these paths simultaneously; that is, for the point pair (𝑠,𝑡), the distance of paths outgoing from 𝑠 is computed, and the distance of paths ingoing at 𝑡 is computed at the same time. Denote the shortest distance over the whole dataset as matrix 𝑆ℎ𝑜𝑟𝑡𝐷, then the previous computation fills the 𝑠th row and the 𝑡th column of 𝑆ℎ𝑜𝑟𝑡𝐷. When it comes to the undirected graph, the entries of the 𝑠th column and the 𝑡th row get values simultaneously.

For the 𝑁-sized dataset 𝑋, we compute the distance of pairs of points in the following order: (𝑥₁,𝑥₂),(𝑥₃,𝑥₄),(𝑥₅,𝑥₆),…,(𝑥_𝑁−1,𝑥_𝑁). After all distances of paths between the prvious pairs of points are obtained, the whole 𝑆ℎ𝑜𝑟𝑡𝐷 can be specified. That idea forms an iterative procedure of the shortest distance computation as follows.

ShortDistance Computation(1)for (𝑖=1;𝑖<𝑁;𝑖=𝑖+2); (2){𝑃𝑎𝑖𝑟𝑆ℎ𝑜𝑟𝑡𝐷(𝑖,𝑖+1);}.

Therein, procedure 𝑃𝑎𝑖𝑟𝑆ℎ𝑜𝑟𝑡𝐷 computes distance values of point pair (𝑥_𝑖,𝑥_𝑖+1), (𝑖=1⋯𝑁−1). Its details are listed in Algorithm 1.

𝑃𝑎𝑖𝑟𝑆ℎ𝑜𝑟𝑡𝐷(𝑠,𝑡) (1)  𝐷𝑠[1∶𝑁]=∞; 𝐷𝑡[1∶𝑁]=−∞; 𝑠𝐿𝑒𝑓𝑡=𝑋; 𝑡𝐿𝑒𝑓𝑡=𝑋; (2)  while (𝑁𝑜𝑡𝐸𝑚𝑝𝑡𝑦(𝑠𝐿𝑒𝑓𝑡)) (3)   {𝑢=min𝑗∈𝑠𝐿𝑒𝑓𝑡{𝐷𝑠(𝑗)}; 𝑠𝐿𝑒𝑓𝑡=𝑠𝐿𝑒𝑓𝑡−{𝑢}; (4)     for (𝑖=1; 𝑖<=𝑁; 𝑖++) (5)     if (𝐷𝑠(𝑖)>𝐷𝑠(𝑢)+𝑊(𝑢,𝑖)) (6)       𝐷𝑠(𝑖)=𝐷𝑠(𝑢)+𝑊(𝑢,𝑖);} (7)  while (𝑁𝑜𝑡𝐸𝑚𝑝𝑡𝑦(𝑡𝐿𝑒𝑓𝑡)) (8)   {𝑢=max𝑗∈𝑡𝐿𝑒𝑓𝑡{𝐷𝑡(𝑗)}; 𝑡𝐿𝑒𝑓𝑡=𝑡𝐿𝑒𝑓𝑡−{𝑢}; (9)     for (𝑖=1; 𝑖≤𝑁;𝑖++) (10)    if (𝐷𝑡(𝑖)<𝐷𝑡(𝑢)−𝑊(𝑖,𝑢)) (11)      𝐷𝑡(𝑖)=𝐷𝑡(𝑢)−𝑊(𝑖,𝑢);} (12) Fill 𝐷𝑠 into 𝑆ℎ𝑜𝑟𝑡𝐷's 𝑠𝑡ℎ row and 𝑠𝑡ℎ column; (13) Fill 𝐷𝑡 into 𝑆ℎ𝑜𝑟𝑡𝐷's 𝑡𝑡ℎ row and 𝑡𝑡ℎ column.

Algorithm 1 

In 𝑃𝑎𝑖𝑟𝑆ℎ𝑜𝑟𝑡𝐷, 𝐷𝑠 and 𝐷𝑡 are 𝑁-sized arraye, and they keep track of the latest shortest distance with respect to 𝑠 and 𝑡, respectively. They are updated all the time. 𝑠𝐿𝑒𝑓𝑡 and 𝑡𝐿𝑒𝑓𝑡 preserve the points to be explored. Note that 𝐷𝑡 uses minus value to show that it is the length of ingoing path that has the opposite direction with the outgoing path. The employment of minus value in 𝐷𝑡 makes 𝑃𝑎𝑖𝑟𝑆ℎ𝑜𝑟𝑡𝐷 procedure able to be applied in the directed graph.

5.3. Test New Metric

To check the performance of ‖⋅‖^∗, it is compared with Euclidean metric and ‖⋅‖₀ in synthetic datasets, as shown in Figures 3, 4, 5, and 6. ‖⋅‖₀ is defined as follows without using relative density ratio 𝜂_𝑥𝑦: 𝑊₀𝑥,𝑦=exp𝑥−𝑦²−1. (8)

(a)

(b)

(c)

Figure 3 

Dataset2 (a), clusters of Euclidean (b), and clusters of ‖⋅‖0, ‖⋅‖∗ (c).

(a)

(b)

(c)

(d)

Figure 4 

Dataset3 (a), clusters of Euclidean (b), clusters produced by ‖⋅‖0 (c), and clusters produced by ‖⋅‖∗ (d).

(a)

(b)

Figure 5 

The shortest path produced by Euclidean (a) and the shortest path produced by ‖⋅‖0 and ‖⋅‖∗ (b).

(a)

(b)

(c)

Figure 6 

The shortest path produced by Euclidean (a), the shortest path produced by ‖⋅‖0 (b), and the shortest path produced by ‖⋅‖∗ (c).

Three metrics are introduced into 𝐾-means method to observe clustering results. Clearly, Euclidean metric fails in dataset2 and dataset3 due to its rigid dependence on the sole path. ‖⋅‖₀ works well in dataset2 but fails in dataset3. The outperformance of ‖⋅‖_∗ over ‖⋅‖₀ shows the importance of 𝜂_𝑥𝑦. That can be explained clearer by the shortest paths produced by three metrics, as shown in Figure 5(b) and Figure 6. Obviously, in dataset3, the shortest path yielded by ‖⋅‖₀ is attracted by the dense cluster, while the path yielded by ‖⋅‖_∗ can cross the same cluster region.

6. Fast Training Approach

The DRs’ extraction process of SDSN is essentially a quadratic optimization problem, which consumes huge cost. So, we give a fast training approach based on incremental learning to facilitate the efficiency.

Like the common incremental method, our approach sets a working set and divides remainder data into batches. Those batches will be enriched into the working set incrementally. The novel in our approach is that the batches are sorted in an order that benefits the incremental development of the problem solution. For SVDD optimization problem, the solution is the support values {𝛽_𝑖}. Therefore, we select those data that could do significance on solution development to form the initial working set and specify the batches according to their importance to the model formulation. Those data batches will be used to update the working set in turn. The reason to consider the importance of data lies in the fact that if the important data are processed firstly, they can assist to build a promising solution script that readily accepts the following gentle adjusting. Yet, if important data are inputted late, sharp adjusting operations might happen when the ending batches are enriched into the working set. That could bring undesired affect to the formulation of final solution. Based on this thinking, we give the method to generate the sorted list of data batches based on Schrödinger equation [9], whose brief introduction is given firstly.

6.1. Schrödinger Equation

Schrödinger equation (SE) [9] comes from quantum mechanics of physics field. It is written as (𝑥)=−𝜎𝐻⋅Ψ²2∇²+𝑃(𝑥)Ψ(𝑥)=𝑒⋅Ψ(𝑥). (9)

In quantum mechanics, SE describes the law of energy conservation of a particle, where 𝑒 is the energy, 𝑃(𝑥) is the Schrödinger potential, and ∇ is the Laplacian. Ψ(𝑥) corresponds to the state of a quantum system, and Ψ(𝑥) can be explained as the wave function of one particle. Conventionally, given 𝑃(𝑥), the equation is solved to find solution Ψ(𝑥), so particle orbits are obtained.

To apply SE in machine learning, Ψ(𝑥) is regarded as probability distribution of dataset, and then the maxima of Ψ(𝑥) associate with cluster centers. Instead of looking for Ψ(𝑥)’s maxima, potential 𝑃(𝑥)’s minima are probed because under physical meaning the latter’s minima correspond to the former’s maxima. The advantage of investigating 𝑃(𝑥) rather than Ψ(𝑥) lies in the robustness of 𝑃(𝑥) towards the variation of parameter 𝜎. As long as 𝜎 is within a moderate range, 𝑃(𝑥)’s minima can reflect cluster centers correctly. That forms an inverse fashion of SE solving process. For a Ψ(𝑥), 𝑃(𝑥) is solved as 𝑃(𝑥)𝜎=𝑒+²(𝑥)∇2Ψ²Ψ(𝑥). (10) Usually, Ψ(𝑥) is simulated by the empirical distribution Ψ(𝑥)=_𝑖exp−𝑥−𝑥_𝑖²2𝜎². (11) Then (10) is rewritten as 𝑃(𝑥)𝑛=𝑒−2+12𝜎²⋅∑_𝑖𝑥−𝑥_𝑖²exp−𝑥−𝑥_𝑖²/2𝜎²∑_𝑖exp−𝑥−𝑥_𝑖²/2𝜎². (12) Let us furthermore require min𝑃(𝑥)=0. That sets the value of 𝑒 as 𝜎𝑒=−min²(𝑥)∇2Ψ²Ψ(𝑥). (13) If we restrict the computation of 𝑃(𝑥) within the finite dataset, there is 𝑃𝑥_𝑖𝑛=𝑒−2+12𝜎²⋅∑_𝑖𝑥_𝑖−𝑥_𝑗²exp−𝑥_𝑖−𝑥_𝑗²/2𝜎²∑_𝑖exp−𝑥_𝑖−𝑥_𝑗²/2𝜎². (14)

From geometry meaning, 𝑃(𝑥)’s minima tell cluster centers, so 𝑃(𝑥)’s maximum indicates the boundary information of clusters. That inspires us to use Schrödinger equation to define reduction strategy.

6.2. Fast Training Approach

It has been known that 𝑃(𝑥_𝑖) tells the possibility of 𝑥_𝑖 how much it gets close to cluster boundaries. The bigger 𝑃(𝑥_𝑖) is, the more possibly 𝑥_𝑖 gets close to boundaries, and the more possible it is an SV. This means that 𝑥_𝑖 makes importance in formulating solution {𝛽_𝑖} and 𝑥_𝑖 should be added into working set early. According to 𝑃(𝑥_𝑖) value, all data are sorted as {𝑥₍₁₎,…,𝑥_(𝑁)}, subject to {𝑃(𝑥₍₁₎)≥𝑃(𝑥₍₂₎)≥⋯≥𝑃(𝑥_(𝑁))}. Specify the size of working set as 𝑀 and the size of batch as 𝐵, with 𝐵<𝑀. 𝑀 and 𝐵 can be determined by storage requirement. Dataset 𝑋 is divided into 1 working set and 𝑇 batches, 𝑈₀,𝑈₁,…,𝑈_𝑇, and there is 𝑈₀=𝑥₍₁₎,…,𝑥_(𝑀),𝑈₁=𝑥_(𝑀+1),…,𝑥_(𝑀+𝐵)𝑈,…,_𝑇=𝑥_{(𝑀+(𝑇−1)𝐵)},…,𝑥_{(𝑀+𝑇𝐵)}. (15)

In our fast training approach, 𝑈₀ is the initial working set. The initial solution {𝛽_𝑖} is obtained on 𝑈₀. Then sort the data of working set according to their support values {𝛽_𝑖} in the ascending order. Replace the top 𝐵 data of the sorted working set with 𝑈₁, so as to formulate the updated working set. The new solution is computed on the new working set again. Repeat the previous procedure until all data batches are input.

6.3. Test Fast Training Approach

Now check the quality of fast training approach. In experiments, SVDD optimization process is conducted in three ways: (1) nonincremental training, (2) the incremental training, and (3) the proposed fast training. In case (2), dataset is divided into batches randomly. For two incremental approaches, the working set size 𝑀 is 20% of dataset size, and the batch size is the 10% of working set size. Experiments aim to compare (1) versus (2) and (1) versus (3), respectively. The criterion is the number of shared SVs between (1) versus (2) and between (1) versus (3). The number of SVs and the number of shared SVs are recorded in column “no. of SV” and “no. of shared” of Table 1.

Table 1 

Performance comparison of fast training approaches.

Datasets are taken from UCI benchmark machine learning repository. In Table 1, sizes of datasets are indicated in the bracket following the names of datasets. Pay attention to letter dataset. It has 20000 data covering 26 letters from “𝑎” to “𝑧”. Here, 260 random samples are selected from classes “𝑎” and “𝑏”, respectively, to form the 520-sized experimental subset.

From Table 1, it is easy to find that the number of SVs generated by incremental approaches is less than that of nonincremental approach. Between (2) and (3), (3) has shared more SVs with approach (1) than (2). So, the proposed approach holds better fast training ability than the usual method.

Besides, we record the time consumption of three processes in Table 2 to check the speed of our fast training method. According to Table 2, the obvious improvement of the proposed fast training method over the other two processes in efficiency is found.

Table 2 

Time comparison of fast training approaches (seconds).

7. Experiments

7.1. SDSN versus SVC and SVC Variants

Since SDSN is designed based on support vector clustering method, experiments will be done to compare the performance of SDSN with support vector clustering (SVC) as well as SVC variants firstly. To be self-included, here SVC and some variants are introduced in brief. SVC consists of optimization piece and labeling piece. The original labeling procedure is named as complete graph (CG). CG identifies clusters by constructing a complete graph and taking connected components of graph as clusters [3]. The adjacency matrix 𝐴 is 𝐴_𝑖𝑗=1∀𝑦,𝑦∈line𝑥_𝑖,𝑥_𝑗,𝑅𝑦≤𝑅0otherwise. (16)

Therein, line[𝑥_𝑖,𝑥_𝑗] is the line segment between 𝑥_𝑖 and 𝑥_𝑗. 𝑅(𝑦) is the distance between 𝜙(𝑦) and the center of hypersphere in the feature space. 𝑅 is the radius of the hypersphere. Clusters are defined as the connected components of the graph induced by 𝐴. Clearly, randomness encoded in CG degrades clustering accuracy. Moreover, the number of sampling points creates the dilemma between clustering quality and time cost.

Variants of SVC focus on revising its labeling process. support vector graph (SVG) [3] is an example that computes the adjacent matrix and connected components only with respect to SVs. So, it obtains dramatic time reduction but simultaneously sees a decrease in clustering accuracy. Proximity Graph (PG) [14, 15] also computes adjacent matrix among SVs, but it takes some simulation approach to derive connected components. Usually, PG employs Delaunay triangulation (DT) [15] as the simulation approaches. PG consumes less time than SVG and produces poorer result than SVG. Another method, gradient descendent (GD) [16], builds adjacent matrix and connected components based on stable equilibrium points (SEPs). Each SEP represents some data within its neighborhood. Data is labeled the same membership as its representative SEP. All these methods are all encoded with randomness, and they gain the improvement of time cost at the price of the decrease of clustering quality. Recently, cone cluster labeling (CCL) is given in [10] to do label assignment based on geometric observation. CCL overcomes randomness and achieves better performance than other methods.

Besides the previous datasets, news groups dataset is tested. This dataset contains about 20,000 articles divided into 20 news groups. They are named as NG1,NG2,…,NG20. The usual 𝑡𝑓.𝑖𝑑𝑓 weighting schema is exploited to express documents, andsome words that appear very few times are deleted. Normalize each document vector to make them have the unit length. For empirical ease, some classes are sampled randomly to form six experimental sets: (1)   {NG1, NG2, NG8} (180); (2)   {NG2, NG7, NG9, NG14} (220); (3)   {NG1, NG3, NG5, NG6} (260); (4)   {NG17, NG18, NG19} (220); (5)   {NG12, NG13, NG14, NG15} (200). The number in the bracket indicates the number of random samples of each class. Clustering accuracies are recorded in Table 3. Note that PG has no result in news group, because DT used by PG is infeasible in high-dimensional data space. In computing adjacent matrix, CG runs with 3 different sampling numbers: CG1 with sampling 15 data; CG2 with sampling 23 data; CG3 with sampling 30 data. The aim is to observe the influence of sampling number on clustering accuracy. For SVG, PG, and GD, their sampling number is set as 15.

Table 3 

Clustering accuracy comparison (%).

From Table 3, it is clear that for CG family, with sampling number increasing, clustering accuracy goes up, which is natural since higher sampling number can reduce randomness. Of course, this improvement is at the price of the huge increase of time cost. Note that sometimes the difference between three CG methods is subtle. One reason is that sometimes data distribution is bound to make influences on clustering results no matter how many samples are selected. The other reason lies in the randomness coming from random sampling, which always leads to unexpected results. To some extent, CG’s performance is controlled by sampling number. If the sampling number gets to the infinity, CG would ideally present good result. SVG is a rough version of CG1, so its work is not as good as CG1. PG follows SVG due to its approximation operations. There is clear instability in GD’s behaviors since it does well in datasets (1), (3) and (4), while it does poorly in (2), (5), and letter. Contributing factor for that is GD’s susceptibility on data distribution and dimensionality. Only in some certain datasets, GD can generate qualified SEPs to present accurate neighborhood information and consequently to produce good results. Compared with these methods, CCL and SDSN show clear advantage due to their removing of randomness. Basically, CCL achieves the top place by doing the best job in most datasets. However, CCL’s good behavior is at the price of expensive cost that is spent on investigating Kernel scale parameter list. SDSN’s behaviors follow CCL by producing the optimal result in 4 of 9 datasets and doing well in the remainder datasets except Wine dataset. The reason of SDSN’s failure is the fact that Wine data has 178 points but covers 13 dimensions, which makes the neighborhood information uninformed, and thus the tuning strategy cannot play its role. Generally speaking, SDSN has outperformance over the original CG approach, which illustrates the improvement of the proposed clustering idea over the classical unsupervised learning idea. And SDSN has the competitive performance with the popular SVDD variant, which verifies SDSN’s fine clustering ability. Figure 7 illustrates the time consumption of these methods in one running, which shows SDSN’s time efficiency as competitive with CCL.

Figure 7 

One-run time consumptions of methods (second).

7.2. SDSN versus Clustering Methods

Now, compare SDSN with some clustering methods: 𝐾-means [17], Girolami [18], NJW [4], and NI [19]. Girolami is a Kernel-based method that depends on an expectation-maximum process. NJW is a representative of spectrum clustering methods. NI is an agglomerative clustering method. It creates the distance definition according to information entropy and takes this information-based distance as clustering criteria to accumulate subclusters. Clustering accuracies of these methods are listed in Table 4.

Table 4 

Comparison of clustering accuracies of methods (%).

Among five methods, it can be found that NJW is the best clustering tool. It probes spectra of all data, to obtain data inherent distribution directions, which assists much in detecting correct clusters. But NJW consumes huge cost, with 𝑂(𝑁³) time complexity. For Girolami, NI, and 𝐾-means, their performance decreases in turn. NI shows instability in experiments. It is because, as an agglomerative approach, NI is fairly easy to be affected by data input order. Although the information-based metric helps a lot to provide accurate distance information, NI still exhibits unstable behaviors. 𝐾-means is a hard partition process completely dependent on Euclidean metric. When the Euclidean metric does not match data distribution, more errors will happen. So, 𝐾-means does not behave well. SDSN does the best job in Monk3 and follows closely NJW in (1), (2), and (5), which demonstrates SDSN’s fine clustering performance. In the remainder cases, the difference between SDSN and NJW is acceptable. These experiments verify the correctness and validation of the proposed clustering idea. And if we take both the performance and the efficiency into consideration, we will find that SDSN is of higher popularity than its peers.

8. Conclusion

This paper presents a general clustering framework that extracts and clusters DRs firstly and then classifies non-DRs. The idea is implemented by an algorithm SDSN. SDSN consists of a tuning-scaled SVDD optimization that extracts DRs, a spectrum-analysis-based method to cluster DRs, the nearest neighbor classifier that works with an informed metric to classify non-DRs, and the fast training approach that facilitates the computation ease. To illuminate the validation of the spectrum-analysis-based method, the geometric property of Gaussian Kernel feature space is explored and proofed. Experiments demonstrate the performance of the tuning-scaled SVDD, of the new metric, and of the fast training approach. The improved popularity of SDSN over the pure clustering approach is verified too.

Since the supervised information can be utilized in the unsupervised learning process, it is well expected to obtain help from clustering information for the classification mining task. The future work direction is focused on solving multi-classification issue based on the geometric properties of multihyperspheres of the feature space, as mentioned in Section 4.2.

Acknowledgments

This research is partially supported by the National Natural Science Foundation of China under Grants nos. 61105129, 61304174, and 11226146 and the Natural Science Foundation of Jiangsu Province of China under Grant no. BK2011581.