Clustering by Fuzzy Neural Gas and Evaluation of Fuzzy Clusters

Abstract

We consider some modifications of the neural gas algorithm. First, fuzzy assignments as known from fuzzy c-means and neighborhood cooperativeness as known from self-organizing maps and neural gas are combined to obtain a basic Fuzzy Neural Gas. Further, a kernel variant and a simulated annealing approach are derived. Finally, we introduce a fuzzy extension of the ConnIndex to obtain an evaluation measure for clusterings based on fuzzy vector quantization.

1. Introduction

Prototype based vector quantization (VQ) is an approved method to cluster and compress very large data sets. Prototype based implies that the data are represented by a much smaller number of prototypes. Famous methods are c-means [1], self-organizing maps (SOM) [2], and neural gas (NG) [3]. These methods have in common that each data point is uniquely assigned to its closest prototype. Therefore, they are also called crisp vector quantizers. Yet, in practical applications, data are often overlapping making it hard to separate clusters. For this kind of data fuzzy vector quantizing, algorithms have been developed, for example, fuzzy c-means (FCM) [4] and fuzzy SOM (FSOM) [5]. Now, each datapoint can be partially assigned to each prototype. The FSOM is an extension of the FCM taking the neighborhood cooperativeness into account. Yet, as common to SOM, this neighborhood is bound to an external topological structure like a grid. In this paper we combined FCM with NG, thus exploiting the advantages of each: fuzziness from FCM and dynamic neighborhood cooperativeness without structural restrictions from NG. Our new approach is called Fuzzy Neural Gas (FNG).

Beside its basic functionality we also introduce some variations of FNG. First, we propose the kernel fuzzy neural gas (KFNG) where we consider differentiable kernels to adapt the metric. This allows the algorithm to operate in the same structural space as support vector machines (SVM) [6], which are known to deliver respectable results [7]. In [6], it has been shown that this modified optimization space is equivalent and isometric to a reproducing kernel Hilbert or Banach space, which proves to be beneficial for unsupervised VQ, that is also for FNG.

For another variant of FNG we were inspired by simulated annealing (SA), a method which allows temporary deterioration of an optimization process to stabilize its long term behavior. To obtain an SA-like approach, we introduce negative learning and call the new method pulsing Neural Gas (PNG). The idea can also be transferred to FNG resulting in Pulsing Fuzzy Neural Gas (PFNG).

Clustering in general is an ill-posed problem and it is difficult to validate a cluster solution. Specification the validation of very large data sets, where a cluster might be represented by more than one prototype, turns out to be a challenge. There exist a number of validity measures based on separation and compactness, yet most of them presume that each cluster should be represented by exactly one prototype. Taşdemir and Merényi proposed the ConnIndex [8], which is suited to evaluate crisp clusterings, where each cluster contains more than one prototype. This ConnIndex takes the neighborhood structure between the learned prototypes into account to transfer the information of the full data set to the cluster validation process. We propose a modification for fuzzy cluster solutions and use this Fuzzy ConnIndex in the experimental section.

In the experimental section, we use three different data sets, an artificial one and two real world problems, to compare the cluster solutions obtained by FNG with those obtained by FCM. For evaluation purposes the Fuzzy ConnIndex is applied. Further, we demonstrate the performance of Pulsing Neural Gas on a checkerboard data set. This type of problem is highly multimodal and usually the algorithms do not find all clusters.

2. Fuzzy Neural Gas

The Fuzzy Neural Gas algorithm is a vector quantizer suitable for overlapping data resulting in fuzzy cluster solutions. It is a combination of the Neural Gas (NG) algorithm which incorporates neighborhood relations between data points and prototypes and the Fuzzy c-Means (FCM) which provides a way to obtain fuzzy data point assignments. In the following section, the NG and the FCM are presented shortly to reproduce the derivation of the FNG originally published in [9]. Besides providing an understanding for the principle functioning of the FNG, the description of the basic algorithms is also useful in preparation of Section 4, where a fuzzy cluster validation method called Fuzzy ConnIndex (fConn) is presented.

2.1. Neural Gas

The Neural Gas vector quantizer [3] is an approach which utilizes the dynamic neighborhood between the prototypes 𝑊={𝐰_𝑗}^𝑁𝑃_𝑗=1, 𝐰_𝑗∈ℝ^𝑑, to obtain a clustering of data samples 𝐯_𝑖∈ℝ^𝑑, 𝑖=1,…,𝑁_𝑉, from a data set 𝑉. This neighborhood function is based on a winner ranking of the prototypes for each data point. The rank of prototype 𝐰_𝑗 is obtained by 𝑟𝑘_𝑗𝐯_𝑖,𝑊=^𝑁𝑃_𝑙=1Θ𝑑𝐯_𝑖,𝐰_𝑗−𝑑𝐯_𝑖,𝐰_𝑙, (1) with the heaviside function Θ(𝑥)=0, if and only if 𝑥≤0 and 1 else, and a dissimilarity measure 𝑑(𝐯_𝑖,𝐰_𝑗) which determines the distance between data point 𝐯_𝑖 and prototype 𝐰_𝑗. Usually the Euclidean distance is used for 𝑑(𝐯_𝑖,𝐰_𝑗).

The neighborhood ℎ^NG_𝜎(𝐯_𝑖,𝐰_𝑗,𝑊) of a data point 𝐯_𝑖 is specified by ℎ^NG_𝜎𝐯_𝑖,𝐰_𝑗,𝑊=𝑐^NG_𝜎⋅exp−𝑟𝑘_𝑗𝐯_𝑖,𝑊²2𝜎², (2) where the rank 𝑟𝑘_𝑗 of prototype 𝐰_𝑗 is an essential part. For the neighborhood only the prototypes within a certain range 𝜎 according to their rank are considered, giving the closest prototype the highest emphasis. The constant 𝑐^NG_𝜎 is arbitrarily chosen.

The neighborhood can be used to calculate the local costs: 𝑙𝑐^NG𝐯_𝑖,𝐰_𝑗,𝑊=ℎ^NG_𝜎𝐰_𝑗,𝐯_𝑖⋅𝑑𝐯_𝑗,𝐰_𝑗, (3) which resemble the local distortions around prototype 𝑗 weighted by the neighborhood cooperativeness.

The Neural Gas cost function which has to be minimized for optimal clustering directly embeds the local costs: 𝐸_NG=12𝐾(𝜎)^𝑁𝑃_𝑗=1𝑃𝐯_𝑖ℎ^NG_𝜎𝐯_𝑖,𝐰_𝑗,𝑊⋅𝑑(𝐯_𝑖,𝐰_𝑗)_𝑙𝑐NG𝑑𝐯_𝑖. (4) The normalisation constant 𝐾(𝜎) depends on 𝑐^NG_𝜎 and 𝑃(𝐯_𝑖) is the data density.

The minimization of the cost function (4) is performed by stochastic gradient descent with respect to the prototypes. Given a data point 𝐯_𝑖 the prototype update rule yields Δ𝐰_𝑗=−𝜀⋅ℎ_𝜎𝐯_𝑖,𝐰_𝑗⋅𝐯,𝑊𝜕𝑑_𝑖,𝐰_𝑗𝜕𝐰_𝑗, (5) where 𝜀>0 is the learning rate [3].

After convergence of the algorithm the whole data set 𝑉 is approximated by the set 𝑊 of prototypes. The receptive field Ω_𝑗 of each prototype 𝐰_𝑗 is defined as Ω_𝑗=𝐯_𝑖∣𝑑𝐯_𝑖,𝐰_𝑗<𝑑𝐯_𝑖,𝐰_𝑘,∀𝑘. (6)

For crisp clusterings it has been shown in [3] that the NG algorithm results in better cluster solutions than Self-Organizing Maps (SOM) [2] due to its flexible neighborhood compared to the fixed grid of a SOM.

2.2. Fuzzy c-Means

The Fuzzy c-Means [10] is also a vector quantizer where each cluster Ω_𝑗 is represented by a prototype 𝐰_𝑗 located in its center of gravity. Yet contrary to NG, a data point can be assigned to more than one prototype. The cost function to minimize is given by 𝐸_FCM=^𝑁𝑉𝑁_𝑖=1^𝑃_𝑗=1𝑢_𝑗𝐯_𝑖^𝑚𝑑𝐯_𝑖,𝐰_𝑗, (7) where the fuzzy assignment of data point 𝐯_𝑖 to prototype 𝐰_𝑗 is described by 𝑢_𝑗(𝐯_𝑖)≥0. If the restriction ∑^𝑁𝑃_𝑗=1𝑢_𝑗(𝐯_𝑖)=1 is valid, the clustering is called probabilistic, otherwise possibilistic. The exponent 𝑚>1 regulates the fuzziness and according to [10] it should be set to 1.2≤𝑚≤2. Again, the distance 𝑑(𝐯_𝑖,𝐰_𝑗) is usually chosen to be the Euclidean distance.

The algorithm itself is an alternating optimization of prototypes and fuzzy assignments. The update of the prototypes is carried out by keeping the assignments fixed and vice versa the assignments are adapted based on fixed prototypes: 𝐰_𝑗=∑^𝑁𝑉_𝑖=1𝑢_𝑗𝐯_𝑖^𝑚⋅𝜕𝑑𝐯_𝑖,𝐰_𝑗/𝜕𝐰_𝑗∑^𝑁𝑉_𝑖=1𝑢_𝑗𝐯_𝑖^𝑚,𝑢_𝑗𝐯_𝑖=1∑^𝑁𝑃_𝑘=1𝑑(𝐯_𝑖,𝐰_𝑗)/(𝑑(𝐯_𝑖,𝐰_𝑘)^1/(𝑚−1). (8)(9) Since the definition of the receptive field (6) does not reflect the information contained in the fuzzy assignments, we define the fuzzy receptive field as Ω^𝐹_𝑘=𝐯_𝑖∣𝑢_𝑘𝐯_𝑖>0. (10)

2.3. Combining NG and FCM to the Fuzzy Neural Gas

As mentioned above the Fuzzy Neural Gas can now be obtained by combining NG and FNG. Thereby, the FCM distance function in (7) is replaced by local costs similar to the NG local costs (3): 𝑙𝑐^FNG_𝜎𝐯_𝑖,𝐰_𝑗=^𝑁𝑃_𝑙=1ℎ^FNG_𝜎𝐰_𝑗,𝐰_𝑙⋅𝑑𝐯_𝑖,𝐰_𝑗², (11) yielding the cost function 𝐸_FNG=^𝑁𝑉𝑁_𝑖=1^𝑃_𝑗=1𝑢_𝑗𝐯_𝑖^𝑚𝑁𝑃_𝑙=1ℎ^FNG_𝜎𝐰_𝑗,𝐰_𝑙⋅𝑑𝐯_𝑖,𝐰_𝑗²_{𝑙𝑐FNG𝜎𝐯𝑖,𝐰𝑗}. (12) The local costs (11) take the dynamic neighborhood structure according to ℎ^FNG_𝜎𝐰_𝑗,𝐰_𝑙=𝑐_𝜎⋅exp−𝑟𝑘_𝑗𝐰_𝑙,𝑊²2𝜎² (13) into account, where the value 𝜎>0 is the neighborhood range and 𝑐_𝜎 assures that ∑_𝑙ℎ^FNG_𝜎(𝐰_𝑗,𝐰_𝑙)=1. For optimal performance 𝜎 should be decreased adiabatically in the course of optimization. Note that the neighborhood contrary to the NG neighborhood is based on the winning ranks according to the best matching prototype and not as known from NG according to the data. The ranks are calculated similar to (1): 𝑟𝑘_𝑗𝐰_𝑙,𝑊=^𝑁𝑃_𝑘=1Θ𝑑𝐰_𝑙,𝐰_𝑗−𝑑𝐰_𝑙,𝐰_𝑘, (14) where Θ(𝑥) again is the heaviside function.

Analogous to FCM, the update of the prototypes and the fuzzy assignments follows an alternating optimization scheme to minimize the FNG cost function (12). The update scheme consists of two update steps: updating the prototypes while keeping the fuzzy assignments fixed and updating the assignments while retaining the prototypes. The update rules are obtained by Lagrange optimization taking the side condition ∑^𝑁𝑃_𝑗=1𝑢_𝑗(𝐯_𝑖)=1 into account.

A batch update considering all the data samples at once is possible if the Euclidean distance is used for the calculation of the local costs (11). The resulting equations can be solved for 𝐰_𝑗 and 𝑢_𝑗(𝐯_𝑖), respectively, yielding 𝐰_𝑗=∑^𝑁𝑉_𝑖=1∑^𝑁𝑃_𝑙=1𝑢_𝑙𝐯_𝑖^𝑚⋅ℎ^FNG_𝜎𝐰_𝑗,𝐰_𝑙⋅𝐯_𝑖∑^𝑁𝑉_𝑖=1∑^𝑁𝑃_𝑙=1𝑢_𝑙𝐯_𝑖^𝑚⋅ℎ^FNG_𝜎𝐰_𝑗,𝐰_𝑙,𝑢_𝑗𝐯_𝑖=1∑^𝑁𝑃_𝑙=1𝑙𝑐^FNG_𝜎𝐯_𝑖,𝐰_𝑗/𝑙𝑐^FNG_𝜎𝐯_𝑖,𝐰_𝑙^1/(𝑚−1). (15) Note that the update of the fuzzy assignments is similar to the FCM assignment update (9) yet instead of the distances 𝑑(𝐯_𝑖,𝐰_𝑗) the local costs (11) are considered.

For other distances besides the Euclidean distance, the equation obtained by Lagrange optimization might not be solvable for 𝐰_𝑗. In that case, the prototypes have to be adapted online via stochastic gradient descent in order to minimize the FNG cost function (12). The corresponding update rule is Δ𝐰_𝑗=^𝑁𝑃𝑁_𝑙=1^𝑉_𝑖=1𝑢_𝑙𝐯_𝑖^𝑚ℎ^FNG_𝜎𝐰_𝑗,𝐰_𝑙𝜕𝑑𝐯_𝑖,𝐰_𝑗𝜕𝐰_𝑗. (16) Since the derivative of the distance 𝜕𝑑(𝐯_𝑖,𝐰_𝑗)/𝜕𝐰_𝑗 has to be considered, the distance measure is required to be differentiable with respect to 𝐰_𝑗. Any measure fulfilling this restriction is a suitable measure; that is, alternative to the commonly used Euclidean distance generalized divergences as well as (differentiable) kernels might be used depending on the specific problem at hand. The latter aspect concerning (differentiable) kernels is investigated in detail in the next subsection.

2.4. Fuzzy Neural Gas with Differentiable Kernels

For vector quantizers the distance between prototypes and data samples is determined by a distance measure 𝑑(𝐯_𝑖,𝐰_𝑗). For FNG this distance has to be differentiable, since the derivative of the distance function 𝜕𝑑(𝐯_𝑖,𝐰_𝑗)/𝜕𝐰_𝑗 is considered in the prototype update rule (16) to minimize the cost function. This implies that basically any differentiable distance measure is applicable. The common Euclidean distance can be used as well as generalized divergences [11] or (differentiable) kernels [12]. Each reproducing kernel uniquely corresponds to a kernel feature map Φ∶𝑉→𝐻, where 𝐻 is a Hilbert space in a canonical manner [13]. Denote 𝐻^′=Φ(𝑉) to be the image of 𝑉. The inner product of 𝐻 is consistent with a kernel; that is, 𝜅_Φ(𝐯_𝑖,𝐰_𝑗)=⟨Φ(𝐯_𝑖),Φ(𝐰_𝑗)⟩_𝐻. Universal continuous kernels ensure the injectivity and continuity of the map. Further, in that case 𝐻^′ is a subspace of 𝐻 [13]. The inner product defines a metric by 𝑑_𝐻Φ𝐯_𝑖,Φ𝐰_𝑗=𝜅_Φ𝐯_𝑖,𝐯_𝑖−2𝜅_Φ𝐯_𝑖,𝐰_𝑗+𝜅_Φ𝐰_𝑗,𝐰_𝑗. (17) The nonlinear mapping into the Hilbert space provides large topological richness for the mapped data, which is used for classification in SVMs. However, this topological structure of the image 𝐻^′ may result in better clustering abilities for unsupervised vector quantization.

An example of a universal kernel is the widely known Gaussian kernel: 𝜅_Φ𝐯_𝑖,𝐰_𝑗=exp^{(−‖𝐯𝑖−𝐰𝑗‖2/𝜎2𝑔)}, (18) where ‖⋅‖ is the Euclidean norm. This kernel and the distance metric based thereon can be differentiated easily and is therefore suitable to be used with FNG. A disadvantage is that the parameter 𝜎_𝑔 has to be estimated, which is known to be a crucial task.

Another simple yet effective kernel is the ELM kernel (extreme learning machine) [14]. The kernel function is defined as 𝜅_Φ𝐯_𝑖,𝐰_𝑗=1𝑝Φ𝐯_𝑖,Φ𝐰_𝑗 (19) and is simply the normalized dot product in the feature space. In the context of FNG the number 𝑝 of hidden variables corresponds to the number 𝑍(𝐻^′) of intrinsic dimensions [15] of 𝐻^′ with 𝑍(𝐻^′)≤𝑁. In case that the mapping Φ(𝑥) is not known, for 𝑝→∞ the kernel can be estimated by an analytic expression [16]: 𝜅_Φ𝐯_𝑖,𝐰_𝑗=2𝜋𝐯arcsin1+_𝑖,𝐰_𝑗1/2𝜎²+1+𝐯_𝑖,𝐯_𝑖1/2𝜎²+1+𝐰_𝑗,𝐰_𝑗, (20) which is the so-called asymptotic ELM kernel, where 𝜎 is the Gaussian distribution of the data.

3. Pulsing Neural Gas

It has been shown that the Neural Gas algorithm converges to a global minimum in infinite time [3]. Yet in practice, time is limited and prototypes might only have reached a local minimum by the time the algorithm stops.

The proposed method in this section called Pulsing Neural Gas is a combination of NG and Simulated Annealing (SA), another widely known technique for solving optimization problems. SA is a probabilistic metaheuristics which accepts a random solution with a certain probability following the Boltzmann-Gibbs distribution 𝑝(Δ,𝑇)∼exp(−Δ/𝑇). This probability 𝑝 depends on the difference Δ between a random solution and the former accepted solution and a temperature 𝑇 which is decreasing over time and convergese to zero. Caused by the cooling respective annealing of the temperature 𝑇, towards the end of the optimization process a deterioration of the cost function is accepted with lower probability than at the beginning. This leads to a stable behavior in the periphery of the global minimum.

To transfer this idea to (Fuzzy) Neural Gas a correspondent to the deterioration in SA has to be found. For the common NG the cost function (4) is minimized by performing stochastic gradient descent learning. Although it cannot be guaranteed, on average the value of the cost function decreases which we consider as positive learning. We now introduce negative learning; that is, we allow the algorithm to perform a negative learning step, which increases the cost function temporarily. Hence, on average, the algorithm performs positive learning, but once in a while with a certain decreasing probability following a Gibbs-distribution a negative learning step causes a disturbance. Possibly this helps to overcome local minima and speeds up convergence to the global minimum.

First considerations took gradient ascent learning into account. However, investigations have shown that this strategy leads to an unstable learning behavior. Instead we suggest a reverse prototype ranking: 𝑟𝑘⁻_𝑗𝐯_𝑖,𝑊=𝑁_𝑝−1−^𝑁𝑝_𝑘=1𝐻𝑑𝐯_𝑖,𝐰_𝑗−𝑑𝐯_𝑖,𝐰_𝑘, (21) for a given data point 𝐯_𝑖. This ranking reverses the known (positive) ranking (1) such that the prototype with the largest distance now becomes the best (lowest) rank (see Figure 1); that is, the update of the prototypes is performed in reverse order and in opposite direction. The prototype update rule is formulated as Δ𝐰_𝑗=𝜀⋅ℎ⁻_𝜎⋅𝜕𝑑𝐯_𝑖,𝐰_𝑗𝜕𝐰_𝑗, (22) where the neighborhood function ℎ⁻_𝜎𝑗,𝑙=𝑐_𝜎⋅exp−𝑟𝑘⁻_𝑗𝐰_𝑙,𝑊²2𝜎² (23) depends on the reverse rankings 𝑟𝑘⁻_𝑗 (21). Now, in contrast to the common positive NG update step, the prototypes are not moved towards the presented data point. Yet instead, according to their reverse ranks they are pushed away, causing little change on the prototypes close to the data point and larger shifts of the prototypes located farther away. Figure 1 depicts this difference between the common NG and the Pulsing NG incorporating negative learning motivated by Simulated Annealing.

(a)

(b)

Figure 1 

Prototype update Neural Gas (a) versus Pulsing Neural Gas (b). NG: the closer the prototype the lower its rank and the stronger the update in direction of the data point. PNG: at random time steps negative learning is performed. The closer the prototype the higher its rank and the weaker the update in the opposite direction of the data sample.

Unfortunately, this strategy is not directly transferable to the batch variants of NG [17] and FNG (12). Here all the data points are presented at once and the relocation of the prototypes at each update step depends on all data points. For this variant the idea of Simulated Annealing is performed differently. Instead of a reverse ranking, now only a random subset of the data samples is presented at a randomly chosen update step: 𝐰_𝑗=∑_𝐯∈𝐴ℎ_𝜆𝑟𝑘_𝑗(𝐯,𝑊)⋅𝐯∑_𝐯∈𝐴ℎ_𝜆𝑟𝑘_𝑗(𝐯,𝑊), (24) where 𝐴⊂𝑉 is a nonempty subset. The probability for performing this update step again follows a Gibbs-distribution decreasing with proceeding training. This way, the trend of the relocations is interrupted enabling the prototypes to leave prospective local minima yet possibly causing higher costs temporarily.

One can visualize this procedure as a more or less smooth process approximating some local optimum and once in a while the whole system is shaken up resulting in a temporary increase of the cost function and causing a reorientation of the whole adaptation process. We name this modification of the NG algorithm Pulsing Neural Gas (PNG) and for the fuzzy variant FPNG.

4. Fuzzy ConnIndex for the Evaluation of Fuzzy Clusterings

A strategy to cluster very large data sets is to perform vector quantization followed by a clustering of the obtained prototypes. If it can be assured that each of the resulting clusters Ξ_𝑙 is represented by more than one prototype, the ConnIndex [8] as proposed by Taşdemir and Merényi can be used for validation purposes. Yet, the ConnIndex is suitable only if crisp vector quantization has been performed in the first step. Since we need a method to evaluate cluster solutions based on fuzzy vector quantization we modified the original ConnIndex. In the following, first we recapitulate the index as proposed by Taşdemir and Merényi and subsequently we derive a fuzzy version of the ConnIndex.

Original ConnIndex. In general, the original ConnIndex balances the overall cluster compactness and separation by combining the intercluster connectivity 𝐶_inter∈[0,1] and the intracluster connectivity 𝐶_intra∈[0,1]∶𝐶=𝐶_{𝑖𝑛𝑡𝑟𝑎}⋅1−𝐶_{𝑖𝑛𝑡𝑒𝑟}. (25) Thereby, 𝐶_{𝑖𝑛𝑡𝑟𝑎} measures the compactness of the clusters and 𝐶_{𝑖𝑛𝑡𝑒𝑟} evaluates the separation between them. A value of 𝐶 close to one suggests a good cluster solution.

For the estimation of the connectivity a nonsymmetric cumulative adjacency matrix 𝐀𝐀=^𝑁𝑉_𝑖=1Ψ𝐯_𝑖 (26) with respect to the receptive fields Ω_𝑗 (6) is considered. Here, Ψ(𝐯_𝑖) is the zero (𝑁_𝑃×𝑁_𝑃)-matrix except the element Ψ_𝑠0,𝑠1 which refers to the best matching unit 𝑠₀(𝐯_𝑖)=argmin_𝑗(𝑑(𝐯_𝑖,𝐰_𝑗)) and the second best matching unit 𝑠₁(𝐯_𝑖)=argmin_{𝑘∣𝑘≠𝑠0}(𝑑(𝐯_𝑖,𝐰_𝑘)) for data point 𝐯_𝑖. The value of this element is set to a positive constant 𝛾₁ usually chosen as 𝛾₁=1. The matrix Ψ(𝐯_𝑖) is called the response matrix with respect to the data vector 𝐯_𝑖. As pointed out in [8], the row vector 𝐚_𝑗=(𝑎_𝑗,1,…,𝑎_{𝑗,𝑁𝑃}) of 𝐀 describes the density distribution within the receptive field Ω_𝑗 with respect to the other 𝑁_𝑃−1 prototypes.

The symmetric connectivity matrix 𝐂=𝐀+𝐀^Τ (27) reflects the topological relations between the prototypes based on the receptive field evaluation. Thereby, the elements 𝑐_𝑗,𝑘 reflect the dissimilarities between the prototypes based on the local data densities.

Now, having the matrices 𝐀 and 𝐂 defined, the before mentioned connectivities 𝐶_{𝑖𝑛𝑡𝑟𝑎} and 𝐶_{𝑖𝑛𝑡𝑒𝑟} can be evaluated. The intracluster connectivity 𝐶_{𝑖𝑛𝑡𝑟𝑎} is based on the cumulative adjacency matrix 𝐀 (26): 𝐶_{𝑖𝑛𝑡𝑟𝑎}(𝑙)=∑_{𝑗,𝑘∣𝑗≠𝑘}𝑎_𝑗,𝑘∣𝐰_𝑗,𝐰_𝑘∈Ξ_𝑙∑_{𝑗,𝑘∣𝑗≠𝑘}𝑎_𝑗,𝑘∣𝐰_𝑗∈Ξ_𝑙 (28) for each cluster Ξ_𝑙. The greater the compactness of a cluster Ξ_𝑙 the closer its intraconnectivity is to one. Note again that, as mentioned above, each cluster is made up of more than one prototype 𝐰_𝑗.

The inter-cluster connectivity 𝐶_{𝑖𝑛𝑡𝑒𝑟} evaluates the separation between the clusters. Analogously, it is the average over the local inter-cluster connectivities 𝐶_{𝑖𝑛𝑡𝑒𝑟}(𝑙)=max_{1≤𝑚≤𝐾,𝑚≠𝑙}𝐶_{𝑖𝑛𝑡𝑒𝑟}(𝑙,𝑚) (29) of all clusters evaluating the separation of each cluster Ξ_𝑙 to the other clusters Ξ_𝑚,𝑚≠𝑙. Thereby, 𝐶_{𝑖𝑛𝑡𝑒𝑟}(𝑙,𝑚) judges the separation of cluster Ξ_𝑙 to cluster Ξ_𝑚 based on the connectivity matrix 𝐂 (27) and is defined as 𝐶_{𝑖𝑛𝑡𝑒𝑟}()=𝑙,𝑚0if𝑆_𝑙,𝑚∑=0_{𝑗,𝑘∣𝑗≠𝑘}𝑐_𝑗,𝑘∣𝐰_𝑗∈Ξ_𝑙,𝐰_𝑘∈Ξ_𝑚∑_{𝑗,𝑘∣𝑗≠𝑘}𝑐_𝑗,𝑘∣𝐰_𝑗∈𝑆_𝑙,𝑚if𝑆_𝑙,𝑚≠0, (30) where the sets 𝑆_𝑙,𝑚=𝐰_𝑗∣𝐰_𝑗∈Ξ_𝑙∧∃𝐰_𝑘∈Ξ_𝑚∶𝑎_𝑗,𝑘>0 (31) describe the neighborhood relations between the clusters Ξ_𝑙 and Ξ_𝑚 based on the contained prototypes. In contrast to 𝐶_{𝑖𝑛𝑡𝑟𝑎}, the value of 𝐶_{𝑖𝑛𝑡𝑒𝑟} decreases with better separability.

Generalization of the ConnIndex. The ConnIndex by Taşdemir and Merényi considers the best and second best matching units 𝑠₀(𝐯_𝑖) and 𝑠₁(𝐯_𝑖) only, discarding any information provided by higher ranked prototypes. A generalized version of the index is obtained by incorporating higher winning ranks as known from Neural Gas [3]; see (1). Obviously 𝑟𝑘_{𝑠0(𝐯𝑖)}(𝐯_𝑖,𝑊)=0 is the rank of the best matching prototype. Analogously, the 𝑝th winner is denoted by 𝑠_𝑝−1(𝐯_𝑖) with rank 𝑟𝑘_𝑠𝑝(𝐯_𝑖,𝑊)=𝑝. If it is clear from the context, we will abbreviate 𝑠_𝑝=𝑠_𝑝(𝐯_𝑖) in the following.

To incorporate the higher winning ranks the response matrix Ψ(𝐯_𝑖) has to be redefined to involve the full response of the whole vector quantizer model for a given input 𝐯_𝑖. The new response matrix Ψ(𝐯_𝑖) is a zero matrix of the same size as Ψ(𝐯_𝑖) except the row vector regarding the winner 𝑠₀(𝐯_𝑖). The new response matrix is set to Ψ_𝑠0𝐯_𝑖=𝐫𝐯_𝑖, (32) where 𝐫(𝐯_𝑖) is the so-called response vector of all prototype responses for a given input 𝐯_𝑖. The vector elements 𝑟_𝑗(𝐯_𝑖) of the 𝑗th prototype are defined as 𝑟_𝑗𝐯_𝑖=𝜑_𝜔𝑟𝑘_𝑗𝐯_𝑖,𝑊, (33)

with 𝜑_𝜔(𝑥) being an arbitrary monotonically decreasing function in 𝑥. A simple choice for this function is the exponential function 𝜑_𝜔(𝑥)=exp(−(1−(𝑖𝑑(𝑥))/(𝑛−1))/2𝜔²). The parameter 𝜔 determines the range of influence and should be determined carefully. If for the vector quantization an algorithm incorporating neighborhood cooperativeness in learning like Neural Gas [3] or self-organizing maps [18] was used, the 𝜔-parameter should be chosen according to the neighborhood range used there. Yet, an alternative approach could be the direct utilization of the distances 𝑑(𝐯_𝑖,𝐰_𝑗) instead of the winning ranks and 𝜑_𝜔(𝑥).

This generalized version of the ConnIndex 𝐶_𝑔 uses for the calculation of the cumulative adjacency matrix 𝐀 in (26) the new response matrices Ψ(𝐯_𝑖) instead of the original response matrices Ψ(𝐯_𝑖).

Note that this version is in concordance with the original version, if 𝜑(𝑥) is chosen as 𝜑(𝑥)=𝛾₀𝛾for𝑥=0₁for𝑥=10else. (34)

Fuzzy ConnIndex. Up to now we assumed that the vector quantization model is based on a crisp mapping. For these models a winner ranking is available and the response information of the network is collected in the response vector 𝐫(𝐯_𝑖), reflecting the topological relation between the prototypes. In fuzzy vector quantization algorithms this information is no longer available because each data point is gradually assigned to all prototypes. Yet, the fuzzy data point assignments 𝑢_𝑗(𝐯_𝑖) which can be stored in a 𝑁_𝑉×𝑁_𝑃 assignment matrix 𝑈 also reflect the topography of the underlying data. The assignment vector 𝐮_𝑖 is then the specific vector of 𝑈 which contains the assignment value of a data point 𝐯_𝑖 to all of the prototypes and is comparable to the response vector 𝐫(𝐯_𝑖) used for the Generalized ConnIndex 𝐶_𝑔. Therefore, the assignments can be used directly to determine the response matrix Ψ(𝐯_𝑖) by substituting the response vector Ψ_𝑠0(𝐯_𝑖) in (32). Consequently, the best matching prototype 𝑠₀(𝐯_𝑖) for a given data vector can be seen as the prototype with the highest fuzzy assignment 𝑢_𝑗(𝐯_𝑖): 𝑠₀=max_𝑗𝑢_𝑗𝐯_𝑖. (35) Now, the row vector Ψ_𝑠0(𝐯_𝑖) of the redefined response matrix Ψ(𝐯_𝑖) can simply be chosen as the fuzzy response vector 𝐮_𝑖: Ψ_𝑠0𝐯_𝑖=𝐮_𝑖. (36) Again, the cumulative adjacency matrix 𝐀 is calculated as before for the original ConnIndex 𝐶 and the Generalized ConnIndex 𝐶_𝑔 according to (25). Further calculations remain unaffected.

Hence, the resulting new fuzzy ConnIndex 𝐶_𝑓 is the counterpart of the generalized ConnIndex 𝐶_𝑔 in case of fuzzy vector quantization models.

5. Performance

To evaluate the performance of FNG we designed different experiments to compare this method with crisp vector quantizers and the fuzzy c-Means. We also conducted an experiment examining the pulsing FNG. To perform the tests we used artificial and real world data sets.

For the evaluation of the cluster results we used the ConnIndex or the Fuzzy ConnIndex, respectively. This evaluation measure, described in the previous section, is relatively new [8, 19]. But it seems to be well suited for the evaluation of cluster solutions in terms of separation and compactness.

Additionally, for the first Smiley dataset we also calculated the Kappa value 𝜅_𝐶 [20] which is a measure to judge the agreement of two cluster solutions. A variant thereof 𝜅_𝐹 is suitable for fuzzy data [21]. Unfortunately, this measure can be used only for cluster solutions with a low number of clusters, since the clusters of the different solutions have to be matched, which is hard for clusterings containing a higher number of clusters.

5.1. Artificial Dataset: Smiley

In the first setting we used the Smiley data set [19]. This two-dimensional data set consists of three clusters with varying shapes, number of data samples, variances, and distances to each other (see Figure 2). It contains a total of 809 data points.

(a)

(b)

Figure 2 

Smiley dataset clustered by Fuzzy Neural Gas (FNG). (a) Obtained prototypes. (b) Fuzzy cluster assignments of the data points.

In the first step we apply c-Means and NG to perform crisp vector quantization and FCM and FNG to perform fuzzy vector quantization with the fuzziness parameter 𝑚 set to different values 𝑚={1.1,…,2.0}. All algorithms result in acceptable solutions. For 𝑚=1.4 the FNG cost function settles at the lowest value; FCM reaches the lowest costs for 𝑚=1.5. The obtained FNG prototypes are depicted in Figure 2; the FCM results look similar. Visual evaluation confirms an intuitively good distribution in the data space.

A more objective evaluation is obtained with the help of the (Fuzzy) ConnIndex. Yet, to apply this measure the prototypes themselves have to be grouped to clusters of at least two prototypes each. In this simple experiment this step is done manually following the inherent obvious structure of the data set consisting of three clusters.

The obtained ConnValues are listed in Table 1 and show as expected a clear discrepancy between the ConnIndex values obtained by crisp and those obtained by fuzzy vector quantization. This is due to the influence of the data points located in the gaps between the main clusters on the calculation of the index. It is also evident that NG and FNG perform better than c-Means and Fuzzy c-Means, respectively. The overall best ConnIndex value is obtained for FNG, which is less surprising since this algorithm is a combination of FCM and NG, taking beneficial features of each: NG neighborhood and FCM fuzzy assignments.

Table 1 

ConnIndex and 𝜅-values for clusterings of different datasets obtained by crisp and fuzzy vector quantizers. Due to the low number of clusters of the Smiley dataset the 𝜅-values can be calculated. There is almost no difference between the crisp methods but a substantial discrepancy between the fuzzy vector quantizer. FNG performs way better than FCM. This observation is supported by evaluating the ConnIndex. To evaluate the cluster solutions of the other two datasets only the ConnIndex was used: for crisp methods the generalized ConnIndex 𝐶_𝐺 and for fuzzy methods the Fuzzy ConnIndex 𝐶_𝐹.

The Kappa values 𝜅_𝐶 [20] and 𝜅_𝐹 [21] measure the agreement of two cluster solutions. The closer the values are to one, the higher is the agreement. Comparing the given data structure with the results obtained by the four different clustering methods yields high values indicating substantial to perfect agreement (according to [22]); see Table 1. It can be observed that the two crisp methods NG and c-Means performed almost equally, while the discrepancy between FCM and FNG is remarkable, indicating superior performance of FNG. Note that the values of the crisp and fuzzy solutions cannot be compared to each other since two different 𝜅-measures are applied.

5.2. Practical Example: Indian Pine

Indian Pine is a publicly available data set taken by the NASA Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) consisting of 145×145 pixels [23]. Data samples cover 220 bands of 10 nm width from 400 to 2500 nm. Due to the atmospheric water absorption 20 noisy bands can be identified (104–108, 150–163, and 220) and removed safely [24]. The data set is labeled according to 16 identified classes, but we do not use this information for the current experimental setting.

The processing of the data consists of two steps. First vector quantization is performed to position the prototypes. For this step the same algorithms as in the last experiment are used: crisp c-Means and NG and fuzzy FCM and FNG. In the second step the obtained prototypes are grouped by affinity propagation (AP) [25] to be able to apply the (Fuzzy) ConnIndex for evaluation. Special care has to be taken to fulfill the requirement that each cluster (i.e., prototype cluster) is represented by more than one prototype. For this reason a sufficiently high number of prototypes has to be chosen. We set this number to 64 (four times the number of known classes).

The calculation of the Generalized ConnIndex 𝐶_𝐺 for the crisp methods is straightforward. For the fuzzy variants again the fuzziness parameter 𝑚 has to be considered carefully. A value of 𝑚=1.5 has proven to be favourable. The respective obtained ConnIndex values 𝐶_𝐺 and 𝐶_𝐹 are listed in Table 1.

Although the prototype clustering by Affinity Propagation always results in crisp cluster assignments, the clustering based on FNG vector quantization still yields better ConnIndex values than the other methods.

5.3. Practical Example: Colorado

The Colorado data set [26] is a LANDSAT TM image from the Colorado area, USA. The image covers a region of about 50×50 kilometers yielding approximately 2 million data points. These are labeled by experts according to 14 different vegetation types and geological formations found in this region. Among them are aspen, mixed pine forest, water, moist meadow, and dry meadow to name a few. The original data samples are 7-dimensional, yet one band (thermal band) is removed due to its low resolution. Generally, the bands are highly correlated [26].

For the experiment we neglected the class information and selected randomly 10% of the data with a representative class distribution. The number of prototypes is set to 56. Besides that, the setup of the experiment is identical to the setup for the Indian Pine and consists of the two there described processing steps.

It can be observed that the FCM training is much faster than FNG training and requires less training cycles, about 35 versus 90. Yet, the Fuzzy ConnIndex yields much better results for FNG than FCM (see Table 1) indicating a better prototype distribution in terms of inter- and intraconnectivity of the obtained clusters. The reason for the prolonged processing time can be found in the computational costs to calculate all neighborhood relations anew in each processing step.

5.4. Artificial Data Set: Checkerboard

This artificial data set [27] consisting of compact yet well-separated clusters arranged in a checkerboard-like manner is well suited to demonstrate the performance of the Pulsing Neural Gas compared to the common Neural Gas. The data set contains 11.250 two-dimensional data vectors, which are grouped in 15×15 normally distributed clusters with a standard deviation of 𝜎=0.3. The mean distance between two neighboring cluster centers is 2.5. Due to the low dimensionality the data set is well suited for visualization; see Figure 3(a).

(a)

(b)

Figure 3 

Final cluster solutions (a) and cost functions (b) for the Checkerboard data set using NG and PNG. (a) It can easily be verified that the PNG algorithm has more prototypes placed within the clusters than common NG. The number of training steps for both algorithms was the same. (b) The blue dots refer to the costs for Pulsing Neural Gas. The little bumps at time steps 41 and 43 indicate that a negative learning step has occurred.

For both algorithms NG and PNG all 225 prototypes are initialized in the center of the data set. In the following the algorithms are run both for the same number of steps. For comparison the values of the energy functions according to (4) are used. The experiment showed that on the long run both algorithms performed well. For online learning the effect of the pulsing variant is neglectable, yet the batch version shows significant improvements. The cost function of the Pulsing Neural Gas reaches lower values. The negative learning steps show as little bumps in the plot of the energy functions (see Figure 3(b)), indicating a temporary deterioration. In Figure 3(a) the prototype distribution after 50 learning steps is visualized. Obviously the number of misplaced NG prototypes is higher than the number of misplaced PNG prototypes. This finding is in accordance with the lower value of the PNG energy function.

6. Conclusion

We proposed in this paper a fuzzy version of the Neural Gas. By combining the concept of neighborhood cooperativeness as known from NG with the FCM fuzzy assignments we obtain the Fuzzy Neural Gas. This algorithm outperforms FCM by taking dynamic neighborhood relations into account, a paradigm proven to be well suited for crisp vector quantization. The resulting FNG shows good performance compared to standard FCM and crisp NG. Due to the neighborhood cooperativeness this algorithm is insensitive to the initialization of the prototypes.

It is straight forward to introduce other distance measures besides the commonly used Euclidean distance. The only prerequisite is that the measure has to be differentiable; for example, differentiable kernels might be used.

A further variant of NG, respectively, its fuzzy version, is the Pulsing Neural Gas imitating a Simulated Annealing-like behaviour. This modification which allows temporary deterioration of the cost function stabilizes in the long run the learning procedure and helps the algorithm to overcome local minima more easily. This effect was demonstrated on a checkerboard data set, for which it is known that usually the algorithms do not find all clusters.

And finally, we extended the original crisp cluster evaluation ConnIndex [21] to be used for fuzzy clustering. It is based on a generalization of the index considering all prototypes instead of first and second best matching units only. The fuzzy version additionally takes the fuzzy information provided by the fuzzy data point assignments into account. As the original, the Fuzzy ConnIndex requires more than one prototype per cluster. The index was used for the evaluation of the experiments.

Acknowledgment

Marika Kaden was funded by the European Social Fund (ESF), Saxony.