Similarity Measure Learning in Closed-Form Solution for Image Classification

Abstract

Adopting a measure is essential in many multimedia applications. Recently, distance learning is becoming an active research problem. In fact, the distance is the natural measure for dissimilarity. Generally, a pairwise relationship between two objects in learning tasks includes two aspects: similarity and dissimilarity. The similarity measure provides different information for pairwise relationships. However, similarity learning has been paid less attention in learning problems. In this work, firstly, we propose a general framework for similarity measure learning (SML). Additionally, we define a generalized type of correlation as a similarity measure. By a set of parameters, generalized correlation provides flexibility for learning tasks. Based on this similarity measure, we present a specific algorithm under the SML framework, called correlation similarity measure learning (CSML), to learn a parameterized similarity measure over input space. A nonlinear extension version of CSML, kernel CSML, is also proposed. Particularly, we give a closed-form solution avoiding iterative search for a local optimal solution in the high-dimensional space as the previous work did. Finally, classification experiments have been performed on face databases and a handwritten digits database to demonstrate the efficiency and reliability of CSML and KCSML.

1. Introduction

Pairwise matching, which is based on a measure (similarity or dissimilarity), is ubiquitous in multimedia applications. The performances of multimedia learning techniques depend sensitively on the selected measure [1–3]. Recently, measure learning has become an active research problem for multimedia learning tasks, for example, image classification [4, 5]. The previous measure learning studies mainly focused on distance (dissimilarity) learning. One of the earliest distance learning algorithms was presented by Xing et al. [6], where a parameterized Mahalanobis distance was learned. Many distance learning studies were followed [7–10], which would be overviewed later. There are two aspects of disadvantages for a distance metric. On the one hand, a distance learning task results in an optimization problem which is usually not easy to give a closed-form solution. Xing et al. [6], Lee et al. [11], Kumar and Kummamuru [12], Jin et al. [13], and Yin et al. [14] all described the distance metric learning through iterative process. The iterative methods are difficult to be extended to kernel versions. Moreover, the iterative procedure is inefficient and unstable. On the other hand, several recent studies suggest that the strict metric axioms (self-similarity, symmetry, and triangle inequality) are epistemologically invalid for perceptual distance of human beings [15, 16] and not so suitable for robust pattern recognition [17].

The other aspect of the relationship between two objects in learning tasks is similarity. Since the measure models vary in engineering practice, dissimilarity and similarity are not simply complementary. The similarity cannot be simply viewed as the negative or reciprocal dissimilarity. It is necessary to distinguish these two notions. The similarity measures include two categories: inner product based and kernel function based, which were both considered in this work.

Many publications support that the intrinsic structure of the feature space for image classification lies on low-dimensional manifolds [18–20]. Compared with Euclidean distance, correlation has some competitive abilities to capture the intrinsic structure embedded in the high-dimensional data. Correlation is a type of normalized inner product and a scale invariant index. It corresponds to the notion of “angle” in geometrical theory. In recent years, some studies have used correlation as a similarity measure for dimension reduction [21–23]. However, since correlation is in the fraction form, the existing correlation-based dimension reduction algorithms, such as correlation embedding analysis (CEA) [21], canonical correlation analysis (CCA) [22], and correlation discriminant analysis (CDA) [23], constructed low-dimensional embeddings through the iterative procedures.

In this work, we presented a similarity measure learning framework for supervised classification. Particularly, under this framework, a correlation similarity measure learning algorithm was constructed with a closed-form solution. It did not need iterative update process and is only involved in eigenvalue decomposition operations. Furthermore, it was extended into a kernelized version.

In order to learn an appropriate similarity measure, dissimilarity metric (distance) learning and dimensionality reduction can bring us much inspiration. Here, we provided a concise review on them.

1.1. Dissimilarity Metric Learning

Many dissimilarity metric learning algorithms have been presented in a variety of application areas. From the diverse points of view, these methods can be divided into different ways. Generally, there are two ways to categorize them: (1) unsupervised learning and supervised learning and (2) global method and local method. In this work, the latter categorization scheme is adopted.

For global methods, the well-known one is the earlier distance metric learning algorithm Xing et al. presented [6], which will be shown in detail later. This algorithm is further extended to the nonlinear case in [24] by the introduction of kernels, where a given kernel is idealized such that it becomes more similar to the ideal kernel also leads to a quadratic programming problem. Relevant component analysis (RCA) [25] learns a global linear transform from the equivalent constraints. Instead of iterative solution in [6], it only uses closed-form expressions of data and is based on subsets of points so-called chunklets. However, RCA has two important disadvantages. One is the lack of exploiting negative constraints which can also be informative, and the other is its incapability of capturing complex nonlinear relationships between data instances with the contextual information [8]. Discriminative component analysis (DCA) and kernel DCA [8] improve RCA by exploring negative constraints with contextual information. Kernel RCA [26] and kernel DCA use kernel trick to discover the nonlinear structures of the given data. Recently, Wang [9] proposed a method to learn Mahalanobis distance metric in semisupervised mode by maximizing the so-called constraint-margin maximization (CMM) criterion. CMM is based on graph embedding framework [27] often used in dimensionality reduction problems.

All the global methods are based on global constraints or side information. However, the real-world data may not satisfy the global linear assumption. So more approaches fall into local based category which approximates global nonlinear data structures based on local linear alignment. Discriminant adaptive nearest neighbor [10] estimates a local distance metric using the local linear discriminant analysis. Neighbourhood components analysis [28] learns a Mahalanobis distance metric by directly maximizing a stochastic variant of the leave-one-out KNN score on the training set. The maximum-margin nearest neighbor (LMNN) classifier [29] extends NCA through a maximum-margin framework. It reformulated the optimization problem as an instance of semidefinite programming, which was also solved by iterative process. Many other recent studies [29–31] also focus on neighbor information.

1.2. Dimensionality Reduction

Most algorithms above are based on so-called Mahalanobis distance function framework, which may be viewed as constructing a global linear transformation of the data and then applying the Euclidean distance over the transformed data. Mahalanobis distance is as follows: 𝑑𝑥,𝑦=𝑥−𝑦_𝐴=𝑥−𝑦^𝑇𝐴𝑥−𝑦. (1)

It requires 𝐴⪰0 to ensure that this can be used as a metric. So 𝐴 can be represented as 𝐴=𝑊𝑊^𝑇. Then to learn Mahalanobis distance is equivalent to finding a transform matrix 𝑊 (𝑦=𝑊^𝑇𝑥). Learning the transformation matrix 𝑊 can yield the Mahalanobis metric 𝐴=𝑊𝑊^𝑇 according to 𝑦_𝑖−𝑦_𝑗²=𝑥_𝑖−𝑥_𝑗^𝑇𝑊𝑊^𝑇𝑥_𝑖−𝑥_𝑗=𝑥_𝑖−𝑥_𝑗^𝑇𝐴𝑥_𝑖−𝑥_𝑗=𝑥_𝑖−𝑥_𝑗²_𝐴. (2)

For those dimensionality reduction methods without explicit transformation, they may also be viewed as searching appropriate embedding in a lower-dimensional space. So distance metric learning has an affinity with dimensionality reduction.

Methods on dimensionality reduction can be divided into two categories: (1) with explicit transformation and (2) with implicit transformation. The former includes almost all subspace learning algorithms. PCA, LDA, NMF, LPP [32], Laplacian Eigenmap [33], and their extended visions [34–36], all result in a transform matrix with optimizing some objective criterions. Most of classical manifold learning algorithms, such as LLE, ISOMAP, and LTSA, belong to the latter category. Since being without explicit transformation, manifold-based methods are more suitable for data visualization than classification. Inspired by NMF and LPP, graph embedding framework [27] becomes popular [37, 38]. It provides more flexibility through designing diverse graphs and weight matrices.

This paper aims at solving the following problem: given a set of sample data with class labels or pairwise constraints, the task is to learn an appropriate similarity measure for classification. In the beginning of this paper, several related distance learning and dimensionality reduction algorithms will be introduced. The previous work of correlation usage in classification will be also discussed in Section 2. The methods of Xing et al. and Xiang et al. are both used to learn the distance metric. However, Xing et al. mainly concentrated on clustering application. CEA [21], CCA [22], and CDA [23] all apply correlation for classification. Moreover, CEA [21] is also based on graph embedding framework [27] as our method does. These algorithms will be all described detailedly in Section 2.

Sections 3, 4, and 5 form the core of this paper. Section 3 gives the precise definition of similarity measure learning problem and introduces a general formulation for it. This formulation can be specified to diverse measure learning algorithms depending on the determination of neighbor graph, affinity weights, and similarity measure. In Section 4, a strategy is given to form specific similarity measure learning algorithm. Firstly, a generalized correlation 𝜌 is defined. After that, two kinds of constraints are introduced, which are based on two kinds of neighbor graphs and corresponding affinity weights. Most importantly, an approximate optimization and its closed-form solution are presented. Following that, it is extended to the nonlinear version in Section 5. Experiments have been conducted to prove the effectiveness of these new measure learning approaches for classification. They will be reported in Section 7. Additionally, discussions and conclusions will be given, respectively, in Sections 6 and 8.

The overall sequence of the core sections in this paper can be illustrated as follows.

Similarity Measure Learning for Classification SML—A framework The definition of SML problem General framework for SML CSML—An algorithm Generalized correlation ρ Optimization problem for CSML The closed-form solution of CSML KCSML—A nonlinear extension of CSML.

2. Related Work

This section provided a brief overview of closely related studies. From this analysis, our work would be placed in the context of other algorithms.

2.1. Xing and Xiang’s Methods

Consider the form of a distance metric as follows: 𝑑𝑥,𝑦=𝑥−𝑦_𝐴=𝑥−𝑦^𝑇𝐴𝑥−𝑦, (3) where 𝐴⪰0. Xing et al. introduced one of the earliest distance metric learning methods using both positive and negative constraints [6]. They posed distance metric learning as the following convex optimization problem: min𝐽=_{𝑥𝑖,𝑥𝑗∈𝑆}𝑥_𝑖−𝑥_𝑗²_𝐴s.t._{(𝑥𝑖,𝑥𝑗)∈𝐷}𝑥_𝑖−𝑥_𝑗𝐴≥1𝐴⪰0, (4) where 𝑆 was the set of positive constraints and 𝐷 was the set of negative constraints. The optimal metric was found by minimizing the distances between data points in affinity-link constraints and simultaneously maximizing the distances between data points in apart-link constraints. Xing et al. [6] used the gradient descent and the idea of iterative projection to solve the problem (4). Although the presented optimization problem was convex, it was a hard problem to solve. And the introduced solution in [6] was slow and somewhat unstable [8].

Xiang et al. [39] introduced the trace-ratio objective function (with the constraint 𝑊^𝑇𝑊=𝐼) as a more appropriate objective function: 𝑊^∗=argmax_{𝑊𝑇𝑊=𝐼}tr𝑊^𝑇𝑆_𝑏𝑊tr𝑊^𝑇𝑆_𝑤𝑊. (5) However, this problem cannot be directly solved by eigenvalue decomposition approaches. To solve the problem (5), Xiang et al. [39] had constructed an iterative framework, in which a lower bound and an upper bound including the optimum were estimated for initialization. Their proposed method provides a heuristic search to solve the problem (5). In this work, we propose a generalized form of similarity measure learning rather than dissimilarity measure learning and provide a closed-form solution of objective function with correlation similarity.

2.2. CEA

Fu et al. [21] introduced correlation embedding analysis (CEA) for dimensionality reduction. Firstly, two undirected weighted graphs, the intrinsic graph 𝐺^𝐼=(𝑋,𝑊^𝐼) and the penalty graph 𝐺^𝑃=(𝑋,𝑊^𝑃), were constructed. 𝑋 was a set of data vertexes and 𝑊^𝐼,𝑊^𝑃∈𝑅^𝑛×𝑛 are weight matrices. The intrinsic graph characterizes data links that the algorithm favors and the penalty graph describes relationships that the algorithm tries to avoid. Then, a graph-preserving criterion is imposed for these two objectives as argmax_𝑊𝐹(𝑊)=_𝑖≠𝑗𝑊^𝑇𝑥_𝑖𝑊^𝑇𝑥_𝑖−𝑊^𝑇𝑥_𝑗𝑊^𝑇𝑥_𝑗²⋅𝑤^𝑃_𝑖𝑗−𝑤^𝐼_𝑖𝑗, (6) where 𝑤^𝑃_𝑖𝑗 and 𝑤^𝐼_𝑖𝑗 are the elements of weight matrices 𝑊^𝑃 and 𝑊^𝐼, respectively. It can be viewed as finding transformation matrix 𝑊 in the linear transformation space of normalized samples. The formulation (6) can be rewritten as argmax_𝑊𝐹(𝑊)=2_𝑖≠𝑗𝑥1−^𝑇𝑖𝑊𝑊^𝑇𝑥_𝑗𝑥^𝑇_𝑖𝑊𝑊^𝑇𝑥_𝑖𝑥^𝑇_𝑗𝑊𝑊^𝑇𝑥_𝑗⋅𝑤^𝑃_𝑖𝑗−𝑤^𝐼_𝑖𝑗. (7)

This objective function is nonlinear and not convex. Fu et al. [21] used the gradient descent rule for optimization by differentiating 𝐹(𝑊) with respect to matrix W. As pointed in [21], the gradient descent may not be deep enough to converge to a good solution when the dimension of the data space is too large. So the iterative process is sensitive on the initial point although the method to find a good initialization was proposed. In this paper, we transform the problem (7) into another optimization problem which can be solved with closed-form solution.

2.3. Correlation in Classification

Next, we will focus on the usage of correlation in classification. Hardoon et al. [22] introduced canonical correlation analysis (CCA). It can be viewed as the problem of finding basis vectors for two sets of variables such that the correlations between the projections of the variables are mutually maximized. If one set of variables is taken as class labels, CCA can be used to realize a supervised linear feature extraction and subsequent classification. It has been extended to a nonlinear version kernel CCA by kernel trick. However, there are some problems when it is used in classification application as pointed in [23], which limits its utilization in practice.

Ma et al. [23] introduced correlation discriminant analysis (CDA) which sought a global linear transformation to maximize the correlation of samples from different classes in the transformed space. Its optimization problem was max_𝐴𝑆_𝑤−𝑆_𝑏ormax_𝐴𝑆_𝑤−𝑆_𝑡s.t.𝐴⪰0, (8) where 𝑆_𝑤=1𝑁_{𝑤𝑡𝑖=𝑡𝑗}𝑦^𝑇_𝑖𝑦_𝑗𝑦_𝑖𝑦_𝑗=1𝑁_{𝑤𝑡𝑖=𝑡𝑗}𝑥^𝑇_𝑖𝑤^𝑇𝑤𝑥_𝑗𝑥^𝑇_𝑖𝑤^𝑇𝑤𝑥_𝑖𝑥^𝑇_𝑗𝑤^𝑇𝑤𝑥_𝑗,𝑆_𝑏=1𝑁_{𝑤𝑡𝑖≠𝑡𝑗}𝑦^𝑇_𝑖𝑦_𝑗𝑦_𝑖𝑦_𝑗=1𝑁_{𝑤𝑡𝑖≠𝑡𝑗}𝑥^𝑇_𝑖𝑤^𝑇𝑤𝑥_𝑗𝑥^𝑇_𝑖𝑤^𝑇𝑤𝑥_𝑖𝑥^𝑇_𝑗𝑤^𝑇𝑤𝑥_𝑗,𝑆_𝑡=1𝑁_𝑤+𝑁_{𝑏𝑡𝑖,𝑡𝑗}𝑦^𝑇_𝑖𝑦_𝑗𝑦_𝑖𝑦_𝑗=1𝑁_{𝑤𝑡𝑖,𝑡𝑗}𝑥^𝑇_𝑖𝑤^𝑇𝑤𝑥_𝑗𝑥^𝑇_𝑖𝑤^𝑇𝑤𝑥_𝑖𝑥^𝑇_𝑗𝑤^𝑇𝑤𝑥_𝑗𝐴=𝑊^𝑇𝑊. (9)

In [23], this problem was also solved by gradient-based optimization method. However, the extension of CDA to kernel CDA was not very easy to be implemented, as pointed in [23].

3. General Framework

Similarity measure learning (SML) is a general framework for similarity measure learning problem. In the context of general supervised classification, the SML problem may be formulated as follows: given a labeled sample set {(𝑋,𝑌)}, with 𝑛 instances, {𝑥_𝑖}^𝑛_𝑖=1∈𝑅^𝐷, and 𝐷 is the feature dimension. The corresponding class label is {𝑦_𝑖}^𝑛_𝑖=1∈{1,…,𝑐}, where 𝑐 is the number of classes. Suppose that the similarity measure between arbitrary two objects 𝑥_𝑖 and 𝑥_𝑗 is 𝜌(𝑥_𝑖,𝑥_𝑗,𝑊), where 𝑊 is a set of parameters to be learned. The goal of SML is to learn the parameter set 𝑊 from the sample set {(𝑋,𝑌)}.

We now introduce SML problem from the novel point of view of graph embedding. Let 𝐺={(𝑋,Δ)} be an undirected weighted graph with vertex set 𝑋 and relation matrix Δ∈𝑅^𝑛×𝑛. We define an intrinsic graph 𝐺^𝐼={(𝑋,Δ^𝐼)}, where Δ^𝐼=[𝛿^𝐼_𝑖𝑗]_𝑛×𝑛, and a penalty graph 𝐺^𝑃={(𝑋,Δ^𝑃)}, where Δ^𝑃=[𝛿^𝑃_𝑖𝑗]_𝑛×𝑛. Vertices 𝑋 of graph 𝐺^𝐼 are the same as those of graph 𝐺^𝑃, but the matrix Δ^𝐼 corresponds to the relations that are to be strengthened and the matrix Δ^𝑃 corresponds to the relations that are to be suppressed in the learning process.

Based on the above evidences, we get the formal definition of the similarity measure learning.

Definition 1. The similarity measure learning (SML) problem is to learn an optimal similarity measure [𝜌_𝑖𝑗]_𝑛×𝑛 from a collection of data points 𝑋 on a vector space 𝑅^𝐷 together with a set of intrinsic pairwise constraints 𝐺^𝐼 and a set of penalty pairwise constraints 𝐺^𝑃, which can be formally formulated into the following optimization framework: min(ormax)𝑓𝑊,𝐺^𝐼,𝐺^𝑃,𝑋, (10) where 𝑊 is a set of parameters to be learned and 𝑓 is some objective function defined over the given data.
Inspired by graph embedding learning in dimensionality reduction, SML can be formulated as the following two objectives based on graph-preserving criterion: max_{𝑊𝑖≠𝑗}𝜌²𝑥_𝑖,𝑥_𝑗,𝑊𝛿^𝐼_𝑖𝑗min_{𝑊𝑖≠𝑗}𝜌²𝑥_𝑖,𝑥_𝑗,𝑊𝛿^𝑃_𝑖𝑗. (11) To combine these two objectives into a unique optimization problem, there exist several different ways [27]. In this work, we consider the difference-form formulation; namely, 𝑊^∗=argmax𝐿(𝑊)=_𝑖≠𝑗𝜌²𝑥_𝑖,𝑥_𝑗,𝑊𝛿^𝐼_𝑖𝑗−𝛿^𝑃_𝑖𝑗. (12) It can be seen from Definition 1 that the method proposed in the next section will be also suitable for classification problem with pairwise constraints instead of labels.

4. Correlation Similarity Measure Learning

In this section, we introduce a generalized correlation measure 𝜌. Based on the generalized correlation, an algorithm of SML, called correlation similarity measure learning (CSML), is proposed. It aims at learning a correlation similarity measure for classification. The details are summarized in Algorithm 1.

Input: the sample set 𝑋={𝑥1,𝑥2,…,𝑥𝑛} with labels 𝑌={𝑦1,𝑦2,…,𝑦𝑛} Output: the parameter set 𝑀 for the generalized correlation 𝜌    Step  1. Construct the intrinsic graph 𝐺𝐼 and penalty graph 𝐺𝑃;    Step   2. Compute the affinity weights: Δ𝐼 and Δ𝑃;    Step   3. Construct the optimization problem (17);    Step   4. For 𝑘=1 To 𝑑 Do       If 𝑘=1 Then        Solve 𝑆𝛿𝑤1=𝜆𝑆𝑒𝑤1 to obtain 𝑤1       Else        Compute 𝐿 and solve 𝐿𝑤𝑘=𝜆𝐿𝑤𝑘 to obtain 𝑤𝑘;    Step  5. Final output 𝑀=𝑊𝑊𝑇, where 𝑊=[𝑤1,𝑤2,…,𝑤𝑑].

Algorithm 1 

The details of algorithm CSML.

4.1. Objective Function

“Correlation” is one of widely used measures to reflect the similarity between two random variables. Correlation is also termed as normalized correlation, correlation coefficient, Pearson’s correlation, or cosine similarity, and hereafter correlation for simplicity. Two samples (e.g., images) are represented as two vectors 𝑥_𝑖 and 𝑥_𝑗 in a feature space, and then the standard form of correlation is corr𝑥_𝑖,𝑥_𝑗=𝑥^𝑇_𝑖𝑥_𝑗𝑥^𝑇_𝑖𝑥_𝑖𝑥^𝑇_𝑗𝑥_𝑗. (13) In learning tasks, to make the similarity measure flexible to sample data, we define a generalized correlation.

Definition 2. The generalized correlation of random vectors 𝑥_𝑖 and 𝑥_𝑗 is defined as 𝜌𝑥_𝑖,𝑥_𝑗=𝑥^𝑇_𝑖𝑀𝑥_𝑗𝑥^𝑇_𝑖𝑀𝑥_𝑖𝑥^𝑇_𝑗𝑀𝑥_𝑗, (14) where 𝑀∈𝑅^𝐷×𝐷 is a parameter matrix and symmetric positive semidefinite; for example, 𝑀⪰0.
So, in the paper, let 𝑀=𝑊𝑊^𝑇, where 𝑊=[𝑤₁,𝑤₂,…,𝑤_𝑑]∈𝑅^𝐷×𝑑 and 𝑑 is an alternative parameter. Generally, matrix 𝑀 parameterizes a family of the correlations on the vector space 𝑅^𝐷. Specifically, when 𝑀 is an identity matrix 𝐼_𝐷×𝐷, the generalized correlation in (14) becomes the standard correlation.
This type of correlation measure assigns different importance on series of features rather than equally processing as standard correlation coefficient does. It enhances the flexibility of the similarity measure. The parameter matrix 𝑀 could be adaptive for sample data.
Equation (14) can be modified to its equivalent form as 𝜌𝑥_𝑖,𝑥_𝑗=𝑊,𝑊tr^𝑇𝑥^𝑇_𝑖𝑥_𝑗𝑊tr𝑊^𝑇𝑥^𝑇_𝑖𝑥_𝑖𝑊tr𝑊^𝑇𝑥^𝑇_𝑗𝑥_𝑗𝑊, (15) where tr(⋅) denotes the trace of a matrix.
Substitute (15) into the optimization problem (12) and then we obtain the objective function as follows: 𝐿(𝑊)=_𝑖≠𝑗tr𝑊^𝑇𝑥_𝑖𝑥^𝑇_𝑗𝑊tr𝑊^𝑇𝑥_𝑖𝑥^𝑇_𝑖𝑊tr𝑊^𝑇𝑥_𝑗𝑥_𝑗𝑊⋅𝛿^𝐼_𝑖𝑗−𝛿^𝑃_𝑖𝑗. (16)

4.2. Intrinsic Graph and Penalty Graph

Intrinsic graph 𝐺^𝐼 and penalty graph 𝐺^𝑃 are both structural representations based on pairwise object comparisons. For such a representation, class overlap does not exist if the objects can be unambiguously labeled; there are no real-world objects in the application which belong to more than one class. Moreover, this structural representation can utilize the prior knowledge or supervised information in an alternative way, which will be discussed later.

4.2.1. Global Constraints

For 𝐺^𝐼, the node 𝑥_𝑖 and the node 𝑥_𝑗 are connected by an edge if 𝑥_𝑖 and 𝑥_𝑗 belong to the same class, otherwise not connected. For 𝐺^𝑃, the edge between 𝑥_𝑖 and 𝑥_𝑗 is constructed if 𝑥_𝑖 and 𝑥_𝑗 belong to different classes, otherwise not constructed. In our experiments, the global scheme is adopted.

4.2.2. Local Constraints

For 𝐺^𝐼, only consider each pair of 𝑥_𝑖 and 𝑥_𝑗 from the same class. The node 𝑥_𝑖 and the node 𝑥_𝑗 are connected if 𝑥_𝑗 is among the most nearest 𝑘^𝐼 nodes from 𝑥_𝑖 or 𝑥_𝑗 is in the circle neighbor region 𝜀^𝐼_𝛾 of 𝑥_𝑖. This is based on neighbor Euclidean distance. For 𝐺^𝑃, only consider each pair of 𝑥_𝑖 and 𝑥_𝑗 from the different classes. The edge between 𝑥_𝑖 and 𝑥_𝑗 is constructed if 𝑥_𝑗 is among the most nearest 𝑘^𝑃 nodes from 𝑥_𝑖 or 𝑥_𝑗 is in the circle neighbor region 𝜀^𝑃_𝛾 of 𝑥_𝑖. Here, 𝑘^𝐼, 𝜀^𝐼_𝛾, 𝑘^𝑃, and 𝜀^𝑃_𝛾 are all alternative parameters. It is obvious that the local scheme is appropriate for unsupervised learning and semisupervised learning.

4.3. Affinity Weights

After determining the edge distribution of constraint graph, it needs to assign affinity weights to the edges. Suppose 𝑑(𝑥_𝑖,𝑥_𝑗)=(𝑥_𝑖−𝑥_𝑗)^𝑇(𝑥_𝑖−𝑥_𝑗) is the distance between 𝑥_𝑖 and 𝑥_𝑗. Following existing work, here we have two variations for weighting the edges: (1) Gaussian kernel or other kernels: 𝛿_𝑖𝑗=𝑒^{−𝑑2(𝑥𝑖,𝑥𝑗)/𝑡} and (2) simple-minded: 𝛿_𝑖𝑗=1, if 𝑖 and 𝑗 are connected; otherwise, 𝛿_𝑖𝑗=0. These two ways have respective advantages and shortages. Δ^𝐼 and Δ^𝑃 can choose different weighting schemes and parameters. It provides the flexibility in practical application. The latter scheme is adopted in our method.

4.4. Closed-Form Solution for CSML

Motivated by [40], in this section, we will give a closed-form solution for SML to avoid the iterative optimization over high-dimensional space. For the optimization problem (12), additionally, we introduce an orthogonal constraint; that is, 𝑤^𝑇_𝑖𝑤_𝑗=0, for all 𝑖≠𝑗. The problem (12) may be transformed into the following maximum optimization: max𝐽=_𝑖≠𝑗tr𝑊^𝑇𝑥_𝑖𝑥^𝑇_𝑗𝑊𝛿^{(𝐼−𝑃)}_𝑖𝑗s.t.tr𝑊^𝑇𝑥_𝑖𝑥^𝑇_𝑖𝑊𝑤=1,∀𝑖_𝑇𝑖𝑤_𝑗=0,∀𝑖≠𝑗, (17) where 𝛿^{(𝐼−𝑃)}_𝑖𝑗 is short for 𝛿^𝐼_𝑖𝑗−𝛿^𝑃_𝑖𝑗. For simplicity, introduce the matrix notation 𝑆_𝛿=_𝑖≠𝑗𝛿^{(𝐼−𝑃)}_𝑖𝑗𝑥_𝑖𝑥^𝑇_𝑗. (18)

In fact, 𝑆_𝛿 is a weighted sum of covariance matrices of sample data. Next, 𝑤_𝑖 will be computed, respectively. To obtain the best discriminant vector 𝑤₁, we introduce the following Lagrange function with multipliers 𝜆_𝑖: 𝐽_𝐿1=𝑤^𝑇₁𝑆_𝛿𝑤₁−𝜆₁𝑤^𝑇₁𝑥₁𝑥^𝑇₁𝑤₁−1−⋯−𝜆_𝑛𝑤^𝑇₁𝑥_𝑛𝑥^𝑇_𝑛𝑤₁−1. (19) Considering 𝜆₁=𝜆₂=⋯=𝜆_𝑛=𝜆, compute the partial derivative of 𝐽_𝐿1 with respect to 𝑤₁ and set it to zero; then 𝑆_𝛿𝑤₁=𝜆𝑆_𝑒𝑤₁, (20) where 𝑆_𝑒=^𝑚_𝑖=1𝑥_𝑖𝑥^𝑇_𝑖. (21) Here, 𝑤₁ is the eigenvector of 𝑆_𝑒⁻¹𝑆_𝛿 associated with the largest eigenvalue.

To obtain other 𝑤_𝑖, we introduce the following Lagrange function with multipliers 𝜆 and 𝜇_𝑖: 𝐽_𝐿=^𝑚_𝑙=1𝑤^𝑇_𝑙𝑆_𝛿𝑤_𝑙−𝜆^𝑚_𝑙=1𝑤^𝑇_𝑙𝑆_𝑒𝑤_𝑙−𝑛−𝜇₁𝑤^𝑇_𝑚𝑤₁−𝜇₂𝑤^𝑇_𝑚𝑤₂−⋯−𝜇_𝑚−1𝑤^𝑇_𝑚𝑤_𝑚−1. (22)𝑤_𝑚 can be obtained by maximizing the above Lagrange function. As the above process, compute the partial derivative of 𝐽_𝐿 with respect to 𝑤_𝑚 and set it to zero: 𝜕𝐽_𝐿𝜕𝑤_𝑚=2𝑆_𝛿𝑤_𝑚−2𝜆𝑆_𝑒𝑤_𝑚−𝜇₁𝑤₁−𝜇₂𝑤₂−⋯−𝜇_𝑚−1𝑤_𝑚−1=0. (23)

Multiply the two sides of (23) by 𝑤^𝑇_𝑚; then 𝑤𝜆=^𝑇𝑚𝑆_𝛿𝑤_𝑚𝑤^𝑇_𝑚𝑆_𝑒𝑤_𝑚. (24) Thus 𝜆 represents the expression to be maximized. Considering (23), multiply its two sides successively by 𝑤^𝑇₁𝑆_𝑒⁻¹,…,𝑤^𝑇_𝑚𝑆_𝑒⁻¹, and then obtain 𝑚−1 equations: 𝜇₁𝑤^𝑇₁𝑆_𝑒⁻¹𝑤₁+⋯+𝜇_𝑚−1𝑤^𝑇₁𝑆_𝑒⁻¹𝑤_𝑚−1=2𝑤^𝑇₁𝑆_𝑒⁻¹𝑆_𝛿𝑤_𝑚,𝜇₁𝑤^𝑇₂𝑆_𝑒⁻¹𝑤₁+⋯+𝜇_𝑚−1𝑤^𝑇₂𝑆_𝑒⁻¹𝑤_𝑚−1=2𝑤^𝑇₂𝑆_𝑒⁻¹𝑆_𝛿𝑤_𝑚,⋮𝜇₁𝑤^𝑇_𝑚−1𝑆_𝑒⁻¹𝑤₁+⋯+𝜇_𝑚−1𝑤^𝑇_𝑚−1𝑆_𝑒⁻¹𝑤_𝑚−1=2𝑤^𝑇_𝑚−1𝑆_𝑒⁻¹𝑆_𝛿𝑤_𝑚. (25)

If we use matrix notations, 𝜇^𝑚−1=𝜇₁,𝜇₂,…,𝜇_𝑚−1^𝑇,𝑊^𝑚−1=𝑤₁,𝑤₂,…,𝑤_𝑚−1,𝐷^𝑚−1=𝐷^𝑚−1_𝑖𝑗=𝑊^𝑚−1𝑇𝑆_𝑒⁻¹𝑊^𝑚−1,𝐷^𝑚−1_𝑖𝑗=𝑤^𝑇_𝑖𝑆_𝑒⁻¹𝑤_𝑗. (26) The previous set of (𝑚−1) equations can be represented in a single matrix relationship: 𝐷^𝑚−1𝜇^𝑚−1=2𝑊^−1𝑇𝑆_𝑒⁻¹𝑆_𝛿𝑤_𝑚 (27) or in another form 𝜇^𝑚−1=2𝐷^𝑚−1−1𝑊^−1𝑇𝑆_𝑒⁻¹𝑆_𝛿𝑤_𝑚. (28) Let us multiply the two sides of (23) by 𝑆_𝑒⁻¹: 2𝑆_𝑒⁻¹𝑆_𝛿𝑤_𝑚−2𝜆𝑤_𝑚−𝜇₁𝑆_𝑒⁻¹𝑤₁−𝜇₂𝑆_𝑒⁻¹𝑤₂−⋯−𝜇_𝑚−1𝑆_𝑒⁻¹𝑤_𝑚−1=0. (29) This can be expressed using matrix notation as 2𝑆_𝑒⁻¹𝑆_𝛿𝑤_𝑚−2𝜆𝑤_𝑚−𝑆_𝑒⁻¹𝑊^𝑚−1𝑇𝜇^𝑚−1=0. (30) Including (28), we have 𝐿𝑤_𝑚=𝜆𝑤_𝑚, (31) where 𝐿=(𝐼−𝑆_𝑒⁻¹[𝑊^𝑚−1]^𝑇[𝐷^𝑚−1]⁻¹𝑊^𝑚−1)𝑆_𝑒⁻¹𝑆_𝛿. Considering 𝜆 as the criterion to be maximized, 𝑤_𝑚 is the eigenvector of 𝐿 and is associated with the largest eigenvalue of 𝐿.

4.5. Singularity of 𝑆_𝑒

To model the similarity measure, it only needs to obtain the 𝑑 largest eigenvalues of 𝐿 to constitute 𝑊=[𝑤₁,…,𝑤_𝑑]. However, involving with the inverse of 𝑆_𝑒, it cannot be applied when 𝑆_𝑒 is singular due to the small sample size problem. The small sample size problem occurs frequently in practice. In many applications, the dimensionality of the sample features is extraordinarily high while the number of samples is much small in comparison. When the number of samples is smaller than that of features, the small sample size problem occurs, for example, face recognition, text document classification, image retrieval, and cancer classification with gene expression profiling. The dimensionality of input space is high while the sample is often lacking. To handle this problem, the direct method is to replace 𝑆_𝑒⁻¹ with the pseudoinverse matrix 𝑆^†_𝑒. However, it does not guarantee that graph-preserving criterion is still optimized by the largest eigenvectors involved with 𝑆^†_𝑒. Here, the problem is similar to that in LDA. For the singularity in LDA, there are several frequently used methods, which can be modified for SML. The common way is to add a singular value perturbation to 𝑆_𝑤 to make it nonsingular [41]. Null subspace method and direct LDA [42] are both well known. Another one is kernel Fisher’s discriminant (KFD), which is a nonlinear extension to LDA. Maximum margin criterion (MMC) [43] modified the criterion in the fraction form into a difference one, which avoids the small sample size. In this work, we first employ PCA to reduce the dimensionality of the feature space to 𝑛−1, where 𝑛 is the number of samples and then apply SML on the dimensionality-reduced subspace.

5. Kernel CSML

CSML is used to find a global linear transformation matrix although the graph with local constraints may capture local nonlinear properties. In many cases, kernel trick is an efficient technique to extend a linear method to its nonlinear version.

To perform our linear method in reproducing kernel Hilbert space (RKHS), we consider the problem in a feature space ℱ induced by a nonlinear mapping 𝜙∶𝑅^𝑛→ℱ. We can define a Mercer’s kernel function: 𝑘(𝑥,𝑦)=⟨𝜙(𝑥),𝜙(𝑦)⟩=𝜙^𝑇(𝑥)𝜙(𝑦), where 𝑘(⋅,⋅) is a positive semidefinite kernel.

In the feature space ℱ, the generalized correlation similarity measure has the form 𝜌𝜙𝑥_𝑖,𝜙𝑥_𝑗=𝜙,𝑊_𝑇𝑥_𝑖𝑊𝑊^𝑇𝜙𝑥_𝑗𝜙^𝑇𝑥_𝑖𝑊𝑊^𝑇𝜙𝑥_𝑖𝜙^𝑇𝑥_𝑖𝑊𝑊^𝑇𝜙𝑥_𝑗. (32) Of course, it has the equivalent form 𝜌𝜙𝑥_𝑖,𝜙𝑥_𝑗=𝑊,𝑊tr^𝑇𝜙𝑥_𝑖𝜙^𝑇𝑥_𝑗𝑊tr𝑊^𝑇𝜙𝑥_𝑖𝜙^𝑇𝑥_𝑖𝑊tr𝑊^𝑇𝜙𝑥_𝑗𝜙^𝑇𝑥_𝑗𝑊. (33) Since, in the feature space ℱ, 𝑊 lies in the linear combination of 𝜙(𝑥₁),𝜙(𝑥₂),…,𝜙(𝑥_𝑛), it can be defined as 𝑊=Φ(𝑋)𝛼, (34) where Φ(𝑋)=[𝜙(𝑥₁),𝜙(𝑥₂),…,𝜙(𝑥_𝑛)] represents the training data in feature space ℱ and 𝛼=[𝛼_𝑖𝑗]_𝑛×𝑑. Specially, 𝛼_𝑗=[𝛼_1𝑗,𝛼_2𝑗,…,𝛼_𝑛𝑗]^𝑇 and 𝑤_𝑗=∑^𝑛_𝑖=1𝛼_𝑖𝑗𝜙(𝑥_𝑖)=Φ(𝑋)𝛼_𝑗. Substitute (33) and (34) into (17) and obtain max𝐽=_𝑖≠𝑗tr𝛼^𝑇Φ(𝑋)^𝑇𝜙𝑥_𝑖𝜙𝑥_𝑗^𝑇Φ(𝑋)𝛼𝛿^{(𝐼−𝑃)}_𝑖𝑗s.t.tr𝛼^𝑇Φ(𝑋)^𝑇𝜙𝑥_𝑖𝜙𝑥_𝑖^𝑇Φ(𝑋)𝛼𝛼=1,∀𝑖_𝑖𝑇Φ(𝑋)^𝑇Φ(𝑋)𝛼_𝑗=0,∀𝑖≠𝑗. (35) Define kernel matrix as 𝐾=𝑘𝑥_𝑖,𝑥_𝑗𝑛𝑛=𝑘_{𝑖𝑗𝑛𝑛}. (36) Let 𝑆^𝑘_𝛿=_𝑖≠𝑗𝛿_𝑖𝑗Φ(𝑋)^𝑇𝜙𝑥_𝑖𝜙𝑥_𝑗^𝑇Φ(𝑋)𝑆=𝐾Δ𝐾,_𝑘𝑒=^𝑛_𝑖=1Φ(𝑋)^𝑇𝜙𝑥_𝑖𝜙𝑥_𝑖^𝑇Φ(𝑋)𝑆=𝐾𝐾,_𝑘=Φ(𝑋)^𝑇Φ(𝑋)=𝐾. (37) The following Lagrange function with multipliers 𝜆 and 𝜇_𝑖 is introduced: 𝐽_𝐿=^𝑑_𝑙=1𝛼^𝑇_𝑙𝑆^𝑘_𝛿𝛼_𝑙−𝜆^𝑑_𝑙=1𝛼^𝑇_𝑙𝑆^𝑘_𝑒𝛼_𝑙−𝑛−𝜇₁𝛼^𝑇_𝑚𝛼₁−𝜇₂𝛼^𝑇_𝑚𝛼₂−⋯−𝜇_𝑚−1𝛼^𝑇_𝑚𝛼_𝑚−1. (38) Comparing with the analysis of CSML, some notations are introduced: 𝜇^𝑚−1=𝜇₁,𝜇₂,…,𝜇_𝑚−1^𝑇,𝛼^𝑚−1=𝛼₁,𝛼₂,…,𝛼_𝑚−1,𝐵^𝑚−1=𝐵^𝑚−1_𝑖𝑗=𝛼^𝑚−1𝑇𝑆^𝑘_𝑒⁻¹𝑆^𝑘𝛼^𝑚−1,𝐵^𝑚−1_𝑖𝑗=𝛼^𝑇_𝑖𝑆^𝑘_𝑒⁻¹𝑆^𝑘𝛼_𝑗. (39)

Note that the above notations are a little different from (26). Similarly, the final solution is obtained: 𝛼₁ is the largest eigenvector of 𝑆^𝑘_𝑒⁻¹⋅𝑆^𝑘_𝛿 and 𝛼_𝑚 is the largest eigenvector of the matrix 𝐼−𝑆^𝑘_𝑒⁻¹𝑆^𝑘𝛼^𝑚−1𝐵^𝑚−1−1𝛼^𝑚−1𝑇𝑆^𝑘_𝑒⁻¹𝑆^𝑘_𝛿. (40)

Here, we note that the problem of the eigenvalue decomposition of (40) is ill-posed because the rank of the square matrix 𝑆^𝑘_𝑒 is less than or equal to 𝑛−1 and then 𝑆^𝑘_𝑒 is singular. To handle the singularity of 𝑆^𝑘_𝑒, we simply add a small positive perturbation to 𝑆^𝑘_𝑒, that is, replay 𝑆^𝑘_𝑒 by 𝑆^𝑘_𝑒, where 𝑆^𝑘_𝑒=𝑆^𝑘_𝑒+𝜇𝐼. (41)

We set 𝜇=10⁻³ in this work.

6. Discussion

6.1. The Trace-Ratio, Ratio-Trace, and Trace-Difference

It is known that the trace-ratio optimization problem is nonconvex and has no closed-form solution. CSML is the typical one of this type of problem. To solve such a problem, there have been some attempts. The most popular is to transform such problems into the ratio-trace problem. For (16), the corresponding ratio-trace form is 𝐿(𝑊)=_𝑖≠𝑗𝑊tr^𝑇𝑥_𝑖𝑥^𝑇_𝑗𝑊𝑊^𝑇𝑥_𝑖𝑥^𝑇_𝑖𝑊𝑊^𝑇𝑥_𝑗𝑥_𝑗𝑊𝛿^𝐼_𝑖𝑗−𝛿^𝑃_𝑖𝑗, (42) which can be approximately solved with the general eigenvalue decomposition (GEVD) method: 𝑆_𝛿𝑤_𝑘=𝜏_𝑘𝑆_𝑒𝑤_𝑘, (43) where 𝜏_𝑘 is the 𝑘th largest eigenvalue of the GEVD associating with the eigenvector 𝑤_𝑘 and 𝑤_𝑘 constitutes the 𝑘th column vector of the matrix 𝑊. Finally, 𝑀=𝑊𝑊^𝑇 and the measure is learned. It can be seen that it is a suboptimal solution of the optimal problem (17) proposed in this paper. As pointed in [44], despite the existence of a closed-form solution for ratio-trace optimization problem, its approximation may sacrifice the potential classification capability of the derived low-dimensional feature spaces and is unstable for supervised classification. Guo et al. [45] converted such trace-ratio problem to a trace-difference one. However, it is solved by the iterative algorithm. For the detailed analysis on these attempts, we will refer the readers to the prior work [44, 45].

In this work, an alternative approximate optimization problem and its solution are presented. The denominator of the original trace-ratio objective function is fixed and then the numerator is maximized alone. In fact, the problem (16) can be approximated to the trace-difference one as follows: 𝐿(𝑊)=tr𝑊^𝑇𝑆_𝛿−𝜆𝑆_𝑒𝑊, (44) which is the same as the objective function in [45]. However, the following operations of CSML are very different from those in [45]. In CSML it just involves the eigenvalue decomposition, which is more simple and comprehensible.

6.2. Computational Complexity

The computational cost of CSML mainly comes from two parts. The first part is graph construction, that is, connecting each sample with its nearest neighbors, and its computational cost is 𝑂(𝑛²𝐷). The next part is the matrix eigenvalue decomposition, and its computational cost is 𝑂(𝑑𝐷³). So the overall cost is 𝑂(𝑑𝐷³+𝑛²𝐷). For comparison, Table 1 illustrates the computational costs of several distance metric learning and dimensionality reductions related to CSML, where 𝑇 is the number of iterations. We can see that Xing’s method is most expensive on computational cost. Our approaches CSML and KCSML are both more efficient than other several related algorithms.

Table 1 

Computational costs of related algorithms.

7. Experiments

To evaluate proposed algorithms CSML and KCSML, in this section, we perform several image classification experiments on diverse databases and compare them with another popular related work. These comparable methods include principal component analysis (PCA), random subspace two-dimensional PCA (RS-2DPCA), linear discriminant analysis (LDA), local preserving projection (LPP), marginal fisher analysis (MFA) [27], correlation embedding analysis (CEA), correlation discriminant analysis (CDA), improved similarity measure-based graph embedding (ISM-GE) [46], and maximal similarity embedding (MSE) [47]. PCA is taken as a baseline method. RS-2DPCA stands for the state of the art of unsupervised dimensionality reduction technique. LDA is a basic supervised discriminant technique. LPP and MFA stand for the state of the art of dimensionality reduction technique. CEA and CDA use standard correlation as their measure and closest to our method. Particularly, CEA is also designed in the graph-preserving framework. ISM-GE and MSE are both the most recent achievements of embedding learning based on the correlation metric, which are close to our method. ISM-GE defines a new improved similarity measure by fusing the Euclidean metric and the correlation metric and then performs graph embedding learning with the new measure. MSE searches for global linear dimensional reduction directions which preserve the local pairwise correlation similarity.

Face recognition is the classical application of image classification, which depends critically on a measure. The face databases Yale [48] and CMU PIE [49] are adopted. The MNIST is a popular handwritten digits database. We choose a subset from it as our experimental database. The image samples from the three databases are shown in Figures 1, 2, and 3, respectively. All the methods in experiments use centering and normalization as their preprocessing. The final classification is based on the simple nearest neighbor (NN) classifier. In all experiments, Gaussian kernel is adopted and the kernel width is set to the standard variance 𝜎=sqrt(∑^𝑛_𝑗=1‖𝑥_𝑗−𝑥‖²/𝑛). All of the results reported for those algorithms in comparison are from the best tuning of their parameters. 𝑑 in the table denotes the projection dimension when the best performance is got in PCA, RS-2DPCA, LDA, LPP, MFA, CDA, CEA, ISM-GE, and MSE. For CSML and KCSML, 𝑑 denotes the number of largest eigenvectors to constitute the parameter matrix 𝑊 for CSML and the matrix 𝛼 for KCSML, respectively.

Figure 1 

Samples from the Yale database.

Figure 2 

Samples from the CMU PIE database.

Figure 3 

Samples from the MNIST database.

7.1. Classification on the Yale Database

The Yale Face Database contains 165 grayscale images of 15 individuals. There are 11 images per subject, varying on facial expression and configuration: center-light, w/glasses, happy, left-light, w/no glasses, normal, right-light, sad, sleepy, surprised, and wink. The images are cropped and resized into 32 × 32 pixels. The feature of each image is represented by a 1,024-dimensional column vector. A random subset with 𝑝 images per individual is taken with labels to form the training set, where 𝑝=2,3,4,5,6,7. The rest of the database is considered to be the testing set. For each given 𝑝, 50 randomly splits are constituted. The results reported in Table 2 are the average values for 50 splits.

Table 2 

Classification performance comparison on the Yale database.

From comparisons in Table 2, we can observe that all the supervised methods outperform the unsupervised method PCA. It is easy to understand it since more class label information is introduced. We also see that CSML and KCSML both outperform the other competitive methods under all configurations, particularly, no matter being with sufficient or insufficient quantity of sample data. It confirms that the proposed generalized correlation similarity measure can effectively capture the intrinsic affinity structure of the data. The more experimental results in [21] show that PCA, LDA, and LPP perform better based on the correlation NN classifier. In our experiments, the correlation based methods (CDA, CEA, CSML, and KCSML) outperform the other methods based on Euclidean distance. In most cases, the kernel extension of CSML is better than its original version. From these results, it is obvious that the similarity measure is more effective in recognition tasks than Euclidean distance.

7.2. Classification on the CMU PIE Database

The CMU PIE database contains 41,368 images of 68 people, each person under different poses, illumination conditions, and expressions. We select a subset, which contains images under five near frontal poses, different illuminations, and expressions. There are 170 images for each individual and 11,554 images in all. The images are cropped and resized to be 32×32 pixels. As processed in the former experiment, each image is unfolded as a column vector. A random subset with 𝑝=5,10,20,30 images per individual is taken to form the training samples. The results in Table 3 are also average results of 50 splits for each 𝑝.

Table 3 

Classification performance comparison on the CMU PIE database.

From Table 3, CSML and KCSML greatly outperform the other competitive methods. PCA still performs worst. The selected subset used in the experiment contains more than 10 thousand images. From the experimental results in Table 3, we can conclude that the proposed similarity measure is effective and reliable on large scale databases. It further demonstrates the ability of the generalized correlation measure to capture the intrinsic structure of high-dimensional data.

7.3. Classification on the MNIST Database

The MNIST database consists of 60,000 handwritten digit images from the larger database NIST. We select randomly 500 images for each digit and then 5000 images in total from MNIST, to constitute a smaller subset as our experimental database. The images have been normalized into 28×28 pixels. The feature of each digit is represented by a 784-dimensional vector. As processed in the former experiments, the subset with 𝑝=50,100,150,200,250,300 images per digit was taken to form the training sample set. And the all left images are taken as testing samples for each training subset. Also, for each 𝑝, 50 random splits are constituted. The results in Table 4 are also average results of 50 splits for each 𝑝.

Table 4 

Classification performance comparison on the MNIST database.

With the experimental results in Table 4, the similar conclusion can be obtained.

7.4. Effects of Parameter Selection

In our proposed algorithm, the 𝑘-nearest neighbor search is twice applied. The first one is used for affinity graph construction in terms of the local constraints. 𝑘^𝐼 for intrinsic graph and 𝑘^𝑃 for penalty graph can be different and chosen with empirical values. Here, we assume 𝑘^𝐼=𝑘^𝑃=𝑘₁ to simplify the analysis. In the above experiments, we adopt the global scheme to construct pairwise affinity graphs (intrinsic graph and penalty graph) avoiding tedious tuning work. The other one, denoted by 𝑘₂-NN, is used as the final classifier. For fairness, in the above experiments, all the compared methods uniformly use the same simple 1-nearest neighbor as the final classifier. In this subsection, to show more details of our proposed algorithm, we analyze the effects of these two types of parameters on the recognition performance.

Figure 4 shows the error rate variations of CSML and KCSML with different 𝑘₁ on the three databases. The corresponding 𝑑’s are set to be the selected best ones in the above corresponding experiments. We find that the recognition performances of our proposed methods have a similar trend, where the error rate becomes approximately stable when the 𝑘₁ are relatively large. This result confirms our intuition that the larger 𝑘₁ covers the more constraints which are beneficial to the description of embedded relations. So, in practice, we suggest to choose a relatively larger 𝑘₁, which is also the reason why we choose the global scheme for affinity graphs construction in the above comparison experiments.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 4 

The behavior of the proposed methods under various 𝑘1. (a) CSML on Yale database, (b) KCSML on Yale database, (c) CSML on CMU PIE database, (d) KCSML on CMU PIE database, (e) CSML on MNIST database, and (f) KCSML on MNIST database.

Figure 5 shows the best dimension number 𝑑’s under different 𝑘₂ on the Yale database with “7 Train,” the CMU PIE with “30 Train,” and the MNIST with “300 Train.” We find that the 𝑑’s have a very small variation with the changing of the 𝑘₂; that is, we can choose a similar 𝑑 for different 𝑘₂. This result suggests that, for a given dataset, its intrinsic dimension number is determined no matter which classifier is selected. However, this has no benefit to parameter selection in practice. Generally, the parameter 𝑑 is set through time-consuming cross-validation tests with empirical experiences. In the above experiments, all the parameters 𝑑’s are chosen by the threefold cross-validation tests in the empirical value ranges.

(a)

(b)

(c)

Figure 5 

The best dimension number 𝑑 for various 𝑘2: (a) on Yale database, (b) on CMU PIE database, and (c) on MNIST database.

7.5. Executive Time

In our experiments, we also consider the comparison of computational efficiency of these algorithms. The CPU times of these methods are executed for fifty runs on Yale with 𝑝=7, CMU with 𝑝=30, and MNIST with 𝑝=300. The result in log scale is summarized in Figure 6. It shows that the executive times of CSML and KCSML are closest to each other and both are far less than those of CEA and CDA, which have comparable recognition rates with our presented methods. It agrees with the theoretical analysis result in Table 1. We could conclude that our presented methods are more efficient than CEA and CDA.

Figure 6 

Comparison on CPU time in log scale.

8. Conclusion

In this paper, we have presented a general framework for similarity measure learning (SML). The proposed generalized correlation 𝜌 improves the flexibility of standard correlation. Based on the generalized correlation, a specific algorithm of SML, called CSML, and its kernel extension KCSML are proposed. Their objective functions are in trace-ratio form, which have no closed-form optimal solution. We transform the two objective functions to their approximate optimization problems and give their closed-form solutions. The experiments on face recognition database and handwritten digits database indicate that the proposed similarity measure is effective to capture the intrinsic affinity structure of high-dimensional data. Because of the flexibility of this general framework, the CSML can also be modified to the semisupervised version under local or global constraint information, which is not contained in this paper. Other experiments critically depending on the adopted measure, such as clustering and image retrieval, will be performed as the future work.

Conflict of Interests

The authors declare that there is no conflict of interests regarding the publication of this paper.

Acknowledgments

This work was supported by the Research Grants MYRG205(Y1-L4)-FST11-TYY and MYRG187(Y1-L3)-FST11-TYY and Chair Professor Grants RDG009/FST-TYY of University of Macau as well as Macau FDC Grants T-100-2012-A3 and 026-2013-A. This research project was also supported by the National Natural Science Foundations of China (61273244, 60873092, 90820306, and 61100114) and the Fundamental Research Funds for the Central Universities (CDJXS10182216).