Fuzzy One-Class Classification Model Using Contamination Neighborhoods

Abstract

A fuzzy classification model is studied in the paper. It is based on the contaminated (robust) model which produces fuzzy expected risk measures characterizing classification errors. Optimal classification parameters of the models are derived by minimizing the fuzzy expected risk. It is shown that an algorithm for computing the classification parameters is reduced to a set of standard support vector machine tasks with weighted data points. Experimental results with synthetic data illustrate the proposed fuzzy model.

1. Introduction

A main goal of the statistical machine learning is to predict an unobserved output value 𝑦 based on an observed input vector 𝐱. A special very important problem of the statistical machine learning is the classification problem which can be regarded as a task of classifying some objects into classes in accordance with their properties or features. A lot of models have been constructed for solving the machine learning problems in last decades. However, a large part of models is based on restrictive assumptions, for instance, the large amount of training data, the known type of the noise probability distribution, point-valued observations, and so forth. At the same time, real applications cannot satisfy all these assumptions or their part due to several reasons. In order to relax some of the restrictive assumptions, many methods have been proposed. One of the directions for developing the corresponding methods is the fuzzy classification which applies the main ideas of fuzzy set theory to various classification problems.

Most fuzzy classification models can conditionally be divided into three groups. Two groups suppose that there are precise observations, but fuzzy sets are used to take into account different contributions of the observations or in order to take into account imprecision of results, for instance, imprecision of separating surfaces. According to models of the first group with precise observations, a membership value or a membership function is assigned to every point from the training set in accordance with some rules [1–5]. As a result, different input points can make different contributions to the learning of decision surface. Then the initial classification problem is solved as a weighted classification problem by means of known methods, for example, by means of the support vector machine (SVM) approach proposed by Vapnik [6]. According to models of the second group, a fuzzy separating surface, for instance, a fuzzy hyperplane in the feature space, is constructed [7, 8]. In other words, a result of the classification problem solution is a set of hyperplanes with a membership function derived from the model. The third group of the models supposes that learning observations are interval-valued or fuzzy-valued themselves [9–11] due to imperfection of measurement tools or imprecision of expert information if used as data. It should be noted that methods based on fuzzy sets can be used for generalizing the two-class classification into multiclass classification. In particular, Wilk and Wozniak [12] proposed the corresponding method by exploiting a fuzzy inference system.

The main difficulty of most fuzzy models is how to determine the membership functions for points from the training set. Therefore, a fuzzy classification model is proposed in the given paper which constructs the membership function on the basis of available statistical data by using an extension of the well-known 𝜀-contamination neighborhood or 𝜀-contaminated (robust) models [13]. Main ideas underlying the proposed model are the following.

The empirical probability distribution accepted for deriving the empirical expected risk (a measure of the classification error) and used in the standard SVM is replaced by a set of probability distributions, which is produced by applying the 𝜀-contaminated model. At that, it is assumed that the contaminating probability distribution can be arbitrary. As a result, we could construct an imprecise classification model for a predefined value 𝜀. However, the main difficulty of using the 𝜀-contaminated model is that the value 𝜀 is unknown, and there are no rules or methods for its determination. Therefore, the next idea is to use all values 𝜀 and to construct a model which covers all values 𝜀.

Note that the produced set of probability distribution on the basis of the 𝜀-contaminated model for a fixed 𝜀 is convex. This implies that the expected risk as an expectation of a loss function lies in an interval with some lower and upper bounds. Moreover, we get a set of nested intervals for different values of 𝜀 ranging from 0 to 1, which can be regarded as a fuzzy set. This implies that we obtain the fuzzy expected risk parametrized by classification parameters or by parameters of a separating surface. The next task is to find the classification parameters which minimize the fuzzy expected risk. This task is solved in the framework of SVM by taking a special ranking index for ranging the fuzzy expected risk measures. The fuzziness of the expected risk measure strongly depends on the number of points in the training set. It is assumed that the fuzziness increases with decrease of the number of data points.

It can be seen from the previous idea that the proposed model encompasses peculiarities of the first and the second group of models.

In order to simplify the description of the proposed classification model, we consider the one-class classification model or the well-known novelty detection model [14–17]. Moreover, we restrict ourselves by studying a model proposed by Schölkopf et al. [16, 18]. Nevertheless, this model can easily be extended in the case of binary classification models. It is important to show main principles for constructing the fuzzy classification model in the paper.

The paper is organized as follows. Section 2 presents the standard one-class classification problem proposed by [16, 18]. The 𝜀-contaminated robust model and its peculiarities are considered in Section 3. A set of probability distributions produced by the model and the fuzzy expected risk measure are studied in the same section. The SVM approach for computing the optimal classification parameters is provided in Section 4. In this section, one of the possible ways for decomposing the quadratic programming problem is considered. Numerical experiments with synthetic and some real data illustrating accuracy of the proposed model are provided in Section 5. In Section 6, concluding remarks are given.

2. One-Class Classification

Suppose we have unlabeled training data 𝐱₁,...,𝐱_𝑛⊂𝒳, where 𝑛 is the number of observations, 𝒳 is some set; for instance, it is a compact subset of ℝ^𝑚. According to papers [16, 18], a well-known one-class classification (novelty detection) model aims to construct a function 𝑓 which takes the value +1 in a “small” region capturing most of the data points and −1 elsewhere. It can be done by mapping the data into the feature space corresponding to the kernel and by separating them from the origin with maximum margin.

Let 𝜙 be a feature map 𝒳→𝐺 such that the data points are mapped into an alternative higher-dimensional feature space 𝐺. In other words, this is a map into an inner product space 𝐺 such that the inner product in the image of 𝜙 can be computed by evaluating some simple kernel 𝐾(𝐱,𝐲)=(𝜙(𝐱),𝜙(𝐲)), such as the Gaussian kernel: 𝐾()−‖‖𝐱,𝐲=exp𝐱−𝐲²𝜎². (1)𝜎 is the kernel parameter determining the geometrical structure of the mapped samples in the kernel space.

The aim of classification is to find a hyperplane 𝑓(𝐱,𝐰)=⟨𝐰,𝜙(𝐱_𝑖)⟩−𝜌=0 that separates the data from the origin with maximal margin. We use the parameter 𝜈∈[0;1] which is analogous to 𝜈 used for the 𝜈-SVM [19]. It denotes the fraction of input data for which ⟨𝐰,𝜙(𝐱_𝑖)⟩≤𝜌.

To find the optimal parameters 𝐰 and 𝜌, the following quadratic program has to be solved: min_{𝑤,𝜉,𝜌}12‖𝐰‖²+1𝜈𝑛^𝑛_𝑖=1𝜉_𝑖−𝜌, (2) subject to 𝐰,𝜙𝐱_𝑖≥𝜌−𝜉_𝑖,𝜉_𝑖≥0,𝑖=1,…,𝑛. (3)

Slack variables 𝜉_𝑖 are used to allow points to violate margin constraints. Using multipliers 𝛼_𝑖,𝛽_𝑖≥0, we introduce a Lagrangian: 𝐿=1𝐰,𝜉,𝜌,𝛼,𝛽2‖𝐰‖²+1𝜈𝑛^𝑛_𝑖=1𝜉_𝑖−−𝜌_𝑛𝑖=1𝛼_𝑖𝐰,𝜙𝐱_𝑖−𝜌+𝜉_𝑖−^𝑛_𝑖=1𝛽_𝑖𝜉_𝑖. (4)

It is shown in [19] that the dual problem is of the form min_𝛼12^𝑛𝑛_𝑖=1𝑗=1𝛼_𝑖𝛼_𝑗𝐾𝐱_𝑖,𝐱_𝑗, (5) subject to 0≤𝛼_𝑖≤1,𝜈𝑛^𝑛_𝑖=1𝛼_𝑖=1. (6) The value of 𝜌 can be obtained as 𝜌=𝐰,𝜙𝐱_𝑗=^𝑛_𝑖=1𝛼_𝑖𝐾𝐱_𝑖,𝐱_𝑗. (7)

After substituting the obtained solution into the expression for the decision function 𝑓, we get 𝑓(𝐱,𝐰)=sgn^𝑛_𝑖=1𝛼_𝑖𝐾𝐱_𝑖,𝐱−𝜌. (8)

3. Robust Models and Fuzzy Expected Losses

3.1. Robust Models and the Expected Risk

Robust models have been exploited in classification problems due to the opportunity to avoid some strong assumptions underlying the standard classification models. As pointed out by Xu et al. [20], the use of robust optimization in classification is not new. There are a lot of published results providing various robust classification and regression models (see, e.g., [21–24]) in which box-type uncertainty sets are considered. One of the popular robust classification models is based on the assumption that inputs are subject to an additive noise, and every data point is only known to belong to the interior of an Euclidian ball. Another class of robust models is based on relaxing strong assumptions about a probability distribution of data points (see, e.g., [25]).

We consider a model which can be partially regarded as a special case of these models and is based on using the framework of 𝜀-contaminated (robust) models ([13]). They are constructed by eliciting a Bayesian prior distribution 𝑝=(𝑝₁,…,𝑝_𝑛) as an estimate of the true prior distribution. The 𝜀-contaminated model is a class of probabilities which for fixed 𝜀∈(0,1) and 𝑝_𝑖 is the set ℳ(𝜀)={(1−𝜀)𝑝_𝑖+𝜀𝑞_𝑖}, where 𝑞_𝑖 is arbitrary and 𝑞₁+⋯+𝑞_𝑛=1. The rate 𝜀 reflects the amount of uncertainty in 𝑝 [26]. In other words, we take an arbitrary probability distribution 𝑞=(𝑞₁,…,𝑞_𝑛) from the unit simplex denoted by 𝑆(1,𝑛). According to these models, for 0<𝜀<1, ℳ(𝜀) is the set of all probabilities with the lower bound (1−𝜀}𝑝_𝑖 and the upper bound (1−𝜀}𝑝_𝑖+𝜀. Of course, the assumption that 𝑞 is restricted by the unit simplex 𝑆(1,𝑛) is one of possible types of 𝜀-contaminated models. Generally, there are a lot of different assumptions which produce specific robust models.

Let us rewrite the problem in a general form of minimizing the expected risk [6] 𝑅𝐰,𝜌=_ℝ𝑚𝐿(𝐱)d𝐹₀(𝐱). (9) Here the loss function 𝐿(𝐱) is the hinge loss function which is represented as 𝐿(𝐱)=max0,𝜌−𝐰,𝜙(𝐱)−𝜌𝜈. (10)

The standard SVM technique is to assume that 𝐹₀ is empirical (nonparametric) probability distribution whose use leads to the empirical expected risk 𝑅_emp=1𝐰,𝜌𝑛^𝑛_𝑖=1𝐿𝐰,𝜙𝐱_𝑖−𝜌𝜈. (11)

The assumption of the empirical probability distribution means that every point 𝐱_𝑖 has the probability 𝑝_𝑖=1/𝑛. This is a too strong assumption when the number of points is not large. Its validity might give rise to doubt in this case. Therefore, in order to relax the strong condition for probabilities of points, we apply the 𝜀-contaminated model. According to the model, we replace the probability distribution 𝑝=(1/𝑛,…,1/𝑛) by the set of probability distributions ℳ(𝜀)={(1−𝜀)𝑛⁻¹+𝜀𝑞_𝑖}. In other words, there is an unknown precise “true” probability distribution in ℳ(𝜀), but we do not know it and only know that it belongs to the set ℳ(𝜀).

3.2. The Fuzzy Expected Risk

Let ℎ=(ℎ₁,…,ℎ_𝑛) be a probability distribution which belongs to the set ℳ(𝜀). Since the set ℳ(𝜀) is convex, then the values of the expected risk for every value 𝜀 are restricted by some lower and upper bounds such that every point of the interval of expected risk corresponds to a probability distribution from ℳ(𝜀). Note that ℳ(𝜀₁)⊆ℳ(𝜀₂) by 𝜀₁≤𝜀₂. This implies that there holds 𝑅_𝐿⋅,𝜀₁;𝑅_𝑈⋅,𝜀₁⊆𝑅_𝐿⋅,𝜀₂;𝑅_𝑈⋅,𝜀₂. (12)

If we consider all possible values of the parameter 𝜀, then we get the set of nested intervals [𝑅_𝐿(𝐰,𝜌,𝜀);𝑅_𝑈(𝐰,𝜌,𝜀)] with the parameter 𝜀. Moreover, the expected risk interval is reduced to a point by 𝜀=0, and the largest interval takes place by 𝜀=1. This set of intervals can be viewed as a fuzzy set 𝑅(𝐰,𝜌) of the expected utility with the membership function 𝜇. The similar idea to obtain fuzzy sets by means of the contamination robust model has been mentioned by Utkin and Zhuk [27].

The upper bound for the expected risk by fixed 𝜀 can be found as a solution to the following programming problem: 𝑅_𝑈𝐰,𝜌,𝜀=max_(𝜀)^𝑛_{ℎ∈ℳ𝑖=1}ℎ_𝑖𝐿𝐰,𝜙𝐱_𝑖−𝜌𝜈=max_(𝜀)^𝑛_{ℎ∈ℳ𝑖=1}(1−𝜀)𝑛⁻¹+𝜀𝑞_𝑖𝐿𝐰,𝜙𝐱_𝑖=()𝑅−𝜌𝜈1−𝜀_emp𝐰,𝜌+𝜀⋅max_𝑞^𝑛_𝑖=1𝑞_𝑖𝐿𝐰,𝜙𝐱_𝑖−𝜌𝜈, (13) subject to 0≤𝑞_𝑖≤1,𝑞₁+⋯+𝑞_𝑛=1. (14)

The obtained optimization problem is linear with optimization variables 𝑞₁,…,𝑞_𝑛, but the objective function depends on 𝐰. Therefore, it cannot be directly solved by well-known methods. In order to overcome this difficulty, note, however, that all points 𝑞 belong to the simplex 𝑆(1,𝑛) in a finite dimensional space. According to some general results from linear programming theory, an optimal solution to the previously mentioned problem is achieved at extreme points of the simplex, and the number of its extreme points is 𝑛. Extreme points of the simplex 𝑆(1,𝑛) are of the form (1,0,…,0),(0,1,…,0),…,(0,0,…,1). (15) This implies that there holds 𝑅_𝑈𝐰,𝜌,𝜀=(1−𝜀)𝑛^𝑛−1_𝑖=1𝐿𝐰,𝜙𝐱_𝑖+𝜀⋅max_{𝑖=1,…,𝑛}𝐿𝐰,𝜙𝐱_𝑖−𝜌𝜈. (16)

The same can be written for the lower bound: 𝑅_𝐿𝐰,𝜌,𝜀=(1−𝜀)𝑛^𝑛−1_𝑖=1𝐿𝐰,𝜙𝐱_𝑖+𝜀⋅min_{𝑖=1,…,𝑛}𝐿𝐰,𝜙𝐱_𝑖−𝜌𝜈. (17)

3.3. The Fuzzy Decision Problem and a Way for Its Solving

Now we can write a new criterion of decision making about optimal parameters 𝐰,𝜌. The parameters 𝐰₀,𝜌₀ are optimal iff for all 𝐰,𝜌, there holds 𝑅(𝐰₀,𝜌₀)≤𝑅(𝐰,𝜌). The next question is how to compare the fuzzy sets. This is one of the most controversial questions in fuzzy literature. Most ranking methods are based on transforming a fuzzy set into a real number called by the ranking index. Here we use the index proposed by [28], which can be written in terms of the considered decision problem as 𝑅𝐰,𝜌=¹₀𝜂⋅𝑅_𝐿𝐰,𝜌,𝜀+1−𝜂𝑅_𝑈𝐰,𝜌,𝜀d𝜇. (18) Here 𝜂∈[0,1] is a parameter of pessimism. It can be regarded as a caution parameter proposed by [29]. The caution parameter reflects the degree of ambiguity aversion. The more ambiguity averse the decision maker is, the higher is the influence of the lower interval limit of generalized expected utility. 𝜂=1 corresponds to strict ambiguity aversion; 𝜂=0 expresses maximal ambiguity seeking attitudes. It is assumed here that the variable 𝜇 is a function of 𝜀, for example, 𝜇=1−𝜀.

Then we write a new criterion of decision making. Parameters 𝐰₀,𝜌₀ are optimal iff for all 𝐰,𝜌, there holds 𝑅𝐰₀,𝜌₀≤𝑅𝐰,𝜌. (19)

It follows from the above that the optimal parameters 𝐰,𝜌 can be obtained by solving the following optimization problem: 𝑅𝐰₀,𝜌₀=min𝑅𝐰,𝜌. (20)

Denote 𝐴=∫¹₀𝜀d𝜇. The optimal expected risk is now of the form 𝑅𝐰_opt,𝜌_opt=min_𝐰,𝜌𝑅𝐰,𝜌, (21) where 𝑅𝐰,𝜌=1−𝐴𝑛^𝑛_𝑖=1𝐿𝐰,𝜙𝐱_𝑖−𝜌𝜈+𝐴𝜂⋅min_{𝑖=1,…,𝑛}𝐿𝐰,𝜙𝐱_𝑖+𝐴1−𝜂max_{𝑖=1,…,𝑛}𝐿𝐰,𝜙𝐱_𝑖. (22)

We can see that the previous objective function 𝑅(𝐰,𝜌) consists of two main parts. The first part is the modified empirical expected risk. The second part can be regarded as Hurwicz criterion with optimism parameter 1−𝜂, which is exploited in decision problems when we do not have information about states of nature. If 𝐴 takes values from 0 to 1, then the objective function is a convex combination of the Hurwicz criterion and the expected utility under condition that probabilities of states of nature are identical 1/𝑛. Moreover, 𝑅(𝐰,𝜌) can be also regarded as the objective function with respect to the Hodges-Lehmann criteria [30] by 𝜂=0. This means that we are not sure that the probability of every point is 1/𝑛, and this disbelief is compensated by a more guaranteed Hurwicz approach with some coefficient depending on the function 𝜇(𝜀). It is interesting that the fuzzy classification problem has transformed to the standard decision problem comprising several decision criteria.

Before solving the optimization problem for computing parameters 𝐰,𝜌, we have to define the function 𝜇(𝜀) which determines the membership function of fuzzy expected losses. As we have pointed out, the simplest way is to assume that 𝜇=1−𝜀. Then we get 𝐴=1−¹₀𝜇d𝜇=0.5. (23)

However, this way does not take into account the possible dependence of the fuzzy set (its fuzziness) on the number of points in the training set. We assume that the fuzziness has to increase with decrease of 𝑛.

Let us consider the meaning of 𝐴 in the objective function (22). If 𝐴=1, then the optimal parameters are defined only by two single points maximizing and minimizing 𝐿(𝐰,𝜙(𝐱_𝑖)). This is an extreme case when we suppose that the empirical probability distribution is totally wrong. This takes place when we have a small number of points in the training set. Another extreme case is 𝐴=0. This case corresponds to the standard approach in classification based on the empirical expected risk. It was shown by Vapnik [6] that the second case can be applied by large values of observations. Hence, we can state that 𝐴=0 by 𝑛→∞ and 𝐴→1 by small values of 𝑛.

One of the possible functions satisfying the above conditions for 𝐴 is 𝜇=1−(𝑛(1−𝜀))^𝑛 or 𝜀=1−(1/𝑛)𝜇^1/𝑛. Hence 1𝐴=𝑛¹₀𝜇^1/𝑛1d𝜇=.𝑛+1 (24)

The next task is to minimize the function 𝑅(𝐰,𝜌) over the parameters 𝐰 and 𝜌. This task will be solved in the framework of the SVM.

4. The SVM Approach

In order to use the SVM approach to the fuzzy classification problem, we use the hinge loss function (10). Let us add the standard Tikhonov regularization term (1/2)⟨𝐰,𝐰⟩ (this is the most popular penalty or smoothness term) [31] to the objective function (22). The smoothness (Tikhonov) term can be regarded as a constraint which enforces uniqueness by penalizing functions with wild oscillation and effectively restricting the space of admissible solutions (we refer to [32] for a detailed analysis of regularization methods). Moreover, we introduce the following optimization variables: 𝜉_𝑖=max0,𝜌−𝐰,𝜙𝐱_𝑖,𝐺=max_{𝑖=1,…,𝑛}𝜉_𝑖. (25)

This leads to the quadratic programming problem 𝑅𝐰_opt,𝜌_opt=12⟨𝐰,𝐰⟩+1−𝐴𝑛^𝑛_𝑖=1𝜉_𝑖+𝐴𝜂min_{𝑖=1,…,𝑛}𝜉_𝑖+𝐴1−𝜂𝐺−𝜌𝜈→min_{𝐰,𝜌,𝐺,𝜉𝑖}, (26) subject to 𝜉_𝑖≥𝜌−𝐰,𝜙(𝐱),𝜉_𝑖≥0,𝑖=1,…,𝑛,𝐺≥𝜉_𝑖,𝑖=1,…,𝑛. (27)

If we suppose that min_{𝑖=1,…,𝑛}𝜉_𝑖=𝜉_𝑘, then the objective function can be decomposed to 𝑛 objective functions of the form 𝑅_𝑘𝐰^(𝑘)_opt,𝜌^(𝑘)_opt=12⟨⟩+𝐰,𝐰1−𝐴𝑛^𝑛_𝑖=1𝜉_𝑖+𝐴𝜂𝜉_𝑘+𝐴1−𝜂𝐺−𝜌𝜈. (28)

The optimal parameters 𝐰_opt and 𝜌_opt correspond to the smallest value of 𝑅_𝑘, 𝑘=1,…,𝑛.

Instead of minimizing the primary objective function, a dual objective function, the so-called Lagrangian, can be formed of which the saddle point is the optimum. Moreover, if we minimize the primary objective function, the dual objective function denoting 𝐿_𝑘 has to by maximized. The Lagrangian is 𝐿_𝑘=12⟨𝐰,𝐰⟩+1−𝐴𝑛^𝑛_𝑖=1𝜉_𝑖+𝐴𝜂𝜉_𝑘−+𝐴1−𝜂𝐺−𝜌𝜈_𝑛𝑖=1𝜁_𝑖𝜉_𝑖−^𝑛_𝑖=1𝜑_𝑖𝜉_𝑖−𝜌+𝐰,𝜙𝐱_𝑖−^𝑛_𝑖=1𝜓_𝑖𝐺−𝜉_𝑖. (29) Here 𝜁_𝑖,𝜓_𝑖,𝜑_𝑖, 𝑖=1,…,𝑛 are Lagrange multipliers. Hence, the dual variables have to satisfy positivity constraints 𝜁_𝑖≥0,𝜑_𝑖≥0,𝜓_𝑖≥0 for all 𝑖=1,…,𝑛. The saddle point can be found by setting the derivatives equal to zero: 𝜕𝐿𝜕𝜌=−𝜐+^𝑛_𝑖=1𝜑_𝑖=0,𝜕𝐿−𝜕𝐺=𝐴1−𝜂^𝑛_𝑖=1𝜓_𝑖=0,𝜕𝐿𝜕𝜉_𝑖=1−𝐴𝑛+𝟏_𝑘(𝑖)𝐴𝜂−𝜁_𝑖−𝜑_𝑖+𝜓_𝑖=0,𝜕𝐿𝜕𝑤_𝑗=𝑤_𝑗−^𝑛_𝑖=1𝜑_𝑖𝜙𝑥_𝑗^(𝑖)=0,𝑗=1,…,𝑚. (30)(31)(32)(33)

Here 𝟏_𝑘(𝑖) is the indicator function taking the value 1 if 𝑖=𝑘. Using (30)–(33), we get the following dual optimization problem: 𝐿_𝑘1=−2^𝑛𝑛_𝑖=1𝑗=1𝜑_𝑖𝜑_𝑗𝐾𝐱_𝑖,𝐱_𝑗, (34) subject to 0≤𝜑_𝑖≤1−𝐴𝑛+𝟏_𝑘(𝑖)𝐴𝜂+𝜓_𝑖,𝑖=1,…,𝑛,_𝑛𝑖=1𝜑_𝑖=𝜈,^𝑛_𝑖=1𝜓_𝑖,𝜓=𝐴1−𝜂_𝑖≥0,𝑖=1,…,𝑛. (35)(36)

The function 𝑓(𝐱) can be rewritten in terms of Lagrange multipliers as 𝑓(𝐱,𝐰)=^𝑛_𝑖=1𝜑_𝑖𝐾𝐱_𝑖,𝐱−𝜌. (37)

Hence, we find the optimal value of 𝜌 by taking 𝑓(𝐱,𝐰)=0; that is, there holds 𝜌=^𝑛_𝑖=1𝜑_𝑖𝐾𝐱_𝑖,𝐱_𝑗. (38)

Let us write the Karush-Kuhn-Tucker complementarity conditions: 𝜑_𝑖𝜉_𝑖−𝜌+𝐰,𝜙𝐱_𝑖𝜓=0,_𝑖𝐺−𝜉_𝑖=0. (39)

It follows from the second condition that 𝐺=𝜉_𝑖 for a single value of 𝑖=𝑙 such that 𝑙=argmax_{𝑖=1,…,𝑛}𝜉_𝑖. Here we assume that 𝜉₁,…,𝜉_𝑛 do not coincide. Therefore, 𝜂_𝑖=0 for all 𝑖≠𝑙. Returning to the constraints (35)–(36), we get 𝜓_𝑙=𝐴(1−𝜂) and 0≤𝜑_𝑖≤1−𝐴𝑛+𝐴𝜂⋅𝟏_𝑘(𝑖)+𝐴1−𝜂⋅𝟏_𝑙(𝑖),𝑖=1,…,𝑛 (40)

It follows from the previous constraints that the optimization problem (34)–(36) can be decomposed into 𝑛 problems: 𝐿_𝑘,𝑙1=−2^𝑛𝑛_𝑖=1𝑗=1𝜑_𝑖𝜑_𝑗𝐾𝐱_𝑖,𝐱_𝑗, (41) subject to (40) and ∑^𝑛_𝑖=1𝜑_𝑖=𝜈.

The optimal values of 𝜑_𝑖, 𝑖=1,…,𝑛 correspond to the smallest value of objective function 𝐿_𝑘,𝑙, 𝑙=1,…,𝑛 and to the largest value of objective functions 𝐿_𝑘,𝑙, 𝑘=1,…,𝑛.

So, we have 𝑛² simple quadratic programming problems whose solution can be obtained by means of well-known methods and tools.

5. Experiments

We illustrate the method proposed in this paper via several examples; all computations have been performed using the statistical software R. We investigate the performance of the proposed method and compare it with the standard SVM approach by considering the accuracy (ACC), which is the proportion of correctly classified cases on a sample of data and is often used to quantify the predictive performance of classification methods. ACC is an estimate of a classifier's probability of a correct response, and it is an important statistical measure of the performance of a one-class classification test. In novelty detection, ACC is the sum of two accuracy measures: the normal accuracy rate which measures how well the algorithm recognizes new examples of the known examples and the novelty accuracy rate which does the same for examples of an unknown novel example. ACC can formally be written as 1ACC=𝑁^𝑁_𝑖=1𝐼𝑦_𝑖⋅𝑓𝐱_𝑖,𝐰≥0, (42) where 𝑦_𝑖 is the label of the 𝑖th test example 𝐱_𝑖.

We will denote the accuracy measure for the proposed model as ACC_fuzzy and for the standard SVM as ACC_st.

All the experiments use a standard Gaussian radial basis function (GRBF) kernel with the kernel parameter 𝜎=0.04. Different values for the parameter 𝜎 have been tested, choosing those leading to the best results.

We consider the performance of our method with synthetic data having two features 𝑥₁ and 𝑥₂. The training set consisting of two subsets is generated in accordance with the normal probability distributions such that 𝑁₁=(1−𝜀₀)𝑁 examples (the first subset) are generated with mean values 𝐦₁=(4,4) and 𝑁₂=𝜀₀𝑁 examples (the second subset) have mean values 𝐦₂=(12,12). The standard deviation is 𝑠=2 for both subsets and both features. Here 𝜀₀ is a portion of abnormal examples in training set. The parameters 𝜈 and 𝜀₀ are 0.3. The parameter 𝐴 is taken 1/(𝑛+1).

Figure 1 illustrates how the contours 𝑓(𝐱,𝐰)=0 depend on the parameter 𝜂 for the fuzzy model (thick curve) and for the standard SVM (dashed curve) by the very small number of points in the training set (𝑛=10). The parameter 𝜂 takes values 0, 0.5, 1. One can see from the pictures that the region bounded by the “fuzzy” contour decreases with 𝜂. The value 𝜂=0 provides the cautious strategy of decision making. It can be seen from the first picture that the region bounded by the “fuzzy” contour is larger than the same region corresponding to the standard model. The regions almost coincide for optimistic strategy when 𝜂=1 (see Figure 1(c)). Figures 2 and 3 illustrate similar dependencies by 𝑛=20 and 𝑛=40, respectively.

(a)

(b)

(c)

Figure 1 

The contours 𝑓(𝐱,𝐰)=0 by different values of 𝜂=0, 0.5, 1 and by 𝑛=10.

(a)

(b)

(c)

Figure 2 

The contours 𝑓(𝐱,𝐰)=0 by different values of 𝜂=0, 0.5, 1 and by 𝑛=20.

(a)

(b)

(c)

Figure 3 

The contours 𝑓(𝐱,𝐰)=0 by different values of 𝜂=0, 0.5, 1 and by 𝑛=40.

The accuracy measure for different conditions is shown in Table 1. It can be seen from the results given in the table that the accuracy measure of the fuzzy model is larger than the same measure of the standard SVM. Of course, this relation takes place when 𝑛 is rather small, and the contamination of the normal data has a large (in examples 0.3) ratio.

Table 1 

Accuracy measures for different 𝑛 and 𝜈.

As a further example we applied all analyzed models to the well-known “Iris” data set from the UCI Machine Learning Repository [33]. The data set contains 3 classes (Iris Setosa, Iris Versicolour, Iris Virginica) of 50 instances each. The number of features is 4 (sepal length in cm, sepal width in cm, petal length in cm, petal width in cm). We suppose that examples from the Iris Setosa class are abnormal. For the experiment, we randomly select 𝑛 points such that (1−𝜀₀)𝑛 points are taken from the set of positively labelled examples and 𝜀₀𝑛 points are from negatively labelled examples. Here 𝜀₀=50/150≃0.333. The parameters for modelling are 𝑣=0.333, 𝜎=0.001. It is investigated how the accuracy measures depend on the amount 𝑛 of training data. In particular, if we take 𝑛=50, then ACC_fuzzy=0.834, ACC_st=0.773. If we take 𝑛=80, then ACC_fuzzy=0.833, ACC_st=0.866. One can see that the fuzzy approach provides better numerical results in comparison with the standard approach when the number of examples 𝑛 is rather small.

6. Conclusion

In this paper, a fuzzy one-class classification model has been proposed, which is based on applying the 𝜀-contaminated model. The algorithm for computing the optimal parameters of classification is reduced to a finite number of standard SVM tasks with weighted data points, where the weights are assigned in accordance with a predefined rule derived from comparison of fuzzy expected risk measures. It is easy to be implemented with standard functions of the statistical software package R.

Experimental results with synthetic data and with the well-known “Iris” data set from the UCI Machine Learning Repository reported have illustrated that the proposed fuzzy model outperforms the standard approach by the small number of observations. Due to the proposed fuzzy model, we do not need to assign a certain value for the contamination parameter 𝜀. However, we have to define the function 𝜇(𝜀) and the parameter 𝜂.

It should be noted that the proposed model can easily be extended on the case of binary or multiclass classification. This is a direction for future work.

We have investigated only one fuzzy ranking index for comparison fuzzy numbers. However, there are many efficient indices whose application to the classification problems in the framework of the proposed model could give better accuracy of classification and better models. This is also another direction for future work.

Acknowledgment

The author would like to express his appreciation to the anonymous referees and the Editor of this Journal whose valuable comments have improved the paper.