An Application of Monte-Carlo-Based Sensitivity Analysis on the Overlap in Discriminant Analysis

Abstract

Discriminant analysis (DA) is used for the measurement of estimates of a discriminant function by minimizing their group misclassifications to predict group membership of newly sampled data. A major source of misclassification in DA is due to the overlapping of groups. The uncertainty in the input variables and model parameters needs to be properly characterized in decision making. This study combines DEA-DA with a sensitivity analysis approach to an assessment of the influence of banks’ variables on the overall variance in overlap in a DA in order to determine which variables are most significant. A Monte-Carlo-based sensitivity analysis is considered for computing the set of first-order sensitivity indices of the variables to estimate the contribution of each uncertain variable. The results show that the uncertainties in the loans granted and different deposit variables are more significant than uncertainties in other banks’ variables in decision making.

1. Introduction

The classification problem of assigning observations to one of different groups plays an important role in decision making. When observations are restricted to one of two groups, the Binary classification has wide applicability in business environments.

Discriminant analysis (DA) is a classification method that can distinguish the group membership of a new observation. A group of observations for which the memberships have already been identified is used for the estimation of a discriminant function by some criteria, such as the minimization of misclassification. A new sample is classified into one of the groups based on the gained results [1].

Mangasarian [2] identified that linear programming (LP) could be used to determine separating hyperplanes, namely, when two set of observations are linearly separable use linear discriminant function. Freed and Glover [3] and Hand [4] using objectives such as minimization of the sum of deviations (MSD) or maximization of the minimum deviation (MMD) of misclassified observations from the separating hyperplane, when sets of observations that are not necessarily linearly separable, proposed LP methods for generating linear discriminant function.

Then a model is based on the goal programming (GP) extension of LP by choosing different criteria, such as minimizing the maximum deviation, maximizing the minimum deviation, minimizing the sum of interior deviation, minimizing the sum of deviations, minimizing misclassified observations, minimizing external deviations, maximizing internal deviations, maximizing the ratio of internal to external, and hybrid models for which there are both advantages and deficiencies [5–9].

In DA, LP and other mathematical programming (MP) based approaches are non-parametric and more flexible than statistical methods [6, 10]. Retzlaff-Roberts [11, 12] and Tofallis [13] proposed the use of DEA-ratio model for DA. Sueyoshi [14] using a data envelopment analysis (DEA) additive model, described a goal programming formulation of DA in which the proposed model is more directly linked to minimizing the sum of deviations from the separating hyperplane; this method was named DEA-DA to distinguish it from other DA and DEA approaches. The original GP version of DEA-DA could not deal with negative data. Therefore, Sueyoshi [15] extended DEA-DA to overcome this deficiency. This approach was designed to minimize the total distance of misclassified observations and formulated by two-stage GP formulations. The number of misclassifications can, however, be considered as measures of misclassification, in which binary variables indicate whether observations are correctly or in correctly classified. Bajgier and Hill [16] proposed a Mixed Integer Programming (MIP) model that included the number of misclassifications in the objective function for the two-group discriminant problem. Gehrlein [17] and Wilson [18] introduced a MIP approach for minimizing the number of misclassified observations in multigroup problems. Chang and Kuo [19] proposed a procedure based on benchmarking model of DEA to solve the two-group problems. Sueyoshi [20] reformulated DEA-DA by MIP to minimize the total number of misclassified observations. When an overlap between two groups is not a serious problem, dropping the first stage of the two-stage MIP approach simplifies the estimation process [21].

Sensitivity analysis provides an understanding of how the model outputs is affected by changes in the inputs, therefore it can assist to increase the confidence in the model and its predictions. sensitivity analysis can use in deciding whether inputs estimates are sufficiently precise to give reliable predictions or we can find the model parameters that can be eliminated.

Two classes in sensitivity analysis have been distinguished [22].

Local Sensitivity Analysis
Studies how some small variations of inputs around a given value, change the value of the output. This approach is practical when the variation around a baseline of the input-output relationship to be assumed linear.

Global Sensitivity Analysis
Takes into account all the variation range of the inputs, and has for aim to apportion output uncertainty to inputs’ ones. It quantified the output uncertainty due to the uncertainty in the input parameters. Global sensitivity analysis apportions the output uncertainty to the uncertainty in the input factors, described typically by probability distribution functions that cover the factors’ ranges of existence.
Local methods are less helpful when sensitivity analysis is used to compare the effect of various factors on the output, as in this case the relative uncertainty of each input should be weighted. A global sensitivity analysis technique thus incorporates the influence of the whole range of variation and the form of the probability density function of the input. The variance-based methods can be considered as a quantitative method for global sensitivity analysis. In this Study, the Sobol’ decomposition in the framework of Monte Carlo simulations (MCS), [22], which is from the family of quantitative methods for global sensitivity analysis, is applied to study of the effect of the variability in DA due to the uncertainty in the variables. The results of the sensitivity analysis can determine which of the variables have a more dominant influence on the uncertainty in the model output.

This paper is organized as follows: Section 2 briefly introduces the DEA-DA model; Section 3 describes the sensitivity analysis based on a Monte-Carlo simulation. Section 4 contains an example and the conclusion is provided in Section 5.

2. Data Envelopment Analysis-Discriminant Analysis (DEA-DA)

The two-stage MIP approach [20] is used in this study to describe DEA-DA. We considered two groups (𝐺₁ and 𝐺₂) for which the sum of the two groups has 𝑛 observations (𝑗=1,…,𝑛). Each observation has 𝑚 independent factors (𝑖=1,…,𝑚), denoted by 𝑟_𝑖𝑗. It is necessary to identify the group membership of each observation before its computation. In the two stage approach, the computation process consists of classification and overlap identification, and handling overlap. The first stage is formulated as follows [20].

Stage 1. Minimize𝑠,Subjectto^𝑚_𝑖=1𝜆⁺_𝑖−𝜆⁻_𝑖𝑟_𝑖𝑗−𝑑+𝑠≥0,𝑗∈𝐺₁,_𝑚𝑖=1𝜆⁺_𝑖−𝜆⁻_𝑖𝑟_𝑖𝑗−𝑑−𝑠≤−𝜀,𝑗∈𝐺₂,_𝑚𝑖=1𝜆⁺_𝑖+𝜆⁻_𝑖𝜉=1,_+𝑖≥𝜆⁺_𝑖≥𝜀𝜉⁺_𝑖,𝜉⁻_𝑖≥𝜆⁻_𝑖≥𝜀𝜉⁻_𝑖𝜉,𝑖=1,…,𝑚,_+𝑖+𝜉⁻_𝑖≤1,𝑖=1,…,𝑙,_𝑚𝑖=1𝜉⁺_𝑖+𝜉⁻_𝑖=𝑙,𝑑,𝑠∶unrestricted,𝜉⁺_𝑖,𝜉⁻_𝑖∈{0,1}andallothervariables≥0. (2.1) Here “𝑑” indicates a discriminant score for group classification and “𝑠” indicates the size of an overlap between two groups.
Let 𝜆^∗_𝑖 (=𝜆_𝑖^+∗−𝜆_𝑖^−∗) and 𝑑^∗ and 𝑠^∗ be an optimal solution of the model (2.1). Then, the original data set (𝐺) is classified into the following subsets 𝐺=𝐺₁∪𝐺₂=𝐶₁∪𝑁₁∪𝐶₂∪𝑁₂, where 𝐶₁=𝑗∈𝐺₁^𝑚_𝑖=1𝜆^∗_𝑖𝑟_𝑖𝑗>𝑑^∗+𝑠^∗,𝐶₂=𝑗∈𝐺₂^𝑚_𝑖=1𝜆^∗_𝑖𝑟_𝑖𝑗<𝑑^∗−𝑠^∗,𝑁₁=𝐺₁−𝐶₁,𝑁₂=𝐺₂−𝐶₂. (2.2)
Then, we determine that observations in 𝐶₁ belong to 𝐺₁ and the observations of 𝐶₂ belong to 𝐺₂ because their location is identified from model (2.1). The two subsets 𝑁₁ and 𝑁₂ consist of the observations have not yet been classified in the first stage.

Stage 2. If 𝑠^∗≥0, then the existence of an overlap is identified in the fist stage. In this stage, we reclassify all of the observations belonging to the overlap (𝑁₁∪𝑁₂) because the group membership of these observations is still undetermined. The second stage is reformulated as follows [20]: Minimize_𝑗∈𝐷1𝑙_𝑗+𝑤_𝑗∈𝐷2𝑙_𝑗,Subjectto^𝑚_𝑖=1𝜆⁺_𝑖−𝜆⁻_𝑖𝑟_𝑖𝑗−𝑐+𝑀𝑙_𝑗≥0,𝑗∈𝑁₁,_𝑚𝑖=1𝜆⁺_𝑖−𝜆⁻_𝑖𝑟_𝑖𝑗−𝑐−𝑀𝑙_𝑗≤−𝜀,𝑗∈𝑁₂,_𝑚𝑖=1𝜆⁺_𝑖+𝜆⁻_𝑖𝜉=1,_+𝑖≥𝜆⁺_𝑖≥𝜀𝜉⁺_𝑖,𝜉⁻_𝑖≥𝜆⁻_𝑖≥𝜀𝜉⁻_𝑖𝜉,𝑖=1,…,𝑚,_+𝑖+𝜉⁻_𝑖𝜆≤1,𝑖=1,…,𝑚,_+𝑖+𝜆⁻_𝑖≥𝜀,forsome𝑖,𝑐∶unrestricted,𝜉⁺_𝑖,𝜉⁻_𝑖,𝑦_𝑗∈{0,1},andallothervariables≥0. (2.3)
Here, the binary variable (𝑙_𝑗) counts the number of observations classified incorrectly. The objective function minimizes the number of such misclassifications. The weight (𝑤) identifies the importance between 𝐺₁ and 𝐺₂ in terms of the number of observations. In the presented model (2.3), it is necessary to prescribe a large number (𝑀) and a small number (𝜀). The equation 𝜆⁺_𝑖+𝜆⁻_𝑖≥𝜀(forsome𝑖) indicates that some pairs avoid the occurrence of 𝜆⁺_𝑖=0 and 𝜆⁻_𝑖=0.
After gaining an optimal solution on 𝜆^∗_𝑖 and 𝑐^∗, the second stage classifies observations in the overlap as follows: if ∑^𝑚_𝑖=1𝜆^∗_𝑖𝑟_𝑖𝑗≥𝑐^∗, then the 𝑗th observation belongs to 𝐺₁, or if ∑^𝑚_𝑖=1𝜆^∗_𝑖𝑟_𝑖𝑗≤𝑐^∗−𝜀, then it belongs to 𝐺₂. Thus, all of the observations in 𝐺 are classified into 𝐺₁ or 𝐺₂ at the end of the second stage.

3. Sensitivity Analysis Based on Monte-Carlo Simulation (MCS)

Sensitivity analysis was created to deal simply with uncertainties in the input variables and model parameters [22]. The results of an sensitivity analysis can determine which of the input parameters have a more dominant influence on the uncertainty in the model output [23]. A variance-based sensitivity analysis, which addresses the inverse problem of attributing the output variance to uncertainty in the input, quantifies the contribution that each input factor makes to the variance in the output quantity of interest. A global sensitivity analysis of complex numerical models can be performed by calculating variance-based importance measures of the input variables, such as the Sobol’ indices. These indices are calculated by evaluating a multidimensional integral using a Monte-Carlo technique. This approach allows analyzing the influence of different variables and their subsets, the structure of 𝑓(𝑥), and so forth.

It is assumed that a mathematical model having 𝑛 input parameters gathered in an input vector with a joint probability density function (pdf) can be presented as a model function: 𝑠=𝑓(𝑟),𝑟=𝐫₁,𝐫₂,…,𝐫_𝑚∈𝑈^𝑛≡[0;1]^𝑛,that𝑟_𝑖𝑗∈𝑈^𝑛≡[0;1]^𝑛𝑖=1,…,𝑚;𝑗=1,…,𝑛, (3.1) where 𝐫_𝑖=(𝑟_1𝑖,𝑟_2𝑖,…,𝑟_𝑛𝑖)^𝑇. Because of the variables are affected by several kinds of heterogeneous uncertainties that reflect the imperfect knowledge of the system, it is assumed that input variables are independent and that the probability density function 𝑝(𝑟) is known, even if the 𝑟_𝑖𝑗 are not actually random variables.

The Sobol’ sensitivity method explores the multidimensional space of the unknown input parameters 𝑥 with a certain number of MC samples. The sensitivity indices are generated by a decomposition of the model function 𝑓 in an 𝑛-dimensional factor space into summands of increasing dimensionality [22]: 𝑓(𝑟)=𝑓𝐫₁,…,𝐫_𝑚=𝑓₀+^𝑚_{𝑖=1𝑞1<⋯<𝑞𝑖}𝑓_{𝑞1,…,𝑞𝑖}𝐫_𝑞1,𝐫_𝑞2,…,𝐫_𝑞𝑖. (3.2) where the constant 𝑓₀ is the mean value of the function, and the integral of each summand over any of its independent variables is zero. Due to this property, the summands are orthogonal to each other in the following form: _𝑈𝑛𝑓_{𝑖1,…,𝑖𝑡}𝑓_{𝑗1,…,𝑗𝑞}𝑑𝑟=0,𝑖₁,…,𝑖_𝑡≠𝑗₁,…,𝑗_𝑞. (3.3)

The sensitivity index, SI_𝑖, represents the fractional contribution of a given factor 𝐫_𝑖 to the variance in a given output variable, 𝑠. To calculate the sensitivity indices, the total variance, 𝑉_𝑠, in the model output, 𝑠, is apportioned to all of the input factors, 𝐫₁,…, 𝐫_𝑚, as follows: 𝑉_𝑠=_𝑈𝑛𝑓²(𝑟)𝑑𝑟−𝑓²₀. (3.4)

By integrating the square of (2.2) and with (2.3), it is possible to decompose the total variance (3.1) as follows [24]: 𝑉_𝑠=_𝑖𝑉_𝑖+_𝑖<ℎ𝑉_𝑖ℎ+_{𝑖<ℎ<𝑘}𝑉_𝑘+⋯+𝑉_12⋯𝑜, (3.5) where 𝑉_𝑖=𝑉[𝐸(𝑠|𝐫_𝑖=𝐫^∗_𝑖)], 𝑉_𝑖ℎ=𝑉𝐸𝑠𝐫_𝑖=𝐫^∗_𝑖,𝐫_ℎ=𝐫^∗_ℎ−𝑉_𝑖−𝑉_ℎ, (3.6) and so on. 𝑉[𝐸(𝑠∣𝐫_𝑖=𝐫^∗_𝑖)] is referred to as the variance of the conditional expectation and is the variance over all of the values of 𝐫^∗_𝑖 in the expectation of 𝑠 given that 𝐫_𝑖 has a fixed value??𝐫^∗_𝑖. This is an intuitive measure of the sensitivity of 𝑠 to a factor 𝐫_𝑖, as it measures the amount by which 𝐸(𝑠∣𝐫_𝑖=𝐫^∗_𝑖) varies with the value of 𝐫^∗_𝑖 whilst averaging over the 𝐫_ℎ’s, ℎ≠𝑖. Following the above definition for the partial variances, the sensitivity indices are defined as SI_𝑖=𝑉_𝑖𝑉_𝑠 (3.7)

Higher order indices can be calculated with a similar approach. With regard to (3.2), the decomposition of the sensitivity indices can be written in the following form: ^𝑚_𝑖=1SI_𝑖+_{1≤𝑖≤ℎ≤𝑚}SI_𝑖ℎ+⋯+SI_1,2,…,𝑚=1. (3.8)

The Sobol’ indices are usually computed with a MC simulation. The mean value and total and partial variance can be derived with 𝑁_sim samples in the following [22]: 𝑓₀=1𝑁_sim^𝑁sim_𝑝=1𝑓𝐫_𝑝,𝑉_𝑠=1𝑁_sim^𝑁sim_𝑝=1𝑓²𝐫_𝑝−𝑓₀²,𝑉_𝑖=1𝑁_sim^𝑁sim_𝑝=1𝑓²𝐫_()𝑝⁽¹⁾_∼𝑖,𝐫⁽¹⁾_𝑖𝑝𝑓𝐫⁽²⁾_{(∼𝑖)𝑝},𝐫⁽¹⁾_𝑖𝑝−𝑓₀². (3.9) In the later equations, 𝐫_𝑚 is a sampled variable in 𝑈^𝑛, and 𝑟_{(∼𝑖)𝑝}=𝑟_1𝑝,𝑟_2𝑝,…,𝑟_(−1)𝑝,𝑟_(𝑖+1)𝑝,…,𝑟_𝑚𝑝. (3.10) The superscripts (1) and (2) indicate that two different samples are generated and mixed.

4. Illustrative Examples

Classification methods are widely used in economic and finance. They are useful for classifying the sectors based upon their performance in different groups and predict the group memberships of new firms. Most of researchers used classification methods to classifying the firms based upon their performance assessment. DA is a classification method that is used in this study. The purpose of the first stage in DEA-DA is to determine whether there is an overlap between the two groups. The existence of an overlap is the main source of misclassification in DA. By identification of the overlap between two groups, it is possible to increase the number of observations classified correctly. If there is no overlap, any DA method may produce an almost perfect classification. However, if there is an overlap, an additional computation process is needed to deal with such an overlap [20]. So, there is a tradeoff between computational effort/time and a high level of classification capability.

Misclassification can result as a consequence of an intersection between two groups. Many researchers have proposed approaches that try to reveal the advantage of identifying the minimized overlap of two groups for risk management on the classification problem [19, 20, 25, 26]. Given the importance of the banking sector for, in general, the whole economy and, in particular, for the financial system, in this section, we present an application of the sensitivity analysis to overlaps, 𝑠, on data from a commercial bank of Iran. This assertion is illustrated numerically for bank branches that have more than 20 and 30 personnel, in two different examples.

If we wish to take into account the inherent randomness with respect to what the criteria might experience, we have to bring stochastic characterization into play. The stochastic efficiency assessment of banking branches normally requires performing a set of analyses on DMUs with a suit of variables as criteria.

At first, we use the additive model to discriminate banking branches. Most models need to examine both a DEA efficiency score and slacks or an efficiency score measured by them, which depends upon input-based or output-based measurement. The additive model [27] aggregates input-oriented and output-oriented measures to produce the efficiency score. Consequently, the efficiency status is more easily determined by the additive model than by the radial model.

In two different examples, the real data set consists of 78 and 18 banking branches. This study selects 31 and 8 branches as inefficient branches, and 47 and 10 branches as efficient branches, respectively, in example 1 and 2, documented in Tables 1 and 2. For determining the classifications based on the additive model, three variables of personnel, payable interest, and non-performing loans are considered as inputs. Nine variables of loans granted, long-term deposit, current deposit, non-benefit deposit, short-term deposit, received interest, and received fee are assumed as outputs.

Table 1 

Table 2 

Then, for sensitivity analysis in DEA-DA, each observation is modeled as random parameters as follows [28]: 𝑟_𝑖𝑗=𝜇_𝑟𝑖𝑗1+𝑉_𝑟𝑖𝑗𝛼_𝑙, (4.1) where 𝜇_𝑟𝑖𝑗 and 𝑉_𝑟𝑖𝑗 are the mean value and the coefficients of variation (COV) of the random parameters, respectively, and 𝛼_𝑙 is generated random parameter with a zero mean that is used in MC simulation. The determination of the bank branches’ parameters carries a high degree of uncertainty, and the specification of these parameters can involve a significant degree of expert judgment. Additionally, the COV of these variables plays an important role in the variation of the efficiency. Here, the COV of the all of the parameters are assumed to equal 0.05.

To compute the sensitivity indices, the Sobol’ sampling scheme has been used. Sequences of Sobol’ sampling vectors are essentially quasi-random sequences that are defined as sequences of points that have no intrinsic random properties. In this study, a sensitivity analysis is applied to assess the influence of the banks’ variables on the overlap.

After one hundred estimates with a sample size of 5000, convergence was seen in the first-order Sobol’ indices derived by a Sobol’ sampling of the uniform criteria spaces for different banks’ variables. The sensitivity indices, 𝑆_𝑖, are depicted in Figures 1 and 2. These figures present the comparison of the first-order indices of the banks’ variables. The total fraction of the variance captured by the first order functions is approximately 99%. This indicates that, for this problem, higher order contributions to the Sobol’ series are relatively small. The overall variance in banks’ efficiency is affected by the variances in each of the random variables. Figure 1 indicates that 54% and 33% of the overall variance in overlap in DEA-DA is attributable to the variance in loans granted and different deposits, respectively, while the personnel, received interest, fee, and non-performing loans variables have little effect. Also, Figure 2 indicates that uncertainties in loans granted and different deposits are the dominant variables in overlap in DEA-DA.

Figure 1 

The relative contributions of different variables on the overlap in DEA-DA in example 1 that the real data set consists of 78 banking branches.

Figure 2 

The relative contributions of different variables on the overlap in DEA-DA in example 1 that the real data set consists of 18 banking branches.

5. Conclusions

Due to the inherent complexity and randomness of the data in DEA and problems involving unpredictable or stochastic variables, a probabilistic analysis may be the most rational method of analysis. Therefore, in a probabilistic-based approach, the results open the door to understanding the appropriate estimation of the deciding variables in DEA. In the overlap in DA, the analytical results show that the loans granted and different deposit variables are the main sources of uncertainty, while other variables have a relatively small effect. The main advantage of the used sensitivity analysis approach is that it provides quantified evaluation of the influence of individual variables in DA, and results may be used for decision making.

Acknowledgment

The authors would like to thank the anonymous reviewers for comments which helped to improve the paper.