Abstract
Uncertainty theory is a branch of axiomatic mathematics dealing with experts’ belief degree. Considering the uncertainty with experts’ belief degree in the evaluation system and the different roles which different indices play in evaluating the overall goal with a hierarchical structure, a new comprehensive evaluation method is constructed based on uncertainty theory. First, index scores and weights of indices are described by uncertain variables and evaluation grades are described by uncertain sets. Second, weights of indices with respect to the overall goal are introduced. Third, a new uncertainty comprehensive evaluation method is constructed and proved to be a generalization of the weighted average method. Finally, an application is developed in evaluating software quality, which shows the effectiveness of the new method.
1. Introduction
Due to human language and technology difficulties, it is difficult to provide an effect and objective evaluation for a system. Therefore, many scholars attempt to establish new mathematical methods to make evaluation results consistent with actual situations. Accordingly, Saaty [1] proposed the analytic hierarchy process (AHP), a method to address multicriteria decision analysis with quantitative and qualitative information. In AHP, human judgments are represented as exact numbers. However, some decision-makers may be reluctant or unable to assign exact numbers to comparison judgments because some evaluation criteria are subjective and qualitative. Therefore, Wu and Tsai [2] used both AHP and decision-making trial and evaluation laboratory methods to evaluate the criteria in autospare parts industry in Taiwan. Kumar et al. [3] presented a new general procedure to construct the membership and nonmembership functions of the fuzzy reliability using time-dependent intuitionistic fuzzy set. By using the finite Markov chain imbedding approach, Zhao and Cui [4] presented a unified formula with the product of matrices for evaluating the system state distribution for generalized multistate -out-of-: systems. Chen et al. [5] provided an evaluation method for enterprisers making investment decisions under hybrid cloud environment using grey system theory. Lee et al. [6] proposed a systematic approach to evaluation of new service concepts by integrating the merit of group analytic hierarchy process in modeling multicriteria decision-making problems. Geng et al. [7] presented a new integrated design concept evaluation approach based on vague sets in order to provide a method for complicated multicriteria decision-making problem under uncertain environments. Aiming to evaluate the government and the monopolist about the consumer’s taste, literature [8] was devoted to the characterization and quantitative representation of imprecise and vague uncertainties and measures of information produced by sources of the considered type. Kramosil [9] introduced the possibilistic variants of both the minimax (the worst case) and the Bayesian optimization principles and applied them in decision-making under uncertainty processed. Using finite-time control and backstepping control approaches, Li et al. [10] proposed a new robust adaptive synchronization scheme to make the synchronization errors of the systems with parameter uncertainties zero in a finite time. Lan et al. [11] presented a bilevel fuzzy principal-agent model for optimal nonlinear taxation problems with asymmetric information and so on.
The above methods address imprecise information, such as human language or experts’ degree of belief using fuzzy set theory (see Zadeh [12]), vague set theory, and grey system theory. However, for the evaluation system, the observed data are often not adequate and we have no choice but to invite some domain experts to evaluate the belief degree that an index belongs to an evaluation grade. In this situation, many surveys show that this imprecise information behaves like neither fuzziness nor randomness (see Liu [13] and Liu [14]). And it was showed by Kahneman and Tversky [15] that human beings usually overweight unlikely events. This fact makes the personal belief degree have much larger variance than the frequency.
In this case, Liu [13] proposed uncertainty theory to deal with belief degree, and Liu [14] refined uncertainty theory. Nowadays, uncertainty theory has become a branch of axiomatic mathematics. The first fundamental concept in uncertainty theory is the uncertain measure, used to measure the degree of belief in an event. The second concept is the uncertain variable, used to represent quantities with imprecise information (e.g., the exact value of oil field reserve). The third concept is uncertainty distribution, which is used to describe uncertain variables. Uncertainty theory has been applied to many areas. Liu [16] established a theory and practice of uncertain programming, Liu [17] applied uncertainty theory to risk analysis and reliability analysis, Liu [18] studied hybrid logic and uncertain logic, Liu [19] proposed inference rule with applications to uncertain control, and Liu [20] studied uncertain process with applications to inference risk model. To explore the recent developments in uncertainty theory, readers may consult Liu [21].
Subsequently, Liu [22] described the weights of indices and the score values of indices with uncertain variables and proposed a comprehensive evaluation method based on uncertainty theory. However, in some certain kinds of assessment domains, we find that different bottom indices play different roles in the evaluation of the overall goal. For example, suppose that and are two students whose four features are shown in Table 1.
Therefore, the feature vectors of and are and . Because and are with the same “Gender” and “Age,” we cannot identify them from the two features. In other words, “Gender” and “Age” do not take effect at all in the identification. Furthermore, because the “Body Height” of and is 180 and 170, respectively, they can be identified by “Body Height.” Of course, they can also be identified by “Body Weight.” In a word, the four features play different roles in the identification of and . For this reason, a weight for each bottom index with respect to the overall goal is introduced to show the different roles of different bottom indices. And without loss of generality, when different bottom indices play the same role in other assessment domains, weights of indices with respect to the overall goal are equal. Considering these reasons, a new evaluation method is proposed based on uncertainty theory.
The structure of this paper is as follows. Section 2 provides some relevant concepts about uncertainty theory. Section 3 establishes a new uncertainty evaluation method based on uncertain sets and uncertain variables. Section 4 gives an application of the proposed method in evaluating software quality, and the conclusions are presented in Section 5.
2. Preliminaries
In this section, we provide some useful definitions of uncertainty theory.
Let be a nonempty set, let be a -algebra on , and let be a set of real numbers. Each element is called an event. The uncertain measure , defined on , was proposed by Liu [13] as follows.
Definition 1 (see Liu [13]). The set function is called an uncertain measure if it satisfies the following.
Axiom 1 (normality axiom). Consider .
Axiom 2 (duality axiom). Consider for any event .
Axiom 3 (subadditivity axiom). For every countable sequence of events , we have
The triplet is called an uncertainty space. Let be uncertainty spaces for . Write Then the product uncertain measure on the product -algebra is defined by the following product axiom [23].
Axiom 4 (product axiom). Let be uncertainty spaces for . The product uncertain measure is an uncertain measure satisfying where are arbitrarily chosen events from for , respectively.
Remark 2. Uncertain measure is interpreted as the personal belief degree (not frequency) of an uncertain event that may occur. It depends on the personal knowledge concerning the event. The uncertain measure will change if the state of knowledge changes.
Definition 3 (see Liu [13]). An uncertain variable is a measurable function from an uncertainty space to the set of real numbers. That is, for any Borel set of real numbers, the set is an event.
Definition 4 (see Liu [14]). Let be a real-valued measurable function, and let be uncertain variables on . Then is an uncertain variable defined by
Definition 5 (see Liu [14]). The uncertainty distribution of an uncertain variable is defined by for any real number (see Figure 1).
Definition 6 (see Liu [14]). An uncertain variable is called zigzag if it has a zigzag uncertainty distribution denoted by , where , , are real numbers with .
Definition 7 (see Liu [23]). The uncertain variables are said to be independent if for any Borel sets of real numbers.
Theorem 8 (see Liu [21]). Assume that and are independent zigzag uncertain variables and , respectively. Then the sum is also a zigzag uncertain variable ; that is, The product of a zigzag uncertain variable and a scalar number is also a zigzag uncertain variable ; that is,
Definition 9 (see Liu [13]). Let be an uncertain variable. Then the expected value of is defined by provided that at least one of the two integrals is finite.
Remark 10. Expected value is the average value of uncertain variable in the sense of uncertain measure and represents the size of uncertain variable.
Example 11 (see Liu [21]). The zigzag uncertain variable has an expected value
Theorem 12 (see Liu [14]). Let and be independent uncertain variables with finite expected values. Then for any real numbers and , one has
An uncertain set is a set-valued function on an uncertainty space that attempts to model “unsharp concepts,” which are essentially sets but their boundaries are not sharply described (because of the ambiguity of human language), such as “young” and “tall.” A formal definition is given as follows.
Definition 13 (see Liu [19]). An uncertain set is a measurable function from an uncertainty space to a collection of sets of real numbers. For any Borel set of real numbers, that is, for any Borel set of real numbers, both of are events.
Definition 14 (see Liu [24]). An uncertain set is said to have a membership function if for any Borel set of real numbers one has
The above equations will be called measure inversion formulas.
Remark 15. Let be a set of real numbers. When an uncertain set has a membership function on , we immediately have Liu [24] proved that a real-valued function is a membership function if and only if (see Figure 2).
Example 16 (see Liu [21]). By a triangular uncertain set we mean the uncertain set fully determined by the triplet of crisp numbers with , whose membership function is
Definition 17 (see Liu [24]). Let be an uncertain set with a membership function . Then the set-valued function is called the inverse membership function of .
Definition 18 (see Liu [25]). The uncertain sets are said to be independent if for any Borel sets one has where are arbitrarily chosen from , , respectively.
Theorem 19 (see Liu [24]). Let and be independent uncertain sets with membership functions and , respectively. Then their union has a membership function
Theorem 20 (see Liu [24]). Let and be independent uncertain sets with membership functions and , respectively. Then their intersection has a membership function
Theorem 21 (see Liu [24]). Let be an uncertain set with membership function . Then its complement has a membership function
Theorem 22 (see Liu [24]). Let be independent uncertain sets with inverse membership functions , respectively. If the function is a measurable function, then is an uncertain set with inverse membership function
Definition 23 (see Liu [19]). Let be a nonempty uncertain set. Then the expected value of is defined by provided that at least one of the two integrals is finite.
Please note that represents “ is imaginarily included in ” and represents “ is imaginarily included in .”
Example 24 (see Liu [21]). The triangular uncertain set has an expected value
3. Uncertainty Evaluation Method
When making a comprehensive evaluation, factors influencing the grade of the overall goal should be considered. The index system is often represented by a three-layer hierarchical structure with the overall goal, the second layer, and the bottom layer (namely, the factors influencing the overall goal’s scaling). Experts always intend to show their own opinions and expectations of each evaluation index, so the evaluation results represent human uncertainty and belief degree. Therefore, the score value and weight of each evaluation index are represented by uncertain variables, and evaluation grades are represented by uncertain sets. Therefore, a new evaluation method based on uncertain variables and uncertain sets is proposed.
3.1. Establishment of Hierarchical Index Structure
Based on evaluation criteria, a hierarchical index structure is established by experts as in Figure 3. There are indices in the second level and indices immediately below index (), where . Let the index (in the bottom level) set be .
3.2. Weights of Indices
Suppose that there are experts in the evaluation group and the degrees of importance are represented by uncertain variables . Each expert gives a degree of importance for each index. For index (; ), experts select , experts select experts select , where . Weight of index is represented by According to Definition 3, weight is an uncertain variable and can be described by its uncertainty distribution .
3.3. Grade Vectors of Bottom Indices
The score values from experts can be represented by uncertain variables. Let the evaluation grades (e.g., Poor, Fair,…, Excellent) be represented by uncertain sets , with membership functions , respectively. Let be the index score of index . Then, the grade vector of index is represented by where () is the membership degree to which index belongs to grade .
The next step is to construct a method to realize the transformation from the grade vector of the bottom index to the grade vector of the overall goal .
3.4. Transformation Method
The transformation method can be obtained with the following three steps.
Step 1. To determine the importance of index in the grading of the overall goal, the weight of index with respect to the overall goal is introduced beginning with the following formulas: where is the weight of index with respect to . From the above formulas, we have
Step 2. Because shows the degree of the effective information offered by in the evaluation of , we calculate
Step 3. By weight of index with respect to , we have From the above formulas, we have Thus, we obtain the grade vector of index in the second level, where indicates the degree to which belongs to evaluation grade .
The grade vector of the overall goal can be obtained in the same way.
Remark 25. From the property of , it is obtained that the larger the is, the more important the role plays in the evaluation of is. If , then the evaluation grade of can be determined only by index . If , then index does not play any role in determining the grade of .
Theorem 26. The weighted-average method is a special case of the above method.
Proof. For each row vector , if there is one component for some and the rest four components are 0, then , , and . Thus, by formulas we have That is to say, the weighted-average method is a special case of the above method.
3.5. Identification
Sometimes, the evaluation scales are comparable (e.g., Poor, Fair,…, Excellent scale; “Fair” is better than “Poor”), and a partial order “≻” can be defined according to the actual situation. If evaluation scale is better than , we denote ; otherwise we denote . Of course, whether is better than depends on the actual situation. To address the ordered division (e.g., Poor, Fair,…, Excellent), Cheng [26] proposed the confidence degree principle.
Confidence Degree Principle (Cheng [26]). Let “≻” be a partial order, let be an ordered division, and let () be the confidence level. If , and if , and then belongs to with at least the confidence level .
4. An Application in the Evaluation of Software Quality
In this section, an application of the proposed method in evaluation of software quality is given.
4.1. Evaluation System and Some Data
A problem of evaluating software quality was discussed in Li [27]. Next, we apply the proposed method to this evaluation problem, and the evaluation system is shown as in Table 2. Based on the evaluation criteria, the experts provided the scores of the bottom indices in Column 5 of Table 2. The evaluation grades are “Poor,” “Fair,” “Good,” “Very good,” and “Excellent.” They are represented by uncertain sets with membership functions , respectively, where membership functions are given by experts according to their personal knowledge and actual situation of the evaluation. According to the scores given by experts (Column 5 in Table 2) and the above membership functions, the grade vectors (Column 4 in Table 2) can be obtained (see Section 4.3).
4.2. Weights of Indices
There are ten experts in the evaluation group, and there are five degrees of importance (namely, “most important,” “more important,” “important,” “less important,” and “unimportant”), which are represented by uncertain variables with the same uncertainty distribution . Taking indices , , , , for examples, the evaluation results from the experts are shown as in Table 3. Weight of index is an uncertain variable . By , , and Theorem 8, we have . The vector is obtained by the 99-table, and the values of are shown in Table 4.
By Example 11 and Theorem 12, the expected value of is . Normalizing the expected value, the weights of indices , , , are . The weights of the other indices can be obtained in the same way (the numbers in Column 3 of Table 2).
4.3. Grade Vectors of Bottom Indices
Taking the grade vector of index as an example, the score value of index is 2.50 and the membership functions of evaluation grades are , respectively. Therefore, the grade vector of index is Similarly, the other grade vectors of the bottom indices can be obtained by the scores of the bottom indices (Column 5 in Table 2).
4.4. Grade Vector of Software Quality
In this subsection, the transformation algorithm established above is used to realize the transformation from the grade vectors of the bottom indices to the grade vector of software quality.
4.4.1. Grade Vectors of Indices in the Second Level
We take the calculation of grade vector of index (install section) as an example.
There are four indices, , , , , immediately below , and the grade-transition matrix of is
With the weights of indices , , , , we can obtain Similarly, we can obtain the grade vectors
4.4.2. Grade Vector of Software Quality
With the grade vectors of indices , we can obtain the grade-transition matrix of as Similar to calculation (1), the grade vector of the overall goal is obtained as
4.5. Identification
In this example, the confidence level of belonging to (Very good) is no less than 0.53 () and the confidence level of belonging to (Good) is no less than 0.96 (). Comparatively, it is more reasonable to define in grade . Therefore, the quality of this software is “Very good.”
4.6. Comparative Analysis
In Li [27], the fuzzy max-min method is used in the software quality evaluation, and we calculate the results via the fuzzy weighted-average method with the same data in Table 2. The results are shown in Table 5. Table 5 shows that the proposed method and fuzzy max-min method produced the same result (i.e., the evaluation result is Very good) and the evaluation result from fuzzy weighted method is Good. And also there is some difference in the membership degree of each grade. Furthermore, the literatures [28, 29] discussed the algorithms for maximizing the soft margin, which show that the larger the difference between two adjacent grades is, the stronger the classification ability of the method is. Therefore, we analyzed the differences between two adjacent grades in Table 6. From Table 6, we can see that there are three differences (between and , and , and and ) and the proposed method is larger than the other two methods. Thus, the proposed method has a stronger classification ability in the evaluation of software quality.
5. Conclusions
In this paper, weights for bottom indices with respect to the overall goal in an evaluation system are introduced and a new uncertainty evaluation method is proposed based on uncertain sets and uncertain variables. This method generalizes the weighted average method, and it is applied in the evaluation of software quality. Comparative analysis with other two methods shows that the proposed method has a stronger classification ability in the evaluation of software quality. More importantly, the proposed method in this paper can also be used in other evaluation systems with a hierarchical structure. Therefore, more applications of the proposed method can be underdeveloped further.
Conflict of Interests
The authors declare that there is no conflict of interests regarding the publication of this paper.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (no. 61073121), the Natural Science Foundation of Hebei Province of China (nos. F2012402037, A2012201033), and the Natural Science Foundation of Hebei Education Department (no. Q2012046). The authors also thank anonymous reviewers for their constructive comments and suggestions and acknowledge the English language editing by Elseviers WebShop.