Weight Analysis for Multiattribute Group Decision-Making with Interval Grey Numbers Based on Decision-Makers’ Psychological Criteria
To address the problem of multiattribute group decision-making with interval grey numbers, decision matrices are adjusted using kernels of interval grey numbers to reduce the psychological effects of decision-makers. The comprehensive weights of attributes are obtained by aggregating the subjective weights with objective weights, which are calculated based on the accuracy and difference of attributes. Considering the consistent, best, and worst decision-making abilities of decision-makers, grey incidence models are established to obtain the consistency weights and individual bipolar weights of decision-makers; then, the comprehensive weights of decision-makers are determined. A clustering approach of interval grey numbers is presented, and overall evaluations are obtained. Finally, an example is provided and its validity is tested to verify the feasibility of the proposed method.
Multiattribute group decision-making (MAGDM) is a kind of decision-making method by which multiple experts rank, optimize, and classify a limited number of alternatives with multiple attributes according to certain criteria. MAGDM has been widely used in engineering , management , society [3, 4], and other fields . The efficiency of the weights of decision-makers and attributes in MAGDM significantly affects the correctness of the results. Therefore, a reliable methodology for determining the weights of decision-makers (DMs) and attributes is essential.
There are many studies on the weights of DMs. Chen and Yang  emphasized the weights of DMs according to the proximity of the evaluation value of each DM to the average evaluation value of the group. Yue  determined the weights of DMs by using an extended TOPSIS method with interval numbers. Yue  used the projection rule to determine the weight of each DM. Meng et al.  determined the weights of DMs based on the distance between the decision matrices of individual and other DMs. Wan and Dong  proposed a method based on similarity for determining the weights of DMs. Meng et al.  established a group consensus-based model to determine the weights of the DMs with respect to each object. Cheng et al.  studied expert weights from incomplete linguistic preference relations based on uniform consistency. Abootalebi et al.  proposed a linear programming model based on a deviation function to find the optimal expert weights.
Attribute weights have also been studied by many scholars. Xu and Da  determined the attribute weights for a problem in which the information about the attribute weights is completely unknown. Li et al.  used a programming model to determine attribute weights by minimizing group inconsistency. Rao and Patel  determined the objective weights of attributes according to the ratio of data variances and combined the objective weights with subjective weights in different proportions. Wei  determined attribute weights according to the maximum disparity. Qi et al.  proposed a weight determination method by maximizing the entropy values of interval-valued intuitionistic fuzzy numbers. Zhang and Guo  developed a programming model to calculate attribute weights based on the principle that the evaluation value of each DM represents the smallest deviation from that of the group. Zhou et al.  used the attribute evaluation value entropy as a measure of data stability to obtain attribute weights. Liu et al.  computed means, variances, and correlation coefficients of attributes to determine attribute weights. Lin et al.  proposed an attribute weight optimization model based on the hesitant fuzzy symbol distance to determine attribute weights. Yin et al.  calculated the weight values of decision attribute indexes by using the improved fuzzy entropy formula. Lu et al.  obtained comprehensive weights of indexes according to the principle of vector similarity. Zhou et al.  considered the dissimilarity of risk preferences among different DMs in generating the attribute weights.
There are experts who study DM and attribute weights together. According to the problem of grey relational information decisions, Yan et al.  established a planning model based on the grey incidence degree and principle of maximum entropy to obtain attribute weights. The weights of DMs were determined according to the consistency of group opinions and information distribution. Li et al.  obtained attribute weights based on the principle of entropy maximization and acquired the weight of each DM based on the grey incidence degree between the opinions of individuals and groups. Zhao et al.  determined the weight of each DM by simultaneously considering similarity and proximity and developed a programming model with interval-valued intuitionistic fuzzy values based on crossentropy values to obtain attribute weights.
Although the aforementioned studies have made significant advances, there are still some unresolved issues to be addressed in this field. (1) Traditional decision-making methods do not consider the psychology of the DMs. When different evaluators subjectively assign multiple indicators of the object being evaluated, evaluators likely have different psychological evaluation criteria on one or several indicators, which would reduce the reliability of the decision results. (2) Research on the weights of DMs is based on the consistency of group opinions, but to pursue the consensus of opinions, the influence of individual evaluators on the results is often neglected, which is clearly one-sided. (3) In the actual decision-making process, the evaluation values of DMs on attributes are often not crisp values, and DMs rarely consider the reliability of evaluation values. (4) When attribute values are interval grey numbers, the method of interval numbers is generally used. In fact, there is an essential difference between interval grey numbers and interval numbers. It is not appropriate to use the interval number method to study interval grey numbers, and it may result in the decision information becoming insufficiently utilized. An interval grey number can be represented as a kernel and its associated degree of greyness. Without comprehensively considering the kernel and its degree of greyness, conclusions are biased and cannot truly reflect the essential characteristics of interval grey numbers.
To address these problems, a new method of MAGDM was studied, in which interval grey numbers were treated as attribute values. The evaluation matrices were adjusted according to the psychological factors of DMs, and attribute weights were modified with respect to the accuracy of and difference between attributes. The proposed methodology considered the DMs’ consistency and bipolar judgement on the best and worst alternatives and improved the weights of DMs. In the method, a new technique for weight determination of MAGDM with interval grey numbers is proposed.
The remainder of this paper is set out as follows. Basic definitions and operations of interval grey numbers are presented in Section 2. The problem of MAGDM with interval grey numbers is proposed and the attributes are adjusted based on psychological criteria of DMs in Section 3. The weights of attributes determined in Section 4 and the weights of DMs are calculated in Section 5. In Section 6, an algorithm for the process of MAGDM is provided. An illustrated example is furnished in Section 7. Finally, conclusions are drawn in Section 8.
2. Basic Definitions and Operations of Interval Grey Numbers
In some cases, it is difficult to determine exact decision information, and, as a result, the obtained information can be uncertain or incomplete. Therefore, it is necessary to extend applications from precise numbers to interval grey numbers for practical applications.
Here, some basic definitions and operations of interval grey numbers are presented.
Definition 1. (see ). Suppose that the background, which results in the occurrence of grey number , is , and is a measure of . Then, the kernel of can be defined as ; is the degree of greyness of grey number .
Definition 2. If is the kernel of interval grey number , and is the degree of greyness of the interval grey number . Then, is called the reduced form of the interval grey number.
If distribution information of the interval grey number, , is lacking, then is called the kernel of the interval grey number.
For two interval grey numbers, and , and represent their reduced forms, respectively, and the following algorithms apply : Rule 1: Rule 2: Rule 3: Rule 4: Rule 5: if is a real number, then ,where , which is subject to , is the weight of The algorithm of grey numbers can be extended to cases in which there are several grey numbers to be operated on.
Definition 3. Let and . Then, the distance between and is defined as follows:where is the distance between the kernels of and and is the distance between deviations of the two grey numbers.
In Definition 3, both the distribution of the kernel and the magnitude of the degree of greyness are considered. It can be proved that the distance formula satisfies the following:(1)Nonnegative.(2)Symmetry.(3)Triangle inequality. If is an interval grey number without any limits, then
Definition 4. For any two interval grey numbers and , and are their reduced forms, respectively, and the following applies:(1)If , then .(2)If , then .(3)If , then(i)If , then .(ii)If , then .(iii)If , then .If the degrees of greyness of interval grey numbers are zero, then the comparison between interval grey numbers is converted into the comparison between real numbers.
3. MAGDM with Interval Grey Numbers
It is assumed that is a group of DMs, is a discrete set of feasible alternatives, is a finite set of attributes, and is the weight vector of attributes given by , with , and . The evaluation value of alternative under attribute given by is an interval grey number ; and are the lower and upper limits of the interval grey number, respectively. Then, the evaluation matrix of is .
3.1. Attribute Adjustments Based on the Psychological Criteria of DMs
In the evaluation process, DM tend to have psychological tendencies that are either too strict or too loose, resulting in different evaluation criteria, which leads to the deviation of the evaluation value . The inconsistency of the strictness or leniency of the DMs indicates that the understanding of DMs regarding evaluation criteria is not very clear. Tajeddin and Alemi  found that the individual characteristics of a DM, such as familiarity with the related knowledge, are one of the factors affecting the evaluation bias of DMs. In addition, personality characteristics and professional attitudes of DMs are also influencing factors that cause DMs to be strict or lenient.
The existence of evaluation bias will affect the result of the decision, and thus, it is not appropriate to ignore evaluation bias when making decisions. For example, there are three DMs evaluating three alternatives. For one benefit attribute, the evaluation matrix of DM is [85, 90, 95], the evaluation matrix of DM is [80, 85, 90], and the evaluation matrix of DM is [75, 80, 85]. The maximum value given by DM is equal to the minimum value given by DM . This situation shows that DM is too lenient, while DM is too strict. In this case, it is unreasonable to treat the evaluation values given by DM in a similar manner as those given by DM .
Therefore, in the MADGM process, the psychology of DMs should be considered, and attribute values should be adjusted accordingly. The size of the interval grey number is represented primarily by the kernel; therefore, the overall average kernel of one attribute provided by all DMs is used as the benchmark. The single average kernel of the attribute provided by each DM is compared with the benchmark. If the average kernel is larger than the benchmark, it will be adjusted downward; if the average kernel is smaller than the benchmark, it will be adjusted upward. Hence, adjusted attribute values that exclude the subjective psychological effects of the DMs are obtained.
The average kernel value of attribute given by DM is calculated as follows:
The average kernel value of attribute given by all DMs is calculated as follows:
The average kernel value of attribute given by DM minus that provided by all DMs is then calculated as follows:
Thus, the kernel of the adjusted evaluation value is defined as follows:
When the average evaluation value provided by DM is smaller than that given by all DMs, is negative and its absolute value should be added to each evaluation value of DM . When the average evaluation value provided by DM is larger than that given by all DMs, is positive and should be subtracted from the evaluation value of DM . The adjusted individual decision matrix of DM is denoted as .
3.2. Normalization of the Adjusted Evaluation Value
To measure all attributes and render them dimensionless to facilitate interattribute comparisons, it is necessary to normalize the decision matrices. Calculation equations of decision matrices are provided below:
For benefit attribute , the following applies:with regards to cost attribute , the following applies:
Then, the standardized decision matrix of DM is , where ; the simplified form of is .
The elements of normalized decision matrices are standard interval grey numbers, and the degree of greyness of each standard grey number is the same as that of the original interval grey number.
4. Determination of the Attribute Weights
4.1. Accuracy Weights of the Attributes
The degree of greyness of an interval grey number can be used to express the uncertainty and accuracy of the DM about the attribute value. The larger the degree of greyness of an interval grey number, the higher the uncertainty and the lower the accuracy of the attribute value, and vice versa.
The average value of the degree of greyness of attribute is calculated as follows:
The standard deviation of the degree of greyness of attribute can be calculated as follows:
The accuracy of attribute is defined as follows:
Therefore, the accuracy weight of attribute is derived as follows:
4.2. Difference Weights of the Attributes
For problems in MAGDM, the larger the difference between values of the same attribute for different alternatives, the more information the attribute provides and the greater effect the attribute has on the decision. In contrast, the smaller the difference between values of the same attribute for different alternatives, the smaller the effect the attribute has on the decision. For example, if the same attribute for all alternatives has the same value, then this attribute has no effect on the decision and the corresponding weight can be set to zero.
In this article, distance is used to characterize the difference degree of the attributes. The distance of attribute between all alternatives is calculated as follows:
Therefore, the difference weight of attribute can be calculated by the following equation:
4.3. Integration of the Attribute Weights
After deriving the accuracy weight vector and the difference weight vector of the attributes given by DM , the two weight vectors need to be combined with the governing subjective weight vector into a comprehensive weight by the following equation:where and , , and are the proportion of subjective weight, accuracy weight, and difference weight of attributes provided by DM , respectively.
5. Determination of the Weights of DMs
5.1. Consistency Weights of DMs
In group decision-making, there is generally considered to be a tendency of consistency between individual and group decision-making. If the comprehensive evaluation of a DM is similar to that of the group, which indicates that the decision of this DM is more consistent with the view of other DMs, a higher weight can be assigned to the DM . The grey incidence degree method can be employed to analyse the similarity between the considered and reference sequences by calculating the degree of grey incidence. The greater the degree of grey incidence, the stronger the correlation is between related sequences. The comprehensive evaluation of the group is considered to be the concerned sequence and that of each DM is considered to be the comparison sequence. The greater the degree of relevance between the individual and group comprehensive evaluations, the more consistent the DM’s decision is with that of the group and the higher the objective weight assigned to the DM.
The average comprehensive attribute value of alternative given by DM is calculated as follows:
The average comprehensive attribute value of alternatives of given by the group is calculated as follows:
Definition 5. The grey incidence coefficient between the decision of DM and the decision of the group on each alternative is defined as follows:where is the resolution coefficient subject to . The grey incidence between DM and the group is calculated as follows:The normalized consistency weight of DM is defined as follows:
5.2. Bipolar Weights of DMs
The consistency weight of a DM represents the uniformity between individual DM and the group. However, if we emphasize the consistency of DM decisions too much and ignore disagreements of certain DMs, a “herd effect” may occur, and the results may be unreasonable or distorted. To prevent DMs from excessively pursuing a high degree of consistency of opinions, it is necessary to assign weights to DMs according to the information contained in the evaluations.
The best decision weights and the worst decision weights are determined according to the DM’s judgement on the best alternatives and the worst alternatives, respectively. If a DM has a high evaluation value on the best alternative, it means that the DM has the best decision-making ability and should be assigned a high weight. If a DM evaluates the worst alternative very accurately, the DM’s decision weight should also be increased. The DM’s decision weight should reflect not only the DM’s ability to choose the best alternative but also his/her ability to choose the worst alternative.
Definition 6. If , then is called the best alternative in the evaluation matrix given by DM . If , then is referred to as the worst alternative in the evaluation matrix provided by DM .
Definition 7. If , then is the best alternative in the evaluation matrices provided by all DMs. If , then is referred to as the worst alternative in the evaluation matrices given by all DMs.
Definition 8. The grey incidence coefficient between the best decision of DM and that of the group is defined as follows:where is the resolution coefficient subject to . The grey incidence between the best decision of DM and that of the group is calculated as follows:Then, the best weight of DM is derived as follows:
Definition 9. The grey incidence coefficient between the worst decision of DM and that of the group is calculated as follows:where is the resolution coefficient subject to . The grey incidence between the worst decision of DM and that of the group is defined as follows:Then, the worst weight of DM is derived as follows:The bipolar weight of DM can be calculated by integrating the best decision weight with the worst decision weight as follows:where and represent the DM’s ability to choose the best alternatives and the worst alternatives, respectively.
Then, the comprehensive weight of DM is defined as follows:where and are proportion of consistency weight and bipolar weight of DM , respectively.
6. Proposed Algorithms
In summary, an algorithm for the process of MAGDM to determine weights of attributes and DMs, when interval grey numbers are involved, is provided in the following steps. Step 1. Establish the individual decision matrix. Each DM provides a decision matrix that is based on alternatives with respect to attributes. Step 2. Adjust the individual decision matrix. Adjust the individual decision to for reducing the psychological impacts of DMs using equations (2)–(5). Step 3. Normalize the individual decision matrix. Normalize the adjusted decision matrix to and transform the normalized matrix into a standard form of the interval grey number based on the kernel and degree of greyness using equations (7) and (8). Step 4. Calculate the accuracy weights of attributes. Calculate the average value and standard deviation of the degree of greyness of attribute using equations (9) and (10), respectively. The accuracy weight is obtained using equations (11) and (12). Step 5. Calculate the difference weights of attributes. Compute the distance between all alternatives under attribute and obtain the difference weight using equations (13) and (14). Step 6. Determine the comprehensive weights of attributes. Aggregate the accuracy weight , the difference weight , and the subjective weight to obtain the comprehensive weight of attribute using equation (15). Step 7. Compute the consistency weights of DMs. Construct the grey incidence model to calculate the consistency incidence coefficient between the decision of DM and that of the group based on each alternative with equations (16)– (18) and calculate the consistency weight of DM using equations (19) and (20). Step 8. Calculate the bipolar weights of DMs. The grey incidence coefficient and the degree of grey incidence between the best decision of DM and that of the group are calculated with equations (21) and (22), respectively. The grey incidence coefficient and the degree of grey incidence between the worst decision of DM and that of the group are determined by equations (22) and (25), respectively. Then, the best decision weight and the worst decision weight of DM can be calculated by equations (23) and (26); the bipolar weight of DM is obtained by equation (27). Step 9. Determine the comprehensive weights of DMs. Aggregate the consistency weight and the bipolar weight to obtain the comprehensive weight of DM with equation (28). Step 10. Calculate the overall evaluations of alternatives. Calculate the sum of all interval grey numbers in each line of each normalized decision matrix. The overall evaluation of alternatives of , which is expressed as the reduced forms of the interval grey numbers, is obtained according to the following equation: Step 11. Rank the overall assessments of alternatives. Rank the alternatives in descending order according to the values of .
7. Illustrative Example
To illustrate the abovementioned approach for solving problems in MAGDM, we consider the example used in  for analysis. The problem is described in the following section.
7.1. Example Analysis
Recently, the Ministry of Transport of the People’s Republic of China started a very large road construction project. A core enterprise became aware of this market opportunity but did not possess all the competencies and resources needed; therefore, partner selection was required. There were five main attributes in the process of the partner selection, namely, cost, time, trust, risk, and quality. Cost, time, and risk were cost types, while trust and quality were benefit types. There were four alternatives and four DMs. The objective here was to select a partner that could best satisfy all attributes.
Each DM provided a decision matrix and attribute weights according to Step 1, as shown in Table 1.
The psychological deviations of DMs could be obtained in Step 2, as provided in Table 2.
The decision matrices were adjusted to reduce the psychological deviations of DMs in Step 2, as summarized in Table 3.
The accuracy weights of attributes were obtained in Step 4, while the difference weights of attributes were derived in Step 5; the comprehensive weights of attributes were calculated in Step 6 (where ), as summarized in Table 5.
The weights of DMs could be determined according to individual decisions. First, the grey incidence coefficients and consistency weights of DMs were calculated in Step 7. Second, the degrees of grey incidence of the DMs’ abilities to choose the best and worst decisions were determined, and the bipolar weights of DMs were obtained in Step 8 (where ). Third, the comprehensive weights of DMs were obtained in Step 9 (where ). The results are summarized in Table 6.
The comprehensive evaluation values of the four partners could be calculated in Step 10 and were , , , and .
Because , the alternatives could be ranked in descending order according to the value of as in Step 11; thus, the best alternative is .
7.2. Validity Test
Because different methods of MAGDM may result in different rankings when applied to the same decision-making problem, uncertain results are obtained; Wang and Triantaphyllou  proposed the following test criteria to evaluate the reliability and validity of MAGDM methods. Test criterion 1: a method of MAGDM is effective if the indication of the best alternative remains the same upon replacing a nonoptimal alternative by a worse alternative without changing the relative importance of each decision criteria. Test criterion 2: an effective method of MAGDM should be transitive. Test criterion 3: a method of MAGDM is effective if the combined ranking of alternatives remains similar to the ranking of the original problem upon decomposing the multicriteria decision-making (MCDM) problem into smaller problems and by applying the same MAGDM method to these subproblems to rank the alternatives.
The validity of the proposed approach is tested by using these criteria as follows.
7.2.1. Validity Check with Criterion 1
To test the validity of the proposed approach under criterion 1, we replaced the nonoptimal alternative with the worse alternative in the original decision matrix of each expert, and their rating values are summarized in Table 7.
Now, by applying the proposed method to the modified data, we obtained the collective values of alternatives as follows: , , , and . Therefore, the ranking order of the alternatives was , which indicated that was still the best alternative, and hence, the proposed approach satisfied test criterion 1.
7.2.2. Validity Check with Criteria 2 and 3
To evaluate the proposed approach of MAGDM under criteria 2 and 3, we decomposed the original decision-making problem into three decision-making subproblems, consisting of alternatives , , and . We applied the proposed approach of MAGDM to these subproblems and determined the ranking order of alternatives as , and . After combining the ranking of alternatives of these smaller problems, we determined the final ranking order as , which was the same as the original problem; the latter shows the transitive property of the proposed approach. Hence, the proposed approach of MAGDM was valid under criteria 2 and 3.
7.3. Result Analysis
In order to further validate the significance and rationality of our method, we compare the results obtained from the method proposed in this paper with results from other methods. Following the proposed algorithms in Section 6, we recalculated the results without adjusting the individual decision matrix (psychological criteria of DMs excluded). The overall evaluations of alternatives are , , , and . Therefore, the ranking order of the alternatives would be and was determined to be the best alternative. In addition, the results derived from the method in , in which psychological deviations of DMs are not taken into account, is another object selected to be compared with. Based on the method from , the raking order is , and was the best alternative.
The results of the two comparative methods are different from the result of our proposed approach. It demonstrates that the consideration of the psychological factors does make difference and bring new results into decision making. Moreover, our method also considered other factors, such as the best and worst decision-making abilities of DMs, to coordinate and unify the evaluation information of the group of DMs. Therefore, the proposed method in this paper covers more situations and the results are more reasonable.
For MAGDM with interval grey numbers, evaluation values are adjusted to reduce the psychological deviations of DMs. To solve the inconsistencies between subjective weights and objective weights computed from attribute values provided by the DMs, comprehensive weights in which subjective weights and objective weights are combined are used as attribute weights; the objective weights of the attributes are obtained based on the accuracies of and differences between DMs. Based on the consensus between the individual DMs and the group of DMs, as well as the best and worst decision-making abilities of individual DMs, grey incidence models are established to obtain the weights of the DMs. The application example demonstrates the feasibility of the proposed model and its strength in terms of the effective usage of available information.
All data can be accessed in the illustrative example section of this article.
Conflicts of Interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
This paper was sponsored by the Project of Education Department of Henan Province (no. 2017-ZZJH-227), China Scholarship Council (CSC, no.  5087), and Royal Society and NSFC (no. IEC\NSFC\170391).
S. P. Wan and J. Y. Dong, “A possibility degree method for interval valued intuitionistic fuzzy multi-attribute group decision making,” Journal of Computer ＆ System Sciences, vol. 80, no. 1, pp. 237–256, 2014.View at: Google Scholar
X. W. Qi, C. Y. Liang, E. Q. Zhang, and Y. Ding, “Approach to interval-valued intuitionistic fuzzy multiple attributes group decision making based on maximum entropy,” Systems Engineering-Theory & Practice, vol. 31, no. 20, pp. 1940–1948, 2011.View at: Google Scholar
R. X. Zhou, F. Y. Fan, D. Y. He, and W. H. Qiu, “Integrated entropy weight method based on data stability and subjective preference in multi-attribute group decision-making,” Control and Decision, vol. 27, no. 7, pp. 1169–1174, 2012.View at: Google Scholar
X. D. Liu, J. J. Zhu, S. T. Zhang, and G. D. Liu, “Hesitant fuzzy multiple attribute decision making method based on optimization of attribute weights,” Control and Decision, vol. 31, no. 2, pp. 297–302, 2016.View at: Google Scholar
S. Lin, X. D. Liu, J. J. Zhu, and S. T. Zhang, “Hesitant fuzzy decision making method with unknown weight information based on an improved signed distance,” Control and Decision, vol. 33, no. 1, pp. 186–192, 2018.View at: Google Scholar
S. Yin, Z. S. Yang, and Y. Chen, “Interval-valued intuitionistic fuzzy multiple attribute decision making based on the improved fuzzy entropy,” Systems Engineering and Electronic, vol. 40, no. 5, pp. 1080–1084, 2018.View at: Google Scholar
Y. F. Lu, L. L. Li, and Z. Zhang, “Research and application of decision index combination weighting method,” Computer Engineering, vol. 44, no. 1, pp. 84–90, 2018.View at: Google Scholar
S. L. Yan, S. F. Liu, Z. G. Fang, and L. F. Wu, “Method of determining weights of decision makers and attributes for group decision making with interval grey numbers,” System Engineering-Theory & Practice, vol. 34, no. 9, pp. 2372–2378, 2014.View at: Google Scholar
Y. L. Li, X. L. Yin, and J. Yang, “Method of determining weights of DMs and attributes for interval-valued intuitionistic fuzzy decision making,” Computer Engineering and Applications, vol. 52, no. 18, pp. 158–161, 2016.View at: Google Scholar
M. Zhao, S. S. Qin, J. H. Xie, F. B. Zhang, and G. Li, “Interval-valued intuitionistic fuzzy multi-attribute group decision making considering risk preference of decision maker,” Operations Research and Management Science, vol. 27, no. 1, pp. 7–16, 2018.View at: Google Scholar
S. F. Liu, Z. G. Fang, and N. M. Xie, “Algorithm rules of interval grey numbers based on the “kernel” and the degree of greyness of grey numbers,” Systems Engineering and Electronics, vol. 32, no. 2, pp. 313–316, 2010.View at: Google Scholar
A. H. Cash, B. K. Hamre, R. C. Pianta, and S. S. Myers, “Rater calibration when observational assessment occurs at large scale: degree of calibration and characteristics of raters associated with calibration,” Early Childhood Research Quarterly, vol. 27, no. 3, pp. 529–542, 2012.View at: Publisher Site | Google Scholar
Z. Tajeddin and M. Alemi, “Pragmatic rater training: does it affect non-native L2 teachers’ rating accuracy and bias,” Iranian Journal of Language Testing, vol. 4, no. 1, pp. 66–83, 2014.View at: Google Scholar
L. Yang, “Group Decision Theory and Application: Research on Individual Preference Aggregation Methods in Group Decision Making, Economic Science Press, Beijing, China, 2004.