Abstract
There are many consensus measures that can be computed using Likert data. Although these measures should work with any number n of choices on the Likert scale, the measurements have been most widely studied and demonstrated for n = 5. One measure of consensus introduced by Akiyama et al. for n = 5 and theoretically generalized to all n depends on both the mean and variance and gives results that can differentiate between some group consensus behavior patterns better than other measures that rely on either just the mean or just the variance separately. However, this measure is more complicated and not easy to apply and understand. This paper addresses these two common problems by introducing a new computational method to find the measure of consensus that works for any number of Likert item choices. The novelty of the approach is that it uses computational methods in -dimensional space. Numerical examples in three-dimensional (for n=6) and four-dimensional (for n=7) spaces are provided in this paper to assure the agreement of the computational and theoretical approach outputs.
1. Introduction
The significance of reaching consensus in a group or among groups can easily be appreciated by anyone who has ever been involved in a group decision-making process. Indeed, some researchers believe that coming to consensus plays a key role in group decision-making [1, 2].
The importance of consensus can be further appreciated when we consider the vast number of fields, other than just group decision-making, that use consensus. For example, politics, economics, social choice, and science all use the idea of consensus [3–5]. When you consider the work of Lehrer and Wagner [6], it can even be seen how some level of agreement (or consensus) is necessary for the central concepts of thinking.
Since consensus is an important idea in many major areas, there is a demand for an accurate way to measure consensus. The central idea of many researchers working in this field of study concerns how to build consensus (or diminish the disagreement) among all people in one group or in more than one group. Many of these measures have an iterative process to try to come to agreement or build consensus [2, 7–9]. If your task is to build consensus by using a multistage process within a group as in [10] or even for each person individually as in [11], you will need at the end of each stage to have a useful instrument or method to measure consensus.
There are several different meanings or definitions in the literature for the term “consensus” or “disagreement.” However, these two concepts are always the compliment of each other. In this paper, we use the term consensus to mean an opinion or belief reached by a group of persons who can agree on Likert scale items, while disagreement refers to a difference of opinion or perception.
The research to find a mathematical or statistical measure of consensus (or disagreement) started with simple measures and has progressed over time to more sophisticated techniques. The simplest and most widely used measures of consensus and disagreement are the percent agreement measure and the variance. The percent agreement measure has been used in different cases and has been applied to small group consensus. This measure gives only a percentage of team members who accept a particular opinion [12, 13]. The variance measure, evaluated from Likert items, is usually used to talk about the disagreement (or lack of consensus) [14]. This statistical measurement is significant in precise comparison situations. However, the variance can be useless in cases of comparing different sizes of groups or for groups that have different means [15].
Another common approach to calculating the consensus is , introduced in [16] by James et al. This measure is based on the variance and is sometimes called inter-rater reliability. Two other measures of consensus are presented by Kendall [17] and Alcalde-Unzu and Vorsatz [18]. The measure Kendall proposes, usually known as Kendall’s tau (), is limited to the status of two individuals and directly calculates the ratio of pairwise comparisons of how two people agree. The second measure, symbolically known as (), was introduced by Alcalde-Unzu and Vorsatz. This measure determines, for any pair of alternatives, the absolute value of the difference between the ratio of persons who choose one alternative and the proportion of individuals who choose the other option and then takes the average of these numbers over all possible pairs of alternatives [18]. The concepts, comparisons, and properties of these two measures are well introduced in [19].
There are many other more advanced methods presented to measure the consensus within a group of individuals. The method of Beliakov et al. [20] concentrates on structuring a function that can measure the degree of consensus from a set of inputs provided as numbers from the unit interval. In the information theory field, Shannon’s [21] formula for Entropy, or the measure of the degree of disorder of a system [22], is widely used. The formula is (where is the probability of event’s outcome). Tastle and Weirman [23, 24] apply Shannon’s formula to Likert scales responses. González-Arteaga [1] provides an approach to consensus measurement based on the Pearson correlation coefficient.
Based on what is presented above, there are several ways to define consensus and various ways to measure consensus within and among groups. However, none of these approaches use computational geometry concepts in -dimensional space as introduced in this paper. In this article, we consider a new method to determine the measure of consensus (or conversely the index of disagreement) presented by Akiyama et al. [25] and generalized by Tsuchiya and Hiramoto [26] with the key to this new method using computational geometry concepts presented in Abdal Rahem and Darrah [27]. The researchers in [25] present the sophisticated index of disagreement, called , and the measure of consensus, , that use the conditional distribution of the variance for a given mean [28]. The distinction of this index is that it allows the comparison of consensus values of various questions for the same group or the same question for diverse groups, even comparisons of groups with different sizes and the questions with different means. However, the way this index is presented for in [25] for n = 5 and generalized in [26] to all n is difficult to understand and not easy to apply.
Consequently, Abdal Rahem and Darrah in [27] provide a straightforward algorithmic method to compute the reliable measure introduced in [25]. The idea of this new approach is much easier to use and understand than the way of calculating the index of disagreement presented in [25]. However, in [27], the approach is shown for only two-dimensional space, which relates to Likert scale. The aim of the research presented in this article is to break the limitation of working with Likert scale only. Likert items are used in many disciplines to measure attitudes, preferences, and subjective reactions [14, 29, 30]. For this discussion, a global Likert scale is used, with the integers 1 through corresponding to the words strongly disagree through strongly agree or any other words the researcher prefers or the question suggests.
The rest of the paper is organized as follows. Section 2 presents the theoretical foundation of our work and defines the index of disagreement for any number of Likert scale and provides a visual illustration of how the equations in Section 2 look for any dimension. In Section 3 the techniques are applied to . Section 4 provides examples of special cases of sets of probabilities as well as specific means with different variances of each fixed mean. Finally, some concluding remarks and future works are discussed in Section 5.
2. Theoretical Foundations
The main theoretical foundations of the consensus measurement are introduced in Akiyama et al. [25]. We introduce there the idea that variance is always a function of the mean. For any fixed mean, there will be a range of variances that will be possible. Then given a mean and a variance , we find the consensus measure for this two numbers by using conditional probability to determine the “ratio of part of the range of variance to the total range of variance.” In the first paper [25], this measurement is computed by a series of analytical steps for . We first algebraically reduce the computation to a set of three equations in two dimensions. Then by finding the areas between pairs of these equations and subtracting these two areas, we were able to find a number that represents the area given a particular mean, , and variance, . We used this method to define a function and integrate this function from the minimum to the given . Finally, we divide this number by the integration over the total range of variance, minimum to maximum , to get the index of disagreement . The measure of consensus, , will then be 1-.
In the second paper, Abdal Rahem and Darrah [27], to simplify the calculation of these areas and make it possible to generalize to any , two algorithms using computational geometry are presented to replace the complicated analytical steps involving calculus. The first algorithm focuses on computing and the second algorithm is for using numerical methods to compute the ratio of the integration. Also, all the background from computational geometry and conditional probability are located in [27]. The second paper provides a basis for this generalization work to that follows in the remainder of this manuscript.
2.1. Basic Structure for Consensus Measure (Conversely, Index of Disagreement)
Let us start by introducing the notation for the two main variables we will work with. Suppose and refer to the mean and the variance computed from the survey wherein the respondents are asked to pick out exactly one answer from the items: .
Other common notations used throughout are the probabilities , , which means all points will first satisfy . Moreover, refer to a set of points in space and is a function from space to .
As utilized in [25], the basic equations for the foundation to compute the index of disagreement are the sum of all probabilities equal to one, the equation for computing the mean, and the equation derived from computing the variance of random variables. Mathematically, we can write the system of equations as follows:
Using the notion , , we can solve (1) for in terms of as below:
where , and are functions of variables. Since we have all the probabilities greater than zero , we can rewrite the system above as follows:
or equivalently
In order to simplify (4), let us define and a function as below:
where is our fixed mean and is any variance in the range with respect to that mean. Therefore, (4) will be
can be defined as the set of all pairs in the -dimensional space that satisfy (6). Due to the symmetry of the mean with respect to the midpoint, in the subsequent discussions we will restrict the range of the mean to . The remaining portion can be treated as the symmetric reflection of .
In order to find minimum () and the maximum (), substitute the minimum () and maximum (), for a given mean , respectively, in . Consequently, is a linear (one-to-one) mapping of the interval onto . The inverse of is also a linear mapping of onto given by .
For purposes of finding the area for fixed , we derive three equations from (6) to get three lines, planes, or hyperplanes in space; say where that subdivided the space. For example, in we have
And in we get
We set which leads to defining , so we compare and to . If we only compare and to find an area between them (if it exists), this will not give us any information related to . Hence, we define as follows:
whereand
Note that all are probabilities, so we restrict .
Now that we have presented all the equations required for restricting the target area, the next step is to compute these areas in any dimension.
2.2. Determine the Index of Disagreement
Although mathematicians may prefer an analytical process to get an exact solution, in many cases it is extremely hard or even impossible to find one. The way of finding the area introduced in Akiyama et al. [25] for uses an analytical method to find the exact solution; however, it has many cases that must be considered. Therefore, the new computational approach presented in this paper finds a very good solution (as good as is required) and can be generalized to any .
To examine what it means to determine these areas or volumes in -dimensional space, we can first look at the problem in 2-dimensional space. The equations are line segments and so the intersections are points. To find the desired area, we simply find the area bounded between two lines and that satisfies and , since x and y are probabilities. In this case, there are two ways to compute the area, analytically [25] and by using the computational geometry concepts [27]. See Figure 1 for an example.
(a)
(b)
By adding one more probability, the volume is then in three dimensions. In this case, the equations are planes. This means the intersections are lines instead of points. Moreover, for , The work will be in four dimensions. Thus, the equations are hyperplanes and the intersection of any two hyperplanes is a 3-dimensional object, and so on. See Figure 2 for an example.
(a)
(b)
Consequently and since the region has such a strange shape in general, calculating its area or volume proves to be very difficult with analytical methods, especially as we go to higher dimensions, but calculating the area of rectangles or cubes is simple. We will use this method to simplify our calculations by subdividing the region into small rectangles or cubes, as is a common method for approximating an area or volume. One of the popular approaches to finding the area under a curve numerically is using the Riemann sum. Employing the same idea as the Riemann sum, with some modification to make it more accurate and fit with our problem, we can determine the area or volume of the required region.
Before presenting the steps of the algorithms, recall that the area for any is bounded by (i.e., for any ).
Algorithm
(1)Divide the interval for every axis into subintervals of length , (), where is a positive integer(2)For to For to For to .(3)If satisfy (10) then If satisfy (11) then and avoid all others.(4).
Note that in this approach of finding the area, the larger the (i.e., the smaller ) is, the closer the estimate gets to the exact area.
Once you get , you can use the same algorithm for determining the integration as we presented in [27] to compute the index of disagreement and the consensus value.
3. Index of Disagreement in 3D
In order to ensure that the methods described above produce acceptable results, they are applied to different cases. For , you can find the examples with details in Akiyama et al. [25] or Abdal Rahem and Darrah [27].
For , our example is in 3-dimensional space, and the equations are as follows:
which means that the system of inequities in (6) becomes
For determining the volume restricted by two planes and the axis, apply the algorithm above. This will find the volume by adding each small cube when satisfies one of the systems of equations to get and and then use those to find .
4. Numerical Example
Now we can look at some examples to see if the method produces the desired results. First, we consider some special cases of sets of probabilities. We use the probabilities to determine the mean and the variance. We apply the algorithms above to get the consensus values for these special cases to show that they make sense.
The first case we consider is the case when all the respondents have chosen the same response. In other words, when one of the probabilities, say , is one and all others are zeros. This case gives us and , and therefore the ratio of the index is which gives the consensus value of one.
The second case is when the responses are evenly distributed at opposite ends of the scale. For example, probabilities are for and , and all others are zeros. This time we will get the mean and the variance . These values of the mean and variance lead to full disagreement (i.e., ) since the ratio is , which then means the consensus is zero. Notice that the result in these two cases is exactly what we would expect and also what we got for with the original method.
Before looking at the different values for the mean and variance, we will look at one more special case. Now, suppose all probabilities are equal. Symbolically, , . The mean in this case is and the variance is . These two values for mean and variance imply . Subsequently, the consensus compared to and for .
For selected values of and , Table 1 shows the different cases of index of disagreement, for , , and the consensus .
The results of all examples above are reasonable and acceptable especially if we compare the results with the same cases for two dimensions, , in [27].
5. Conclusions and Future Work
A new approach for generalizing the consensus measure (or index of disagreement) presented in Akiyama et al. [25] has been developed to work for Likert scales with any number of choices. By using computational geometry and -dimensional space concepts, this paper not only has been able to generalize all the work constructed in [25, 27] but also shows comparable results to those obtained in the last two articles. The difference of this approach compared to the other consensus measures, which also work for any Likert items, is that this measure considers the conditional distribution of the variance for a given mean. In other words, this method provides more information than others that are only based on just the mean or just the variance.
Another distinguishing feature of this new method is that it is easy to understand and apply. In fact, we can summarize this work as easy as A, B, C. () Set up the equations , () determine the areas, and () calculate the integrations and then the index of disagreement. Additionally, by using these simple steps in an Excel spreadsheet, you can get all the variables defined above, and for () and (), all you need to do is to enter the algorithms into any software you prefer. For instance, all the examples in this paper were computed using Microsoft Excel spreadsheet and Matlab.
In future studies, we plan to develop the same ideas to compute the consensus measure for continuous scales. We also plan to continue looking for easier and faster methods to find the areas and volumes described in the paper by using advanced computation geometry and high-dimensional statistics concepts [31, 32].
Data Availability
The data used are available upon request.
Conflicts of Interest
The authors declare that they have no conflicts of interest.