On the Equivalence of Multirater Kappas Based on 2-Agreement and 3-Agreement with Binary Scores

Warrens, Matthijs J.

doi:https://doi.org/10.5402/2012/656390

International Scholarly Research Notices

On this page

Abstract Introduction Discussion References Copyright Related Articles

Research Article | Open Access

Volume 2012 | Article ID 656390 | https://doi.org/10.5402/2012/656390

On the Equivalence of Multirater Kappas Based on 2-Agreement and 3-Agreement with Binary Scores

Matthijs J. Warrens¹

Academic Editor: J. Hu, O. Pons

Received07 Aug 2012

Accepted25 Aug 2012

Published15 Oct 2012

Abstract

Cohen’s kappa is a popular descriptive statistic for summarizing agreement between the classifications of two raters on a nominal scale. With raters there are several views in the literature on how to define agreement. The concept of g-agreement refers to the situation in which it is decided that there is agreement if g out of m raters assign an object to the same category. Given raters we can formulate multirater kappas, one based on 2-agreement, one based on 3-agreement, and so on, and one based on m-agreement. It is shown that if the scale consists of only two categories the multi-rater kappas based on 2-agreement and 3-agreement are identical.

1. Introduction

In social sciences and medical research it is frequently required that a group of objects is rated on a nominal scale with two or more categories. The raters may be pathologists that rate the severity of lesions from scans, clinicians who classify children on asthma severity, or competing diagnostic devices that classify the extent of disease in patients. Because there is often no golden standard, analysis of the interrater data provides a useful means of assessing the reliability of the rating system. Therefore, researchers often require that the classification task is performed by raters. A standard tool for the analysis of agreement in a reliability study with raters is Cohen’s kappa [5, 28, 34], denoted by [2, 12]. The value of Cohen’s is 1 when perfect agreement between the two raters occurs, 0 when agreement is equal to that expected under independence, and negative when agreement is less than expected by chance. A value .60 may indicate good agreement, whereas a value .80 may even indicate excellent agreement [4, 16]. A variety of extensions of Cohen’s have been developed [19]. These include kappas for groups of raters [24, 25], kappas for multiple raters [15, 29], and weighted kappas [26, 30, 31]. This paper focuses on kappas for raters making judgments on a binary scale.

With multiple raters there are several views on how to define agreement [13, 21, 22]. One may decide that there is only agreement if all raters assign a subject to the same category (see, e.g., [27]). This type of agreement is referred to as simultaneous agreement, -agreement, or DeMoivre’s definition of agreement [13]. Since only one deviating rating of a subject will lead to the conclusion that there is no agreement with respect to the subject, -agreement looks especially useful in case the researchers demands are extremely high [22]. Alternatively, a researcher may decide that there is already agreement if any two raters categorize an object consistently. In this case we speak of pairwise agreement or 2-agreement. Conger [6] argued that agreement among raters can actually be considered to be an arbitrary choice along a continuum ranging from 2-agreement to -agreement. The concept of -agreement with refers to the situation in which it is decided that there is agreement if out of raters assign an object to the same category [6].

Given raters we can formulate multirater kappas, one based on 2-agreement, one based on 3-agreement, and so on, and one based on -agreement. Although all these kappas can be defined from a mathematical perspective, the multirater kappas in general produce different values (see, e.g., [32, 33]). The difficulty for a researcher is to decide which form of -agreement should be used in case one is looking for agreement between ratings when the raters are assumed to be equally skilled. Popping [22] notes that in a considerable part of the literature multirater kappas based on 2-agreement are used. Conger [6] notes that especially coefficients based on 3-agreement may be useful in case the researchers demands are slightly higher. Stronger forms of -agreement may in many practical situations be too demanding. However, it turns out that with ratings on a dichotomous scale the multirater kappas based on 2-agreement and 3-agreement are equivalent. This fact is proved in Section 3. First, Section 2 is used to introduce notation and present definitions of 2-, 3-, and 4-agreement. The multirater kappas and the main result are then presented in Section 3. Section 4 contains a discussion.

2. 2-, 3- and 4-Agreement

In this section we consider quantities of -agreement for . Suppose that observers each rate the same set of objects (individuals and observations) on a dichotomous scale. The two categories are labeled 0 and 1, meaning, for example, presence and absence of a trait or a symptom. So, the data consist of binary variables of length . Let , let , and let denote the number of times rater used category . Furthermore, let denote the number of times rater assigned an object to category and rater assigned an object to category . The quantities and are defined analogously. For notational convenience we will work with the relative frequencies , , , and .

For illustrating the concepts and results presented in this paper we use the study presented in O’Malley et al. [20]. In this study four pathologists (raters 1, 3, 5, and 8 in Figure 6 in [20]) examined images from 30 columnar cell lesions of the breast with low-grade/monomorphic-type cytologic atypia. The pathologists were instructed to categorize each as either “Flat Epithelial Atypia” (coded 1) or “Not Atypical” (coded 0). The results for each rater for all 30 cases are presented in Table 1. The 4 columns labeled 1 to 4 of Table 1 contain the ratings of the pathologists. The frequencies in the first column of Table 1 indicate how many times on a total of 30 cases a certain pattern of ratings occurred. Only five of all theoretically possible patterns of 1s and 0s are observed in these data. Values of various multirater kappas for these data are presented on the right-hand side of the table. The formulas of the multirater kappas are presented in Section 3.

We can think of the four proportions , , and as the elements of a table that summarizes the 2-agreement between raters and [10]. Proportions , , , and are quantities of 2-agreement, because they describe information between a pair of raters. In general we have Summing over the rows of this table we obtain the marginal totals and corresponding to rater .

Example 2.1. For raters 1 and 2 in Table 1 we have illustrating identity (2.1). The marginal totals indicate how often raters 1 and 2, used the categories 0 and 1.

We can think of the eight proportions as the elements of a table that summarizes the 3-agreement between raters , and . We have Summing over the direction corresponding to rater , the table collapses into the table for raters and .

Example 2.2. For raters 1, 2 and 3 in Table 1 we have and . Furthermore, we have illustrating identity (2.4).

The 2-agreement and 3-agreement quantities are related in the following way. For we have the identitiesFor example, we have . Moreover, we have an analogous set of identities for products of the marginal totals. That is, for we have the identitiesUsing the relations between the 2-agreement and 3-agreement quantities in (2.7a), (2.7b), and (2.7c) and (2.8a), (2.8b), and (2.8c) we may derive the following identities. Proposition 2.3 is used in the proof of the theorem in Section 3.

Proposition 2.3. Consider three raters , , and . One has

Proof. We can express the sum of the 2-agreement quantities: in terms of 3-agreement quantities using the identities in (2.7a), (2.7b), and (2.7c). Doing this we obtain Applying identity (2.4) to (2.12) we obtain identity (2.9). Using the identities in (2.8a), (2.8b), and (2.8c) identity (2.10) is obtained in a similar way.

We can think of the sixteen proportions as the elements of a table that summarizes the 4-agreement between raters , , , and . We have

Example 2.4. For raters 1, 2, 3, and 4 in Table 1 we have The remaining 4-agreement quantities are zero. Furthermore, we have illustrating identity (2.13).

The 3-agreement and 4-agreement quantities are related in the following way. For we have the identitiesFor example, we have . There is also an analogous set of identities for products of the marginal totals.

The identities in (2.16a), (2.16b), (2.16c), and (2.16d) do not lead to a result analogous to Proposition 2.3. We have however the following less general result.

Proposition 2.5. Consider four raters , ,, and . Suppose One has

Proof. We can express the sum of the 3-agreement quantities in terms of 4-agreement quantities using the identities in (2.16a), (2.16b), (2.16c), and (2.16d). Doing this we obtain Combining (2.13) and (2.17) we obtain the identity Applying (2.21) to (2.20) we obtain identity (2.18).

The 4-agreement quantities , , , , , and are in general not zero. Even if we would require that condition (2.17) holds, we would not obtain an identity similar to (2.18) for the products of the marginal totals.

3. Kappas Based on 2-, 3-, and 4-Agreement

In this section we present the main result. We introduce Cohen’s [5] and three multirater kappas, one based on 2-agreement, one based on 3-agreement, and one based on 4-agreement. For two raters and Cohen’s is defined as

Example 3.1. For raters 1 and 2 in Table 1 we have

There are several ways to generalize Cohen’s to the case of multiple raters. A kappa for raters based on 2-agreement between the raters is given by The in denotes that this coefficient is a measure for raters. The 2 in denotes that the coefficient is a measure of 2-agreement, since the and describe information between pairs of raters.

Coefficient is a special case of a multicategorical kappa that was first considered in Hubert [13] and has been independently proposed by Conger [6]. Hubert's kappa is also discussed in Davies and Fleiss [7], Popping [21], and Heuvelmans and Sanders [11]. Furthermore, Hubert’s kappa is a special case of the descriptive statistics discussed in Berry and Mielke [3] and Janson and Olssen [14]. Standard errors for can be found in Hubert [13].

Example 3.2. For the four raters in Table 1 we have

A kappa for raters based on 3-agreement between the raters is given by For raters we have the special case Coefficient was first considered in Von Eye and Mun [8]. It is also a special case of the weighted kappa proposed in Mielke et al. [17, 18]. The coefficient is a measure of simultaneous agreement [18]. Standard errors for can be found in [17, 18].

Example 3.3. For the four raters in Table 1 we have Interestingly, we have (Example 3.2).

Examples 3.2 and 3.3 show that the multirater kappas based on 2-agreement and 3-agreement produces identical values for the data in Table 1. This equivalence is formalized in the following result.

Theorem 3.4. for all .

Proof. Given raters, a pair of raters and occur times together in a triple of raters. Hence, using identities (2.9) and (2.10) we have Multiplying all terms in by , and using identities (3.8) in the result, we obtain Since in the denominator of (3.9), coefficient (3.9) is equivalent to .

Finally, a kappa for raters based on 4-agreement between the raters is given by The special case extends the kappa proposed in Von Eye and Mun [8] and Mielke et al. [17, 18].

Example 3.5. For the four raters in Table 1 we have Note that for these data we have (Examples 3.2 and 3.3), although the difference between the values of the multirater kappas is negligible.

4. Discussion

Cohen’s kappa is a standard tool for summarizing agreement ratings by two observers on a nominal scale. Cohen’s kappa can only be used for comparing raters at a time. Various authors have proposed extensions of Cohen's kappa for raters. The concept of -agreement with refers to the situation in which it is decided that there is agreement if out of raters assign an object to the same category [6, 22]. Given raters we can formulate multirater kappas, one based on 2-agreement, one based on 3-agreement, and so on, and one based on -agreement. Although all these kappas can be defined from a mathematical perspective, the multirater kappas in general produce different values (see, e.g., [32, 33]). In this paper we considered multirater kappas based on 2-, 3-, and 4-agreement for dichotomous ratings.

As the main result of the paper it was shown (Theorem 3.4, Section 3) that the popular concept of 2-agreement and the slightly more demanding but reasonable alternative concept of 3-agreement coincide for dichotomous (binary) scores, that is, the multirater kappas based on 2-agreement and 3-agreement are identical. Hence, for ratings on a dichotomous scale the problem of which form of agreement to use does not occur. The key properties for this equivalence are the relations between the 2-agreement and 3-agreement quantities in Proposition 2.3 (Section 2). The O’Malley et al. data in Table 1 and the hypothetical data in Table 2 show that 2/3-agreement is not equivalent to 4-agreement. This is because there is no result analogous to Proposition 2.3 between 2/3-agreement and 4-agreement quantities. The data examples in, for example, Warrens [32, 33] show that the equivalence also does not hold for multirater kappas for more than two categories. Furthermore, the data examples in Table 2 show that the 2/3-agreement and 4-agreement kappas can produce quite different values.

Another statistic that is often regarded as a generalization of Cohen’s is the multirater statistic proposed in Fleiss [9]. Artstein and Poesio [1] however showed that this statistic is actually a multirater extension of Scott’s pi [23] (see also [22]). Using instead of in we obtain a special case of the coefficient in Fleiss [9], which shows that the coefficient is a special case of Hubert’s kappa [6, 13, 29]. It is possible to formulate an analogous multirater pi coefficient based on 3-agreement. This pi coefficient is equivalent to the coefficient based on 2-agreement.

Acknowledgment

This paper is a part of project 451-11-026 funded by The Netherlands Organisation for Scientific Research.

References

R. Artstein and M. Poesio, “Kappa³=Alpha (or beta),” NLE Technical Note 05-1, University of Essex, 2005.
View at: Google Scholar
M. Banerjee, M. Capozzoli, L. McSweeney, and D. Sinha, “Beyond kappa: a review of interrater agreement measures,” The Canadian Journal of Statistics, vol. 27, no. 1, pp. 3–23, 1999.
View at: Publisher Site | Google Scholar | Zentralblatt MATH
K. J. Berry and P. W. Mielke, “A generalization of Cohen’s kappa agreement measure to interval measurement and multiple raters,” Educational and Psychological Measurement, vol. 48, pp. 921–933, 1988.
View at: Google Scholar
D. Cicchetti, R. Bronen, S. Spencer et al., “Rating scales, scales of measurement, issues of reliability: resolving some critical issues for clinicians and researchers,” The Journal of Nervous and Mental Disease, vol. 194, no. 8, pp. 557–564, 2006.
View at: Publisher Site | Google Scholar
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement, vol. 20, pp. 37–46, 1960.
View at: Publisher Site | Google Scholar
A. J. Conger, “Integration and generalization of kappas for multiple raters,” Psychological Bulletin, vol. 88, no. 2, pp. 322–328, 1980.
View at: Publisher Site | Google Scholar
M. Davies and J. L. Fleiss, “Measuring agreement for multinomial data,” Biometrics, vol. 38, pp. 1047–1051, 1982.
View at: Publisher Site | Google Scholar | Zentralblatt MATH
A. Von Eye and E. Y. Mun, Analyzing Rater Agreement. Manifest Variable Methods, Lawrence Erlbaum Associates, 2006.
J. L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin, vol. 76, no. 5, pp. 378–382, 1971.
View at: Publisher Site | Google Scholar
J. L. Fleiss, “Measuring agreement between two judges on the presence or absence of a trait,” Biometrics, vol. 31, no. 3, pp. 651–659, 1975.
View at: Publisher Site | Google Scholar
A. P. J. M. Heuvelmans and P. F. Sanders, “Beoordelaarsovereenstemming,” in Psychometrie in De Praktijk, P. F. Sanders and T. J. H. M. Eggen, Eds., pp. 443–470, Cito Instituut voor Toestontwikkeling, Arnhem, The Netherlands, 1993.
View at: Google Scholar
L. M. Hsu and R. Field, “Interrater agreement measures: comments on ${kappa}_{n}$ , Cohen’s kappa, Scott’s π and Aickin’s α,” Understanding Statistics, vol. 2, pp. 205–219, 2003.
View at: Publisher Site | Google Scholar
L. Hubert, “Kappa revisited,” Psychological Bulletin, vol. 84, no. 2, pp. 289–297, 1977.
View at: Publisher Site | Google Scholar
H. Janson and U. Olsson, “A measure of agreement for interval or nominal multivariate observations,” Educational and Psychological Measurement A, vol. 61, no. 2, pp. 277–289, 2001.
View at: Publisher Site | Google Scholar
J. R. Landis and G. G. Koch, “An application of hierarchical kappatype statistics in the assessment of majority agreement among multiple observers,” Biometrics, vol. 33, pp. 363–374, 1977.
View at: Google Scholar
J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, pp. 159–174, 1977.
View at: Google Scholar
P. W. Mielke, K. J. Berry, and J. E. Johnston, “The exact variance of weighted kappa with multiple raters,” Psychological Reports, vol. 101, no. 2, pp. 655–660, 2007.
View at: Publisher Site | Google Scholar
P. W. Mielke, K. J. Berry, and J. E. Johnston, “Resampling probability values for weighted kappa with multiple raters,” Psychological Reports, vol. 102, no. 2, pp. 606–613, 2008.
View at: Publisher Site | Google Scholar
J. C. Nelson and M. S. Pepe, “Statistical description of interrater variability in ordinal ratings,” Statistical Methods in Medical Research, vol. 9, no. 5, pp. 475–496, 2000.
View at: Publisher Site | Google Scholar | Zentralblatt MATH
F. P. O'Malley, S. K. Mohsin, S. Badve et al., “Interobserver reproducibility in the diagnosis of flat epithelial atypia of the breast,” Modern Pathology, vol. 19, no. 2, pp. 172–179, 2006.
View at: Publisher Site | Google Scholar
R. Popping, Overeenstemmingsmaten voor Nominale Data [Ph.D. thesis], Rijksuniversiteit Groningen, Groningen, The Netherlands, 1983.
R. Popping, “Some views on agreement to be used in content analysis studies,” Quality & Quantity, vol. 44, no. 6, pp. 1067–1078, 2010.
View at: Publisher Site | Google Scholar
W. A. Scott, “Reliability of content analysis: the case of nominal scale coding,” Public Opinion Quarterly, vol. 19, no. 3, pp. 321–325, 1955.
View at: Publisher Site | Google Scholar
S. Vanbelle and A. Albert, “Agreement between an isolated rater and a group of raters,” Statistica Neerlandica, vol. 63, no. 1, pp. 82–100, 2009.
View at: Publisher Site | Google Scholar
S. Vanbelle and A. Albert, “Agreement between two independent groups of raters,” Psychometrika, vol. 74, no. 3, pp. 477–491, 2009.
View at: Publisher Site | Google Scholar
S. Vanbelle and A. Albert, “A note on the linearly weighted kappa coefficient for ordinal scales,” Statistical Methodology, vol. 6, no. 2, pp. 157–163, 2009.
View at: Publisher Site | Google Scholar | Zentralblatt MATH
M. J. Warrens, “κ-adic similarity coefficients for binary (presence/absence) data,” Journal of Classification, vol. 26, no. 2, pp. 227–245, 2009.
View at: Publisher Site | Google Scholar
M. J. Warrens, “Inequalities between kappa and kappa-like statistics for $κ \times κ$ tables,” Psychometrika, vol. 75, no. 1, pp. 176–185, 2010.
View at: Publisher Site | Google Scholar
M. J. Warrens, “Inequalities between multi-rater kappas,” Advances in Data Analysis and Classification, vol. 4, no. 4, pp. 271–286, 2010.
View at: Publisher Site | Google Scholar
M. J. Warrens, “Cohen's linearly weighted kappa is a weighted average of $2 \times 2$ kappas,” Psychometrika, vol. 76, no. 3, pp. 471–486, 2011.
View at: Publisher Site | Google Scholar
M. J. Warrens, “Weighted kappa is higher than Cohen's kappa for tridiagonal agreement tables,” Statistical Methodology, vol. 8, no. 2, pp. 268–272, 2011.
View at: Publisher Site | Google Scholar | Zentralblatt MATH
M. J. Warrens, “Equivalences of weighted kappas for multiple raters,” Statistical Methodology, vol. 9, no. 3, pp. 407–422, 2012.
View at: Publisher Site | Google Scholar
M. J. Warrens, “A family of multi-rater kappas that can always be increased and decreased by combining categories,” Statistical Methodology, vol. 9, no. 3, pp. 330–340, 2012.
View at: Publisher Site | Google Scholar
M. J. Warrens, “Conditional inequalities between Cohen's kappa and weighted kappas,” Statistical Methodology, vol. 10, pp. 14–22, 2013.
View at: Publisher Site | Google Scholar

Copyright

Copyright © 2012 Matthijs J. Warrens. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

1008

Downloads

1256

Citations