Peirce's <svg     style="vertical-align:-0.216pt;width:6.25px;" id="M1" height="15.075" version="1.1" viewBox="0 0 6.25 15.075" width="6.25"  xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg">
	
		
			<g transform="matrix(.022,-0,0,-.022,.062,14.75)"><path id="x1D456" d="M244 607q0 -25 -15.5 -43t-37.5 -18q-19 0 -32 13t-13 35q0 21 15 41t39 20q20 0 32 -14t12 -34zM222 91q-29 -33 -79 -68t-75 -35q-13 0 -19 7.5t-6 31t10 65.5l62 253q5 26 -1 26q-21 0 -72 -43l-13 24q43 40 91 68t71 28q30 0 10 -78l-71 -274q-8 -30 3 -30
q16 0 76 48z" /></g>

		
	
</svg> and Cohen's <svg     style="vertical-align:-0.216pt;width:11.4875px;" id="M2" height="10.4375" version="1.1" viewBox="0 0 11.4875 10.4375" width="11.4875"  xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg">
	
		
			<g transform="matrix(.022,-0,0,-.022,.062,10.1)"><path id="x1D705" d="M484 406q0 -21 -22 -38q-10 -7 -18 -1q-15 11 -30 11q-30 0 -77.5 -37t-74.5 -67l79 -168q11 -23 19.5 -34.5t13 -14t10.5 -2.5q9 0 30 17.5t40 42.5l17 -23q-31 -45 -57.5 -69.5t-40.5 -29.5t-31 -5q-18 0 -32 13.5t-35 55.5q-34 71 -69 160q-36 -32 -59 -56
q-12 -27 -36 -158q-44 -7 -77 -15l-5 7l82 349q6 25 -2 25q-16 0 -72 -41l-14 22q44 41 89 69.5t66 28.5q19 0 19 -32q0 -16 -6 -43l-35 -160l146 143q94 92 136 92q20 0 33 -12t13 -30z" /></g>

		
	
</svg> for <svg     style="vertical-align:-0.0pt;width:44.762501px;" id="M3" height="14.3625" version="1.1" viewBox="0 0 44.762501 14.3625" width="44.762501"  xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg">
	
		
			<g transform="matrix(.022,-0,0,-.022,.062,14.3)"><path id="x32" d="M412 140l28 -9q0 -2 -35 -131h-373v23q112 112 161 170q59 70 92 127t33 115q0 63 -31 98t-86 35q-75 0 -137 -93l-22 20l57 81q55 59 135 59q69 0 118.5 -46.5t49.5 -122.5q0 -62 -29.5 -114t-102.5 -130l-141 -149h186q42 0 58.5 10.5t38.5 56.5z" /></g><g transform="matrix(.022,-0,0,-.022,15.799,14.3)"><path id="xD7" d="M528 54l-36 -38l-198 201l-198 -201l-36 38l197 200l-197 201l36 38l198 -202l198 202l36 -38l-197 -201z" /></g><g transform="matrix(.022,-0,0,-.022,33.933,14.3)"><use xlink:href="#x32"/></g>

		
	
</svg> Measures of Rater Reliability

<table>Gold standard data generating mechanism.</table>

Journal of Probability and Statistics

fig1

Figure 1

Figure 1: Peirce's <svg     style="vertical-align:-0.216pt;width:6.25px;" id="M1" height="15.075" version="1.1" viewBox="0 0 6.25 15.075" width="6.25"  xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg">
	
		
			<g transform="matrix(.022,-0,0,-.022,.062,14.75)"><path id="x1D456" d="M244 607q0 -25 -15.5 -43t-37.5 -18q-19 0 -32 13t-13 35q0 21 15 41t39 20q20 0 32 -14t12 -34zM222 91q-29 -33 -79 -68t-75 -35q-13 0 -19 7.5t-6 31t10 65.5l62 253q5 26 -1 26q-21 0 -72 -43l-13 24q43 40 91 68t71 28q30 0 10 -78l-71 -274q-8 -30 3 -30
q16 0 76 48z" /></g>

		
	
</svg> and Cohen's <svg     style="vertical-align:-0.216pt;width:11.4875px;" id="M2" height="10.4375" version="1.1" viewBox="0 0 11.4875 10.4375" width="11.4875"  xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg">
	
		
			<g transform="matrix(.022,-0,0,-.022,.062,10.1)"><path id="x1D705" d="M484 406q0 -21 -22 -38q-10 -7 -18 -1q-15 11 -30 11q-30 0 -77.5 -37t-74.5 -67l79 -168q11 -23 19.5 -34.5t13 -14t10.5 -2.5q9 0 30 17.5t40 42.5l17 -23q-31 -45 -57.5 -69.5t-40.5 -29.5t-31 -5q-18 0 -32 13.5t-35 55.5q-34 71 -69 160q-36 -32 -59 -56
q-12 -27 -36 -158q-44 -7 -77 -15l-5 7l82 349q6 25 -2 25q-16 0 -72 -41l-14 22q44 41 89 69.5t66 28.5q19 0 19 -32q0 -16 -6 -43l-35 -160l146 143q94 92 136 92q20 0 33 -12t13 -30z" /></g>

		
	
</svg> for <svg     style="vertical-align:-0.0pt;width:44.762501px;" id="M3" height="14.3625" version="1.1" viewBox="0 0 44.762501 14.3625" width="44.762501"  xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg">
	
		
			<g transform="matrix(.022,-0,0,-.022,.062,14.3)"><path id="x32" d="M412 140l28 -9q0 -2 -35 -131h-373v23q112 112 161 170q59 70 92 127t33 115q0 63 -31 98t-86 35q-75 0 -137 -93l-22 20l57 81q55 59 135 59q69 0 118.5 -46.5t49.5 -122.5q0 -62 -29.5 -114t-102.5 -130l-141 -149h186q42 0 58.5 10.5t38.5 56.5z" /></g><g transform="matrix(.022,-0,0,-.022,15.799,14.3)"><path id="xD7" d="M528 54l-36 -38l-198 201l-198 -201l-36 38l197 200l-197 201l36 38l198 -202l198 202l36 -38l-197 -201z" /></g><g transform="matrix(.022,-0,0,-.022,33.933,14.3)"><use xlink:href="#x32"/></g>

		
	
</svg> Measures of Rater Reliability 

Journal of Probability and Statistics

Peirce's and Cohen's for Measures of Rater Reliability

Figure 1