Intratherapist Reliability in the Rating of Scapula Posture in Multiple Planes of Reference
Background. Evaluation of scapular posture is a fundamental component in the clinical evaluation of the upper quadrant. This study examined the intrarater reliability of scapular posture ratings. Methods. A test-retest reliability investigation was undertaken with one week between assessment sessions. At each session physical therapists conducted visual assessments of scapula posture (relative to the thorax) in five different scapula postural planes (plane of scapula, sagittal plane, transverse plane, horizontal plane, and vertical plane). These five plane ratings were performed for four different scapular posture perturbating conditions (rest, isometric shoulder; flexion, abduction, and external rotation). Results. A total of 100 complete scapular posture ratings (50 left, 50 right) were undertaken at each assessment. The observed agreement between the test and retest postural plane ratings ranged from 59% to 87%; 16 of the 20 plane-condition combinations exceeded 75% observed agreement. Kappa (and prevalence adjusted bias adjusted kappa) values were inconsistent across the postural planes and perturbating conditions. Conclusions. This investigation generally revealed fair to moderate intrarater reliability in the rating of scapular posture by visual inspection. However, enough disagreement between assessments was present to warrant caution when interpreting perceived changes in scapula position between longitudinal assessments using visual inspection alone.
Painful disorders of the upper quadrant are among the most commonly reported musculoskeletal complaints [1, 2]. The scapulae are an important consideration in upper quadrant disorders on the basis of their integral role in the function of the upper limb and cervico-thoracic spine. Changes in scapular posture and motion (often referred to as scapular dyskinesis)  are proposed to detrimentally alter the distribution of mechanical forces within the upper quadrant during upper limb activities due to their extensive muscle attachments to the axial skeleton [4, 5] and impact on glenohumeral joint biomechanics . In support of this are studies showing scapular dyskinesis in upper quadrant disorders such as subacromial impingement [6–8] and chronic neck pain [9, 10] that may also be associated with altered axio-scapular muscle function [6, 7, 11, 12]. However, as highlighted in a review paper, there are inconsistencies between studies as to the most common pattern of scapular dyskinesis present in painful shoulder girdle disorders . Irrespective of which scapular dyskinesis is most common, these studies reinforce that scapular posture and motion are fundamental to clinical evaluation of the upper quadrant [14, 15].
The intuitive starting point for assessment of the scapular is visual inspection of scapular posture [14–16]. Clinicians evaluate posture of the scapulae relative to the thorax using defined bony landmarks of the scapular, clavicles, and thorax, to rate orientation of the scapular bone relative to the thorax in multiple planes of reference  consistent with contemporary biomechanical descriptions of scapular and clavicular kinematics [17, 18]. While lacking the capacity of electromagnetic or optical tracking devices to quantify scapular posture [7, 8, 17], visual inspection does permit judgement of scapular posture that appears to be clinically informative in directing postural correction strategies to alleviate symptoms  and improve muscle function . Furthermore, sophisticated electromagnetic and optical tracking devices currently have limited practical application in clinical settings due to the expense of the equipment, the time required to set up equipment and analyse data, and required technical expertise. Lower-tech methods using clinically accessible equipment, such as inclinometers , to quantitate certain elements of scapular posture, have been described [6, 22]. However, these lower-tech options usually offer a unidimensional measurement of scapular posture and are not able to capture all the dimensions that can be rated using visual inspection.
Visual ratings of scapular posture have the potential to be time efficient and descriptive but also have notable limitations. The subjective interpretation of the ratings underpinned by the lack of consensus regarding “normal” scapular posture at rest [22, 23] may lead to inconsistencies of ratings between therapists, arguably irrespective of experience or training, when rating scapular posture by visual inspection alone. This has been evident in the modest findings of intertherapist reliability studies utilising visual inspection of scapular orientation [16, 24, 25]. Similarly, it is plausible the subjective nature of ratings may also cause inconsistent descriptions of scapular posture by an individual clinician assessing the same patient at multiple assessments (intratherapist reliability). The ability to provide a reliable scapular posture rating within an individual is required in clinical practice to enable judgement of legitimate changes in response to clinical interventions. Therefore, the purpose of this investigation was to evaluate the intratherapist reliability of rating scapular posture over two occasions using visual inspection alone to rate scapular posture in multiple planes of reference. It was anticipated that the findings of this study would assist in understanding the degree of test-retest agreement for this clinical assessment approach to clarify its utility and limitations as a measure of outcome in clinical practice.
An intrarater reliability investigation (test-retest) was undertaken with one week between visual assessments of scapular posture (relative to the thorax) in five different scapular postural planes (plane of scapular, sagittal plane, transverse plane, horizontal plane, and vertical plane) rated over four different scapular posture perturbating conditions (rest, isometric shoulder; flexion, abduction, external rotation). Assessment sessions were spaced one week apart to replicate a usual time interval between clinical consultations so as to test the intratherapist reliability of the measure over a clinically relevant time period.
2.2. Participants and Setting
A total of 100 unique scapular posture observations were obtained from subject-physical therapist pairs (50 left scapular ratings and 50 right scapular ratings) for initial test and retest assessments. The physical therapists () were qualified and registered to practice (one to four years of experience). Subjects were a convenience sample that did not have a history of shoulder or neck pain. Subjects without shoulder or neck pain were considered the most appropriate for this investigation to mitigate the chance of change in scapular posture occurring between the test and retest assessments as a result of clinical interventions, natural recovery, or worsening pathology.
Ethical approval for the study was granted by the Institutional Human Research Ethics Committee and all procedures were conducted according to the Declaration of Helsinki. Participants provided informed consent.
Scapular posture was rated by observation only. The posture of the scapular relative to the thorax was rated in five different postural planes (Table 1, Figure 1) by visualising the relationship between visible bony landmarks of the scapular bone, clavicles, and thorax, to judge relative postural orientation. These planes of scapular posture are consistent with contemporary descriptions of scapular kinematics [17, 18]. As shown in Table 1 for each of the five scapular posture planes evaluated, therapists had to nominate one out of a possible three ordinal ratings, with the middle rating (neutral) always representing “normal” scapular orientation for that postural plane, with ratings either side of the neutral position representing deviation of the scapular towards opposite directions within the plane. The criteria by which the therapist rated a “normal” scapular posture in each plane were based on literature concerning the resting position of the scapular [23, 26].
Ratings of scapular posture were performed in standing during four postural perturbating tasks of the scapular. These included a rest condition (arm positioned at rest by the subjects’ side), and three isometric shoulder tasks in the directions of shoulder flexion, abduction, and external rotation. These four postural perturbating conditions were chosen to challenge the maintenance of scapular posture under unloaded (rest) and directionally loaded (isometric tasks) conditions that replicate typical tests that may be performed in clinical practice to evaluate scapular postural control . The isometric tasks were performed against the resistance of a load cell connected to a computer that provided visual feedback of effort intensity to the participant permitting the performance of all isometric tests at a consistent low intensity (20% of maximal voluntary contraction (MVC)) (Figure 2).
All subjects attended three sessions. This included a preliminary session and the two experimental (test-retest) sessions. During the preliminary session participants were familiarised with the test procedure and the reference MVC values for each of the three isometric shoulder conditions (shoulder; flexion, abduction, external rotation) were recorded. The MVC measures were performed in an identical setup to that for the isometric test condition during the experimental session (Figure 2). All testing was performed in standing against the resistance of a load cell connected to a computer. The load cell was applied just above the lateral elbow crease for abduction, above the anterior elbow crease for flexion, and just proximal to the wrist crease for external rotation. For each of the isometric shoulder conditions the peak value of three MVC trials was documented as the reference value for that condition to be utilised in the experimental (test-retest) sessions. This was repeated for each shoulder. The same member of the research team supervised the preliminary testing session for all subjects.
A rest interval of at least 48 hours was permitted before subjects attended the first experimental session to permit recovery from any fatiguing effects of the preliminary session. Before the commencement of the experimental session, markings were placed on the participants’ thorax to assist the therapists in identifying and estimating scapular posture from the reference bony landmarks. These markings included a 3 cm horizontal line between the T3/4 and T8/9 interspinous spaces, and a vertical line between the T3–T8 spinous processes (Figure 2). Participants were then positioned in a relaxed upright standing position and asked to focus their gaze at a point directly in front of them. Participants were aware that they were being observed, but were blinded to the component of their posture (scapular posture) that was being evaluated. The therapists stood approximately two meters behind the participants. Palpation of the scapulae (or any other body region) was not permitted.
Scapular posture ratings were conducted in a standardised order at both assessment sessions. Ratings in each postural plane were first made for the rest condition, followed by postural ratings for the flexion, abduction, and external rotation isometric conditions, respectively. Therapists were given a five-second window per postural plane rating with short rest periods in between. This resulted in a total of 20 five-second windows in which the observing therapist made 20 ratings (five postural planes rated over the four different scapular posture perturbating conditions) for each scapular at both assessment sessions. A designated research therapist signalled which postural plane was to be judged, as well as when to commence and terminate a specific rating. The side tested first (left or right scapular) was alternated between participants, but was the same at the test and retest assessments. Preliminary piloting of this procedure prior to the commencement of the study indicated this time was sufficient for therapists to make a scapular posture rating decision. Therapists were permitted to ask for an additional repetition(s) if they were not able to determine their rating within the allotted time. However, no additional repetitions were requested by any therapist for any participant. Each rating was hand written onto a paper-based assessment form.
In total, each therapist recorded 40 ratings of scapular posture for each participant at the initial test assessment (five postural plane ratings for each of the rest, flexion, abduction, and external rotation conditions for left (20) and right (20) scapulae). This process was then repeated using an identical procedure at the retest assessment one week later. Therapists were not permitted to view their ratings from the initial test assessment during the retest assessment.
Data from the left and right scapular observations were included (100 unique therapist-subject-scapular observations with equal left and right sided representation) at each assessment. To provide an overview of the nature of ratings across the two assessments, the number (and percentage) of paired ratings that were neutral at both assessments, not neutral at either assessment, neutral at the initial assessment but non-neutral at the retest, and not neutral at the initial but neutral at the retest were tabulated (Table 2). Kappa coefficients were used to examine the agreement between ratings provided in the test and retest assessments for each of the planes and conditions (Table 3). The number of expected and actual matches were also tabulated for each plane and test condition (Table 3). Due to the high prevalence of neutral ratings across the planes and conditions, a prevalence adjusted bias adjusted kappa (PABAK) was also calculated and presented along with the respective prevalence and bias indices (Table 3) [27, 28].
3.1. Scapular Ratings
All participants completed both assessment sessions resulting in a total of 100 unique scapular posture ratings (50 left, 50 right) at each assessment. Table 2 displays the number (and percentage) of paired ratings that were rated neutral at both assessments, not rated neutral at either assessment, rated neutral at the initial assessment but not neutral at the retest, and not rated neutral at the initial but neutral at the retest. The number of ratings in the neutral category ranged from 10 (10%) to 85 (%) across the five postural planes and four posture perturbating conditions. The external rotation condition was associated with the most nonneutral ratings at both assessments, specifically for ratings in the Scapular 60 (60%) and Sagittal 60 (60%) postural planes. The number of ratings that changed from neutral to nonneutral (or nonneutral to neutral) across the test and retest sessions ranged from 13 (13%) on the vertical plane under the flexion isometric condition to 41 (41%) on the scapular plane under the abduction isometric condition.
3.2. Test-Retest Agreement
The percentages of observed agreement and expected agreement between the initial assessment and the retest assessment sessions are shown in Table 3. The expected agreement due to chance between the test and retest ratings ranged from 49% to 84% across the postural planes and during the different test conditions which reflected the high prevalence of neutral ratings. The observed agreement between the test and retest was consistently higher than the expected agreement for each postural plane and test condition and ranged from 59% to 87%; 16 of the 20 plane-condition combinations exceeded 75% observed agreement. Kappa (and prevalence adjusted bias adjusted kappa) values were inconsistent across the postural planes and test conditions. The horizontal and vertical planes had the lowest kappa values despite having the highest levels of exact agreement across the test conditions (due to the high prevalence of neutral ratings in these planes).
This study has shown that the test-retest reliability for visual ratings of scapular posture performed one week apart was somewhat inconsistent across the different postural planes evaluated, and inconsistent between the different test conditions utilised to perturbate scapular posture. Exact observed agreement seemed high with most postural planes and test conditions exceeding 75% agreement (Table 3). However, after adjusting for expected agreement and the high prevalence of neutral ratings, the kappa and prevalence adjusted bias adjusted kappa values indicated that agreement between ratings at the initial assessment and retest assessment were generally only fair to moderate  (Table 3). The three rotational planes (Scapular, Sagittal, and Transverse) generally had higher kappa scores than the translational planes (Horizontal and Vertical). However, this may be a reflection of the high prevalence of neutral ratings having a deflating influence on the kappa coefficient in comparison to the percentage exact agreement, which was actually highest for these translation planes (Horizontal and Vertical). Therefore it is difficult to draw any firm conclusion about superior rating reliability being evident in one plane versus another.
The overall findings of this study are similar to previously reported intratherapist reliability coefficients for the observational rating of scapular movement patterns ()  and spinal posture ()  and highlight the subjective nature and limitations of this type of clinical evaluation.
On the basis of the participant sample in this study reporting no history of painful shoulder or neck disorders it is not surprising that the most consistent pattern of agreement between assessment sessions was for a neutral scapular posture rating (Table 2). It is noteworthy, however, that for each postural plane-condition combination, a substantial proportion of scapular posture ratings were different between sessions. This ranged from 13% and 41% across the planes and conditions (Table 2). The cases of disagreement in this study may be due to either inconsistency in ratings from the therapist or an actual difference in scapular posture between sessions due to many potential factors such as natural variation in postural orientation over time, unintentional variations in the experimental set-up, participant mood and fatigue, or variations due to the level of conscious attention given to postural orientation by the participant between sessions. The test-retest design implemented in this study was premeditated to simulate observation that occurs in a clinical setting and did not permit analyses of the cause of this disagreement (i.e., inconsistency in rating versus inconsistency in scapular position).
There are some interesting findings from this study that have direct relevance to clinical practice. While descriptions such as that provided in Table 1 provide a guide to therapists, the criteria for “normal” scapular posture still lack clear definitive boundaries [22, 23, 26]. Therefore it may be difficult for clinicians to confidently nominate a “normal” or “abnormal” state in patients who do not display an obvious deviation in scapular posture, as evident by the modest reliability observed in this study. Findings from this research suggest that while visual assessment of scapular posture may still be an informative measure between treatment sessions for therapists, it should not be used in isolation to inform decision making regarding the response to treatment between sessions. These decisions should instead be founded on a battery of information from relevant tests that include the observation of scapular posture as one piece of information only. In clinical practice information addition to visual inspection to evaluate the contribution of the scapular to a disorder of the upper quadrant would be assisted with a range of scapular tests that have been examined in recent reviews [6, 22, 23].
There are several strengths and limitations of this investigation as well as important future research directions following this study. The inability to determine whether any observed disagreement was due to inconsistency in observation rating or inconsistency in actual scapular posture between sessions could have been overcome by having therapists rate scapular posture from identical video footage at both sessions. While this may be a potential future study to tease out sources of error, we elected the current study design because it more accurately reflected the clinical setting (and therefore capacity to generalise findings of this study to clinical contexts). Clinicians should therefore be aware of the test-retest error of an assessment tool not only due to the “clinician error” component but also due to the “patient performance” variation between sessions. Similarly, determining the most appropriate time duration between the test and retest assessments for investigations of this nature is impacted by two considerations . First, a longer time between assessments may lead to a greater risk of a true change in scapular positioning occurring. Second, a short time between assessments may increase the risk of recalling ratings from the first assessment. In this investigation, therapists completed 40 ratings for each participant at each assessment, even if a therapist recognised a patient from their previous assessment, the number of ratings completed provides some natural protection against recalling first test ratings for any one plane or condition. Therefore, the authors contend that the one week interval between assessments was appropriate for this investigation. It is also noteworthy that within an individual clinician, the scapular rating at rest may have influenced the ratings provided under the subsequent postural perturbating conditions (isometric shoulder flexion, abduction, or external rotation) within the same assessment. However, observing the scapular position at rest before introducing perturbating conditions is consistent with the way routine clinical examinations of the upper quadrant are performed .
This study has examined intrarater reliability and has only examined reliability in individuals with no history of painful upper quadrant disorders. We are now undertaking studies examining inter-rater reliability of the visual assessment of scapulae with and without painful disorders of the upper quadrant. Additionally, studies in the future may wish to consider linear modelling approaches to examine the relationship between shoulder pathology, scapular positioning (at rest and under isometric moment generation), and other clinically relevant factors (such as dynamic rotator cuff functioning) that may influence the management of patients with painful conditions of the shoulder and neck region.
This investigation generally revealed fair to moderate intrarater reliability in the rating of scapular posture by visual inspection. However, enough disagreement between assessments was present to warrant caution when interpreting perceived changes in scapula position between longitudinal assessments using visual inspection alone.
S. M. McPhail contributed to study design, analysis, paper drafting, editing, and appraisal. L. N. Dalland, K. Naess, M. Lund, T. J. Ytre-Hauge, and S. R. Holm contributed to study design, data collection, and paper editing and appraisal. S. O’Leary contributed to study design, data collection, analysis, paper drafting, editing, and appraisal.
S. M. Mcphail is supported by an NHMRC of Australia Fellowship. S. O’Leary was supported by an NHMRC of Australia Research Training Fellowship, and a Health Practitioner Research Fellowship (Queensland Health and University of Queens land (CCRE Spinal Pain, Injury and Health)).
L. L. Andersen, K. Hansen, O. S. Mortensen, and M. K. Zebis, “Prevalence and anatomical location of muscle tenderness in adults with nonspecific neck/shoulder pain,” BMC Musculoskeletal Disorders, vol. 12, no. 1, p. 169, 2011.View at: Publisher Site | Google Scholar
L. Smith, Q. Louw, L. Crous, and K. Grimmer-Somers, “Prevalence of neck pain and headaches: impact of computer use and other associative factors,” Cephalalgia, vol. 29, no. 2, pp. 250–257, 2009.View at: Publisher Site | Google Scholar
J. J. P. Warner, L. J. Micheli, L. E. Arslanian, J. Kennedy, and R. Kennedy, “Scapulothoracic motion in normal shoulders and shoulders with glenohumeral instability and impingement syndrome: a study using Moire topographic analysis,” Clinical Orthopaedics and Related Research, no. 285, pp. 191–199, 1992.View at: Google Scholar
J. F. Behrsin and K. Maguire, “Levator scapulae action during shoulder movement: a possible mechanism for shoulder pain of cervical origin,” Australian Journal of Physiotherapy, vol. 32, no. 2, pp. 101–106, 1986.View at: Google Scholar
S. O'leary, D. Falla, J. M. Elliott, and G. Jull, “Muscle dysfunction in cervical spine pain: implications for assessment and management,” Journal of Orthopaedic and Sports Physical Therapy, vol. 39, no. 5, pp. 324–333, 2009.View at: Publisher Site | Google Scholar
P. M. Ludewig and J. F. Reynolds, “The association of scapular kinematics and glenohumeral joint pathologies,” Journal of Orthopaedic and Sports Physical Therapy, vol. 39, no. 2, pp. 90–104, 2009.View at: Publisher Site | Google Scholar
P. M. Ludewig and T. M. Cook, “Alterations in shoulder kinematics and associated muscle activity in people with symptoms of shoulder impingement,” Physical Therapy, vol. 80, no. 3, pp. 276–291, 2000.View at: Google Scholar
P. W. McClure, L. A. Michener, and A. R. Karduna, “Shoulder function and 3-dimensional scapular kinematics in people with and without shoulder impingement syndrome,” Physical Therapy, vol. 86, no. 8, pp. 1075–1090, 2006.View at: Google Scholar
H. Helgadottir, E. Kristjansson, S. Mottram, A. Karduna, and H. Jonsson Jr, “Altered alignment of the shoulder girdle and cervical spine in patients with insidious onset neck pain and whiplash-associated disorder,” Journal of Applied Biomechanics, vol. 27, no. 3, pp. 181–191, 2011.View at: Google Scholar
H. Helgadottir, E. Kristjansson, S. Mottram, A. Karduna, and H. Jonsson Jr, “Altered scapular orientation during arm elevation in patients with insidious onset neck pain and whiplash-associated disorder,” Journal of Orthopaedic and Sports Physical Therapy, vol. 40, no. 12, pp. 784–791, 2010.View at: Publisher Site | Google Scholar
H. Helgadottir, E. Kristjansson, E. Einarsson, A. Karduna, and H. Jonsson Jr, “Altered activity of the serratus anterior during unilateral arm elevation in patients with cervical disorders,” Journal of Electromyography and Kinesiology, vol. 21, no. 6, pp. 947–953, 2011.View at: Google Scholar
J. J. Lin, W. P. Hanten, S. L. Olson et al., “Functional activity characteristics of individuals with shoulder dysfunctions,” Journal of Electromyography and Kinesiology, vol. 15, no. 6, pp. 576–586, 2005.View at: Publisher Site | Google Scholar
P. M. Ludewig and J. F. Reynolds, “The association of scapular kinematics and glenohumeral joint pathologies,” Journal of Orthopaedic and Sports Physical Therapy, vol. 39, no. 2, pp. 90–104, 2009.View at: Publisher Site | Google Scholar
G. Jull, M. Sterling, D. Falla, J. Treleaven, and S. OLeary, Whiplash, Headache and Neck Pain: Research Based Directions For Physical Therapies, Elsevier, Edinburgh, UK, 2008.
S. A. Sahrmann, Diagnosis and Treatment of Movement Impairment Syndromes, Mosby, St. Louis, Mo, USA, 2002.
W. B. Kibler, T. L. Uhl, J. W. Q. Maddux, P. V. Brooks, B. Zeller, and J. McMullen, “Qualitative clinical evaluation of scapular dysfunction: a reliability study,” Journal of Shoulder and Elbow Surgery, vol. 11, no. 6, pp. 550–556, 2002.View at: Publisher Site | Google Scholar
A. R. Karduna, P. W. McClure, L. A. Michener, and B. Sennett, “Dynamic measurements of three-dimensional scapular kinematics: a validation study,” Journal of Biomechanical Engineering, vol. 123, no. 2, pp. 184–190, 2001.View at: Publisher Site | Google Scholar
P. W. McClure, L. A. Michener, B. J. Sennett, and A. R. Karduna, “Direct 3-dimensional measurement of scapular kinematics during dynamic movements in vivo,” Journal of Shoulder and Elbow Surgery, vol. 10, no. 3, pp. 269–277, 2001.View at: Publisher Site | Google Scholar
S. M. Ha, O. Y. Kwon, C. H. Yi, H. S. Jeon, and W. H. Lee, “Effects of passive correction of scapular position on pain, proprioception, and range of motion in neck-pain patients with bilateral scapular downward-rotation syndrome,” Manual Therapy, 2011.View at: Publisher Site | Google Scholar
S. Wegner, G. Jull, S. O'Leary, and V. Johnston, “The effect of a scapular postural correction strategy on trapezius activity in patients with neck pain,” Manual Therapy, vol. 15, no. 6, pp. 562–566, 2010.View at: Publisher Site | Google Scholar
L. Watson, S. M. Balster, C. Finch, and R. Dalziel, “Measurement of scapula upward rotation: a reliable clinical procedure,” British Journal of Sports Medicine, vol. 39, no. 9, pp. 599–603, 2005.View at: Publisher Site | Google Scholar
F. Struyf, J. Nijs, S. Mottram, N. A. Roussel, A. M. Cools, and R. Meeusen :, “Clinical assessment of the scapula: a review of the literature,” British Journal of Sports Medicine. In press.View at: Google Scholar
J. Nijs, N. Roussel, F. Struyf, S. Mottram, and R. Meeusen, “Clinical assessment of scapular positioning in patients with shoulder pain: state of the art,” Journal of Manipulative and Physiological Therapeutics, vol. 30, no. 1, pp. 69–75, 2007.View at: Publisher Site | Google Scholar
B. W. Hickey, S. Milosavljevic, M. L. Bell, and P. D. Milburn, “Accuracy and reliability of observational motion analysis in identifying shoulder symptoms,” Manual Therapy, vol. 12, no. 3, pp. 263–270, 2007.View at: Publisher Site | Google Scholar
P. McClure, A. R. Tate, S. Kareha, D. Irwin, and E. Zlupko, “A clinical method for identifying scapular dyskinesis, part 1: reliability,” Journal of Athletic Training, vol. 44, no. 2, pp. 160–164, 2009.View at: Google Scholar
D. C. Sobush, G. G. Simoneau, K. E. Dietz, J. A. Levene, R. E. Grossman, and W. B. Smith, “The Lennie test for measuring scapular position in healthy young adult females: a reliability and validity study,” Journal of Orthopaedic and Sports Physical Therapy, vol. 23, no. 1, pp. 39–50, 1996.View at: Google Scholar
J. Sim and C. C. Wright, “The kappa statistic in reliability studies: use, interpretation, and sample size requirements,” Physical Therapy, vol. 85, no. 3, pp. 257–268, 2005.View at: Google Scholar
T. Byrt, J. Bishop, and J. B. Carlin, “Bias, prevalence and kappa,” Journal of Clinical Epidemiology, vol. 46, no. 5, pp. 423–429, 1993.View at: Publisher Site | Google Scholar
J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics, vol. 33, no. 1, pp. 159–174, 1977.View at: Google Scholar
C. Fedorak, N. Ashworth, J. Marshall, and H. Paull, “Reliability of the visual assessment of cervical and lumbar lordosis: how good are we?” Spine, vol. 28, no. 16, pp. 1857–1859, 2003.View at: Publisher Site | Google Scholar
S. McPhail, P. Lane, T. Russell et al., “Telephone reliability of the Frenchay activity index and EQ-5D amongst older adults,” Health and Quality of Life Outcomes, vol. 7, p. 48, 2009.View at: Publisher Site | Google Scholar