Abstract
Background. Standardized patient (SP) clinical assessments have limited utility in assessing higher-level clinical competencies. This study explores the value of including simulation exercises and postencounter notes in an SP clinical skills examination. Methods. Two exercises involving cardiac auscultation and ophthalmic funduscopy simulations along with written post encounter notes were added to an SP-based performance examination. Descriptive analyses of students' performance and correlations with SP-based performance measures were obtained. Results. Students' abilities to detect abnormalities on physical exam were highly variable. There were no correlations between SP-based and simulation-derived measures of physical examination competency. Limited correlations were found between students' abilities to perform and document physical examinations and their formulation of appropriate differential diagnoses. Conclusions. Clinical simulation exercises add depth to SP-based assessments of performance. Evaluating the content of post encounter notes offers some insight into students' integrative abilities, and this appears to be improved by the addition of simulation-based post encounter skill exercises. However, further refinement of this methodology is needed.
1. Introduction
Assuring the acquisition of fundamental clinical skills by medical students has represented a major focus for medical educators during the past thirty years. A central facet of this process has been an effort to better understand how clinicians acquire, assimilate, and integrate clinical information. This work, in turn, has had significant implications for education directed toward the development of clinical expertise [1].
During this period, there has been a parallel effort to improve the assessment of clinical skills. The most significant development in this respect has been the remarkable evolution of standardized patient-(SP-) based clinical skills examinations, culminating in the implementation of a high stakes SP-based clinical skills examination as a component of Step 2 of the United States Medical Licensing Examination [2].
Extensive research has demonstrated the reliability, validity, and utility of SP-based clinical skills examinations. Yet, concerns about the limitations of this assessment methodology persist [3–5]. In particular, questions remain as to whether SP clinical-based assessments can adequately address higher levels of clinical functioning, including competencies such as detecting the presence of abnormal findings on physical examination, and students’ abilities to assimilate, integrate, and utilize clinical data during the course of a clinical encounter [6].
Due to constraints that limit the availability of faculty in the United States, most SP-based clinical skill examinations use SP completed checklists to assess student data gathering skills (history and physical examination). In contrast to more global approaches to assessment that utilize expert observers to rate student performance, such SP-derived checklists do not necessarily allow for any hierarchical interpretation of student thought processes as they proceed in gathering historical data, and how these data affect how students go on to formulate the content of their physical examinations. In fact, checklist-based scoring may actually penalize students who demonstrate more advanced clinical reasoning [7]. Furthermore, physical examination checklists are only able to score the “content” of the students’ exams. Unless there are specific prompts built into the checklist, the SP has no way of knowing whether the student saw, heard, or felt what she/he was actually supposed to detect.
In the United States, the typical SP-based clinical skills examination consists of multiple cases, each followed by a “postencounter exercise.” Postencounter exercises usually involve the student writing a clinical encounter note or performing some other type of knowledge, skill, interpretive, or reflective exercise. While the SP-based components of clinical skill assessment examinations have been rigorously studied, comparatively little attention has been paid to postencounter exercises. The purpose of this study is to explore the added value that two specific post-encounter exercises, completion of a simulation-based clinical skills exercise and completion of a written clinical encounter note (PEN), might bring to assessing medical students’ physical diagnosis and clinical information integration competencies.
2. Methods
This study was reviewed and approved by the University of California Irvine Human Subjects Institutional Review Board. Data collection for the study occurred in conjunction with the annual University of California Irvine School of Medicine (UCI SOM) administration of the California Consortium for the Assessment of Clinical Competence Clinical Practice Examination (CPX). The consortium consists of medical educators from the eight California allopathic medical schools. The consortium CPX is an eight-case SP-based clinical assessment developed collaboratively by the California schools to assess the clinical skills of rising fourth year medical students. The examination has demonstrated psychometric characteristics suitable for a high stakes assessment during its 12 plus years of existence, and passing it is a graduation requirement at each school.
This study focused on postencounter exercises following two specific cases: a middle-aged female who presented to “clinic” following a prolonged episode of chest pain and a young adult female who presented to “clinic” with a severe headache. Each standardized patient encounter was fifteen minutes in duration. Following each clinical encounter, students were required to write a postencounter “progress” note. In addition, following the chest pain case, students were asked to describe the findings present on an auscultatory simulation recording that featured the typical findings associated with Aortic Stenosis [8]. Following the headache case, students were also asked to describe findings present on a normal fundus and on a fundus demonstrating changes consistent with acute papilledema using an Ophthalmology Funduscopy simulator [9].
The auscultation and funduscopy postencounter skill exercises, and the diagnoses produced during these exercises, were scored using a set of predefined criteria for each case. The postencounter note history, physical examination, and differential diagnosis sections were scored for content using a “keywords and phrases” identification process modeled after the approach described by Ben-David et al. [10]. The postencounter note keywords and phrases were identified by the case authors, and the skill exercise scoring criteria were identified by the individual faculty responsible for creating each skill exercise. Both the notes and the skill exercise responses were captured electronically, numerically coded, and scored in a de-identified manner by one of two study authors (SA and MDP).
In order to qualitatively explore the potential value of the simulation-based skill exercises, descriptive results for each exercise are reported. To determine if these exercises resulted in added value to the SP-based assessments of physical examination skills, scores for the SP-based physical exam checklists were compared to the simulation-based skill exercise scores (a lack of correlation between the SP-based and simulation exercise-based scores indicating that the skills exercise added additional value when taken together with the SP-based score). To determine whether the data acquisition competncies of students, as measured by SP scores of history and physical exam performance, was accurately reflected in their postencounter notes, as measured by history and physical exams sections, was similarly assessed by comparison of scores. Finally comparisons of SP-derived history and physical exam scores to postencounter note differential diagnosis scores, as well as simulation exercise-based skills scores, postencounter note differential diagnoses scores, and simulation-based differential and final diagnosis scores were compared in order to assess whether higher levels of skill proficiency as reflected by either SP-or simulation-based scores produced correspondingly higher differential or final diagnoses scores. In order to perform these comparisons, Bivariate correlation coefficients (Spearman’s Rho) were computed to examine associations between the various scores. All analyses were performed using SPSS version 13.0 (SPSS, Inc., Chicago, IL, USA).
3. Results
Eighty-seven students were eligible to participate in the study. Of these students, 84 (97%) completed the chest pain auscultation skills exercise, 52 (60%) completed the chest pain postencounter progress note, 85 (98%) completed the funduscopy skill exercise, and 51 (59%) completed the headache postencounter note. During early examination sessions, students were noted to be experiencing some difficulty completing both elements of the postencounter exercises. During subsequent sessions, students were advised to complete the skill exercise first and then complete the postencounter note as time allowed. For purposes of this study, all students completing the skill exercises are included in the descriptive analysis, and all students who completed both the skill exercises and the postencounter notes for each of the each of the cases were included in the comparative analyses.
The descriptive analysis of the students’ postencounter simulation-based skills exercise performance found the following: 91% of students identified the presence of a murmur, 77% correctly identified the timing of the murmur as systolic, 41% correctly identified the character of the murmur as being crescendo-decrescendo, and 37% correctly identified the etiology of the murmur as being likely due to aortic stenosis. Only 2% of students of students linked the presence of aortic stenosis to the occurrence of acute coronary syndrome. In terms of funduscopy, 44% of students were able to correctly identify a normal fundus, while 26% were able to correctly identify a fundus featuring the characteristic findings of papilledema. Of the students who correctly identified a normal fundus, 90% associated this finding with a correct diagnosis of migraine headache. Of the students who correctly identified papilledema, 89% appropriately modified their diagnosis to a headache caused by an acute increase in intracranial pressure. However, none of the students who incorrectly described either the normal or abnormal fundus arrived at appropriate post exercise diagnoses.
Bivariate correlation coefficients are presented in Table 1. The SP scores include those from history and physical examination checklists. Student postencounter scores include those for skill exercises, progress note history, and physical examination, and the differential diagnosis sections. There were no correlations found between SP-based physical examination scores and skill exercise-based physical exam scores on either case. However, for the headache case there were significant positive correlations () for the physical exam items documented in the postencounter note and the postencounter note differential diagnosis score, the postencounter skill exercise score and the postencounter note differential diagnosis score, and the postencounter skill exercise score and the final diagnosis score. Significant () positive correlations on the chest pain case were found for the postencounter skill score and the final diagnosis score, and the differential diagnosis portion of the postencounter note score and the postencounter skill exercise final diagnosis score.
4. Discussion
The relatively small size of our student sample, the limited number of postencounter exercises studied, and the fact that the study was conducted in a single institution, all dictate that the findings of this study are tentative in nature and should be interpreted with caution. In our view, the lack of correlation between the SP-derived physical examination scores and the postencounter simulation skills exercise-based physical exam scores suggests that each measures a different aspect of clinical competency. At the very least, the results reported here pertaining to the use of postencounter simulation-based skills exercises in a standardized patient-based clinical skills exam extends the limited literature in this area, adds depth to our understanding of the complexity of ascertaining clinical skills oriented competencies, and lends additional support to expanding the use of hybrid standardized patient and simulator-based cases [9, 11, 12]. In short, students need to know what to examine, how to perform the examination, and how to recognize abnormalities detected during the examination.
That the students faired relatively poorly on the simulation-based skills exercises is not entirely surprising. Other investigators have demonstrated that mastering a complex skill such as cardiac auscultation can be daunting [13–15]. Wilkerson and Lee reported an interesting experiment comparing student performance when students are asked to examine a patient presenting with chest pain and then asked to simply perform a cardiovascular exam. Their results suggest that such performance for late third year early fourth year medical students is actually consistent with being at “can show what to do” when prompted but not yet at “the does” stage of clinical skill development [16]. In other words, it may simply in our institution that, at this stage of training, students have not had sufficient practice with these skills in clinical contexts so that their performance becomes “automatic.”
Boulet and colleagues have reported extensively on postencounter clinical notes primarily in conjunction with the development and implementation of the ECFMG clinical skills assessment [10, 17–19]. In contrast to our findings, their results indicated relatively high levels of correlation between SP measurements of student data gathering abilities and expert ratings of postencounter progress notes. However, their data analyses combined SP history and physical examination scores and expert postencounter progress note scores from multiple cases, and their progress note rubric used a more holistic approach that considered organization and language use which was pertinent to the ECFMG objective of assuring that the candidate could produce understandable written communication. In our study, we were most interested in assessing how the postencounter clinical note might be useful in attempting to understand student clinical reasoning, at least in terms of whether the student actually performed and then documented the performance of case-specific critical tasks. A study more analogous to the one reported here was undertaken by Macmillan and colleagues in conjunction with development of USMLE Step 2 CS [20] In their study, they compared item-specific content derived scores for the history and physical exam sections of postencounter progress notes to item-specific SP-derived history and physical exam scores for the corresponding clinical cases. Their study produced mixed results with good overall correlation in four cases and poor overall correlation in two cases. They also found that history items in general were more likely to correlate than were physical examination-related items. They concluded that it was likely that the SP-derived scores and postencounter progress note scores are also measuring somewhat different aspects of students’ performance. They went on to suggest that poor levels of agreement between postencounter progress note scores and SP derived scores reflect students’ difficulties in appropriately assimilating and interpreting clinical data.
Although our study includes a very limited number of cases and a small student sample size, making comparisons to the work of Boulet and colleagues, and Macmillan and colleagues at best tenuous, we believe our data does also suggest that history and physical examination performance as measured by SPs does not correspond to scores derived from students’ written documentation of these activities. Our data do suggest, at least for the headache case, that postencounter notes can afford some insight regarding the students’ ability to assimilate and integrate clinical data but, in contrast to Macmillan and colleagues, we found a stronger correlation between documentation of physical exam information and arrival at an appropriate differential diagnosis. However, this may simply reflect variability between the types of cases utilized in the two studies. Our data also suggest that a significant relationship exists between students’ abilities to actually detect an abnormality on physical examination and then arrive at a correct diagnosis. This seems intuitively obvious but may not be so, particularly if it reflects the more proficient students’ ability to use hypothesis generation to direct them toward searching for specific abnormalities when performing a physical exam. Indeed our results seem to reinforce the notion that such competencies involve performing the correct task, performing the task correctly, and interpreting the results appropriately. The combination of student encounters with standardized patients, simulation-based skill exercises which introduce significant clinical findings to the case, and postencounter clinical notes, when taken together, may provide an important instrument for exploring students’ clinical reasoning abilities. A next logical step would be to analyze the organization of the students’ patient interview and exam with the aim of establishing some sense of link between what the student asks or performs and at least an implicit sense of the students’ real-time clinical reasoning process (3). This will likely necessitate more intensive faculty interaction with students immediately after the clinical counter, or perhaps the use of structured student postencounter self-reflection exercises.
Implementation of USMLE Step 2 CS has left many educators wondering whether there is a continuing value to offering institutionally administered SP-based clinical skills examinations, beyond the opportunity they afford for students to practice prior to taking Step 2CS. Petrusa has argued that standardized patient-based examinations should be taken to a next level with “professional development, acquisition of expertise, and team functioning” offering new areas for potential exploration [21]. Understanding the complexities of higher-level clinical skills oriented competencies will be one major element of this research agenda. The study reported here represents a small and preliminary step in that direction.