Abstract
International experiences are increasingly popular among medical students, but few have been rigorously evaluated. We used ESM to characterize the experience of medical students participating in the one-month University of Chicago Geographic Medicine Scholars Program (GMSP) in Hyderabad, India. Participants wore watches programmed to randomly beep eight times daily, during the first and third weeks. When alerted, students reported their activity and rated their level of engagement and emotion in booklets. All 11 student participants responded to 59% of 816 alerts. 41% of beeps occurred during GMSP formal structured activities (i.e., lecture, clinic). Students reported structured activities to be more interesting (6.05 versus 4.14; ), more important to their future goals (6.48 versus 5.71; ), more challenging (3.75 versus 2.48; ), and more enjoyable (6.08 versus 5.36; ) than unstructured activities. Our results show that future global health efforts should augment the use of structured activities.
1. Introduction
International experiences are becoming increasingly popular among medical students. To respond to this increased interest, academic health centers began introducing some component of global medicine into their curriculum. Whereas a mere 6% of medical students participated in global activities in 1984, up to 40 percent were anticipated to participate in 2011 [1]. Evidence to date evaluating global health programs for medical students have reported increased skill and confidence in students and increased appreciation for cross-cultural communication [2]. A variety of retrospective surveys and interviews have demonstrated that medical students find these experiences to be meaningful [3, 4]. A study that employed a follow-up survey for senior medical students who had participated in a global health rotation demonstrated that the experience was highly influential on medical careers (67%) and future intentions to work overseas (60%) [5].
In general, there is the assumption that international health experiences enhance personal development and academic potential. However, few medical student programs in global health have been rigorously evaluated. Feedback from students following these experiences has largely focused on the positives of their experience—largely autonomous medical training in a resource-poor environment, traveling to a different country, cultural enrichment, and tourism during downtime [4, 6]. One possible reason that predominantly positive benefits of these experiences are reported is that most studies ask students to retrospectively rate their global health experience. Retrospective reports may be susceptible to recall bias that favors reporting only positive experiences [4]. In addition, students may also recognize the need to provide positive evaluations to ensure that programs continue to be offered.
Unfortunately, it is likely that students face hardships during these global health experiences that are not reported such as hazardous living conditions, dangerous exposures, and fear of safety and illness. These hardships may be especially prevalent when students are not in structured out-of-country program activities. It is also possible that students acclimate to their new environment as they spend more time abroad in their rotation. Therefore, it is likely that student perceptions may change during the duration of the global health experience. To capture data that may reflect a student’s true experience during global health experience, real-time experiential survey methods may be especially helpful. Real-time survey data can not only minimize the potential of recall bias, but also enable programs to examine how student perceptions of their global health experience change depending on the activities they are participating in. For example, it is currently unclear if medical student interest in global health rotations are fueled by the inherent interest in structured global health activities or by the desire to travel abroad more generally. Real-time sampling may provide a way to differentiate between student perceptions of formal structured program activities versus unstructured activities (i.e., tourism, personal activities). To date, no study has used real-time experiential sampling to collect data on medical student experiences during global health programs. Therefore, the aim of this pilot study is to use real-time Experience Sampling Method (ESM) to assess student feelings and activities when participating in a structured global health experience during medical school.
ESM is a data collection method to capture self-reported, subjective, and objective real-time data. Developed by Mihaly Csikszemenmihalyi, ESM provides detailed information about participants’ subjective interpretations of their experiences, including their levels of interest and engagement. ESM is especially useful for studying how individuals feel when participating in specific activities. ESM was initially used in education to study how teenagers feel when in class, school activities, with friends, at home, or on the job [7]. ESM has been adapted to healthcare and medical education to study a variety of topics including patient mood, resident fatigue, as well as physician activities [8–12]. Due to subject burden with ESM, published acceptable response rates range from 11% to 60%. Therefore, the advantage to using ESM is that real-time data is more reliable that retrospective data. The disadvantage is that it is cumbersome to administer and there is, by the nature of the method, a greater fraction of missing data, when compared to more traditional cross-sectional methods.
The study aims to determine whether ESM can be used to assess students’ real-time feelings and activities during a global health activity. This may include, but is not limited to, concentration, difficulty, interest, challenging, expectations, enjoyment, goals, success, tired/alert, bored/excited, passive/active, sad/happy, and sick/healthy. And from this gathered data, we hope to determine how to improve future global health activities.
2. Methods
2.1. Participants
In April of 2006, the University of Chicago Department of Medicine developed the Geographical Medical Scholars Program (GMSP)—which provided Pritzker medical students a domestic lecture series on global health, clinical rotations in travel clinic and the emergency department and culminating with an international out-of-country experience. GMSP participants were specifically selected based on their skills, goals, and global health aspirations reflected through an application and interview.
In 2007, a total of seven students participated in the program (6 female, 1 male). Of those students, four were second-year medical students, one was a third-year student and the remaining three were fourth-year students. There were six women and one man. In 2008, a total of five students, participated (2 female, 3 male). There were three second-year medical students and two fourth years.
2.2. Program
Students enrolled in the 2007 and 2008 GMSP spent their one-month out-of-country experience in Hyderabad, India. Students spent most of their time the first two weeks on site at Share India, a partnering institution, where they rotated through the hospital’s wards, emergency room, and Intensive Care Unit (ICU). They also participated in the rural community medicine program, perinatal care program, rotated through Fever Hospital, Sivananda Rehabilitation Home (Leprosy clinic), and two Chest/Tuberculosis Hospitals. The students were also able to participate in one-day activities, such as immunization clinics held in remote villages and volunteering at an HIV orphanage. Students were encouraged to propose a project or an interest that they would focus most of their time on during the final two weeks of the program. The program was supervised by three full-time University of Chicago faculty members and two house staff who accompanied the students during the program.
2.3. Procedure
To conduct ESM, we opted for wristwatch alerts using Digital Casio watches (Casio Databank, Model number DBC-150). The watches were set to beep randomly 8 times a day and 5 days per week during week 1 and week 3 of a 1-month experience. These weeks were chosen to capture the start of the experience, provide a rest period, and then recapture data after acclimation in the environment.
2.4. Measures
To develop the Experience Sampling Survey for GMSP, we modified an existing survey that captures student experience with a research program in the US Domains for objective data collection included the student’s location, activity, and time. Students were also asked to complete subjective information including, but not limited to whether the activity was GMSP related (clinic, lecture, etc.) or not (i.e., personal time, tourism or cultural enrichment, travel, and social activities), the engagement of the student (i.e., interest, importance to future, concentration, etc.) in the activity from 0 (not at all) to 9 (very much). They also reported their perception of feelings (self and other), concentration, thoughts, self-assessment of performance, perceived assessment of self by others (students, faculty), and their emotional state using dichotomous scales ranging from one extreme to another (i.e., healthy to sick, alert to tired, etc.). “Perception of self and other” and “perceived assessment of self by others” gave the student the opportunity to explain how he/she felt about themselves at a given time and how they felt others may have felt about them at a given time. We also incorporated domains relevant to an international rotation, such as fear of illness and personal safety.
Upon acceptance into the GMSP for 2007 and 2008, all medical students were approached to participate in ESM. A trained research assistant (M. Darcy, R. Farr) provided all students with a formal introduction to ESM followed by an explanation of the study. At this time, written informed consent was also obtained. To train students to use ESM, all students were asked to practice wearing the wristwatch for 2 days and completing the booklets while they were in the United States, before the out-of-country experience. Students were able to report back any glitches they noticed or questions they had, so that their concerns could be addressed before their departure. Prior to departure, one of the students was identified by program faculty (J. Schneider, V. Arora) and approached to be the “ESM Champion.” The ESM Champions (A. Ahmad, I. Nwadei) were assigned to manage the day-to-day on-site implementation of ESM and to ensure compliance.
For the out-of-country experience, watches were programmed in Chicago for India time by trained research assistants and then given to one of the ESM student champions for transport to India. Upon arrival to Hyderabad, the ESM Champion distributed watches and booklets to the student participants. Each ESM participant was given a 1-week supply of booklets at the start of week 1 and week 3. Each booklet contained 8 identical Experience Sampling surveys that would be filled out during a day. Stress was placed on the importance of filling out data in booklet quickly after each beep (<5 min) to limit retrospective bias. Students were also instructed to keep their booklets and answers private so as not to influence others or generate interpersonal interpretations of the experience at the time of the beep. To encourage student participation and motivation, a faculty champion (J. Schneider) also wore the watch and completed ESM questionnaires. Books were collected daily by the ESM Champion and returned to the United States to the investigative team for data entry and analysis.
2.5. Statistics
Descriptive statistics were used to summarize response rates and overall experiences. Activities were coded as either GMSP or non-GMSP using data from participants and input from ESM champions and faculty. GMSP activities included, but were not limited to direct and indirect participation in patient care in the hospital or rural clinics, attending lectures, travel to and from GMSP activity sites, preparation of GMSP presentation materials, attending health care conferences, and volunteering at GMSP-approved sites.
Because our data were clustered by subject, we used multivariate linear regression models using subject as the fixed effect. All models were also adjusting for day of week and ESM period (week 1 versus 3) to ascertain the association between formal structured GMSP activities and engagement/emotional state. In addition, trends over time with increasing exposures were compared by examining day of week and week effect covariates in regression models. For binary outcome variables, conditional logistic regression also controlling for subject, day of week, and ESM period was used. Due to multiple comparisons, statistical significance after Bonferroni correction was defined as . This study was approved by the University of Chicago Institutional Review Board.
3. Results
During the 2007 and 2008 GMSP, all eleven of eleven students participated in ESM. This included six second years, two third years, and three fourth years. A total of 59% of beeps were answered (481/816) overall with a response rates per students varying from 25% to 85% ( by chi-square test). Of note, only 3 students had a response rate below 50% (Table 1). Response rates for second years were lower (52%) than for third and fourth years (69%) (.67, ). This includes 259 of 416 beeps (62%) in 2007 and 222 of 400 beeps (56%) in 2008. A significant year effect was found, such that students were more likely to answer a beep in 2007 versus 2008 , ). Fewer beeps were also answered in the second period for both years highlighting the possible fatigue with this method (week 1 65% versus week 3 53%, ; ).
During 20% of alerts, students reported use of technology (i.e., email, phones, cameras). Students also reported fear of getting sick 13% of the time and someone helping them find their way around (also 13% of the time). Less than half (41%) of beeps actually occurred during structured GMSP activities (i.e., lecture, clinic, etc.). The majority of structured activities in GMSP included patient care (20%), lecture/education (43%), or talking/discussion (22%). The bulk of unstructured activities mapped to subcategories of personal activities (i.e., eating, etc.) (40%), traveling (14%), computer work (7%), talking/listening (14%), and tourist activity (3%). There was no difference in rates of technology use, fear of illness, or in someone helping find way around between formal structured GMSP activities and unstructured.
In multivariate analysis, students were found to perceive GMSP activities to be more interesting (6.05 versus 4.14; ), more important to their future goals (6.48 versus 5.71; ), more challenging (3.75 versus 2.48; ), and more enjoyable (6.08 versus 5.36; ) than non-GMSP activities (Table 2). Although not statistically significant, students felt more excited (4.80 versus 4.53; ), more alert (4.65 versus 4.25; ), and more proud (4.67 versus 4.47; ) during formal structured GMSP activities versus unstructured GMSP activities (Table 3).
In restricting the analysis to the responses for students when they were in GMSP activities, we found that patient care was the most engaging activity within the GMSP program. In fixed effects regression models controlling for day of week and period, students in patient care GMSP activities were more likely to report higher levels of interest ; 95% CI 0.93–2.54, ), enjoyment (; 95% CI 0.60–2.01, ), and relation to future goals ( = 1.84; 95% CI 1.03–2.66, ) than when they were participating in other formal activities in GMSP.
There were also significant relationships between engagement and emotions and time in country. Compared to week 1, students in week 3 reported their activities as less enjoyable (week 1 6.07 versus week 3 5.22; ) and less important to their future (week 1 6.44 versus week 3 5.61; ). In addition, students in week 3 were more likely to report feeling healthier (week 1 4.46 versus week 3 4.88; ). Although not statistically significant, students were less afraid of getting sick in week 3 (adjusted OR 0.42, 95% CI 0.24–0.77, ) and less likely to report someone helping them find their way than in week 1 (adjusted OR 0.49, 95% CI 0.26–0.91, ). There was no significant interaction between period and formal GMSP activity.
4. Discussion
To our knowledge, this is the first real-time assessment of a medical student experience in global health using experience sampling method. This small pilot study demonstrates that medical student engagement in global health experiences can be assessed using real-time assessment methods, such as Experience Sampling Method. Our preliminary findings demonstrate that students are more engaged when participating in formal structured program activities, such as clinical work or education, than unstructured activities in their out-of-country experience. Specifically, during a global health rotation, students report a greater sense of excitement, pride, and fulfillment when participating in structured program activities compared to unstructured activities. Moreover, students’ engagement appears to wane over time. Particularly, students are less engaged in their out-of-country global health experience in the middle of the program, compared to the start of the program. Students report their activities as less enjoyable, less important to their future.
It is important to explore the possible mechanisms and potential implications for these findings. It is reassuring that, during a global health experience, formal structured activities were rated higher than unstructured activities. This is not surprising, because structured activities include things that are medically related and academically challenging. Furthermore, in our subset analyses examining alerts only from GMSP activities, students were more engaged when they were in patient care GMSP activities. This finding may be related to the interest of preclinical students in seeking clinical experiences, which may be easier to participate in when abroad then during their preclinical curriculum. Future global health programs for medical students should ensure maximal participation in formal structured activities, including patient care, to capitalize on student growth in these experiences.
Moreover, as students acclimate to their environment, fear of illness may decrease. However, the lower levels of excitement about the experience in week 3 are concerning. At a time when students should be acclimating to their environment and at greatest productivity in their out-of-country experience, they appear less engaged. This may be the case for several reasons. First, it is possible that as the initial excitement about being abroad fades, student engagement wanes as well. For example, students may feel bored after the newness and challenge of the global health rotation has worn off. It is also possible that the activities of week 3 were less interesting. Lastly, ESM burden, as evidenced by the lower response rate, may have played a role in the decreased interest of participants in week 3.
Overall, the response rate was 59% for ESM over the two years being evaluated and may have been so for several reasons. For instance, a student may not have heard the beep, the watch may have malfunction and skipped a beep(s), the student may have taken the watch off for a period of time, or the student may have been too busy to stop and fill out the survey or may have felt it would be intrusive or impolite to do so at the time. Taking this into account, the response rate was still very high for this methodology, and thus, we were able to obtain a relatively significant amount of information with ESM.
There were several limitations to this study. First and foremost, it was an evaluation of one global health program at one medical school. This limits generalizability, but we encourage additional programs and sites to also use this method to assess student engagement in global health activities for comparative purposes. Second, although we have data on many different time points, they are collected as part of a pilot study from a total of 11 individuals, which limits generalizability. Third, this study was largely descriptive and not an intervention study, but provides a starting point for considering future intervention studies comparing different global health programs. Formal validation and reliability testing of our ESM instrument has not yet been established. In addition, the study may also contain an element of reporting bias as the students involved in the evaluation were those interested in global health activities. Therefore, the reported comments may have had more positive elements than those obtained from students who were not interested in global health. Although it is possible that students may have erred on the side of reporting more positive reporting as a form of bias, they were encouraged to be honest and assured that their comments would be kept confidential and in no way would be used as a form of evaluation of their work. Finally, the data missing from the period of time that students did not respond to the beep could introduce an element of bias if this were more likely to occur during certain activities. For example, we cannot discern response rates by whether a student was in a structured activity or not, because the response to the beep is necessary to determine how to code the activity.
5. Conclusion
As global health rotations and programs are becoming a more common component of medical education, it is becoming increasingly important to have a methodology in place to evaluate these experiences. Using Experience Sampling Method in our pilot study, we showed that formal structured global health activities provided students with a greater sense of excitement, pride, and fulfillment than unstructured activities. These results highlight the importance of global health programs to include more structured activities. In addition, responses from students in our study suggest that global health programs for medical students should consider how to prepare and counsel students for their experiences. From our results, this would specifically mean addressing fear of illness and maintaining student engagement after the initial excitement of out-of-country wanes so that they can feel more productive as they acclimate to their environment. Moreover, attempts could be made to assure that more academically challenging activities be available from week 3 onwards, to keep students engaged, interested, and motivated to continue learning.
Among other methods, Experience Sampling Method can be a powerful tool to help assess student engagement in global health experiences in real time. ESM may be a useful adjunct to more conventional data collection methods, such as interview and observation. future work will aim to triangulate results of ESM with other more conventional methods.
Conflict of Interests
The authors declare no conflict of interests.
Authors’ Contribution
Drs. A. Ahmad, J. Schneider, V. Arora, and Ms. I. Nwadei, R. Farr, and R. Thal, and Mr. M. Darcy all made substantial contributions to the conception and design, acquisition of data, or analysis of interpretation of data, have all been involved in drafting the article or revising critically it for important intellectual content, and have given final approval of the version to be submitted.
Ethical Approval
The IRB approved this study.
Acknowledgment
This study was funded by the Pritzker School of Medicine. The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the paper.