Perception of the Use of the Direct Observation of Procedural Skills in Occupational Therapy Postgraduate Year Training in Taiwan: Survey of the Perspectives of Trainees and Supervisors

Liu, Yi-chia; Lee, Yi-chen; Huang, Pai-chuan; Lee, I-Hui; Lin, Keh-chung

doi:https://doi.org/10.1155/2023/8013086

Occupational Therapy International

On this page

Abstract Literature Review Methods Results Discussion Conclusion Abbreviations Data Availability Ethical Approval Consent Disclosure Conflicts of Interest Authors’ Contributions Acknowledgments References Copyright Related Articles

Research Article | Open Access

Volume 2023 | Article ID 8013086 | https://doi.org/10.1155/2023/8013086

Perception of the Use of the Direct Observation of Procedural Skills in Occupational Therapy Postgraduate Year Training in Taiwan: Survey of the Perspectives of Trainees and Supervisors

Yi-chia Liu,^1,2Yi-chen Lee,^3,4Pai-chuan Huang,^1,2I-Hui Lee,^1,5and Keh-chung Lin^3,6

Academic Editor: Claudia Hilton

Received18 Apr 2022

Revised27 Feb 2023

Accepted08 May 2023

Published24 May 2023

Abstract

Introduction. Assessment of clinical competence is a significant part of the training for young occupational therapists (OTs). Objective and systematic assessment allows both supervisors and trainees to be aware of the training objectives and monitor the progress. The direct observation of procedural skills (DOPS) is a work-based assessment to evaluate professional knowledge, skills, and attitude in clinical training. This study investigated the perspectives of OT educators and trainees on using DOPS and their discrepancy for OT postgraduate year (PGY) training. Methods. This study used a quantitative online survey. Eighty-six supervisors and 41 trainees of OT PGY training programs from 95 hospitals returned the questionnaire (a 90.5% return rate), and 64 supervisors and 30 trainees who used DOPS were analyzed. Outcomes included the practicality in using the DOPS in clinical settings, the ease of rating the DOPS, and advantages and the disadvantages of the DOPS. Results. Most respondents reported that completing one DOPS required at least 11 minutes for direct observation (11-40 minutes: teacher 92.2%; trainee 80.6%). Most respondents (teacher 96.9%, trainee 96.8%) had feedback after direct observation of DOPS, and about half of the feedback assessments took 5 to 10 minutes (teacher 53.1%, trainee 48.4%). Most OT educators and trainees agreed that clinical resources were sufficient and that DOPS matched with OT training goals, benefited OT competence training, and had a fair, objective, and consistent scoring system. Significantly higher percentages of OT trainees felt stressed in and satisfied with the DOPS assessment than trainers. Differences between teachers and trainees regarding easiness of rating DOPS items were not significant. Conclusion. Most OT educators and trainees agreed that DOPS was a practical and appropriate assessment for OT PGY training.

1. Literature Review

Clinical education is the most important part of health professional education [1] and also one of the strategies for developing continuing competence in occupational therapy (OT) [2]. The postgraduate year (PGY) training program for general medicine was implemented by the Taiwanese government in 2003 [3] and for other health care professions in 2007, including OT [4], pharmacy [5], and nursing [6], to address a need for improved professional training and the quality of health care services. The programs are implemented by the Joint Commission of Taiwan under the supervision and instruction of the relevant professional associations to equip PGY trainees with (1) professional knowledge, the ability to provide quality care, evidence-based skills, and resource management ability; (2) patient-centered, comprehensive, and holistic perspectives and skills; (3) professional ethical reasoning and communication skills; and (4) the ability to work in a team [7, 8].

Assessment of clinical competence is a significant part of the training for young OTs. Objective and systematic assessment allows both supervisors and trainees to be aware of the training objectives and monitor the progress. The assessment of practical performance in clinical conditions (does) is ranked as the highest level in Miller’s hierarchical model for the assessment of clinical competence [9]. Direct observation of procedural skills (DOPS) is a workplace-based assessment tool specifically designed to assess practical skills. It requires the assessor to directly observe a trainee conducting a procedure, document the evaluation in a checklist, and give feedback to the trainee based on objective findings [10].

Several studies have reported the experiences of supervisors and trainees with the DOPS in health professional training, such as medical, nursing, and veterinary students, and showed that examiners and examinees were both very satisfied with the DOPS [10, 11]. However, the feasibility and user experience of the DOPS for OT PGY training has not been examined. We investigated the perception of OT supervisors and trainees on the DOPS in Taiwan for PGY training. Therefore, the aims of our study were to investigate (1) the experience of using the DOPS in clinical settings, (2) the ease of rating clinical performance with the DOPS, and (3) the advantages and disadvantages of the DOPS for OT PGY training.

2. Methods

2.1. Procedures

This is a cross-sectional survey study. Data were collected from January 2017 to December 2018. We invited the teaching directors of each hospital to participate in this project from January 2017 to June 2017. A link to the online questionnaire was sent from July 2017 to December 2017. As of January 2018, the response rate was not good. The second recruitment and first reminder were carried out from January 2018 to July 2018. Follow-up reminders were sent one and two months later. Information in the database of the Ministry of Health and Welfare, Taiwan, was used to invite OT chiefs in charge of OT PGY training programs in 2017 to participate in this study. The online questionnaire was sent by email along with regular postal mail to participants. Participants signed an informed consent to be included in this study, and their data coding and analysis were anonymized. In our first part of the electronic questionnaire, there was the background of the questionnaire, the purpose of the research, the use of data, and the informed consent form. After downloading the filled forms, any identified information was removed, and then the deidentified data were used for data coding and analysis. When the online electronic questionnaire URL was given, each respondent was asked to report a unique user number to be used while filing the survey, which was convenient for distinguishing the source and avoiding repeated collection of operations.

2.2. Instrument

Prior to using DOPS for assessment, training included a 2 hours of general introduction to DOPS and a 2-hour scoring consistency workshop. This questionnaire was developed by two senior OTs (Y. C. Liu and P. C. Huang) based on the research questions and literature of medical education with work-based assessment in 2015 [11–14]. Five experts in OT PGY training were invited to assess and discuss the questionnaire in a focus group, and revised this questionnaire accordingly. A second group of five external experts in OT fieldwork education and research were invited to revise and finalize the questionnaire.

The final questionnaire had four sections. The first section included 11 items of basic sociodemographic information of the participants, including job position, age, sex, highest educational level, teaching/working years, specialty area, status of hospital, and service location. The second section used a 5-point Likert scale to rate 21 items related to experiences with using DOPS (i.e., practicality, subjective opinion, cover domain, and scoring system). The third section assessed trainer’s and trainee’s perception of ease of using the DOPS in rating the clinical performance of PGY OT trainees with a 5-point Likert scale of 11 items. The fourth part included 16 items of the advantages and disadvantages of the DOPS for the evaluation of OT PGY trainees.

2.3. Statistical Analyses

Demographic characteristics and distribution of participants’ responses in perception of DOPS are presented with descriptive statistics (i.e., frequency and percentage and mean and standard deviation). Comparisons between supervisors and trainees in experience of using DOPS, ease of rating the DOPS, and advantages and disadvantages of DOPS were assessed with the and tests. Data were analyzed with SPSS 18.0 software (SPSS, Chicago, IL, USA).

3. Results

Completed questionnaires were returned by 86 supervisors and 41 trainees of the 95 hospitals (individuals) who received the invitational letter and survey by emails and regular mail for a 90.5% return rate. The response rate was the number of questionnaires returned divided by the number of that have applied for PGY teaching hospitals times 100%. Data from 64 supervisors and 30 trainees who had used the DOPS and completed surveys were included in the analysis. 37.2% of the participants were located in the northwestern Taiwan, 12.8% of the participants were located in the middle western Taiwan, 43.7% of the participants were located in the southwestern Taiwan, and 6.4% of the participants were located in the eastern Taiwan. The distribution of location of participants represents the population of OT in Taiwan. The demographic characteristics of the participants are presented in Table 1. Significantly more women, being younger, and lower levels of education were noted in the trainee group than in the supervisor group. There were no differences in specialty areas and classification of hospital in the two groups.

Most respondents reported that completing one DOPS required at least 11 minutes for direct observation (11-20 minutes: teacher 26.6% () trainee 29.0% (); 21-30 minutes: teacher 34.4% (), trainee 38.7% (); 31-40 minutes: teacher 15.6% (), trainee 12.9% ()). Most respondents (teacher 96.9% (), trainee 96.8% ()) had feedback after direct observation of DOPS, and about half of the feedback assessments took 5 to 10 minutes (teacher 53.1% (), trainee 48.4% (n =15)). There were no differences in time for direction observation (, ) and for feedback (, ) between teachers and trainees.

The percentage of trainees who received DOPS training was 70% () compared with 56.3% () for teachers, which was significantly higher (, ). There was no significant difference (, ) in terms of discussion of DOPS rating criteria among teachers before formal assessment between teachers (75%, ) and trainees (90%, ).

Regarding experience with DOPS, more than 70% of trainees (range: 72.4% to 93.3%) and supervisors (range: 71.3% to 85.9%) responded “strongly agree” or “agree” with most items except for teacher’s rating of sufficient teachers (67.2%), assessing clinical reasoning ability (62.6%), consistency in scoring standard (61.0%), and both teachers’ and trainees’ rating of stressfulness in assessment (17.2% and 41.4%, respectively) and needing further training (54.7% and 55.1%, respectively). In addition, less than 10% of trainees (range: 0 to 6.8%) and supervisors (range: 0 to 9.4%) reported “strongly disagree” or “disagree” with most items except for teachers’ rating of need further training (20.6%) and both trainees’ and teachers’ rating of stressfulness in assessment (10.3% and 23.5%, respectively). The use experience of DOPS of the trainees and supervisors and their comparisons are presented in Table 2.

Results of the tests indicated that there were significant differences between trainees and teachers on most items of experiences of using the DOPS for OT PGY training except for items of low cost, better than traditional teaching assessment, fairness, objectiveness, and consistency in scoring standard. Significantly higher percentages of “strongly agree” were noted in trainee’s perception (range: 20.7% to 41.4%) than in supervisor’s perception (range: 1.6% to 12.5%).

Most respondents reported “simple” (trainee range: 10.7% to 34.5%; teacher range: 23.0% to 35.0%) or “neutral” (trainee range: 17.2% to 53.6%; teacher range: 27.9% to 47.5%) in the easiness of rating most of the DOPS in clinical settings. Results of tests indicated there were no significant differences between trainees and teachers on all items of easiness of rating DOPS for OT PGY training program. The easiness of rating DOPS as reported by the trainees and supervisors and their comparisons are presented in Table 3.

The advantages of the DOPS assessment tool were ranked as follows: direct observation of clinical performance (trainee: 96.6%, supervisor: 96.8%), immediacy of feedback (trainee: 75.0%, supervisor: 76.2%), flexibility (trainee: 75.9%, supervisor: 76.2%), finishing assessment in a short period of time (trainee: 69.0%, supervisor: 69.8%), no additional funding or resource burden (trainee: 62.1, supervisor: 55.6%), and no space restriction (trainee: 58.6%, supervisor: 55.6%). Having a valid and reliable scoring system may be an issue for DOPS in the assessment of clinical competence in OT PGY training because more than 75% of respondents reported subjectivity as one of the disadvantages. The results of the tests revealed no significant differences on perceptions in advantages and the disadvantages of DOPS between trainees and supervisors. Table 4 presents the comparison of advantages and disadvantages between trainees and supervisors.

4. Discussion

To the best of our knowledge, this is one of the few studies to examine the feasibility of the DOPS and explore the use experience, advantages, and disadvantages of the OT PGY training programs by including the perspectives of both trainees and supervisors. The results found that the DOPS tool was a practical measurement tool to assess the competence of OT PGY trainees.

Time for implementing DOPS and providing feedback is a significant factor for feasibility and acceptability [11]. Some studies [15, 16] found a lack of time or being seen as an additional workload in administrating the DOPS and giving feedback in physician training. This may reflect busy the clinical hours of the physicians and time constraints for using the DOPS in the clinical training of medical doctors. In contrast to findings of the above studies [11, 15, 16], approximately 90% of the respondents in this study spent at least 11 minutes for rating DOPS, and half of the respondents reported to have at least 6 to 10 minutes for feedback. Most of the trainees (86.7% agree or strongly agree) and supervisors (75% agree or strongly agree) agreed time for implementation of DOPS was sufficient for OT PCY training. In addition, more than 75% of trainees and supervisors agree that immediate feedback was one of the advantages of using DOPS in OT PGY training.

Training of rating DOPS is another significant issue for consideration in the assessment for OT PGY training. Half of the trainees (55.1%) and supervisors (54.7%) claimed that they need further training in using DOPS. Proper implementation is one of the significant factors related to the educational impacts of administrating DOPS [17] and relies on sufficient training in using assessment tools such as DOPS.

Approximately 40% of supervisors (35.9% neutral, 1.6% disagree, and 1.6% strongly disagree) did not support consistency in the scoring standard for rating DOPS in this study. Intercase variation may affect the reliability [11]. Training is needed not only on the procedure of administrating DOPS but also on scoring training on different patients or conditions. In addition, a study suggested rating of trainees’ performance by at least three supervisors observing at least two procedures each to achieve good reliability [11].

Consistent with previous studies [10, 11], some trainees (41.4% agree or strongly agree) felt that taking DOPS is a stressful experience. However, the trainee’s satisfaction with DOPS is high and even higher than the supervisor’s. It is possible that attending a work-based examination such as the DOPS is stressful, but the trainees appreciate this experience as learning and positive although stressful experience.

4.1. Implications for Occupational Therapy Education

The DOPS is a practical assessment tool for OT PGY training to monitor the trainee’s knowledge, skill, and attitude. Sufficient time for examination and feedback with DOPS is an essential component for successful implementation. Training for using DOPS is needed not only for the implementation procedure but also for scoring criteria across patients and conditions.

4.2. Strengths and Limitations

The strength of this survey included a nationally representative sample and a high response rate (more than 90%). Participants included OTs from different areas of Taiwan, clinical specialty areas of OT, and hospital settings.

The limitations included the following. This study recruited one trainer who was the major supervisor of the trainee during the period of study from each hospital. Other therapists who had once been supervisors before the study period only were excluded. Selection bias might exist. The experience of using the DOPS might be subject to recall bias because this survey was not filled out immediately after any specific DOPS assessment.

4.3. Future Study

In addition to trainees and supervisors in the OT PGY training programs, the DOPS is also used for the fieldwork education of senior OT students and their clinical supervisors. Future studies may investigate the user experiences of the senior OT students and their clinical supervisors in fieldwork education.

5. Conclusion

Most OT educators and trainees agreed that the DOPS is a practical and appropriate assessment for the OT PGY training. Sufficient time for rating the DOPS and feedback about performance is essential to learning effects. Training in using the DOPS is necessary and should include rating criteria with different conditions and patients with the same procedures.

Abbreviations

OT:	Occupational therapy
DOPS:	Direct observation of procedural skills
PGY:	Postgraduate year.

Data Availability

The data sets generated and analyzed during the present study are not publicly available due to participant confidentiality but are available from the corresponding author on reasonable request.

Ethical Approval

The study protocol was reviewed and approved by the Ethical Committees for Human Research of the National Cheng Kung University Hospital in Taiwan. The study procedures, risks, and benefits were explained to all participants.

The participants signed an informed consent before enrolling in this study.

Disclosure

A preprint has previously been published [18].

Conflicts of Interest

The authors declare that they have no competing interests.

Authors’ Contributions

YCLiu, PCH, and KCL designed this study and the questionnaire. YCLiu, YCLee, and IHL collected and analyzed data. YCLiu, KCL, and YCLee interpreted the results and drafted the manuscript. All authors read and approved this manuscript. Yi-chia Liu and Yi-chen Lee contributed equally to this work.

Acknowledgments

This study was supported in part by grants from National Cheng Kung University Hospital (NCKUH-10607022), National Health Research Institute (NHRI-EX109-10929PI, NHRI-EX110-10929PI, NHRI-EX111-10929PI), and the Health Research (Grant Number MOST 108-2314-B-002-165-MY3) in Taiwan. The authors wish to thank colleagues of the Department of Psychiatry, National Cheng Kung University Hospital, for their valuable comments on this work.

References

American Occupational Therapy Association, “Occupational therapy fieldwork education: value and purpose,” American Journal of Occupational Therapy, vol. 70, p. 7012410060, 2016.
View at: Google Scholar
American Occupational Therapy Association, “Standards of practice for occupational therapy,” American Journal of Occupational Therapy., vol. 75, p. 7513410050, 2021.
View at: Google Scholar
P. W. Hsu, M. J. Hsieh, R. H. Fu, J. L. Huang, M. C. Liao, and S. T. Lee, “Comparing the outcomes of different postgraduate year training programs in Taiwan,” Biomed J., vol. 38, no. 6, pp. 544–549, 2015.
View at: Publisher Site | Google Scholar
Y. C. Liu, P. C. Huang, Y. C. Lee et al., “The use of the mini-CEX in occupational therapy postgraduate year training programs in Taiwan,” Journal of Medical Education, vol. 25, pp. 49–60, 2021.
View at: Google Scholar
C. J. Wei, T. H. Lu, S. C. Chien et al., “The development and use of a pharmacist specific Mini-CEX for postgraduate year trainees in Taiwan,” BMC Medical Education, vol. 19, no. 1, pp. 165–172, 2019.
View at: Publisher Site | Google Scholar
Y. C. Yin, “The two-year post graduate training program for nurses: implementation status and personal perspectives,” Hu Li Za Zhi, vol. 60, no. 3, pp. 11–16, 2013.
View at: Publisher Site | Google Scholar
Joint Commission of Taiwan, Guidelines for the 2-Year Occupational Therapist Training Course, Joint Commission of Taiwan, 2018.
Taiwan Occupational Therapy Association, Core Competence, Taiwan Occupational Therapy Association, 2019.
G. E. Miller, “The assessment of clinical skills/competence/performance,” Academic Medicine, vol. 65, no. 9, pp. S63–S67, 1990.
View at: Publisher Site | Google Scholar
M. Erfani Khanghahi and F. Ebadi Fard Azar, “Direct observation of procedural skills (DOPS) evaluation method: systematic review of evidence,” Med J Islam Repub Iran, vol. 32, p. 45, 2018.
View at: Google Scholar
N. Naeem, “Validity, reliability, feasibility, acceptability and educational impact of direct observation of procedural skills (DOPS),” Journal of the College of Physicians and Surgeons–Pakistan, vol. 23, no. 1, pp. 77–82, 2013.
View at: Google Scholar
N. Bindal, H. Goodyear, T. Bindal, and D. Wall, “DOPS assessment: a study to evaluate the experience and opinions of trainees and assessors,” Medical Teacher, vol. 35, no. 6, pp. e1230–e1234, 2013.
View at: Publisher Site | Google Scholar
J. Hayashi, C. Christmas, and S. C. Durso, “Educational outcomes from a novel house call curriculum for internal medicine residents: report of a 3-year experience,” Journal of the American Geriatrics Society, vol. 59, no. 7, pp. 1340–1349, 2011.
View at: Publisher Site | Google Scholar
B. L. Hoseini, F. Jafarnejad, S. R. Mazloum, and M. Foroughipour, “Practical experience of the mini-CEX in undergraduate trainees,” Procedia - Social and Behavioral Sciences, vol. 83, no. 4, pp. 803–807, 2013.
View at: Publisher Site | Google Scholar
J. R. Wilkinson, J. G. Crossley, A. Wragg, P. Mills, G. Cowan, and W. Wade, “Implementing workplace-based assessment across the medical specialties in the United Kingdom,” Medical Education, vol. 42, no. 4, pp. 364–373, 2008.
View at: Publisher Site | Google Scholar
A. Miller and J. Archer, “Impact of workplace based assessment on doctors' education and performance: a systematic review,” BMJ, vol. 341, no. sep24 1, article c5064, 2010.
View at: Publisher Site | Google Scholar
A. C. Lorwald, F. M. Lahner, Z. M. Nouns, C. Berendonk, J. Norcini, R. Greif et al., “The educational impact of mini-clinical evaluation exercise (mini-CEX) and direct observation of procedural skills (DOPS) and its association with implementation: a systematic review and meta-analysis,” PLoS One, vol. 13, no. 6, article e0198009, 2018.
View at: Publisher Site | Google Scholar
Y. C. Liu, Y. C. Lee, P. C. Huang, I. H. Lee, and K. C. Lin, Perception of the Use of the DOPS in Occupational Therapy PGY Training in Taiwan: The Perspectives of Trainees and Supervisors.

Copyright

Copyright © 2023 Yi-chia Liu et al. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

PDF Download Citation

Download other formats

Order printed copies

Views

293

Downloads

283

Citations