Development and Testing of Screen-Based and Psychometric Instruments for Assessing Resident Performance in an Operating Room Simulator
Table 4
Interrater agreement for the Crisis Management Checklist.
Item
Circuit disconnect
Bradycardia
Endobronchial intubation
Hypovolemia
Pulmonary embolism
Light anesthesia
All scenarios
State change detection
0.699/0.732
0.521/0.749
0.807/0.839
0.856/0.817
0.495/0.489
0.121/0.273
0.639/0.674
Timely/prompt detection
0.730
0.463
0.791
0.733
0.333
0.301
0.593
Complete detection
0.640
0.838
0.729
0.618
0.506
0.649
0.655
Missed detection
0.248
†
0.518
0.000
0.487
(−)0.366
0.088
Situational awareness
0.835/0.838
0.946/0.952
0.834/0.856
0.773/0.773
0.907/0.937
0.798/0.803
0.844/0.852
Complete/correct differential
0.753
0.885
0.820
0.724
0.889
0.790
0.821
Prioritized differential list
0.710
0.913
0.790
0.825
0.857
0.794
0.807
Reassesses situation
0.733
0.387
0.739
0.533
0.647
0.708
0.620
One or more incorrect diagnoses
0.910
0.654
0.158
†
0.627
0.557
0.565
Therapy/resource utilization
0.917/0.917
0.886/0.886
0.705/0.711
0.888/0.888
0.763/0.784
0.934/0.934
0.842/0.852
Timely therapy
0.945
0.647
0.594
0.795
0.798
0.871
0.793
Prioritized actions
0.857
0.681
0.570
0.914
0.733
0.840
0.780
Appropriate therapy/action
0.770
0.432
0.609
0.681
0.515
0.851
0.658
One or more inappropriate actions
†
†
0.000
†
0.487
†
0.314
Total
0.903/0.898
0.954/0.965
0.871/0.860
0.917/0.931
0.870/0.908
0.850/0.881
0.878/0.890
Interrater agreement assessed by calculating the intraclass correlation coefficient (ICC) using a two-way mixed effects for consistency between two expert rater responses. Not significant. Item responses were all zero. The second ICC does not include the “missed detection” item. The second ICC does not include the “one or more incorrect diagnoses” item. The second ICC does not include the “one or more inappropriate actions” item. The second ICC does not include the previously excluded items.