Research Article

Development and Testing of Screen-Based and Psychometric Instruments for Assessing Resident Performance in an Operating Room Simulator

Table 4

Interrater agreement for the Crisis Management Checklist.

ItemCircuit disconnectBradycardiaEndobronchial intubationHypovolemiaPulmonary embolismLight anesthesiaAll scenarios

State change detection0.699/0.7320.521/0.7490.807/0.8390.856/0.8170.495/0.4890.121/0.2730.639/0.674
 Timely/prompt detection0.7300.4630.7910.7330.3330.3010.593
 Complete detection0.6400.8380.7290.6180.5060.6490.655
 Missed detection0.2480.5180.0000.487(−)0.3660.088
Situational awareness0.835/0.8380.946/0.9520.834/0.8560.773/0.7730.907/0.9370.798/0.8030.844/0.852
 Complete/correct differential0.7530.8850.8200.7240.8890.7900.821
 Prioritized differential list0.7100.9130.7900.8250.8570.7940.807
 Reassesses situation0.7330.3870.7390.5330.6470.7080.620
 One or more incorrect diagnoses0.9100.6540.1580.6270.5570.565
Therapy/resource utilization0.917/0.9170.886/0.8860.705/0.7110.888/0.8880.763/0.7840.934/0.9340.842/0.852
 Timely therapy0.9450.6470.5940.7950.7980.8710.793
 Prioritized actions0.8570.6810.5700.9140.7330.8400.780
 Appropriate therapy/action0.7700.4320.6090.6810.5150.8510.658
 One or more inappropriate actions0.0000.4870.314
Total0.903/0.8980.954/0.9650.871/0.8600.917/0.9310.870/0.9080.850/0.8810.878/0.890

Interrater agreement assessed by calculating the intraclass correlation coefficient (ICC) using a two-way mixed effects for consistency between two expert rater responses.
Not significant.
Item responses were all zero.
The second ICC does not include the “missed detection” item.
The second ICC does not include the “one or more incorrect diagnoses” item.
The second ICC does not include the “one or more inappropriate actions” item.
The second ICC does not include the previously excluded items.