Opportunities and challenges of using simulation technology and wearables for skill assessment

Introduction
Virtual reality (VR) simulators and wearables offer opportunities for more objective and timely feedback on skill acquisition. They are promising tools for real-time performance assessment at the level of the individual and the team.
VR simulators and wearables allow for fine-grained analyses of patterns in performance and continuous monitoring of learning processes. For example, specific performance measures can be used to monitor and predict trainees’ learning curves for open and minimally invasive surgery skills. Critical moments in team interactions can be identified by detecting physiological changes or speech features with wearables. However, much is still unknown about using and combining these technologies with traditional measures like expert judgment. Our goal is to clarify the opportunities and challenges of using technology for individual and team skill assessment by presenting our experiences with using these tools in various settings.

MethodsWe present key technologies and their (dis)advantages for performance assessment of various skills, including VR simulators, motion sensors, sociometric badges, and physiological data sensors. We show applications of these technologies for 1) examining individual differences in learning curves of basic laparoscopic skills, 2) team interactions and effectiveness during a simulated advanced life support scenario, and 3) assessing smoothness of movement during suturing and flexible bronchoscopy. Using these technologies we were able to estimate individual performance levels, monitor changes in performance over time, and identify critical moments in team interactions. In all studies we encountered similar challenges: 1) translating low-level data into relevant assessment measures; 2) a need for
computational models to arrive at unbiased predictions and effectively filter out noise to support real-time feedback.
Results & Discussion
Simulator technology and wearables add unique assessment information to traditional measures. They have the potential to provide relevant and objective performance feedback to trainees during unsupervised training in a simulated environment, e.g. repeated rehearsal of basic flexible bronchoscopy tasks on a VR simulator. This also allows examining the effectiveness of novel training paradigms. Integrating these measures into standard educational practice is a challenge and requires technical solutions for data synchronization, processing, and analysis. Also, an assessment
framework needs to be developed to understand how low-level data from these tools correspond to clinical performance. Integrating the abundance of measures into meaningful performance outcomes is an innovation in its own right.