Due to the haptic experience in the virtual environment, MED1stMR decided for a mixed reality (MR) training approach to create a better learning experience for medical first responders.
MED1stMR project goals
The project develops a mixed reality solution that incorporates large movement for the participants, two high-fidelity simulation manikins, and a biosignals measurement suite including a trainer dashboard to facilitate alterations to the training flow in accordance with the stress level of the trainees during their mass casualty incidents in training. This solution is built to be exploitable after the end of the project.
Research beyond project scope
Additionally, MED1stMR conducts research to explore possibilities of mixed reality for future training options. One approach is the “Green Manikin” developed by AIT. It started with a simple torso simulation manikin that used chroma-keying to enable the trainees to use real tools and see them in their hands in the virtual environment. The mannikin patient is haptic and at the same time overlayed with different injuries and animations as well as virtual backgrounds and other environmental effects. Using real tools and the trainee’s own hand is a major advantage of this prototype.
First results
The first studies were conducted with the Green Manikin, which combines a physical manikin with a virtual avatar, allowing trainees to interact with both real and virtual elements by using the Varjo XR-3 mixed-reality headset. In the designed scenario, trainees deal with a car accident involving multiple injured people. The research team measured the participants’ levels of presence, stress, and technology acceptance, as well as their feedback on the MR system and the training scenario. Results showed the participants had high levels of physical- and self-presence, increased stress levels, and high technology acceptance during the simulation. The MR approach was well received, and they particularly appreciated the haptic and multi-sensory feedback and the realism and complexity of the training scenario. The study concluded that MR could be seen as a viable technological solution to create immersive skill training, perceived by end-users as a useful tool and addition to existing training solutions.
Future of simulation in MR
In the course of the project, the Green Manikin became a full-body simulation manikin enhanced with full-body tracking and operated with a new approach: communication via speech recognition and a Large Language Model (LLM) for suitable answers. This gives first insights into how the lack of communication with patients, especially the response from patients as a clear indication of the mental state and injuries can be solved in the future of simulation training with virtual environments.
At the end of the project, even more results will be available as the Green Manikin is also touring Europe to some of the field trial locations in order to gather more data on future training options in the field of mixed reality training for medical first responders.
