Usability Assessment of an Augmented Reality Application for Procedural Guidance on Cricothyrotomy and Thoracostomy

Successful management of the airway is vital for trauma patients, as failed treatment can result in severe hypoxia, hypotension, aspiration, or cardiac arrest [1]. This is evident in a study of battlefield deaths that reported, following haemorrhage, airway obstruction and tension pneumothorax were the next most common focuses of potentially survivable deaths [2]. Clinicians caring for patients with airway and chest injuries have several lifesaving interventions available for treating these types of conditions, including Cricothyroidotomy (Cric), needle decompression and Chest Tube (CT) insertion. While Crics are within the scope of practice for some prehospital professionals, it is a high acuity, low occurrence procedure for which many are not comfortable performing [3]. In the civilian sector, surgical airway is seen as a last-ditch effort and is done only when other methods of securing a patent’s airway are unsuccessful. Similarly, CT insertions are not typically performed by emergency medical technicians, medics, or paramedics. Current guidance for the battlefield treatment of tension pneumothorax includes up to two rounds of needle decompression [4]. If unsuccessful, medics are trained to proceed to management of circulation. The complication rate for trauma-related CT insertion averages 19%, which may be a reason for limiting the procedure to advanced providers [5]. Insertion is typically performed in the emergency room by physicians; however, other clinicians such as paramedics and nurses with advanced training can sometimes perform the procedure if warranted by patient condition and setting. This leaves to question how teams can safely care for patients with airway or chest trauma when advanced providers are unavailable for extended periods.

When a physician is not available in-person, telemedicine has been utilized to assist clinicians in caring for complex, critical patients. Telemedicine or telehealth, refers to the use of technology to deliver health services from a distance [6]. While this capability has emerged as a valuable resource, it is not always available in remote locations where network connectivity may be limited. Localized Clinical Decision Support Systems (CDSS) have been proposed to guide medical personnel by reducing the complexity of injury management and providing critical information in real time, without the need for an outside connection [7]. A literature review by Panicker and George evaluated adoption of CDSS in numerous use cases, including reducing prescription errors by general practitioners [8,9], improving management of antibiotics and genome-guided prescriptions [10,11], prompting physicians of the need for immunizations [12], and management of patients with diabetes [13]. Most CDSS have been developed to integrate with familiar interfaces, such as a patient health dashboard or web/mobile application [14,15]. However, these interfaces require physical inputs from the user via touchscreen or keyboard to interact with the system. In contrast, Augmented Reality (AR) systems overlay virtual content onto a user’s real-world view through glasses or other heads-up displays, allowing users to receive hands-free information [16]. Overlay can include text or images, which may float in the air or overlay onto patients and equipment. AR applications are currently being used for medical training, surgical assistance, and patient monitoring and can be delivered via head-mounted displays or other mobile devices for untethered, remote use [17-19]. Research shows AR increases performance accuracy and reduces cognitive burden when used for medical training and surgical assistance [20,21].

Use of CDSS in the military will need to support Large Scale Combat Operations (LSCO) that will be resource-limited and access to communications networks will be denied due to electronic warfare, such as signal jamming [22-24]. Additionally, it is anticipated that the number of military surgeons will be inadequate to care for the number of anticipated casualties [25]. Consequently, clinicians will be forced to provide care and critical interventions beyond their normal scope of practice or expertise. While network disruptions preclude the use of telehealth, self-contained CDSS could mitigate this issue by providing users with embedded guidance materials, including written and visual instruction. Voiceand gesture-controlled AR CDSS could provide robustness in these environments, along with a host of other benefits such as sterile field maintenance, step-by-step guidance in text and video formats, and holographic visualizations directly onto casualties.

We hypothesize that AR-based CDSS can help novice clinicians perform tasks beyond their usual scope of practice, helping them to meet the challenges of an austere environment. The Augmented Reality Surgical Assist Manager (ARSAM) was developed to assist combat medics with emergency surgical interventions, such as the Cric and CT insertion, which are commonly performed due to airway and chest trauma. ARSAM utilizes instructional text, pictures, videos, and holographic overlays to aid medical personnel in performing surgical procedures. Instructions are simplified from what would normally be provided in medical literature and clinical practice guidelines, and users are allowed to move backwards or forwards through the steps of each intervention using hand gestures or voice commands. While ARSAM was developed with combat medics in mind, the application could also be used by civilian personnel in rural or remote environments or in mass casualty environments where there are more patients than skilled providers. If ARSAM is to be adopted and fielded for high-stakes critical care scenarios, it must meet a high standard for usability and performance. The purpose of this paper is to present the results of a study conducted to evaluate ARSAM’s overall usability in the simulation of two lifesaving surgical interventions: Cric and CT insertion.

Regulatory approval and standards

This research study was conducted under a protocol (M- 11021), approved by the headquarters’ Institutional Review Board.

Inclusion/exclusion criteria

The study sought to evaluate the design and functionality of our clinical decision support software, which was intended for clinicians with limited experience. For this reason, participants were recruited from the staff working at our medical center, however, no medical experience was required to participate. It was anticipated that a mixed volunteer population would optimize feedback by collecting both clinical content and interface design feedback. Participants had to be at least 18 years of age and willing to dedicate up to four continuous hours to training, simulation, and providing feedback.

Materials

ARSAM was developed and tested on the Microsoft® HoloLens 2™ (Redmond, CA) mixed reality system. The system consists of an untethered, hands-free, head-mounted display that is controlled through voice or hand gestures. Clinical content was created using military training resources, skill checklists, and clinical practice guidelines, and was reviewed by medical subject matter experts within the organization. Instructions and graphics were developed using Unity® with the Mixed Reality Tool-Kit (MRTK). Unity® projects were deployed to the HoloLens 2TM through Visual Studio. The Laerdal Sim Man® 3G (Wappingers Falls, NY) and Syn Daver® Adult Cric Trainer (Tampa, FL) were used (with slight design modifications) for performing surgical simulations.

Application design

The software was designed as a stand-alone, executable application for the Microsoft® HoloLens™ using the Unity™ platform. The main menu consisted of virtual buttons that allowed users to select the surgical procedure for which they needed assistance. After selection, a holographic phantom appeared in the users’ visual field, either a head (for Cric) or a chest cavity (for CT), depending on the procedure selected. Users had to manually overlay the phantom onto the applicable manikin (Figure 1). This step was critical to ensuring that instructional holograms appeared in the correct locations during procedures. For usability evaluation, a member of the research team placed the phantoms prior to handing the headsets off to participants. Within each subapplication, there were two main screens: one with contextual instructions and a second (when applicable) with either a video or image. Interactive buttons with “next” and “back” were displayed below the instructional screen to assist users in advancing to the next steps or going back to previous steps. These buttons could be selected or spoken. Some steps included additional holographic overlays on the surface of the patient manikin.

Figure 1:Manual placement of the phantom figures (left: head for cricothyrotomy procedure, middle: chest for chest tube insertion) over the mannikin so that instructional holograms appear in the correct place (right: thoracostomy cut line).

Study design

Participants were oriented to the AR device using an embedded training application, which was designed to look and operate the same as the surgical application being tested. Using the headset, they were provided written instructions and pictures or short video clips which demonstrated how to operate the device and software. Instructions were followed by a series of practice exercises. Participants were also given a brief questionnaire to complete which captured demographics and experience. After training, participants were asked to perform two manikinsimulated procedures: Cric and CT insertion. Both procedures were performed using the ARSAM software, however, some participants also completed the procedures using non-AR tools, if they wanted and if time allowed. To assist participants with non-AR tasks, the Deployed Medicine [26] application library was available via tablet. This military medical reference tool contained both clinical practice guidelines and videos. Since the study aimed to evaluate usability of the software, we did not need comparison data between modalities. However, having a few participants complete the procedures using both AR and non-AR allowed us to prepare for performance testing that would take place after usability testing and software updates were complete. Notes and videos were taken by the research team throughout all AR and non-AR procedures to record observations, participant reactions, and problems that were encountered. For safety, participants could skip steps they did not feel comfortable performing, such as those involving the use of sharps.

Following each AR-assisted procedure, participants completed a System Usability Scale (SUS) survey and a custom Participant Usability Survey. The SUS is a validated, ten-item Likert questionnaire with positively and negatively worded questions to quickly assess usability of a system [27]. A SUS score above 68 (range 1-100) implies acceptable software usability. The Participant Usability Survey consisted of 57 questions for each AR procedure. Questions were developed around 9 usability heuristics adapted from Jakob Nielsen’s usability design principles [28]: Help/documentation, feedback, satisfaction, integrating physical and virtual worlds, user interactions, consistency, recognition, cognitive overload, and comfort. After both simulation tasks were performed, participants completed a Post-Simulation Survey. This survey was developed by the research team and gathered data on each participant’s previous experience with the procedures, overall study experience, and opinions regarding how the technology might be used. Survey responses were used to guide an after-action discussion on more granular aspects of the surgical tasks, AR software, hardware, and overall study experience. Figure 2 shows the complete study design, including procedures and survey instruments.

Figure 2:Study design flow.

Usability analysis

Data gathered in this study were used to provide a quantitative analysis of the ARSAM software for the two surgical tasks. Survey data using continuous variables are expressed as mean +/- standard deviation. Analysis was completed using JMP®, Version 16 (SAS Institute Inc., Cary, NC). Usability heuristic data from the Participant Usability Survey were analyzed to identify areas of consensus and improvement needs. “Strong” consensus, where the percentage of agreement was 60-100% for positively worded statements, required no additional evaluation for software improvement. Statements with very low consensus (0%-40% agreement) were prioritized areas for improvement. For negatively worded statements, “strong” consensus identified areas that did require improvement. For mixed consensus (40-60% agreement), we recorded those statements for further evaluation by the research team.

Ten participants were recruited and enrolled in the study, consisting of 6 females and 4 males with an average age of 40.8 years. The majority (n=8) were familiar with or had knowledge of augmented reality devices. While most (n=7) had never used the Microsoft HoloLens, half of the participants had used some other head-mounted display device (such as another augmented, mixed, or virtual reality device) (Table 1).

Table 1:Participant demographics.

From the SUS survey, the lowest-rated positive statement concerned participant confidence using the ARSAM software (Cric=3.7±1.2, CT=4.1±0.7) (Table 2). The highest-rated negative statement indicated that participants needed assistance when using the app (Cric=3.5±1.1, CT=2.9±1.4). Overall, SUS scores demonstrated ARSAM was a usable application for guiding inexperienced clinicians in performing a Cric or CT insertion procedure (Cric=74.5±15.1, CT=79.0±15.7).

Table 2:System usability survey results.

SUS=System Usability Scale
SUS scores were compared to the standard passable score [68].
(note: SUS questions used Likert scale, 1 thru 5)

Of the 57 statements analyzed from the Participant Usability Survey, mixed agreement was found amongst participants on 9 statements, indicating areas of the software that could be improved (Table 3). These statements fell under the heuristic principles of feedback, integration of physical and virtual worlds, user interactions, consistency, recognition rather than recall, and cognitive overload. Feedback, user interactions, and consistency had the most statements that needed to be addressed (n=2). We compiled both heuristics survey data and after-action review notes to determine specific areas of software changes for each heuristic principle (Supplemental Data Table 1). After-action discussion topics centered around encountered problems with the software, concerns and frustrations, areas for improvement, clinical value, enjoyment, and side effects (Supplemental Data Table 2). Problems that were encountered included application closure during a procedure, difficulty with voice recognition and gesturing or manipulation of screens. While most participants responded that instructions were understandable, some requested improved resolution of the videos. Several also stated they would prefer instructions be broken into smaller steps, reducing the volume of words and information being displayed at once.

Table 3:Participant usability survey results.

Mixed response=Statements where 60%-40% of the respondents did not agree with the statement
Strong disagreement=Statements where more than 60% of the respondents did not agree with the statement
**Questions that were worded in the opposite/negative tone were analyzed in reverse.

Supplemental Data Table 1:Participant demographics.

All participants were pleased with taking part in the study and felt they learned something, whether it be about the procedures or the technology. However, participants were split about the ideal use for ARSAM. Some felt it was optimal for real-time decision support, while others saw it being used for training. Concerns regarding real-time use centered around unfamiliarity with the system, potential crashes or system instability, and difficulty with voice recognition in noisy environments. No cyber sickness or other side effects were reported among the 10 ARSAM users who participated in this study (Supplemental Data Table 2). This study evaluated the usability of ARSAM, a prototype AR CDSS for guiding novice users through two high-acuity, low-frequency trauma procedures. Despite the complexity of the tasks and the mixed levels of user experience, SUS scores indicated that ARSAM was generally wellreceived, with average scores of 74.5 for Cric and 79.0 for CTboth exceeding the accepted threshold of 68 for usability. These results support the feasibility of using AR-based support tools to guide critical interventions in resource-limited or austere environments. While usability metrics were favorable, heuristic feedback and after-action reviews highlighted important areas for refinement. Specifically, reported issues related to interface interactions such as challenges with gesture and voice command recognition and application crashes, which can be critical barriers in high-stakes clinical environments. Although distrust in CDSS is generally attributed to errors in content, rigidity in systems, or alert fatigue [29], our application’s challenges arose primarily from the hardware/software interface rather than the content itself. However, improvements in content presentation were noted for software development planning to improve cognitive overload in users. Streamlined content for rapid assimilation is essential, especially under stressful conditions. Therefore, a revision of the content delivery was essential for the next version of ARSAM. Additionally, technical instabilities, though expected in prototype systems, underscore the need for improved system robustness prior to deployment for simulated clinical performance testing.

Supplemental Data Table 2:User after action compilation.

Interestingly, while participants recognized ARSAM’s potential for real-time procedural guidance, a majority expressed strong support for its use as a just-in-time training tool. This aligns with growing evidence to support AR in medical education, where it has been shown to enhance procedural accuracy, improve user confidence, and increase knowledge retention during simulationbased learning [30]. Unlike active clinical use, training scenarios benefit from more lenient time constraints and allow users to engage with the system without time-critical consequences or harm to patient safety. Additionally, participants expressed interest in just-in-time training applications-an approach increasingly endorsed for preparing providers in unpredictable or infrequent scenarios, such as those encountered during battlefield medicine or disaster response [31,32].

This study was a first step to understanding user satisfaction and engagement with the ARSAM application prior to evaluating clinical performance. Given our findings, the next phase of development should prioritize interface optimization, system reliability, and instructional design tailored to reduce cognitive burden. Although the application is strongly tied to the hardware of choice in this study, it is important to note any potential limitations in device durability and stability, especially in non-hospital settings. Furthermore, future validation efforts should assess the impact of ARSAM on procedural accuracy, task completion time, and user performance under field or simulation conditions. Expanding testing to include combat medics and advanced practice clinicians in operational environments will be critical to ensuring clinical utility and field readiness.

Limitations

A limitation of this study was simulation fidelity. We aim to make this technology usable in high stress surgical, non-hospital, battlefield environments. However, our simulated surgical bay lacked the environmental stressors necessary to capture such highpressure responses from participants. For example, there were no detrimental consequences if the procedure was not performed in a timely or accurate manner. The simulation environment also had comfortable conditions for the participant. This could conceivably result in more favorable usability metrics than if the technology had been tested in a synthetic battlefield setting. It may be more conducive to conduct future performance assessments of ARSAM in a fully immersive training scenario to accurately capture highstress environmental responses in a repeatable fashion [33].

Software usability and user interaction were the primary focuses of this evaluation, as our development efforts were limited to software refinement. However, some of the feedback we received focused on the device, such as difficulty with voice and gesture recognition or the speed/volume of auditory instruction. Although the AR device’s physical design and system limitations were outside of our control, we were able to make software adjustments to address some of these issues, such as enlarging buttons and disabling verbal instruction. Ultimately, we wanted to create an application that was device agnostic, with the recognition that each hardware device will have its own unique challenges.

Finally, our participant recruitment pool was limited to a single institution and included those both with and without medical expertise. While non-medical feedback helped in terms of improving usability, this cohort is not representative of our target user population and may have limited the amount of medically relevant feedback gathered. Future studies with a more relevant cohort will enable us to confirm clinical content accuracy, assess clinical performance, and conduct a secondary usability assessment.

Our study demonstrates the potential use of ARSAM in the performance of two emergency interventions. Although users found the clinical decision support software to be acceptable in its current state, feedback was obtained to increase usability and likelihood of adoption. User comments and survey data will direct improvements, bolstering its potential for real-time guidance.

Funding provided by:
i. JPC6 Combat Casualty Research Program Project Number IS220009
ii. Oak Ridge Institute for Science and Education

SM, NC, AS developed the study design; SM, AW, NC, JR, DL, AS conducted the study, SM, NC, AW, AS collected and analyzed the primary data; SM, NS, NC, AW, JR, SC drafted the initial manuscript; SM, AW, NC, JR, DL, SC, AS provided key revisions and final edits.

None to declare

The views expressed in this article are those of the authors and do not reflect the official policy or position of the Defense Health Agency, Department of the War, nor the U.S. Government. This study was conducted under a protocol reviewed and approved by the US Army Medical Research and Development Command Institutional Review Board and in accordance with the approved protocol.