Transcript Slide 1 - University of Utah

Situation Awareness Assessment to Determine the Usability of a Graphical Anesthesia
Display During a Simulated Anesthetic: Study Results
Parley Williams, MSII, David Lamborn, MSII, Noah Syroid, M.S. Dwayne Westenskow, PhD,
Department of Anesthesiology, University of Utah, Salt Lake City
Introduction
(Results Continued)
In most anesthetic cases, clinicians are typically guided in their management of
patients by a highly limited set of observed variables. No real-time data
projections are used to estimate real-time physiologic drug concentrations or
effects. Researchers at the University of Utah have developed a graphical
display to provide clinicians with real-time modeled predictions of physiologic
drug concentrations and effects, including synergism. The purpose of this study
is to use a situational awareness global assessment technique (SAGAT) to
determine the usability of this display during a simulated total intravenous
anesthetic (TIVA) case1,2,3.
When asked to rate the usefulness of the display, the participants reported an average
score of 7.28 out of 9 (9 representing most useful). The most frequent comments
reported by the participants are summarized as follows:
Strengths:
1. Display was helpful in managing the patient.
2. Display was helpful in planning drug delivery and emergence.
3. Display was easy to read, including the future component.
Concerns/Suggestions:
1) Concerned that NMB reversal was not depicted on the display.
2) Pharmacokinetic lines were indistinguishable in the analgesia plot.
3) Would like to be able to move grey bands up or down based on individual
patients and the degree of stimulus of the procedure.
4) Would like to be able to administer a simulated dose and observe how that
might change the overall predicted effect.
Methods
Following a 15-minute training session on the display, 4 residents and 4
attending physicians were presented with a typical TIVA case. The simulated
case was a laparoscopic hernia repair with a surprise switch to open hernia
repair. At predetermined intervals the simulation was paused to ask the
clinician several questions designed to assess their level of understanding as it
related to individual components and applications of the display. The three
levels of understanding assessed included: 1) Basic perception, 2)
Understanding meaning, and 3) The ability to predict future trends1.
•Hypothetical
drug
administration
•Adaptable
therapy windows
Figure 1: This figure shows a sample view of the display. Arrows pointing to various places on the display illustrate
components of the display corresponding to the major recommendations to improve the design of the display as summarized
in figure 3. These recommendations are intended to help improve the safety and usability of the display. The display is
divided into 3 main sections: sedation, analgesia, and neuromuscular blockade. Each individual section displays the
predicted effect-site concentration and overall effect of administered drugs. Colored lines correspond to individual drug
concentrations, while white lines correspond to the overall synergistic effect of combined drugs. Vertical and horizontal bars
correspond to the administration of a drug bolus and infusion respectively.
Results
Average SAGAT Score by Level of
Situational Awareness
100%
95%
Recommendations
(See Figure 1)
Improve the general appearance of therapy bands to help clinicians feel
more comfortable titrating above them, and not strictly within them.
90%
Average SAGAT Score
SAGAT Analysis
After using the SAGAT to evaluate their performance during the simulated case,
the average participant scored 95.2% correct overall with a standard deviation
of 2.9%. The average score for questions pertaining to level 1 situational
awareness was 97.7% with a standard deviation of 4.2%. Similarly, the average
score for questions pertaining to level 2 situational awareness was 97.8% with
a standard deviation of 2.6%. Differing slightly more, the average score for
questions pertaining to level 2 situational awareness was 83.8% with a standard
deviation of 10.6% (Figure 2). An analysis the questions most commonly
missed revealed 5 components of the display that were uniformly difficult for
participants to interpret. Four out of these five areas stem from questions
probing level 2 situational awareness, and one area from level one situational
awareness. These five components included the following
1. The general appearance of a Bolus and Infusion on the display.
2. The EC50 T1% line to which the NMB graph is normalized.
3. The measures of the Y-axis.
4. The pharmacokinetic lines in the analgesia graph.
5. The way the display would look without synergism.
The performance on most topics was statistically equivalent for experienced
and inexperienced clinicians alike. The only exception was a statistically
significant difference between experienced and inexperienced clinicians when
asked the first set of questions regarding the EC50 T1% line (p=0.043).
Comment Analysis
An analysis of comments provided during the simulation and on the
questionnaire reported several strengths and weaknesses of the display.
85%
Adapt the appearance of drug administration so the administration of a
bolus and infusion can be more readily distinguishable to the clinician.
80%
75%
Improve the appearance of, and/or the training on the EC50 T1% line
so it can be more intuitively applied to the management of patients.
70%
65%
60%
Clarify and label the measures of the Y-axis.
55%
50%
Level 1
Level 2
Level 3
Levels of Situational Aw areness
Figure 2: This figure depicts the average performance of all participants on
questions pertaining to Level 1, Level 2, and Level 3 situational awareness
respectively. Scores are shown as a percentage correct out of the total number of
questions asked for each level of situational awareness. The standard deviation for
each level of situational awareness is also depicted.
Provide means to administer a hypothetical drug dosage, and allow
clinicians to adapt the therapy windows based on the history of the
patient and the stimulus of the surgery.
Figure 3: A summary of recommendations to improve the display based on the
SAGAT results, statistical analysis, comments from clinicians, and general
observations.
References
1. Endsley MR: Situation awareness global assessment technique (SAGAT): Air to air tactical version user
guide. Hawthorne (CA), Northorp Corp., 1990
2. Endsley MR: Measurement of situation awareness in dynamic systems. Human Factors 1995; 37: 65-84
3. Endsley MR: Situation awareness analysis and measurement. Manwah (NJ), Lawrence Erlbaum Assoc.,
Inc., 2000
Discussion
The relatively high performance on most of the SAGAT questions suggests that
clinicians generally understood the display and were able to properly integrate it in
the management of the simulated patient. This result indicates that the presentation
was adequate in acquainting clinicians with the basic features of the display and also
suggests that the display itself is largely intuitive. The positive feedback from
clinicians also suggests the display was relatively user-friendly. While clinicians
typically did very well with the first and third levels of situational awareness, it
appears there were a few components that were more difficult for the second level of
situational awareness. These results may indicate that while subjects may be aware of
certain components of the display, and they may be able to identify and predict trends,
they may not fully understand what certain components of the display actually mean
or represent. For instance, perhaps the visual representation of the display was
intuitive enough that they could visualize trends without necessarily having to
understand what drove those trends, or what they meant. In certain instances where
increased experience enhanced the performance of the clinician, this may simply be a
result of clinicians using their experience to fill in the gaps where the display may not
be clear. In other instances where concepts were equally difficult for experienced and
inexperienced clinicians alike, this may indicate areas of the display that may actually
be misleading. Although not directly addressed in the SAGAT questions or the
comments submitted by the participants, it was observed that nearly all participants
felt compelled to titrate drugs in a way that kept the overall drug effect within the grey
bands themselves. This finding suggests a fundamental lack of understanding about
the meaning of the gray bands or a misleading design flaw. Titrating within these lines
corresponds to less than 95% of patients being adequately sedated and having no
response to intubation. This constitutes a potential safety concern inherent in the
design of the display. If the display were to be relied upon to make dosing decisions,
this lack of understanding could potentially lead to under-sedation and/or inadequate
pain management of the extremely drug tolerant patient. The 5 main
recommendations for improving the display have been summarized in figure 3.