Transcript 3-Rustia CASNUR-IFA2012x

Greater
healthcare
demands
!!
Healthcare meltdown
Aging and nursing
shortage
400,000
shortage
by 2020
1:20 vs
ideal of
1:10
INCREASING
Retention
DEMANDS
of Workers
on SENIOR
Strategy
NURSES
At least 1
adverse
effect
70% of
deaths
INCREASED INCIDENCES of ERROR
Heterogeneous concept that must be broken down
There is no specialized concept or metric for the
cognitive aspect of work ability and there are also no
methods at present to measure / assess this.
Measure Cognitive Work Ability
Job reassignment & interventions
Predicting stress levels and likelihood of
adverse effects (ADEs)
Existing methods are insufficient &
subject to bias.
Automated and Performance-Based
CASNUR: Developing a software to assess cognitive
work ability among aging nurses
by Maria Dominique Rustia & Rosemary Seva
Human Factors and Ergonomics Center, De La Salle University, Manila, Phil.
[email protected]
May 28, 2012
IFA 2012 Global Conference on Ageing
Background
Method
Concept & Design
Testing
PROBLEM STATEMENT
Conclusion
1
• Existing assessment methods for work ability among nurses are
subject to response bias and fail to consider the cognitive
dimension of tasks.
• There is a need for a performance-based assessment method
for cognitive work ability (CWA) to be established, especially in
the context of ICU nursing.
Background
Method
Concept & Design
Testing
Conclusion
OBJECTIVES, SCOPE & LIMITATIONS
1
OBJECTIVES
• to provide a reliable and valid measure of CWA among hospital nurses
by developing a software application that can assess CWA among them.
1.
2.
3.
4.
To test the validity of the CWA assessment software
To perform an initial evaluation of the usability of the software
To determine the reliability & effectiveness of software in capturing data for CWA
metric derivation
To identify areas for improvement in the software
SCOPE & LIMITATIONS
Hospital nurses, ages 40-65, Intensive Care Unit (ICU) setting
Philippine setting
Will only test variables of CWA model of Rustia & Seva (2011)
Will not propose any interventions
Conceptualization and design of software
Gender, affective & physical needs not considered
Background
Method
Concept & Design
Testing
Conclusion
1
COGNITIVE WORK ABILITY
Work Ability Index (WAI)  “how good workers are at present and in the
near future, and how they will be able to do their work with respect to work
demands, health and mental resources” (Ilmarinen & Tuomi, 2003)
speed
accuracy
Cognitive Work Ability (CWA)  the ability of a person to fulfill
mentally demanding tasks efficiently and effectively
Background
Method
Concept & Design
Testing
Conclusion
EXISTING ASSESSMENT METHODS
Work Ability
Index (Ilmarinen
& Tuomi, 1992)
WHO Quality of
Life (WHO,
1991)
Expanded
Nursing Stress
Scale (French et
al., 2007)
TISS-28
(Miranda et al.,
1996)
Revised Nursing
Work Index
(Aiken &
Patrician, 2000)
1
• Reliability and validity have
been verified but coverage of
components of CWA is poor
• Questionnaire-based, risk of
biased and inaccurate answers
• No performance-based
evaluation methods to date
SIMULATION OF THE
NURSING ENVIRONMENT
(low cost  software)
CASNUR
Cognitive work ability Assessment
Software for NURses
Background
Method
Concept & Design
METHODOLOGY
Testing
Conclusion
2
Background
Method
Concept & Design
Testing
Conclusion
METHODOLOGY
Systems Development Life Cycle
2
Method
Background
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN- Software Concept
1
SIMULATION
2
DATA-GATHERING
3
EVALUATION
3
Background
Method
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN – Requirements Planning
ACCURACY
3
SPEED
PERFORMANCE
QUALITY
• Applied Cognitive Task Analysis (ACTA) applied to determine cognitive
processes in regular ICU nursing tasks.
Background
Method
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN – Requirements Planning
3
SIMULATION INTERVIEW &
KNOWLEDGE AUDIT
MONITOR and ALARM
hard to say immediately
where the problem is
EMERGENCIES lead to
improvising
presence of mind and
alertness, prioritization, good
memory, speed
checking, finding
discrepancies in medicine
arrangement and labeling
of medicine, better systems
Background
Method
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN – Requirements Planning
3
Background
Method
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN – Requirements Planning
3
Constantly
changing
E-cart to
concentrate on
individual skill
Computerized to
remove bias
related to
handwriting &
technique
Other Characteristics:
Abnormal
event
SIMULATION
• 1
Simulation
must also have the same layout
as actual
ICU generation
in the Philippines
• Emergency ICU case  feeling of “time pressure” and urgency
2
DATA-GATHERING
Errors, Actions, Responses
Background
Method
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN - Conceptual Model
3
Background
Method
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN - Software Features
3
Background
Method
Concept & Design
Testing
Conclusion
SOFTWARE DESIGN – Software Features
3
Background
Method
Concept & Design
Testing
Conclusion
EXPERIMENTATION- Factor Selection
3
Measurable by CASNUR
Measurable by CASNUR
and eye-tracker software
Measurable by CASNUR
and eye-tracker software
Measurable by CASNUR
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST – Subjects & Assumptions
SUBJECTS & ENVIRONMENT
• Filipino nurses, ages 45+
• Fluent in English and Filipino
• Capable of using computers
• No physical or mental diseases / impairments
• Must have had experience in ICU / emergencies
• Day shift to avoid influence of night shifts
• Pre-scheduled appointments to avoid rushing
ASSUMPTIONS
The participants selected are representative of the
targeted users of CASNUR.
The medications are assumed to be correct even if
they medically are not.
The frequency of errors is assumed to be correct.
The subjects were physically, mentally and
emotionally fit before testing.
4
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Methodology
Subject Profile Survey
CASNUR software
Usability Plan, CASNUR
Usability Questionnaire
Dikablis Eye-Tracker & Software
CASNUR + Eye-Tracker + Morae
Simulation Evaluation Survey
4
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Methodology
RELIABILITY
VALIDITY
4
USABILITY
EFFICIENCY
Testing of successful
generation of variables
and events
How close the simulation
is to real-life situations
EFFECTIVENESS
SATISFACTION
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
4
RELIABILITY
SIMULATION
Vital Signs Monitor changes,
alarms and indicates abnormality
• Emergency cart with full
functionality of selecting and
adding medicine to table
• Medicine Table with maximum
DATAGATHERING
Error identification,
counting and recording
Response / action
identification, counting and
recording
capacity of 3 medications
Visual timer on interface
• Administration of oxygen,
Export feature to EXCEL
dextrose and medicine (input of
dosage)
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
RELIABILITY
10 variables measured
4
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
4
VALIDATION OF CWA MODEL FROM RESULTS
Attentive Resources
Working Memory
Capacity
Perception
Responsiveness
TESTS
CONDUCTED:
Collinearity
Heteroskedasticity
Normality
Task structure
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
4
VALIDITY
Shapiro-Wilk W-test for normality
Correlation analysis
Simulation evaluation survey
Rating of how realistic simulation is
Mostly normal in distribution
Number of wrong medicine types = positive
correlation with unnecessary actions
Response time = +correlation with suffering
duration
Correct responses = -correlation w/ wrong
type errors
4 / 5 (5 = very realistic)
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
USABILITY
EFFECTIVENESS
High Learnability (12.03%
unnecessary actions)
• 40.66% of abnormal events
solved by the users
EFFICIENCY
“Normal” task completion
times
4
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
USABILITY
SATISFACTION
50% satisfaction rating
5.25 / 6 in ease of use
Second trial slightly easier than first
4
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
DEBRIEFING RESULTS
4
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
4
AREAS FOR IMPROVEMENT
NORMALITY OF DATA
More runs / replications
PREPARATION
1. Only qualitative testing for presence
of mental impairments
2. Inconsistencies in pacing and tone of
briefing voice.
Use of COGLAB Memory Test
Design of a briefing video for CASNUR
SOFTWARE
1. Unstoppable alarms make nurses more irritable or tense.
2. Nurses tend to forget to select the medicine first before clicking on GIVE MEDICINE.
3. Confusion with color on vital signs monitor.
4. Nurses forget to click DONE / CANCEL to return to main interface from e-cart.
5. Scattering of physician’s orders during experiment
6. Too many abnormal events occurring within the span of 5 minutes.
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
AREAS FOR IMPROVEMENT
SOFTWARE
ON/OFF toggling of alarm
Reprogramming of functions
Use of brighter color (YELLOW)
Placing of “GO BACK TO PATIENT”
button in e-cart and changing CANCEL
to “CANCEL addition”
4
Background
Method
Concept & Design
Testing
Conclusion
PILOT TEST - Results and Analysis
4
AREAS FOR IMPROVEMENT
EXPERIMENT
1. Difficulty moving mouse around.
2. More than one run per setting is
needed.
3. Subjects tend to get significantly
better after 4th / 5th try.
4. Need a basis / ideal performance
value for the task times.
5. Usability testing needs more
standardization.
Touchscreens
More subjects and runs
Do not use beyond 3 tries per
subject.
Compute for performance
value through control limit
computation & interviews with
nursing educators.
Inclusion of NASA TLX test.
Background
Method
Concept & Design
Testing
Conclusion
CONCLUSION & RECOMMENDATIONS
• Cognitive Work Ability (CWA) is a specialized
concept for the ability of a person to fulfill
mentally demanding tasks efficiently and
effectively.
• Simulation as performance-based method for
assessing CWA among ICU nurses.
• A simulation software has been successfully
designed with acceptable usability ratings,
reliability in data collection & simulation, and
validity (face and construct).
• Recommended use of software by the
academe, health professionals, nurses,
hospitals.
5
• Larger experiment with more
samples and actual derivation
and evaluation of CWA index.
• Incorporation of eye-tracking and
screen-recording features in
system to reduce CASNUR’s
usage costs.
• Testing by wider demographic
(young and old).
• Longitudinal testing of software
and CWA index.
“With society becoming older and more active, will
we change the way we work…? Does it make any
sense to stop being productive at a particular age?”
– Ken Dychtwald (1990)
TRY OUT CASNUR
Look for me after the presentation / session /
anytime during the IFA 2012 Conference.
Questions / Comments?
Feel free to e-mail them to:
[email protected]
Thank you.