Transcript NLR - Eurocontrol

WP 3:
Human
Performance
Metrics
10 Oct 2002
Brian Hilburn
Goals
Extract set of ASAS applicable HumPerf Metrics
Outline application methods (guidance)
Provide primary references
Mapped onto:
- Application (eg Airborne Sep.)
- Validation Type (eg Realtime, etc)
- Agent (ATC vs Flightdeck)
- Human Performance Area
Human Performance Areas
– Workload
– Situation Awareness
– System Monitoring
– Teamwork
– Trust
– Usability and User Acceptance
– Human Error
Parallel WP2 (System Metrics)
– SYSTEM versus HUMAN performance?
Distinction sometimes arbitrary
– Performance Areas differ
– Implicit links to Goals
human performance influences SAFETY etc
WP 3 Deliverable
Fast t
Obj
Air
ASAS f1
.
.
ASAS fn
Subj
Ground
Wrkld
SA
etc
Real t
Methods
Literature review
- ASAS sources (transatlantic FreeFlight)
- ATM validation efforts (INTEGRA, RHEA,…)
- Aviation Human Factors
- Human Factors general
Distillation of methods and metrics
Guidelines
Including general Measurement Issues
General Measurement Issues
Validation test subjects
Experimental design in validation
Analytical techniques
Balancing validation realism & data collection
Format of Metrics
Function
Validation
Technique
Agent
Human
Performance
Area or
Concept
Data Type
Workload
Subjective
Objective
What?
Situation
Awareness
Fast Time
Airborne surveillance
functions
Airside
Metric
How?
What?
How?
Comments
Guidance:
Literature Consensus and Expert Opinion...
Pupil diameter
Validation
Technique
Subjective/Objective
Intrusive
Cost of Equipment
Reliability
Validity
Expertise Required
Resource Intensity
RTS
Good temporal resolution,
but also costly in terms of
Objective expertise. Equipment for eye
Med-High tracking expensive and
generally not portable;
High
Ocular measures currently
High
quite intrusive. Subject to
Med*
light (eg probably not
High
suitable for use in daylight
High
cockpit settings) and other
artefacts. (Beatty, 1986)
100+ Human Performance Metrics
•
•
•
•
•
•
•
•
NASA Task Load Index
Heart Rate Variability
Pupil Diameter
Human Computer Trust Scale
Auditory Choice Secondary Task Response Time
Situation Awareness Rating Technique
Blink latency
Choice Reaction Time secondary task
• etc………...
Choosing a Metric
– Objectivity
Objective / Subjective
– Intrusive
High / Low
– Cost
High / Low
– Reliability
High / Low
– Validity
High / Low
– Utility
High / Low
– Expertise
High / Low
– Resource
High / Low
A Worked Example...
• Given: Airborne Spacing application
(in-descent spacing scenario)
Controlled airspace
• Study: Real time
• Question: Transient workload peaks?
• Issues: Time resolution, intrusiveness, cost
Applying the Framework
General Guidance
Human System
Workload Error Monitoring Usability
Analytic
Fast Time
Real Time
Survey
*
*
***
**
***
***
*
*
**
*
***
*
**
**
***
***
SA
Teamwork
Trust
*
*
***
**
**
**
***
*
*
*
*
***
Conclusions

Link between System
and Human
Performance metrics

What are my goals?

Relationship to higher
validation goals

Guidance, not a
cookbook

Ideally used in
conjunction w SYSTEM
metrics