Transcript converging-technology-kautz-v3 - Computer Science

Recognizing Human Activity
from Sensor Data
Henry Kautz
University of Washington
Computer Science & Engineering
graduate students: Don Patterson, Lin Liao
CSE faculty: Dieter Fox, Gaetano Borriello
UW School of Medicine: Kurt Johnson
Intel Research: Matthai Philipose, Tanzeem Choudhury
Converging Trends…
Pervasive sensing infrastructure
GPS enabled phones
RFID tags on all consumer products
Wireless motes
Breakthroughs in core artificial intelligence
After “AI boom” fizzled, basic science went on…
Advances in algorithms for probabilistic reasoning and
machine learning
Bayesian networks
Stochastic sampling
Last decade: 10 variables  1,000,000 variables
Healthcare crisis
Epidemic of Alzheimer’s Disease
Deinstitutionalization of the cognitively disabled
Nationwide shortage of caretaking professionals
...An Opportunity
Develop technology to
Support independent living by people
with cognitive disabilities
At home
At work
Throughout the community
Improve health care
Long term monitoring of activities of daily
living (ADL’s)
Intervention before a health crisis
The University of Washington
Assisted Cognition Project
Synthesis of work in
Ubiquitous computing
Artificial intelligence
Human-computer interaction
ACCESS
Support use of public transit
CARE
ADL monitoring and assistance
This Talk
Building models of everyday plans and
goals
From sensor data
By mining textual description
By engineering commonsense knowledge
Tracking and predicting a user’s behavior
Noisy and incomplete sensor data
Recognizing user errors
First steps toward proactive assistive technology
ACCESS
Assisted Cognition in Community, Employment, &
Support Settings
Supported by
The National Institute on Disability & Rehabilitation
Research (NIDDR)
The National Science Foundation (NSF)
Learning & Reasoning About
Transportation Routines
Task
Given a data stream from a
wearable GPS unit...
Infer the user’s location and mode of
transportation (foot, car, bus, bike, ...)
Predict where user will go
Detect novel behavior
User errors?
Opportunities for learning?
Why Inference Is Not Trivial
People don’t have wheels
Systematic GPS error
We are not in the woods
Dead and semi-dead zones
Lots of multi-path propagation
Inside of vehicles
Inside of buildings
Not just location tracking
Mode, Prediction, Novelty
GPS Receivers We Used
GeoStats wearable
GPS logger
Nokia 6600 Java Cell
Phone with Bluetooth
GPS unit
Geographic Information
Systems
Street map
Data source: Census 2000
Tiger/line data
Bus routes and bus stops
Data source: Metro GIS
Architecture
Learning Engine
 Goals
 Paths
 Modes
 Errors
GIS
Database
Inference Engine
Probabilistic Reasoning
Graphical model:
Dynamic Bayesian network
Inference engine:
Rao-Blackwellised particle filters
Learning engine:
Expectation-Maximization (EM) algorithm
Graphical Model (Version 1)
Transportation Mode
Velocity
Location
Block
Position along block
At bus stop, parking lot, ...?
GPS Offset Error
GPS signal
Rao-Blackwellised Particle
Filtering
Inference: estimate current state
distribution given all past readings
Particle filtering
Evolve approximation to state distribution using
samples (particles)
Supports multi-modal distributions
Supports discrete variables (e.g.: mode)
Rao-Blackwellisation
Each particle includes a Kalman filter to represent
distribution over positions
Improved accuracy with fewer particles
Tracking
blue = foot
green = bus
red = car
Learning
User model = DBN parameters
Transitions between blocks
Transitions between modes
Learning: Monte-Carlo EM
Unlabeled data
30 days of one user, logged at 2
second intervals (when outdoors)
3-fold cross validation
Results
Model
Mode Prediction
Accuracy
Decision Tree
(supervised)
55%
Prior w/o bus info
60%
Prior with bus info
78%
Learned
84%
Probability of correctly
predicting the future
Prediction Accuracy
How can we
improve predictive
power?
City Blocks
Transportation Routines
Goals
A
B
Work
work, home, friends, restaurant, doctor’s, ...
Trip segments
Home to Bus stop A on Foot
Bus stop A to Bus stop B on Bus
Bus stop B to workplace on Foot
“Learning & Inferring Transportation Routines”, Lin Liao, Dieter
Fox, & Henry Kautz, AAAI-2004 Best Paper Award
Hierarchical Model
gk-1
gk
Goal
tk-1
tk
Trip segment
mk-1
mk
Transportation mode
xk-1
xk
x=<Location, Velocity>
zk-1
zk
GPS reading
Hierarchical Learning
Learn flat model
Infer goals
Locations where user is often motionless
Infer trip segment begin / end points
Locations with high mode transition probability
Infer trips segments
High-probability single-mode block transition
sequences between segment begin / end
points
Perform hierarchical EM learning
Inferring Goals
Inferring Trip Segments
Going to work
Going home
Correct goal
and route
predicted 100
blocks away
Novelty & Error Detection
Approach: model-selection
Run several trackers in parallel
Tracker 1: learned hierarchical model
Tracker 2: untrained flat model
Tracker 3: learned model with clamped final
goal
Estimate the likelihood of each tracker given
the observations
Detect User Errors
Untrained
Trained
Instantiated
Application:
Opportunity
Knocks
Demonstration (by Don
Patterson) at AAHA
Future of Aging Services,
Washington, DC, March,
2004
CARE
Cognitive Assistance in Real-world Environments
supported by the Intel Research Council
Learning & Inferring Activities
of Daily Living
Research Hypothesis
Observation: activities of daily
living involve the manipulation of
many physical objects
Cooking, cleaning, eating, personal
hygiene, exercise, hobbies, ...
Hypothesis: can recognize
activities from a time-sequence of
object “touches”
Such models are robust and easily
learned or engineered
Sensing Object Manipulation
RFID: Radiofrequency ID
tags
Small
Semi-passive
Durable
Cheap
Where Can We Put Tags?
How Can We Sense Them?
coming... wall-mounted “sparkle reader”
Example Data Stream
Making Tea
Building Models
Core ADL’s amenable to classic
knowledge engineering
Open-ended, fine-grained models:
infer from natural language texts?
Perkowitz et al., “Mining Models of
Human Activities from the Web”,
WWW-2004
Experimental Setup
Hand-built library of 14
ADL’s
17 test subjects
Each asked to perform
12 of the ADL’s
Data not segmented
No training on
individual test subjects
Activity
Prior Work
CARE
Accuracy/Recall
Personal Appearance
92/92
Oral Hygiene
70/78
Toileting
73/73
Washing up
95/84
100/33
Appliance Use
100/75
Use of Heating
84/78
Care of clothes and linen
100/73
Making a snack
100/78
Making a drink
75/60
Use of phone
64/64
Leisure Activity
100/79
Infant Care
100/58
Medication Taking
100/93
Housework
100/82
General Solution
Quantitative Results
Point Solution
Quantitative Results
General Solution
Anecdotal Results
Point Solution
Anecdotal Results
Pervasive Computing, Oct-Dec 2004
Current Directions
Affective & physiological state
agitated, calm, attentive, ...
hungry, tired, dizzy, ...
Interactions between people
Human Social Dynamics
Principled human-computer interaction
Decision-theoretic control of interventions
Why Now?
A goal of much work of AI in the 1970’s
was to create programs that could
understand the narrative of ordinary
human experience
This area pretty much disappeared
Missing probabilistic tools
Systems not able to experience world
Lacked focus – “understand” to what end?
Today: tools, grounding, motivation
Challenge to Nanotechnology
Community
Current sensors detect physical or
physiological state: user mental
state must be indirectly inferred
To what can extend can
nanotechnology afford direct
access to a person’s emotions and
intentions?