Transcript Datasets-ParticipantSlides

Fabio Pianesi
Massimo Zancanaro
FBK-irst
Alessandro Cappelletti, Bruno Lepri,
Nadia Mana
Research questions

Recognition of is happening at a given time
slice (mainly) from audio-visual signals
Fabio Pianesi & Massimo Zancanaro FBK
Data sharing requirements

Indipendent modules for

Visual recognition robust for lighting for



Detecting and recognizing objects
Attentional module to focus cameras, mics and other sensors where
“the action is”


Detecting parts of the body
Avoids continuous streams of data from the environment
Standards


Annotation of activities
Meta-data descriptions



Type of sensors, their relative position in the environment
Environment description (relative to sensors: riverberation, …)
Standard for data storing
Fabio Pianesi & Massimo Zancanaro FBK
Gregory D. Abowd
Georgia Tech
Research questions

How to address questions of health and (to a
lesser degree) sustainability through
instrumentation and augmentation of the home.
How to enable others to collect data in real
homes.

Chronic care management


Early detection and monitoring of
interventions for autism

Video data and sensor data
Room #1
Sensing for the masses

Infrastructure mediated sensing data from
real homes

Energy awareness, location-tracking
Gregory D. Abowd, Georgia Tech
Room #n
Room #2
Bus
Monitoring
Sensors
Electrical Machine
Outlets and Learning
Appliances System
Air Ducts
Inferred
Plumbing Human
Fixtures
Activity
Data sharing requirements

I want annotated home movies of young child behavior, or at least
movies that I can annotate and make available as a shared data set for
the vision community to work on.

I want to provide (through commercial efforts) the ability to collect lowlevel sensor data of home activity so that you can collect data in real
homes.
Gregory D. Abowd, Georgia Tech
Aaron Crandall
Washington State University
D.J. Cook, M. Schmitter-Edgecombe,
Chad Sanders, Brian Thomas
Collecting and Disseminating Smart Home
Sensor Data in the CASAS Project
D.J Cook, M. Schmitter-Edgecombe, Aaron Crandall, Chad
Sanders and Brian Thomas
[email protected]
CASAS Testbed
• Comprehensive
• Agent-oriented
• Both office and living spaces
• Scripted and unscripted data
• Focused on ADL detection
The Space & Sensors
• Describing the physical space:
• Implications on resident behavior
• Issues with changes
• The sensors:
• Location
• Relationships
• Implications
• Configurations
• Versions
The Data Fields and Format

When collected, the CASAS data is very simple:
Annotation & ADLs
• Correct annotation is still a limiting factor
• Detail of annotation drives cost of effort and accuracy
• Proper notation of both correct activity completion
and activity errors
Final Core Issues
• Ensuring clean data
• Annotation accuracy & length
• Generating sufficiently varied data
• Properly describing test bed configurations
WSU Smart Home Dataset
Available Now
Thank you
Shared Datasets:
http://www.ailab.wsu.edu/casas/datasets.html
Contact info:
Aaron S. Crandall
[email protected]
Diane J. Cook
[email protected]
Lorcan Coyle
[email protected]
Lero – The Irish Software Engineering
Research Centre
University of Limerick
Juan Ye, Susan McKeever, Stephen
Knox, Matthew Stabeler, Simon
Dobson, and Paddy Nixon
University College Dublin
Research questions

we are interested in activity recognition



Bayesian networks & lattice theory,
Dempster Shafer evidence theory, casebased reasoning
more realistic and/or more crisp datasets for
evaluations
we are also gathering our own datasets


based on best principles? - CASL
(also we have some “toy datasets”)
Lorcan Coyle, Lero@UL

McKeever et al.,
Pervasive LBR 2008

Stabeler et al.,
Pervasive LBR 2008

Knox et al., RIA 2008

Ye et al., RIA 2008

Ye et al., ICPS 2008

Ye et al., Percom
2009
Data sharing requirements

there should be a web-based repository like the UCI ML repository

we need a common language for datasets


algorithms should be released!


like Weka or in Weka?
results need to be published beyond the paper!



and parsers to allow interoperability
put results up with the datasets
tag datasets with 3rd party opinions and cite the paper where the
results are presented
ultimately we need to make it transparent for reviewers/scientists to
understand a “good result”
Lorcan Coyle, Lero@UL
Fernando De la Torre
Jessica Hodgins
Javier Montano
Sergio Valverde
Carnegie Mellon
University
http://kitchen.cs.cmu.edu/
Research questions

How to build good computation models to characterize subtle human motion?

Develop machine learning algorithms for activity recognition and temporal
segmentation (supervised/unsupervised) of human motion

Judgments about the quality of motion

How to select or fuse multimodal data for activity recognition?

What should be a good protocol for multimodal data capturing?
Fernando, Carnegie Mellon University
Data sharing requirements

Shared datasets:
 45 people cooking 5 different recipes (brownies, salad, pizza, sandwich,
eggs)
 Each recipe is about 22 minutes and 5 synchronized modalities are
recorded (audio, video, motion capture, inertial measurement units)
 Anomalous situations (falling, fire, mistaken putting soap rather than
salt, …)
 Camera calibration parameters, time stamps for each modality
 Shared labels for object recognition, temporal segmentation and activity
recognition

Shared code:
 Multimodal data visualization toolbox (Matlab).
 Baseline experiments for activity recognition and temporal
segmentation.
 Aligned Cluster Analysis: Clustering of time series.
Fernando, Carnegie Mellon University
James Fogarty
Assistant Professor
Computer Science & Engineering
Research questions

Attacking human-computer interaction problems
using statistical machine learning

Previously with a significant focus on sensing
 Sensor-based human interruptibility models
 Privacy-sensitive approach to collecting sensed
data in location-based applications
 Unobtrusive home activity sensing
(collaborations pulling me back into this)

More recently focused on domains where it is
actually possible to attack the entire problem
 End-user interactive concept learning
(with application in Web image search)
 Mixed-initiative information extraction
(with application to semantifying Wikipedia)
James Fogarty, University of Washington
Data sharing requirements

Convincingly answering compelling HCI questions typically requires
some custom data collection (either formative or summative data)




Those datasets are expensive and difficult to collect
We therefore look for the minimal collection to answer our question
Rendering the collected data largely useless for other questions
Data sharing can have important value, but we also need to examine
other approaches to achieving the same intended benefits





Work on different problems (like the Web, where there’s lots of data!)
Improved coordination of collection (work with others to reduce costs)
Improved standardization of collection (agree what’s important to collect)
Improved collection tools (lower barrier to getting it in the first place)
Improved annotation tools (lower barrier to coding it later)
James Fogarty, University of Washington
Stephen Intille
Massachusetts Institute of Technology
Research questions

How can just-in-time information presented by context-aware
technology in the home and worn on the body help people stay
healthy as they age?

How do we make activity detection algorithms that work for nontechies in real life in complex situations using practical and affordable
sensor infrastructures?
End-user concerns/challenges that have not been adequately
addressed…
* Practical sensor installation
* Maintaining sensors
* Fixing mistakes
* Adding activities
Toothbrushing
Stephen Intille (MIT)
Data sharing requirements

What shared datasets or tools, if any, would best advance your
work (on automatic detection of activity for health systems) ?

Datasets of 10-100 families in their homes doing everyday activities for
months with accurate labeling of activity, postures, and audio
transcription and synchronized with data from 3-axis accelerometers
on each limb, object usage data on as many objects as possible,
current flow sensing on electrical devices, and indoor position
information on each occupant (1m accuracy).

Datasets of 10-100 people doing everyday activities in natural settings
for weeks or months with accurate labeling of type and intensity
(energy expenditure) of physical activity while wearing 3-axis
accelerometers on each limb.
Stephen Intille (MIT)
Taketoshi MORI
Mechano-Informatics,
The University of Tokyo
Masamichi Shimosaka,
Akinori Fujii,
Kana Oshima,
Ryo Urushibata,
Tomomasa Sato,
Hajime Kubo,
Hiroshi Noguchi
Sensing Room and Its Resident Behavior Mining
CHI 2009 Workshop
Developing Shared Home Behavior Datasets to Advance HCI and Ubiquitous Computing Research
Research questions
Sensing Room and Its Resident Behavior Mining

We have constructed several room-type
human behavior sensing environments. These
used many distributed sensors. The key was
location sense. The problems were




A long-term recording is difficult,
Time synchronization is difficult,
Annotating is such a bother!
Based on the collected behavior data, we
have been constructing services such as action
anticipation, beat-one information display and
robotic support.

We introduced for these problems,



Multi-layered network system,
Distributed object software scheme,
RDF/OWL knowledge representations.
Taketoshi Mori, the University of Tokyo
Data sharing requirements
Sensing Room and Its Resident Behavior Mining

Developing algorithms to detect unusual behavioral phenomenon or to
foresee stereotyped frequently occurring behaviors for supporting
human, it is necessary to obtain human’s position in the space with
timestamp. Distributed sensors should supply sufficient information to
estimated the human position. It may help if the timestamp is marked
both at the sensed time by sensors and the recorded time by the home
server.

The datasets with many additional data,
such as the resident’s profile, 3D room
models, the wall and floor textures, the
weather and temperature help to construct
an appropriate behavior estimation method.

The datasets should be constructed based on
some tagged formats as XML or YAML, and
preferably the tags are added following RDF.
Taketoshi Mori, the University of Tokyo
Tim van Kasteren
Intelligent Systems Lab Amsterdam
University of Amsterdam
Co-author: Ben Kröse
Research questions


Which probabilistic model is best for modeling human behavior?

How to deal with unsegmented data?

How to capture long term dependencies?

How to deal with the large number of ways in which activities can be performed?
How can we apply these models on a large scale, without the necessity of
training data from each house they are applied?

How to deal with different layout of houses?

How to deal with different behavior of people?
Tim van Kasteren (University of Amsterdam)
Data sharing requirements



To validate the effectiveness of our models, we need:

Datasets consisting of several days (weeks) of data recorded in a real
world setting.

We have mainly used wireless sensor networks, but we are interested in
validating our models on other sensing modalities as well.
To validate the application of our models on a large scale, we need:

Datasets from multiple houses.

Ideally consisting of a fixed set of sensors and labeled activities.
We offer:

Several real world datasets consisting of at least two weeks of fully labeled
data each.
Tim van Kasteren (University of Amsterdam)
Sumi Helal
University of Florida, Andres MendesVazquez, Diane Cook and Shantonu
Hussein
www.icta.ufl.edu
Research questions

How can we synthesize sensory datasets either from scratch or by “stemcelling” existing actual datasets?

Synthesis is necessary to enable researchers with limited resources but
with great ideas and algorithms that need to be thoroughly tested.

Synthesis could also be needed by the owner of an actual dataset, to
enable/him/her to go back in time and explore additional concerns/goals
not thought of during data collection.

What are the synthesis strategies/algorithms?

How good are synthesized datasets? How can we assess our success or
failure in this direction.

What does “Sensory Dataset Description Language” standard has to do
with data synthesis?
Sumi Helal, University of Florida
Data sharing requirements

Simply, access to a database of well documented datasets will
advance our research and tool development in sensory data synthesis.

What is of great importance to us is documentation of the “protocol”
used to collect the data, not just the data itself.

To be able to utilize other people datasets, and to foster a greater level
of interoperability and cross use of data sets, we have been working
on defining a standard to propose to the community. We call the
standard: “Sensory Dataset Description Language” or SDDL. The
SDDL specification proposal can be downloaded from:
http://www.icta.ufl.edu/persim/sddl/

We have utilized 4 datasets in defining this standard proposal. We
wish to consider many more datasets in refining this proposal. Your
comments AND contributions to SDDL are sought.
Sumi Helal, University of Florida
Allen Yang
with Phil Kuryloski and Ruzena Bajcsy
UC Berkeley
DexterNet: A Wearable Body Sensor System


Primary Goals
1.
Real-time control & sampling of
heterogeneous body sensors
2.
3.
Secured surveillance in indoors and outdoors
System Architecture
1.
2.
3.

Provides geographical and social data
Body Sensor Layer (BSL)
Personal Network Layer (PNL)
Global Network Layer (GNL)
Prototype Systems
1.
2.
3.
Human action recognition
State-of-the-art security features
Real-time communication between Berkeley
and Vanderbilt Hospital tested
Allen Yang, UC Berkeley
Reference: BSN Workshop, 2009.
Wearable Action Recognition
Database (WARD), version 1

Free for noncommercial users

5 motion sensors, each carries an accelerometer
and gyroscope sampled at 30 Hz

20 test subjects (13 male & 7 female) ages 19-75

13 action categories collected in an indoor lab

standing
sitting
sleeping
walking
running
jumping
turning
upstairs/downstairs
pushing objects
Data processed in Matlab. Visualization tool is
included
Allen Yang, UC Berkeley
Workshop schedule












9:00 Overview and goals
9:15 Introductions by attendees
10:30 Break
10:45 Targeted questions and answers
12:00 State-of-the-art in data collection
12:30 Lunch
14:00 Discussion: What's possible?
14:20 Group exercise
15:20 Group presentations
16:00 Break
16:15 Next steps
17:30 End of workshop
Gregory D. Abowd
Georgia Tech
Question & Answers #1

Why do you want family home movies?

Sufficient retrospective research in the autism domain has shown
that there is evidence of developmental delay in home movies. This
has value for early detection and early intervention.

We have shown that you can encourage the collection of relevant
developmental milestone behavior from parents, but not of rich
evidence like video.

We are working on filtering techniques to pull out the relevant
snippets of social interaction.

Ultimately, I envision a way to upload home movies to a secured
service that can then extract relevant portions to share with a
pediatrician or other professional for screening purposes.
Gregory D. Abowd, Georgia Tech
Question & Answers #2

What is the value of infrastructure mediated sensing to other
researchers?


This is a way to gather low-level sensing data from real homes.
There is both commercial and research opportunities here and I
think the commercial opportunities in demand-side energy
management may be able to drive the ability to provide valuable
resources for researchers to leverage.
Gregory D. Abowd, Georgia Tech
Fabio Pianesi
Massimo Zancanaro
FBK-irst
Alessandro Cappelletti, Bruno Lepri,
Nadia Mana
Question & Answers #1

How would low-bandwidth sensing (e.g. passive infrared motion
detection, object movement sensors, RFID) complement the methods
used in the NETCARITY project

Attention mechanism to that activates/deactivates camera/mikes when
someone enters a room

Fusion of multiple modalities


Recognition of objects (manipulation)
Information about body (and body segments) activity levels, posture
changes, etc.
Fabio Pianesi & Massimo Zancanaro FBK
Question & Answers #2

How could the data on target behaviors in NETCARITY be used to improve
segmentation of activities in recordings of ongoing natural behavior?

Segmentation is a ill-posed problem because it confuses two level: the
description of an activity and the intention of the performer

Example:


While I cook spaghetti, I go to the restroom. A friend call and I say “I’m
cooking” (still in the restroom!)

I grab an hammer and my wife asks me about what I’m doing: “I’m hanging
the painting” but I have not yet started (or not?)
Telic events have a clear end but still lack a clear start


If the apple is finished, you have ate an apple
But it’s hard to agree on the start (or if you leave the apple on the table)
Fabio Pianesi & Massimo Zancanaro FBK
Question & Answers #3

How might recordings from the high density microphone arrays used in
this project provide value to other researchers? Would this justify the
cost?

For what concerns event detection, we had disappointing results from
microphone

They can be useful to monitor verbal and para-verbal activities to
estimates:


personality traits (Pianesi et al. 2008; Lepri et al. 2009)
mood
Fabio Pianesi & Massimo Zancanaro FBK
Aaron Crandall
Washington State University
D.J. Cook, M. Schmitter-Edgecombe,
Chad Sanders, Brian Thomas
Lorcan Coyle
[email protected]
Lero – The Irish Software Engineering
Research Centre
University of Limerick
Juan Ye, Susan McKeever, Stephen
Knox, Matthew Stabeler, Simon
Dobson, and Paddy Nixon
University College Dublin
Combining Redundant Data
“In the CASL Dataset, how might overlapping data from Ubisense locator,
pressure mats, and Bluetooth spotters be used to good advantage?”

tells us a lot about the data quality

reveals when sensors are not operating optimally

allows us to make more educated guesses

we can test with subsets

the sensors aren’t where the cost is (imho)

without redundant data streams there are certain algorithms we cannot
test


voting/weighting strategies?
it’s easier to play with DS evidence theory
Lorcan Coyle, Lero@UL
Bootstrapping Users to a Dataset
“Describe the concept of “bootstrapping” datasets for new users and
discuss how this might be done efficiently for large long-term datasets”

release parsers/interfaces to deal with your dataset

how about sample experiments?

really simple worked-through tutorial examples subsets of the dataset

e.g., using only RFID and object sensors:




10:12pm: Prof. Plum enters kitchen
10:14pm: candlestick sensor active
10:16pm: Prof Plum enters hallway
reducing the learning curve
Lorcan Coyle, Lero@UL
Fernando De la Torre
Jessica Hodgins
Javier Montano
Sergio Valverde
Carnegie Mellon
University
Question & Answers #1

How might body motion capture be most practically implemented in a
natural home environment?

Wearable:


Small wireless Inertial Measurement Units distributed through the
body.
Instrumented environment:


Sparse information: motion sensors around the house.
Rich information: multiple cameras (at least 3)
Fernando, Carnegie Mellon University
Question & Answers #2

Given the choice between high resolution (1024x768, 30fps) or high
frame rate (640x480, 60fps) video, which do you think would be more
beneficial to the greatest number of researchers?

It depends on the task

Subtle activity recognition (e.g. grasping a fork) or object recognition
probably is better higher resolution

Egomotion computation from wearable camera or fast activities such
as cutting a cucumber, probably better higher frame-rate
Fernando, Carnegie Mellon University
James Fogarty
Assistant Professor
Computer Science & Engineering
Question & Answers #1

Discuss why you believe it is or is not possible to collect
general-purpose shared datasets on home behavior.

Very simple to collect shared datasets on home behavior
(see the website for this workshop, we’ve already succeeded!)

The notion that a dataset is general purpose is what makes it difficult

Data collection is hard enough and expensive enough
when focused on answering your own research questions

Asking the question also implies that recognition is the goal

Is that what we’re doing here in the HCI community?

Maybe we should be looking for the HCI contributions we can make
without solving the hard general activity recognition problem
James Fogarty, University of Washington
Question & Answers #2

Describe one way researchers might solicit and distill community input
before undertaking a data collection project.

Focus on researcher awareness of the benefits they personally might
obtain from collecting data, then encourage them to share

Tag existing datasets by the types of data they contain

When designing a new data collection, a researcher can easily see
what kinds of data other people have previously collected in tandem
with what you are already planning to collect

Can also see why they collected it, imagine what additional benefit its
collection would have to your current research

Try to identify a way to solicit wishlists or shortcomings of datasets

Make it really easy to search, link to related tools, etc.
James Fogarty, University of Washington
Stephen Intille
Massachusetts Institute of Technology
Question & Answers #1

Do you think low cost, off-body sensors be used to detect
postural transitions or physical activity with any useful degree of
accuracy?
Best hope: computer vision (technically and socially tough)
Interesting question: how close can you get when other sensors are
ubiquitous?
Stephen Intille (MIT)
Question & Answers #2

How are you addressing the logistical challenges associated with
mobile computing research (e.g., user compliance, comfort,
battery life)?
- One night, one recharge
- Same thing every day
- Phone prompting if no compliance
- Looking for apps to inspire compliance
- Leave stuff by the door (retrain user)
- Sensors: small enough to wear under clothing
- Tricky IRB/social issue: what to do outside home
Stephen Intille (MIT)
Taketoshi MORI
Mechano-Informatics,
The University of Tokyo
Masamichi Shimosaka,
Akinori Fujii,
Kana Oshima,
Ryo Urushibata,
Tomomasa Sato,
Hajime Kubo,
Hiroshi Noguchi
Sensing Room and Its Resident Behavior Mining
CHI 2009 Workshop
Developing Shared Home Behavior Datasets to Advance HCI and Ubiquitous Computing Research
Question & Answers #1
Sensing Room and Its Resident Behavior Mining

What challenges would you anticipate for installing magnetic
motion capture in natural homes? What alternative strategies for
capturing bodily motion would you consider?

Magnetic motion capture systems work poorly when
there exist many metallic things. Also, there may be
troublesome cables between magnetic sensors
distributed on human body and the controller. We do
not expect the magnetic capture systems as the usual
behavior collecting way, but it may be used to prepare
ground truth motion/posi-tion since other mo-caps such
as optical/super-sonic-based are greatly influenced by
occlusions.

2D/3D stationary laser range sensors may be used to
measure human position and pose. Appliances usage
tells a lot about home behaviors. Some people may wear
wrist-watch type accelerometer with gyros.
Taketoshi Mori, the University of Tokyo
Question & Answers #2
Sensing Room and Its Resident Behavior Mining

Describe your schema for annotating behaviors in Sensing Room.
What were strengths and weaknesses of the annotation
procedure?

Sensing Room accumulates the resident’s place by its floor distributed
pressure sensors, object carrying actions by RFID readers, and many
other acts by electric switches or electric current sensors. All the data
are put together and can be displayed as 3D-CG images. Watching the
CG video, several researchers write down the behavior annotation by
hand.

Our procedure has advantage that it can be
done offline. No camera surveillance is
required. But, of course, it has the
weakness that the correctness depends
both on the lucidness of the graphics and
the interpretation of the annotators.
Taketoshi Mori, the University of Tokyo
Tim van Kasteren
Intelligent Systems Lab Amsterdam
University of Amsterdam
Co-author: Ben Kröse
Question & Answers #1

Q: How might you change the real-time voice-activated annotation
procedure to reduce the burden on the user?

A: The current system requires the user to constantly be aware of the
activity he/she is involved in and report this.


If the system would ask the user what activity is being performed
this would reduce the burden.

The system could ask the user at times when sensor patterns are
most ambiguous with respect to the activities annotated.
However, the question remains how this effects the reliability of the
annotation method.
Tim van Kasteren (University of Amsterdam)
Question & Answers #2

Q: Describe how the activities to be annotated were selected. How and
why would you change this list in future data collection?

A: Activities were selected based on previous work, literature on
activities of daily living (ADLs) and based on what would seem
challenging yet feasible using the sensor platform used.


In future data collection more detailed activities would be
annotated. For example, annotating getting tea and getting juice,
instead of getting a drink.

More detailed activities can always be grouped into a collective
activity afterwards, but extend the lifetime of a dataset.
However, there is always a trade-off between cost and gain.
Tim van Kasteren (University of Amsterdam)
Allen Yang
with Phil Kuryloski and Ruzena Bajcsy
UC Berkeley
Question & Answers #1

Advantages of wearable sensors over environmental sensors?
1. Cost less to instrument, especially in outdoors
2. Richer interaction with subjects (physiological sensors, message feedback)

Integration of wearable and environmental sensors?
1. Certain environmental sensors are not portable (size and battery)
2. Wearable sensors can provide localization services, which then correlate with
environmental information.
Airborne Particulate Matter Concentrations
Allen Yang, UC Berkeley
Question & Answers #2

Plan for future databases?
1. Human Motion Interaction: WARD version 2.
2. Integrating Geographic Data (with Oakland Children’s Hospital): Long-term
monitoring of 160 obese patients correlated with environmental factors.
3.
Integrating Social Interaction: Port DexterNet platform to consumer-ready smart
phones (iPhone and gPhone).
Allen Yang, UC Berkeley