Transcript Situational awareness in an Internet of Things Environment

Smart Textiles & Wearable Intelligence
Anna Hristoskova
1st International Summer School on eCare, August 25 - 29, 2014, Ghent, Belgium
Home automation
Light/Temp/Hum sensor
 Sensors/devices
Movement detection
integrated into the
environment
Sensor gateway
 Focus is on
integration and
standardized
communication
 Automation and
monitoring of
RFID reader
person activities
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
Magnetic door sensor
2
Smart Textiles and Wearable Intelligence
Environment:
- Can contain sensors / actuators
- Electronics + Data processing
- Wireless communication
- Powering
-…
Mattress:
- Can contain sensors/ actuators
- Conductive wiring
- Electronics + Data processing
- Wireless communication
- Powering
-…
Bedlinen:
- Can contain sensors/ actuators
- Conductive wiring
- Electronics + Data processing
- Wireless communication
- Powering
-…
Wardrobe:
- Can contain sensors/
actuators
- Electronics + Data processing
- Wireless communication
- Powering/Charging
-…
Garment:
- Can contain sensors/ actuators
- Conductive wiring
- Electronics + Data processing
- Wireless communication
- Powering
-…
Smartphone:
- Can contain sensors/ actuators
- Electronics + Data processing
- Wireless communication
- Powering
-…
Bed:
- Can contain sensors/ actuators
- Electronics + Data processing
- Wireless communication
- Powering
-…
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
3
SMARTpro – the project
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
4
T
Vision
Integration
Identify and address
industrial needs
and challenges
w.r.t. the realization,
integration and
usability
of smart textiles &
wearable
intelligence
 Combine
intelligently
and efficiently
textiles,
electronics
and ICT
Business
Opportunity
Manufacturing
processes
Communication
Industry
Users
Personalization
Innovative solutions
Components
E
Domain
Interoperability
ICT
Stimulate
collaboration
across ICT,
electronics and
textile sectors
 Manufacturing
on an industrial
scale
 Reliable and
durable
Build proof-of-concept demonstrators
in target domains such as
 Safety and intervention
 (Home) care
 Sport and leisure
 Technical applications
Involve end-users
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
5
SMARTpro – the project
Home/care
Textile sector
Intervention
Sports & leisure
Technical
Textile
Oct. 1st 2013 - Oct. 1st 2017
CROSS
SECTORIAL
COOPERATION
Electronics sector
ICT-sector
Multi-disciplinary
consortium of
research partners with
complementary expertise,
coordinated by industrial
research centers.
A steering group of core
industrial players is being
established.
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
6
Approach to selected
targeted & collective challenges
End user 1 requirements
End user 2 requirements
Company 1 Technological/manufacturing challenges
Company 2 Technological/manufacturing challenges
Taken up by company
(together with SmartPro
partner) in
separate trajectory
End user requirements +
Common technological manufacturing
challenges that can be addressed
with innovative technologies from
SmartPro partners
End user 3 requirements
Company 3 Technological/manufacturing challenges
The aim is to find a critical mass in
requirements + technological/manufacturing challenges
that is as much as possible relevant for all member companies
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
7
SMARTpro
Partner introduction
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
8
Textiles - design
 Design, development and
prototyping of textiles
products
 Digital printing & coating,
i.e., conductive inks
 Production of electrically
conductive yarns and
fabrics
 Integration of components
in textiles through various
confection techniques
 Comfort
 Body Scanning
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
9
Textiles - Integration
 Lab production of smart textiles
components
 Sensors for monitoring heart rate,
respiration rate, sweat
 Textile antenna, textiles batteries
 Textiles for heating and electrostimulation
 Printing of conductive inks
 Conductive yarns & coatings
 Optical fibres & LEDs
 Fibre transistors
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
10
Electronics - design
 Prototyping following end-user
specs
 Definition of feasible integration
prototypes
 Identification of manufacturability
and quality risks of prototypes
 Analysis of the risk factors of
electronics in textiles
 Industrial available electronic
component and PBA technology
for the intended applications
 Basic elements of design
guidelines for integrating
electronics into textiles
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
11
Electronics - Integration
Conductive yarns
(acting as sensors) in
combination with integrated
readout electronics
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
12
Textiles & Electronics
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
13
Data Intelligence
Paradigm shift to
wearable and
embedded
devices
Allow to access
and manipulate
diverse
information about
humans
Intelligent sensorbased applications
enhance decisionmaking for
professionals and
individuals
 Carried by humans or worn in/on the body as smart phones, heart rate
meters, Google Glass or the Nike+ FuelBand
 Sensors integrated into objects, such as textiles, toothbrushes,
mattresses, mirrors, thermostats, doorways, steering wheels, …
 Physiological measurements and daily habits
 Context and surroundings
 Lifestyle management for chronic illnesses such as diabetes
 Emergency situation such as fire or flooding
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
14
Data Intelligence
While hardware and software design remain important, capturing,
processing and interpreting data becomes the main concern
• The collection, modeling, and processing of large amounts of sensor data,
• The extraction of knowledge from these data and
• Its presentation to the end users
Data opportunities
• Uncover hidden insights and infer additional knowledge from data
• Enable advanced visualization of trends and patterns
• Reduce information overload and target proactive information delivery
Data challenges
• Data integration from heterogeneous sources
• Real time information processing and event recognition
• Consider information from human behaviours and multi-modal interactions
• Act on behalf of the users’ intentions
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
15
Applications
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
16
Interdisciplinary collaboration
Challenges:
Sensors, packaging and reliability
Flexible and stretchable
Substrates
… other platforms
Challenges:
Conductive yarns
Interconnection with
flexible and stretchable
substrates
Washable -> packaging
Integration
Electronics
Textiles
ICT
Monitoring,
sensing,
light
Applications
Challenges:
Comfort
Maintenance friendly
Reliable
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
17
SmartPro project
Scenarios
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
18
Home / care
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
19
Home/Care
 Concept
 continuous monitoring of the patient and his environment, at home
and on-the-go
 Target users
 Elderly
 Patients in rehabilitation phase after some medical intervention
 Formal and informal caregivers
 Objective
 Provide personalised monitoring and remote care with the goal to
 increase autonomy and quality of life
 facilitate caregivers in providing optimal support on demand
 Predict deteriorations of the patient’s state through
 Stimulation and detection of movement and inactivity
 Monitoring the evolution of important predictors (e.g. walking speed)
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
20
Home/Care
Demonstrator
Light/Temp/Hum sensor
Environment/On-the-go:
- With unobtrusive and subtle monitoring solution
e.g. Position monitoring
Movement detection
Home
- With unobtrusive and subtle monitoring solution
Sensor gateway
Rooms:
-
Equipped with sensors/ actuators
Position monitoring system
,,,
Pressure sensor
RFID reader
Magnetic door sensor
Person: - Elderly
- Patients in rehabilitation phase after some medical intervention
 To be monitored
1st International Summer School on eCare,
 Information to formal and informal caregivers - 29, 2014, Ghent, Belgium
August 25
21
Home/Care
 Monitored parameters
 Physiological
 heart rate, body temperature, galvanic skin response, EMG
 Activity
 type (walking, biking, running, ...), distance, speed, steps, ...
 Diet
 fruit and vegetable intake, total calorie estimation, ...
 Environment
 temperature, noise, light, ...
 Derived knowledge
 Habits
 sleeping, resting, eating, ...
 Anomalies
 elevated heart rate, abnormal wake-ups, too low intensity levels, ...
 Patterns
 causal relationships between activities and physical/mental condition
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
22
Home/Care
 Use cases
 Wander detection
 recognize where a patient is located and whether or not he or she is
supposed to be there or feels lost
 Fall detection - recognize if a patient made a fall.
 Monitoring evolution of important predictors (e.g. walking speed) for fall prevention for
elderly
 Activity monitoring
 recognize whether or not a patient is acting according to his or her normal
habits
 Physiological state
 monitor the patient’s health state, create a long term profile and track
possible deteriorations
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
23
Intervention
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
24
Intervention
 Concept
 monitoring target users’ status, environment and behavior
 Target users




Team of emergency workers (e.g. firefighters, police, ...)
Coordinated by a commander
On-site stewards
Evacuees
 Objective
 increase situational awareness and support decision making
in case of an incident
 Controlling the incident and get back to normal as soon as possible
 Ensuring safe and efficient evacuation of the building(s) and site
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
25
Intervention
Demonstrator
Environment:
- With monitoring solution
e.g. temperature, fire, dangerous goods
Smartphone:
- With monitoring solution
e.g. location
Garment:
- unobtrusive reliable monitoring
e.g. temperature, oxygen levels,
heart rate, stress
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
26
Intervention
Demonstrator
Fire fighters
On-sleeve
computation
Commander tablet
remote overview
Server inside
fire truck
Crisis team web-based
generalized overview
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
27
Intervention
 Monitored parameters
 Physiological
 heart rate, body temperature, galvanic skin response, blood
oxygen level
 Environment
 indoor/outdoor location, people movement, temperature, noise
& lighting levels, traffic information, presence and location of
dangerous goods
 Resources
 oxygen bottles, firearms, mobile devices, GSM/WiFi
infrastructure
 Derived knowledge
 Stress & fatigue, overcrowding of emergency exits, panic
reactions, ...
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
28
Sports & leisure
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
29
Sports & leisure

Concept


Target users




monitoring athletes during races, training and everyday life
Athletes
Coach/personal trainer
Science lab personnel
Objective

Provide personalised and actionable insights beyond mere tracking and visualisation
 No average human, different individuals have different needs and respond differently
 Knowing in retrospect that you slept bad is not so useful, knowing what you can/should (not)
do upfront in order to sleep well is more important

Improve performance and prevent injuries
 Monitor correct execution of exercises, adherence to prescribed plan, intensity levels,
recovery & sleep, muscle imbalances, ...

Detected anomalies
 high resting HR, abnormal sleep patterns, failure to meet intensity levels, incorrect posture

Provide feedback on potential causes
 cafeine intake, insufficient time between exercise, too high training volume, too high/low
intensity level, ..

Predict future evolution
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
30
Sports & leisure
Demonstrator
Garment:
- Unobtrusive monitoring of
performance
e.g. heart rate, fatigue
Environment/On-the-go:
- With unobtrusive and subtle monitoring solution
e.g. aerodynamic posture during time trials
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
31
Sports & leisure
 Monitored parameters
 Physiological
 heart rate, body temperature, galvanic skin response, EMG
 Activity
 distance, speed, steps, cadence, movement, rest, altitude, geographic
locations
 Environment
 weather, noise, light
 Derived knowledge
 Habits
 sleeping, resting, eating, travelling
 Anomalies
 elevated heart rate, abnormal wake-ups, too high/low intensity levels, tactics
 Patterns
 causal relationships between activities and anomalies
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
32
Sports & leisure
 Example use cases
 Real-time analysis of
 aerodynamic posture during time trials, in order to provide
immediate corrective feedback
 fatigue and stress during game, in order to recommend
substitutions
 Analysis of lifestyle outside training in the build-up phase
towards a big event,
 guarantee optimal recovery and to appropriately tune training
schedules
 Monitoring of correct execution of exercises and
adherence to prescribed plans
 speed up rehabilitation and prevent injuries
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
33
Technical
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
34
Technical

Concept

Indoor
 intelligent flooring/wall/car interior

Outdoor
 monitoring critical infrastructure

Target users

Large area structures for interior use
 carpets, curtains, mattress covers

Outdoor structures
 geotextiles, tents, sunscreens

Transport & Logistics
 automotive, packaging

Objective(s)

Outdoor
 improve safety by hazard & structural health monitoring of constructions

Indoor
 safety & prevention, fall and inactivity detection or prevention of inhabitants

Integration of new features, e.g. lighting, heating, cooling, energy scavenging
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
35
Technical
Virtual demonstrator
Large textile structure:
- Unobtrusive monitoring of activities and state
e.g.fall detection, structural health monitoring
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
36
Technical
 Monitored parameters
 Environment
 weather (ex. wind, precipitation), moisture, pressure, temperature
 Activity
 Outdoor
 safety, data for structural health monitoring, capture structural response to
extreme events (e.g. earthquake, storm), speed of degradation, energy
applications (flexible solar cells)
 Indoor
 safety, lighting (LED, OLED, optical fibers integration), heating or sensor
applications, location
 Derived knowledge
 Durability and safety limits of outdoor structures and
constructions
 Indoor
 Support for the other cases
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
37
Technical
 Example use cases
 Monitoring of wind/snow stress on tent construction
 Monitoring evolution of important predictors (e.g.
walking speed, sleep quality) for fall prevention for
elderly
 Monitoring crowded indoor events in case of crowd,
control, intervention during emergency
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
38
SMARTpro project
Data Intelligence
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
39
Technological challenges
(Home) care
Technical
Unobrusive monitoring
Sport & leisure
Intelligent data processing
Emergency/Intervention
Data
fusion &
1st International Summer School on eCare,
mining
August 25 - 29, 2014, Ghent, Belgium
Electronics
into textile
Textile as
carrier for
electronics
40
Intelligent data processing
– challenges
 Capture, store and fuse large amounts of data obtained from
continuously monitoring people and their environment
 physiological as well as non-physiological measurements
 context and surroundings
 Identify trends in order to construct user profiles
 typical amount of sleep, most active periods of day/week, frequently visited
POIs with typical visiting times, heart rate zones, resting heart rate, muscle
activitation, ...
 Explore methods to build prediction models
 estimated recovery period, overtraining, evolution in walking speed,
predicted energy consumption for weight loss, physiological evolution, ...
 Detect anomalies in real-time
 geo-fencing, fall detection, abnormal inactivity, increased stress level, ...
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
41
Intelligent data processing
– needs
 Historical data
 for training robust models that can predict the future based
on the past
 Sufficient & suitable data
 not necessarily ‘big’ but naturally ‘relevant’ w.r.t.
requirements
 Reliable data
 reduced level of noise
 appropriate frequency
 standardized format
 End user requirements
 Lacking knowledge of the application domain and end-user
needs, even the most intelligent algorithms are useless
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
42
Context modelling method
Observations
Software
Engineer
User
researchers

Context definition by domain experts






Field studies execution
Mental models creation
Scenario description
Co-creation workshops
Goal-directed task analysis
Co-creation
workshops
Scenarios
Domain
experts
Formal translation into an ontology & rules by ontology engineers in
collaboration with domain experts


long track record of developing ICT solutions for specific domain
accurate and complete reflection of the daily work practices

Deployment of the architecture by domain experts

Processing of usability feedback
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
43
Formal domain definition
supported by semantics
 Common vocabulary and a shared understanding of the
structure of the information
 Support communication
 Explicit domain description of
Domain
concepts
Rule set





Concepts
Properties and Attributes of concepts
Constraints on properties and attributes
Individuals
Rules to express general workflows & intelligent behavior
 Enable reuse of domain knowledge
 Introduce standards allowing interoperability
 Capture knowledge formally
 Reasoning and extraction of new knowledge
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
44
Interpretation & decision
support
Domain
concepts
Interpretation
has_symptom *
The flu
IS Curable Disease
Symptom
Is a
Fever
Disease
Reasoning
on the data
Temp
Treatment
has_treatment *
Antibiotics
Domain
Modelling
Rule
definition
Curable Disease
== Disease
AND has_treatment.Treatment
IF night
→ close the blinds
AND dim the lights
Rule set
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
Recovery
in a week
Support
actions
Stay at
home
Treatment:
Stay at home
Rest
Antiviral medications
Antibitiotics
Decision
support
45
Electronics layer
Data processing steps
1.
Pre-processing and cleaning of the sensor and device data



2.
Averages of measurements
Combination of multiple data points
Interpret on the exact meaning of the data
Evaluate the user’s context
Decision support based on the user context

5.
37°C
15°C
X,Y
36°C
37°C
X,Y
Interpretation of the situation based on the gathered data


4.
36°C
Aggregation of several data measurements from various sources


3.
Collect distributed heterogeneous data from various devices and
sensors
Clean up the data
Remove faulty measurements
Depending on the interpretation, the person’s situation can be inferred
and decision taken on which type of actions should be taken
36.5°C
X,Y
Temperature
Position
Triggering of actions depending on the situation




User profiling
Alarms
Predictions
Recommendations
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
46
Home/Care: data processing
 Interpret sensor data ...
 recognises a person is wearing the textile (based on temp. data),
where he or she is (based on localisation data) and
what he or she is doing (based on activity data)
 ... and turn it into knowledge
 identifies patterns & trends




sleep/awake pattern
typical POIs and visiting times
walking speed/steps per day
health state
 identifies anomalies/deteriorations
 abnormal sleeping behaviour (e.g. during the day)
 geo-fencing
 moving from more active to more passive lifestyle (e.g. prolonged sitting
instead of walking)
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
47
Technological challenges
Vakgroep Textielkunde
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
48
Anna Hristoskova
Questions?
1st International Summer School on eCare,
August 25 - 29, 2014, Ghent, Belgium
49