Transcript Personal Assistant System .(English)

Personal Assistant System: Use of
Computer Technologies to Facilitate
Elderly People with Assisted Living
Presenter: Jennifer C. Hou‡
Collaboration with Linda Ball*, Stanley Birge*, Marco Caccamo‡, Carl
Gunter‡, Karrie G. Karahalios‡, Narasimhan Nitya #, Lui Sha‡, Yang Yu#
‡ Dept of Computer Science University of Illinois at Urbana Champaign
*College of Medicine, Washington University in Saint Louis
#Pervasive Platforms and Architecture Lab, Motorola Labs
[email protected], 217-265-6329
Population Aging



Aging of the baby boomer has
become a social and
economical issue.
In the United States alone, the
number of people over age 65
is expected to hit 70 million by
2030, almost doubling from 35
million in 2000.
Table compiled by the U.S. Administration on Aging based on data from the U.S.
Census Bureau.
Percentage of People of 65+ and 85+



People over age 65 are expected
to constitute 20% of the
population in 2030.
Similar increases are expected
worldwide.
Table compiled by the U.S. Administration on Aging based on data from the U.S.
Census Bureau.
Similar Expenses Worldwide
2002
SOURCE: United Nations ▪ “Population Aging ▪ 2002”
Similar Expenses Worldwide
2030
SOURCE: United Nations ▪ “Population Aging ▪ 2002”
Consequences Are …..


Along with the increase of elderly people population, the
expenditures of the United States for health care will project
to rise to 15.9% of the GDP ($2.6 trillion) by 2010.
-- Health care industry study, Digital Foresight
Many elderly people will stay at home, rather than being
consigned to expensive retirement homes.


Even today, only 10% of elderly people of age 65-85 and 25% of
those of age >= 85 are institutionalized.
Many elderly people choose to stay at home also for privacy/dignity
issues.
Overview of the Talk



Why Do We Care About Senior Care?
Overview of Personal Assistant System (PAS)
Underlying Research that Makes Things Work







Do Elderly People Accept This Technology?


Systems Architecture
Wireless communications
Tracking and localization
Security and privacy
Software safety, reliability, and availability
Human computer interface
Pilot studies
Where Do We Go From Here?
Why Do We Care About Senior Care


Because your parents are the next in line to be qualified as
seniors, and you are next to the next in line.
Can advances in sensing, object localization, wireless
communications technologies


enable elderly people to regain their capability of independent
living?
make possible unobtrusive supervision of basic needs of frail elderly
and thereby replicate services of on-site health care providers?
We believe the answers are Yes!
We Believe Technology Can Help

Time Driven Reminders of
Daily Activities:
Home PC




Serves as the intelligence.
Sends reminder messages
to wireless-enabled
appliances.
Closes the loop with HCI/
localization techniques.
Takes action in the lack of
response: A reminder can
be sent more times, after
which a designated on-site
personnel or a healthcare
provider is notified.
Jennifer, It is
8:30am.
Time to take your
Insulin injection
before breakfast.
Overview of PAS
Assisted Living Hub (ALH)
ALSP
Clinician
Service
Gateway
Medical
Device

A drop-box architecture in which

A security-enhanced, assisted living device called Authentication Manager
for You (AMY) co-exists with a home PC, called the Assisted-Living Hub
(ALH).

The ALH is equipped with multiple wireless interfaces and serves as the
local intelligence. It also communicates with the Assisted Living Service
Provider (ALSP).

Through web interfaces, ALSP allows healthcare providers and clinicians to
retrieve/analyze data and give instructions.
A More Technical View of PAS
Home Environment
ALSP
ALH
IEEE 802.11
WLAN
.
AMY
Internet
• Easy to deploy.
• Few entities that are subject to security attacks.
• Numerous applications can be built on top of it.
Monitoring
Service
Clinicians
What Applications Are Most Critical
Factors Contributing to Loss of Independence and Institutionalization*
% of Residents
% of Residents as
Need
With Need
Primary Cause
________________________________________________________________________
Needs prompting to take medications
95
42
Risk of injury due to falls
42
17
Unable to get up after a fall
20
17
Monitoring of vital signs too labor intensive
12
20
Needs physical assistance with Activities of Daily Living
90
67
Needs prompting to toilet on a schedule
67
17
Needs prompting to go to meals
33
10
Needs prompting to bathe
75
0
Gets lost in apartment
17
0
May wander out of facility
12
10
Needs monitoring of blood sugar frequently
20
8
Needs monitoring of weight daily/weekly
25
0
* Respondents include 8 geriatricians, 10 nurse administrators of assisted living
facilities, and 6 home health clinicians at Washington University in Saint Louis
Application I: Time-based Reminder Services

PAS can help reminding residents of daily activities.



ALH obtains from the ALSP updated prescription and appointment
records of a resident.
When it is time for the resident to carry out their time driven
routines, the ALH locates active wireless-enable devices and sends
reminder messages to one or more devices that are in the
proximity of the resident.
Whether or not these routines are followed as advised is detected
in a non-intrusive manner by exploiting sensor localization
techniques.
Assisted Living Hub (ALH)
ALSP
Gateway
Medical
Device
Clinician
Service
Application II: Monitoring of Physiological Functions



A number of physiological functions critical to maintaining homeostasis
for different medical conditions can be measured by Bluetooth-enabled
medical devices, transmitted to the ALH and then to the ALSP to be
evalauted by healthcare providers.
Measures will have a prescribed desired range and deviations from that
range will generate an alert from the ALSP to the health care provider.
This enables prompt intervention before the situation deterioates to a
point requiring hospitalization and may simply be in the form of
additional instructions to the resident.
ALSP
Home Environment
IEEE 802.11
WLAN
. AMY
.
Internet
Monitoring
Service
Application III: Non-intrusive Monitoring of Daily
Activities and locations

Detection of early warning signs for depression and/or
other chronic diseases:


The location of a resident and the duration in which he/she stays
at that location are profiled by Motes-based (or Ubisense-based)
sensing and tracking in a privacy-preserving manner.
Movement profiles are transmitted to the ALSP and analyzed (if
necessary) for
 Early warning signs for severe depression (e.g., not taking
medicine, not eating, staying in bed for very long time)
 Preventive measures (behavior changes) for chronic diseases
commonly seen in elderly people (such as Parkinson’s disease
and/or Alzheimer’s disease).
Application IV: Fall Detection



Residents wear sensors equipped with accelerometers (with fall
detection algorithms that detect falls with the combination of speed and
orientation changes).
In the case of fall detection, the sensor device beeps and an alert
message is sent to the ALSP and to a designated healthcare provider.
In the case of false alarm, the resident can press a button and disable
the message sending.
Overview of the Talk



Why Do We Care About Senior Care?
Overview of Personal Assistant System (PAS)
Underlying Research that Makes Things Work







Do Elderly People Accept This Technology?


Systems Architecture
Wireless communications
Tracking and localization
Security and privacy
Software safety, reliability, and availability
Human computer interface
Pilot studies
Where Do We Go From Here?
Software Infrastructure Needed

A software infrastructure that integrates sensing,
communication, and event/information management.

Understand, analyze, structure and control the complex interactions
across the layers of computing, communication and sensing along
the dimensions of robustness, reliability, QoS, security and privacy.
Reminders/social
interaction
Monitoring
Emergency event processing
Telemedicine
Interface mgmt
Events/data management
Dependability Evolvability
Communication
Sensing
Real-time
Security
&
Privacy
Networked Software Architecture
Socket s = new
Socket(myaslp.com:90)
Active Application A
ss.accept();
ALSP Server
Under Normal Situation
J2ME API for Accessing
BT Nodes and Internet
TCP/IP
Stack
ServerSocket ss = new
ServerSocket(90);
When the Gateway is Available
Fixed
ALH BT Stack
OS
TCP/IP
Comm.
API
TCP/IP
Stack
Bluetooth
Vital Sign Meters
IP Routing Service
802.11
802.11
MAC
Wireline
MAC
Gateway
Cellphone
Network
+
Internet
ALSP
Server
OS
System Architecture of the ALH
OS Layer in the ALH Architecture

The OS layer


Is equipped with various
communication stacks
and corresponding
platform-dependent APIs
Depending on the family
of peripherals to
support, other stacks
and APIs may include
Bluetooth, Zigbee, and
Infrared.
Middleware Layer in the ALH Architecture

Middleware



Device monitoring
daemons: monitor the
join/leave of peripheral
devices in the environment
and registers/de-registers
the devices in the Device
Registry Services.
The register-de-register
process includes
creating/destroying proxy
or singular proxy stub
objects for the device.
The proxy/proxy stub allows applications to be built upon well-known
device APIs instead of vendor specific APIs/semantics. It also allows offthe-shelf devices to be integrated into the PAS system, as long as the
vendor provides the semantics specifications on how to communicate
with the device.
Middleware Layer in the ALH Architecture


Device registry service:
maintains a database of
peripherals available in the
environment. Each entry of
the database is a proxy (or
proxy stub) object created for
a specific peripheral device.
Applications will query this
service for appropriate
proxies. When an appropriate
proxy is found, a clone of the
proxy migrates to the
application.
Middleware Layer in the ALH Architecture

The Unified Peripheral
Communication APIs



Abstracts different network
stack programming interfaces
into a consistent paradigm,
which basically follows
java.net.* APIs.
Allows users to specify (in
XML) their QoS demands
when establishing
communication links, and the
middleware returns approved
QoS guarantees.
This allows proxies to be
developed independently of
specific network stack
programming interfaces.
Middleware Layer in the ALH Architecture

Internet Heartbeat Daemon:



periodically checks the
availability of Internet
access through the gateway
router.
When the gateway router
fails/recovers, this daemon
activates/deactivates the
Bluetooth cell phone to
access to the ALSP.
ALH Main Daemon: is in
charge of managing (start,
suspend, stop, restart etc.)
all the application daemons
and middleware daemons on
the ALH.
Overview of the Talk



Why Do We Care About Senior Care?
Overview of Personal Assistant System (PAS)
Underlying Research that Makes Things Work







Do Elderly People Accept This Technology?


Systems Architecture
Wireless communications and networking
Tracking and localization
Security and privacy
Software safety, reliability, and availability
Human computer interface
Pilot studies
Where Do We Go From Here?
Networked Software Architecture
TV
Peripheral
Bluetooth
Network
earplug
ALSP
ALH
Internet
Gateway
WLAN
(e.g. IEEE 802.11)
Internet-capable
Medical
Meters
Networked Software Architecture
Socket s = new
Socket(myaslp.com:90)
Active Application A
ss.accept();
ALSP Server
Under Normal Situation
J2ME API for Accessing
BT Nodes and Internet
TCP/IP
Stack
ServerSocket ss = new
ServerSocket(90);
When the Gateway is Available
Fixed
ALH BT Stack
OS
TCP/IP
Comm.
API
TCP/IP
Stack
Bluetooth
Internet
Vital Sign Meters
IP Routing Service
802.11
802.11
MAC
Wireline
MAC
Gateway
ALSP
Server
OS
ServerSocket ss = new
ServerSocket(90);
s = new
Networked Software Architecture Socket
Socket(128.174.11.11:90)
ss.accept()
Passive Application A
ALSP Server
Under Normal Situation
J2ME API for Accessing
BT Nodes and Internet
TCP/IP
Stack
When the Gateway is Available
Fixed
ALH BT Stack
OS
TCP/IP
Comm.
API
TCP/IP
Stack
PC IP: 128.174.11.11
Bluetooth
Dumb Dev
IP Routing Service
802.11
802.11
MAC
Wireline
MAC
Gateway
Cellphone
Network
+
Internet
ALSP
Server
OS
What If the Internet Gateway Is Down?



One major deficiency of PAS is that all the traffic is transported through
the gateway AMY, which becomes unavailable when it fails or when the
resident is away from home.
To enhance robustness and ubiquity of PAS, we have used cell phones
(transparent to users) as both a backup AMY and the local intelligence
for data aggregation and acquisition.
we have leveraged the programming capability of Motorola EZX
platforms (mainly the A780, E680 and E680i phones) to incorporate the
following novel features that are not currently present in any cell
phones:


Enabling cell phones to serve as local data storage/fusion intelligence.
Enabling cell phones as a delivery endpoint for reminder messages
Networked Software Architecture
mobile medical
TV
Peripheral
devices
Bluetooth
Network
earplug
ALH
WLAN
(e.g. IEEE 802.11)
ALSP
cellphone
Internet
GPRS network
or WiFi
Internet-capable
Medical
Meters
Base Station
When the Gateway is Not
Available
Socket s = new
Socket(myaslp.com:90)
The cellphone modem is activated
Cellphone DUN Modem
(service provided by most BT enabled cellphones)
Active Application A
J2ME + Bluetooth Dial-Up
Networking Profile
(BT-DUN) for Internet Access
APIs (TAPI/NAPI, File system, etc.)
ServerSocket ss = new
ServerSocket(90);
ss.accept();
ALSP Server
TCP/IP
Comm.
API
Service Layer (Network, Connectivity, Telephony, etc)
Fixed
ALH
OS
BT Stack
Cellphone
OS
BT Stack
GPRS
TCP/IP
Stack
Bluetooth
Bluetooth Medical
Meter
IP Routing Service
802.11
802.11
MAC
Wireline
MAC
A. L. Device
Cellphone
Network
+
Internet
ALSP
Server
OS
ServerSocket ss = new
ServerSocket(90);
The cellphone modem is activated
ss.accept()
Cellphone DUN Modem
(service provided by most BT enabled cellphones)
Active Application A
J2ME + Bluetooth Dial-Up
Networking Profile
(BT-DUN) for Internet Access
Socket s = new
Socket(128.174.11.12:90)
ALSP Server
TCP/IP
Comm.
API
APIs (TAPI/NAPI, File system, etc.)
Service Layer (Network, Connectivity, Telephony, etc)
Fixed
Butler
OS
BT Stack
Cellphone
OS
BT Stack
GPRS
TCP/IP
Stack
Bluetooth
Bluetooth Medical
Meter
IP Routing Service
802.11
802.11
MAC
Wireline
MAC
Gateway
Cellphone
Network
+
Internet
ALSP
Server
OS
Networked Software Architecture
mobile medical
Peripheral
devices
Bluetooth
Network
earplug
Cellphone
ALSP
GPRS
or WiFi
Internet
Base Station
WLAN
(e.g. IEEE 802.11)
Internet-capable
Medical
Meters
When cell phone takes
the roll of ALH
Socket s = new
Socket(myaslp.com:90)
The cellphone is used as a smart device
QoS
based
Application 1
(Listening to PC)
Active Application A
Application 2
(Sending data to ALSP
Server)
APIs (TAPI/NAPI, File system, etc.)
J2ME
ServerSocket ss = new
ServerSocket(90);
ss.accept();
ALSP Server
TCP/IP
Comm.
API
Service Layer (Network, Connectivity, Telephony, etc)
Fixed
Butler
OS
BT Stack
Cellphone
OS
BT Stack
GPRS
Or WiFi
TCP/IP
Stack
Bluetooth
Bluetooth Medical
Meter
IP Routing Service
802.11
802.11
MAC
Wireline
MAC
A. L. Device
Cellphone
Network
+
Internet
ALSP
Server
OS
Networked Software Architecture
Socket s = new
Socket(myaslp.com:90)
Active Application A
ServerSocket ss = new
ServerSocket(90);
ss.accept();
ALSP Server
When cell phone takes the roll of ALH
TCP/IP
Comm.
API
J2ME API for Accessing
BT Nodes and Internet
GPRS
Or WiFi
Fixed
ALH BT Stack
OS
TCP/IP
Stack
Bluetooth
Vital Sign Meters
Internet
GPRS network
IP Routing Service
or 802.11
Base
Station
ALSP
Server
OS
Overview of the Talk



Why Do We Care About Senior Care?
Overview of Personal Assistant System (PAS)
Underlying Research that Makes Things Work







Do Elderly People Accept This Technology?


Systems Architecture
Wireless communications and networking
Tracking and localization
Security and privacy
Software safety, reliability, and availability
Human computer interface
Pilot studies
Where Do We Go From Here?
Real-Time Tracking and Localization
We have built a system that uses a combination of ultrasonic and
RFID technologies as the underlying sensing mechanism for real-time
tracking of both human and objects.
Human Tracking:





Ultrasonic transmitters, called
beacons and denoted by Bi,
occasionally sends out an
ultrasonic pulse and a short RF
signal at the same time.
Due to the speed difference
between an RF signal and an
ultrasonic signal, the listener
denoted by Lj on the resident
can infer its distance from the
beacon using Time Difference Of
Arrivals of the two signals.
This distance measurement can
be reported to the ALH for realtime tracking of the use (marked
as (2) and (3)).
Real-Time Tracking and Localization

Object tracking



The RFID (radio
frequency identification)
reader attached on the
wristband can read
RFID tags of objects
touched by the user.
The objects can be
tracked, whenever the
user touches and/or
carries them.
The whereabouts of the
objects are logged.
Overview of the Talk



Why Do We Care About Senior Care?
Overview of Personal Assistant System (PAS)
Underlying Research that Makes Things Work







Do Elderly People Accept This Technology?


Systems Architecture
Wireless communications and networking
Tracking and localization
Security and privacy
Software safety, reliability, and availability
Human computer interface
Pilot studies
Where Do We Go From Here?
How Medical Data Can Be Securely
Transmitted ?
PubAS/PrivAS
PubC/PrivC
SOAP Envelope
Header
Medical
Information
PubC Pub
KFAS
Sign. on Msg.
PW
AP
Sign. on Key
PW
AP
PWAP
[Michael et al. 06]
KF
How Medical Data Can Be Securely
Transmitted ?
PubAS/PrivAS
SOAP Envelope
PubC/PrivC
SOAP Envelope
Header
Header
Medical
Information
Medical
Information
PubC KF
PubAS
Sign. on Msg.
Sign. on Msg.
PW
AP
Sign. on Key
PW
AP
PWAP
KF
How Medical Data Can Be Securely
Transmitted ?
PubAS/PrivAS
PubC/PrivC
SOAP Envelope
Header
Medical
Information
PubC KF
PubC
Sign. on Msg.
Priv
AS
Sign. on Key
Priv
AS
PWAP
KF
How Medical Data Can Be Securely
Transmitted ?
PubAS/PrivAS
PubC/PrivC
SOAP Envelope
Header
Medical
Information
PubC KF
PubCPub
KF C
Sign. on Msg.
Priv
AS
Sign. on Key
Priv
AS
PWAP
KF
Implementation
Assisted
Person
Medical
Device
AL Hub
ALSP
Company
Clinicians / Family or
Friends / Assisted Person
ALSP
Server
Computer
Drop-box Prototype
Message
Generator
Message
Generator
DB Access
User
Interface
AL Messages
SAAJ
Encrypt
Data
Add
Signatures/
Encrypt
Envelope
Message
Generator
Decrypt
Envelope/
Verify
Signatures
Add
Signatures/
Encrypt
Envelope
Decrypt
Data
Decrypt
Envelope/
Verify
Signatures
Add
Signatures/
Encrypt
Envelope
Decrypt
Envelope/
Verify
Signatures
AL Security
Engines
XMLSec
JCE
Apache Axis
(WSDD / SOAP, XML-RPC / Java RPC, EJB, HTTP / WSDL Tools)
Implementation

H/W





Device: Digital Pulse Oximeter
AL Hub: A labtop, Windows
ALSP: A desktop, Linux
Clinician’s computer: A labtop, Windows
S/W




Language: Java
Database: MySQL
Web service: Apache web server, tomcat, axis 1.4
Libraries: XML-SEC, SAAJ, avetanaBluetooth, etc.
Security tokens

The following security tokens are set up





Doctor’s public key certificate
The ALSP’s public key certificate
A secret key shared among AP and AP’s family members or friends
AP’s and family/friends’ password to ALSP
Access control matrix (ALSP), DataID table (AL Hub)
GDoc, GAS, KAP, U*; V*, PPA , PFA , ACLAP , ΔAP
• GDoc: Doctors office’s public key certificate.
• GAS: The ALSP's public key certificate.
• U*; V*: URL pointers to the ALSP's public key certificate. U*; V* are the ones which AP
and clinician have, respectively.
• KAP: A secret key shared the AP and her family members.
• PPA; PFA: AP's and Family's secret passwords (respectively) shared with the ALSP.
• ACLAP: An access control list for the AP's records at the ALSP.
• ΔAP: A mapping table from a serial number of a medical device to DataID
Bootstrapping
1)
2)
3)
4)
5)
6)
7)
Clinician sets up relationship with ALSP / Exchange URL pointers
Assisted person (AP) subscribes to an ISP / Set up a gateway for WPA
network
AP visits clinician. Clinician recommends a list of devices, hubs, and ALSPs.
AP buys devices and hub and adds them to home network
AP subscribes to ALSP. ALSP sends a URL pointer and a password token. AP
loads them into the hub.
AP creates a shared “family key” for family members’ and friends’ access to
data on the ALSP. Family members register passwords to ALSP.
Hub generates ΔAP , a set of DataID. A dataID is mapped to a device serial
number and recipients’ URL pointers. Each entry is a tuple (serial number,
dataID, URL pointer)
With the recipient information (clinicians and family members), ALSP creates
access control list ACLAP.




 



Gdoc, GAS, KAP, U*, V*, PPA, PFA, ACLAP, ΔAP
Clinician
AP
ALSP
Family
Overview of the Talk



Why Do We Care About Senior Care?
Overview of Personal Assistant System (PAS)
Underlying Research that Makes Things Work







Do Elderly People Accept This Technology?


Systems Architecture
Wireless communications and networking
Tracking and localization
Security and privacy
Software safety, reliability, and availability
Human computer interface
Pilot studies
Where Do We Go From Here?
Pilot Study at Nazareth Living Center*



With the help of geriatricians at Washington University in Saint Louis,
we have carried out a 3-week pilot study at the Nazareth Living Center
for Assisted Living in June-July 2006.
This facility houses 110 well-educated, predominantly female residents,
whose average age was 88. Of 30 residents who attended a
presentation, 14 agreed to participate.
After administering a standard cognitive assessment, two residents
were consented to test the PAS prototypes (application 1 and
applicatoin2) and ten residents to carry/wear a placeholder device for
a period of two weeks.
* Nazareth Living Center is a
Catholic, nonprofit skilled
nursing and assisted living
community, sponsored by
the Sisters of St. Joseph of
Carondelet.
Pilot Study at Nazareth Living Center

Residents found PAS useful and were willing to wear the sensing
apparatus:



The two residents using the PAS prototype found it to be quite useful.
Residents not chosen to use the working prototype expressed their desire to
use the working version as well.
Residents lack in confidence in PAS when it
did not work properly:



Wireless connectivity is made possible by a
low-end Linksys WRT54G wireless router. This,
coupled with concrete walls between residents'
rooms and the nurse station, led to intermittent
connectivity.
Residents, as a result, were not confident in
relying solely on PAS for medical monitoring.
To address this problem for the time being, we
will add a wireless repeater (i.e., an additional
Linksys WRT54G router configured in the
client-mode and forwarding all packets to
another, master router) between residents'
rooms and the nurse station.
Pilot Study at Nazareth Living Center

Nurses/caretakers desire interfaces that provide
security/privacy.


With a high resident-to-nurse ratio, nurses were usually very busy and
could not ensure that the information being displayed would not be
viewed by unauthorized personnel. The need for privacy should be
addressed by designing adequate access control to PAS.
Nurses inquired whether or not medical data could be securely
transmitted via wireless technology. This implies they also had
concerns about PAS security.
Overview of the Talk



Why Do We Care About Senior Care?
Overview of Personal Assistant System (PAS)
Underlying Research that Makes Things Work







Do Elderly People Accept This Technology?


Systems Architecture
Wireless communications and networking
Tracking and localization
Security and privacy
Software safety, reliability, and availability
Human computer interface
Pilot studies
Where Do We Go From Here?
Where Do We Go From Here?


Technology enhancement with respect to software robustness/reliability,
security/privacy, and HCI.
Extensive, hypothesis-driven clinical trials with Washington University in
Saint Louis and Buehler Center on Aging, Health & Society, Northwest
University


The average cost of nursing home stay is about $8,300/month*.
Can PAS delay transfers of elderly people to skilled nursing facilities (SNFs)
and improve the quality of their lives (by preserving independence)?
Delaying institutionalization by simply one month would cover the cost of
deploying PAS.
*Phibbs CS, Holtz J-EC, Goldstein MK et al. The effect of geriatrics evaluation and management on nursing home use and health care
costs: Results from a randomized trial. Med Care 2006;44:91-5.
Concluding Remark

With most of the PIs being baby boomers, it is our sincere
hope that we will be the first generation to live
independently and with dignity with the help of PAS when
we age, and the project will benefit the healthcare system
and the society as a whole.