Transcript Agent Layer

智慧型家庭網路之技術與應用
Professor Yau-Hwang Kuo
Director
Center for Research of E-life Digital Technology
(CREDIT)
National Cheng Kung University
Tainan, Taiwan
1
Outline









Introduction
Structure of Smart Home Network
Realization of Device & Network Layers
Agent-based Platform
Affective HCI
Integrated Perception
Cognition Layer
Smart Home Services
Conclusion
Trend of Digital Home






House_n (MIT)、Aware Home (Geogria
Tech.)、Interactive Workspace (Stanford
Univ.)、MavHome (UTA)。
Digital Home Working Group: HP, Intel, IBM,...
ECHONET: Energy Conversation and
Homecare Network.
CELF: Consumer Electronic Linux Forum.
OSGi: Open Service Gateway Initiative
Easy Living: Microsoft
Scenarios of Digital Life
smart digital housekeeper.
2. ubiquitous digital nursing agent.
3. affective digital tutor.
4. ubiquitous home security monitor.
5. ubiquitous home content service.
6. universal cyber circles.
7. ubiquitous universal messaging service.
8. personal knowledge warehouse/navigation.
9. nomadic personal digital secretary.
10. secure traffic navigator.
1.
Microsoft’s View for
Digital Home Solution

Total connectivity


Personalized experiences


No more islands of functionality
Customized entertainment,
communications, and control
Ubiquitous access

Your PCs, devices, and content,
securely accessible everywhere
Microsoft’s View for
Digital Home Solution
Technology “by invitation only”, not
imposed
 Highly personal and personalized space
 Virtually random, unmanaged “build out”
 Complex mix of products and services

Issues of Digital Home

人機互動能否人性化？
robustness、adaptability、multi-modal
collaboration 人性化互動特質。
 感官、認知、情緒、協調、合作實現人性
化互動的技術要素。
 ubiquitous multi-modal affective
human-machine interaction 數位家庭的
人性化互動需求。

Issues of Digital Home (cont.)

人際互動能否得到提昇擴大？


去空間限制、去時間限制、去工具限制、去
安全限制。
家電間的協力合作能力能否得到提昇?

connectivity among appliances、
autonomous collaboration of appliances、
interoperability of appliances。
Issues of Digital Home (cont.)

人在數位生活空間的自由度是否得到提
昇？
可移動性、可轉移性、可調整性。
 ubiquitous integration home network、
location-awareness、universal access、
multi-modal human-machine interaction

Issues of Digital Home (cont.)

人在數位生活空間的便利度是否得到提
昇？

生活機能完整性、設備與網路無縫結合度、
生活機能可獲性(availability)、用戶干預
度、操作易度、穩私與安全等。
人在數位生活空間所獲得的生活輔助機
能能否得到提昇?
 Smart home network is necessary!

Goals:
Infrastructure & applications

Create a new life space supported by a smart
home service network and attached digital
appliances.
 Develop e-services over the smart home
network and digital appliances to realize a
new life style.
 Develop a service modeling and execution
environment over the smart home network to
realize various e-services.
Goals:
technologies

Develop nomadic HCI technology

Speech, vision, physiology, sensors.
Develop affective HCI technology
 Develop agent-based home service
network middleware.
 Develop embedded platform & SoC for
smart appliances.

Layered Structure of Smart Home Service Network
Applications (health care, entertainment, surveillance, etc.)
Application Layer
Service Model Execution Platform (script translation, scheduling, QoS)
Emotion / Semantics / Behavior / Intention Understanding
Cognition / Affection
Layer
Perception Layer
Corpus of Knowledge
(Ontology)
Natural Language
Processing
(text, spoken)
Inference
Engine
Integrated Perception
Speech
Vision (face) Vision (gesture) Physiology
Smell
Agent Layer
Mobile Agent Platform
Network Layer
Home Network (802.11, Bluetooth, HomePlug) + Mobile Internet (SIP +3G)
Device Layer
Home Comm. Gateway;
Home Perception Server;
Home Media Center
Networked Physiology
& Environment
Monitoring Appliances
Networked
Microphones;
Cameras;
Speakers
Wireless
A/V Streaming
Appliances
阿桂的工作環境：Layered Structure of Smart Home Network
Application Scripts for Various Living Support Functions
Web Activity Awareness Service
Service Appliance Online Activity Telephony/ Content Content
Service
Prediction Handling Aggregation Discovery Control Trading Scheduling Messaging Retrieval Delivery
Layer
S
Agent-based Task Scheduling/Dispatch/Migration
E
R
V
Semantics/Behavior/Intention/Emotion/Context Understanding
Cognition/ I
Affection C
Corpus of Knowledge
Human-Machine
Natural Language
(Ontology)
Interaction Engine
Processing
Layer
E
P
Perception L
Layer
A
T
F
Adaptation O
R
Layer
M
Content & Context Information Collection
Location Sensing Physiology Sensing Environment Sensing
Device and Media Management
Device Bridge Protocol Bridge Media Adaptation UI Adaptation
Speech
Vision
OSGi Gateway
Text
Remote Access
Management
HTTP/RTP/RTSP (streaming) + SCP/RTCP/UPnP/SOAP (control) + RDP (UI remoting) + SIP (messaging)
Network
Layer
Device
Layer
IPv4/IPv6
Home Network (802.11 + Home Plug)
電子丫環 資訊伺服器 屋控伺服器
阿桂
阿文
阿金
硬體設施 硬體設施 硬體設施
家電
設備
Access Networks (FTTH + 3G)
通訊伺服器
感應器
阿銘
硬體設施
CO
Routers
Backbone
Internet/
WWW
Servers
(阿美)
Device & Network Layers:
types of digital appliances

Client-type devices





Gateway-type devices


802.11g-based multifunctional audio/voice adaptor
802.11g/MPEG-4-based multifunctional video adaptor
802.11g/MPEG-4-based smart IP camera
Bluetooth-based ECG device
Multimedia communication gateway
Server-type devices




House control server
Human-machine interaction server
Content server
Application server
Device & Network Layers:
relationship among server appliances
house control &
housekeeping
devices
主
人
用
戶
端
設
備
屋
控
伺
服
器
應
用
伺
服
器
FTTH/
3G/
WiMAX
WiFi/
Home
Plug
WiFi/Home Plug
WiFi/
Home
Plug
Internet/
WWW
CO
通
訊
伺
服
器
WiFi/
Home
Plug
內
容
伺
服
器
Telephony
A/V devices
data
store
Architecture of agent platform
ASI_1_1
ASI_1_2
BIS_1_1
ASI
SDH
Scenario
Server
DB
Register
Script
What to do ?
BIS
BIS_2_1
User
Request
PMS_1_1
PMS_1_2
PMS
LKN_1_1
LKN_2_1
LKN
FEA_1_1
FEA_2_1
FEA
Service
Server
Scheduling
Algorithm
XML
How to do ?
Service
Agent
Location
Server
XML
Where to do ?
Common
API
Task
agent
Task
agent
Task
agent
Agent-based Runtime Environment

Execution environment: IBM Aglets system
 Common API
getSubList (void)
getSDHStatus(void)
getSubList (Int subsystemId )
getFunctionList (void)
getScenarios (void)
getUserLoc (Int userId)
Event_Trigger
Start_Service_Agent (Int subsystemId,
Int deviceLocation, String text)
Start_Service_Agent (Int subsystemId,
Int deviceLocation, File file)
getDataFromSub(Int subsystemId,
Int destSubsystemId
String[][] function_name, parameters)
Adaptive Service Provider:
architecture
ASI_1_1
ASI
BIS
ASI_1_2
BIS_1_1
Register
BIS_2_1
PMS
PMS_1_1
PMS_1_2
LKN
LKN_1_1
LKN_2_1
FEA
FEA_1_1
FEA_2_1
Service
Service
Server
Server
Scheduling
Scheduling
Algorithm
Algorithm
XML
XML
Task
List
Task
Task
agent
agent
Service
Service
Agent
Agent
Service
Service
Agent
Agent
Task
Task
agent
agent
Task
Task
agent
agent
User request
(data, args)
Adaptive Service Provider:
functionalities

Functionalities
Registry mechanism for subsystem, device and
functionalities
 Service provider for user requests
 Load balanced service scheduling algorithm
according to system resources
 Agent cooperation mechanism

Adaptive Service Provider:
components

Service server
Subsystem and devices functionalities registration
 Service portal for users
 Monitoring each subsystem and device


Service agents
Provide service for each user request
 Service composition
 Task assignment and task agent dispatch according
to predefined XML-based scenarios

Adaptive Service Provider:
components (cont.)

Task agent
Execute each functionality on each subsystem
 Common API


Service scheduling algorithm
Provide a task list for service agent according to
registry and pre-defined scenarios in database
 A Petri net based & load balanced scheduling
algorithm for adaptive service path in each
subsystem and device

Agent-based Middleware:
mobility management

Location detection
Device-followed type: mobile IP; signal
analysis
 Device-free type: speech interaction; vision
monitoring.

Seamless handoff and transcoding for
ubiquitous service following
 Roaming path tracking and prediction

Agent-based Middleware:
appliance collaboration management
Collaboration among homogeneous
appliances: data fusion, task migration.
 Collaboration among heterogeneous
appliances: multi-modal HCI.
 Scheduling, concurrency control &
synchronization of collaborative tasks.
 Self-organization for service deployment

Agent-based Middleware:
interoperability management
Device bridge
 Protocol bridge
 Transcryption
 Transcoding
 Content translation & adaptation

Agent-based Middleware:
remote access management
Remote service deployment
 remote service access
 remote service management
 auto-configuration
 service re-direction
 service aggregation
 UI remoting

Agent-based Middleware:
other management functions

load management:
Client-server load partition
 Server load sharing
 Load scheduling of appliance farm


availability management
Fault tolerance
 Just-in-time activation of appliances


service quality management
Affective Speech Conversation
Synthesis
ASR
Speech
Text
Emotion
Dialog
System
Text
Speech
Emotion
Emotional Speech Synthesis
Text
Input
Text
Analysis
Emotion
Selection
Database
Selection
Syntactic
Analysis
Unit
Selection
Sad
Happy
Neutral
Angry
User’s Action
Emotional
Speech Database
Speech
Smoothing
Speech
Segmentation
Behavior Understanding by Vision

High-Level behavior understanding from
videos


Human Activity Recognition


State Machine
Two-Stage recognition process
Accident/Abnormal behavior detection

Context & domain knowledge Combination
System Architecture
Image
Image
Image
Video Stream
Segmentation &
Tracking
Feature Extraction
Activity Recognition
Background
(Update)
Posture
Recognition
Postures
Analysis
Foreground
detection
HistoryMap
Size
Analysis
Tracking
Motion
Estimation
Motions
Analysis
Daily life
information
State Transition
Violent Motions
Temporal
information
Context
Combination
Lying & Static
Abnormal Detection
Normal Detection
Accident Detection
Method – Activity Recognition

Activity Recognition

Level 1 - postures


Posture Sequence
Level 2 – motion/history

History Map Matching
Method – Behavior understanding

Behavior

Normal behavior

State Machine


Abnormal behavior


Activity + Contexts
Normal behavior + domain knowledge
Accident

Unreasonable activity + domain knowledge
Facial Expression Analysis
Face Acquisition
Acquisition
Segmentation
Eye
Region
Facial Feature Extraction
Deformation
Extraction
Motion
Extraction
Facial Expression
Classification
Representation
Key frame
Selection
Recognition
Eye
Points
Displacement
Vectors
YCbCr
Image
Sequence
Color
Mouth
Region
Mouth
Points
Fuzzy
Neural
space
Network
Invariant
Moments
Region
Of
Interest
Optical
Flow
Key
Frame
Results
Integrated Perception:
fuzzification of reference perceptual models

Manipulate all kinds of perception in a
uniform process to ease the perceptual
integration.
 Due to high vagueness of perception, fuzzy
logic based approach is a good choice to
establish the reference models of perception.
 The reference models which fuzzify
perceptual attributes and perceptual decision
subspaces will be embedded into the
integrated perception model.
FL-based Acoustic Reference Model
for Emotion Recognition
speech corpus
feature
extraction
SVM
clustering
for emotion 1
SVM
clustering
for emotion 2
fuzzification of
acoustic
features (AFs)
and
construction of
acoustic action
units (AAUs)
AAU2
model
…
…
SVM
clustering
for emotion V
AAU1
model
AAUS
model
FL-based Acoustic Reference Model
for Emotion Recognition (cont.)



Adopt SVM clustering approach in the subspace of
each emotion type to gather the clusters of acoustic
training patterns.
Inspect all produced SVM clusters in the whole
feature space and merge the highly overlapped
clusters.
Each cluster is modeled as an AAU represented with
its fuzzy cluster center where each feature is a fuzzy
set whose membership function is determined by the
least-square curve fitting approach on the feature
values of training samples included in the cluster.
FL-based Acoustic Reference Model
for Emotion Recognition (cont.)

The mapping between AAUs and emotion
types is dependent on the SVM clustering
result of each emotion type.
 Each emotion type is associated with a set of
clusters of acoustic samples. The weight of
each cluster is determined by the ratio of the
number of samples it contains with respect to
the total amount of samples of the same
emotion.
FL-based Facial Reference Model for
Emotion Recognition
graphical head
model
morphological
process to
simulate AUs
FACS AUs
identification
process
correspondence
feature points (FPs)
extraction process
Membership
grade
fuzzy logic based
reference model for
FACS
Membership
grade
FAU1
FAUi
FAU2
FAUj
FP1 value
FAUk
FP2 value
FL-based Facial Reference Model for
Emotion Recognition (cont.)

Intend to construct a computational reference
model for FACS action units based on the
measurable features of facial expression.
 An approach similar to the construction of
acoustic reference model is adopted.
 The training samples are generated from a
generic head model with necessary
morphological manipulation.
FL-based Facial Reference Model for
Emotion Recognition (cont.)
The membership functions will be
determined by the least-square curve
fitting approach according to the sample
patterns produced from the
morphological process.
 Each AU may just represent a partial
facial expression and relate to more
than one emotion.

Fuzzy Neural Network for Integrated
Emotion Recognition
{< total ordering of emotion types>, group level of agreement}
Fuzzy group decision process
emotion type
layer
representative
concept layer
Fear
FAU1
Anger
FAU2
Surprise
Fear
Anger
FAUK
AAU1
AAU2
Surprise
AAUS
scaled
feature layer
primary
feature layer
FPn
FP1
Face Features Expression
AF1
AFm
Acoustic Features
Fuzzy Neural Network for Integrated
Emotion Recognition (cont.)




All kinds of perceptual information are fused by the
FNN model to realize emotion recognition.
Each appliance will have an instance of the
corresponding FNN to join the emotion recognition job.
A two-layered (emotion type & concept layers) BP
learning algorithm is adopted by using the training
samples in constructing reference models. The fuzzy
group decision process does not join the learning.
Scaling input value to [0,1] in the second layer is
realized by the membership function of the
corresponding fuzzy set.
Fuzzy Neural Network for Integrated
Emotion Recognition (cont.)

The links between AUs and scaled features
are not fully connected.
 The FAU/AAU nodes realize normalized
weighted sum for the membership grades of
input features weighted by their respective
link strength.
 Each emotion type node determines output
value by the normalized weighted sum of its
inputs from the representative concept layer.
Cognition Layer:
understanding and response
Understand the semantics of multimodal expression.
 Classify and recognize the intention/
need/emotion of semantic expression.
 Summarize the semantics of multimodal expression according to
classified result.

Cognition Layer:
understanding and response (cont.)
Predict the user behavior sequence
according to the classified result.
 Schedule the response sequence
according to the prediction result.
 Determine the instant response.

Stimulus
spoken
language
Perception
Speech
Processing
Cognition
Semantic
Expression
Semantic
Feature
Extraction
Features
Conceptualization
Concepts
Event Detector
(Neural Networkbased Approach)
Events
gesture
face
expression
physiological
signals
Emotion
Attributes
Signal
Processing
text
Response
Personal
Event /
Emotion Log
Vision
Processing
Video
Processing
Speech
Processing
Application
Control
Emotion
Recognition
Ontology
Contextual
Rules
Semantic
Summary
Emotion Types
Emotion
Event
Sequence Sequence
Case base Case base
Emotion
Episode
Discovery
Response
Stimulus
Semantic
Summary
Extraction
Stimulus
Response
Templates
Emotion Episodes
Instant
Response
Determination
Response
Roadmap
User Behavior
Prediction
(Episode-based
Approach)
Prediction
Result
Response
Scheduling
Smart Home Services

nomadic content services
 health care by integrated perception
 smart home surveillance
 smart e-mail and calendar arrangement
Conclusion
Life style of human being will be heavily
affected by ICT, but the technological
gap is still big.
 Ubiquitous HCI and OCI technologies
will be important to realize digital life
style.
 Cognitive computing and affective
computing are important to improve the
effectiveness of HCI technology.

Description of Context-Aware Middleware
User
Profile
Admission
Control
Personal
Agent
Context
Reasoning
Context
Aggregator
Resource
Management
Wrapper
Wrapper
Device
Service
Service
Agent
Context-Aware Middleware Architecture
Location
Detection
Speech
Recognition
Posture
Recognition
Identification
Interface
Context
Resource
JADE
UPnP Wrapper
Agent Platform Bundle Bundle Management Reasoning
Bundle
Bundle
(Service Scenario)
OSGi Platform
JAVA Virtual Machine
Operation System
Bundle Repository
Context Aggregator
and
Ontology Reasoning