Transcript Robot Theatre Research at PSU

Robot Theatre
Research at
PSU
Hough Transform
for Radiation Therapy
Intelligent
Robotics
PCA
for facial emotions
Emotional Robot
Computer
Vision
Computational
Intelligence
Motion
Generation
Robot theatre
Zhao, Hun
Labunsky
Lukac
Constructive Induction
Architecture with
Perception of emotions
Gebauer
Labunsky
Emotion Based Motion
Generation
Sunardi
Common Robot
Language for
Humanoids
Lukac
Evolutionary generation Raghuvanshi
of robot motions
Robot
Design : three
theatres
The Narrator
Sonbi the Confucian
Scholar
KHR-1
Pune the Courtisane
KHR-1
iSOBOT
iSOBOT
The orchestra
The hexapod
dancers
Improvisational
Musical of small
robots
Paekchong the Butcher
Interactive Hahoe
Theatre
Of Large Robots
Physics Debate
Theatre of medium
robots
Professor Albert Einstein
Dr. Niels Bohr
Schroedinger Cat
What is
robot
theatre?
Theory of Robot Theatre?
1. Motion Theory:
–
Motions with symbolic values
2. Theory of sign
–
Creation of scripts, generalized events, motions
to carry meaning
3. Robot theories that may be used:
1.
2.
3.
4.
5.
6.
Machine Learning
Robot Vision
Sensor Integration
Motion: kinematics, inverse kinematics, dynamics
Group dynamics
Developmental robots
Realizations of Robot Theatres
• Animatronic “Canned” Robot theatre of
humanoid robots
– Disneyworld, Disneyland, Pizza Theatre
• Theatre of mobile robots with some
improvisation
– Ullanta 2000
• Theatre of mobile robots and humans
– Hedda Gabler , Broadway, 2008
– Phantom in Opera, 2008
– Switzerland 2009
Models of
Robot Theatre
Animatronic
Theatre
Actors: robots
Directors: none
Public: no feedback
Action: fixed
Example: Disney World
controller
Robot
Canned code
Editor
motion
controller
Motion
language
Robot
Inverse
Kinematics
Forward
Kinematics
Motion
Capture
Editor
Evolutionary
Algorithms
motion
controller
Motion
language
Robot
1. A very sophisticated system can be used to create motion but all events are
designed off-line.
2. Some small feedback, internal and external, can be used, for instance to avoid
robots bumping to one another, but the robots generally follow the canned
script.
Universal Event Editor
Motion
Capture
Inverse
Kinematics
Universal
Event
Editor
Initial
events
script
Forward
Kinematics
Events
language
events
controller
Robots
Lighting
System
Sound
System
Curtain and all
equipment
Universal Editors
for Robot Theatre
Universal Perception Editor
cameras
Speech
input
Sensors
Principal
Component
Analysis
Various Feature
Extracting
Methods
Neural
Nets
Constructive
Induction
Clustering
Perception
Editor
Examples – input output pairs
Feedback from the environment
controller
controller
Robot
Robot
Critic
The environment
includes:
1. Other robots
2. Human actors
3. Audience
4. The director
Interaction
Theatre
Actors: robots
Directors: none
Public: feedback
Action: not fixed
Example: Hahoe
Perception
Machines
Input text from
keyboard
Face Detection and
Tracking
Behavior
Learning
Architecture
for Interaction
Theatre
Motion
Machines
Output text i
Output speech i
Face Recognition
Facial Emotion
Recognition
Hand gesture
recognition
Behavior
Machine
Output robot motion i
Output lights i
Sonar, infrared,
touch and other
sensors
Output special effects i
Speech recognition
Output sounds i
Improvisational
Theatre
Actors: robots
Directors: humans
Public: no feedback
Action: not fixed
Example: Schrödinger Cat
Improvisational Theatre “What’s That? Schrödinger Cat”
Schrödinger Cat
Professor Einstein
Motion
Motion
e1
c1
Motion
e2
Motion
c1
Motion
Motion
cm
en
Siddhar
Motions of Einstein
Motions of Schrödinger
Cat
Arushi
Theatre of Robots
and Actors
(contemporary)
Actors: robots
Actors: humans
Directors: humans
Public: traditional feedback, works only for human
actors
Action: basically fixed, as in standard theatre
Theatre of Robots
and Actors (future)
Actors: robots
Actors: humans
Directors: humans + universal editors
Public: traditional feedback, like clapping, hecking,
works for both robot and human actors
Action: improvisational, as in standard improvisational
theatre
Research
Topics in Robot
Theatre
Perception
Face Recognition as a learning
problem
Face Image 1
John Smith
Face Image 2
Face Image 3
Face Image 4
Marek Perkowski
Face Emotion
Recognition as a learning
problem
Face Emotion
(Gesture)
Recognition
as
a learning problem
Face Image 1
happy
Face Image 2
Face Image 3
sad
Face Image 4
Face
person
Face
person
Face
emotion
Face
age
Face
gender
Face
gesture
Recognition
Problems =
Who? What?
How?
Learning problems in Human-Robot Interaction – Perception problems
Motion
Mouth
Motion
text
Hexapod
walking
Distance
evaluation
Biped walking
Number of falls
evaluation
Biped
Gestures
Comparison to
video evaluation
Hand gestures
Subjective human
evaluation
Motion
Problems =
examples of
correct motions
– generalize
and modify,
interpolate
Learning problems in Human-Robot Interaction – Motion Generation problems
The concept of generalized motions
and universal event editor to
edit:
– robot motions,
– behaviors,
– lightings and automated events
Languages to describe all kinds of
motions and events
• Labanotation
• DAP (Disney Animation Principles) and
• CRL (Common Robot Language)
Theory of Event Expressions
• Tool to design motions directly from symbols.
• This theory is general enough to allow arbitrary motion
to be symbolically described but is also detailed
enough to allow the designer or the robot to precise
the generated behavior to the most fundamental
details.
• Our main concept is that the motion is a sequence of
symbols, each symbol corresponding to an elementary
action such as shaking head for answering “yes”.
• We will call them primitive motions.
• The complex motions are created by combining
primitive motions.
• What are the factors of motion that are important to
generation of symbolic motions, especially in dance
and pantomime?
• The research issues are:
(1) What should be the primitive, basic or atomic motions?
(2) What should be the operations on motions that combine
primitive motions to complex motions?
(3) How symbolic motions are reflected/realized as physical
motions of real robots?
(4) How to create existing and new symbolic motions
automatically using some kind of knowledge-based
algebraic software editor.
• Human-robot emotion-augmented communication systems.
• The new extended communication media in addition to:
–
–
–
–
–
speech
will include prosody
facial gestures
hand gestures
body language
•
•
•
•
–
–
–
–
head and neck,
legs,
bending of full body,
muscle-only motions
The gestures used in daily communication,
ritual gestures of all kinds,
theatric and
dance motions
• will be captured or translated to certain algebraic notations.
• the same grammar.
• The unified notations will allow also to transform between various media.
• For instance, a captured motion of a flower can be presented as motion of
a pantomime actor.
• A sound pattern can be transformed to a pattern of colored laser lights.
• Sound, light, theatric stage movements, special effects and robot motions
have all the same nature of sequences of symbolic event-related atoms.
• Thus the theater itself becomes a super-robot with the ability to express
its emotional state through various motions, lights and theater plays
orchestration.
• These sequences can be uniformly processed.
• We propose to create universal editors that will be not specialized to any
particular medium or robot.
• They will use algebraic and logic based notations.
on-line versus off-line creation of motions
• Film is an art that does not require feedback from the audience and every
presented event is always exactly the same.
– Creation has been done earlier, off-line with the director and his associates as the
audience during shooting and editing.
•
In contrast, the theatre performance is always slightly different and it
depends on the feedback from the audience.
• The public may say “Actor X had a good day today”.
• The research questions are:
– (1) “will the future robot theatre be more similar to films or theatrical
performances?”,
– (2) “can one create a realistic robot theatre without viewer’s feedback?”
– (3) “What type of tools do we need to create a theater play off-line versus online?”,
– (4) “Is an off-line sequence generator enough to create art?”
– (5) “Do we need a sequence generator and a virtual reality simulator before
actuate the play on the robot”?
– (6) “How really important is the feedback from the audience for a human singer or
conference speaker to modify their on-stage behaviors? a
• How it should be reflected in the on-line robot theatre?
• Should we have a “character” of each robot simulated
together with the performance?
– (So that a shy robot should run from the stage if it is
booed.
– A strong robot should remain.
– But maybe the shy robot will know much better how to
express the meaning of the play.)
• In general, do we need a “character” to use the
feedback from audience or should be this simulated
otherwise?”
Event Expressions to specify
languages of motions and behaviors
• A regular expression is a pattern that matches one or more strings of
characters from a finite alphabet.
•
Individual characters are considered regular expressions that match
themselves.
• Original regular expressions used union, concatenation and iteration
operators.
• Regular expressions were next extended by adding negation and
intersection to a Boolean Algebra.
• Observe that X1 ∩ X2 is an empty set for atoms X1  X2 , as the meaning of
intersection operator is set-theoretical.
• Similarly the interpretation of  operator is set-theoretical in regular
expressions, thus new symbols are not being created.
•
•
Greeting_1 = (Wave_Hand_Up o
Wave_Hand_Down ) (Wave_Hand_Up
o Wave_Hand_Down ) * 
Wave_Hand_Up o Say_Hello
Wave_Hand_Up
Which means, to greet a person the
robot should execute one of two
actions:
–
–
•
Initial state
Action 1: wave hand up, follow it by waving
hand down. Execute it at least once.
Action 2: Wave hand up, next say “Hello”. The
same is true for any complex events.
Wave_Hand_Up
Wave_Hand_Down
As we see, the semantics of regular
expressions is used here, with atomic
symbols from the terminal alphabet of
basic events {Wave_Hand_Down,
Wave_Hand_Down
Wave_Hand_Up , Say_Hello}.
The operators used here are:
concatenation (o), union () and
iteration (*). Each operator has one or
Wave_Hand_Up
two arguments.
So far, these expressions are the same
as regular expressions.

•
•
Say_Hello
Final state
Extending to Event Expressions
• They are now expanded to event expressions by recursively
adding more deterministic and probabilistic operators on
event expressions
• For instance, if we agree that the meaning of every
operator in Greeting_1 is that it executes its first
argument with probability ½ and its second argument with
the same probability, each possible sequence from the
infinite set of motions of Greeting_1 will have certain
probability of occurrence.
• One can create event expressions using both deterministic
and probabilistic, single-argument and two-argument
operators.
Event Expressions to specify
languages of motions and behaviors
• The base ideas of event expressions are these:
– (1) Symbol (represented by sequence of characters) is
a basic event or a set of basic events synchronized and
executed in parallel.
– (2) Symbols can be connected in parallel (for instance,
(a) text spoken, (b) leg motion, and (c) hand motion
can be created relatively independently and combined
in parallel). Connecting symbols in parallel creates
new symbols that can be used as macros
• For instance, assuming that concatenation is a deterministic
operator and union is a probabilistic operator with equal
probabilities of both arguments, there is a probability of ½ that the
robot will greet with motion Wave_Hand_Up o Say_Hello .
• Assuming that all operators are probabilistic operators with equal
probabilities of both arguments, for event expression
Wave_Hand_Up o Say_Hello there is a probability of ½ that the
robot will greet with one of the following four motions, each with
the same probability:
–
–
–
–
(1) Wave_Hand_Up o Say_Hello ,
(2) Wave_Hand_Up ,
(3) Say_Hello ,
(4) Nothing will happen.
Extending to Event Expressions
• As we see, in this case for each of two arguments of concatenation
there is the probability of ½ that it happens and probability of ½ that
it does not happen.
• Similarly, the user of the editor can use many operators,
deterministic or probabilistic to define event expressions.
• Several such operators are created for any standard operator of
regular expression.
• Next, the user can define his/her own new operators.
• These operators can have temporal, multiple-valued and
probabilistic/deterministic nature.
• Our system of Event Expressions and corresponding state machines
uses an expanded set of operators taken from multiple-valued logic;
literals, MAX, MIN, truncated-sum, Modulo-addition and others.
• The symbols are interpreted as having numerical values for them.
• This allows also for interpolation (Hermite, Spline, Radial Basis) and
spectral operators based on Fast Fourier Transform (FFT).
Brzozowski’s derivatives
• All words of E, starting from letter Xj є X. If the letter is
removed from the front of each word from the set, a new
language is created, referred to as left-side-derivative of
language E by letter Xj.
• This new language is now denoted by E/Xj.
• A derivative for word s = Xi1, Xi2…Xin is defined as follows:
E/Xj = {s є X* : Xjs є E}.
• As inherent laws of Brzozowski’s derivative method, the
following properties Pi always hold true.
Recursive Rules for PMG design
•
•
•
•
•
•
•
•
•
•
•
•
•
P1. Xi/Xj = e for i = j
= ø for i ≠ j
P2. (E1 ∪ E2)/Xi = E1/Xi ∪ ∈ E2/Xi
P3. ∈(E) = e when e ∈ E
= ø when e ∉ E
P4. E1E2/Xi=(E1/Xi)E2∪∈ (E1)(E2/Xi)
P5. E/s=E1/Xi1Xi2…Xin=[[E/Xi1]/Xi2]…]/Xin
P6. E*/Xi = (E/Xi)E*
P7. E/e = E
P8. (E1 ∩ E2)/Xi = (E1/Xi) ∩ (E2/Xi)
P9. (-E)/Xi = -(E/Xi)
P10. (E1 max E2)/Xi = (E1/Xi) max (E2/Xi)
In our system there are many other rules similar to rule P10 for MIN, MAX and
other MV operators.
• There are many rules similar to P9 for literals and rules similar to P8, P2, P6 and
P4 for probabilistic variants of operators ∩,∪, and concatenation, respectively
Example of designing PMG from
Event Expression
• Language is given E1 = (X2X1* ∪ X1X2).
• Applying the left-side-derivative with respect to
first character in string, X1
E1/X1 = (X2X1* ∪ X1X2)/X1
= (X2X1*)/X1 ∪ (X1X2)/X1
by P2
= (X2/X1)X1* ∪ ∈ (X2)X1* ∪ (X1/X1)/X2 ∪ ∈ (X1)(X2/X1)
by P4
= ø X1 ∪ ø(X1/X2) ∪ eX2 ∪ ø ø
by P1
= X2
Fig. 2. Graph for regular language
E1 = (X2X1* ∪ X1X2).
It can be interpreted as PMG, EEAM or BM
depending on meaning of symbols Xi
Acceptor, generator and
transformer
• Observe that this graph can be interpreted as an
acceptor, when symbols Xi are inputs.
• It can be interpreted as a generator when symbols
Xi are outputs.
• The graph can be thus used to recognize if some
motion belongs to some language and can
generate a motion belonging to the language.
• This graph is realized in software
Input text
Output text
Hexapod
walking
Distance
evaluation
Biped walking
Number of falls
evaluation
Biped
Gestures
Comparison to
video evaluation
Hand gestures
Subjective human
evaluation
Behavior
Problems =
examples of
correct
motions –
generalize and
modify,
interpolate
Learning problems in Human-Robot Interaction – Motion Behavior
(input/output) generation problems