Transcript Introduction to Embodied Intelligence

Cognitive Architecture
How to Motivate a Machine ?
Janusz Starzyk
School of Electrical Engineering and Computer
Science, Ohio University, USA
www.ent.ohiou.edu/~starzyk
EE141
Outline









Traditional Artificial Intelligence
Embodied Intelligence (EI)
Embodiment of Mind
EI Interaction with Environment
How to Motivate a Machine
Goal Creation Hierarchy
GCS Experiment
Motivated Learning
Challenges of EI
 We need to know how to organize it
 We need means to implement it
 We need resources to build and
sustain its operation

Promises of EI
 To economy
EE141 To society
Intelligence
AI’s holy grail
From Pattie Maes MIT Media Lab

“…Perhaps the last frontier of science – its
ultimate challenge- is to understand the biological
basis of consciousness and the mental process by
which we perceive, act, learn and remember..”
from Principles of Neural Science by E. R. Kandel et al.
 E. R. Kandel won Nobel Price in 2000 for his work on physiological
basis of memory storage in neurons.
 “…The question of intelligence is the last great
terrestrial frontier of science...” from Jeff Hawkins On
Intelligence.
 Jeff Hawkins founded the Redwood Neuroscience Institute devoted
to brain research
EE141
Is what is intelligence?
EE141
Various Definitions of Intelligence









The American Heritage Dictionary:
 The capacity to acquire and apply knowledge.
 The faculty of thought and reason.
Webster Dictionary:
 The act or state of knowing; the exercise of the understanding.
 The capacity to know or understand; readiness of comprehension;
Wikipedia – The Free Encyclopedia:
 The capacity to reason, plan, solve problems, think abstractly, comprehend ideas
and language, and learn.
Kaplan & Sadock:
 The ability to learn new things, recall information, think rationally, apply knowledge
and solve problems.
On line dictionary dict.die.net
 The ability to comprehend; to understand and profit from experience
The classical behavioral/biologists:
 The ability to adapt to new conditions and to successfully cope with life situations.
Dr. C. George Boeree, professor in the Psychology Department at Shippensburg University:
 A person's capacity to (1) acquire knowledge (i.e. learn and understand), (2) apply
knowledge (solve problems), and (3) engage in abstract reasoning.
Stanford University Professor of Computer Science Dr. John McCarthy, a pioneer in AI:
 The computational part of the ability to achieve goals in the world.
Scientists in Psychology:
 Ability to remember and use what one has learned, in order to solve problems, adapt
to new situations, and understand and manipulate one’s reality.
EE141
Intelligence
Mainstream Science on Intelligence
December 13, 1994:
An Editorial With 52 Signatories, by
Linda S. Gottfredson, University of
Delaware
Intelligence is a very general
mental capability that, among
other things, involves the ability
to reason, plan, solve problems,
think abstractly, comprehend
complex ideas, learn quickly and
learn from experience.
From http://www.indiana.edu/~intell/map.shtml
EE141
Animals’ Intelligence
 Defining intelligence
through humans is not
appropriate to design
intelligent machines:
– Animals are intelligent too
 Dog IQ test:
 Dogs can learn 165 words (similar to 2 year olds)
 Average dog has the mental abilities of a 2-year-old child (or better)
 They would beat a 3- or 4-year-old in basic arithmetic,
 Dogs show some basic emotions, such as happiness, anger and disgust
 “The social life of dogs is very complex - more like human teenagers interested in who is moving up in the pack, who is sleeping with who etc,“
says professor Stanleay Coren from University of British Columbia
 Border collies, poodles, and german shepards are the smartest dogs
EE141
Traditional AI

Abstract intelligence
Embodied Intelligence

 attempt to simulate
“highest” human faculties:
– language, discursive
reason, mathematics,
abstract problem solving

Environment model
 Condition for problem
solving in abstract way
 “brain in a vat”
EE141
Embodiment
 knowledge is implicit in the
fact that we have a body
– embodiment supports brain
development

Intelligence develops
through interaction with
environment
 Situated in environment
 Environment is its best model
Design principles of intelligent systems
from Rolf Pfeifer “Understanding of Intelligence”
Design principles
synthetic methodology
time perspectives
emergence
diversity/compliance
frame-of-reference
complete agent
principle
From: www.spectrum.ieee.org/.../biorobot11f-thumb.jpg
EE141
Design principles of intelligent systems
from Rolf Pfeifer “Understanding of Intelligence”, 1999








EE141
Interaction with
complex environment
ecological balance
redundancy principle
parallel, loosely
coupled processes
asynchronous
sensory-motor
coordination
value principle
cheap design
Agent
Drawing by Ciarán O’Leary- Dublin Institute of Technology
The principle of “cheap design”

intelligent agents: “cheap”
 exploitation of ecological
niche
 economical (but redundant)
 exploitation of specific
physical properties of
interaction with real world
EE141
Principle of “ecological balance”

balance / task distribution
between
 morphology
 neuronal processing (nervous
system)
 materials
 environment

balance in complexity
 given task environment
 match in complexity of sensory,
motor, and neural system
EE141
The redundancy principle



redundancy prerequisite for
adaptive behavior
partial overlap of
functionality in different
subsystems
sensory systems: different
physical processes with
“information overlap”
EE141
Generation of sensory stimulation
through interaction with environment
multiple modalities
 constraints from
morphology and
materials
 generation of
correlations through
physical process
 basis for crossmodal associations

EE141
The principle of sensory-motor
coordination
Holk Cruse



self-structuring of
sensory data through
interaction with
environment
physical process —
not „computational“
prerequisite for
learning
EE141
•no central control
•only local
neuronal
communication
•global
communication
through
environment
neuronal
connections
The principle of parallel, loosely
coupled processes
Intelligent behavior emergent
from agent-environment
interaction
 Large number of parallel,
loosely coupled processes
 Asynchronous
 Coordinated through agent’s
–sensory-motor system
–neural system
–interaction with environment

EE141
So what is an Embodied
Intelligence ?
EE141
Embodied Intelligence
 Definition
 Embodied Intelligence (EI) is a mechanism that learns
how to survive in a hostile environment
– Mechanism: biological, mechanical or virtual agent
with embodied sensors and actuators
– EI acts on environment and perceives its actions
– Environment hostility is persistent and stimulates EI to act
– Hostility: direct aggression, pain, scarce resources, etc
– EI learns so it must have associative self-organizing memory
– Knowledge is acquired by EI
EE141
Embodied Intelligence




EI mimics biological intelligent systems,
extracting general principles of intelligent
behavior and applying them to design intelligent
agents.
Knowledge is not entered into such systems, but
rather is a result of their successful interaction
with the environment.
Embodied intelligent systems adapt to
unpredictable and dynamic situations in the
environment by learning, which gives them a
high degree of autonomy.
Learning in such systems is incremental, with
continuous prediction of the input associations
based on the emerging models - only new
information is registered in the memory.
EE141
What is Embodiment of a Mind?
EE141
Embodiment of a Mind





Embodiment of a mind is a
part of environment under
control of the mind
It contains intelligence core
and sensory motor interfaces
to interact with environment
It is necessary for
development of intelligence
It is not necessarily constant
or in the form of a physical
body
Boundary of embodiment
transforms modifying brain’s
self-determination
EE141
Embodiment
Sensors
channel
Environment
Intelligence
core
Actuators
channel
Embodiment of Mind




Brain learns own body’s dynamic
Self-awareness is a result of
identification with own embodiment
Embodiment can be extended by
using tools and machines
Successful operation is a function
of correct perception of
environment and own embodiment
EE141
Requirements for Embodied Intelligence

State oriented
 Learns spatio-temporal patterns
 Situated in time and space

Learning
 Perpetual learning
 Screening for novelty

Value driven
 Pain detection
 Pain management
 Goal creation
 Competing goals

Emergence
 artificial evolution
 self-organization
EE141
EI Interaction with Environment
Agent Architecture
Reason
Short-term Memory
Perceive
Act
RETRIEVAL
LEARNING
Long-term Memory
INPUT
OUTPUT
Task
Environment
Simulation or
Real-World System
EE141
From
Randolph M. Jones, P : www.soartech.com
Sensory Inputs Coding
Kandel Fig. 30-1
Kandel Fig. 23-5
Richard Axel, 1995
Visual, auditory, olfactory,
tactile, smell -> motor
How do we process and
represent sensory
information?
EE141
Hip
Trunk
Arm
Hand
Foot
Face
Tongue
Larynx
Challenges of Embodied Intelligence
EE141
Challenges of Embodied Intelligence

Development of sensory interfaces
 Active vision
 Speech processing
 Tactile, smell, taste, temperature, pressure
sensing
 Additional sensing
– Infrared, radar, lidar, ultrasound, GPS, etc.
– Can too many senses be less useful?

Development of pain sensors
 Energy, temperature, pressure, acceleration
level
 Teacher input

Development of motor interfaces
 Arms, legs, fingers, eye movement
EE141
Embodiment
Sensors Intelligence
core
Actuators
Environment
Challenges of Embodied Intelligence (cont.)

Finding algorithmic solutions for
 Association, memory, sequence
learning, invariance building,
representation, anticipation, value
learning, goal creation, planning

Development of circuits for neural
computing
 Determine organization of artificial
minicolumn
 Self-organized hierarchy of
minicolumns for sensing and motor
control
 Self-organization of goal creation
pathway
EE141
Human Intelligence – Cortex Uniform Structure




V. Mountcastle argues that all
regions of the brain perform the
same computational algorithm
Groups of neurons (minicolumns)
connected in a pseudorandom way
Same structure
Minicolumns organized in
macrocolumns
VB Mountcastle (2003). Introduction [to a special issue of
Cerebral Cortex on columns]. Cerebral Cortex, 13, 2-4.
EE141
Mini Columns
“The basic unit of cortical operation
is the minicolumn …
It contains of the order of 80-100
neurons except in the primate striate
cortex, where the number is more
than doubled.
The minicolumn measures of the
order of 30-50 m in transverse
diameter, separated from adjacent
minicolumns by vertical, cell-sparse
zones …
The minicolumn is produced by the
iterative division of a small number
of progenitor cells in the
neuroepithelium.”
(Mountcastle)
EE141
Copyright © 2006-2008, all rights reserved, Visualbiotech
Artificial Minicolumn Organization

Sensory neurons are responsible for representing environment





receive inputs from sensors or sensory neurons on lower level
represent the environment
receive feedback input from motor and higher level neurons
help to activate motor and reinforcement neurons
Motor neurons are responsible for actions and skills
 are activated by reinforcement and sensory neurons
 activate actuators or provide an input to lower level motor neurons
 provide planning inputs to sensory neurons

Reinforcement neurons are responsible for building the value
system, goal creation, learning, and exploration




EE141
receive inputs from reinforcement neurons on the lower level
receive inputs from sensory neurons
provide inputs to motor neurons
initiate learning and force exploration
How to Motivate a Machine ?
A fundamental question is what
motivates an agent to do anything,
and in particular, to enhance its
own complexity?
What drives an agent to explore
the environment and learn ways to
effectively interact with it?
EE141
How to Motivate a Machine ?

Pfeifer claims that an agent’s motivation should emerge
from the developmental process.
 He called this the “motivated complexity” principle.
 Chicken and egg problem? An agent must have a motivation to
develop while motivation comes from development?

Steels suggested equipping an agent with self-motivation.
 “Flow” experienced when people perform their expert activity well
would motivate to accomplish even more complex tasks.
 Humans get internal reward for activities that are slightly above their
level of development (Csikszentmihalyi).
 But what is the mechanism of “flow”?

Oudeyer proposed an intrinsic motivation system.
 Motivation comes from a desire to minimize the prediction error.
 Similar to “artificial curiosity” presented by Schmidhuber.
EE141
How to Motivate a Machine ?
Can a machine that only implements
externally given goals be intelligent?
If not how these goals can be created?
•There is a need for a hierarchy of values.
•Not all values can be predetermined by
the designer.
•There is a need for motivation to act,
explore and learn.
•As machine makes new observations
about the environment, there is a need to
relate them to goals and values and create
new goals and values.
EE141
How to Motivate a Machine ?

Although artificial curiosity helps to
explore the environment, it leads to
learning without a specific purpose.
 It may be compared to exploration in
reinforcement learning.
 internal reward motivates the machine to
perform exploration.

Exploration is needed in order to learn and to model the
environment.
 But is this mechanism the only motivation we need to develop
intelligence?
 Can “flow” ideas explain goal oriented learning?
 Can we find a more efficient mechanism for learning?

I suggest a simpler mechanism to motivate a machine.
EE141
How to Motivate a Machine ?

I suggest that it is the hostility of the environment, in the
definition of EI that is the most effective motivational factor.
 It is the pain we receive that moves us.
 It is our intelligence determined to reduce this pain that motivates us
to act, learn, and develop.

Both are needed - hostility of the environment and
intelligence that learns how to reduce the pain.
 Thus pain is good.
 Without pain there would be no intelligence.
 Without pain we would not be motivated to develop.
Fig. englishteachermexico.wordpress.com/
EE141
Motivated Learning

I suggest a goal-driven mechanism to motivate
a machine to act, learn, and develop.




A simple pain based goal creation system is
explained next.
It uses externally defined pain signals that are
associated with primitive pains.
Machine is rewarded for minimizing the primitive
pain signals.
Definition: Motivated learning (ML) is learning based on the
self-organizing system of goal creation in embodied agent.



Machine creates higher level (abstract) goals based on the primitive
pain signals.
It receives internal rewards for satisfying its goals (both primitive and
abstract).
ML applies to EI working in a hostile environment.
EE141
EI Interaction with Environment
EI Architecture
Pain
Perceive
Goal
Creation
Competing
goals
Act
Planning
INPUT
OUTPUT
Task
Environment
Simulation or
Real-World System
EE141
EI machine
interacts with environment using its three pathways
Goal Creation Experiment in ML
Pain
Primitive Hunger
1
Pain
0
0
200
300
400
Lack of Food
500
600
100
200
300
400
Empty Gorcery
500
600
100
200
300
400
Discrete time
500
600
0.5
0
0
Pain
100
0.5
0
0
Pain signals in CGS simulation
EE141
Goal Creation Experiment in ML
Goal Scatter Plot
40
35
30
Goal ID
25
20
15
10
5
0
0
100
200
300
400
Discrete time
500
600
Action scatters in 5 CGS simulations
EE141
Goal Creation Experiment in ML
Pain
Pain
Pain
Pain
Pain
Primitive Hunger
0.5
0
0.2
0.1
0
0.2
0.1
0
0.2
0.1
0
0.1
0.05
0
0
100
200
300
Lack of Food
400
500
600
0
100
200
300
Empty Gorcery
400
500
600
0
100
200
300
Lack of Money
400
500
600
0
100
200
300
400
Lack of JobOpportunitites
500
600
0
100
200
500
600
300
Discrete time
400
The average pain signals in 100 CGS simulations
EE141
Compare RL (TDF) and ML (GCS)
Mean primitive pain
Pp value as a function
of the number of
iterations:
- green line for TDF
-blue line for GCS.
Primitive pain ratio with
pain threshold 0.1
EE141
Compare RL (TDF) and ML (GCS)

Comparison of
execution time on
log-log scale
 TD-Falcon green
 GCS blue

Combined
efficiency of GCS
1000 better than
TDF
Problem solved
Conclusion: embodied intelligence, with motivated learning based on
goal creation system, effectively integrates environment modeling
and decision making – thus it is poised to cross the chasm
EE141
Reinforcement Learning


Single value function
Measurable rewards
 Can be optimized



Predictable
Objectives set by
designer
Maximizes the reward
Motivated Learning

 One for each goal


Learning effort increases
with complexity
Always active
EE141
Internal rewards
 Cannot be optimized



 Potentially unstable

Multiple value functions
Unpredictable
Sets its own objectives
Solves minimax problem
 Always stable


Learns better in complex
environment than RL
Acts when needed
How can we make human level
intelligence?
 We
need to know how
 We need means to
implement it
 We need resources to
build and sustain its
operation
EE141
Resources – Evolution of Electronics
EE141
From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
EE141
From Ray Kurzwail, The Singularity Summit at Stanford, May 13, 2006
Software or hardware?
Software





Sequential
Error prone
Require programming
Low cost
Well developed
programming methods
EE141
Hardware





Concurrent
Robust
Require design
Significant cost
Hardware prototypes
hard to build
Promises of embodied intelligence

To society
 Advanced use of technology
– Robots
– Tutors
– Intelligent gadgets
 Intelligence age follows
– Industrial age
– Technological age
– Information age
 Society of minds
– Superhuman intelligence
– Progress in science
– Solution to societies’ ills

To industry
 Technological development
 New markets
 Economical growth
EE141
ISAC, a Two-Armed Humanoid Robot
Vanderbilt University
Sounds like science fiction


EE141
If you’re trying to look far
ahead, and what you see
seems like science fiction,
it might be wrong.
But if it doesn’t seem like
science fiction, it’s
definitely wrong.
From presentation by Feresight Institute
Embodied Artificial Intelligence
Based on:
[1] E. R. Kandel et al. Principles of Neural Science,
McGraw-Hill/Appleton & Lange; 4 edition, 2000.
[2] F. Inda, R. Pfeifer, L. Steels, Y. Kuniyoshi, “Embodied Artificial
Intelligence,” International seminar, Germany, July 2003.
[3] R. Chrisley, “Embodied artificial intelligence, ” Artificial
Intelligence, vol. 149, pp.131-150, 2003.
[4] R. Pfeifer and C. Scheier, Understanding Intelligence, MIT
Press, Cambridge, MA, 1999.
[5] R. A. Brooks, “Intelligence without reason,” In Proc. IJCAI-91.
(1991) 569-595 .
[6] R. A. Brooks, Flesh and Machines: How Robots Will Change Us,
(Pantheon, 2002).
[7] R. Kurzweil The Age of Spiritual Machines: When Computers
Exceed Human Intelligence, (Penguin, 2000).
EE141
Questions?
EE141
EE141