Transcript Machine Intelligence

How to Motivate
Machines to Learn and
Help Humans in Making
Water Decisions?
Janusz Starzyk
School of Electrical Engineering and
Computer Science, Ohio University,
USA
www.ent.ohiou.edu/~starzyk
EE141
Outline







Embodied Intelligence (EI)
Embodiment of Mind
EI Interaction with Environment
How to Motivate a Machine
Goal Creation Hierarchy
Goal Creation Experiment
Promises of EI
 To economy
 To society
EE141
Intelligence
AI’s holy grail
From Pattie Maes MIT Media Lab

“…Perhaps the last frontier of science – its
ultimate challenge- is to understand the biological
basis of consciousness and the mental process by
which we perceive, act, learn and remember..”
from Principles of Neural Science by E. R. Kandel et al.
 E. R. Kandel won Nobel Price in 2000 for his work on physiological
basis of memory storage in neurons.
 “…The question of intelligence is the last great
terrestrial frontier of science...” from Jeff Hawkins On
Intelligence.
 Jeff Hawkins founded the Redwood Neuroscience Institute devoted
to brain research
EE141
Traditional AI

Abstract intelligence
Embodied Intelligence

 attempt to simulate
“highest” human faculties:
 knowledge is implicit in the
fact that we have a body
– language, discursive
reason, mathematics,
abstract problem solving

Environment model
 Condition for problem
solving in abstract way
 “brain in a vat”
EE141
Embodiment
– embodiment is a
foundation for brain
development

Intelligence develops
through interaction with
environment
 Situated in a specific
environment
 Environment is its best
model
Design principles of intelligent systems
from Rolf Pfeifer “Understanding of Intelligence”, 1999








EE141
Interaction with
complex environment
cheap design
ecological balance
redundancy principle
parallel, loosely
coupled processes
asynchronous
sensory-motor
coordination
value principle
Agent
Drawing by Ciarán O’Leary- Dublin Institute of Technology
Embodied Intelligence
Definition
 Embodied Intelligence (EI) is a mechanism that learns
how to survive in a hostile environment
– Mechanism: biological, mechanical or virtual agent
with embodied sensors and actuators
– EI acts on environment and perceives its actions
– Environment hostility is persistent and stimulates EI to act
– Hostility: direct aggression, pain, scarce resources, etc
– EI learns so it must have associative self-organizing memory
– Knowledge is acquired by EI
EE141
Embodiment of a Mind




Embodiment contains
intelligence core and
sensory motor interfaces
under its control to interact
with environment
Necessary for development
of intelligence
Not necessarily constant or
in the form of a physical
body
Boundary transforms
modifying brain’s selfdetermination
EE141
Embodiment
Sensors
channel
Environment
Intelligence
core
Actuators
channel
Embodiment of a Mind




Brain learns own body’s dynamic
Self-awareness is a result of
identification with own embodiment
Embodiment can be extended by
using tools and machines
Successful operation is a function
of correct perception of
environment and own embodiment
EE141
EI Interaction with Environment
Agent Architecture
Reason
Short-term Memory
Perceive
Act
RETRIEVAL
LEARNING
Long-term Memory
INPUT
OUTPUT
Task
Environment
Simulation or
Real-World System
EE141
From
Randolph M. Jones, P : www.soartech.com
How to Motivate a Machine ?
The fundamental question is how to
motivate a machine to do anything, in
particular to increase its “brain”
complexity?
How to motivate it to explore the
environment and learn how to
effectively work in this environment?
Can a machine that only implements
externally given goals be intelligent?
If not how these goals can be
created?
EE141
How to Motivate a Machine ?

I suggest that hostility of environment motivates us.
 It is the pain that moves us.
 Our intelligence that tries to minimize this pain motivates our actions,
learning and development

We need both the environment hostility and the mechanism
that learns how to reduce inflicted by the environment pain

I propose based on the pain
mechanism that motivates the
machine to act, learn and develop.
So the pain is good.
Without the pain there will be no intelligence.
Without the pain there will be no motivation to
develop.
EE141
Pain-center and Goal Creation
Dual pain level
Pain increase
Sensor
(-)





Simple Mechanism
Creates hierarchy of
values
Leads to formulation of
complex goals
Reinforcement :
• Pain increase
• Pain decrease
Forces exploration
+
(+)
Environment
(+)
(-)
Pain level
Wall-E’s goal is to keep
his plants from dying
EE141
(-)
-
(+)
Motor
Pain decrease
Excitation
Primitive Goal Creation
faucet
refill
garbage
w. can
sit on
water
tank
Dual pain
Dry soil
EE141
+
Pain
Primitive
level
open
Abstract Goal Creation
 The goal is to reduce
the primitive pain level
 Abstract goals are
created to reduce abstract
pains in order to satisfy the
primitive goals
 Abstract pain center
Sensory pathway Motor pathway
(perception, sense) (action, reaction)
faucet
“water can” –
sensory input
to abstract pain
w. can
center
Activation
Stimulation
Inhibition
Reinforcement
Echo
Need
Expectation
EE141
open
-
Dry soil
+
Abstract pain
water
Dual pain
Level II
Level I
+
Pain
Primitive
Level
Abstract Goal Hierarchy
Sensory pathway
(perception, sense)
 A hierarchy of
abstract goals is
created - they satisfy
the lower level goals
Motor pathway
(action, reaction)
tank
refill
-
+
faucet
open
-
Activation
Stimulation
Inhibition
Reinforcement
Echo
Need
Expectation
Dry soil
EE141
Level II
+
w. can
water
-
Level III
Level I
+
Primitive
Level
GCS vs. Reinforcement Learning
States
Policy
Desired
action
&state
Pain
Critic
States
Value
Function
action
GCS
Sensory
pathway
Action
decision
Motor
pathway
reward
Environment
Gate control
Environment
Action
Actor-critic design
Goal creation system
Case study: “How can Wall-E water his plants if
the water resources are limited and hard to find?”
EE141
Goal Creation Experiment
SENSORY
MOTOR
INCREASES
DECREASES
1
water can
water the plant
moisture
water in can
8
faucet
open
water in can
water in tank
15
tank
refill
water in tank
reservoir water
22
pipe
open
reservoir water
lake water
29
rain
fall
lake water
-
PAIR #
Sensory-motor pairs and their effect on the environment
EE141
Results from GCS scheme
Dry soil
pain
4
2
0
0
100
200
400
500
600
300
400
500
600
300
400
500
600
300
400
500
600
300
400
500
600
300
No water in can
pain
2
1
0
0
100
200
100
200
100
200
100
200
pain
2
1
0
0
pain
1
No water in reservoir
0.5
0
0
pain
4
No water in lake
2
0
0
EE141
No water in tank
GCS vs. Reinforcement Learning
Averaged performance over 10 trials:
GCS:
Primitive pain
pain
1
0.5
0
0
100
200
400
500
600
400
500
600
400
500
600
Lack of food
RL:
1
pain
300
30
0.5
20
0
0
100
200
300
10
Lack of money
0.4
0
pain
0
100
200
300
0.2
Machine using GCS learns to control all abstract pains and
0
maintains
the primitive
pain
0
100
200
300 signal on400a low level
500 in
Lack of bank savingsconditions.
demanding environment
0.4
EE141
600
Goal Creation Experiment
Goal Scatter Plot
40
35
30
Goal ID
25
20
15
10
5
0
0
100
200
300
400
Discrete time
500
600
Action scatters in 5 CGS simulations
EE141
Goal Creation Experiment
Pain
Pain
Pain
Pain
Pain
Primitive pain – dry soil
0.5
0
0.2
0.1
0
0.2
0.1
0
0.2
0.1
0
0.1
0.05
0
0
100
200
300
400
Lack of water in can
500
600
0
100
200
300
400
Lack of water in tank
500
600
0
100
200
300
400
Lack of water in reservoir
500
600
0
100
200
300
400
Lack of water in lake
500
600
0
100
200
300
Discrete time
500
600
400
The average pain signals in 100 CGS simulations
EE141
Promises of embodied intelligence

To society
 Advanced use of technology
– Robots
– Tutors
– Intelligent gadgets
 Intelligence age follows
– Industrial age
– Technological age
– Information age
 Society of minds
– Superhuman intelligence
– Progress in science
– Solution to societies’ ills

To industry
 Technological development
 New markets
 Economical growth
EE141
ISAC, a Two-Armed Humanoid Robot
Vanderbilt University
Biomimetics and Bio-inspired Systems
Mission Complexity
Impact on Space Transportation, Space Science and Earth Science
2002
2010
2020
2030
Embryonics
Self Assembled Array
Space Transportation
Memristors
Biologically inspired
aero-space systems
Sensor Web
Extremophiles
Mars in situ
life detector
Brain-like
computing
Skin and Bone
Self healing structure
and thermal protection
systems
EE141
Biological nanopore
low resolution
Artificial nanopore
high resolution
DNA
Computing
Biological Mimicking
Sounds like science fiction


EE141
If you’re trying to look far
ahead, and what you see
seems like science fiction,
it might be wrong.
But if it doesn’t seem like
science fiction, it’s
definitely wrong.
From presentation by Feresight Institute
Questions?
EE141