Transcript Biologically inspired robotics

Controlling Mobile Robots
with Distributed
Neuro-Biological Systems
Sebastian Gutierrez-Nolasco (UCI)
Nalini Venkatasubramanian (UCI)
Alfredo Weitzenfeld (ITAM)
Contact info
[email protected]
Biologically Inspired Robotic Systems

Nature has always been a source of inspiration in the
development of autonomous robotic systems

Ethology

Animal behavior-based simulation


Interaction with the environment is usually oversimplified

Lack of strong biological basis for their working assumptions

Lack of any formal underpinnings for the simulation results
Neuroethology

behavior related to neurobiological structure


Replicate brain models to provide credible and general animal behavior

Provide inspiration for further robotics architectures
More complex and accurate than ethology systems

Enable experimentation

Experimentation requires real-time performance
Neuroethological robotic systems

Super Robots

Incorporate extensive
processing capabilities



Bulky
Expensive
Inexpensive Robots


Smaller and inexpensive robots
connected to a network of
processing nodes
Concerns


Real-time performance
Unpredictable communication
environment affects robot
performance
Challenges of Biologically Inspired Autonomous Robots

A neural model may take hours of processing time

Simulation of multiple neural networks require a distributed
processing environment


Biologically inspired robotics demand sophisticated
image processing techniques


A typical retina model may consist of more than 100,000 neurons and
500,000 interconnections
Communication intensive tasks are required
Autonomous robotic agents have real-time and
processing restrictions, as well as power awareness
requirements

Battery usage is a major concern in mobile robots
Developing Biologically Inspired Robot Architectures

Developing software for autonomous mobile robots is
complex

Highly heterogeneous methods for capturing and processing
sensor information



Multiple sensory input devices
Sensory input multi-granularity
Communication is error-prone due to unpredictable interference
and failures



partial and complete failures
Unreliability and disconnection
Varying available bandwidth
Our Approach

Develop an embedded architecture capable of
conducting neuroethological robotic experimentation

Inexpensive small robots communicate (wireless) with distributed
computational resources

Neural models are distributed in multiple processing nodes

Adaptive robotic middleware optimizes robot
communication in response to varying network conditions
Structure of the Talk
Neuroethological Modeling
1.

Study animal behavior and corresponding neural structure
as inspiration to robotic architectures
Embedded Mobile Robots
2.

Develop distributed wireless robot architectures capable of
efficient neural processing
Adaptive Middleware
3.

Achieve real-time computation and adapt embedded
architecture to varying network conditions
Internet Based Robotics
4.

Enable remote robot task development and
experimentation
Neuroethological Modeling
1.

Study animal behavior and corresponding neural structure
as inspiration to robotic architectures
Embedded Mobile Robots
2.

Develop distributed wireless robot architectures capable of
efficient neural processing
Adaptive Middleware
3.

Achieve real-time computation and adapt embedded
architecture to varying network conditions
Internet Based Robotics
4.

Enable remote robot task development and
experimentation
Behavior:
Frog and Toad - Rana Computatrix [Arbib 1987, Cervantes 1990]
-
PS
MO
+
+
-
PS
Mate
PS
Prey
Mate-Pair
S
Prey-Acq
S
Moving-Object
S
Perceptual Schema (PS)
Main Schema (S)
PS
NMO
+
-
PS
Pred
Pred-Av
S
Find-Loc
S
Non-Moving-Object
S
Behavior:
Toad Prey Acquisition [Cervantes 1985]
Stimulus
Response
Mobile visual stimulus
in lateral visual field
(monocular perception)
Orientation
Mobile visual stimulus
in binocular visual field
(short distance)
Binocular
fixation
Attack
Mechanic stimulus
in mouth and pharynx
receptors
Snap
Clean
Behavior:
Toad Prey Acquisition with Detour Behavior Before and After Learning
[Corbacho and Arbib 1995]
10cm Barrier
20cm Barrier Before learning
20cm Barrier After learning
Schema Computational Model
...
...
din n
Schema Level 1
Schema
...
...
dout 1
din 1
dout m
data in
data out
Schema Level 2
Schema Level
Neural Level
Neural
Other Processes
Neural based Behavior :
Toad Prey Acquisitions and Predator Avoidance
Prey Recognizer
Prey Approach
Moving Stimulus Selector
Predator
Recognizer
Depth
Forward
Predator Avoid
Visual
Orient
Backward
Sidestep
Static Object
Recognizer
Tactile
Static Object
Avoidance
Schema Level
Tectum
Retina
R1-R2
R1-R2
R3
R3
R4
R4
Stereo
T5_2
R1-R2
Motor Heading Map
R3
R4
MaxSelector
Neural Level
TH10
PreTectum/Thalamus
Toad Prey Acquisition with Detour:
Simulation Results
16
15
14
4 7 10 13
12
6 9
3
5 8 11
2
1
10cm barrier
20cm barrier
Before learning
20cm barrier
After learning
Neuroethological Modeling
1.

Study animal behavior and corresponding neural structure
as inspiration to robotic architectures
Embedded Mobile Robots
2.

Develop distributed wireless robot architectures capable of
efficient neural processing
Adaptive Middleware
3.

Achieve real-time computation and adapt embedded
architecture to varying network conditions
Internet Based Robotics
4.

Enable remote robot task development and
experimentation
Embedded Mobile Robots:
Robot Hardware
Sensors
Vision
Tact
Actuators
Legs
Wheels
LEGO
OOPIC
Embedded Mobile Robots:
Distributed Embedded Architecture
Wireless
Remote
Computaional
System
Autonomous Robot 1
...
...
Internet
Server
...
Internet
Instance 1
Autonomous Robot N
Remote
Computational
System
Instance N
Embedded Mobile Robots:
Distributed Embedded Architecture
 Time consuming processes are carried out in the
(neural) computational system



Limited task are carried out in the robot hardware




Neural processing
Image processing
Sensory input
Motor output
Default behavior
Communication and data transformation is managed by
the adaptive middleware
Embedded Mobile Robots:
Distributed Embedded Architecture
camera
Remote Computaional System
servo
Robot
Wireless
Frame Grabber
Transceiver
PC
CPU
(OOPic)
Transceiver
Power stage
motor
motor
Sensors
(tact)
Embedded Mobile Robots:
Distributed Embedded Architecture
Remote Computational System
Wireless
Video/
Image
Processing
ASL
ASL
ASL
NSL
NSL
NSL – Neural Simulation Language
ASL – Abstract Schema Language
Robot
Video
Server
Tactile
Server
Motor
Server
transceiver
ASL
NSL
transceiver
NSL
camera
tactile
motor
Embedded Mobile Robots:
Processing cycle
Video capture
Video processing
Model simulation
Model output
Navigation control
(d , r , c)
Neuroethological Modeling
1.

Study animal behavior and corresponding neural structure
as inspiration to robotic architectures
Embedded Mobile Robots
2.

Develop distributed wireless robot architectures capable of
efficient neural processing
Adaptive Middleware
3.

Achieve real-time computation and adapt embedded
architecture to varying network conditions
Internet Based Robotics
4.

Enable remote robot task development and
experimentation
Distributed Systems Middleware

Enables the modular interconnection of distributed
software


Concurrent Object Oriented Model


Abstract over low level mechanisms used to implement resource
management services
Separation of concerns and reuse of services
Customizable, Composable Middleware Frameworks


Provide for dynamic network and system customizations,
dynamic invocation/revocation/installation of services
Concurrent execution of multiple resource management policies
Core Resource Management Services

Core Services - basic services where interactions
between the application and system can occur.



Building blocks for other services
Reduce interactions among many services to interactions
between a few simple services
Choosing core services - commonly observed patterns



Recreation of data/services at a remote site
Capturing approximation of distributed state at multiple sites
Interactions with a global repository
TLAM: The Two Level Meta-architecture
System (Meta) Level
Replication
Access
Control
DGC
Migration
Remote
Creation
Checkpointing
Distributed
Snapshot
Directory
Services
Application (Base) Level
Adaptive Robotic Middleware (ARM)

Extends the TLAM to

Optimize information flow between robots and the computational
system


Compose communication protocols to obtain the combined
benefits - conflicting requirements


Determine how, when and what information should be modified in
order to match fluctuations in the communication environment
Explicit knowledge of how communication protocols compose and
interact is required
Adapt protocols and mechanisms to changing communication
and power constraints
ARM: Distributed Embedded Architecture
Remote Computational System
Wireless
ARM
Video/
Image
processing
Robot
Video
Server
ASL
ASL
ASL
NSL
NSL
NSL – Neural Simulation Language
ASL – Abstract Schema Language
Tactile
Server
Motor
Server
ARM
transceiver
ASL
NSL
transceiver
NSL
camera
tactile
motor
ARM: Components

Communication manager


Provide and enforce application level requirements
Components

Oracle





determine most suitable protocol implementation in terms of coverage and efficiency
Set of communication protocols
Protocol installer/uninstaller
Resident ARM module running in the robot (resident evil)
Adaptation manager

Provide adaptation and monitor mechanisms operating at different levels of
abstraction

Reactive


Proactive


Triggered when failure to achieve intended communication goal is detected
Triggered when a more efficient communication can be achieved under the current
environment conditions
Adaptation Repository

Determine most suitable adaptation strategy to be applied
ARM: Example
Communication Requirement
Secure compressed
video capture
Communication Manager
Protocol
(un)installer
Protocol
Oracle
ARM manager
Protocol
loader
set of
protocols
Robot’s
resident
ARM
Battery
Monitors
Adaptation
Profiler
Security
Protocol
Compression
Protocol
Timeliness
Protocol
Communication Performance Monitor
Adaptation
Repository
Adaptation Manager
Neuroethological Modeling
1.

Study animal behavior and corresponding neural structure
as inspiration to robotic architectures
Embedded Mobile Robots
2.

Develop distributed wireless robot architectures capable of
efficient neural processing
Adaptive Middleware
3.

Achieve real-time computation and adapt embedded
architecture to varying network conditions
Internet Based Robotics
4.

Enable remote robot task development and
experimentation
Interned based Robotics: Web Access
Experimental Results: 2 Preys
Experimental Results: 2 Preys and Predator
Embedded Mobile Robots:
Experimental Results: Prey Acquisition with 10 cm Barrier
(A)
(B)
(C)
(D)
Embedded Mobile Robots:
Experimental Results: Prey Acquisition with 20 cm Barrier
(A)
(B)
(C)
(D)
(E)
(F)
(G)
(H)
Neural based Behavior: Prey Acquisition (10cm barrier)
• Barrier (PreTectum)
• Prey (Tectum)
• Integrated (MHM)
• Heading (MHM)
•Tactile
Visual Fields
Neural based Behavior: Prey Acquisition
(20cm barrier before bumping)
• Barrier (PreTectum)
• Prey (Tectum)
• Integrated (MHM)
• Heading (MHM)
•Tactile
Visual Fields
Neural based Behavior: Prey Acquisition
(20cm barrier after bumping)
• Barrier (PreTectum)
• Prey (Tectum)
• Integrated (MHM)
• Heading (MHM)
•Tactile
Visual Fields
Neural based Behavior: Prey Acquisition
(20cm barrier after learning)
• Barrier (PreTectum)
• Prey (Tectum)
• Integrated (MHM)
• Heading (MHM)
•Tactile
Visual Fields
Future Work






Complete Internet based System
Develop middleware adaptation capabilities
Build smaller robotic systems
Extend to multiple robot tasks
Extend vision system to “true” moving forms
Extend biological models
Video
Bonus Section
Research Cycle
Formal Models
Data, Hypotheses
Neuroscience
(Experiments)
Brain Theory
(Modeling)
New Hypotheses
New Ideas
(Results from
Experiments with
Physical Devices)
New Hypothesis
Gaps in Knowledge
Robotics
Neural Maps
T - Temporal, D - Dorsal, N - Nasal, V - Ventral
O - Optic Tectum, B - Nucleus of Belonci
C - Lateral Geniculate Nucleus,
P - Thalamic Pretectal Neuropil
X - Basal Optic Root
[Scalia and Fite 1974]
Neuron Model
s
mp
mf
input
neuron
output



mp - membrane potential : dmp(t)/dt = f(s,mp,t)
mf - firing rate : mf(t) =s(mp(t))
Leaky Integrator : t dm(t)/dt = -m(t) + s
Retina-Thalamus-Tectum
+
+
Retina
Input
TP
R4
R3
R2
+
+
+
TP – ThalamusPreTectum
GL – Glomerelus
SN – Stellate Neurons
SP – Small Pear
LP – Large Pear
PY - Pyramidal
+
GL
SN
-
-
SP
+
+
+
-
LP
Synapsis
+
Excitation
-
Inhibition
+
-
+
+ +
+
+
+
-
PY
Output
Max Selector [Didday 1976]
s0
up1
up0
wu0
uf0
wu1
wm
upi
........
uf1
w0
tu
si
s1
wun-1
ufn-1
wi
w1
wn-1
vp
vf
dui t 
 ui  wu f ui   wm g v   h1  si
dt
n
dv
t v  v  wn  f ui   h2
dt
i 1
upn-1
........
ufi
wui
sn-1
1 ui  0
f (ui )  
0 ui  0
v v  0
g(v)  
0 v  0
Max Selector Model
MaxSelectorModel
MaxSelector
out
MaxSelector
Stimulus
s_out
in
Vlayer
Ulayer
s_in
uf
u_in
vf
uf
s_in
MaxSelector
Output
v_in
nslModel MaxSelectorModel () extends NslModel()
{
private MaxSelector maxselector(10);
private MaxSelectorStimulus stimulus(10);
private MaxSelectorOutput output();
public void initSys() {
system.setRunTime(10.0);
system.setRunDelta(0.1);
}
public void makeConn() {
nslConnect(stimulus.s_out,maxselector.s_in);
nslConnect(stimulus.s_out,output.s_in);
nslConnect(maxselector.out, output.uf);
}
}
Max Selector Module
MaxSelector
out
Vlayer
Ulayer
in
s_in
uf
u_in
vf
v_in
nslModule MaxSelector (int size) extends NslModule()
{
public Ulayer u1(size);
public Vlayer v1(size);
public NslDinDouble1 in(size);
public NslDoutDouble1 out(size);
public void makeConn(){
nslRelabel(in,u1.s_in);
nslConnect(v1.vf,u1.v_in);
nslConnect(u1.uf,v1.u_in);
nslRelabel(u1.uf,out);
}
}
Ulayer and Vlayer Modules
Vlayer
Ulayer
s_in
u _in
uf
vf
v _in
nslModule Ulayer(int size) extends NslModule ()
{
public NslDinDouble1 s_in(size);
public NslDinDouble0 v_in();
public NslDoutDouble1 uf(size);
private NslDouble1 up(size);
private NslDouble0 hu();
private double tau;
public void simRun() {
up =0;
uf = 0;
hu = 0.1;
tau =1.0;
}
public void simRun() {
up = nslDiff(up,tau, -up + uf - v_in – hu + s_in);
uf = nslStep(up,0.1,0.1.0);
}
}
nslModule Vlayer(int size) extends NslModule ()
{
public NslDinDouble1 u_in(size);
public NslDoutDouble0 vf();
private NslDouble0 vp();
private NslDouble0 hv();
private double tau;
public void initRun() {
vp =0;
vf = 0;
hv=0.5;
tau=1.0;
}
public void simRun() {
vp = nslDiff(vp,tau,-vp+nslSum(u_in) – hv);
vf = nslRamp(vp);
}
}
Neuron (detailed)
Neuron
Dendrite
Axon
Soma
Axon
Dendrites
Synapse
Soma
Synapses
Axon
Terminal
Spine
Synaptic
Cleft
Calcium
Mechanism
Receptor
Vescicle Interacellular
Element
Channel
Diffusion
Buffer
Pump
AMPA
Receptor
NMDA
Receptor
Neuroscience:
Autonomous Biological Agents
Sensors
Vision
Sound
Smell
Touch
Actuators
Legs
Wings
Fins
Robotics:
Autonomous Robotic Agents
Sensors
Vision
Sound
Smell
Touch
Actuators
Legs
Wings
Fins
Wheels