Transcript Neural Coalition and Main Theorem

SHANNON LECTURE
LIVING INFORMATION
THEORY
Toby Berger
Cornell University
IEEE ISIT – LAUSANNE
4 July 2002
PART III:
NEURAL COALITIONS
AND MAIN THEOREM
DEFINITION OF A COALITION OF SENSORY NEURONS
THE AXONS IN A COALITION OF SENSORY
NEURONS FORM MANY OF THEIR SYNAPSES
WITH OTHER NEURONS IN THE COALITION
(INTER-COALITION FEEDBACK). SOMETIMES
THE LOCAL CONNECTIVITY IS CLOSE TO 50%, AS
OPPOSED TO ONLY 10-7 BRAINWIDE.
THE REMAINDER OF THE SYNAPSES IN WHICH
THE AXONS OF A COALITION ARE INVOLVED ARE
SPLIT ON AVERAGE ABOUT EQUALLY BETWEEN
“LOWER” NEURONS AND “HIGHER” NEURONS.
HERE, “LOWER” MEANS CLOSER TO THE
SENSORY ORGAN AND “HIGHER” MEANS
CLOSER TO THE ‘TOP BRAIN’.
BOTTOM-UP v. TOP-DOWN
INFORMATION TRAVELING ALONG AXONS
THAT GO FROM LOWER TO HIGHER IS
CALLED BOTTOM-UP, OR FEEDFORWARD,
INFORMATION. THAT WHICH TRAVELS
ALONG AXONS THAT GO FROM HIGHER TO
LOWER IS CALLED TOP-DOWN, OR
FEEDBACK, INFORMATION.
Real neural spikes occur in continuous time.
We introduce a time-discrete model with a
time step of circa 2 ms (typical spike width).
This captures the essence, provided the
exact time of occurrence of a neural spike is
not significant to an accuracy much greater
than 1 ms.
“FIGURE 1” OF NEURAL INFORMATION THEORY
SCHEMATIC OF
A COALITION
PSPs
THE COALITION CHANNEL
P( y (1),..., y (n) | y (0), x(1),..., x(n))





n
  P( y (k ) | x(k ), y (k  1))



k 1
NEXT SLIDE SHOWS THAT A NEURAL COALITION IS A LINEAR
(BUT THRESHOLDED) RANDOMIZED DYNAMICAL SYSTEM
COALITION EQUATION
Y (k )  U ( PSP (k )  T )
m
m
Y (k )  U ( Y (k  1)W Q (k ) S (k ) 
m
i
im im
im
i
l X l (k )WlmQlm (k )Slm (k )  T )
where
W
im
Q (k )
im
S (k )
im
=
Signed weight of synapse (i,m)
=
Random size of quantal contribution of synapse (i,m) to
PSP of coalition member m at time k
= 0 if synapse (i,m) fails at time k, 1 if not. (The fascinating
phenomenon of “quantal failure” deserves a full lecture of its
own – see Levy and Baxter, JCN 2002 and Proc. ISIT ’02, p. 18.)
W , Q (k ), S (k )
lm lm
lm
T T
(k ) 
m
analogously defined
Spiking threshold of neuron m at time k
MAXIMUM INFORMATION RATE HYPOTHESIS
The input (afferent) process {X(k))} that drives a coalition will
have the property that it maximizes the mutual information rate
n  1I ( X (1),..., X (n); Y (1),..., Y (n))




between itself and the output (efferent) process {Y(k)} that it
generates over all processes that lead to the same or smaller
energy expenditure in the Y-neurons.
Remark: Energy is expended both in the synapses in receiving
and responding to afferent excitation and in the axons both to
restore chemical concentrations during refractory periods
following action potential generation and, to a lesser extent, to
drive spikes down the axonal ‘transmission lines’.
JUSTIFICATIONS FOR THE MAX INFO RATE HYPOTHESIS
• Brain consumes 25% to 50% of metabolic energy under low
physical exertion, so its computations should be energy-aware*.
• Inefficient to build and maintain a coalition and then expend
more energy than necessary to transfer whatever information
about its afferent process it conveys to its efferent process.
• The max info rate hypothesis leads to Markovianness of
{X(k),Y(k)} which reduces latency by encapsulating memory.
*L. Sokoloff (1989), “Circulation and energy metabolism of the brain,” in Basic
Neurochemistry: Molecular, Cellular and Medical Aspects, 4th ed., G. Siegel et al., Eds.
The Brain as a Markov Chain
MAIN THEOREM:
IF MAXIMUM INFORMATION RATE HYPTOTHESIS IS TRUE, THEN:
• {(Xk,Yk} is a first-order (non-homog) Markov chain
• {Yk} is a first-order (non-homog) Markov chain
• {Xk} is not necessarily Markovian
PROOF: IF TIME PERMITS
(This is joint work with Yuzheng Ying with thanks to Claudio Weidmann.)
REMARKS:
• Max info rate hypothesis says the source {X(k)} is robustly
“matched” to the channel’s transition matrix, P(y|x).
• If double matching prevails, as we suspect it does, then the
failure rate parameterizes the rate-distortion function, and
distortion is measured by a Weber-Fechner fidelity criterion
of the form
n

1
n
 d ( x(k ), y (k ), y (k  1)).
k 1
• The Markovianness of the Main Theorem is essential to
the brain’s low-latency processing of sensory information.
Without it, bottom-up delay would accumulate too fast to
allow for the number of coalitions needed to achieve the
sophisticated distinctions of which the brain is capable.
Temporal Dynamics of a V1 Neuron’s Response to Real
and Illusory Contours
From T. S. Lee and M. Nguyen, Dynamics of subjective contour formation in the early visual cortex. PNAS 98(4):1907-1911, 2001.
OPEN PROBLEMS
• All problems in bio-IT are open, including those I may have
implied are solved.
•What is memory? How is it physically stored and accessed?
• Can the max information rate hypothesis be proved by
appealing to a least action principal in chemical statistical
mechanics? (Perhaps this can be approached via the fact
that the solution of multiphase chemical equilibrium problems
is obtained by solving for the minimum of the
Gibbs/Helmholtz Free Energy which, in turn, is
mathematically fully analogous to calculation of a point on a
rate-distortion function.)
CLOSING CONJECTURE
The most remarkable forms of information
processing – those we call consciousness,
intelligence and creativity – all are deeply
reflexive. They are products of our selfawareness. It seems reasonable to conjecture
that these forms of thinking emerge from the
brain’s pervasive and intricate use of feedback.