Transcript Presentation

FUSION OF
LANGUAGE AND THOUGHT
PROCESSES FOR CTS
Collaborative Technologies and
Systems Conference 2003, Orlando,
Florida
Leonid Perlovsky
Technical Advisor, AFRL
CTS AND INTELLIGENT AGENTS

Collaborative systems include multiple interacting
intelligent agents
– a human, machine, device or software code

Agents are
– significantly autonomous and goal-oriented, perform
various functions, and communicate with other agents
– equipped with sensors or collect data, receive
communications, extracts information
– use existing knowledge, integrate new information into
producing new knowledge, send communications
– embody the concept of life and intelligence

What are the required intelligent agent
technologies?
INTELLIGENT AGENT
TECHNOLOGIES

Interfaces and access
– man-machine and machine-machine interfaces
– knowledge and data access

Understanding of
– language, situations, and environment

Fusion
– knowledge and data from diverse sources and disciplines,

Decision making
– heterogeneous environment, with inaccurate data,
uncertain knowledge and intuitions
– information exchange, knowledge management

Abilities for thinking and language
– a mysterious territory
LANGUAGE AND THINKING
PAST

Artificial Intelligence, 1950s-1980s
– logical rules
– no principal difference between thinking and language
– failed, yet no replacements to logic when combining L. & T.

Linguistics
– Chomskyan linguistics, computational linguistics, cognitive
linguistics: relate words to words
– No relation to surrounding world

Closely related and intertwined evolution in
human mind
– science cannot tell us yet what is language without thinking
or v.v.
LANGUAGE AND THINKING
FUTURE

Thinking and understanding
– identify “concepts” of objects, relationships and situation
in sensory data
– relate concepts to needs and emotions
– relate concepts to behavior

Language is a part of mind
– involved in concepts, emotions, instincts
– closest to concepts
– words are combined into phrases like retinal signals are
combined into objects (?)

Where language and thoughts come
together?
– Concepts? Not logical rules and artificial intelligence
again?
PHYSICS AND MATHEMATICS OF MIND
RANGE OF CONCEPTS

Logic is sufficient to explain mind
– [Newell, “Artificial Intelligence”]

No new specific mathematical concepts are needed
– Mind is a collection of ad-hoc principles, [Minsky]

Specific mathematical constructs describe the
multiplicity of mind phenomena
– “first physical principles of mind”
– [Grossberg, Zadeh, Perlovsky,…]

Quantum computation
– [Hameroff, Penrose, Perlovsky,…]

New unknown yet physical phenomena
– [Josephson, Penrose]
GENETIC ARGUMENTS
FOR THE “FIRST PRINCIPLES”

There is about 30,000 genes in human genome
– relate concepts to needs

Only about 2% difference between human and apes

Say, 1% difference between human and ape minds
– Only about 300 proteins

It is likely, that few general principles of concept
learning are required to explain our ability to operate
with concepts
– If we count “a protein per concept”
– If we count combinations: 300300 ~ unlimited => all
languages could have been genetically h/w-ed (!?!)

Languages are not genetically hardwired
– Because they have to be flexible and adaptive
INFORMATION PROCESSING
AND UNDERSTANDING

Understanding the meaning of signals (visual,
acoustic, text)
– Identify objects in signals
• signals -> concepts; or words -> phrases
– Associate relevant objects
• objects -> scenes;

or phrases -> more general concepts
In this task the human mind is by far superior
qualitatively to existing mathematical methods
– Effort has been devoted toward incorporating “biological
lessons” into smart algorithms, yet success has been
limited
– Why is this so and how to overcome existing limitations?
REASONS FOR PAST LIMITATIONS

The basis of human intelligence is in combining
conceptual understanding with emotional evaluation
– became exceedingly well appreciated among psychologists and
neurobiologists during the last ten years
– human understanding without emotional involvement is basically
flawed [Damasio]

This new understanding has not been accepted by
mathematical and engineering community
– mathematical laws governing emotional involvement into thinking
process have not been well known
– there is a long-standing cultural belief that emotions are opposite
to thinking and intellectually inferior
• Socrates, Plato, Aristotle
• reiterated by founders of Artificial Intelligence [Newell]
FUNDAMENTAL MATHEMATICAL
PROBLEM

Combinatorial Complexity (CC):
– understanding involves evaluating a large number
of combinations
– words into sentences, pixels or samples into
objects, objects into scenes
– a general problem of all data association methods

CC was encountered for over 50 years
– statistical pattern recognition and neural networks:
CC of learning requirements
– rule-based systems, expert systems, and AI, in the
presence of variability: CC of rules
– model-based systems, utilizing adaptive models:
CC of computations (N and NP complete)
– Chomskyan linguistics: (1-1957) rule-based, (21981) model-based (rules and parameters)
CC AND TYPES OF LOGIC

CC is related to formal logic
– law of excluded third
• every logical statement is either true or false
– there could be no similarity/distance measure between
logical statements
– CC is Godel’s “incompletenes” in a finite system

Multivalued and fuzzy logic eliminated the “law of
excluded third”
– yet, they are based on the math. of formal logic
– e.g. fuzzy logic systems are either too fuzzy or too crisp

A similarity measure between logical statements was
not introduced
STRUCTURE OF MIND

Concepts
– Models of objects, their relations, and situations
– Evolved to satisfy instincts

Instincts
– Internal sensors (e.g. sugar level in blood)

Emotions
– Neural signals connecting instincts and concepts
• e.g. a hungry person sees food all around

Behavior
– Models of goals (desires) and muscle-movement

Hierarchy
– Concept-models and behavior-models are organized in a “loose”
hierarchy
SIMILARITY MEASURE AND
DYNAMIC LOGIC

A similarity measure
– based on a similarity between models and data (words)
• equivalent to emotional evaluative signals
– leads to dynamics (equations of motions) improving
concept-models by maximizing similarity (dynamic logic)

Dynamic Logic unifies formal and fuzzy logic
– initial “fuzzy model-concept” dynamically evolve into
“formal-logic or crisp model-concept”

Overcomes CC of model-based recognition
– fast algorithms
• low-polynomial M*N, instead of combinatorial MN
– associates pixels into objects (or words into phrase-models)
without CC
THINKING

Understanding and learning
– Model-concepts always have to be adapted to incoming
signals or data
• lighting, surrounding, new objects and situations

Instinct for knowledge and understanding
– Models-concepts are adapted and improved even when there
is no concrete “bodily” needs
– Increase similarity between models and world

Emotions related to knowledge instinct
– Satisfaction or dissatisfaction
• change in similarity between models and world
– Harmony or disharmony: aesthetic emotion

Behavior related to knowledge
– Adaptation and learning of concepts (behavior in the mind)
MODELING FIELD THEORY
basic two-layer hierarchy: from signals to objects

Signals and Concept-Models
– signals x(n), n = 1,…,N
– model-object Mm(Sm,n), parameters Sm, m = 1, …;

Goal: learn object-models and understand signals
– associate samples n with models m and find parameters Sm
– learn signal-contents of objects (and object properties)

Maximize similarity, between signals and models, L
– knowledge instinct
– Likelihood or mutual information, L = l({x}) =  l(x(n))
– l(x(n)) =  r(m) l(x(n) | Mm(Sm,n)) (M may depend on n)
– CC: L contains MN items: all associations of words and models
n
m
DYNAMIC LOGIC ALGORITHM (DLA)
(non-combinatorial solution)

Start with a set of signals and unknown object-models
– any parameter values Sm
– associate fuzzy object-model with its contents (signal composition)
– (1)
f(m|n) = r(m) l(n|m) /  r(m') l(n|m')
m'

Improve parameter estimation
– (2)
Sm = (1- a) Sm + a  f(m|n) [ll(n|m)/Mm]*[Mm/Sm]
n
• (a determines speed of convergence)
– learn signal-contents of objects

Continue iterations (1)-(2). Theorem: MFT is converging
- similarity increases on each iteration
- aesthetic emotion is positive during learning

Each concept-model is an agent
– semi-independent, interacting with other agents
• competing for evidence (among signals)
– learning its properties and recognizing its signals
LINGUISTICS
WORDS, CONCEPTS AND GOALS

Text is a (loose) hierarchy of concepts
– Word is a concept; it acquires meaning in a phrase
– Phrase-concept acquires meaning in a “paragraph”,…

Model-concepts (e.g. phrases made up of words)
– Simplistic “bag”-model
• a set or collection of words
– More complex models: word order and relationships
– Real-language models
• grammar (Chomsky, Pinker, Jackendoff, Rieger, Mehler…)

Goal-instinct
– A search engine: find conceptual similarity between a query and
text
• analyze both in terms of concepts
– Learn and identify model-concepts in texts (language instinct =
knowledge instinct)
MODELING FIELD THEORY
basic two-layer hierarchy: words and phrase-concepts

Words and Concept-Models
– words w(n), n = 1,…,N
– model-phrase Mm(Sm,n), parameters Sm, m = 1, …;
– Simplistic “bag”-model: Mm = Sm = {w}m

Goal: learn phrase-models
– associate words n with models m and find parameters Sm
– learn word-contents of phrases (and grammatical relationships)

Maximize similarity, between words and models, L
– language instinct = knowledge instinct
– Likelihood or mutual information, L = l({w}) =  l(w(n))
– l(w(n)) =  r(m) l(w(n) | Mm(Sm,n)) (M may depend on n)
– CC: L contains MN items: all associations of words and models
n
m
DYNAMIC LOGIC ALGORITHM (DLA)
(non-combinatorial solution)

Start with a large body of text and unknown phrase-models
– any parameter values Sm
– associate fuzzy phrase-model with its contents (words)
– (1)
f(m|n) = r(m) l(n|m) /  r(m') l(n|m')
m'

Improve parameter estimation
– (2)
Sm = (1- a) Sm + a  f(m|n) [ll(n|m)/Mm]*[Mm/Sm]
n
• (a determines speed of convergence)
– learn word-contents of phrases (and grammatical relationships)

Continue iterations (1)-(2). Theorem: MFT is converging
- similarity increases on each iteration
- aesthetic emotion is positive during learning

Each phrase-concept-model is an agent
– semi-independent, interacting with other agents
• competing for evidence (among words)
– learning its properties and recognizing its words
CATCH ME BY HAND

If I give you enough time you’ll be able to catch me
on the previous slide
– the “bag”-model is non-differentiable
– this is a principal moment, learning non-differentiable models requires
sorting through combinations
– lead to combinatorial complexity
– differentiable models can be defined with a little trick
INTEGRATED
LANGUAGE AND THINKING

Where language and thoughts come together?
– concept-models have linguistic and objective aspects
– a fuzzy concept m has sensory and linguistic model
• Mm = { Mmsensory,Mmlinguistic };
– language and thoughts are fused at fuzzy pre-conscious level
• before concepts are learned

Understanding language and sensory data
– baby learning: “look, this is a car”
– each linguistic model has an empty “slot” for objects and
situations in surrounding world
– each sensory or situational model has an empty “slot” for a word
or phrase
– language participates in thinking and v.v.

Two types of information help learning and
understanding each other
– help associating signals, words, models, and behavior
INNER LINGUISTIC FORM
HUMBOLDT, the 1830s

In the 1830s Humboldt discussed two types of
linguistic forms
– words’ outer linguistic form (sounds) – a formal designation
– and inner linguistic form (???) – creative, full of potential

This remained a mystery for rule-based AI,
structural linguistics, Chomskyan linguistics
– rule-based approaches using the mathematics of logic make
no difference between formal and creative

In MFT and DLA there is a difference
– static form of learned (converged) concept-models
– dynamic form of fuzzy concepts, with creative learning
potential and emotional content
WHY MIND AND EMOTIONS?

A lot about the mind can be explained: concepts, instincts,
emotions, conscious and unconscious, intuition, aesthetic ability,…
– but, isn’t it sufficient to solve mathematical equations or to code a
computer and execute the code?

A simple yet profound question
–
–
–
–
–

the answer is in history and practice of science
Newton laws do not contain all of the classical mechanics
Maxwell equations do not exhaust radars and radio-communication
a physical intuition about the system is needed
an intuition about MFT was derived from biological, linguistic,
cognitive, neuro-physiological, and psychological insights into
human mind
Practical engineering applications of DLA require
biological, linguistic, cognitive, neuro-physiological,
and psychological insights in addition to mathematics
MFT THEORY OF MIND

MFT dynamics: elementary thought process
– a large number of model-concepts compete for incoming signals
– uncertainty in models corresponds to uncertainty in associations f(m|n)
– eventually, one model (m') wins a competition for a subset {n'} of input
signals w(n), when parameter values match object properties, and f(m'|n)
values become close to 1 for n{n'} and 0 for n{n'}
– upon convergence, the entire set of input signals {n} is divided into subsets,
each associated with one model-object
– fuzzy a priori concepts (unconscious) become crisp concepts (conscious)
• dynamic logic

Elementary thought process, consciousness and unconscious
– Aristotle: in thinking, an a priori form-as-potentiality (fuzzy model) meets
matter (signals) and becomes a form-as-actuality (a concept)
– Jung: conscious concepts are developed by mind based on inherited
structures of mind, archetypes, inaccessible to consciousness
– Grossberg: models attaining a resonant state (winning the competition for
signals and becoming crisp in this process) reach consciousness
MFT THEORY OF MIND
-understanding
Incoming signals, {x,w} are associated with model-concepts (m)
– creating phenomena (of the MFT-mind), which are understood as objects, situations,
phrases,…
– in other words signal subsets acquire meaning (e.g., a subset of retinal signals
acquires a meaning of a chair)

Several aspects of understanding and meaning
– concept-models are connected (by emotional signals) to instincts and to behavioral
models that can make use of them for satisfaction of bodily instincts
– an object is understood in the context of a more general situation in the next layer
consisting of more general concept-models (satisfaction of knowledge instinct)
• each recognized concept-model (phenomenon) sends (in neural terminology: activates) an
output signal
• a set of these signals comprises input signals for the next layer models, which ‘cognize’ more
general concept-models
• this process continues up and up the hierarchy towards the most general models: models of
universe (scientific theories), models of self (psychological concepts), models of meaning of
existence (philosophical concepts), models of a priori transcendent intelligent subject
(theological concepts)
– neural brain organization: individual modules, which form approximate hierarchies,
along with a number of “parallel” and “loop-like” pathways
SIGNS AND SYMBOLS
mathematical semiotics

Signs: stand for something else
– non-adaptive entities (mathematics, AI)
– brain signals insensitive to context (Pribram)

Symbols
– signs (mathematics, AI)
– psychological processes connecting conscious
and unconscious (Jung)
– brain signals sensitive to context (Pribram)
– processes of sign interpretation

Mathematics of symbol-processes
– relationships to thinking
– relationships to language
MFT THEORY OF SYMBOLS
-mathematical semiotics
Semiotics studies symbol-content of culture

Example: consider a written word "chair"
– It can be interpreted by mind to refer to something else: an entity in the
world, a specific chair, or the concept "chair" in the mind
– In this process, the mind, or an intelligent system is called an interpreter,
the written word is called a sign, the real-world chair is called a
designatum, and the concept in the interpreter's mind, the internal
representation of the results of interpretation is called an interpretant of the
sign
– The essence of a sign is that it can be interpreted by an interpreter to refer
to something else, a designatum

This is a simplified description of a thinking process, called semiosis
– its mechanism is given by the elementary thought process

Elementary thought process involving consciousness and unconscious,
concepts and emotions, is a dynamic symbol process
– a much more complicated entity than was originally envisioned by founders
of “symbolic AI”
MFT THEORY OF MIND
- aesthetic emotions and beauty 
Aesthetic emotions (not related to bodily satisfaction)
– Instincts for knowledge and language (learning concept-models)
– Emotions (satisfaction-dissatisfaction): harmony-disharmony
– Maximize similarity between models and world
• between our understanding of how things ought to be and how they actually are
in the surrounding world; Kant: aesthetic emotions

Beauty
– Harmony is an elementary aesthetic emotion; higher aesthetic emotions
• development of more complex “higher” models
– The highest forms of aesthetic emotion, beauty
• related to the most general and most important models
• models of the meaning of our existence, of our purposiveness or intentionality
• beautiful object stimulates improvement of the highest models of meaning
– Beautiful “reminds” us of our purposiveness
• Kant called beauty “aimless purposiveness”: not related to bodily purposes
• he was dissatisfied by not being able to give a positive definition
– knowledge instinct
• absence of positive definition remained a major source of confusion in
philosophical aesthetics till this very day
MFT THEORY OF MIND
- physical intuition 
Intuitive perception (imagination) of object-models and
their relationships with objects in the world
– involves fuzzy unconscious concept-models
– in process of being learned and adapted
• toward crisp and conscious models, a theory
– such models satisfy or dissatisfy the knowledge instinct before they
are accessible to consciousness, hence the complex emotional feel of
an intuition

Beauty of a physical theory discussed often by physicists
– related to satisfying our feeling of purpose in the world
– satisfying our need to improve the models of the meaning in our
understanding of the universe
REAL WORLD APPLICATIONS

Many applications have been developed
– Government
– Medical
– Commercial

Sensor signals processing and object recognition
– Variety of sensors

Internet search engines
– Based on text understanding

Financial market predictions
– Market crash on 9/11 predicted a week ahead
BACK UP
MFT
and Buddhism
MFT
vs. biology
Classical
methodology flowchart
MFT
flowchart
MFT
vs. inverse problems
MFT
predictions and testing
MFT
future directions
Publications
MFT AND BUDDHISM

Fundamental Buddhist notion of “Maya”
– the world of phenomena, “Maya”, is meaningless deception
– penetrates into the depths of perception and cognition
– phenomena are not identical to things-in-themselves

Fundamental Buddhist notion of “Emptiness”
– “consciousness of bodhisattva wonders at perception of emptiness
in any object” (Dalai Lama 1993)
– any object is first of all a phenomenon accessible to cognition
– value of any object for satisfying the “lower” bodily instincts is
much less than its value for satisfying higher needs, knowledge
instinct
– Bodhisattva’s consciousness is directed by knowledge instinct
– concentration on “emptiness” does not mean emotional emptiness,
but the opposite, the fullness with highest emotions related to the
knowledge instinct, beauty and spiritually sublime
MFT vs. BIOLOGY OF EYE
 Human eye is part of the brain
- Integrated sensor-processor system in both design and in operations
• multi-layer hierarchical system
• integrated adaptive optimized resource allocation
- Feedback from higher layers to lower layers is essential
• eye-brain neural pathway contains more feedback connections than feedforward
ones
 Adaptive, joint optimization of sensor-processor system network
- Based on a hierarchy with feedback among layers and modules
- What is the nature of this feedback?
 MFT hierarchy with feedback
- Every layer has 5 basic modules/elements:
(1) incoming signals (structured at lower layer, unstructured at the current layer)
(2) models: phenomenology (emissivity, geometry) and simulation codes
(3) similarity measure between signals and models
(4) adaptation mechanism
(5) outgoing signals (a structure: sign-concept)
CLASSICAL METHODOLGY
Result: Conceptual objects
Recognition
signals
MODELS/templates
•objects, sensors
•physical models
Sensors /
Effectors
Input: World/scene
MFT
basic two-layer hierarchy: signals and concepts
Result: Conceptual objects
Correspondence /
Similarity measures
signals
Attention / Action
Sim.signals
MODELS
•objects, sensors
•physical models
Sensors /
Effectors
signals
Input: World/scene
MFT VS. INVERSE SCATTERING

Inverse Scattering in Physics
– reconstruction of target properties by “propagating back” scattered fields
– usually complicated, ill-posed problems (exception: CATSCAN)

Biological systems (mind) solve this problem all the time
– by utilizing prior information (in feedback neural pathways)

Classical Tikhonov’s inversion cannot use knowledge
– regularization parameter () is a constant
– Morozov’s modification can utilize prior estimate of errors for 

Inverse Scattering using MFT
– can utilize any prior knowledge ( became an operator)
– utilizes prior knowledge adaptively ( depends on parameters)
MFT PREDICTIONS AND TESTING

General neural mechanisms of the elementary thought process
– confirmed by neural and psychological experiments
– includes neural mechanisms for bottom-up (sensory) signals, top-down
(“imagination”) model-signals, and the resonant matching between the two

Adaptive modeling abilities
– well studied: adaptive parameters are synaptic connections

Instinctual learning mechanisms
– studied in psychology and linguistics

Ongoing and future research will confirm, disprove, or suggest
modifications to
–
–
–
–
mechanisms of language and thinking integration
mechanisms of model parameterization and parameter adaptation
reduction of fuzziness during learning
similarity measure as a foundation of knowledge and language instincts
MFT FUTURE DIRECTIONS

Developing MFT models based on known linguistic
models

Differentiated forms of knowledge instinct
– highly differentiated emotions are involved in human
conversation and human thinking
– multiple measures of similarity, differentiated knowledge
instinct
• differentiated emotional concepts

Quantum Computing MFT devices
PUBLICATIONS
OXFORD UNIVERSITY PRESS
www.oup-usa.org