Transcript PPT - Ubiquitous Computing Lab

Intelligent Systems
Colloquium 3
Turing test and problem of testing
of Intelligence
Acting humanly: Turing Test
• Turing (1950) "Computing machinery and
intelligence":
• "Can machines think?"  "Can machines
behave intelligently?"
• Operational test for intelligent behavior: the
Imitation Game
Two versions of Turing test
• Susan Sterrett’s careful reading of Turing’s 1950 paper reveals a
significant distinction between two different versions of what has come to
be known as the Turing Test (Sterrett 2000). The first version, dubbed the
Original Imitation Game (OIG), appears on the very first page of
Computing Machinery and Intelligence (Turing 1950). The OIG has three
players: a man (A), a woman (B), and a third person (C) of either sex. The
third player (C) is called the interrogator, and his function is to
communicate with the other two, through what would nowadays be called
a text-only instant messaging chat interface, using two terminals (or today
perhaps, two windows) labeled (X) and (Y). The interrogator must decide
whether (X) is (A) and (Y) is (B), or (X) is (B) and (Y) is (A), in other words
which is the man and which is the woman. The interrogator’s task is
complicated by the man (A), who Turing says should reply to the
interrogator with lies and deceptions. For example, if the man is asked,
“are you a man or a woman?,” he might reply, “I am a woman.”
• Putting aside the gender and social issues raised by the OIG, consider the
OIG as an actual scientific experiment. Turing’s point is that if we were to
actually conduct the OIG with a sufficiently large sample of subjects
playing the parts of (A), (B), and (C), then we could measure a specific
percentage M of the time that, on average, the interrogator misidentifies
the woman, so that 100-M% of the time she is identified correctly. Given
enough trials of the OIG, at least in a given historical and cultural context,
the number M ought to be a fairly repeatable measurement.
Two versions of Turing test (2)
• Turing describes a second game more like the concept of a
“Turing Test” as most engineering schools teach it. The setup
is similar to the OIG, but now gender plays no role. The
player (B) is called “a man” and the player (A) is always a
computer. The interrogator must still decide whether (X) is
(A) and (Y) is (B), or (X) is (B) and (Y) is (A), in other words
which is the man and which is the machine. Sterrett calls this
second game the Standard Turing Test (STT).
• Unlike the OIG, the STT is not a good scientific experiment.
What does it mean to “pass” the STT? Must the interrogator
identify the machine correctly 50% of the time, or 100%? For
how long must the machine deceive the interrogator? Finally,
does the interrogator know in advance that he is trying to
“out”(Zdenek 2000) the robot, or that one of the players is a
machine at all?
• Unfortunately the STT, though flawed as an experiment, has
come to be popularized as the modern “Turing Test.” The
STT is the basis of real-world Turing Tests including the
Loebner Prize
Loebner Prize
• The Loebner Prize Medal and cash award is presented annually to the
designer of the computer system that best succeeds in passing a
variant of the Turing Test.
• The Loebner Prize Competition in Artificial Intelligence was established
in 1990 by Hugh Loebner and was first held at the Boston Computer
Museum, Boston, Massachusetts, USA in 1991. From 2001, the
Competition is to be run by the Science Museum, London.
• In accordance with the requirements of the Donor, as published in the
June 1994 Communications of the ACM, the winner of the US$100,000
Gold Prize must be prepared to deal with audio visual input, and
appropriate competitions will be held once Competitors have reached
Turing's 50:50 likelihood level of being mistaken for a human. An
intermediate Silver Prize of US$25,000 will be offered for reaching this
level in a text-only test. There is also an Annual Bronze Prize which is
awarded to the designer of the "most human computer" as rated by a
panel of Judges. In 2001, this prize will be worth $2,000
• Applications must be accompanied by e-mail protocols recording
interactions between the computer system to be entered and one or
more human subjects. Protocols must not exceed 2500
words. Applications must be submitted by e-mail to the Science
Museum. The Science Museum may request an opportunity to interact
with Computer Entries.
Searle’s Chinese room
Operator’s Manual (dictionary)
English language
Chinese language
She is understanding
Chinese language?
• Searle’s Chinese room provides a good
metaphor for thinking about A.L.I.C.E.-like
systems. Indeed the AIML contents of the
A.L.I.C.E. brain is a kind of “Chinese
Room Operator’s Manual.”
Learning and Turing test
• Some have argued that Turing, when he
predicted that a machine could play his game in
“50 years” after his 1950 paper, envisioned
something more like a general purpose learning
machine, which does not yet exist. The concept
is simple enough: build a robot to grow like a
child, able to be taught language the way we
are. In our terms, the role of the botmaster would
be fully automated. But even a child does not, or
at least should not, go forth into the world,
unprotected, to learn language “on the street,”
without supervision.
Problems of testing of AI
• Definition of Intelligence is absent. What is Intelligence?
It is possible three approaches to define it:
– Intelligence – knowledge (data or facts about world)
– Intelligence – capability to solve tasks (logical, mathematical,
searching, games and so on)
– Intelligence – capability to adapt and learn
• Human Intelligence is different too
– Different intelligence of different people in above 3 sense
• Different intelligence (in above 3 sense) of tested human
and testing human. So lack of understanding is possible
and relatively usual.
– Different knowledge (facts in memory), in particular, knowledge
about language
– Different knowledge about solving of tasks
– Different capabilities to adapt
Kinds of testing of AI
• So testing of AI is task of estimation of similarity between
behavior of Human and Computer (language-based test
is more simple and standartiziable)
• May be other tests, for example, learning to play in game
with previously unknown rules (http://www.a-i.com, 2001)
• For AI in mobile robots and animats may be other tests
with any movements, for example:
–
–
–
–
–
Navigation in labyrinth
Avoiding of any moving object (Gavrilov A.V., 1990)
Pursuit of any object
Searching of any objects in unknown or changing environment
Navigation in human-like complex and partly unknown
environment (DARPA, 2003)
– Different games, for example, soccer (from 1995,
http://www.robocup.org)
Learning to play in game with
previously unknown rules
• Challenge: Create a program that can win at a
game without knowing the game's rules
• Program will compete against the programs of
other contestants in simple turn-based games.
The rules of the games will not be explained
beforehand. Your program will be notified of its
opponent's previous move, and after your
program makes a move, it will be notified of a
win, loss, or draw. Games will consist of many
thousands of moves
• Turns of games are words from determined set
of words
• Program must to learn to get win as result of turn
Avoiding of any moving object
• In closed space any object is moving by
any rules (law)
• Task for robot is to avoid collisions or
nearness with this object
• Control system must to learn any behavior
corresponding to law of movement of
object
• Criterion of quality of control system may
be frequency of collisions or average
distance between object and system