Transcript Robot Intelligence

Turing Test: Mindless Game?
Kevin Warwick
Classical/Traditional AI
• The branch of computer science concerned with
making computers behave like humans
• Getting a machine to do things that if a human
did them, you would regard them as an
intelligent act
• MIT AI – Minsky, McCarthy, Winston
• Webopedia says “Currently, no computers
exhibit full artificial intelligence (that is, are able
to simulate human behaviour)”
Turing’s prediction
• “I believe that in about fifty years’ time it
will be possible to programme computers
… to make them play the imitation game
so well that an average interrogator will
not have much more than a 70% chance
of making the right identification after five
minutes of questioning” (Turing, 1950).
•
TIG (Turing Test)
More Intelligent Machines?
• If we look at machine performance in the
Loebner competition over several years –
how much are machines improving?
• Will it be long before machines win at the
TIG?
Turing’s Imitation Game
• According to some TIG should be killed
because it offers nothing that furthers
understanding the complexity underlying
human intelligence and the nature of
consciousness.
Human Intelligence
• According to French (1990) “the test
provides a guarantee not of intelligence but
of culturally-oriented human intelligence”
• But Turkle (1997) clearly assigns
intelligence to machines “our general
tendency to treat responsive computers as
more intelligent …”
Turing & Intelligence
• Circumscribed issue by positing:
“Intelligent behaviour presumably consists
in a departure from the completely
disciplined behaviour involved in
computation, but rather a slight one, which
does not give rise to random behaviour, or
to pointless repetitive loops”
(Turing, section 7-Learning Machines, 1950)
AI’s Goals
• Hanard (1992):the Turing Test “sets AI’s
empirical goal” – it is not a mindless
parlour game.
Indistinguishability
• Turing’s Test affords a means to measure
“indistinguishability” (Turing, 1950,
Hanard, 2001) between human and
machine.
Loebner Prize
• A unique aspect of the situation can be
gleaned from the annual Loebner Prize for
Artificial Intelligence, a contest based on
the Imitation Game.
Loebner 2004-2006
• Format: parallel-paired comparison of
each of four hidden-machines against
each of four hidden-humans
• Task of each Interrogator, in four tests is to
identify the machine/human in each pair –
assigns a mark out of 100 to each (e.g.
Entry A 49/Entry B 51 – B is deemed to be
slightly more human than A)
Turing’s Interrogators
• Each year 4 new interrogators/judges are
selected
• Unique experience of the competition as a
Turing Interrogator twice, in 2001 and
2006
Judging behaviour
• Block (1981) reminds us that the role of
interrogators in Turing’s Test should be to
“avoid the problem of actually specifying
the behaviour or behavioural dispositions
thought to constitute intelligence”.
Negative Feature
• Downward trend, year on year, in the
highest score awarded to machines.
Downward Trend
2004
Machine’s
highest Score
from a Judge
48
Highest totals to
all 4 machines
from a Judge
123 – Judge 2
2005
45
100 – Judge 2
2006
28
97 – Judge 3
Loebner
Contest
Expectations high?
• Expectation of Loebner Judges may be
increasing, or the machines themselves
are getting worse, or …?
• No machine since 2003 has deceived any
Judge that it is human, whereas this was a
regular occurrence up to that time.
• Conversation for 25 minutes (not 5)
• What about networking???
Third Party
• Is the unknown entity a human or a
machine?
• Turing only considered view of
interrogator/judge
• Turing did not consider view of unknown
entities
• Turing did not consider a third party view
Third Party Views - 1
• I will now present some actual transcripts
from the Loebner competition
• In each case the Judge is a Human
• In the first transcript the unknown entity
(E) was a male human – the judge (J) felt
it/he was “probably” a machine
• 3rd party interviews (over 90%) thought the
entity was human but that the judge was a
machine
• E: Hello there. What has four legs and an arm?
• J: I don't know. Maybe a minotaur with a vitamin B
deficiancy.
• E: That's not the punchline I was expecting. I suspect
you know the real answer and you're just messing about.
What's your name by the way?
• J: My name is Adam. Some call me DJ Longman.
• E: That's nice. You can call me..er...James.
• J: What genre of music are you into?
• E: Serious music. Does that sound terribly pretentious?
• J: Have you heard about the DMC competition?
• E: I'm trying to work out what the TLA DMC might stand
for. I'm not getting very far. The C probably stands for
competition.
• J: It is the Deejay Mixing Competition. I hop eto do a
scratch set in it next year?
• E: Well, that's very good on your part. What exactly will
you have to do?
• J: you get five minuites to do a set based on the style of
music you are into.
• E: I see. And is the type of music you are into `serious'
music?
• J: What do you think? Weather it is hot or not I do not
know.
• E: I'm beginnig to suspect that you're ... how can I put
this politely? ... French. There, I've said. Sorry, but there
it is.
• J: No, I was just trying to catch you out.#
• E: Are we supposed to be playing cricket, or baseball, or
something like that?
• J: I am into basketball. What relevance does that have to
the last questionm?
• E: Very nearly none at all. Where do you play?
• J: I quite like air jordan trainers, especially because they
are limited edition. I played abit at school.
• E: Did you know that "abit" is Latin for "goes away"?
Third Party Views - 2
• In the second transcript the unknown
entity (E) was a machine (the 2003
winner) – the judge (J) felt it/he was
“definitely” a human
• 3rd party interviews (100%) thought the
entity was human and that the judge was a
machine
• E: Hello, anybody there? What's your name?
• J: I'm Ray who are you?
• E: Be welcome, Ray. You know who I am. Why the
charade?
• J: I do not know who you are- what is your name?
• E: My name is Gren
• J: What is that short for ?-or do you mean Glen?
• E: Yes, that's what I mean, isn't it clear?
• J: It is now - never mind we all make mistakes.
• E: OK, let's discuss something else. If you wish, of
course
• J: How did you get here today?
• E: I heard about by another fellow who works at the
university, and I was curious whether this robot freaks
could beat us humans or not
• J: So what do you think so far?
• E: I try not to think about things, actually. I don't know, it
just seems that I don't exist for thinking!
• J: So what are you really inerrested in ?
• E: Who already is really any-what?
• J: Try thinking a bit more please- you must be inerrested in
something!
• E: Thanks for asking politely. How did you hit on that? Anyway: Well,
that is your opinion, I suppose! Anyhow, I have bigger fish to fry.
• J: What is the weather where you are?
• E: There is no weather in cyberspace
• J: So you must be a machine are you sure that your name is not
Marvin
• E: What do you think about machines, Ray?
• J: At the moment not a lot but then like you it is best noy to think on
a sunny afternoon.
• E: Quite a bit, yes
• J: Are at last I thnk we have some common ground.
• E: Sure thing! Do you mean you and me?
• J: Yes
• E: Okay, I understand
• J: Good- I must say good-bye!
• E: Ok, Ray, just connect in if you need me!
• J:
• E: You want me to talk while you just sit there, huh? Well it won't
work
Third Party Views - 3
• In the third case the hidden entity was a
female human
• The judge thought the human was
definitely a machine
• 3rd party interviewees thought the entity
was definitely human but that the judge
was definitely a machine
• I have not labelled these cases – just A &
B
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
A: hello
B: hi where do you come from?
A: around here
B: what do you mean by that?
A: I mean I live nearby
B: how long have you lived here for?
A: Since 1978
B: does that mean you are 35 years old?
A: No, much oloder than that
B: where were you born?
A: In Wales
B: why did you come to live near here?
A: My parents moved to the south coast
B: which part of the country would you prefer?
A: You tell me which part you like best
B:
A: Which do you prefer - Brighton or Worthing?
B:
Problem:
• In contrast to Turing’s allusion to intelligent
behaviour, the Loebner machine entries
tend to be:
– Computationally disciplined
– Fooled by nuances
– Relatively random
– Repetitive
Human Intelligence
• Judges/3rd Parties can regard a human
entity as a machine when:
• They try to avoid a direct question
• Repeat themselves
• Respond too quickly
• Throw in a random point
• Do not answer
Conclusions
• The important problem of what exactly is
intelligence and how it works is not answered by
the Loebner competition.
• What we are left with are a series of subjective
scores from Judges.
• With TIG, are we moving the goalposts in what
we expect machines to achieve and/or are we
becoming more streetwise to machine utterings?
• Anyway isn’t TIG restricting machines to
communicate in the same poor way that humans
do? Why not humans/cows?
Comments - 2
• TIG is a lot about first impressions –
importance of only 5 minutes
• How much do we decide on an individual’s
intelligence in the first few minutes?
• How they look, what they wear, what
noises they make, what academic titles or
position they apparently have