Transcript Q-Learning

國立雲林科技大學
National Yunlin University of Science and Technology
Motivated Reinforcement Learning for
Non-Player Characters in Persistent
Computer Game Worlds
Advisor : Dr. Hsu
Presenter : Chia-Hao Yang
Author
: Kathryn Merrick, Mary Lou Maher
SIGCHI 06
1
Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Outline









Motivation
Objective
Introduction
Method
Experiments
Discussion
Conclusions
Habituation SOM
Q-learning
2
Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Motivation

Many NPC possess a fixed set of pre-programmed
behaviors and lack the ability to adapt and evolve in
time with their surroundings.
3
Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Objective

To create NPC that can both evolve and adapt with
their environmental.
4
Intelligent Database Systems Lab
Introduction

N.Y.U.S.T.
I. M.
Current technologies for NPCs
─
Reflexive agents

Only recognized states will produce a response


─
Learning agents

It can modify their internal structure to respect to some task.

─
State machines & rule-based approaches
EX : Baldur Gate & Dungeon Siege
Black and White
Reinforcement learning agents


The agent records the reward signal.
Then chooses an action which attempts to maximize the longrun sum of the values of the reward signal.

Tao Feng
5
Intelligent Database Systems Lab
Method

N.Y.U.S.T.
I. M.
Motivated reinforcement learning agents
─
─
It use a motivation function to directs learning.
Skill development is dependent on the agent’s environment & these
skills are developed progressively over time.
S(t-1) – S(t-2)
S(t) – S(t-1)
Q-learning
6
Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Experiments

In order to experiment with MRL agent, we implemented a village
scenario in Second Life.
─
Support character

Trades people



Location, object, inventory sensor
Move to object, pick up object, use object effector
Ex : the pick, when used on the mine, will produce iron which can converted to
weapons when used near the forge
7
Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Experiments
─
Partner character

Vendor character



Location, object sensor
Move to object effector
Ex : In Ultima Online players can set up vendor characters to sell the goods they have
crafted.
8
Intelligent Database Systems Lab
N.Y.U.S.T.
I. M.
Conclusions


This paper has presented MRL agents as a means of
creating non-player characters which can both evolve
and adapt.
MRL agents explore their environment and learn new
behaviors in response to interesting experiences,
allowing them to display progressively evolving
behavioral patterns.
9
Intelligent Database Systems Lab
Habituation SOM

N.Y.U.S.T.
I. M.
An HSOM consists of a standard Self-Organizing Map with an
additional habituating neuron connected to every clustering
neuron of the SOM.
─
─
10
Intelligent Database Systems Lab
Q-Learning


N.Y.U.S.T.
I. M.
It’s a part of reinforcement learning algorithm which has been widely
used for many applications such as robotics, multi agent system, game,
and etc.
It allows an agent to learn through training without teacher in unknown
environment.
─
─
Modeling the Environment
putting similar matrix name Q in the brain of our agent
11
reference
Intelligent Database Systems Lab
Q-Learning
─
algorithm
─
example
N.Y.U.S.T.
I. M.
……
12
reference
Intelligent Database Systems Lab