Transcript Selection - Universitatea Politehnica din Bucuresti

Sisteme de programe
pentru timp real
Universitatea “Politehnica” din Bucuresti
2003-2004
Adina Magda Florea
http://turing.cs.pub.ro/sptr_2004
Curs Nr. 7 si 8
Genetic Algorithms
•
•
•
•
•
•
•
•
•
•
Introduction
Basic schema
GA Functioning
An example
Selection
Recombination
Mutation
Using GAs to solve the TSP
Parallel implementations of Gas
Co-evolution
2
1. Introduction

Genetic Algorithms (GAs) are adaptive heuristic search
algorithms
 Basic concept = simulate processes in natural systems
necessary for evolution - principles first laid down by Charles
Darwin of survival of the fittest.
 GAs provide an alternative method to solving optimization
problems (but not only)
GAs are useful and efficient when:
• The search space is large, complex or poorly understood.
• Domain knowledge is scarce or expert knowledge is difficult
to encode to narrow the search space.
• No mathematical analysis is available.
• Traditional search methods fail
3
Introduction - cont
Main schools of evolutionary algorithms:
 genetic algorithms, mainly developed in the USA by J. H.
Holland (1975)
 evolutionary strategies, developed in Germany by I. Rechenberg
and H.-P. Schwefel (1975-1980)
 evolutionary programming (1960-1980)
There are many ways to view genetic algorithms.
• Optimisation
tasks:
numerical
optimisation,
and
combinatorial optimisation problems such as traveling
salesman problem (TSP), circuit design, job shop scheduling
and video & sound quality optimisation.
• Automatic Programming: evolve computer programs for
specific tasks or design other computational structures, for
example cellular automata.
4
Introduction - cont
• Machine and robot learning: classification, prediction, protein
structure prediction, evolve rules for classifier systems or
symbolic production systems, and to design and control robots.
• Economic models: model processes of innovation, the
development of bidding strategies, and the emergence of
economic markets.
• Ecological models: biological arms races, host-parasite coevolutions, symbiosis and resource flow in ecologies.
• Population genetics models: study questions in population
genetics, such as “under what conditions will a gene for
recombination be evolutionarily viable?”
• Models of social systems: study evolutionary aspects of social
systems, such as the evolution of cooperation, the evolution of
communication, and trail-following behaviour in ants.
5
Introduction - cont
• GAs operate on a population of potential solutions applying the
principle of survival of the fittest to produce better and better
approximations to a solution.
• At each generation, a new set of approximations is created by
the process of selecting individuals according to their level of
fitness in the problem domain and breeding them together using
operators borrowed from natural genetics.
• This process leads to the evolution of populations of individuals
that are better suited to their environment than the individuals
that they were created from, just as in natural adaptation.
• GAs model natural processes, such as selection, recombination,
mutation, migration, locality and neighbourhood.
• GAs work on populations of individuals instead of single
solutions.
• In this way the search is performed in a parallel manner.
6
2. Basic schema
Problem
representation
Generate
initial
population
of individuals
(genes)
start
Fitness function
Evaluate
objective
function
Are optimization
criteria met?
no
Generate
new
population
yes
Selection
best
individuals
Crossover/Mate
result
offsprings
Mutation
generations
7
2.1 Problem solving using GAs
8
Problem solving using GAs - cont
Better results can be obtained by introducing many
populations, called subpopulations.
Every subpopulation evolves for a few generations
isolated (like the single population evolutionary
algorithm).
Then one or more individuals are exchanged between
the subpopulations.
The Multipopulation GAs models the evolution of a
species in a way more similar to nature than the
single population evolutionary algorithm.
9
3. GA Functioning
Initial Population
• A gene (individual) is a string of bits; some other
representations can be used
• The initial population of genes (bitstrings) is usually created
randomly.
• The length of the individual depends on the problem to be
solved
Selection (1)
• Selection means to extract a subset of genes from an existing
population, according to any definition of quality.
• Selection determines, which individuals are chosen for mating
(recombination) and how many offsprings each selected
individual produces.
10
3.1 Selection
The first step is fitness assignment by:
• proportional fitness assignment or
• rank-based fitness assignment
• Actual selection: parents are selected according to
their fitness by means of one of the following
algorithms:
• roulette-wheel selection
• stochastic universal sampling
• local selection
• truncation selection
• tournament selection [BT95].
11
Selection - cont
Proportional fitness assignment
 Consider the population being rated, that means: each gene
has a related fitness. The higher the value of the fitness, the
better.
 The mean-fitness of the population is calculated.
 Every individual will be copied as often to the new
population, the better it fitness is, compared to the average
fitness. E.g.: the average fitness is 5.76, the fitness of one
individuum is 20.21. This individuum will be copied 3 times.
All genes with a fitness at the average and below will be
removed.
 Following this steps, in many cases the new population will
be a little smaller than the old one. So the new population
will be filled up with randomly chosen individuals from the
old population to the size of the old one.
12
3.2 Reinsertion

If less offsprings are produced than the size of the original
population, the offsprings have to be reinserted into the old
population.
 Similarly, if not all offsprings are to be used at each
generation or if more offsprings are generated than needed
then a reinsertion scheme must be used to determine which
individuals should be inserted into the new population.
The selection algorithm used determines the reinsertion
scheme:
• global reinsertion for all population based on the selection
algorithm (roulette-wheel selection, stochastic universal
sampling, truncation selection),
• local reinsertion for local selection.
13
3.3 Crossover/Recombination
•
Recombination produces new individuals in combining the
information contained in the parents (parents - mating
population).
• Several recombination schemes exist
• The b_nX crossover - random mating with a defined
probability
This type is described most often as the parallel to the
Crossing Over in genetics
1. PM percent of the individuals of the new population will be
selected randomly and mated in pairs.
2. A crossover point will be chosen for each pair
3. The information after the crossover point will be exchanged
between the two individuals of each pair.
14
Crossover
15
3.4 Mutation
•
•
•
•
•
After recombination every offspring undergoes
mutation.
Offspring variables are mutated by small perturbations
(size of the mutation step), with low probability.
The representation of the variables determines the used
algorithm.
Mutation - add some effect of exploration to the
algorithm.
One simple mutation scheme:
– Each bit in every gene has a defined probability P to
get inverted
16
Mutation
17
The effect of mutation is in some way antagonist to selection
18
4 An example
Compute the maximum of a function
• f(x1, x2, ... xn)
• The task is to calculate x1, x2, ... xn for which
the function is maximum
• Using GA´s can be a good solution.
19
Example- Representation
1.
2.
3.
4.
•
•
Scale the variables to integer variables by multiplying them
with 10n, where n is the the desired precision
New Variable = integer(Old Variable ×10 n)
Represent the variables in binary form
Connect all binary representations of these variables to form
an individual of the population
If the sign is necessary two ways are possible:
Add the lowest allowed value to each variable and transform
the variables to positive ones
Reserve one bit per variable for the sign
Usually not the binary representation is used, but the Gray-code
representation
20
Example- Representation
21
Example- Computation
1.
2.
3.
4.
5.
Make random initial population
Perform selection
Perform crossover
Perform mutation
Transform the bitstring of each individual back to
the model-variables: x1, x2, ... xn
6. Test the quality of fit for each individual, e.g., the
f(x1, x2, ... xn)
7. Check if the quality of the best individual is good
enough (not significantly improved over last steps)
8. If yes then stop iterations else go to 2
22
5. Selection



The first step is fitness assignment.
Each individual in the selection pool receives a
reproduction probability depending on the own
objective value and the objective value of all other
individuals in the selection pool.
This fitness is used for the actual selection step
afterwards.
23
Terms
• selective pressure: probability of the best individual being
selected compared to the average probability of selection of all
individuals
• bias: absolute difference between an individual's normalized
fitness and its expected probability of reproduction
• spread: range of possible values for the number of offspring of
an individual
• loss of diversity: proportion of individuals of a population that
is not selected during the selection phase
• selection intensity: expected average fitness value of the
population after applying a selection method to the normalized
Gaussian distribution
• selection variance: expected variance of the fitness distribution
of the population after applying a selection method to the
normalized Gaussian distribution
24
5.1 Rank-based fitness assignment
 The population is sorted according to the objective values
(fitness).
 The fitness assigned to each individual depends only on its
position in the individuals rank and not on the actual objective
value.
 Rank-based fitness assignment overcomes the scaling problems
of the proportional fitness assignment = Stagnation in the case
where the selective pressure is too small or premature
convergence where selection has caused the search to narrow
down too quickly.
 The reproductive range is limited, so that no individuals
generate an excessive number of offsprings.
 Ranking introduces a uniform scaling across the population and
provides a simple and effective way of controlling selective
pressure.
 Rank-based fitness assignment behaves in a more robust
manner than proportional fitness assignment.
25
Rank-based fitness assignment - cont
• Nind - the number of individuals in the population
• Pos - the position of an individual in this population
(least fit individual has Pos=1, the fittest individual
Pos=Nind)
• SP - the selective pressure.
• The fitness value for an individual is calculated as:
Linear ranking
Fitness(Pos) = 2 - SP + 2*(SP - 1)*(Pos - 1) / (Nind - 1)
• Linear ranking allows values of SP in [1.0, 2.0].
26
Rank-based fitness assignment - cont
Non-linear ranking:
Fitness(Pos) =
Nind*X (Pos - 1) /  i = 1,Nind (X(i - 1))
• X is computed as the root of the polynomial:
0 = (SP - 1)*X (Nind - 1) + SP*X (Nind - 2) + ... + SP*X + SP
• Non-linear ranking allows values of SP in
[1.0, Nind - 2.0]
• The use of non-linear ranking permits higher selective
pressures than the linear ranking method.
• The probability of each individual being selected for
mating is its fitness normalized by the total fitness of
the population.
27
Rank-based fitness assignment - cont
Fitness assignment for linear and non-linear ranking
28
Rank-based fitness assignment - cont
Properties of linear ranking
Selection intensity: SelIntRank(SP) = (SP-1)*(1/sqrt()).
Loss of diversity: LossDivRank(SP) = (SP-1)/4.
Selection variance: SelVarRank(SP) = 1-((SP-1)2/ ) = 1-SelIntRank(SP)2.
29
5.2 Roulette wheel selection
The simplest selection scheme is roulette-wheel selection, also called stochastic sampling
with replacement
11 individuals, linear ranking and selective pressure of 2
Number
of individual
1
2
3
4
5
6
7
8
9
10
11
Fitness
value
2.0
1.8
1.6
1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0
Selection
probability
0.18
0.16
0.15
0.13
0.11
0.09
0.07
0.06
0.03
0.02
0.0
sample of 6 random numbers (uniformly distributed between 0.0 and 1.0):
• 0.81, 0.32, 0.96, 0.01, 0.65, 0.42.
30
Roulette wheel selection - cont
• After selection the mating population consists of the
individuals:
1, 2, 3, 5, 6, 9.
• The roulette-wheel selection algorithm provides a zero bias but
does not guarantee minimum spread
31
5.3 Stochastic universal sampling
• Equally spaced pointers are placed over the line as many as there are
individuals to be selected.
• NPointer - the number of individuals to be selected
• The distance between the pointers are 1/Npointer
• The position of the first pointer is given by a randomly generated
number in the range [0, 1/NPointer].
• For 6 individuals to be selected, the distance between the pointers is
1/6=0.167.
• sample of 1 random number in the range [0, 0.167]: 0.1.
32
Stochastic universal sampling - cont
• After selection the mating population consists of the
individuals:
1, 2, 3, 4, 6, 8.
• Stochastic universal sampling ensures a selection of offspring
which is closer to what is deserved then roulette wheel selection
33
5.4 Local selection
• Every individual resides inside a constrained environment
called the local neighbourhood
• The neighbourhood is defined by the structure in which the
population is distributed.
• The neighbourhood can be seen as the group of potential mating
partners.
• Linear neighbourhood: full and half ring
34
Local selection - cont
The structure of the neighbourhood can be:
• Linear: full ring, half ring
• Two-dimensional:
– full cross, half cross
– full star, half star
• Three-dimensional and more complex with any combination of the above
structures.
35
Local selection - cont
• The distance between possible neighbours together with the structure
determines the size of the neighbourhood.


The first step is the selection of the first half of the mating
population uniform at random (or using one of the other
mentioned selection algorithms, for example, stochastic
universal sampling or truncation selection).
Then a local neighbourhood is defined for every selected
individual. Inside this neighbourhood the mating partner is
selected (best, fitness proportional, or uniform at random).
36
Local selection - cont






Between individuals of a population an 'isolation by distance'
exists.
The smaller the neighbourhood, the bigger the isolation
distance.
However, because of overlapping neighbourhoods, propagation
of new variants takes place.
This assures the exchange of information between all
individuals.
The size of the neighbourhood determines the speed of
propagation of information between the individuals of a
population, thus deciding between rapid propagation or
maintenance of a high diversity/variability in the population.
A higher variability is often desired, thus preventing problems
such as premature convergence to a local minimum.
37
5.5 Tournament selection





A number Tour of individuals is chosen randomly from the
population and the best individual from this group is selected as
parent.
This process is repeated as often as individuals to choose.
The parameter for tournament selection is the tournament size
Tour.
Tour takes values ranging from 2 - Nind (number of individuals
in population).
Relation between tournament size and selection intensity
tournament size
1
2
3
5
10
30
selection intensity
0
0.56
0.85
1.15
1.53
2.04
38
Tournament selection - cont



Selection intensity:
SelIntTour(Tour) = sqrt(2*(log(Tour)-log(sqrt(4.14*log(Tour)))))
Loss of diversity:
LossDivTour(Tour) = Tour -(1/(Tour-1)) - Tour -(Tour/(Tour-1))
(About 50% of the population are lost at tournament size Tour=5).
Selection variance:
SelVarTour(Tour) = 1- 0.096*log(1+7.11*(Tour-1)), SelVarTour(2) = 1-1/
39
6. Crossover / recombination
• Recombination produces new individuals
in combining the information contained in
the parents
40
6.1 Binary valued recombination (crossover)

One crossover position k is selected uniformly at random and
the corrsponding parts (or variables) are exchanged between the
individuals about this point  two new offsprings are
produced.
41
Binary valued recombination - cont



Consider the following two individuals with 11 bits each:
individual 1:
01110011010
individual 2:
10101100101




crossover position is 5
After crossover the new individuals are created:
offspring 1:
0 1 1 1 0| 1 0 0 1 0 1
offspring 2:
1 0 1 0 1| 0 1 1 0 1 0
42
2.2 Multi-point crossover

m crossover positions ki
 individual 1:
01110011010
 individual 2:
10101100101
 cross pos. (m=3) 2 6 10
 offspring 1:
0 1| 1 0 1 1| 0 1 1 1| 1
 offspring 2:
1 0| 1 1 0 0| 0 0 1 0| 0
43
2.3 Uniform crossover







Uniform crossover generalizes single and multi-point crossover
to make every locus a potential crossover point.
A crossover mask, the same length as the individual structure,
is created at random and the parity of the bits in the mask
indicate which parent will supply the offspring with which bits.
individual 1:
01110011010
individual 2:
10101100101
mask 1: 0 1 1 0 0 0 1 1 0 1 0
mask 2: 1 0 0 1 1 1 0 0 1 0 1
(inverse of mask 1)
offspring 1:
11101111111
offspring 2:
00110000000
Spears and De Jong (1991) have poposed that uniform
crossover may be parameterised by applying a probability
to the swapping of bits.
44
2.4 Real valued recombination
2.4.1 Discrete recombination
 Performs an exchange of variable values between the
individuals.
 individual 1:
12 25 5
 individual 2:
123 4 34
 For each variable the parent who contributes its variable
to the offspring is chosen randomly with equal probability.
 sample 1: 2 2 1
 sample 2: 1 2 1
 After recombination the new individuals are created:
 offspring 1:
123 4 5
 offspring 2:
12 4 5
45
Real valued recombination - cont
Discrete recombination - cont
 Possible positions of the offspring after discrete
recombination

Discrete recombination can be used with any kind of
variables (binary, real or symbols).
46
Real valued recombination - cont
2.4.2 Intermediate recombination
 A method only applicable to real variables (and not binary
variables).
 The variable values of the offsprings are chosen
somewhere around and between the variable values of
the parents.
 Offspring are produced according to the rule:
offspring = parent 1 + Alpha (parent 2 - parent 1)
where Alpha is a scaling factor chosen uniformly at
random over an interval [-d, 1 + d].
 In intermediate recombination d = 0, for extended
intermediate recombination d > 0.
 A good choice is d = 0.25.
 Each variable in the offspring is the result of combining
the variables according to the above expression with a
47
Real valued recombination - cont
Intermediate recombination - cont
 Area for variable value of offspring compared to
parents in intermediate recombination
48
Real valued recombination - cont
Intermediate recombination - cont
 individual 1: 12 25 5
 individual 2: 123 4 34
Alpha are:
 sample 1: 0.5 1.1 -0.1
 sample 2: 0.1 0.8 0.5
 The new individuals are calculated


(offspring = parent 1 + Alpha (parent 2 - parent 1)
offspring 1:
67.5 1.9 2.1
offspring 2:
23.1 8.2 19.5
49
Real valued recombination - cont

Intermediate recombination is capable of producing any point
within a hypercube slightly larger than that defined by the
parents

Possible area of the offspring after intermediate recombination
50
Real valued recombination - cont
2.4.3 Line recombination
 Line recombination is similar to intermediate recombination,
except that only one value of Alpha for all variables is used.
 individual 1:
12 25 5
 individual 2:
123 4 34
Alpha are:
 sample 1: 0.5
 sample 2: 0.1
The new individuals are calculated as:
 offspring 1:
67.5 14.5 19.5

offspring 2:
23.1 22.9 7.9
51
Real valued recombination - cont
Line recombination - cont
 Line recombination can generate any point on the line defined
by the parents
52
7. Mutation
• After recombination offsprings undergo
mutation.
• Offspring bits or variables are mutated by
inversion or the addition of small random values
(size of the mutation step), with low probability.
• The probability of mutating a variable is set to be
inversely proportional to the number of variables
(dimensions).
• The more dimensions one individual has as
smaller is the mutation probability.
53
7.1 Binary mutation




For binary valued individuals mutation means flipping of
variable values.
For every individual the variable value to change is chosen
uniform at random.
A binary mutation for an individual with 11 variables, variable 4
is mutated.
Individual before and after binary mutation
before mutation
0
1
1
1
0
0
1
1
0
1
0
after mutation
0
1
1
0
0
0
1
1
0
1
0
54
7.2 Real valued mutation

Effect
of mutation


The size of the mutation step is usually difficult to choose. The optimal step
size depends on the problem considered and may even vary during the
optimization process. Small steps are often successful, but sometimes bigger
steps are quicker.
A mutation operator (the mutation operator of the Breeder Genetic
Algorithm) :
mutated variable = variable ± range·delta (+ or - with equal probability)
range = 0.5*domain of variable
delta = sum(a(i) 2-i), a(i) = 1 with probability 1/m, else a(i) = 0; m = 20.
55
Real valued mutation - cont



This mutation algorithm is able to generate most points in the
hypercube defined by the variables of the individual and range
of the mutation.
With m=20, the mutation algorithm is able to locate the
optimum up to a precision of (range*2-19)
Probability and size of mutation steps (compared to range)
56
8. Using GAs to solve the TSP
57
8.1 Problem representation
Evaluation function
• The evaluation function for the N cities is the sum of the
Euclidean distances
N
Fitness   ( xi  xi 1 ) 2  ( yi  yi 1 ) 2
i 1
Representation
• a path representation where the cities are listed in the order in
which they are visited
3 0 1 4 2 5
0 5 1 4 2 3
5 1 0 3 4 2
58
8.2 TSP Genetic operators
Crossover
• Traditional crossover is not suitable for TSPs
• Before crossover
1 2 3 4 5 0 (parent 1)
2 0 5 3 1 4 (parent 2)
• After crossover
1 2 3 3 1 4 (child 1)
2 0 5 4 5 0 (child 2)
• Greedy Crossover which was invented by Grefenstette in 1985
59
TSP Genetic operators - cont
Two parents 1 2 3 4 5 0 and 4 1 3 2 0 5
• To generate a child using the second parent as the template, we select city 4
(the first city of our template) as the first city of the child. 4 x x x x x
• Then we find the edges after city 4 in both parents: (4, 5) and (4, 1) and
compare the distance of these two edges. If the distance between city 4 and city
1 is shorter, we select city 1 as the next city of child 2. 4 1 x x x x
• Then we find the edges after city 1 (1, 2) and (1, 3). If the distance between
city 1 and city 2 is shorter, we select city 2 as the next city. 4 1 2 x x x
• Then we find the edges after city 2 (2, 3) and (2, 0). If the distance between
city 2 and city 0 is shorter, we select city 0. 4 1 2 0 x x
• The edges after city 0 are (0, 1) and (0, 5). Since city 1 already appears in child
2, we select city 5 as the next city. 4 1 2 0 5 x
• The edges after city 5 are (5, 0) and (5, 4), but city 4 and city 0 both appear in
child 2. We select a non-selected city, which is city 3. and thus produce a legal
child 4 1 2 0 5 3
We can use the same method to generate the other child. 1 2 0 5 4 3
60
TSP Genetic operators - cont
Mutation
• We can not use the traditional mutation operator. For example if
we have a legal tour before mutation 1 2 3 4 5 0, assuming the
mutation site is 3, we randomly change 3 to 5 and generate a new
tour 1 2 5 4 5 0
• Instead, we randomly select two bits in one chromosome and
swap their values.
• Swap mutation: 1 2 3 4 5 0
153420
Selection
• When using traditional roulette wheel selection, the best
individual has the highest probability of survival but does not
necessarily survive.
• We use CHC selection to guarantee that the best individual will
always survive in the next generation (Eshelman 1991).
61
8.3 TSP Behavior
• In CHC selection if the population size is N, we generate N
children by using roulette wheel selection, then combine the N
parents with the N children, sort these 2N individuals according to
their fitness value and choose the best N individuals to propagate
to the next generation.
Comparison of
Roulette and CHC selection
With CHC selection the population
converges quickly compared
to roulette wheel selection
and the performance is also better.
62
TSP Behavior - cont
• But fast convergence may mean less exploration.
• To prevent convergence to a local optimum, when the population
has converged we save the best of the individuals and re-initialize
the rest of the population randomly.
• We call this modified CHC selection R-CHC.
Comparison of CHC selection
with or without re-initialization
63
9 Parallel implementations of GAs
• Migration
• Global model - worker/farmer
• Diffusion model
64
9.1 Migration
• The migration model divides the population in multiple
subpopulations.
• These subpopulations evolve independently from each
other for a certain number of generations (isolation
time).
• After the isolation time a number of individuals is
distributed between the subpopulations (migration).
• The number of exchanged individuals (migration rate),
the selection method of the individuals for migration and
the scheme of migration determines how much genetic
diversity can occur in the subpopulations and the
exchange of information between subpopulations.
65
Migration - cont
• The parallel implementation of the migration
model showed not only a speedup in computation
time, but it also needed less objective function
evaluations when compared to a single
population algorithm.
• So, even for a single processor computer,
implementing the parallel algorithm in a serial
manner (pseudo-parallel) delivers better results
(the algorithm finds the global optimum more
often or with less function evaluations).
66
Migration - cont
• The selection of the individuals for migration can take
place:
– uniform at random (pick individuals for migration in a random
manner),
– fitness-based (select the best individuals for migration).
• Many possibilities exist for the structure of the
migration of individuals between subpopulations. For
example, migration may take place:
– between all subpopulations (complete net topology unrestricted)
– in a ring topology
– in a neighbourhood topology
67
Migration - cont
Unrestricted migration topology (Complete net
topology)
68
Migration - cont
Scheme for migration of individuals between
subpopulations
69
Migration - cont
Ring migration topology
Neighbourhood
migration
topology (2-D
implementation)
70
9.2 Global model - worker/farmer
• The global model employs the inherent
parallelism of genetic algorithms (population of
individuals). The Worker/Farmer algorithm is a
possible implementation.
71
• The diffusion
model handles
every individual
separately and
selects the
mating partner in
a local
neighbourhood
similar to local
selection. Thus, a
diffusion of
information
through the
population takes
place. During the
search virtual
islands will
evolve.
9.3 Diffusion model
72
9.2 Global model - worker/farmer
• The selection of the individuals for migration can take
place:
– uniform at random (pick individuals for migration in a random
manner),
– fitness-based (select the best individuals for migration).
• Many possibilities exist for the structure of the
migration of individuals between subpopulations. For
example, migration may take place:
– between all subpopulations (complete net topology unrestricted)
– in a ring topology
– in a neighbourhood topology
73
10 Coevolution
A multi-agent system with competitive
agents that tries to solve the “tragedy of
commons” problem:
 the rational exploitation of natural
renewable resources by self-interested
agents (human or artificial) that have to
achieve a certain degree of cooperation in
order to avoid the overexploitation of
resources.
74
10.1
Problem Description
“Tragedy of commons” instances:
 use
of common pastures by sheep farmers
 fish and whales catching
 environmental pollution
 urban transportation problems
 preservation of tropical forests
 common computer resources used by different processors
The problem instance considered:
 Fishing companies (Ci) owing several ships (NSi) and having the
possibility to fish in several fishing banks (Rp), during different
seasons (Tj).
 Each company aims at collecting maximum assets expressed by
the amount of money they earn (and the number of ships). The
money is generated by fish catching at fish banks.
 A fish bank is a renewable resource and will not be regenerated if
the companies will be catching too much fish, leading to the
exhaustion of the resource, ecological unbalance, and profit loss.
75
10.2
The MAS model
Company agents (CompA)



A company is represented by a cognitive agent following the BDIG
model
A CompA may be ecologically oriented or profit oriented (fix the
goals)
A CompA have a planning component to develop plans for sending
ships to fish banks (in shore or deep sea), under the uncertainty
induced by the fishing actions of the other company agents.
Environment agent (EnvA)



The attributes of the environment
The evolution of the environment status
Each company in the system has to register at the EnvA (EnvA is
also a facilitator in the MAS)
76
Company 1 Agent
Self representation
 World representation
Beliefs,
Goals, Plans,
Request
Negotiation Strategy, Company 3 Agent
Ships, Profile
Accept

ModifyRequest
Request
Result
Fish banks, Declare
Seasons,
Inquiry
Regeneration limit/rate,
Names of other agents,
Beliefs about other agents
Environment Agent
Inform
Companies
 Fish banks
 Environment state

Company 2 Agent
Multi-agent system to model the “tragedy of commons”
77
Agent interactions
Communication between CopmAs and EnvA is aimed at
interacting with and acquiring knowledge on the environment.
 Declare(Fishbank, NoShips, Fc, Time)
 Result(Fishbank, Amount, Time)
 Inquiry(Fishbank, Time)
 Inform(Fishbank, Amount, Time, List)
Communication between one CompA and another CompA is
dedicated to negotiate the fishing amount that will be caught at a
particular time moment by a company agent.
 Request(Fishbank , Amount, Time)
 Accept(Fishbank, Amount, Time)
 Reject(Fishbank , Amount, Time)
 ModifyRequest(Fishbank , Amount, Time)
 Declare(Fishbank, NoShips, Fc, Time)
78
10.3 Genetic representation of plans
The generation of plans of a company is achieved by means of
a GA with genetic coadapted components.
Cooperative Coadapted Species (Potter and De Jong, 2000):

A subcomponent evolves separately by a genetic algorithm, but the
evolution is influenced by the existence of the other subcomponents in
the ecosystem.
 Each species is evolved in its own population and adapts to the
environment through the repeated application of a genetic algorithm.
The influence of the environment on the evolution, namely the
existence of the other species, is handled by the evaluation function of
the individuals, which takes into account representatives from other
species. Thus, the interdependency among subcomponents is
achieved by evolving the species in parallel and evaluating them within
the context of each other.
79
Population of
Company i
Population of
Company 1
Individual1 of Ci
...
Best of C1
Individualj of Ci
...
Individualn of Ci
Population of
Company M
Evaluate individuals
of Company i
Best of CM
Genetic coadapted components to model the companies
80





The fitness of an individual in a population is computed based on its
configuration and on representatives chosen from the other
populations in the ecosystem.
From each population the representative is chosen as the “best”
individual of the population, with two possible interpretations for what
“best” means in the context of a subspecies.
If the company is profit oriented, the best individual is the one that
will bring the maximum profit.
For ecologically oriented companies, the representative is the
individual that will bring maximum profit, while keeping the minimum
amount of fish that allows regeneration. If the profile of the company
is not known, an ecologically profile is assumed by default.
The fitness of the individual is evaluated in the context of the selected
representative and is based on the profit, taking care of the ecological
balance (individuals that do not ensure the minimum regeneration
amount are assigned 0 fitness) or not, depending on the company
profile.
81
Company i
Ship 1
Ship 2
Ship NSi
First representation
(NoBPlace + NoBSeason) * NoShips
Season: T1, T2, T3, T4
Place at Ti: H, R1, R2, R3
Company i
T1
T2
Place of Ship NSi: H, R1, R2, R3
Place of Ship 1: H, R1, R2, R3
Tn
Second representation
NoBPlace * NoShips * NoSeasons
82
10.4 Genetic Model Utilization in
MAS
The genetic model of the ecosystem may be used in
different ways.
 A CompA builds its own plan by genetic evolution
while keeping the ecological balance: it models himself
and the way it believes the other CompAs will act.
 Replanning, after seasons T1..Ti have passed, by the
same approach as above.
Considering the difficult problem of distributed planning and dynamical
replanning, the genetic model offers a simple solution to develop a plan for
sending ships to different places, or, as more knowledge about the evolution of
the world is gathered, to rebuild the plan adequately.
83
 A CompA may keep alternate “good plans” by
selecting the first x “best plans” from genetic
evolution, for negotiation with other CompAs.
 A CompA may model the evolution of the world (the
other CompAs action plans) in case it performs
symbolic distributed planning.
In this case, the criteria for selecting the preferred
solution from several runs are somehow different.

When selecting the presumed course of actions of the
other agents from the several runs of the algorithm, the
agent will prefer the one which brings maximum profit to
the society and, if several such solutions exit, the one
that will bring maximum gain to the profit oriented
companies.

84
 For negotiation, when a CompA has a plan:
Once the CompA has developed a plan, it will use this
plan as the fixed description of the self subspecies in the
population and run the genetic algorithm with this
description. The evolved model of the world and its own
strategy will lead the negotiation.

The agent will also prefer the other agents plans which
brings maximum profit to the society (and maximum
gain to the profit oriented companies).

The negotiation starts either in case the CompA
detects that the ecological balance is threaten by the
actions of the other agents, or when it would like to fish
more in a particular fishing bank and it detects there will
be not enough fish left at a particular time moment.

85
More on negotiation
 A cooperation profile of the other agents in the system
is built to guide the negotiation strategy.
 The cooperation profile is defined by a set of attributes
and is updated constantly while interacting with the other
company agents. The cooperation profile of another agent x
as developed by the current agent A contains the following
attributes:
No of requests accepted by x
 No of requests rejected by x
 No of requests modified by x
 No of x’s requests accepted by A
 No of x’s requests rejected by A

86
 The genetic model may be used by the EnvA also
If we endow the EnvA with more cognitive capabilities,
then it will evolve a model of the world for which it acts
as the environment and will start negotiating with the
company agents in the same way these agents are
negotiating among them.

 Therefore the agents in the MAS are hybrid:
cognitive as they have mental attitudes,
negotiation strategies, and symbolic planning,
and genetic, as they use the coadapted species
genetic model in different ways.
87
10.5
Experimental results
The planning process was tested for several
situations, with different GA and EA parameters.
We present results for 5 fishing seasons, three fishing
banks (R1, R2, R3) and in-port (H), and 3 companies.
The fitness of an individual was computed as 95% profit
and 5% preservation of resources, with a 0 fitness
value if at least one resource goes beyond the
regeneration level.
88
Genetic Algorithm's parameters:
 Two-point crossover rate: 0.6
 Probability of mutation in every individual: 0.1
 Population size: 100
 Length of an individual: 100
 The selection is based on the stochastic remainder
technique
 Number of generations: 20..200
Evolutionary Algorithm’s parameters:
 Crossover rate: 0
 Probability of mutation in every individual: 0.05..0.50
 Population size: 100
 Length of an individual: 100
 Number of generations:150
89
RUN 1
Company 1
Season 1
Season 2
Season 3
Season 4
Season 5
Fitness value
Company 2
Season 1
Season 2
Season 3
Season 4
Season 5
Fitness value
Company 3
Season 1
Season 2
Season 3
Season 4
Season 5
Fitness value
H
R1
R2
3
1
5
2
3
3
1
3
2
2
0
3
0
3
3
0.836166666666
H
R1
R2
2
4
2
1
4
2
1
1
2
3
2
2
0
2
3
0.855166666666
H
R1
R2
1
4
3
3
4
0
2
5
3
1
1
5
2
3
2
0.817166666666
R3
1
2
4
5
4
R3
2
3
6
3
5
R3
2
3
0
3
3
RUN 2
Company1
Season 1
Season 2
Season 3
Season 4
Season 5
Fitness value
Company2
Season 1
Season 2
Season 3
Season 4
Season 5
Fitness value
Company 3
Season 1
Season 2
Season 3
Season 4
Season 5
Fitness value
H
R1
R2
2
2
1
1
3
3
0
3
3
3
4
1
1
3
0
0.856229166666
H
R1
R2
2
3
1
3
3
1
4
2
1
2
3
4
3
2
3
0.730333333333
H
R1
R2
2
4
1
3
2
2
1
2
3
2
0
5
1
0
5
0.826041666666
R3
5
3
4
2
6
R3
4
3
3
1
2
R3
3
3
4
3
4
Results of genetic plan evolution for
3 companies and 2 runs
90
1
Run 1
Run 2
Run 3
Average
0.9
0.8
Fitness
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
20
30
40
50
60
70
80
90
100 110 120 130 140 150 160 170 180 190
Generations No.
Average best fitness of genetic plans
depending on the number of generations
(the average is for 5 runs while only the results of the first three are shown)
91
Evolutionary Algorithm
(150 Generations)
Run 1
Run 2
Run 3
Average
0.900
0.800
0.700
Fitness
0.600
0.500
0.400
0.300
0.200
0.100
0.000
0.05
0.10
0.15
0.20
0.25
0.30
0.35
0.40
0.45
0.50
Mutation Rate
Average best fitness of evolutionary plans
depending on mutation rate
(the average is for 5 runs while only the results of the first three are shown)
92
10.6
Conclusions

We have presented a multi-agent system with competitive
agents that tries to solve the problem of the “tragedy of
commons”
 The self-interested agents in the system have to achieve
a certain degree of cooperation in order to avoid the
overexploitation of resources and, in this way, to ensure
individual and global welfare.
 The multi-agent system proposed is based on agents
which are capable of developing plans, of interactions with
the other agents in the system and with the environment.
The agents are hybrid, containing cognitive components
and a genetic representation of their planned actions and
of those of the other agents in the society.
93
Conclusions



We have shown that the model of coadapted subspecies is
also suited to represent competitive subcomponents
that should collaborate as they are living in a common
environment and they are sharing common resources.
As opposed to other multi-agent approaches for the
“tragedy of commons” problem, in which the ecological
balance is kept by a centralised authority,
the MAS solution we propose is inherently distributed.
The CompAs are the active, distributed elements that are
trying to achieve their goals and to keep the adequate
level of renewable resources, while the EnvA is an agent
which updates the environment status and informs about
the changes in the world around.
94