CS 591

Topics in Complex Adaptive Systems Spring Semester, 2006 Genetic Algorithms Stephanie Forrest FEC 355E
http://cs.unm.edu/~forrest/cas-class-06.html forrest@cs.unm.edu 505-277-7104

Genetic Algorithms
Principles of natural selection applied to computation:
Variation Selection Inheritance

Evolution in a computer:
Individuals (genotypes) stored in computers memory Evaluation of individuals (artificial selection) Differential reproduction through copying and deletion Variation introduced by analogy with mutation and crossover

Simple algorithm captures much of the richness seen in naturally evolving populations.

A Simple Genetic Algorithm

Population at Tn

Population at T n+1 Mutation

00111 11100 01010 ...

F(00111) = 0.1 F(11100) = 0.9 F(01010) = 0.5 Selection

11100 11100 01010 ...

Crossover

01100 11010 01100 ...

Fitness
50 60 70 80 90 100

10

20

30

40

0 20 40 60 80 10 0 12 0 14 0 16 0 18 0 20 0 22

mean fitness

Example Performance Curve

Generation

max fitness

0 24 0 26 0 28 0 30 0 32 0 34 0 36 0 38 0 40 0 42 0 44 0 46 0 48 0 50 0 52 0 53 5

Multi-parameter Function Optimization

F(x, y) = yx2 - x4 Degray 001 001 1 1 1 1 Bit string (Gray coded) 1 0 1 Base 2 5 Base 10

Decimal 0 1 2 3 4 5 6 7

Binary Gray code 000 001 010 011 100 101 110 111 000 001 011 010 110 111 101 100

F(0001111) = F(1,5) = 5 12-14 = 4

Example Applications of Genetic Algorithms

Engineering applications:
Multi-parameter function optimization (e.g., spectroscopic applications, turbine engine design). Sequencing problems (e.g., circuit design, factory scheduling, TSP). Machine and robot learning. Complex data analysis. Automatic programming (e.g., genetic programming).

Modeling:
Rule discovery in cognitive systems. Learning strategies for games. Affinity maturation in immune systems. Ecosystem modeling.

Genetic Programming
Evolve populations of computer programs:
Typically use the language Lisp. Select a set of primitive functions for each problem. Represent program as a syntax tree.

Function approximation vs. function optimization. Crossover operator:

Exchange subtrees between individual program trees. Schema Theorem?

Many applications:
Optimal control (e.g., the pole balancing problem) Circuit design Symbolic regression (data fitting)

Genetic Programming
Expression LISP (= (* x x) (*3 x y) (* y y)) X2 = 3xy = y2

*
x x 3

*
x y y

*
y

Genetic Programming cont.

Consider evolving a program to compute: cos 2x A human-designed program: 1 - 2sin2x In Lisp: (-1 (* 2 (* sin (sin x)))) A genetic programming solution: (sin ( - ( - 2 (* x 2)) (sin (sin (sin (sin (sin (sin ( * (sin (sin 1)) (sin (sin 1))))))))))) Junk DNA?

Where did these ideas originate?

Genetic algorithms (Holland, 1962)
Original idea was to create an algorithm that captured the richness of natural adaptive systems. Emphasized the adaptive properties of entire populations and the importance of recombination mechanisms such as crossover. Application to function optimization introduced by DeJong (1975).

Evolutionstrategie (Rechenberg, 1965)

Emphasized the importance of selection and mutation, as Mechanisms for solving difficult real-valued optimization problems.

Evolutionary programming (Fogel et al., 1966)

Emphasis on evolving finite state machines.

Genetic programming (Koza, 1992) ==> Evolutionary Computation

References
J. H. Holland Adaptation in Natural and Artificial Systems. Univ. of Michigan Press (1975). Second Edition published by MIT Press (1992). D. E. Goldberg Genetic Algorithms in Search, Optimization, and Machine Learning. Addison-Wesley (1989). M. Mitchell An Introduction to Genetic Algorithms. MIT Press (1996). S. Forrest Genetic Algorithms: Principles of natural selection applied to computation. Science 261:872-878 (1993). J. Koza Genetic Programming. MIT Press (1992).

How do Genetic Algorithms Work?

The Central Dogma of Genetic Algorithms
(Holland, 1975)

Schema processing Schema Theorem Implicit parallelism Building block hypothesis K-armed bandit analogy

Genetic Algorithms and Search

High-dimensional search spaces:
All binary strings of length l. All possible strategies for playing the game of chess. All possible tours in the Travelling Salesman Problem.

Genetic algorithms use biased sampling to search high-dimensional spaces:

Independent sampling. Selection biases search towards high-fitness regions. Crossover combines partial solutions from different strings.

Partial solution formalized as schema.

*1** *0**

1*** 0***

Schemas
Schemas capture important regularities in the search space: 1 0 0 1 1 1 0 1 0 0 1 1 * * * * 1 1 * * 0 * * * * * 0 * * 1 * * 0 * 1 1 Implicit Parallelism: 1 individual samples many schemas simultaneously. Schema Theorem: Reproduction and crossover guarantee exponentially increasing samples of the observed best schemas. Order of a schema O(s) = number of defined bits. Defining length of a schema D(s) = distance between outermost bits.

Schema Theorem
(Holland, 1975)
Let: s be a schema in population at time t,

N(s,t) be the number of instances of s at time t. Question: What is the expected N(s,t+1) ? Assume: Fitness-proportionate selection. F ( x) Expected number of offspring(x) = F (t ) Ignoring crossover and mutation,

( s, t ) N ( s, t + 1) = N ( s, t ) F (t )
Note: If
( s, t ) = c , then F (t )

N ( s, t ) = c t N ( s,0)

Crossover and mutation handled as loss terms:

( s, t ) D( s) N ( s, t + 1) = N ( s, t )(1 pc )[(1 pm ) O ( s ) ] l 1 F (t )

Royal Road Schemas

schema 1
100 90 80 70

schema 2

schema 3

Fitness

60 50 40 30 20 10 0
0 20 40 60 80 100 120 140 160 180 200 220 240 260 280 300

Generation

Study Question
Given a population, consisting of
N individuals, Each individual is L bits long,

How many schemas are sampled by the population (in one generation)? Hint:
Minimum value is: 2L Maximum value is Nx2L

Building Blocks Example

Fitness

Genome
0.0

Interpret bit string as a binary decimal

1.0

.1000 = 0.5 1#### 0.5 0#### 0.5

.01000 = 0.25 #1### = even intervals #0### = odd intervals

Questions
When will genetic algorithms work well and when they will not?
Not appropriate for problems where it is important to find the exact global optimum. GA domains are typically those about which we have little analytical knowledge (complex, noisy, dynamic, poorly specified, etc.). Would like a mathematical characterization that is predictive.

What makes a problem hard for genetic algorithms?

Deception, multi-modality, conflicting schema information, noise, ...

What makes a problem easy for genetic algorithms? What distinguishes genetic algorithms from other optimization methods, such as hill climbing? What does it mean for a genetic algorithm to perform well?

Building Blocks Hypothesis

(Holland, 1975)
1. GA initially detects biases in low-order schemas:
GA obtains good estimates of schema average fitness by sampling strings.

2. Over time, information from low-order schemas is combined through crossover, and 3. GA detects biases in high-order schemas, 4. Eventually converging on the most fit region of the space. Implies that crossover is central to the success of the GA. Claim: GA allocates samples to schemas in a near optimal way:
K-armed bandit argument.

The Two-armed Bandit

Originated in statistical decision theory and adaptive control. Suppose a gambler is given N coins with which to play a slot machine with two arms (A1 and A2). The arms have:
Mean payoff (per trial) rates m1 and m2 Variances s12 and s22

The payoff processes from the two arms are each stationary and independent of one another. The gambler does not know these payoffs and can estimate them only by playing coins on the different arms.

Two-armed Bandit cont.

Given that the player wants to maximize total payoff, what strategy is best?
On-line payoff (maximize payoff during N trials) vs. Off-line payoff (determining which arm has the higher payoff rate).

Claim: Optimal strategy is to exponentially increase the sampling rate of the observed best arm, as more samples are collected. Apply to schema sampling in GA:
3L schemas in an L-bit search space can be viewed as 3L arms of a multiarmed slot machine. Observed payoff of a schema H is simply its observed fitness. Claim: The GA is a near optimal strategy for sampling schemas. Maximizes on-line performance.

Most GA theory is not practical or predictive, except in a general way.

Analysis methods:
Walsh polynomials and schema analysis Population genetics models Markov chains Signal-to-noise analysis Statistical structure of fitness landscapes Statistical mechanics models PAC learning

Common traps:
Infinite populations. Analysis intractable except for short strings (16 or less). Enumerate all possible populations. Convergence proofs based on idea that any string can mutate into any other string. Weak bounds.

Implementation Issues

Implementation issues Permutation problems and special operators:

Example: TSP

Genetic programming:
Example: Trigonometric functions

Modeling applications:
Example: Prisoners Dilemma Example: Classifier Systems Example: Echo

Implementation Issues
Data structures:
Packed arrays of bits Byte arrays Vectors of real numbers Lists and trees Feature lists Binary encodings, gray codes Real numbers Permutations Trees

Crossover:
1-point, 2-point, n-point Uniform Special operators

Mutation:
Bit flips Creep (Gaussian noise)

Representation:

Elitism Parameters (rough guidelines):

Bitstring length (32 - 10,000) Population size (100 - 1000) Length of run (50 - 10,000) Crossover rate (0.6 per pair) Mutation rate (0.005 per bit)

Selection
On next slide

Scaling
On next slide

Selection Methods
Fitness-proportionate (used in theoretical studies):
i Expected value of individuali: f exp(i ) = f Implement as roulette wheel:
8% 0% 20%

17% 55%

rank ( i ) 1 Rank-based: f exp (i) = Min + ( Max Min) N 1

(SKIP)

Intended to prevent premature convergence (slow down evolution). Each individual ranked according to fitness. Expected value depends on rank. Min, Max are constants.

Selection Methods cont.

Tournament:
Computationally efficient. Choose T = size of tournament (2 is a common value). Pick subsets of size T from the population randomly (with replacement). Compare fitnesses within the subset and choose the winner (either deterministically or stochastically). Iterate.

Steady state. Implicit (e.g., Echo, immune models, etc.)

Reproduction rate proportional to resources obtained.

Scaling Methods How to maintain a steady rate of evolution? (SKIP)

Linear: Based on max, min, mean: ' ' f f = af + b , such that avg = f avg
Described in Goldberg, 1989.

Sigma: Based on mean and standard deviation

f ' = f ( f c )

with cutoffs for extreme values. c is typically 1.5 or 2.0

Exponential:
f' =kf
Normalizes difference between 1.0 and 1.5, and 1000.0 and 1000.5.

Permutation Problems and Special Operators

Problem: Find an optimal ordering for a sequence of N items. Examples:
Traveling Salesman Problem (TSP) Bin packing problems Scheduling DNA fragment assembly

Traveling Salesman Problem: 6

1
2

3 6 2 5

Using Genetic Algorithms to Find Good Tours for TSP

Natural representation: Permutations in which each city is labeled by a unique integer (bitstring):

3 2 1 4 4 1 2 3
Problem: Mutation and crossover do not produce legal tours:

3 2 2 3
Solutions:
Other representations. Other operators. Penalize illegal solutions through fitness function.

Specialized Operators
What information (schemas) should be preserved?
Absolute position in the sequence Relative ordering in the sequence (precedence) Adjacency relations

How much randomness is introduced? Order crossover (Davis, 1985) Partially-mapped crossover (PMX) (Goldberg, et al. 1985) Cycle crossover (Oliver, et al. 1987) Edge-recombination:
Try to preserve adjacencies in parents Favor adjacencies common to both parents When 1,2 fail, make a random selection.

Edge Recombination Operator

Algorithm:
1. Select one parent at random and assign the first element in its permutation to be the first one in the child. 2. Select the second element for the child, as follows: If there is an adjacency common to both parents, then choose that. If there is an unused adjacency available from one parent, choose it. If (1) and (2) fail, then pick an adjacency randomly. 3. Select the remaining elements in order by repeating step 2.

Edge Recombination Example

362145 521364 Original Individuals

Key 1 2 3 4 5 6

364125 New Individual

Adjacent Keys 2,2,3,4, 1,1,3,6, 1,6,6, 1,5,6 2,4 2,3,3,4

Evolution of Cooperation
(Axelrod, 1984)

What is the Prisoners Dilemma?

Player B Cooperate Player A Cooperate Defect 3,3 5,0 Defect 0,5 1,1

Non-zero sum game:

Total number of points is not constant for each configuration.

What is the best move for Player 1? Player 2?

Prisoners Dilemma cont.

Iterated Prisoners Dilemma:

Play game for an indeterminate number of steps. Raises possibility of strategies based on the behavior of the other player.

What are the possible strategies?

RANDOM Always cooperate, always defect Tit-For-Tat Model other player and try to maximize against that model. etc.

Evolution of Cooperation
~1980 Two tournaments:
Each strategy (entry) encoded as a computer program. Round robin (everyone plays everyone). Tit-For-Tat (TFT) won both times.

1982-85 Learning algorithms:

Can TFT evolve in a competitive environment? How could a TFT strategy evolve in nature? Is there a better strategy? How do strategies develop?

Use genetic algorithm to study these questions,

In environment of tournament. In a changing environment (co-evolution). In a 3-person game. In an n-person game. In a Norms game.

Prisoners Dilemma Using Genetic Algorithm

Population consists of individual strategies:
1 strategy per chromosome. Each chromosome is a 64 (70) bit string. Each bit specifies the strategys next move (cooperate or defect) for one particular history of 3 moves.

Encoding:
Need to remember each players move for 3 time steps ==> 6 pieces of information. At each point, either player could cooperate or defect (binary decision). 26 = 64 possible histories. Value of bit at a given position tells strategy what to do (0 = coop, 1 = defect) in the context of that history. Additional 6 bits encodes assumption about previous interactions before the start of the game.

Prisoners Dilemma Encoding cont.

Let A be the sum formed by,
Counting the others defection as 2 Counting my own defection as 1,

And giving weights of,

16 to the most recent moves, 4 to the move two time steps in the past, and 1 to the move three time steps in the past.

E.g.,
History of mutual cooperation for 3 time steps ==> 0. History of mutual defection for 3 times steps ==> 63.

What history should we assume for first 3 moves?

Mutual cooperation. Genetically determined (need 6 more bits).

Prisoners Dilemma Using GA

Fitness Function
Environment of tournament:
Select 8 strategies from ~160 tournament entries that predict performance, and play each candidate strategy against those 8 representatives.

Co-evolutionary environment:
Strategies play against other strategies in the population.

Results for Fixed Env.

From a random start, the GA evolved populations whose median member was just as successful as TFT. Most of the evolved strategies resemble TFT. In some rules, strategies were discovered that did significantly better than TFT (in tournament env.):
Discriminates between different opponents (based only on its behavior). Adjusts its own behavior to exploit an exploitable opponent. Does this without getting into (too) much trouble with other opponents. Achieved this by defecting on first move, and then apologizing when necessary.

Results in Evolving Environment (From Axelrod, 1997)

Sorted Order Representation

010111001101011 010 2 7 1 5 3 2
Bit String

Key Values

3 1

1 5

5 4

4 2

2 3
Rotated Layout Start at position 2

Sorted Order Representation

010111001101011 010 2 1 3 1 7 2 1 5 1 3 5 4 5 4 3 5 4 2 2 3
Rotated Layout Start at position 2 Bit String

Key Values Position