Neural Network Lab

Particle Swarm Optimization Using C#

11/25/2013
Particle swarm optimization (PSO) is a tecni!ue "or "in#in$ appro%imate
solutions to #i""icult or impossible numeric optimization problems& 'n particular(
PSO can be use# to train a neural network& PSO is loosel) base# on te bea*ior
o" $roups suc as "locks o" bir#s or scools o" "is& +is article e%plains PSO an#
presents a complete #emo pro$ram in ,-&
. neural network is essentiall) a comple% "unction tat accepts numeric inputs
an# $enerates numeric outputs& . "ull) connecte# neural network wit m inputs(
i##en no#es an# n outputs as (m / ) 0 ( / n) 0 ( 0 n) wei$ts an# biases&
+rainin$ a neural network is te process o" "in#in$ *alues "or te wei$ts an#
biases so tat "or a $i*en set o" trainin$ inputs( te compute# outputs o" te
network closel) matc (in oter wor#s( a*e minimal error) te known outputs
associate# wit te trainin$ inputs& So( trainin$ a neural network is essentiall) a
numerical optimization problem&
'n or#er to use PSO to train a neural network( )ou must a*e a soli# $rasp o" PSO&
1ecause neural networks are "airl) complicate#( it2s best to learn about PSO b)
usin$ a simpler problem tan trainin$ a neural network& +ake a look at te
screensot o" a #emo pro$ram in Figure 1( an# te $rap o" a "unction in Figure
2&
[Click on image for larger view.
Figure 1. Particle Swarm Optimization 3emo
[Click on image for larger view.
Figure 2. 3umm) 3ouble43ip 5unction to 6inimize
+e ima$e in Figure 2 is te $rap o" te "unction7
z = x * exp( -(x^2 + y^2) )
+is is 8ust a relati*el) simple "unction tat can be use# to test #i""erent numerical
optimization tecni!ues& +e "unction as a known minimum *alue o" z 9
40&:2;;;1< at % 9 40&=0=10>; an# ) 9 0&0& +e screensot in Figure 1 sows PSO
in action( "in#in$ an appro%imate solution to te minimization optimization
problem&
+e #emo pro$ram be$ins b) settin$ up some PSO parameters& PSO is an iterati*e
process an# a particle represents a possible solution to te optimization problem
bein$ anal)ze#& +e problem #imension is set to 2 because te "unction to
minimize as two *alues( % an# ) (or( e!ui*alentl)( %0 an# %1)& +e number o"
particles is set to 5& 6ore particles $enerall) lea# to a better solution at te
e%pense o" per"ormance& Parameter ma%?pocs is set to 1(000 an# is te
ma%imum number o" processin$ iterations to be use#& .$ain( more epocs can
$i*e a better solution but will take lon$er& Parameter e%it?rror is set to 0&0& +e
e%it error is a *alue tat( i" #urin$ PSO processin$ a solution as an error tis low(
te al$oritm will e%it earl) be"ore reacin$ ma%?pocs iterations& 'n te #emo(
because e%it?rror is set to 0&0( te al$oritm will not terminate earl)&
Parameters min@ an# ma%@ are constraints on te possible *alues o" te
components o" a possible solution& 'n te #emo( min@ an# ma%@ are set to 410&0
an# 010&0( respecti*el)& '" )ou look at te $rap in Figure 2( )ou can see tat te
minimum *alue o" z occurs wen bot % an# ) are between 42&0 an# 02&0( so tose
*alues coul# a*e been use#& Aalues "or min@ an# ma%@ will *ar) "rom problem
to problem& 'n neural network trainin$( i" all input *alues are normalize#( settin$
min@ an# ma%@ to 410&0 an# 010&0 is a reasonable rule o" tumb in most cases&
Once te PSO parameters are set( te #emo pro$rams call te PSO sol*in$
meto#& ."ter 1(000 iterations( te #emo #ispla)s te "inal *alues o" te "i*e
particles in te swarm( alon$ wit te best solution "oun#& 'n tis e%ample( PSO
"oun# a best solution o" %0 9 40&=0=152( %1 9 0&0000:1 wic is *er) close to( but
not !uite( te optimal solution o" %0 9 40&=0=10=( %1 9 0&000000& +e best
solution as a *er) low error o" 0( roun#e# to "i*e places&
+e 3ouble43ip 5unction
Ben e*aluatin$ #i""erent numerical optimization tecni!ues( ' o"ten use te
"unction z 9 % / e%p(4(%C2 0 )C2)) sown in Figure 2& +e "unction #oesn2t a*e
an o""icial name (at least( tat '2m aware o")( so ' call it te #ouble4#ip "unction&
+e "unction is use"ul as a baseline bencmark problem because it as a sin$le
$lobal minimum an# a sin$le $lobal ma%imum( wit easil) compute# known
solutions&
1ecause te #ouble4#ip "unction as a known solution( tere2s no nee# to use
PSO& PSO is best use# "or problems were tere is no eas) classical tecni!ue(
suc as usin$ te calculus #eri*ati*e( "or "in#in$ a solution& +rainin$ a neural
network is an e%ample o" suc a problem& 't2s important to remember tat PSO is
not $uarantee# to "in# an optimal solution to an optimization problem&
O*erall Pro$ram Structure
+o create te #emo pro$ram( ' launce# Aisual Stu#io an# create# a ,- console
application an# name# it ParticleSwarmOptimization& +e #emo as no
si$ni"icant &N?+ #epen#encies so an) *ersion o" Aisual Stu#io will work& ."ter
te template co#e loa#e#( in te Solution ?%plorer win#ow ' rename# "ile
Pro$ram&cs to ParticlePro$ram&cs an# Aisual Stu#io automaticall) rename# class
Pro$ram "or me& .t te top o" te source co#e( ' #elete# all unnee#e# re"erences to
namespaces( lea*in$ 8ust te re"erence to S)stem&
+e o*erall pro$ram structure wit some BriteLine statements remo*e# an#
minor e#its to sa*e space is presente# in !isting 1& 6eto# Sol*e #oes most o" te
work an# uses te PSO al$oritm searc "or a $oo# solution to minimize te
#ouble4#ip "unction& 6eto# ?rror accepts an arra) o" #ouble( wic represents a
possible solution an# returns te s!uare# #i""erence between te compute# *alue
o" te #ouble4#ip "unction an# te known optimal *alue& Pro$ram4#e"ine# class
Particle encapsulates a *irtual particle& ' remo*e# all normal error4ceckin$ to
keep te size o" te #emo pro$ram small an# te main i#eas clear&
Listin$ 17 Particle Swarm 3emo Pro$ram Structure
using System;
namespace ParticleSwarmOptimization

class ParticleProgram

static !oi" #ain(string$% args)

&onsole'(rite)ine(*Particle Swarm Optimization*);

int "im = 2; ++ pro,lem "imensions
int numParticles = -;
int max.poc/s = 0111;
"ou,le exit.rror = 1'1; ++ exit early
"ou,le min2 = -01'1; ++ pro,lem-"epen"ent
"ou,le max2 = 01'1;
&onsole'(rite)ine(*3nStarting PSO*);
"ou,le$% ,estPos = Sol!e("im4 numParticles4
min24 max24 max.poc/s4 exit.rror);
"ou,le ,est.rror = .rror(,estPosition);
&onsole'(rite)ine(*5est position6oun"7*);
6or (int i = 1; i 8 ,estPos')engt/; ++i)

&onsole'(rite(*x* + i + * = *);
&onsole'(rite)ine(,estPos$i%'9oString(*:;*) + * *);
<
&onsole'(rite)ine(**);
&onsole'(rite(*:inal ,est error = *);
&onsole'(rite)ine(,est.rror'9oString(*:-*));
&onsole'(rite)ine(*3n.n" PSO "emo3n*);
&onsole'=ea")ine();
< ++ #ain
static "ou,le .rror("ou,le$% x) ' ' <
static "ou,le$% Sol!e(int "im4 int numParticles4 "ou,le min24
"ou,le max24 int max.poc/s4 "ou,le exit.rror) ' ' <
<
pu,lic class Particle ' ' <
<
+e Particle ,lass
+e Particle class #e"ines a particle "or use b) te PSO al$oritm& ,lass Particle is
presente# in !isting 2& 5or simplicit)( all class members are #eclare# usin$ public
scope& .rra) position represents te current position o" a particle( wic in turn
represents a possible solution to an optimization problem& .rra) *elocit)
represents te current spee# an# #irection o" a particle( presumabl) towar#s a new(
better position/solution& 5iel# error is te error associate# wit te particle2s
current position& .rra) bestPosition ol#s te position o" te particle wic
)iel#e# te best (lowest) error *alue& 5iel# best?rror is te error *alue associate#
wit location bestPosition&
Listin$ 27 +e Particle ,lass
pu,lic class Particle

pu,lic "ou,le$% position;

pu,lic "ou,le error;
pu,lic "ou,le$% !elocity;
pu,lic "ou,le$% ,estPosition;
pu,lic "ou,le ,est.rror;
pu,lic Particle("ou,le$% pos4 "ou,le err4 "ou,le$% !el4
"ou,le$% ,estPos4 "ou,le ,est.rr)

t/is'position = new "ou,le$pos')engt/%;
pos'&opy9o(t/is'position4 1);
t/is'error = err;
t/is'!elocity = new "ou,le$!el')engt/%;
!el'&opy9o(t/is'!elocity4 1);
t/is',estPosition = new "ou,le$,estPos')engt/%;
,estPos'&opy9o(t/is',estPosition4 1);
t/is',est.rror = ,est.rr;
<
pu,lic o!erri"e string 9oString()

string s = **;
s += *==========================3n*;
s += *Position7 *;
6or (int i = 1; i 8 t/is'position')engt/; ++i)
s += t/is'position$i%'9oString(*:>*) + * *;
s += *3n*;
s += *.rror = * + t/is'error'9oString(*:>*) + *3n*;
s += *?elocity7 *;
6or (int i = 1; i 8 t/is'!elocity')engt/; ++i)
s += t/is'!elocity$i%'9oString(*:>*) + * *;
s += *3n*;
s += *5est Position7 *;
6or (int i = 1; i 8 t/is',estPosition')engt/; ++i)
s += t/is',estPosition$i%'9oString(*:>*) + * *;
s += *3n*;
s += *5est .rror = * + t/is',est.rror'9oString(*:>*) + *3n*;
s += *==========================3n*;
return s;
<
<
+e Particle constructor simpl) accepts *alues "or a particle an# copies tem in&
1ecause all "iel#s are public( te constructor coul# a*e been omitte# but( in m)
opinion( usin$ an e%plicit constructor is a cleaner #esi$n& +e +oStrin$ meto# is
cru#e but use"ul "or BriteLine4st)le #ebu$$in$ #urin$ #e*elopment&
+e ?rror 5unction
6eto# ?rror is #e"ine# as7
static "ou,le .rror("ou,le$% x)

"ou,le true#in = -1'>2@@@0A>;

"ou,le z = x$1% *
#at/'.xp( -((x$1%*x$1%) + (x$0%*x$0%)) );
return (z - true#in) * (z - true#in)
<
+e meto# accepts a particle2s position/solution as arra) %( computes te *alue o"
te #ouble4#ip "unction at position %( an# returns te s!uare o" te #i""erence
between te compute# solution an# te known solution& 'n te case o" neural
network trainin$( te error "unction will be some measure o" ow closel) a set o"
network4compute# output *alues matces a set o" trainin$ #ata output *alues&
+e Sol*e 6eto#
6eto# Sol*e implements te PSO al$oritm& +e meto# is presente# in !isting
"( locate# towar# te en# o" tis article& 6eto# Sol*e assumes tat tere is an
accessible error meto#& .n alternati*e is to pass te error meto# into Sol*e as a
#ele$ate& 6eto# Sol*e be$ins b) settin$ up a Dan#om ob8ect7
=an"om rn" = new =an"om(1);
Ob8ect rn# is use# to initialize eac particle2s position an# *elocit) wit ran#om
*alues( an# to ran#oml) kill particles& Ne%t( a collection o" Particle ob8ects is
create# an# an arra) to ol# te best position/solution "oun# is allocate#7
Particle$% swarm = new Particle$numParticles%;
"ou,le$% ,estBlo,alPosition = new "ou,le$"im%;
"ou,le ,estBlo,al.rror = "ou,le'#ax?alue;
Decall tat eac particle as a best position "oun#E arra) bestFlobalPosition is te
best4known position o" an) particle in te swarm&
Ne%t( eac particle in te swarm is initialize# to a ran#om position an# te error
associate# wit te ran#om position is #etermine#& +e ke) co#e is7
"ou,le$% ran"omPosition = new "ou,le$"im%;
6or (int C = 1; C 8 ran"omPosition')engt/; ++C)
ran"omPosition$C% = (max2 - min2) * rn"'DextEou,le() + min2;
"ou,le error = .rror(ran"omPosition);
+e ne%t step in initialization is to create a ran#om *elocit)7
"ou,le$% ran"om?elocity = new "ou,le$"im%;
6or (int C = 1; C 8 ran"om?elocity')engt/; ++C)

"ou,le lo = min2 * 1'0;

"ou,le /i = max2 * 1'0;
ran"om?elocity$C% = (/i - lo) * rn"'DextEou,le() + lo;
<
?ac component o" te initial *elocit) is set to a small ran#om *alue& ' use 0&1
times min@ an# ma%@ as limits( wic are somewat arbitrar)( but wic a*e
worke# well "or me in practice&
Bit a ran#om position( associate# error an# ran#om *elocit) set up( te current
particle can be initialize#7
swarm$i% = new Particle(ran"omPosition4 error4
ran"om?elocity4 ran"omPosition4 error);
+e call ma) look at bit o## at "irst $lance& +e "ourt an# "i"t parameters to te
Particle constructor are te particle2s best4known position an# associate# error(
wic initiall) are te same as te startin$ position an# error&
."ter eac particle is initialize#( eac is cecke# to see i" it as te best position o"
an) o" te particles in te swarm7
i6 (swarm$i%'error 8 ,estBlo,al.rror)

,estBlo,al.rror = swarm$i%'error;
swarm$i%'position'&opy9o(,estBlo,alPosition4 1);
<
."ter swarm initialization( meto# Sol*e prepares te main PSO processin$ loop7
"ou,le w = 1'F2A; ++ inertia
"ou,le c0 = 0'>A>>-; ++ cogniti!e weig/t
"ou,le c2 = 0'>A>>-; ++ social weig/t
"ou,le r04 r2; ++ cogniti!e4 social ran"omizations
"ou,le pro,Eeat/ = 1'10;
int epoc/ = 1;
"ou,le$% new?elocity = new "ou,le$"im%;
"ou,le$% newPosition = new "ou,le$"im%;
"ou,le new.rror;
Ben a particle mo*es in PSO( a new *elocit) is "irst compute#& +en( te new
*elocit) is use# to compute te new position& . particle2s new *elocit) #epen#s on
tree "actors7 te particle2s current *elocit)( te particle2s best4known position an#
te best4known position o" an) particle in te swarm& Aariable w is calle# te
inertia wei$t an# #etermines te in"luence o" te current *elocit)& Aariables c1
an# c2 are calle# te co$niti*e an# social wei$ts an# #etermine te in"luence o"
te particle2s best position an# te $lobal best4known position( respecti*el)& +e
*alues use# ere a*e been su$$este# b) PSO researc an# )ou can tink o" tem
as ma$ic constants&
Aariables r1 an# r2 are *alues tat a## a ran#omization e""ect to a particle2s
mo*ement( wic elps to pre*ent te particle "rom becomin$ stuck in a #ea#
en#& Aariable prob3eat is set to a small *alue an# is use# to ran#oml) kill an#
ten re$enerate a particle& +e #eat4birt process is optional in PSO( but '2*e
"oun# it to be a use"ul enancement&
'n pseu#o4co#e( te main PSO processin$ loop is7
loop until "one
6or eac/ particle
compute new !elocity
use !elocity to compute new position
compute new error at position
c/ecG i6 new error is a particle ,est
c/ecG i6 new error is a swarm ,est
Gill particle wit/ small pro,a,ility
en" eac/ particle
en" loop
return ,est position 6oun"
+e new *elocit) "or te current particle( currP( is compute# like so7
Particle currP = swarm$i%;
6or (int C = 1; C 8 currP'!elocity')engt/; ++C)

r0 = rn"'DextEou,le();
r2 = rn"'DextEou,le();
new?elocity$C% = (w * currP'!elocity$C%) +
(c0 * r0 * (currP',estPosition$C% - currP'position$C%)) +
(c2 * r2 * (,estBlo,alPosition$C% - currP'position$C%));
<
+is is te eart o" te PSO al$oritm& '" )ou look closel)( )ou2ll see ow eac
component o" te current particle2s *elocit) #epen#s on te current *elocit)( te
particle2s best4known position an# te swarm2s best4known position& +e net e""ect
is tat particles ten# to mo*e towar#s better solutions( an# teir *elocit) ten#s to
#ecrease as teir position impro*es&
+e new position "or te current particle( currP( is compute# like tis7
6or (int C = 1; C 8 currP'position')engt/; ++C)

newPosition$C% = currP'position$C% + new?elocity$C%;

i6 (newPosition$C% 8 min2)
newPosition$C% = min2;
else i6 (newPosition$C% H max2)
newPosition$C% = max2;
<
5or e%ample( suppose a particle is at (3&0( =&0) an# te new *elocit) is (1&0( 42&0)&
+e particle2s new position woul# be (:&0( 5&0)& '" a component o" te current
particle2s position $oes out o" ran$e( it2s brou$t back into ran$e&
."ter a particle2s new position is #etermine#( te error associate# wit te new
position is compute# ten cecke# to see i" it2s a new best error "or te particle or
a new best error "or an) particle in te swarm&
+e e""ecti*eness o" PSO #epen#s to some e%tent on te initial positions
$enerate# "or eac particle& ' au$ment basic PSO b) ran#oml) killin$ ten
re$eneratin$ particles& 'n pseu#o4co#e7
generate a ran"om !alue
i6 ran"om !alue small t/en
generate a new ran"om position
compute error o6 new position
(lea!e existing !elocity)
c/ecG i6 new swarm ,est (,y c/ance)
en" i6
+e i#ea o" ran#oml) killin$ particles is to tr) an# pre*ent particles "rom
clumpin$ to$eter near a non4optimal solution& '" te probabilit) o" #eat is too
i$( particles will #ie o"" too !uickl) an# not reac a $oo# position/solution& 1ut
i" te probabilit) o" #eat is too low( repositionin$ will occur so in"re!uentl) tat
it ma) not be use"ul& '2*e "oun# tat a #eat probabilit) between 0&01 an# 0&1 is
$enerall) most e""ecti*e&
Listin$ 37 +e PSO Sol*e 6eto#
static "ou,le$% Sol!e(int "im4 int numParticles4 "ou,le min24
"ou,le max24 int max.poc/s4 "ou,le exit.rror)

++ assumes accessi,le .rror() an" Particle class

=an"om rn" = new =an"om(1);
Particle$% swarm = new Particle$numParticles%;
"ou,le$% ,estBlo,alPosition = new "ou,le$"im%;
"ou,le ,estBlo,al.rror = "ou,le'#ax?alue;
++ swarm initialization
6or (int i = 1; i 8 swarm')engt/; ++i)

"ou,le$% ran"omPosition = new "ou,le$"im%;
6or (int C = 1; C 8 ran"omPosition')engt/; ++C)
ran"omPosition$C% = (max2 - min2) *
rn"'DextEou,le() + min2;
"ou,le error = .rror(ran"omPosition);
"ou,le$% ran"om?elocity = new "ou,le$"im%;
6or (int C = 1; C 8 ran"om?elocity')engt/; ++C)

"ou,le lo = min2 * 1'0;
"ou,le /i = max2 * 1'0;
ran"om?elocity$C% = (/i - lo) * rn"'DextEou,le() + lo;
<
swarm$i% = new Particle(ran"omPosition4 error4
ran"om?elocity4 ran"omPosition4 error);
++ glo,al ,est position+solutionI
i6 (swarm$i%'error 8 ,estBlo,al.rror)

,estBlo,al.rror = swarm$i%'error;
swarm$i%'position'&opy9o(,estBlo,alPosition4 1);
<
< ++ initialization
"ou,le w = 1'F2A; ++ inertia weig/t
"ou,le c0 = 0'>A>>-; ++ cogniti!e weig/t
"ou,le c2 = 0'>A>>-; ++ social weig/t
"ou,le r04 r2; ++ cogniti!e an" social ran"omizations
"ou,le pro,Eeat/ = 1'10;
int epoc/ = 1;
"ou,le$% new?elocity = new "ou,le$"im%;
"ou,le$% newPosition = new "ou,le$"im%;
"ou,le new.rror;
++ main loop
w/ile (epoc/ 8 max.poc/s)

6or (int i = 1; i 8 swarm')engt/; ++i) ++ eac/ Particle

Particle currP = swarm$i%;
6or (int C = 1; C 8 currP'!elocity')engt/; ++C)

r0 = rn"'DextEou,le();
r2 = rn"'DextEou,le();
new?elocity$C% = (w * currP'!elocity$C%) +
(c0 * r0 * (currP',estPosition$C% - currP'position$C%)) +
(c2 * r2 * (,estBlo,alPosition$C% - currP'position$C%));
<
new?elocity'&opy9o(currP'!elocity4 1);
6or (int C = 1; C 8 currP'position')engt/; ++C)

newPosition$C% = currP'position$C% + new?elocity$C%;
i6 (newPosition$C% 8 min2)
newPosition$C% = min2;
else i6 (newPosition$C% H max2)
newPosition$C% = max2;
<
newPosition'&opy9o(currP'position4 1);
new.rror = .rror(newPosition);
currP'error = new.rror;
i6 (new.rror 8 currP',est.rror)

newPosition'&opy9o(currP',estPosition4 1);
currP',est.rror = new.rror;
<
i6 (new.rror 8 ,estBlo,al.rror)

newPosition'&opy9o(,estBlo,alPosition4 1);
,estBlo,al.rror = new.rror;
<
++ "eat/I
"ou,le "ie = rn"'DextEou,le();
i6 ("ie 8 pro,Eeat/)

6or (int C = 1; C 8 currP'position')engt/; ++C)
currP'position$C% = (max2 - min2) *
rn"'DextEou,le() + min2;
currP'error = .rror(currP'position);
currP'position'&opy9o(currP',estPosition4 1);
currP',est.rror = currP'error;
i6 (currP'error 8 ,estBlo,al.rror)

,estBlo,al.rror = currP'error;
currP'position'&opy9o(,estBlo,alPosition4 1);
<
<
< ++ eac/ Particle
++epoc/;
< ++ w/ile
++ s/ow 6inal swarm
&onsole'(rite)ine(*3nProcessing complete*);
&onsole'(rite)ine(*3n:inal swarm73n*);
6or (int i = 1; i 8 swarm')engt/; ++i)
&onsole'(rite)ine(swarm$i%'9oString());
"ou,le$% result = new "ou,le$"im%;
,estBlo,alPosition'&opy9o(result4 1);
return result;
<
.n .lternati*e to 1ack4Propa$ation
+e e%planation o" PSO an# #emo co#e presente# in tis article soul# $et )ou up
to spee# wit a torou$ knowle#$e o" PSO& 1) "ar te most common tecni!ue
"or trainin$ a neural network is te back4propa$ation al$oritm& 1ack4propa$ation
is matematicall) ele$ant but as some serious #rawbacks& 'n particular( "or man)
problems( back4propa$ation ten#s to be e%tremel) sensiti*e to te *alues use# "or
te learnin$ rate an# momentum& PSO is an attracti*e alternati*e to back4
propa$ation "or trainin$ neural networks&
.bout te .utor
3r& Games 6c,a""re) works "or Aolt 'n"ormation Sciences 'nc&( were e mana$es
tecnical trainin$ "or so"tware en$ineers workin$ at te 6icroso"t De#mon#(
Bas&( campus& He2s worke# on se*eral 6icroso"t pro#ucts( inclu#in$ 'nternet
?%plorer an# 6SN Searc& 3r& 6c,a""re) is te autor o" I&N?+ +est .utomation
DecipesI (.press( 200>) an# can be reace# at 8ammcJmicroso"t&com&