HighAvailabilitywithPostgreSQL

InstallPostgreSQLv9.3
SettingupaMasterserver
SettingupaCascading(Upstream)Standbyserver
SettingupaDownstreamStandbyserver
ScripttoconfigureaMaster/HotStandbyserver
Monitor
Failover
TheMasterfails
TheCascadingStandbyfails
TheDownstreamStandbyfails
ClusterDatabaseAdministration

HighAvailabilitywithPostgreSQL

InstallPostgreSQLv9.3
1. Followtheinstructionsinhttp://www.postgresql.org/download/linux/ubuntu/toaddthe
PostgreSQLAptRepository(forUbuntu13.04replaceYOUR_UBUNTU_VERSION_HEREwith
precise)

2. InstallPostgreSQL
sudo apt-get install postgresql-9.3 postgresql-contrib-9.3
3. Afterthis,PostgreSQLcreates,configuresandstartsadefaultdatabasecluster(*)
namedmainusingthefollowinglocations:

Datadirectory /var/lib/postgresql/9.3/main/
Configurationdirectory /etc/postgresql/9.3/main/
Binariesdirectory /usr/lib/postgresql/9.3/
Logfile /var/log/postgresql/postgresql9.3main.log

(*)Don'tbeconfused.Inthatcasetheword"cluster"isusedtoindicateaclusterof
databasesinthesamecomputer(accordingtoPostgreSQLdocumentation),notasetof
connectedcomputersthatworktogetherasawhole.

4. Setuppasswordlessaccessoversshforthepostgresuserusedinthe
archive_command (onlyifnotalreadydone).Youmustcopythegeneratedpublickey
totheothersserverstoallowaccessfromanyservertoanyotherone.Beforecopythe
publickeyyouneedtocreateapasswordfortheuserpostgresforeachhost.
sudo su postgres
ssh-keygen -t dsa
ssh-copy-id -i ~/.ssh/id_dsa.pub <server_to_access_ip>
Toverifyifitworksdothefollowing(youshouldntbeaskedtoenterapassword):
ssh <server_to_access_ip>
5. Repeatthepreviousstepsforeachserver:Master,CascadingStandbyand
DownstreamStandby.

SettingupaMasterserver
1. Changetothepostgresuser(bydefaultthecluster'sowner)
sudo su postgres
2. Createtheuserusedforthereplication(seeScripttoconfigureaMaster/HotStandby
server)
psql -c "CREATE USER replicator REPLICATION LOGIN ENCRYPTED
PASSWORD 'secret'"
3. Stopthecluster
pg_ctlcluster 9.3 main stop(seehowtoshowadbclusterinformation)
4. Editthe/etc/postgresql/9.3/main/postgresql.confconfigurationfileandchangethe
followingsettings:

Name

Value

listen_addresses

'*'

wal_level

hot_standby

archive_mode

on

archive_command

'rsync -av %p
postgres@<standby_ip>:/var/lib/postgresql/wal_archive/%f'

max_wal_senders

wal_keep_segments

32

synchronous_standby_names

'*'

TODO:addabriefexplanationofeachsetting

5. Editthe/etc/postgresql/9.3/main/pg_hba.confconfigurationfileandallowconnections
fromthestandbyserverandCyclos:

host

replication

replicator

<sub_net>/<stand_by_ip>

md5

host

all

<user>/all

<sub_net>/<cyclos_ip>

md5

6. Startthecluster
pg_ctlcluster 9.3 main start

SettingupaCascading(Upstream)Standbyserver
1. FollowthestepsasfortheMaster(SettingupaMasterserver)exceptthelastone(the
Masterinitialization).Thosesettingsinpostgresql.confwillbeignoredwhenrunningasa
standbybutwillbeneededincaseoffailover.Forthearchive_commandREPLACE
the<standby_ip>withtheIPoftheDownstreamStandby.
2. Additionally,thefollowingsettingmustbeenabledin
/etc/postgresql/9.3/main/postgresql.conf
hot_standby = on
3. CreatetheWALdirectory
mkdir /var/lib/postgresql/wal_archive
(thepostgresusermusthavewritepermissionsonthisdirectory.Seethe
archive_commandconfiguredintheMaster)
4. Cleanthedatadirectory
rm -r /var/lib/postgresql/9.3/main/*
5. TakeafreshbackupfromtheMasterserver
pg_basebackup -D /var/lib/postgresql/9.3/main -Fp -P -v -Xs -U
replicator -W -h <master_ip> [-p <master_port>]
6. Createafile/var/lib/postgresql/9.3/main/recovery.confandsetthefollowing:

Name

Value

standby_mode

'on'

primary_conninfo

'host=<master_ip> port=5432 application_name=standby_1

user=replicator password=secret'

restore_command

'cp /var/lib/postgresql/wal_archive/%f "%p"'

archive_cleanup_command

'/usr/lib/postgresql/9.3/bin/pg_archivecleanup
/var/lib/postgresql/wal_archive/ %r'

recovery_target_timeline

'latest'

7. Startthecluster
pg_ctlcluster 9.3 main start
8. ChecktheMasterslogfiletoverifyallisworkingasexpected.Youshouldseesomething
similartothis:
database system is ready to accept connections
...
standby "standby_1" is now the synchronous standby with priority 1

9. Checkthestandbyslog.Youshouldseesomethinglikethis:
database system is ready to accept read only connections
...
started streaming WAL from primary at ... on timeline 1

SettingupaDownstreamStandbyserver
Settingupadownstreamstandbyisbasicallythesameasfortheupstreamstandby(see
SettingupaCascading(Upstream)Standbyserver)withaminorchanges.
1. Replacethe<master_ip>withtheupstream_ip
2. Chooseadifferentname(e.g.:standby_2)fortheapplication_nameattributeofthe
primary_conninfointhe/var/lib/postgresql/9.3/main/recovery.conffile.
3. Startthecluster
pg_ctlcluster 9.3 main start
4. Checkthelogfile.Youshouldseesomethinglikethis:
database system is ready to accept read only connections
...
started streaming WAL from primary at ... on timeline 1

ScripttoconfigureaMaster/HotStandbyserver
Ifyoupreferyoucanrun(loggedasthepostgresuser)thefollowingscriptoneachserverto
configureaMaster/Standbynodetoapplyallofthosechangesautomatically:
#!/bin/bash
# ==================== show_help function =====================================

show_help() {
echo
echo "It configures a PostgreSQL v$PG_VERSION server to run as a Master/Hot Standby"
echo
echo "Usage: " $(basename $0)
"{clean} | {standby_server sub_net (used in
pg_hba.conf to allow access) [master_server standby_name]} "
echo
echo "
If a master_server is specified a standby_name must be specified too
and this means you are configuring a hot standby server."
echo "
The 'clean' parameter will erase the information (all data will be
lost!) related to the '$CLUSTER_NAME' cluster and create a new one."
echo "
If you are configuring a standby server and you don't have another
standby to cascade just put something like"
echo "
standby_ip as the first parameter (this won't be used because a
standby works in recovery mode and that parameter is used in archive mode)."
}
# ==================== configure_master function ==============================
configure_master() {
echo "Configuring Master server..."
echo "Creating replicator user for replication/backup..."
echo -n "Enter password for replicator user:"
read -s REPLICATOR_PWD
echo
# --------------------------------------------------------------------psql -c "CREATE USER replicator REPLICATION LOGIN ENCRYPTED PASSWORD
'$REPLICATOR_PWD'"
# --------------------------------------------------------------------echo "Stopping server..."
pg_ctlcluster $PG_VERSION $CLUSTER_NAME stop
# --------------------------------------------------------------------echo "Modifying settings in $PG_CONFIG_FILE..."
echo "
listen_addresses = '*'
wal_level = hot_standby
archive_mode = on
archive_command = 'rsync -av %p postgres@$STANDBY_SERVER:$WAL_FOLDER/%f'
max_wal_senders = 3
wal_keep_segments = 32
synchronous_standby_names = '*'
" >> $PG_CONFIG_FILE
# --------------------------------------------------------------------echo "Modifying settings in $HBA_CONFIG_FILE..."
echo "

host
all
all
host
replication replicator
" >> $HBA_CONFIG_FILE

$SUB_NET/24
$SUB_NET/24

md5
md5

}
# ================ configure_standby function =================================
configure_standby() {
echo
echo "Configuring Standby server with name $STANDBY_NAME..."
# --------------------------------------------------------------------echo "Modifying settings in $PG_CONFIG_FILE..."
echo "
hot_standby = on
" >> $PG_CONFIG_FILE
# --------------------------------------------------------------------echo "Creating WAL folder $WAL_FOLDER..."
rm -r $WAL_FOLDER
mkdir $WAL_FOLDER
# --------------------------------------------------------------------echo "Cleaning data directory..."
rm -r /var/lib/postgresql/$PG_VERSION/$CLUSTER_NAME/*
echo "Taking a fresh backup from $MASTER_SERVER..."
pg_basebackup -D /var/lib/postgresql/$PG_VERSION/$CLUSTER_NAME -Fp -P -v -Xs -U
replicator -W -h $MASTER_SERVER
echo "Creating recovery file..."
echo "
standby_mode = 'on'
primary_conninfo = 'host=$MASTER_SERVER port=$PG_PORT application_name=$STANDBY_NAME
user=replicator password=$REPLICATOR_PWD'
restore_command = 'cp $WAL_FOLDER/%f \"%p\"'
archive_cleanup_command = '/usr/lib/postgresql/$PG_VERSION/bin/pg_archivecleanup
$WAL_FOLDER/ %r'
recovery_target_timeline = 'latest'
" > $RECOVERY_CONFIG_FILE
}
# ================ reset_cluster function =====================================
reset_cluster() {
echo -n "Reset cluster $CLUSTER_NAME (y/n)?"
read resp
echo
if [ "$resp" == "y" ]
then
echo "Stopping cluster..."

pg_ctlcluster $PG_VERSION $CLUSTER_NAME stop -m immediate

echo "Dropping cluster..."
pg_dropcluster $PG_VERSION $CLUSTER_NAME
pg_createcluster $PG_VERSION $CLUSTER_NAME
echo "Starting cluster..."
pg_ctlcluster $PG_VERSION $CLUSTER_NAME start
pg_lsclusters
else
echo "Clean was canceled!"
fi
exit 0
}
# =============================================================================
# =========================== BEGIN ===========================================
# =============================================================================
# ========================= VARIABLES INITIALIZATION ==========================
STANDBY_SERVER=$1
SUB_NET=$2
MASTER_SERVER=$3
STANDBY_NAME=$4
CLUSTER_NAME=main
PG_VERSION=9.3
PG_PORT=5432
PG_CONFIG_FILE=/etc/postgresql/$PG_VERSION/$CLUSTER_NAME/postgresql.conf
HBA_CONFIG_FILE=/etc/postgresql/$PG_VERSION/$CLUSTER_NAME/pg_hba.conf
RECOVERY_CONFIG_FILE=/var/lib/postgresql/$PG_VERSION/$CLUSTER_NAME/recovery.conf
WAL_FOLDER=/var/lib/postgresql/wal_archive
# ========================= PARAMETERS VERIFICATION ==========================
# expected args are 1, 2 or 4
if [[ $# -ne 1 && $# -ne 2 && $# -ne 4 ]]
then
show_help
exit 0
fi
if [[ $# -eq 1 && "$1" -eq "clean" ]]
then
reset_cluster
exit 0
elif [ $# -eq 1 ]
then

echo "Invalid parameter: '$1'. 'clean' was expected."

exit 1
fi

# ========================= FUNCTIONS INVOCATIONS ==========================

configure_master
if [ -n "$MASTER_SERVER" ]
then
configure_standby
fi
echo "Starting server..."
pg_ctlcluster $PG_VERSION $CLUSTER_NAME start
echo "Done!"
exit 0

Sampleusage
Suposeyouwanttoconfigureasfollow(theexampleisusingthehostnamebutitcanworks
withIPtoo):

Servertype

Hostname

Master

master.server

Cascading

cascading.server

Downstream

downstream.server

Thenyoumustexecutethescriptinthisorder(thescriptwassavedinafilenamedconfigure):
The192.168.0.0isthesubnet_maskofthenetwork(bydefaultitusethefirst24bitsofthemask:
subnet_mask/24).

Onmaser:
configure cascading.server 192.168.0.0
Oncascading:
configure downstream.server 192.168.0.0 master.server standby_1
Ondownstream:
configure to_define 192.168.0.0 cascading.server standby_2

Monitor
Additionally,youcouldexecutethisqueryoneachservertoverifytheoverallconfiguration:
select application_name,client_addr,state,sync_priority,sync_state
from pg_stat_replication;

Theexpectedresultsshouldbe.


OnMaster:
application_name|state|sync_priority|sync_state
+++
standby_1 |streaming|
1|sync

OnCascadingStandby:
application_name|state|sync_priority|sync_state
+++
standby_2 |streaming|
0|async

OnDownstreamstandby:
application_name|state|sync_priority|sync_state
+++
(0rows)

Ateverymomentyoucanrunthefollowingscripttoverifyhowthereplicationisworking(just
copysaveitasshellscriptandsettherightvaluesforthevariables):

#!/bin/bash
USER=cyclos
DB=cyclos4
MASTER=stroit11.local
UPSTREAM_STANDBY=stroit18.local
DOWNSTREAM_STANDBY=stroit15.local
MASTER_QUERY="SELECT pg_xlog_location_diff(pg_current_xlog_location(), '0/0') AS offset;"
STANDBY_QUERY="SELECT pg_xlog_location_diff(pg_last_xlog_receive_location(), '0/0') AS
receive, pg_xlog_location_diff(pg_last_xlog_replay_location(), '0/0') AS replay;"
exec_query() {
h=$1
#host
q=$2
echo $(psql -A -t -U $USER -h $h -c "$q" $DB)
}
show_standby() {
standby=$1
standby_receive=$2
standby_replay=$3
master_offset=$4
receive_diff=$(($master_offset - $standby_receive))
replay_diff=$(($master_offset - $standby_replay))

echo "
echo "
echo "

Standby: "$standby
receive: "$standby_receive bytes". Behind: " $(($receive_diff / 1024)) KB
replay: "$standby_replay bytes". Behind: " $(($replay_diff / 1024)) KB

}
show_replication_status() {
master=$1
#mandatory
standby1=$2
#mandatory
standby2=$3
#optional
TMP=$(exec_query $standby1 "$STANDBY_QUERY")
if [ $# -eq 3 ]
then
TMP2=$(exec_query $standby2 "$STANDBY_QUERY")
fi
master_offset=$(exec_query $master "$MASTER_QUERY")
IFS='|' read -ra STANDBY_VALUES <<< "$TMP"
standby1_receive=${STANDBY_VALUES[0]}
standby1_replay=${STANDBY_VALUES[1]}
if [ $# -eq 3 ]
then
IFS='|' read -ra STANDBY_VALUES <<< "$TMP2"
standby2_receive=${STANDBY_VALUES[0]}
standby2_replay=${STANDBY_VALUES[1]}
receive2_diff=$(($master_offset - $standby2_receive))
replay2_diff=$(($master_offset - $standby2_replay))
fi

echo "------------------------ REPLICATION STATUS -------------------------"

echo "Master: "$master
echo " current: "$master_offset bytes
echo
show_standby $standby1 $standby1_receive $standby1_replay $master_offset
if [ $# -eq 3 ]
then
echo
show_standby $standby2 $standby2_receive $standby2_replay $master_offset
fi
echo "---------------------------------------------------------------------"
}
show_replication_status $MASTER $UPSTREAM_STANDBY $DOWNSTREAM_STANDBY
#show_replication_status $MASTER $UPSTREAM_STANDBY

Failover
Now,itistimetodiscusshowtorecover/reconfigurethesystemifanyoftherunningservers
fails.
TheMasterfails
InthiscasetheonlythingwemustdoispromotetheCascadingStandbyserverasthenew
master.Loggedinthestandbyserverthiscanbedonewiththefollowing:
pg_ctlcluster 9.3 main promote

ifyoulookatitslogfileyoushouldfindsomethinglikethis:
received promote request

standby "standby_2" is now the synchronous standby with priority 1

InthelogoftheDownstreamStandbyserveryoushouldfindthis:
new target timeline is 2
...
restarted WAL streaming at 0/12000000 on timeline 2

Afterthis,wehavetheserversrunningwithsynchronousreplicationenabled.
Ofcourse,weshouldtakeactiontoleavethisdegeneratestateassoonaspossible.To
achievethis,basically,therearetwoalternatives,toknow:
1. SetupanewDownstreamStandbyconnectedtothe(new)CascadingStandby.
2. Recreatetheoldmasterandswitchthecurrentmastertorecoverymode(againsetupto
functionasaCascadingStandby)renamingthe
/var/lib/postgresql/9.3/main/recovery.donefiletorecovery.confandrestartingthe
server.
TheCascadingStandbyfails
Thisconsequencesofthisfailarequiteimportantbecauseacommitofawritetransactionon
themasterwillwaituntilconfirmationisreceivedfromthe(crashed)CascadeStandbycausing
themastertostoptowork.
TosolvethiswemustreconnecttheDownstreamStandbydirectlytothetheMasterserver
editingthe/var/lib/postgresql/9.3/main/recovery.conffileinthedownstreamserverandput
theMasterIP.Alsowemustchangethearchive_commandsettinginthe
/etc/postgresql/9.3/main/postgresql.conffileoftheMastertosendtheWALsegmentstoits
newupstreamserver.
Restartthestandbyandjustreload(restartisnotrequiredinthiscase)theMaster:
pg_ctlcluster 9.3 main restart (onstandby)
pg_ctlcluster 9.3 main reload
(onmaster.Currentconnectionsarenotclosed!)

ChecktheMasterslogandyoushouldseethefollowing:
parameter "archive_command" changed to
...

standby "standby_2" is now the synchronous standby with priority 1

TheDownstreamStandbyfails
Inthiscaseatfirsttimethereisnothingtodobecausethesynchronousreplicationwillcontinue
working.Again,weshouldtrytorestoretheoriginalstatesettingupanewdownstreamstandby
assoonaspossible.

ClusterDatabaseAdministration

Todropanexistingclusteruse:
pg_dropcluster --stop 9.3 main
Tocreateanewclusterandstartitimmediately(bydefaulttheclusterisconfiguredtobe
started/stoppedautomaticallyintheinitscript):
pg_createcluster --start 9.3 main
Tostart/stopanexistingclusteruse:
pg_ctlcluster 9.3 main start/stop

Toshowinformationaboutalldatabaseclustersinahost:
pg_lsclusters

Assumingthisscenario:

Master=192.168.1.11
Cascade=192.168.1.12
Downstream=192.168.1.13

Runthiscommandoneachmachine:
unamen

Thiswillreturntheresolvedhostnameforeachmachine.
Putoneachmachineat/etc/hoststheIPsanditscorrespondingresolvedname

192.168.1.11 master_hostname
192.168.1.12 cascade_hostname
192.168.1.13 downstream_hostname

Itwillbethesameforeachmachine

Atmasterandcascadewewilluseheartbeatformonitoring

Runonmasterandcascademachine

apt-get install heartbeat --no-install-recommends

Inbothservers,createafilenamed/etc/ha.d/ha.cflikethis:

logfacility local0 #
keepalive 2
#
could also use 2000ms
deadtime 5
#
ping 192.168.1.12
#
udpport 694
#
bcast eth0
#
node masterHostname #
`uname -n`
node cascadeHostname#
by `uname -n`
auto_failback off

used to tell heartbeat which log facility to utilize for logging

interval between heartbeat packets currently every 2 secs you
timeout before the other server takes over
physical address of the other server. Set the correct ip
port to listen in on for broadcasts made by heartbeat
device to use for broadcasts
dns name of the main server - should be the same as returned by
dns name of the failover server - should be the same as returned

Then,alsoinbothservers,createafilenamed/etc/ha.d/haresourceslikethis:

master_hostnameIPaddr::192.168.1.15/24/eth0hapostgres

Finally,youneed,inbothservers,the/etc/ha.d/authkeysfile,whichmustbeequalsin
bothservers:

auth 1
1 md5 1234

Setthecorrectpermissionatauthkeysfile:
chmod 600 /etc/ha.d/authkeys

Wewillusethe192.168.1.15fordatabaseconectionatourapplication.
Andincaseoffailurethehapostgresservicewillbelaunched,soweneedtocreateit:

Atmastermachinecreatethisfilewiththecontentbelow
nano /etc/init.d/hapostgres
case $1 in
start)
;;
stop)
;;
restart)
;;
esac
exit 0
Settheexecpermission
chmod+x/etc/init.d/hapostgres

Atthecascademachinecreatethesamefilebutwiththiscontent:
nano /etc/init.d/hapostgres
case $1 in
start)
sudo -u postgres pg_ctlcluster 9.3 main promote
;;
stop)
;;
restart)
;;
esac
exit 0

Settheexecpermission
chmod +x /etc/init.d/hapostgres


Soyoucanstarttheheartbeatprocessonthemastermachineandafteronthecascade:

service heartbeat start

Ifitsokyoushouldseesomethinglikethis:
Starting High-Availability services: INFO:
Done.

Resource is stopped

Nowyouwillhaveanotheripaddress(192.168.1.15)onthemastermachine.

IncaseoffailurethisIPwillbemovedtothecascademachineandthe/etc/init.d/hapostgreswill
beexecutedpromotingthecascademachineasthenewmaster.

Tomonitorthecascadewewilluseanpingscriptbecausewedon'tneedanothervirtualipor
somethingelse.

Asincaseofafailureinthecascade,weneedtomakeareloadatthemasterserverwewill
needanpasswordlesssshforthepostgresuserfromthedownstreamtothemasterserver.

Onthedownstreamserverdothis:

cp /var/lib/postgresql/9.3/main/recovery.conf
/var/lib/postgresql/9.3/main/recovery.conf.ori
cp /var/lib/postgresql/9.3/main/recovery.conf
/var/lib/postgresql/9.3/main/recovery.cascade

Puttheiporthehostnameofthemasterserveronthefilerecovery.cascadeatprimary_conninfo

Onthemastermachinedo:
cp /etc/postgresql/9.3/main/postgresql.conf /etc/postgresql/9.3/main/postgresql.conf.ori
cp /etc/postgresql/9.3/main/postgresql.conf /etc/postgresql/9.3/main/postgresql.cascade

Puttheipofthedownstreamonthearchive_commandatthefilepostgresql.cascade

Atthemasterserverwewillcreateanewscript:(makesurethatthepostgressusercanrunthis
script)/script/failover_cascade.sh
Withthiscontent:

cp/etc/postgresql/9.3/main/postgresql.cascade/etc/postgresql/9.3/main/postgresql.conf
pg_ctlcluster9.3mainreload

settheexecpermissionforthesefile
chmod+x/script/failover_cascade.sh

Atthedownstreamserverwewillcreatethis2scripts:

/script/failover_cascade.shWiththiscontent:

cp/var/lib/postgresql/9.3/main/recovery.cascade/var/lib/postgresql/9.3/main/recovery.conf
pg_ctlcluster9.3mainrestart
sshpostgres@192.168.1.11/script/failover_cascade.sh

settheexecpermissionforthesefile
chmod+x/script/failover_cascade.sh

Createthesescript:
/script/ping_monitor.shwiththecontent:

#!/bin/bash
setx
/bin/pingc1192.168.1.12
if[$?=0]
then
echo"RespostaOk!"
else
/script/failover_cascade.sh
fi

settheexecpermissionforthesefile
chmod+x/script/ping_monitor.sh

puttheping_monitor.shtorunevery2or5secondsatthecronofthepostgresuseronthe
downstreamserver.

Sothiswillbecheckiftheipofthecascadearelive,andincaseoffailuretheywillrunthe
/script/failover_cascade.sh

The/script/failover_cascade.shwillchangetherequiredfilesandwillrunthe
/script/failover_cascade.shatthemastermachine.

Afterthisthedowsntreamserverwillbethenewcascadeserver.