Installing, configuring and operating a

clustered master server cluster with

catalog replication
Issue Date: 19th March 2010
Version number: 1.0
Applies to: NetBackup release 6.5.4 and above
Note: This is a living document and will be subject to periodic updates. Please
check the data and version number to ensure you are referencing the latest version.
Please send any feedback on this document to IMG-TPM-Requests@symantec.com

This document is provided for informational purposes only. All warranties relating to the information in this document, either express
or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without
notice. Copyright 2010 Symantec Corporation. All rights reserved. Symantec, the Symantec logo and NetBackup are trademarks or
registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of
their respective owners.

Contents
1.0
1.1
1.2
2.0
2.1
2.2
3.0
3.1
3.2
3.3
3.4
3.5
3.6

INTRODUCTION .................................................................................................................. 1
GLOSSARY .......................................................................................................................... 1
ADDITIONAL READING........................................................................................................... 1
NETBACKUP CONFIGURATION ........................................................................................ 3
NETBACKUP SERVER REQUIREMENTS ................................................................................... 3
REPLICATED STORAGE CONFIGURATION ................................................................................ 3
INSTALLING AND CONFIGURING THE NETBACKUP SERVERS .................................. 4
INSTALLING THE AND CONFIGURING THE PRIMARY MASTER SERVER CLUSTER .......................... 4
INSTALLING AND CONFIGURING THE SECONDARY MASTER SERVER CLUSTER ........................... 4
STRETCHING AN EXISTING CLUSTER...................................................................................... 4
POPULATING THE SERVER TABLES ........................................................................................ 5
DNS CONSIDERATIONS ........................................................................................................ 5
GLOBAL CLUSTERS .............................................................................................................. 5

4.0

UPGRADING NETBACKUP IN A REPLICATED CONFIGURATION ................................ 6

5.0

OPERATING A REPLICATED ENVIRONMENT ................................................................. 7

5.1
5.2
6.0
6.1
6.2
6.3
6.4

FAILING OVER TO THE SECONDARY NODE .............................................................................. 7

PERFORMING A FIRE DRILL TEST ......................................................................................... 7
CROSS DOMAIN IMPLEMENTATION ................................................................................ 8
CROSS DOMAIN IMPLEMENTATION CONSIDERATIONS .............................................................. 8
PARTIAL CATALOG REPLICATION IN CROSS DOMAIN IMPLEMENTATIONS ................................... 8
VARIATIONS IN INSTALLATION AND UPGRADE PROCEDURES .................................................... 8
ADDITIONAL FAILOVER STEPS FOR CROSS DOMAIN OPERATION ............................................... 8

Installing, configuring and operating a non-clustered master server with catalog replication
Page i

1.0 Introduction
This document expands on the information provided in tech note 287636 and deals specifically
with replication between clustered master server clusters. This document provides a high level
description of the installation and upgrade procedures and operation practices when replicating
catalogs between clustered master server clusters. It does not cover the clustering or replication
technologies used in any detail please refer to your specific clustering and replication
documentation for details on deploying and operating the clustering and replication layers.

1.1 Glossary
The following descriptive terms are used throughout this document:
NetBackup catalog the NetBackup catalog is the set of databases used by NetBackup to
locate data held in backups. It comprises of several separate databases which, on a clustered
master server cluster, are normally co-located on a single volume.
NetBackup domain a group of clients, media servers and storage devices under the control of
a single NetBackup master server cluster. A NetBackup domain may be limited to a single data
center or span multiple data centers in multiple locations.
Production domain - the NetBackup domain, spanning one or more data centers that form the
organizations regular IT infrastructure, including the clients and media servers used to make
backups and restore data under normal operating circumstances.
DR (disaster recovery) domain a separate NetBackup domain at a remote location that is
used for disaster recovery purposes in the event that the production domain/sites are unavailable.
The DR domain has its own set of media servers and is controlled by the DR master server
cluster to which the catalogs of the production master server cluster are replicated.
Primary master server cluster the cluster that usually runs the NetBackup master server
controlling backup and restore operations.
Secondary master server cluster the cluster that the NetBackup master server is failed over
to in the event of a problem at the primary site. The secondary master server cluster may be
located at another site in the same NetBackup domain (single domain replication) or form part of
a separate disaster recovery domain (cross domain replication).
Replication layer the replication layer is the storage layer that sits beneath the NetBackup
catalogs and ensures they are replicated correctly between the sites.
Single domain replication the catalogs are replicated to a secondary master server cluster
that forms part of the same NetBackup domain with access to the same media servers and
storage .
Cross domain replication the catalogs are replicated to a secondary master server cluster
that forms part of a separate NetBackup domain with different media servers and storage
devices.
Cluster service group the construct in the clustering application that includes the resources
required to support the NetBackup master server. This usually includes the virtual IP address
and cluster common storage configuration as well as the resource that control the NetBackup
processes themselves.

1.2 Additional reading

The NetBackup Troubleshooting Guide

(http://seer.entsupport.symantec.com/docs/290230.htm)

NetBackup Catalog Replication: Conditional Support Statement

(http://seer.entsupport.symantec.com/docs/287636.htm)

Installing, configuring and operating a non-clustered master server with catalog replication
Page 1

Installing, configuring and operating a non-clustered NetBackup master server cluster

with catalog replication (http://seer.entsupport.symantec.com/docs/345977.htm )

Recovery procedures when replicating catalogs between domains

(http://seer.entsupport.symantec.com/docs/345979.htm)

Installing, configuring and operating a non-clustered master server with catalog replication
Page 2

2.0 NetBackup Configuration

This section describes the server and replicated storage requirements for catalog replication.

2.1 NetBackup server requirements

Both the primary and secondary master servers must be cluster using the same virtual
name. However the number and names of the member nodes of each master server
cluster can be different.

The primary and secondary master server clusters must be compatible server types
running the same version and patch level of the operating system and the
release/patch/EEB level for the NetBackup server (including the NetBackup binaries and
ICS components) must be kept in sync between primary and secondary sites.

The secondary master server cluster should not have any other NetBackup function,
either in the same domain as the primary master server cluster or in another domain (e.g.
the secondary master server cluster cannot be used as a media server when it is not
being used as a master server cluster, nor can it be the master server cluster for another
NetBackup domain catalogs are replicated but cannot be merged).

The all of the components that make up the NetBackup catalogs entire NetBackup install
path must be located on a single volume or volume set which is replicated between sites.

NetBackup must be configured to be manually started and stopped on the secondary

master server cluster.

The NetBackup catalog mount point must be the same at both primary and secondary
master server clusters.

The NetBackup master server and EMM Server must reside on the same virtual server,
they cannot use different virtual names or IP addresses.

2.2 Replicated storage configuration

The replication technology employed must maintain a consistent (i.e. write-ordered) copy
of the data at all times. If change maps are being applied to perform resynchronization
after a replication problem, then the administrator must take a snapshot of the replicated
dataset at the secondary site before starting the resynchronization step. This is
mandatory to provide a rollback to a consistent point in case of a failure in applying the
change map.

Replication should continuous (i.e. not periodic snapshots) but does not have to be fully
synchronous. Asynchronous and semi-synchronous replication is supported provided the
replication is fully write-ordered.

The ability to bring NetBackup up using the replicated copy on the secondary site should
be verified as part of the initial configuration. This is not a requirement for support.

Site-specific configuration/procedures must make provision for enabling, disabling,

switching, synchronization, and verification as appropriate of the replicated data set in the
course of switching the NetBackup server between sites.

Installing, configuring and operating a non-clustered master server with catalog replication
Page 3

3.0 Installing and configuring the NetBackup servers

3.1 Installing the and configuring the primary master server
cluster
When setting up the primary site follow the standard approach to installing a clustered NetBackup
master server cluster as described in the NetBackup High Availability Guide, specifying the
replicated storage as the mount point for the cluster common storage and specifying all the
servers that will form part of the domain, including those that will form the secondary site master
server cluster.
Once the NetBackup cluster group has been created the storage resources in the cluster must be
reconfigured to include any replication control components. For some replication layers, notably
VVR, best practice dictates that the replication agent should be contained in a separate service
group to the NetBackup application and that the two service groups should be linked so that they
operate together.
If the replication technology being deployed offers bandwidth planning and analysis tools it is a
good idea to configure the NetBackup master server cluster to run on the primary master server
cluster for a period before implementing the replication layer and stretching the cluster to the
secondary site using the sizing information collected by the tools.

3.2 Installing and configuring the secondary master server

cluster
The cluster at the secondary site is installed and configured using the same virtual name and
cluster common mount point as the cluster at the primary site. During the installation process be
sure to specify all the NetBackup servers when prompted, including the master server cluster
nodes and media servers at the primary site.
Note: The secondary master server cluster does not need to have the same number of nodes as
the primary master server cluster but it must be clustered. It is quite acceptable for the secondary
site to be configured with a single node cluster but it cannot be configured as a non-clustered
server if the primary side is clustered.
In order to prevent data loss and access conflicts the initial configuration of the cluster at the
secondary site must be done using a temporary volume as the cluster common volume rather
than the replication target volume from the primary site. This is because the catalog setup,
carried out as part of the installation on Windows and as part of the cluster configuration process
(using the cluster_config script) on UNIX and Linux, creates and seeds the EMM database on the
cluster common storage.
Create a small disk volume (100 MB should suffice) and mount it to the same mount point used
for the replicated volume on the primary server. Once the configuration of the cluster is complete
the cluster should be taken off line and the mapping of the temporary storage replaced with
mapping to the replication target. For Windows environments this volume will be known as the
upgrade volume and will need to be presented every time NetBackup is upgraded on the
secondary master server cluster. For UNIX and Linux environments this volume is not required
after the initial installation and can be destroyed once the installation is complete.
The replication agent on the secondary site should not be configured at automatically reverse the
direction of replication. Replication should not be reversed until the primary site is operational
again and you are satisfied that it is appropriate to re-sync the catalog volumes and fail back to
the primary site.

3.3 Stretching an existing cluster

As discussed in section 3.2 above, the secondary master server cluster in simply installed with
the same virtual name and attributes as the primary master server cluster and the cluster
common storage is replicated to it. This means that it is very simple to stretch an existing cluster
Installing, configuring and operating a non-clustered master server with catalog replication
Page 4

to create a replicated or global cluster environment by following the steps laid out in the earlier
sections.
Remember that if the replication technology provides tools for sizing the replication components
these should be used on the primary master server cluster to determine the replication layer
requirements before attempting to configure the replication layer.

3.4 Populating the server tables

For both new installations and extended existing installations the server tables must be correctly
populated in EMM. This can be done automatically be simply failing the master server cluster
over to each node in turn.
Once the setup of the secondary cluster is complete, NetBackup should be taken off-line on the
primary site, the replication direction should be reversed and NetBackup should be bought on-line
on the secondary site. The act of bringing the NetBackup master server cluster on-line on a
particular node of the cluster automatically adds that node to the list of known servers in the EMM
server tables. Where there are multiple nodes at the secondary site the master server should be
failed over to each node.
Once all member nodes have been added to the cluster NetBackup should be taken off-line on
the secondary site, the replication direction should be reversed and NetBackup should be bought
on-line on the primary site.

3.5 DNS considerations

In most cases the geographical separation between the primary and secondary sites requires the
master server nodes at the secondary site to be on a different subnet to those at the primary site.
In this case a DNS change may need to be carried out as part of the failover process. Where a
DNS change is to be carried out it is important to consider any operational practices around
making such a change. The DNS change may be propagated automatically by the cluster
failover process or it may be manually initiated. The backup system may not function correctly
until the change has been full propagated and that may impact the recovery time in a site failover.
Note: if the DNS change is to be propagated automatically by the cluster service group the DNS
resource must come on-line after NetBackup has started.

3.6 Global clusters

Some clustering technologies such as VCS allow the creation of a global cluster by linking the
clusters at the primary and secondary site at the management and control level. This is achieved
with a public heartbeat connection that can generate an alert when the secondary master server
cluster loses contact with the primary master server cluster. Refer to your clustering
documentation for details on configuring global clusters.

Installing, configuring and operating a non-clustered master server with catalog replication
Page 5

4.0 Upgrading NetBackup in a replicated configuration

The primary and secondary master server clusters must run the same version of NetBackup if
the secondary master server cluster is not running the same version as the primary master server
cluster NetBackup may not start in a failover situation.
For UNIX and Linux environments both the primary and secondary master server clusters are
upgraded following the procedures outlined in the NetBackup installation guides (note that all
nodes on the secondary cluster are considered to be passive for upgrade purposes.
For Windows environments the primary master server cluster is upgraded following the
procedures outlined in the NetBackup installation guides but the secondary master server cluster
must be upgraded as follows:
1. Isolate the secondary master server cluster from the rest of the network
2. Present the upgrade volume to the appropriate mount point
3. Start NetBackup
4. Upgrade NetBackup following the procedures outlined in installation guide
5. Stop NetBackup
6. Dismount the upgrade volume
7. Reconnected the secondary master server cluster to the rest of the network
The reason for these additional steps is that the Windows upgrade procedure requires at least on
node of the cluster to be active at the time of the upgrade even if there are no changes to the
database schemas.

Installing, configuring and operating a non-clustered master server with catalog replication
Page 6

5.0 Operating a replicated environment

5.1 Failing over to the secondary node
In the event of all nodes of the primary master server cluster failing or an incident resulting in a
denial of access to the primary site it will be necessary to failover to the secondary master server
cluster at the secondary location. The exact failover procedure will vary depending on the
replication technology used but the following high level sequence should be followed:
1. Stop NetBackup on the primary master server cluster
2. Stop or reverse the replication of the catalog volume
3. If necessary update DNS to reflect the new virtual IP address for the master server
4. Start NetBackup on the secondary master server cluster
It should now be possible to control the NetBackup domain from the secondary master server
cluster. (Note that the first two steps described here will occur automatically if the primary site is
lost).
For configurations where global clustering is used the steps described here can be automated to
provide a single button failover capability. However unless multiple heartbeat connections exist
between the clusters the failover process should not be completely automated as a failure of the
heartbeat network could cause the secondary master server cluster to come on-line while the
primary master server cluster was still operational.

5.2 Performing a fire drill test

It is often desirable to test the ability to bring up the secondary master server cluster without
going through a full failover operation. As in the case of a full failover the detail steps will vary
depending on the replication technology but the following high level sequence should be followed:
1.

Suspend replication between the primary and secondary master server clusters

2.

Isolate the secondary master server cluster from the network

3.

Start NetBackup on the secondary master server cluster

4.

Carry out any validation checks required

5.

Stop NetBackup on the secondary master server cluster

6.

Reestablish network connections on the secondary master server cluster

7.

Restart the replication

Installing, configuring and operating a non-clustered master server with catalog replication
Page 7

6.0 Cross domain implementation

The information provided in the previous sections assumes that the secondary master server
cluster has access to the same NetBackup domain as the primary master server cluster (i.e. it
can reach the same media servers and clients as the primary master server cluster). In some
cases the secondary master server cluster may be in a disaster recovery or bunker location
where it has access to a different set of servers and clients to those available in the primary
domain. In this case there are some additional considerations and the setup and operation will
be slightly different.

6.1 Cross domain implementation considerations

Most disk based backup technologies cannot be used with cross domain catalog replication
because the disk device presentation in the DR domain will not match the production domain.
For this reason replication of catalogs between domains is only supported for backups written to
tape or BasicDisk (BasicDisk volumes must be replicated and the replicated volumes mount
against the same mount point on media servers in the DR domain).

6.2 Partial catalog replication in cross domain implementations

When replication occurs between domains the media server and device configuration in the DR
domain is not captured in the EMM database replicated from the production domain. This can
result in lengthy recovery procedures as the media server and device configuration must be
rediscovered after a failover. One way to avoid this problem is to use partial catalog replication
as described in tech note 345979.
To implement partial replication in clustered environments the following additional setup steps
need to be followed:
1. Relocate the relational database components of the primary and secondary master
server clusters to a separate cluster common volume as described in Appendix C of the
NetBackup High Availability Administrators Guide
2. Replicate on the volume containing the non-relational database components between the
clusters.
If partial catalog replication is used the tape usage and assignment information will not be
replicated to the DR domain and additional care is required to ensure tapes are not accidentally
overwritten. Partial catalog replication is discussed in more detail in tech note 345979.

6.3 Variations in installation and upgrade procedures

The installation and upgrade procedures are largely the same for single domain and cross
domain configurations, however in a cross domain configuration the server list on the secondary
master server cluster should also include the media servers in the DR domain.

6.4 Additional failover steps for cross domain operation

Following a failover in a cross domain configuration additional steps are required to ensure that
the DR domain media servers and devices configured and the master server cluster does not
attempt to reach media servers in the production domain. This is discussed in more detail in
tech note 345979.

Installing, configuring and operating a non-clustered master server with catalog replication
Page 8