Best Practices for Disk Layout with the Flexible Disk Option

Issue Date: 5th April 2009 Version number: 2.0 Note: This is a living document and will be subject to periodic updates. Please check the data and version number to ensure you are referencing the latest version.

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 1

Contents
INTRODUCTION.............................................................................................................................3 1.1 GLOSSARY...................................................................................................................................3 1.2 SHAREDDISK SUPPORT...................................................................................................................4 1.3 RELATED DOCUMENTS.....................................................................................................................4 FLEXIBLE DISK OPTION DISK TYPES.........................................................................................5 1.4 ADVANCEDDISK............................................................................................................................5 1.5 SHAREDDISK................................................................................................................................5 1.6 FILE SYSTEM RESTRICTIONS............................................................................................................5 1.7 WHICH DISK TYPE SHOULD I BE USING?............................................................................................6 DISK POOL FUNDAMENTALS......................................................................................................6 1.8 CONFIGURING STORAGE SERVERS....................................................................................................9 1.9 CONFIGURING DISK POOLS..............................................................................................................9 1.10 EXCLUSIVE USE OF DISK VOLUMES.................................................................................................13 1.11 DISK VOLUMES WITH DIFFERENT CHARACTERISTICS.............................................................................13 1.12 DISK POOLS AND VOLUME MANAGERS...........................................................................................14 ADVANCED DISK PERFORMANCE CONSIDERATIONS AND BEST PRACTICES....................14 1.13 CONSIDERATIONS FOR DISK VOLUME SIZES IN ADVANCEDDISK............................................................14 1.14 NETWORK FILE SYSTEM CONSIDERATIONS........................................................................................14 1.15 STATE CHANGES IN ADVANCEDDISK..............................................................................................15 SHARED DISK PERFORMANCE CONSIDERATIONS AND BEST PRACTICES.........................16 1.16 PREPARING SHAREDDISK LUNS FOR USE......................................................................................16 1.17 SCALABILITY.............................................................................................................................17 1.18 STATE CHANGES IN SHAREDDISK .................................................................................................19 CAPACITY MANAGEMENT WITH FLEXIBLE DISK OPTION......................................................20 1.19 INTELLIGENT CAPACITY MANAGEMENT AND LOAD BALANCING WITHIN DISK POOLS...................................20 1.20 WATER MARKS AND CAPACITY BASED RETENTION..........................................................................20 1.21 INCREASING AND DECREASING THE SIZE OF DISK POOLS.......................................................................21 STORAGE UNIT CONFIGURATIONS...........................................................................................21 1.22 STORAGE UNIT GROUPS AND MEDIA SERVER LOAD BALANCING..........................................................21 1.23 MULTIPLE STORAGE UNITS SHARING THE SAME DISK POOL................................................................22

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 2

Introduction
The disk types available under the Flexible Disk Option (AdvancedDisk and SharedDisk) as first introduced with NetBackup 6.5 allow NetBackup to fully utilize file systems native to the host operating system of the Media Server. NetBackup assumes full owner ship of these file systems while at the same time using the Storage Server capabilities of the host operating system. The Flexible Disk Option comes in two flavors of disk types. One of these, AdvancedDisk, does not require any specialized hardware, while the other, SharedDisk, depends on the availability of SAN attached storage. Both flavors are managed as Disk Pools within NetBackup. This document will focus on performance issues and best practices related to both these flavors. In practice the entire data path between client and storage including both hardware and software stacks determines the overall performance of the backup and restore process. It is therefore essential that the performance of the disk storage is not considered independently of entire data path and effect of infrastructure when seeking to resolve overall performance issues. While following the guidelines presented in this paper should ensure that the disk storage performs optimally it will not guarantee optimal performance of the entire data path

1.1 Glossary
The following terms are used throughout this document: Storage Server A Storage Server is a logical abstraction of the entity that owns (serves) physical storage. It needs to be implemented on a real system. In the context of this paper a Storage Server is always implemented on a Media Server. Data Mover A Data Mover is a logical abstraction of the entity that moves data between the NetBackup Client and the Storage Server. It needs to be implemented on a real system. In NetBackup 6.5, Data Movers are only implemented on Media Servers. Disk, disk volume or volume in the context of this document the term disk does not refer to a single physical spindle or LUN but is stead refers to a disk volume presenting a file system that can be written to by NetBackup. In a SharedDisk implementation this equates to a single LUN presented from a storage array but that LUN may be made up of multiple physical spindles combined at the hardware level. In an AdvancedDisk implementation this equates to a mount point with a file system which could either be a single physical disk (spindle), a LUN comprised of multiple spindles at the hardware layer or a volume created using a logical volume manager such as VxVM. Disk Pool A collection of Disks Volumes (usually in the same disk array) which provides a target storage area for Storage Units. BasicDisk this is the term used in NetBackup 6.5 to describe the simple mounted file system disk storage unit that has been available in NetBackup releases for some time. BasicDisk storage units are limited to a single mount point or path and cannot make use of many of the new features introduced in NetBackup 6.5 such as Media Server Load Balancing and Storage Lifecycle Policies. AdvancedDisk one of the two types of disk covered by the Flexible Disk Option. AdvancedDisk can make use of any locally presented or network presented disk storage. In the initial implementation AdvancedDisk volumes were dedicated to a single media server but from NetBackup 6.5.2 onward volumes such as NFS volumes that can be presented to multiple servers simultaneously can be shared by multiple Media Servers. SharedDisk the other disk type available with Flexible Disk Option. SharedDisk makes use of disk volumes in SAN attached arrays which can be presented to multiple Media Servers in a similar way to the way in which tape drives are shared in a Shared Storage Option configuration with exclusive access to each volume being granted to a single Media Server at any time. LUN or SAN LUN a LUN or volume presented from a disk array that can be used as a disk volume. This may not correspond directly to a physical disk within the array as hardware Raid and mirroring may be used below the LUN level. Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 3

Capacity management NetBackups Storage Lifecycle Policy system can be used with the Flexible Disk Option to stage data destined for long term storage, providing a faster restore time for recently created backups. The capacity management feature ensures the disks do not fill up by removing older and less important backups when capacity thresholds are reached. Capacity management also ensures that data is distributed evenly across the volumes in a disk pool. Load balancing works in conjunction with capacity management to ensure that backups are directed to the most appropriate volumes using the least heavily loaded Media Server that has access to the Disk Pool. High and low water marks these threshold figures are use to control the capacity management feature of the Flexible Disk Option. They set on a Disk Pool and applied to all the volumes within a pool. nbdevconfig the NetBackup command used to create, import, preview and inventory disk pools. nbdevquery the NetBackup command used to display the characteristics of disk pools and their associated components. nbshareddisk the NetBackup command used to format LUNs in a disk array used in a SharedDisk configuration. VxVM Veritas Volume Manager, a logical volume manager that forms part of the Symantec Storage Foundation product. VxFS Veritas File System, an advanced file system that forms part of the Symantec Storage Foundation product on UNIX and Linux servers.

1.2 SharedDisk support

Note that support for SharedDisk is available only with NetBackup 6.5.x and will not be included in NetBackup 7.0. SharedDisk support will be withdrawn with NetBackup 6.5.x support on 3rd October 2012.

1.3 Related documents

The following documents provide more background on the subjects discussed in this paper: The NetBackup Hardware Compatibility List (http://seer.entsupport.symantec.com/docs/284599) NetBackup 6.5 Shared Storage Guide (http://seer.entsupport.symantec.com/docs/290238) NetBackup 6.5.1 Shared Storage Guide (http://seer.entsupport.symantec.com/docs/293321) NetBackup 6.5.2 Documentation updates (http://seer.entsupport.symantec.com/docs/302438) Upgrading BasicDisk Storage Units to AdvancedDisk Storage Units (http://seer.entsupport.symantec.com/docs/301168) Implementing SCSI persistent reservation in SharedDisk (http://seer.entsupport.symantec.com/docs/297573) Using AdvancedDisk sharing with Storage Foundation Cluster File System (http://seer.entsupport.symantec.com/docs/320797)

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 4

Flexible Disk Option Disk Types

The Flexible Disk Option includes two distinct types of disk, AdvancedDisk and SharedDisk. These disk types differentiate themselves from the traditional BasicDisk storage by the use of disk pools and from OpenStorage devices by being presented directly to the Media Servers rather than through an appliance of some type.

1.4 AdvancedDisk
AdvancedDisk increases the utilization and performance of disk storage by combining disk resources together as a single pool rather than treating them as smaller, separate islands of disk. AdvancedDisk also extends the capability of BasicDisk through the Enterprise Disk Foundation features such as Media Server Load Balancing and Storage Lifecycle Policies which increase Media Server utilization and availability while improving backup success rates and performance. Instead of creating Storage Units from single disk volumes, AdvancedDisk allows you create Storage Units containing multiple disk volumes defined within a disk pool.

1.4.1 AdvancedDisk sharing

The concept of AdvancedDisk sharing was introduced in NetBackup 6.5.2 and allows storage that can be mounted against the same mount point on multiple Media Servers such as storage presented over NFS or using Storage Foundation Cluster File System, Lustre or CXFS, to be placed into disk pools and shared between those Media Servers. Storage presented is this way is accessible to all the Media Servers at the same time and remains mounted at all times while server is up. Some restrictions apply when using network presented storage such as NFS and CIFs and this is discussed in more detail in section 1.13.

1.5 SharedDisk
SharedDisk allows LUNs in a disk array to be shared between multiple Media Servers in the same way that tape drives can be shared with the Shared Storage Option. As in AdvancedDisk, the SharedDisk volumes are defined in a disk pool, however in SharedDisk storage the disks are in a storage array and the disk pool is presented over a SAN connection to multiple Media Servers. Disks in a SharedDisk disk pool are normally dismounted and are automatically mounted on individual Media Servers when required they cannot be mounted on multiple Media Servers at the same time. When a disk from the pool is assigned to a Media Server, the file system is mounted, and the backup image is then read or written. Once the backup or restore operation has completed the disk is dismounted and returned to the pool. Any disk within the disk pool may be presented to any Media Server at any time. Since a disk can be mounted on different Media Servers, the Media Servers participating in the pool must be the same platform type running the same operating system. Support for SharedDisk is limited to certain operating systems and certified disk array and should be checked against the NetBackup Hardware Compatibility List.

1.6 File System Restrictions

Please note that the following restrictions apply when using Flexible Disk Option with particular file systems:

1.6.1 SharedDisk limitations

SharedDisk is only supported with native (UFS and NTFS) and Veritas (VxFS) file systems other file systems such as ZFS are not supported

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 5

1.6.2 AdvancedDisk limitations

For some file system types, notably NFS and ZFS, the lack of full posix compliance of the file system means that full file system conditions cannot be detected correctly, leading to problems when spanning volumes. To avoid problems where these file systems are used, each disk pool should be comprised of only one volume so that no spanning occurs.

1.7 Which Disk Type Should I be using?

AdvancedDisk can be used with any disk type and offers efficiency improvements and other benefits over the traditional BasicDisk model, including the following: Disk Pools which simplify expansion and improve space usage The ability to use Storage Lifecycle Policies The ability to use Storage Unit Groups with Media Server load balancing The ability to use common storage shared across multiple Media Servers for load balancing and redundancy

In most cases you should be considering converting existing BasicDisk storage units to AdvancedDisk to take advantage of these features. SharedDisk is particularly useful in configurations where only some Media Servers have access to tape drives. Storage Lifecycle Policies can be used with SharedDisk to automate the duplication process to tape using different Media Servers to the ones used for the original backups by setting the Alternate Read Server setting in the backup Storage Destination of the policy. SharedDisk implementations generally offer better performance that AdvancedDisk sharing but rely on premium tier 1 storage. This makes SharedDisk more suitable as a staging point for mission critical backups that have a requirement for fast recovery in the short term and longer tem retention on tape or other storage while AdvancedDisk sharing is more suitable for less critical backups with shorter retention requirements as its comparative low cost makes it more cost effective where backup and restore performance is not a primary consideration.

Disk Pool Fundamentals

Disk pools form one of the key concepts underlying the Flexible Disk Option. It significantly changes the way in which available disk space is used. A disk pool groups a set of disks together to form a single block of storage that can be shared among multiple Storage Units and, in some cases, multiple Media Servers. With the Flexible Disk Option disk types of AdvancedDisk and SharedDisk, the disk pool provides a pool of storage for use by the Storage Units, replacing the more conventional one-to-one mapping between disk and Storage Unit provided by the BasicDisk model. In BasicDisk configurations each Storage Unit has access to a single disk volume or part thereof. With the Flexible Disk Option, the Storage Units can access all the disks in a disk pool and the disk used for a particular backup is selected based on the amount of space available. In effect the entire disk pool appears as a single disk to reach Storage Units. The following table summarizes the advantages of using disk pools over BasicDisk volumes. Feature Storage sharing BasicDisk Fixed capacity based on a single disk. Multiple Storage Units can be created on a Single Media Server but each one requires a dedicated disk. No checks on available space before backup starts Disk Pools Multiple Storage Units on the same Media Server can share a disk pool. Space is allocated dynamically as required. Benefit of Disk Pools Automatic space allocation reduces the risk of backup failure.

Intelligent capacity management

Will Fit selection selects the Storage Unit based on available space.

Pre-selection of storage ensures efficient backup operation.

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 6

Feature Simplified growth

BasicDisk No ability to increase the size of individual Storage Units when disks fill up Does not support load balancing in Storage Unit Groups

Disk Pools Disk pool capacity is increased by adding additional disk volumes into the pool.

Media Server load balancing

Supports load balancing directing the backup to the least busy Media Server

Storage Lifecycle Policies

Staging achieved using individual Storage Units with staging schedules

Data can be classified and managed differently throughout its life based on the importance of the data. After it has been duplicated, data can be selectively expired based on the relative importance to the business.

Benefit of Disk Pools Minimal reconfiguration required as environment grows. Improved utilization of disk. Eliminates Media Server bottlenecks to increase the performance and success of backup jobs. Important backups are available for rapid restore for longer periods, thereby improving RTOs.

Table 1 - Disk Pools vs. BasicDisk Storage Figure 1 below shows the difference between a Media Server with access to BasicDisk Storage Units and a Media Server with access to an AdvancedDisk disk pool.

Figure 1 - BasicDisk vs. AdvancedDisk In the BasicDisk configuration the three disks act as separated Storage Units and the backups are written to specific disks. Backup jobs do not use all three disks unless they are configured to form a Storage Unit Group and there is no guarantee that the disk space is used efficiently among the three independent disks. AdvancedDisk utilizes space more efficiently than BasicDisk. In the AdvancedDisk configuration the three disks form a single pool and backups may be written to any of the disks. When a backup runs, Intelligent Disk Capacity Management will automatically estimate the size of the backup and select the disk with the greatest amount of free space and will reserve space to match the estimated size of the backup image so that other backup jobs do not over commit beyond the disks available space. Note how the different colored backup images reside on different disks in the BasicDisk configuration but are distributed across all of the disks in the AdvancedDisk disk pool. Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 7

Storage Units that use AdvancedDisk disk pools on different Media Servers can be configured to form load balanced Storage Unit Groups. When a backup is run using the Storage Unit Group, the Storage Unit on the least busy Media Server is automatically selected and a suitable disk within the AdvancedDisk disk pool on that Media Server is used for the backup. An AdvancedDisk disk pool can be formed from any disks that can be seen by, and are presented to, the Media Server. However it is recommended that all the disks in a particular disk pool should have common characteristics to ensure consistent behavior when different disk are selected for use. In a SharedDisk configuration the disk pool is created using a disk array which can be shared between multiple Media Servers over a SAN. This allows the individual disks within the disk pool to be presented to different Media Servers at different times. As in the case of AdvancedDisk, the disk selection process is based on the will fit model with space being reserved while a backup is in progress to prevent conflicts but with SharedDisk the disks can be unassigned to one Media Server and then presented to another Media Server. SharedDisk disk pools load balance at the Storage Unit level rather than the Media Server. Multiple Media Servers have access to the same Storage Unit. When a backup is run to a particular Storage Unit, the least busy Media Server is automatically selected and a suitable disk is presented to that Media Server for the backup to write the backup image. Figure 2 shows four Media Servers sharing a common SharedDisk disk pool.

Figure 2 - SharedDisk Disk Pool Both AdvancedDisk and SharedDisk rely on collections of disk volumes assembled in Disk Pools. Before a LUN becomes usable, a file system native to the operating system of the Media Server host has to be created on that LUN. This use of native file systems means that the normal rules and guidelines related to file system configuration on the host fully apply. Disk Pools support the concept of Media Server grouping for load balancing purposes. For SharedDisk, grouping refers to the Media Servers defined within a Disk Pool to share the volumes. For the OpenStorage and PureDisk options, grouping refers to the Media Servers defined with credentials to the storage server, and defined within storage units that use a Disk Pool. In the initial 6.5 release AdvancedDisk Disk Pools are restricted to a single Media Server but in later releases Media Server grouping will work in the same way that it does for SharedDisk.

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 8

1.8 Configuring Storage Servers

In order to create Disk Pools the Media Server must first also be configured as a Storage Server. This is done using the command line function nbdevconfig creatests. This command takes four additional qualifiers: -storage_server this is the name of the Storage Server and, for Flexible Disk Option is always the same as the Media Server -stype this is the Storage Server type, for Flexible Disk Option the valid values are AdvancedDisk and SharedDisk. If a Media Server is going to host both types of disk the command must be run twice, once for each value of stype -media_server this is the name of the Media Server -st this is the storage type and is an aggregate value made up from two values added together. These values are either 1 indicating formatted disk or 2 indicating raw disk and 4 indicating direct attached storage or 8 indicating network attached storage. The term raw disk here means a disk that is not pre-mounted but is mounted on demand. For AdvancedDisk the value of st is always 5 and for SharedDisk the value is always 6. In the following examples the Media Server NBMED01 is configured as an AdvancedDisk Storage Server: nbdevconfig -creatests -storage_server NBMED01 -stype AdvancedDisk -media_server NBMED01 -st 5 and as a SharedDisk Storage Server: nbdevconfig -creatests -storage_server NBMED01 -stype SharedDisk -media_server NBMED01 -st 6

1.9 Configuring Disk Pools

Once the Storage Server has been established the Disk Pools can also be created using the nbdevconfig command. Note that before a Disk Pool can be configured the disks in the Disk Pool must be formatted. For SharedDisk the formatting is done using the nbshareddisk command which is described in more detail on section 1.16. For AdvancedDisk the formatting must be done before the disks are mounted and the disks must be mounted before they can be added to the Disk Pool. It is possible to create a Disk Pool using the command line, for example you can create a list of the disks that the Storage Server can see using the command: nbdevconfig -previewdv -storage_servers <storage server name> -stype SharedDisk dvlist <file name> and then use that list to create a Disk Pool using the command: nbdevconfig -createdp -dp <disk pool name> -stype SharedDisk -storage_servers <storage server name> -dvlist <file name> However the Disk Pool Wizard available in the Administration GUI under Media and Device Management -> Devices -> Disk Pools provides a simple, quick way to configure Disk Pools. First select the type of Disk Pool you are creating:

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 9

Figure 3: Pool Type Selection Then select the Media Servers you want to associate with this Disk Pool:

Figure 4: Media Server Selection

The wizard will display the list of arrays (in the case of SharedDisk) or mount points (in the case of

AdvancedDisk) that can be used:

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 10

Figure 5: Array Selection

Figure 6 - Disk selection When the array or disks are selected it is possible to set high and low water mark attributes for the Disk Pool, these values are applied to each disk within the Disk Pool individually:

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 11

Figure 7: Disk Pool Properties Once the Disk Pool is created it is possible to configure a SharedDisk storage unit using some or all of the Media Servers associated with the Disk Pool:

Figure 8: Storage Unit Creation

1.9.1 Patchwork Disk Pool

When configuring an AdvancedDisk Disk Pool it is possible to select the individual disks that are used to make up the pool. In a SharedDisk the selection is normally confined to an array with all of the disks within the array forming the disk pool, however it is also possible to create what is known as a patchwork Disk Pool in which only selected disks in an array are used to make up the disk pool using the nbdevconfig command to obtain a list of the disks in the array and then use the same command to create the disk pool using an edited disk list as follows: Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 12

1. Create a file containing a list of all the available LUNs using the command: nbdevconfig -previewdv -storage_servers -stype SharedDisk dvlist <file name> <storage server name>

2. Edit the file and remove any LUNs that are not required to be in the disk pool. 3. Create the disk pool based on the contents of the file using the command: nbdevconfig -createdp -dp <disk pool name> -stype SharedDisk -storage_servers <storage server name> -dvlist <file name> This allows you to create a disk pool out of any number of volumes from any number of arrays.

1.10 Exclusive use of disk volumes

Once a file system is imported as a disk volume into a NetBackup Disk Pool, NetBackup assumes that no third party applications utilize this file system. However, NetBackup has no mechanism in place to enforce this. Failure to ensure sole use / ownership of a disk volume by NetBackup may and will cause incorrect behavior of the capacity management components in NetBackup resulting in premature image expiration. For example in a Disk Pool containing one disk volume with a high water mark set to 90% and a low water mark set to 70%, if 50% of the available space used by applications other than NetBackup only 40% will be available to NetBackup. When the high water mark is reached 50% of all the backups held on the disk volume will be removed rather than 20% if the disk volume is exclusively available to NetBackup. Recommendation: Disks used as AdvancedDisk and SharedDisk volumes in NetBackup should not be used for any other purpose. If there ever is a need to mount a SAN LUN outside of NetBackup (troubleshooting, file system maintenance, etc.), the related disk volume must first be marked as DOWN within NetBackup the command nbdevconfig changestate with the appropriate qualifiers depending on the Storage Server type, Disk Pool name and disk volume. Once the SAN LUN is dismounted, the disk volume should be marked as UP in order for NetBackup to resume operations on that disk volume.

1.11 Disk volumes with different characteristics

NetBackup allows multiple disk volumes to reside within a single Disk Pool. Load balancing strategies are applied across all disk volumes within the Disk Pool during media and device selection (MDS). In other words, NetBackup assumes that all disk volumes within a Disk Pool are somewhat similar. This includes the areas of size and performance characteristics. While asymmetric configuration will not cause backups to fail, it is likely to result in unpredictable performance. Example 1: Assume a Disk Pool with two disk volumes, one is 800 GB and the other is 80 GB. When NetBackup selects a disk volume to be used for a backup job, this selection is essentially driven by the size (free space) of the disk volume. Because of the size difference in this case, the larger disk volume will initially receive a larger share of the backup traffic. While this is quite correct from the point of view of available space, it will limit the system wide performance as the bulk of the I/O traffic will be directed to the one disk. Example 2: Assume a Disk Pool with two disk volumes of the same size, one has a transfer rate of 100 MB/sec and the other has a transfer rate of 25 MB/sec. Once again load balancing will select a disk volume based on available free space. As a result both disk volumes will see the same amount of traffic but backups and restores will run much faster to one disk volume than the other. Recommendation: All disk volumes within a NetBackup Disk Pool should be of similar size and should have similar performance characteristics. When dealing with disk volumes that are have significantly different characteristics they should be grouped into multiple disk pools, each containing disk volumes with similar characteristics, rather than all being placed in one pool.

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 13

1.12 Disk Pools and Volume Managers

Logical volume managers such Veritas Volume Manager (VxVM) allow abstractions to be created between the underlying disks/spindles and the volume on which a file system resides. Multiple small volumes can be created on a single disk and multiple disks can be combined to form a single large volume. Volume Managers can improve the resilience and data integrity for backups written to disk volumes used in Disk Pools by allowing mirroring and RAID configurations. The AdvancedDisk storage implementation presents all mounted file systems as disk volumes to NetBackup. AdvancedDisk operates naturally with volume managers that work below the level of a mounted file system because it has no visibility of them and thus is not concerned with the underlying geometry of the storage. For the SharedDisk storage implementation NetBackup actively manages mounting and dismounting of file systems (which includes management of the SAN visibility and accessibility of SAN LUNs). The use of Volume Managers is limited to configurations that present a one-to-one mapping between SAN LUNs in the disk array enclosure and the operating system view of these LUNs. Any mirroring or RAID configuration must be done in the array hardware below the level of the LUN presentation. Recommendation: When using Volume Managers with SharedDisk there must be a one-to-one mapping between the LUNs presented by the disk array and the disks that form the SharedDisk Disk Pool.

Advanced Disk Performance Considerations and Best Practices

The AdvancedDisk storage implementation relates mounted files systems within the operating system of the Media Server to disk volumes within NetBackup Disk Pools. The normal rules and guidelines related to file system configuration on the host apply and advanced file systems such as VxFS and XFS can be used on Media Servers that support them. The AdvancedDisk storage implementation enforces the fact that an AdvancedDisk disk volume is a mounted file system on the Storage Server. However, NetBackup does not manage the mounting and dismounting of AdvancedDisk disk volumes. Instead, mounting and dismounting is managed by the administrator and the operating system of the host. Recommendation: File systems that are used as AdvancedDisk disk volumes in NetBackup should be configured in such way that they are automatically mounted upon startup of the operating system of the hosting Media Server.

1.13 Considerations for Disk Volume Sizes in AdvancedDisk

In an AdvancedDisk configuration the disk volumes can be created from individual LUNs using a logical volume manager such as VxVM to create volumes of any size. When creating disk volumes for use in an AdvancedDisk Disk Pool there are two factors to consider for optimal performance: 1. To avoid backups spanning multiple volumes the size of each disk volume should be larger than the largest backup that will be written to it. 2. To avoid write contention between backup jobs the number of volumes in the disk group should be equal to or greater than the total number of maximum concurrent jobs allowed in all the Storage Units using the Disk Pool.

1.14 Network file system considerations

The AdvancedDisk storage implementation presents all mounted file systems as disk volumes to NetBackup. This includes network file systems (such as NFS). Like different types of SCSI and SAN presented disk volumes, network file system disk volumes should be placed in dedicated Disk Pools based on size as well as performance. Disk Pools should not contain a mixture of locally presented and network presented disk volumes. When using network file systems the following should be observed: Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 14

Manual mount points must be used. Automatic mounting and dismounting can change mount points, which may cause disk resources to be unavailable. The server that exports the mount points must be configured to allow root access to the file systems from the Storage Servers.

Although a network file system can be visible across many Media Servers, NetBackup 6.5 is not aware of this capability and network file systems should not be presented to multiple Storage Servers running NetBackup 6.5 or 6.5.1. NetBackup 6.5.2 and subsequent versions are aware of the ability to present network file systems to multiple servers and allows the configuration of multiple Storage Servers within an AdvancedDisk Disk Pool. However NetBackup does not validate the mount points so if the NFS volumes are presented to more than one Storage Server, the following must apply for the file systems of the disk volumes: Each Media Server must mount the file systems of all the disk volumes within a disk pool. The mount points must be valid The mount points must be the same on each Media Server.

Some network file systems, such as NFS, implement a file system behavior that makes them unsafe for use in spanning situations. This restriction does not apply to locally presented common file systems such as Storage Foundation Cluster File System Recommendation: NFS should be configured to disable spanning. For NetBackup 6.5 this is done by limiting the number of disk volumes in a Disk Pool to one. Note: NetBackup does not support the use of CIFS (Common Internet File System) disk volumes with Windows Media Servers because the mapped devices are not visible to the Windows services and thus cannot be discovered by nbdevconfig.

1.15 State Changes in AdvancedDisk

If the mount status of a configured disk volume changes, the state of the disk volume within NetBackup will follow (with a delay of about a minute). The current state of a disk volume can be checked by issuing the command nbdevquery liststs stype AdvancedDisk U as this example from a Windows server shows: Storage Server Storage Type State Flag Flag Flag Flag Flag Flag Flag Flag Flag Flag : wstmas02 : Formatted Disk, Direct Attached : UP : OpenStorage : AdminUp : InternalUp : SpanImages : LifeCycle : CapacityMgmt : FragmentImages : Cpr : RandomWrites : FT-Transfer Storage Server Type : AdvancedDisk

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 15

The mount status is indicated by the InternalUp flag. If InternalUp is not show, the disk volume is marked as DOWN in the output of the command nbdevquery listdp U as shown here: Disk Pool Name Disk Pool Id Disk Type Status Flag Flag Flag Flag Flag Flag Flag Flag Flag Flag Flag Flag Flag Raw Size (GB) Num Volumes High Watermark Low Watermark Comment Storage Server : DPtest : DPtest : AdvancedDisk : DOWN : Patchwork : Visible : OpenStorage : SingleStorageServer : AdminDown : InternalUp : SpanImages : LifeCycle : CapacityMgmt : FragmentImages : Cpr : RandomWrites : FT-Transfer : 4.00 : 1 : 98 : 80 : : wstmas02

Usable Size (GB) : 4.00

The administrator can reset the state of the disk volume by issuing the command: nbdevconfig changestate stype AdvancedDisk dp <disk pool> -dv <disk volume> -state RESET

Shared Disk Performance Considerations and Best Practices

The SharedDisk storage implementation relates LUNs within SAN attached storage arrays (enclosures) to disk volumes within NetBackup Disk Pools. NetBackup actively manages mounting and dismounting of these LUNs within the operating system of the Media Server.

1.16 Preparing SharedDisk LUNs for use

Before the disk volumes in a SharedDisk array can be added to a disk pool they need to be formatted. This is done from the command line using the nbshareddisk command The following steps show the overall process and provide examples of the command line instructions that are needed to format a set of devices that SharedDisk can use: Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 16

1. Issue the list command without any options to display all enclosures. From the list, select an enclosure for further discovery: $ nbshareddisk list Enclosure list: Vendor Array Name Enclosure ID -----------------------------------------------------------------NETAPP ndmpfiler1 UEID##NETAPP##LUN##0a505b7c IBM IBM.1750-13ADDTA UEID##IBM##TOTALSTORAGE##IBM.1750-13ADDTA HP HPEVA4000 UEID##HP##HSV##5000-1FE1-5007-0020 HP VRTS.EVA.ROS UEID##HP##HSV##5000-1FE1-5004-5660 EMC 000182601092 UEID##EMC##SYMMETRIX##000182601092 2. Scan the list to determine on which enclosure you want to perform further discovery. After you select an enclosure for further discovery, issue a command to output the UDIDs. The following example shows the information being redirected to a file so that it can be used with the format option. It lists the UDIDs in a specified enclosure. $ nbshareddisk list -ueid UEID##HP##HSV##5000-1FE1-5007-0020 > mydevlist nbusd_v2 UDID##HP##HSV##5000-1FE1-5007-0020##6005-08B4-0010-5F490000-5000-0B44-0000 nbusd_v3 UDID##HP##HSV##5000-1FE1-5007-0020##6005-08B4-0010-5F49.. nbusd_v4 UDID##HP##HSV##5000-1FE1-5007-0020##6005-08B4-0010-5F49.. 3. Edit the file that you created in step 2 to remove the lines that correspond to UDIDs that are not wanted in the disk group. The format command erases the disk content of the LUN, so be sure to remove any LUNs that may already be in use in NetBackup. Comment out an unwanted UDID line by adding a "#" character at the front of the line. It only examines the lines that contain UDID##. 4. Issue the format command to format all devices in the list. The following example formats all devices that are listed in the mydevlist file that use the VxFS file system. $ nbshareddisk format -udid_file mydevlist -fstype vxfs The format command accepts two values for the fstype qualifier, native which creates a simple UFS partition on Solaris and Linux and an NTFS partition on Window and vxfs which creates a vxfs file system on Solaris and Linux (this value cannot be used on Windows. If vxfs is specified the LUNs are also placed under Veritas Volume Manager control and a VxVM volume is created automatically for each individual LUN.

1.17 Scalability
When configuring a system for SharedDisk storage implementations, there are two dimensions to consider: 1. The number of Media Servers that will be used as Storage Servers to access the SharedDisk Disk Pool 2. The number of SAN LUNs that will be used as disk volumes within a SharedDisk Disk Pool

1.17.1Scalability: Media Server

Ignoring hardware aspects, the number of Media Servers that are configured as Storage Servers for a SharedDisk Disk Pool should be driven by considerations such as: The number of clients to use the Disk Pool I/O bandwidth of the Media Servers (keeping in mind that data has to be transferred in AND out of the Media Server)

The following areas are affected by the number of Storage Servers:

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 17

The device selection becomes more time consuming with a larger number of Storage Servers. This is especially true for multi-copy operations. The disk array enclosure may encounter scalability problems cause by both hardware and software details of their respective implementation. All the Storage Severs sharing a single SharedDisk Disk Pool must be running on the same platform (i.e. all Solaris, all Windows or all Linux). The amount of background monitoring communication traffic will increase, causing more database activities. The communication traffic consists of the following: o o The Disk Polling Service and the Media Performance Monitor Service within nbrmms on the Media Server. The Resource Event Manager service and the Disk Service Manager within nbemm on the EMM server.

Recommendation: A conservative approach is recommended for the number of Storage Servers in a Disk Pool. The number of Storage Servers should be a maximum of 10-20 rather than in the hundreds. With respect to the Disk Pool configuration in NetBackup, it is easy to add or remove Storage Servers from a Disk Pool.

1.17.2Scalability: SAN LUNs

The next thing to consider is the number of SAN LUNs to be used as disk volumes within a SharedDisk Disk Pool. This is based on: Limitations in file system size imposed by the operating system of the Media Server used as Storage Server. For SharedDisk, the maximum size of each LUN that you can format is limited depending on the version of NetBackup, the following table shows the maximum sizes for the versions available at the time of writing: Max LUN Size of Solaris Max LUN Size on Windows 990GB 2047GB 2TB 2047GB 2TB 74TB Max LUN Size on Linux 74TB

NetBackup Version 6.5 6.5.1 6.5.2

The number of Storage Servers within a SharedDisk Disk Pool there should be at least one disk volume per Storage Server. While the assignment of disk volume to Storage Server is not fixed over time, a specific disk volume can only be used at one Storage Server at any point in time. If the total number of disk volumes is smaller than the number of Storage Servers, NetBackup simply runs out of disk volumes to mount. Limitations in the number and size of the SAN LUNs imposed by the array disk enclosure.

Given this lower bound of one disk per Storage Server and the upper bound of the number of LUNs that can be presented from the array, what are the consequences of different disk volume counts? Storage shareability during media and device selection (MDS) NetBackup tries to find the best Storage Server at the time of selection. Having more disk volumes improves the probability of finding an eligible disk volume for that Storage Server resulting in better load balancing across Media Servers. Raw I/O performance I/O contention is more likely to arise when multiple backup jobs are writing to the same disk volume. Having more disk volumes increases the probability that different jobs will use different disk volumes and thus reduce the I/O contention in the disk array.. Image fragmentation given a constant amount of storage space fewer disk volumes will result in less image fragmentation, because images will be less likely to span disk Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 18

volumes. This will reduce the need for multi disk volume restores resulting in a better potential for good restore performance. Media and device selection as with larger Storage Server counts, having large numbers of disk volumes will make the media and device selection process more time consuming. This is especially true for multi copy (in-line copy) operations where different disk volumes need to be selected for the different copies. System monitoring and administration - many system monitoring and administration activities related to Disk Pools have a linear dependency on the number of disk volumes in a Disk Pool and hence the monitoring overhead increase with the volume count

Finding a good balance between these pros and cons for going either towards a larger or a smaller count of disk volumes within a Disk Pool is not easy and the options have been extensively tested.. Recommendation: Based on results from extensive performance modeling it is recommend that the number of disk volumes in a Disk Pool should be configured between 2 and 4 times the number of Storage Servers for LAN Clients. On Solaris and Linux the combination of SharedDisk storage with SAN Clients provides a robust and flexible configuration for the fast backup of large amounts of data. Where FT connections between SAN Clients and Storage Servers are used the number of logical FT connections determines the maximum number of concurrent jobs and thus the maximum number of disk volumes within a SharedDisk Disk Pool that can be used at any one time. For example if two SAN clients are sending two streams of data each over two shared FT connections then the maximum number of disk volumes that can be used is four. Recommendation: Where FT connections between SAN Clients and Storage Servers are used the number of disk volumes within a SharedDisk Disk Pool should not exceed the total number of logical FT connections available between the SAN Clients and the FT Media Servers One final consideration is the time taken to mount the disk volumes. By default NetBackup uses LUN masking to ensure exclusive read/write access for disks in an array. This can result in mount times of the order of 30 seconds to two minutes. It is worth noting, that this delay is not related to the size of the backup job. For longer backup job, the relative effect of this mount delay should be more acceptable. In NetBackup 6.5.1 the option of using SCSI persistent reserve instead of LUN masking was introduced for Solaris Media Servers. This capability was extended to Linux Media Servers in NetBackup 6.5.2 but is not currently available for Windows Media Servers. Using SCSI persistent reserve significantly reduces the mount time and thus the impact it has on overall backup performance. This option is discussed in more detail in the NetBackup 6.5.1 Shared Storage Guide and in tech note 297573. Recommendation: Where LUN masking is used ensure exclusive read/write access for disks in an array the mount time overhead can be reduced by using larger disk volumes and ensuring that only the larger backups are directed to the SharedDisk Storage Units

1.18 State changes in SharedDisk

SharedDisk disk volumes only change their (internal) state upon catastrophic errors. If that occurs, the error needs to get resolved through external means (example: someone unplugged a SAN related cable so that the SAN LUN lost connectivity to the Storage Server; cable should be put back in place). Once the error is resolved, the disk volume status must be reset using the command: $ nbdevconfig -changestate -stype SharedDisk -dp <disk pool> -dv <LUN> -state RESET

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 19

Capacity Management with Flexible Disk Option

1.19 Intelligent Capacity Management and Load Balancing within Disk Pools
Intelligent Capacity Management is used to select the disk volume within a Disk Pool that a given backup will be written to. The disk volume selected will have the most available free space as determined by NetBackup. This does not necessarily mean the most available free space at that point in time. NetBackup calculates the free space on a volume based on two additional factors: Pre-allocated space when a backup is started the amount of space it will occupy is estimated from previous backups of the same type (same policy, type and client). The estimated amount of space is then pre-allocated on the target volume within NetBackup. Thus a disk volume with a backup in progress may appear to have 200 Gbytes free but, because the backup has an estimated size of 100 Gbytes, NetBackup will only consider it to have 100 Gbytes free. Potential free space NetBackup also take account of space that could be released by removing existing backup images that have expired. Thus a disk volume may appear to have 200 Gbytes free but, because it contains 100 Gbytes of expired backup data, NetBackup will consider it to have 300 Gbytes free.

1.19.1How does NetBackup calculate how much space to pre-allocate?

For backup jobs that have run previously the amount of space that is pre-allocated is based on the size of the last backup of the same type made by the same policy on the same client and an overhead of 20% is added. Backup jobs with no previous history the capacity of the disk volume above the high watermark is used (e.g. for a 2TByte volume with a 98% high water mark, 40Gbytes is pre-allocated). For duplication jobs the size of the source image is known and the pre-allocation uses that value. Jobs will never fail due to estimated size. If no disk volume has sufficient free space the volume with the most available space is used.

1.20 Water Marks and Capacity Based Retention

High and low water marks are set on the Disk Pool but apply to each disk volume within the Disk Pool. The water marks are expressed as a percentage of the space on the disk volume. As the disk volumes within a Disk Pool fill up checks are made that the high water mark is not being exceeded. These checks are made at one minute intervals and also whenever a backup job is initiated. The Storage Lifecycle Policies feature in NetBackup allows Flexible Disk Option storage to be used with Capacity Manager Retention. Under Capacity Managed Retention a target value is set for how long the backup images should remain on disk if there is sufficient space available. The backups remain on disk until the disk volume high water mark is reached and are then removed until the low water mark is reached. The oldest images that have been successfully duplicated to another location are automatically removed according to the following rules: 1) Images that are past their try to keep time (the desired cache period specified in the Storage Destination) are expired based on data classification with the images belonging to the lowest data classification being expired first. 2) A check is made after image is removed to see if the disk volume low water mark has been reached. 3) Image expiration continues until either the disk volume low water mark is reached or there are no remaining images that have passed their try to keep time. 4) If the low water mark is not reached after removing all the images that have passed their 'try to keep' date then the expiration process will continue to remove images that have not yet reached their 'try to keep' date. However this will only happen if: Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 20

a. The total Disk Pool is still above the high water mark b. (In the case of a SharedDisk Disk Pool only) the number of disk volumes that are above high water mark is greater than the number of Media Servers sharing the Disk Pool If an image being removed spans two or more disk volumes within the Disk Pool all fragments of the image on all disk volumes are removed.

1.21 Increasing and decreasing the size of disk pools

The amount of space in a disk pool can be increased or decreased simply by adding or removing disks, no logical volume manager is required.

1.21.1Adding disks to a disk pool

To add disks to an existing disk pool first create a new disk pool including the disks you want to add to the existing pool and then merge the two pools using the command nbdevconfig mergedps. This command can only be executed when the both disk pools are off-line so the nbdevconfig changestate command must be used to set the state of both pools to DOWN before they are merged and to set the state of the resulting pool to UP after the merge is complete.

1.21.2Removing disks from a disk pool

Disks can be removed from a disk pool using the nbdevconfig deletedv command. Both the disk pool and disk volume must be off-line at the time so the nbdevconfig changestate command must be used to mark both the pool and disk as DOWN and the disk itself must be empty when it is deleted from the pool. Note: Any backup data on the disk when it is removed from the pool will be lost so enure the disk is empty before issuing this command.

Storage Unit configurations

Storage Units that are used with Flexible Disk Option are based on the Disk Pool rather than individual disks. The high and low water marks are applied at the disk pool level and thus do not need to be applied to individual Storage Units as they are in BasicDisk configurations.

1.22 Storage Unit Groups and Media Server Load Balancing

One key change from the traditional model of Storage Units that occurs when Flexible Disk Option is used is that for Disk Pools that are shared between multiple Media Servers a single Storage Unit can be associated with a number of Media Servers. Where a Storage Unit is configured with multiple Media Servers load balancing across the Media Servers (based on CPU utilization) will occur automatically and a backup job will always be directed to the least busy Media Server. Multiple Media Servers can be selected for Storage Units that use SharedDisk Disk Pools and AdvancedDisk Disk Pools that are configured for sharing. Storage Units that used AdvancedDisk Disk Pools that are associated with a single node (i.e. do not include shared mount points) can be configured into Storage Unit Groups. When a Storage Unit Group is defined Media Server load balancing can be enabled by simply selecting the Load Balance option or, if the bpstuadd command is being used from the CLI, just specifying the qualifier -sm 4.

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 21

Figure 9: Load Balancing in Storage Unit Groups This Load Balance option is not available for BasicDisk Storage Units. Within a Disk Pool load balancing between the disks occurs automatically with backups always being directed to the disk with the most available space expressed as a percentage of the total space.

1.23 Multiple Storage Units sharing the same Disk Pool

A single AdvancedDisk or SharedDisk Disk Pool can support multiple Storage Units. There is no relationship between Storage Units and disk volumes in a configuration of this kind, any Storage Unit will use any disk volume within the Disk Pool depending on the available space. This means that there is generally little benefit in creating multiple Storage Units that use the same Disk Pool. There are a number of exceptions to this rule which are discussed in the following sections:

1.23.1Limiting the number of jobs for a particular data classification.

The Maximum Concurrent Jobs setting on the Storage Units limits the number of jobs using each Storage Unit and thus prioritize one set of jobs (using a Storage Unit that allows a large number of concurrent jobs) over another (using a Storage Unit that allows a smaller number of concurrent jobs). For example a large AdvancedDisk or Disk Pool can be configured as two Storage Units called Gold_DSU and Silver_DSU. Gold_DSU has maximum concurrent jobs set to 10 and Silver_DSU has maximum concurrent jobs set to 5. Backups are configured so that the more critical backups run to Gold_DSU and the less critical run to Silver_DSU. All the backups have a start window between 9:00pm and 5:00am and have the same priority. As the Gold_DSU allows twice as many Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 22

concurrent jobs as Silver_DSU, more of critical backups will run at once and thus they will complete first. Taking this a step further, suppose you have a SharedDisk Disk Pool with two fast fiber connected Media Servers, and three ordinary Media Servers. You could configure a Platinum_DSU to use the two fast servers, with a higher Maximum Concurrent Jobs, and a Gold_DSU and a Bronze_DSU that use the ordinary Media Servers. The following illustration attempts to show how this could lay out:

Figure 10 - Jobs prioritized by concurrency

1.23.2Balancing Backup and Duplication Jobs

In a similar way different Storage Units with different concurrent job counts can be used for pirmary backups and duplicated backups.

1.23.3Using SharedDisk Disk Pools with SAN Media Servers

For operational purposes a SharedDisk Disk Pool can be considered as equivalent to a shared tape drive and the Storage Units that use that Disk Pool can be considered equivalent to individual tapes using a tape drive. In the same way that individual tapes may be dedicated to a single Media Server or shared using the media sharing option so a Storage Unit may be dedicated to a single Media Server or shared among many Media Servers in a SharedDisk configuration. In a SharedDisk configuration Media Server load balancing is carried out once the Storage Unit is selected (rather than before the Storage Unit is selected) and the Storage Unit is then presented on the most appropriate Media Server that is configured to use it. Thus a backup may be directed to any Media Server that shares the Storage Unit and the particular Media Server used will be determined by NetBackup at the time the backup executes. The function of a SAN Media Server, as opposed to a general Media Server, is to back up a local application directly to the backup media. In general SAN Media Servers are deployed in configurations where, either due to data volumes or network bandwidth, it is undesirable to send backup data across a network connection to a remote Media Server.

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 23

For this reason while SAN Media Server may take advantage of the volumes in a SharedDisk Disk Pool it should not share its Storage Unit with any other Media Server as this could result in backups running over the LAN to the remote server rather than going direct to disk. To avoid this problem the Storage Unit used by the SAN Media Server should be set up using the following attributes: Storage Unit Type: Disk Disk Type: SharedDisk Media Server: Use only the following Media Servers Only the SAN Media Server (or, in the case of a cluster, the virtual server name) should be selected from the list of Media Servers displayed. The following command line instruction may be used to add the Storage Unit: # bpstuadd label <STU name> -dp <ShareDisk Disk Pool> -dt 6 hostlist <SAN Media Server>

Best Practices for Disk Layout with the Flexible Disk Option version 2.0 Page 24