Status Code: 219

The required Storage Unit is unavailable

This error occurs when the NetBackup scheduler process ( bpsched) cannot find the specified Storage Units available for use. The policy or schedule for the backup requires a specific Storage Unit, which is currently unavailable. This error also occurs for other attempts to use the Storage Unit within the current backup session. The first priority should be to determine which Media Server and Storage Unit is writing the backup that fails with the Status 219.
Enable/view the following logs : Unix: Syslog and Admin Messages /usr/openv /Netbackup/logs/bptm Windows: Event Viewer System and application logs <install_path>\Netbackup\logs \bptm Also, view the Problems Report in the Administration Console .

Status Code 219

Are tape drives up ?

NO

YES

Storage Unit Configuration Issue Section 1 Drives are down Section 3 Policy configuration Section 2

Verify connection to bpcd Section 4

Enable/view the following logs :

Master Server
Unix: /usr/openv /Netbackup/logs / bpsched Windows: <install_path>\Netbackup\ logs\ bpsched

Name Resolution issues Section 5

Media Server
Unix: /usr/openv /Netbackup/logs/bpcd Windows: <install_path>\Netbackup\ logs\ bpcd Client: None

Status Code: 219 The required Storage Unit is unavailable

Page 1 of 13

Table of Contents
1 Storage Unit(s) are not configured correctly...............................................................................3 1.1 Maximum Concurrent Drives / Jobs.....................................................................................3 1.2 Media Server Specification.................................................................................................. 3 1.3 Robot Number..................................................................................................................... 3 1.4 On Demand Only................................................................................................................. 3 1.5 Multiplexing.......................................................................................................................... 3 2 Policy configuration.................................................................................................................... 4 3 Drives are down ........................................................................................................................ 5 3.1 Checking drive status and bringing drives up.......................................................................5 3.1.1 Checking and upping the drives from the GUI:...............................................................5 3.1.2 Checking and upping the drives from the Command Line:.............................................6 3.2.1 OS configuration issues................................................................................................ 7 3.2.2 SCSI Path issues........................................................................................................... 7 3.3.3 Device issues................................................................................................................ 8 4Verify connection to bpcd on the media server...........................................................................10 4.1 Verify that bpcd is listening................................................................................................10 4.2 Test bpcd connection with Telnet.......................................................................................10 5 Name Resolution between Master and Media Server..............................................................12 6 Links......................................................................................................................................... 13

Status Code: 219 The required Storage Unit is unavailable

Page 2 of 13

1 Storage Unit(s) are not configured correctly

In the Windows or Java Administration Console, identify the Job ID within the Activity Monitor that fails with a Status 219. The Storage Unit used for the job is also indicated here. Once the Storage Unit name has been obtained, view the Storage Units properties by doing the following in the Administration Console: 1. In the left pane, ensure that NetBackup Management is expanded. 2. Click Storage Units. 3. In the right pane, double-click on the Storage Unit name in question to pull up its properties. Below are some common Storage Unit properties that, if not configured properly, can cause a Status 219.

1.1 Maximum Concurrent Drives / Jobs

Verify that no Storage Units have their Maximum concurrent jobs attribute set to 0 (for Disk Storage Units) or Maximum concurrent drives used for backup attribute set to 0 (for Media Manager Storage Units).

1.2 Media Server Specification

Verify that the Media Server name in the Storage Unit configuration is correct. The actual name of the Storage Unit may specify a certain Media Server, but the Media Server configured within the Storage Unit may be different. Ensure that these match.

1.3 Robot Number

Verify that the Robot number in the Storage Unit configuration matches the Media Manager device configuration within Media and Device Management -> Devices in the Administrator Console.

1.4 On Demand Only

If the "On Demand Only" attribute is checked on a Storage Unit, please note that any policies that do not target that Storage Unit directly will not use that Storage Unit. For example, if a policy is set to use Any Available Storage Unit, it will not utilize a Storage Unit that has the On Demand Only box checked.

1.5 Multiplexing
In an SSO environment, when drives are shared, a job will typically requeue with a status code 134 if a drive is over-committed. In some situations, due to timing, this scenario may result in a Status Code 219 (if a specific Storage Unit is designated in the policy). To help minimize this, try increasing the Maximum Multiplexing per drive within the Storage Unit. Allowing more streams per drive will increase drive availability (This also applies to non-SSO environments). Note that Multiplexing can result in significantly longer restore times. See the Media Manager System Administrators Guide for more information on this feature.

Status Code: 219 The required Storage Unit is unavailable

Page 3 of 13

2 Policy configuration
As mentioned above, Storage Units are targeted from a policy. However, it is possible to override the policys Storage Unit designation from within the schedule itself. If a different Storage Unit is being used than expected, ensure that no schedules within the policy are set to override the Storage Unit.

Figure 1. The Override policy storage unit option as shown in the Windows Administration Console.

Status Code: 219 The required Storage Unit is unavailable

Page 4 of 13

3 Drives are down

If all drives are down within a Storage Unit, that Storage Unit is considered unavailable and a Status 219 will be seen. Hardware problems (such as Tape Drive, Robot or SCSI/HBA problems) can result in down tape drives. This means that NetBackup has fewer resources with which to complete the backup jobs. Use the NetBackup Device Monitor within the Windows or Java Administration Console, or the vmoprcmd command to view the status of all the Tape Drives. If there are any down tape drives, examine the operating systems error logs to determine why the drives were downed. Note that in Shared Storage environments (SSO), a tape drive can be DOWN for some hosts, but not others. Running the Problems Report in the NetBackup Administration console may also assist in diagnosis. Additionally, it may be necessary to reconfigure, or verify the configuration of devices. See the Media Manager and Media Manager Device Configuration Guides for more information on how to proceed with this. If all drives are down within a Storage Unit, the bpsched log will show something similar to the following:
11:29:30.600 [11372] <2> get_num_avail_drives: NUM UP 0 0 0 0 0 0 11:29:30.601 [11372] <4> log_in_errorDB: no drives up on Storage Unit rumpunch-dlt> 11:29:30.602 [11372] <2> ?: available drives = 0, shared drives = 0, allow_mult_ret= 0 11:29:32.607 [11372] <2> ?: /usr/openv/netbackup/bin/backup_exit_notify rumpunch policya full FULL 213 0 &

3.1 Checking drive status and bringing drives up.

The drive status can be viewed in the Java or Windows NetBackup Administration Console in the Device Monitor, or from the command line. Check the Control field for the drive and confirm that it is not DOWN. Also, drives in a robot should not be listed as AVR in the control field, as this indicates a possible robot communication issue.

3.1.1 Checking and upping the drives from the GUI:

If the drive status is DOWN, the drive can be brought up in the Java GUI or NetBackup Administration Console in the Device Monitor. Right-click on the drive and choose to up the drive (Figure 2). If the drive comes up, then attempt to run another backup to this Storage Unit..

Status Code: 219 The required Storage Unit is unavailable

Page 5 of 13

Figure 2. A down drive is being upped.

3.1.2 Checking and upping the drives from the Command Line:
To determine the status of the drives, run the vmoprcmd command below on the media server in question: UNIX: /usr/openv/volmgr/bin/vmoprcmd d Windows: <install_path\VERITAS\volmgr\bin\vmoprcmd d Drive status is DOWN as shown by the vmoprcmd command:
Drv Type Control Wr.Enbl. ReqId 0 4mm DOWN DRIVE STATUS User Label RecMID ExtMID Ready No -

Up the drives by running the following: UNIX: /usr/openv/volmgr/bin/vmoprcmd up <drive_index> Windows: <install_path\VERITAS\volmgr\bin\vmoprcmd up <drive_index>

3.2 Drives do not come up

If the drive does not come up, the operating system error logs should be reviewed for errors. The system error logs are located at: Solaris: /var/adm/messages HP-UX: /var/adm/syslog/syslog.log AIX: The syslog log is not turned on by default. Check /etc/syslog.conf file for location of
Status Code: 219 The required Storage Unit is unavailable

Page 6 of 13

*.debug and *.error log file. If it is commented out (if there is a # in front of *.debug and *.error), then syslog is not recording errors at the OS level. To troubleshoot drive issues, turn on syslog logging. The AIXs errpt command can also be used to look for hardware errors: # errpt a (See Appendix for more detailed AIX logging instructions) Linux: /var/log/syslog Windows: 1. Right-Click My Computer. 2. Select Manage. 3. Expand System Tools and then Event Viewer. 4. View the Event Viewer Application and System logs for more details on the failure. As an application, NetBackup has no direct access to a device, instead relying on the operating system (OS) to handle any communication with the device. This means that during a write operation, NetBackup asks the OS to write to the device and report the success or failure of that operation. If the drives are down, there are 3 likely causes. These causes are described below:

3.2.1 OS configuration issues

There are two likely candidates for problems at the OS level; tape drivers and HBA drivers. Tape drivers and configuration Contact the OS or hardware vendor to ensure that an up-to-date driver is installed for the tape device. Contact the hardware vendor to ensure that the driver is configured correctly for the device. Host Bus Adapter (HBA) driver/configuration/firmware Make sure the HBA has an updated driver and firmware and that the configuration is correct.

Example of errors from a UNIX system (syslog, messages):

May 14 16:25:51 server unix: WARNING: /pci@6,4000/scsi@2 (glm12): May 14 16:25:51 server unix: Target 1 reducing sync. transfer rate May 14 16:25:51 server unix: WARNING: ID[SUNWpd.glm.sync_wide_backoff.6014]

3.2.2 SCSI Path issues

Make sure that any devices in the SCSI path to the target device are operating correctly, typically SAN hardware (switches, bridges, etc.). Check for faulty cables, or improper SCSI termination. SAN firmware/configuration Make sure that any switches or bridges are operating correctly, and that they have the latest software updates installed.

Example of SAN/SCSI communication errors from a UNIX system (syslog, messages):

May 14 16:25:51 server unix: WARNING: /pci@6,4000/scsi@2/st@1,0 (st85): May 14 16:25:51 server unix: Error for Command: write Error Level: Fatal May 14 16:25:51 server unix: Requested Block: 1531 Error Block: 1531 Status Code: 219 The required Storage Unit is unavailable

Page 7 of 13

May 14 16:25:51 server unix: Vendor: QUANTUM Serial Number: qj 6 i O May 14 16:25:51 server unix: Sense Key: Aborted Command May 14 16:25:51 server unix: ASC: 0x47 (scsi parity error), ASCQ: 0x0, FRU: 0x0 Jun 20 02:20:09 server scsi: [ID 107833 kern.warning] WARNING: /pci@8,600000/JNI,FCR@2/st@14,0 (st15): Jun 20 02:20:09 server SCSI transport failed: reason 'tran_err': giving up Jun 20 02:26:09 server bptm[29663]: [ID 832037 daemon.error] scsi command failed, may be timeout, scsi_pkt.us_reason = 3 Jun 20 02:26:42 server jnic146x: [ID 362195 kern.notice] jnic146x1: Link not operational. Performing reset. Jun 20 02:26:42 server jnic146x: [ID 133166 kern.notice] jnic146x1: Link Down Dec 18 21:11:59 server vmunix: 0/1/0/0: Unable to access previously accessed device at nport ID 0x11900.

Example of SCSI communication errors from a Windows Event Viewer System log:
20040830 10:36:30 aic78xx E9 NA The device, \Device\Scsi\aic78xx1, did not respond within the timeout period. 20040830 10:46:33 aic78xx E9 NA The device, \Device\Scsi\aic78xx1, did not respond within the timeout period.

3.3.3 Device issues

Here are some things to check for on the tape device. Firmware Check the Device Compatibility list to see what firmware version VERITAS engineering has tested with the device. Contact the hardware vendor to obtain any updates. Bad device Have the hardware vendor check the device to verify that it is operating correctly. Bad media Check to see if the same piece of media has been causing problems. Dirty Drive Check to verify that the drives are being cleaned. Environmental Check power to the device, proper cooling, dust, etc. Example from the UNIX /usr/openv/netbackup/logs/bptm/log.<date> file:
<16> write_data: cannot write image to media id XXXXXX, drive index 2, I/O error <16> io_ioctl: ioctl (MTWEOF) failed on media id XXXXXX, drive index 0, The request could not be performed because of an I/O device error. (../bptm.c.12720) <2> io_ioctl: MTWEOF failed during error recovery, I/O error <16> write_tar_image: cannot write image to /tmp/sync_XXXXXX, Error 0 <16> bpbackupdb: NB database backup to media id XXXXXX FAILED bpbackupdb: EXIT status = 84 <16> write_tar_image: ndmp write of bpbackupdb image to F:\VERITAS\NetBackup\temp\sync_XXXXXX failed, error code 13 (NDMP_EOM_ERR) bpbackupdb: EXIT status = 84 <16> io_ioctl: ioctl (MTWEOF) failed on media id XXXXXX, drive index 0, Data error (cyclic redundancy check). (..\bptm.c.16728) Status Code: 219 The required Storage Unit is unavailable

Page 8 of 13

Example of tape drive and media errors from a UNIX system (syslog, messages):
May 15 16:41:40 Fatal May 15 16:41:40 2181 May 15 16:41:40 O May 15 16:41:40 May 15 16:41:40 FRU: 0x0 server unix: Error for Command: write Error Level: server unix: Requested Block: 2181 Error Block: server unix: Vendor: QUANTUM Serial Number: qj 6 i server unix: Sense Key: Media Error server unix: ASC: 0xc (write error), ASCQ: 0x0,

Nov 5 21:08:47 server avrd[21163]: Tape drive QUANTUMDLT70001 (device 1, /dev/rmt/c9t0d0BESTnb) needs cleaning. Attempting to auto-clean...

Example of tape drive and media errors from a Windows Event Viewer System log:
20040804 22:47:53 dlttape-VRTS E7 NA The device, \Device\Tape2, has a bad block. 20040806 17:52:32 dlttape-VRTS E11 NA The driver detected a controller error on \Device\Tape0. 20040806 17:52:47 dlttape-VRTS E11 NA The driver detected a controller error on \Device\Tape0.

Status Code: 219 The required Storage Unit is unavailable

Page 9 of 13

4 Verify connection to bpcd on the media server

If bpsched on the Master Server cannot communicate with bpcd on the Media Server, bpsched sets the number of available drives in that Storage Unit to 0. Any further backups to that Storage Unit during the backup session will fail. The number of available drives remains at 0 until the scheduler is initialized again.

4.1 Verify that bpcd is listening

If bpcd is not listening, the bpsched log will show something similar to:
00:09:10 [7703] 00:09:10 [7703] 00:09:10 [7703] 00:09:10 [7703] 00:09:10 [7703] 204 00:09:10 [7703] 00:09:10 [7703] 00:09:10 [7703] Units available <16> getsockconnected: exceeded timeout of 30 seconds <2> getsockconnected: sockfd:-1 timo:32 <16> bpcr_connect: Can't connect to client server1 <16> start_bptm: connection refused by host server11 <16> get_stunits: get_num_avail_drives failed with stat <4> get_db_info: no available Storage Units <8> bpsched_main: failed getting database information <16> log_in_errorDB: scheduler exiting - no Storage for use (213)

Run the command below on the Media Server to check if the bpcd port is listening: Windows: netstat -a > c:\netstat.txt UNIX: netstat a > /tmp/netstat.out The text file that gets created should list processes that are running ( bpcd, vnetd, vopied, bpjava-msvc). Search this file to determine if bpcd is in a listen or listening status: Windows: TCP hostname:bpcd hostname.domain.com:0 LISTENING UNIX: *bpcd *.* 0 0 49152 LISTEN If bpcd isnt listening, it may need to be stopped and restarted. On Windows, bpcd can be cycled by stopping and restarting the Client service. If bpcd is listening, yet drives are up and a 219 is still encountered, create a bpsched and bpcd activity log directory and retry the operation. Check the resulting activity logs for records of an earlier failure.

4.2 Test bpcd connection with Telnet

A telnet command can be used from the Master Server to test the connection to the Media Servers bpcd process. Command syntax:
telnet <media server name or IP address> bpcd

Successful telnet output example from UNIX: # telnet alaska bpcd Trying 10.10.100.20... Connected to alaska.min.veritas.com. Escape character is '^]'.
Status Code: 219 The required Storage Unit is unavailable

Page 10 of 13

Hit the Ctrl ] and then type quit to end Unsuccessful telnet output example from UNIX: # telnet alaska bpcd Trying 10.10.100.20... telnet: Unable to connect to remote host: Connection refused Successful telnet output example from Windows: telnet alaska bpcd < If successful no displayed messages will be returned > To stop telnet: hold down Ctrl key and press the ] key, then release and type quit to end telnet session. Unsuccessful telnet output example from Windows: Connecting to Alaska. . .Could not open a connection to host on port 13782 : Connect failed If a telnet test fails, ensure again that bpcd process(this is the client service on Windows) is running and listening. If so, it may be necessary to analyze the network configuration to determine why the bpcd port (13782) can not be reached.

Status Code: 219 The required Storage Unit is unavailable

Page 11 of 13

5 Name Resolution between Master and Media Server

Name resolution should be closely analyzed if drives are up and bpcd is listening, but the Status 219 is still seen.

Using bpclntcmd
bpclntcmd is a useful utility that can be run between the Master and Media Server. It will help determine if name resolution is working properly from NetBackups perspective. It can be run from the Media Server to the Master, or from the Master Server to the Media Server. Windows command location: %install_path%\VERITAS\NetBackup\bin\ UNIX command location: /usr/openv/netbackup/bin Switches and variations:
bpclntcmd bpclntcmd bpclntcmd bpclntcmd bpclntcmd bpclntcmd -pn -self -hn <hostname_of_masterserver> -hn <hostname_of_mediaserver> -ip <ip_of_masterserver> -ip <ip_of_mediaserver>

The goal of these commands is to make sure the hostname is consistently seen the same way after each command. Below is an explanation of what each switch does: -pn - The client process on the host connects to the Master Server and asks the question "Who am I?". The second line of the output is the result. This is how the client process on the host is being seen by the Master Server. -self - Checks how this host can be resolved. Ideally, there should be only 1 unique hostname and 1 unique IP address. -hn - Checks the given hostname and returns an IP. -ip - Checks the given IP and returns a hostname. If there is any inconsistency or errors in the results of these commands, correct the hostname to IP resolution by either editing the local hosts table or by updating the DNS database.

Status Code: 219 The required Storage Unit is unavailable

Page 12 of 13

6 Links
Click here to Search for other documents on Status Code 219
Click below to perform a search on the following relevant items:
Storage unit

Status Code: 219 The required Storage Unit is unavailable

Page 13 of 13