SAS Disk Shelves Universal SAS and ACP Cabling Guide

NetApp, Inc. 495 East Java Drive Sunnyvale, CA 94089 USA Telephone: +1 (408) 822-6000 Fax: +1 (408) 822-4501 Support telephone: +1 (888) 4-NETAPP Documentation comments: doccomments@netapp.com Information Web: www.netapp.com Part number: 215-05500_A0 September 2010

Copyright and trademark information

Copyright information

Copyright 19942010 NetApp, Inc. All rights reserved. Printed in the U.S.A. No part of this document covered by copyright may be reproduced in any form or by any means graphic, electronic, or mechanical, including photocopying, recording, taping, or storage in an electronic retrieval systemwithout prior written permission of the copyright owner. NetApp reserves the right to change any products described herein at any time, and without notice. NetApp assumes no responsibility or liability arising from the use of products described herein, except as expressly agreed to in writing by NetApp. The use or purchase of this product does not convey a license under any patent rights, trademark rights, or any other intellectual property rights of NetApp. The product described in this manual may be protected by one or more U.S.A. patents, foreign patents, or pending applications. RESTRICTED RIGHTS LEGEND: Use, duplication, or disclosure by the government is subject to restrictions as set forth in subparagraph (c)(1)(ii) of the Rights in Technical Data and Computer Software clause at DFARS 252.277-7103 (October 1988) and FAR 52-227-19 (June 1987).

Trademark information

NetApp; the NetApp logo; the Network Appliance logo; Bycast; Cryptainer; Cryptoshred; DataFabric; Data ONTAP; Decru; Decru DataFort; FAServer; FilerView; FlexCache; FlexClone; FlexShare; FlexVol; FPolicy; gFiler; Go further, faster; Manage ONTAP; MultiStore; NearStore; NetCache; NOW (NetApp on the Web); ONTAPI; RAID-DP; SANscreen; SecureShare; Simulate ONTAP; SnapCopy; SnapDrive; SnapLock; SnapManager; SnapMirror; SnapMover; SnapRestore; SnapValidator; SnapVault; Spinnaker Networks; Spinnaker Networks logo; SpinAccess; SpinCluster; SpinFlex; SpinFS; SpinHA; SpinMove; SpinServer; SpinStor; StorageGRID; StoreVault; SyncMirror; Topio; vFiler; VFM; and WAFL are registered trademarks of NetApp, Inc. in the U.S.A. and/or other countries. Network Appliance, Snapshot, and The evolution of storage are trademarks of NetApp, Inc. in the U.S.A. and/or other countries and registered trademarks in some other countries. The StoreVault logo, ApplianceWatch, ApplianceWatch PRO, ASUP, AutoSupport, ComplianceClock, DataFort, Data Motion, FlexScale, FlexSuite, Lifetime Key Management, LockVault, NOW, MetroCluster, OpenKey, ReplicatorX, SecureAdmin, Shadow Tape, SnapDirector, SnapFilter, SnapMigrator, SnapSuite, Tech OnTap, Virtual File Manager, VPolicy, and Web Filer are trademarks of NetApp, Inc. in the U.S.A. and other countries. Get Successful and Select are service marks of NetApp, Inc. in the U.S.A. IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both. A complete and current list of other IBM trademarks is available on the Web at www.ibm.com/legal/copytrade.shtml. Apple is a registered trademark and QuickTime is a trademark of Apple, Inc. in the U.S.A. and/or other countries. Microsoft is a registered trademark and Windows Media is a trademark of Microsoft Corporation in the U.S.A. and/or other countries. RealAudio, RealNetworks, RealPlayer, RealSystem, RealText, and RealVideo are registered trademarks and RealMedia, RealProxy, and SureStream are trademarks of RealNetworks, Inc. in the U.S.A. and/or other countries. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such. NetApp, Inc. is a licensee of the CompactFlash and CF Logo trademarks. NetApp, Inc. NetCache is certified RealSystem compatible.

ii

Copyright and trademark information

Table of Contents
Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v

Chapter 1

Cabling Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Chapter 2

SAS Cabling for New Storage Systems . . . . . . . . . . . . . . . . . . . . 3 SAS cabling rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Cabling SAS connections. . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 SAS cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Chapter 3

ACP Cabling for New Storage Systems . . . . . . . . . . . . . . . . . . . 25 ACP cabling rules. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Cabling ACP connections . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 ACP cabling examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Appendix A

Terms and Symbols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Appendix B

Document Update Record . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Table of Contents

iii

iv

Table of Contents

Preface
About this guide This guide provides information about cabling SAS disk shelves for new storage system installations. This guide does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at now.netapp.com for your disk shelf model. The disk shelf illustrations show a generic SAS disk shelf, which has the two IOMs side-by-side. No power supplies are shown.

Audience

This guide is for system administrators who are familiar with operating systems that run on the storage systems clients, such as UNIX, Windows 95, Windows NT, Windows 2000, and Windows XP. The procedures in this guide describe replacement, upgrade, and maintenance tasks for personnel with the following skills and experience:

Working familiarity with small computer system hardware and operation Basic understanding of common networking concepts and practices Working familiarity with accepted tools and procedures for installing and operating sensitive electronic equipment

Formatting conventions

The following table lists the kinds of formatting this guide uses to identify special information. Formatting convention Italic type

Type of information

Words or characters that require special attention. File names and path names. Man page names. Book titles in cross-references. Placeholders for information you must supply. For example, if the guide says to enter the command arp -d hostname, you enter the characters arp -d followed by the actual name of the host.

Preface

Formatting convention
Monospaced

Type of information

Command and daemon names. Information displayed on the system console or other computer monitors. The contents of files.

font

Bold monospace font

Words or characters you type. What you type is always shown in lowercase letters, unless you must type it in uppercase letters for it to work properly.

Special messages

This guide contains special messages that are described as follows: Note A note contains important information that helps you install or operate the system efficiently. Attention An attention notice contains instructions that you must follow to avoid a system crash, or loss of data, or damage to the equipment. DANGER A danger notice warns you of conditions or procedures that can result in death or severe personal injury. CAUTION A caution notice warns you of conditions or procedures that can cause personal injury that is neither lethal nor extremely hazardous.

vi

Preface

Cabling Requirements
Before you begin

To cable a new storage system with SAS disk drives, your storage system must meet the following requirements:

Your storage system must be running the minimum software release required to support your model of SAS disk shelves. See the Installation and Service Guide for your particular disk shelf model or the System Configuration Guide at now.netapp.com.

If your storage system was not shipped in a system cabinet, the controllers and disk shelves must already be installed in a rack. If your storage system will be using one or more PCIe SAS HBAs, they must already be installed and available for cabling the disk shelves. If you are cabling for the ACP capability and your storage system does not have a native onboard ACP port, you must be prepared to designate a network interface for each controller at system setup. If you are using a four-port Ethernet card for this purpose, this card must already be installed and available in each controller.

Chapter 1: Cabling Requirements

Cabling Requirements

SAS Cabling for New Storage Systems

About this chapter

This chapter provides information about cabling the disk shelf SAS connections for a new storage system installation. This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at now.netapp.com for your disk shelf model. Note References to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.

Topics in this chapter

This chapter discusses the following topics:

SAS cabling rules on page 4 Cabling SAS connections on page 10 SAS cabling examples on page 15

Chapter 2: SAS Cabling for New Storage Systems

SAS cabling rules

Topics in this section

The following topics are covered in this section:

General SAS cabling rules on page 4 Onboard SAS port rule on page 5 Shelf-to-shelf connection rule on page 6 Controller-to-shelf connection rules on page 6

General SAS cabling rules

The following SAS cabling rules apply to all supported storage systems using SAS HBAs and on board SAS ports: Supported configurations: Multipath HA is the supported configuration, with the following exceptions:

For single-controller configurations, 2040 storage systems use singlepath and all other supported platforms use dual-path. 2040 systems in an HA pair configuration use single-path HA.

SAS cables used: QSFP to QSFP SAS cables are used to connect disk shelves together and to connect disk shelves to the SAS ports on the controller.

QSFP to mini-SAS cables are used for 2050 configurations only, and connect disk shelves to the dual-port mini-SAS HBAs on the controller.

SAS cabling rules

SAS port designations: All SAS ports on the disk shelf are designated by a square symbol or a circle symbol. The following illustration shows square and circle labeling on the IOM ports. Note SAS connectivity does not use the concept of In ports and Out ports.
IOM A ACP SAS ACP IOM B SAS

Onboard SAS port rule

To ensure that the SAS cabling rules apply to all SAS HBAs on a controller, whether in a physical PCI slot (slot 1-N) or on board the base controller, an on board HBA is defined as occupying PCI slot 0. Each port is named with sequentially increasing letters (0a, 0b, 0c, and so on). For example, the following illustration shows a controller with one onboard SAS HBA in slot 0 and three SAS HBAs in physical PCI slots 1, 2, and 3.
Slot 0 Slot 1 A B A B C D Slot 2 A B C D Slot 3 A B C D

Exception: The only exception is the 2040 storage system. Treat the onboard SAS port labeled 0d as if it were port 0a. Treating the port as 0a enables you to apply all the SAS cabling rules. (Data ONTAP continues to recognize this port as 0d.)
2040 controller
0a

0d

Chapter 2: SAS Cabling for New Storage Systems

Shelf-to-shelf connection rule

All the disk shelves in a stack are daisy-chained when you have more than one disk shelf in a stack.

IOM A circle port is connected to the next IOM A square port. IOM B circle port is connected to the next IOM B square port.
Stack of three shelves

ACP
Top shelf

SAS

ACP

SAS

IOM A

IOM B

Bottom shelf

Controller-to-shelf connection rules

Each controller connects to each stack of disk shelves in the system through the controller SAS ports, which can be A, B, C, and D, (and can be on a SAS HBA in a physical PCI slot [slot 1-N] or on board the base controller). For quad-port SAS HBAs, the controller-to-shelf connection rules ensure resiliency for the storage system based on the ASIC chip design. Ports A and B are on one ASIC chip, and ports C and D are on a second ASIC chip. By using ports A and C to connect to the top shelf in each stack, and using ports B and D to connect to the bottom shelf in each stack, the controllers maintains connectivity to the disk shelves if an ASIC chip fails. The following illustration shows a quad-port SAS HBA with the two ASIC chips and their designated ports.
Quad-port SAS HBA
ASIC chip A port B port PCI-e chip ASIC chip C port D port

SAS cabling rules

SAS port A and C connections: SAS ports A and C on a controller are connected to the stacks of disk shelves in this way:

SAS ports A and C are connected to the top disk shelf square port in each stack of disk shelves.

The A and C ports on controller 1/A connect to IOM A. The A and C ports on controller 2/B connect to IOM B.
Controller 2/B
Slot 1

Controller 1/A
Slot 1

A B C D

A B C D

Stack 1

Stack 2

ACP

SAS

ACP

SAS

ACP

SAS

ACP

SAS
Top shelf

IOM A

IOM B

IOM A

IOM B

Bottom shelf

SAS ports A and C are connected to the disk shelf stacks in sequential order of the HBA slots (slot 0, slot 1, slot 2, and so on). For example, a controller with slots 1, 2, and 3 occupied with quad-port SAS HBAs is cabled in order of HBAs and ports 1a, 2a, 3a, 1c, 2c, 3c, as shown below.
Controller 1/A
Slot 1 Slot 2 Slot 3

A B C D

A B C D

A B C D

Chapter 2: SAS Cabling for New Storage Systems

SAS port B and D connections: SAS ports B and D on the controller are connected to the stacks of disk shelves in this way:

Ports B and D are connected to the bottom disk shelf circle port in each stack of disk shelves.

The B and D ports on controller 1/A connect to IOM B. The B and D ports on controller 2/B connect to IOM A.
Controller 1/A
Slot 1 Slot 1

Controller 2/B
A B C D

A B C D

Stack 1

Stack 2

ACP
Top shelf

SAS

ACP

SAS

ACP

SAS

ACP

SAS

IOM A

IOM B

IOM A

IOM B

Bottom shelf

SAS ports B and D are connected to the disk shelf stacks by offsetting the order of the HBA slots by one so that the first HBA port is cabled last. The cabling offset ensures system resiliency. For example, a controller with three quad-port SAS HBAs in slots 1, 2, and 3 is cabled in order of HBAs and ports 2b, 3b, 1d, 2d, 3d, 1b, as shown below.
Controller 1/A
Slot 1 Slot 2 Slot 3

A B C D

A B C D

A B C D

SAS cabling rules

A controller with one HBA is cabled in order of port 1d and then 1b, as shown below.
Controller 1/A
Slot 1

A B C D

Stack cabling sequence: Controllers are cabled to the stacks in sequence of the stacks, such as stack 1, stack 2, stack 3, and so on. 20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

Chapter 2: SAS Cabling for New Storage Systems

Cabling SAS connections

Topics in this section

The following topics are covered in this section:

Completing the SAS cabling worksheet on page 10 SAS cabling worksheet on page 12 Example worksheet on page 12 Cabling SAS ports on page 13

Completing the SAS cabling worksheet Step 1 Action

By making a list of the SAS ports for each controller that need to be cabled to the disk shelf IOMs, you can simplify the cabling task of your storage system.

Make a list of the top connections from the controller by listing all SAS HBA A ports and then all SAS HBA C ports on each controller. You can use the SAS cabling worksheet on page 12 to write your information. Note If you have an HA pair configuration, repeat this list for the second controller. Examples: A controller with three quad-port SAS HBAs is listed as 1a, 2a, 3a, 1c, 2c, 3c. A controller with one quad-port SAS HBA is listed as 1a, 1c. A controller with one dual-port SAS HBA is listed as 1a. A 2040 controller is listed as 0a. 2040 system configurations only have a top connection to the disk shelves.

10

Cabling SAS connections

Step 2

Action Make a list of the bottom connections from each controller. You can use the SAS cabling worksheet on page 12 to write your information. a. List the SAS HBA B ports and then all SAS HBA D ports. Examples: A controller with three quad-port SAS HBAs is listed as 1b, 2b, 3b, 1d, 2d, 3d. A controller with one quad-port SAS HBA is listed as 1b, 1d. A controller with one dual-port SAS HBA is listed as 1b. b. Copy the first port in your list to the end of your list, and then cross out the first port in your list.

For example:

1b, 2b, 3b, 1d, 2d, 3d, 1b

Offsetting the list by one (so that the bottom connection cabling begins on the second HBA) ensures system resiliency after you cable your storage system. Examples: A controller with three quad-port SAS HBAs is listed as 2b, 3b, 1d, 2d, 3d, 1b. A controller with one quad-port SAS HBA is listed as 1d, 1b. A controller with one dual-port SAS HBA is listed as 1b. Note If you have an HA pair configuration, repeat this list for the second controller. 3 Go to Cabling SAS ports on page 13 to cable your storage system.

Chapter 2: SAS Cabling for New Storage Systems

11

SAS cabling worksheet

By filling in this worksheet as instructed in the Completing the SAS cabling worksheet on page 10, you identify the SAS ports for each controller that need to be cabled to the disk shelf IOMs.

Connection type Top connections Bottom connections

Controller C1/A C2/B C1/A C2/B

SAS port cabling sequence

Cable to disk shelf IOM To top shelf IOM A To top shelf IOM B To bottom shelf IOM B To bottom shelf IOM A

Example worksheet

This is an example of a completed worksheet for an HA pair configuration with three quad-port SAS HBAs and three stacks of disk shelves. Note The grayed out SAS ports are unused because of the limited number of stacks in this configuration. If there were six stacks, all SAS ports would be used.

Connection type Top connections Bottom connections

Controller C1/A C2/B C1/A C2/B

SAS port cabling sequence 1a, 2a, 3a, 1c, 2c, 3c 1a, 2a, 3a, 1c, 2c, 3c 2b, 3b, 1d, 2d, 3d, 1b 2b, 3b, 1d, 2d, 3d, 1b

Cable to disk shelf IOM To top shelf IOM A To top shelf IOM B To bottom shelf IOM B To bottom shelf IOM A

12

Cabling SAS connections

Cabling SAS ports Step 1 Action

To cable the SAS ports, complete the following steps.

If you have more than one disk shelf in a stack, cable the shelf-to-shelf SAS connections; otherwise, go to Step 2. Using the QSFP to QSFP SAS cables that came with your system: a. b. Connect IOM A circle port to the next IOM A square port. Connect IOM B circle port to the next IOM B square port.

Note The QSFP connectors are keyed and click into place when oriented correctly into a SAS port. Do not force a connector into a port.
Stack of three shelves

ACP
Top shelf

SAS

ACP

SAS

IOM A

IOM B

Bottom shelf

Use your completed worksheet to cable the top connections for each controller. Note If you are cabling a 2050 storage system, use the QSFP to mini-SAS cables for all controller to shelf connections; otherwise, use the QSFP to QSFP SAS cables.

Chapter 2: SAS Cabling for New Storage Systems

13

Step 3

Action Use your completed worksheet to cable the bottom connections for each controller. Note For quad-port SAS HBAs, because ports A and C never connect to the same stack of disk shelves and ports B and D never connect to the same stack of disk shelves, you might have unused HBA ports depending on the number of stacks in your configuration. For example, a single-controller with one quad-port SAS HBA and one stack of shelves only uses HBA ports A and D. 4 Verify the SAS connections by entering the applicable command:

For 7-Mode, enter the following command at the system console:

sasadmin expander_map

For HA pair configurations, you can run this command on either node.

For Cluster-Mode, enter the following command at the clustershell prompt:

node run -node <node name> -command sasadmin expander_map

Result: The output shows information about SAS expanders attached to SAS channels on the controllers. 5 The next step depends on the output. If... The output lists all of the IOMs Any of the IOMs are not shown - either the output will not show the IOM that is cabled incorrectly, or will not show all of the IOMs downstream from it. 6 Then... The IOMs have connectivity. Go to Step 6. Repeat Steps 1 through 5 to correct cabling errors.

The next step depends on whether you are using the ACP capability. If... You are using the ACP capability You are not using the ACP capability Then... Go to Cabling ACP connections on page 28. Boot the storage system and begin setup.

14

Cabling SAS connections

SAS cabling examples

Topics in this section

The following topics are covered in this section: Note The cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.

Examples of HA pair quad-port SAS HBA cabling on page 15 Examples of single-controller quad-port SAS HBA cabling on page 18 Examples of 2040 SAS cabling on page 20 Examples of 2050 SAS cabling on page 22

Examples of HA pair quad-port SAS HBA cabling

The following cabling examples show the SAS cabling sequence for HA pair configurations using quad-port SAS HBAs. One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1d.
Controller 1/A
Slot 1 A B C D A

Controller 2/B
Slot 1 B C D

1.

Ethernet port

2. 4.

Ethernet port

Stack 1
3.
ACP SAS ACP SAS

IOM A

IOM B

Chapter 2: SAS Cabling for New Storage Systems

15

Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a; Bottom: 2b, 1d.
Controller 1/A
Slot 1 Slot 2 A A B C D A

Controller 2/B
Slot 1 Slot 2 A B C D

4. 7.
Ethernet port

1.

B C D

2.
Ethernet port

B C D

3.

8.

Stack 1
ACP SAS ACP SAS ACP

Stack 2
SAS ACP SAS
Top shelf

IOM A

IOM B

IOM A

IOM B

5.

6.

Bottom shelf

16

SAS cabling examples

Three quad-port SAS HBAs and three stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a; Bottom: 2b, 3b, 1d.
Controller 1/A Controller 2/B
Slot 1 Slot 2 A B C D Slot 3 A B C D

1. 9. 7.

Slot 1 A B C D

Slot 2 A B C D

Slot 3 A B C D

6. 11. 12.

4. 8.

A B C D

2.

3.

Stack 1
ACP
Top shelf IOM A IOM B IOM A

Stack 2
ACP SAS ACP SAS ACP

5.

10.

Stack 3
ACP SAS ACP SAS

SAS

SAS

IOM B

IOM A

IOM B

Bottom shelf

Four quad-port SAS HBAs and four stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a; Bottom: 2b, 3b, 4b, 1d. Four quad-port SAS HBAs and eight stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 1c, 2c, 3c, 4c; Bottom: 2b, 3b, 4b, 1d, 2d, 3d, 4d, 1b. Five quad-port SAS HBAs and five stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a; Bottom: 2b, 3b, 4b, 5b, 1d. Five quad-port SAS HBAs and 10 stacks of disk shelves: The SAS port cabling sequence for each controller is Top: 1a, 2a, 3a, 4a, 5a, 1c, 2c, 3c, 4c, 5c; Bottom: 2b, 3b, 4b, 5b, 1d, 2d, 3d, 4d, 5d, 1b.

Chapter 2: SAS Cabling for New Storage Systems

17

Examples of singlecontroller quad-port SAS HBA cabling

The following cabling examples show the SAS cabling sequence for singlecontroller configurations using quad-port SAS HBAs. One quad-port SAS HBA and one stack of disk shelves: The SAS port cabling sequence is 1a, 1d.
Controller 1/A
Slot 1 A B C D

1.

Stack 1

2.
SAS

ACP Top shelf

SAS

ACP

IOM A

IOM B

Bottom shelf

18

SAS cabling examples

One quad-port SAS HBA and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 1c; Bottom: 1d, 1b.
Controller 1/A
1. 3.
Slot 1
A B C D

4.

2.
Stack 1 Stack 2

ACP
Top shelf

SAS

ACP

SAS

ACP

SAS

ACP

SAS

IOM A

IOM B

IOM A

IOM B

Bottom shelf

Two quad-port SAS HBAs and two stacks of disk shelves: The SAS port cabling sequence is Top: 1a, 2a; Bottom: 2b, 1d.
Controller 1/A

1. 4.

Slot 1 Slot 2 A B C D A B C D

3.
Ethernet port

2.
Stack 2

Stack 1
ACP
Top shelf IOM A IOM B IOM A

SAS

ACP

SAS

ACP

SAS

ACP

SAS

IOM B

Bottom shelf

Chapter 2: SAS Cabling for New Storage Systems

19

Examples of 2040 SAS cabling

The following cabling examples show the SAS cabling sequence for 2040 HA pair and single-controller configurations. Note For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A. Note The 2040 storage systems onboard SAS port labeled 0d is treated as port 0a for cabling purposes. When the onboard port is treated as port 0a, all the SAS cabling rules can be applied.

20

SAS cabling examples

2040 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 0a.
2040 configuration
Controller 1/A
0a

Controller 2/B
0a 0d

0d
1.

Ethernet port e0P

Ethernet port e0P

Stack 1

2.

ACP
Top shelf

SAS

ACP

SAS

IOM A

IOM B

Bottom shelf

2040 single-controller with one stack of disk shelves: The SAS port cabling sequence is Top: 0a.
2040 configuration
Controller 2/B
0a 0d Ethernet port e0P
1.

Stack 1
ACP
Top shelf
IOM A IOM B

SAS

ACP

SAS

Bottom shelf

Chapter 2: SAS Cabling for New Storage Systems

21

Examples of 2050 SAS cabling

The following cabling examples show the SAS cabling sequence for 2050 HA pair and single-controller configurations. Note For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A. 2050 HA pair configuration with one stack of disk shelves: The SAS port cabling sequence for each controller is Top: 1a; Bottom: 1b.
2050 configuration
Controller 1/A
Slot 1 A B

Controller 2/B
Slot 1 A B

1.

Ethernet port

2.

Ethernet port

Stack 1

4.

ACP
Top shelf
3.

SAS

ACP

SAS

IOM A

IOM B

Bottom shelf

22

SAS cabling examples

2050 HA pair configuration with one disk shelf: The SAS port cabling sequence for both controllers is Top: 1a; Bottom: 1b.
2050 configuration
Controller 1/A
Slot 1 A B A B

Controller 2/B
Slot 1

Ethernet port
1.

Ethernet port

4.

Stack 1

2.

ACP
3.

SAS

ACP

SAS

IOM A

IOM B

Chapter 2: SAS Cabling for New Storage Systems

23

2050 single-controller configuration with one stack of disk shelves: The SAS port cabling sequence is Top: 1a; Bottom:1b.
2050 configuration
Controller 2/B
Slot 1 A B

1.

2.

Stack 1

ACP
Top shelf

SAS

ACP

SAS

IOM A

IOM B

Bottom shelf

2050 single-controller configuration with one disk shelf: The SAS port cabling sequence is Top: 1a; Bottom: 1b.
2050 configuration
Controller 2/B
Slot 1 A B

Ethernet port
1. 2.

Stack 1

ACP

SAS

ACP

SAS

IOM A

IOM B

24

SAS cabling examples

ACP Cabling for New Storage Systems

About this chapter

This chapter provides information about cabling the disk shelf ACP connections for a new storage system installation. This chapter does not provide cabling information for hot-adding or replacing disk shelves. For those procedures, see the Installation and Service Guide at now.netapp.com for your disk shelf model. Note References to the top and bottom disk shelves in a stack are to the logical top and bottom of a stack. Your physical stack arrangement might be different.

Topics in this chapter

This chapter discusses the following topics:

ACP cabling rules on page 26 Cabling ACP connections on page 28 ACP cabling examples on page 33

Chapter 3: ACP Cabling for New Storage Systems

25

ACP cabling rules

Topics in this section

The following topics are covered in this section:

General ACP cabling rules on page 26 ACP connectivity rules on page 26

General ACP cabling rules

These ACP cabling rules apply to all supported storage systems using SAS storage:

You must use CAT6 Ethernet cables with RJ-45 connectors for ACP connections. If your storage system does not have a dedicated on board network interface for each controller, you must dedicate one for each controller at system setup. You can use a quad-port Ethernet card. All ACP connections to the disk shelf are cabled through the ACP ports, which are designated by a square symbol or a circle symbol.

ACP connectivity rules

All supported storage systems using the ACP capability are cabled in this way: Shelf-to-shelf connections: All the disk shelves in a stack are daisy-chained when you have more than one shelf in a stack.

IOM A circle port is connected to the next IOM A square port. IOM B circle port is connected to the next IOM B square port.

Intrastack connections: The bottom shelf and top shelf in each stack are connected to provide resiliency within each stack of disk shelves. The bottom shelf IOM A circle port is connected to the top shelf IOM B square port. This applies to stacks containing one or more disk shelves. Stack-to-stack connections: When you have two or more stacks of disk shelves, the last shelf in a stack is connected to the first shelf in the next stack until all stacks are connected. The bottom shelf IOM B circle port of a stack is connected to the top shelf IOM A square port of the next stack. Controller-to-stack connections: Each storage system controller is connected to each stack of disk shelves through a dedicated Ethernet port:
26 ACP cabling rules

Controller 1/A always connects to the top shelf IOM A square port in a stack. Controller 2/B always connects to the bottom shelf IOM B circle port in a stack.

20xx storage system connections: For single-controller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A.

Chapter 3: ACP Cabling for New Storage Systems

27

Cabling ACP connections

Cabling ACP

Any storage system supporting the ACP capability can be cabled using this cabling procedure.

Step 1

Action If you have more than one disk shelf in a stack, cable your shelf-to-shelf ACP connections; otherwise, go to Step 2. Using the Ethernet cables that came with your system: a. b. Connect each shelf IOM A circle port to the next shelf IOM A square port until all shelves in each stack are connected. Connect each shelf IOM B circle port to the next shelf IOM B square port until all shelves in each stack are connected.
Daisy-chain connections
ACP SAS ACP SAS

Top shelf
IOM A IOM B

Bottom shelf

28

Cabling ACP connections

Step 2

Action Cable the intrastack connection. If... You have more than one disk shelf in a stack of disk shelves Then... Connect the bottom shelf IOM A circle port to the first shelf IOM B square port and repeat for all stacks.
Intrastack connection

ACP
Top shelf

SAS

ACP

SAS

IOM A

IOM B

Bottom shelf

You have only one disk shelf in a stack of disk shelves

Connect the IOMs within the disk shelf by connecting IOM A circle port to IOM B square of the disk shelf.
Intrastack connection

ACP

SAS

ACP

SAS

IOM A

IOM B

Chapter 3: ACP Cabling for New Storage Systems

29

Step 3

Action If you have more than one stack of disk shelves, cable the stack-to-stack connections; otherwise, go to Step 4. a. b. Beginning with Stack 1, connect the bottom shelf IOM B circle port to the next stack top shelf IOM A square port. Repeat Step a until all stacks are connected.
Stack-to-stack connections
Stack 1
ACP
Top shelf IOM A IOM B IOM A IOM B IOM A IOM B

Stack 2
SAS ACP SAS ACP SAS ACP

Stack 3
SAS ACP SAS

SAS

ACP

Bottom shelf

30

Cabling ACP connections

Step 4

Action Cable controller-to-stack connections. If... If your system has a controller 1/A If your system has a controller 2/B Then... Cable controller 1/A to the Stack 1 top shelf IOM A square port. Cable controller 2/B to the last stack bottom shelf IOM B circle port.

For example, an HA pair configuration with two stacks of disk shelves has the controller-to-stack connections cabled as follows:
Controller 1/A Controller 2/B

Ethernet port

Ethernet port

Stack 1
ACP
Top shelf IOM A IOM B IOM A

Stack 2
SAS ACP SAS ACP SAS

SAS

ACP

IOM B

Bottom shelf

Note 20xx storage systems use the e0P port on each controller. 5 Verify that the ACP cabling is correct by entering the applicable command:

For 7-Mode, enter the following command from the system console:
storage show acp

If you have an HA pair configuration, run this command on both nodes.

For Cluster-Mode, enter the following command from the clustershell:

node run -node local -command storage show acp

Chapter 3: ACP Cabling for New Storage Systems

31

Step 6

Action The next step depends on what the command output is for each node. If the result in the output shows ACP connectivity status... Full Connectivity

Then... ACP cabling is correct. The number of shelf IOMs detected out-of-band (through ACP) and in-band (through SAS) are the same.

No Connectivity

No ACP ports on the shelf IOMs are connected to the storage controllers. Repeat Step 4.

Partial Connectivity

Repeat Steps 1 through 6 to see what ACP connections you missed. Fewer shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS).

Additional Connectivity

Repeat the Cabling SAS connections on page 10 to see what SAS connections you missed. More shelf IOMs are detected out-of-band (through ACP) than in-band (through SAS).

Boot the storage system and begin setup.

See the Installation and Setup Instructions that came with your storage system. Note If your storage system does not have a dedicated ACP on board Ethernet port on each controller for ACP, you will designate them at setup.

32

Cabling ACP connections

ACP cabling examples

Topics in this section

The following topics are covered in this section: Note The cabling examples show cabling to the logical top and logical bottom of a stack of disk shelves. Your physical stack arrangement might be different.

Examples of HA pair configuration ACP cabling on page 33 Examples of ACP cabling for all systems except 20xx on page 35 Examples of 20xx single-controller ACP cabling on page 36

Examples of HA pair configuration ACP cabling

The following cabling examples show ACP cabling for all supported HA pair configurations. HA pair configuration with one disk shelf:
Controller 1/A Controller 2/B

Ethernet port

Ethernet port

Stack 1
ACP
Top shelf IOM A IOM B

SAS

ACP

SAS

Chapter 3: ACP Cabling for New Storage Systems

33

HA pair configuration with two stacks of disk shelves:

Controller 1/A Controller 2/B

Ethernet port

Ethernet port

Stack 1
ACP
Top shelf IOM A IOM B IOM A

Stack 2
SAS ACP SAS ACP SAS

SAS

ACP

IOM B

Bottom shelf

HA pair configuration with three stacks of disk shelves:

Controller 1/A Controller 2/B

Ethernet port

Ethernet port

Stack 1

Stack 2

Stack 3

ACP

SAS

ACP

SAS

ACP

SAS

ACP

SAS

ACP

SAS

ACP

SAS
Top shelf

IOM A

IOM B

IOM A

IOM B

IOM A

IOM B

Bottom shelf

34

ACP cabling examples

Examples of ACP cabling for all systems except 20xx

The following cabling examples show ACP cabling for all supported storage systems other than 20xx systems. Single-controller configuration with one stack of disk shelves (except the 20xx system):
Non 20xx configuration
Controller 1/A

Ethernet port

Stack 1
ACP SAS ACP SAS

Top shelf
IOM A IOM B

Bottom shelf

Single-controller configuration with two stacks of disk shelves (except the 20xx system):
Non 20xx configuration
Controller 1/A

Ethernet port

Stack 1
ACP
Top shelf IOM A IOM B IOM A

Stack 2
SAS ACP SAS ACP SAS

SAS

ACP

IOM B

Bottom shelf

Chapter 3: ACP Cabling for New Storage Systems

35

Examples of 20xx single-controller ACP cabling

The following cabling examples show ACP cabling for 20xx single-controller configurations. Note For single-controller 20xx configurations, the controller is in Slot B (the bottom slot); therefore, the controller needs to be cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A. 20xx single-controller configuration with one stack of disk shelves:
20xx configuration
Controller 2/B

Ethernet port

Stack 1
ACP
Top shelf
IOM A IOM B

SAS

ACP

SAS

Bottom shelf

36

ACP cabling examples

20xx single-controller configuration with two stacks of disk shelves:

20xx configuration
Controller 2/B

Ethernet port

Stack 1
ACP
Top shelf IOM A IOM B IOM A

Stack 2
SAS ACP SAS ACP SAS

SAS

ACP

IOM B

Bottom shelf

Chapter 3: ACP Cabling for New Storage Systems

37

38

ACP cabling examples

Terms and Symbols

About this appendix

This appendix defines terms and port icon symbols particular to SAS disk shelves and also includes common terms about connectivity. The terms are organized by relationship to each other instead of alphabetical order.

List of terms and symbols

You should be familiar with the following terms:

Square and circle symbols: The SAS ports and ACP ports on the disk shelf IOMs are designated by square and circle symbols. All cabling is done in reference to connecting to a square port or a circle port. Note SAS connectivity does not use the concept of In ports and Out ports.

IOM: The SAS shelf I/O module is located in the back of the disk shelf. It connects the individual disk drives to the rest of the storage system and controls the disk shelf operator display panel LEDs. Each disk shelf has two IOMs: IOM A and IOM B. These are also referred to as SBB A and SBB B on the slot map label on the back of the disk shelf.

SAS stack: Also referred to as stack. A group of one or more SAS disk shelves connected (daisy-chained) together and connected to the controller through the top disk shelf in the stack and the bottom disk shelf in the stack (as needed). The maximum number of disk shelves in a stack of disk shelves and the number of disk shelf stacks in a configuration are dependent on the type of storage system.

ACP: Alternate Control Path is a protocol that enables Data ONTAP to manage and control the disk shelf storage subsystem. It uses a separate network from the data path, so management communication is not dependent on the data path being intact and available. Use of ACP requires that all disk shelf IOMs and storage system controllers connect through the ACP ports on the IOMs and the designated network interface on each controller.

QSFP: The standard SAS cable connector for all NetApp SAS systems. QSFP to QSFP SAS cables are used to daisy-chain SAS disk shelf ports and connect

Appendix A: Terms and Symbols

39

to the quad-port SAS host bus adapter (HBA) or onboard SAS ports. QSFP to mini-SAS cables are used to connect disk shelves to the dual-port miniSAS HBA.

Software-based disk ownership: Software-based disk drive ownership is stored on the disk drive rather than determined by the topology of the storage systems physical connections. It gives you increased flexibility and control over disk drive use. SAS and SATA disk drives use software-based disk drive ownership. Therefore, the hardware-based disk ownership concept that the local node owns all direct-attached disk drives connected to it through the A channel (the loop attached to the A module on the disk shelf) and its partner owns the disk drives connected to the local node on the B channel, does not apply. This also means that the hardware-based disk drive ownership terms do not apply: primary path, standby path, redundant path, and redundant standby path. Instead, connectivity between the controller and SAS disk shelf is addressed in terms of: top connections and bottom connections.

Top connection: The cabling connection from the controller to the top disk shelf in a stack of disk shelves. References to the top disk shelf in this document are to the logical top of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.

Bottom connection: The cabling connection from the controller to the bottom disk shelf in a stack of disk shelves. References to the bottom disk shelf in this document are to the logical bottom of a stack of disk shelves. Your physical disk shelf stack might be arranged differently.

Shelf-to-shelf (daisy-chain): The cabling connection between disk shelves in a stack of more than one disk shelf. Each stack of disk shelves is daisy-chained through its SAS ports and, if you are using the ACP capability, each disk shelf is also daisychained through its ACP ports.

Controller: The component of your storage system that runs Data ONTAP software and manages the storage subsystem. The controller located in the top slot in a storage system chassis can be referred to as Controller 1 or Controller A. The controller located in the bottom slot in a storage system chassis can be referred to as Controller 2 or Controller B.

40

Terms and Symbols

In this document controllers are referred to as Controller 1/A and Controller 2/B.

Single-controller configuration: A storage system having one controller. HA pair configuration: A storage system having two controllers. This can be two controllers in one chassis or it can be two controllers, each in a separate chassis.. The two controllers serve data for each other if one of the controllers becomes impaired.

Multipath HA: Each controller in the HA pair configuration has multiple ways to connect to a disk drive. Multipath HA cabling is the most resilient and preferred configuration for HA pair configurations. This is because it takes full advantage of the resiliency capability of SAS, which means that the storage system continues to have access to disk drives in the event of cable, HBA, and IOM failure. A single failure of a cable, HBA, or IOM does not result in a controller failover.

Dual-path: The single-controller storage system has two ways to connect to a disk drive. This is the preferred configuration for single-controller configurations.

Single-path HA: Each controller in the HA pair configuration has one way to connect to a disk drive. This means an IOM or cable failure requires a controller failover.

Single-path: The single-controller storage system has one way to connect to a disk drive. This configuration has multiple single points of failure and is not a preferred configuration.

Configured systems: A new storage system that ships with SAS disk shelves and preinstalled SAS HBAs (if applicable).

Add-on disk shelves: SAS disk shelves that are shipped individuallynot shipped as part of a configured system.

Appendix A: Terms and Symbols

41

42

Terms and Symbols

Document Update Record

Document update history The following table lists and describes the history of changes made to this document. Changes are not dictated by Data ONTAP releases.

B
Feature release date September 2009

Feature updates

Initial release of the SAS and ACP cabling information was part of the DS4243 Installation and Service Guide. Initial release of the SAS and ACP cabling information as a stand-alone document and updates.

September 2010

The SAS disk shelf illustrations are now generic so that they apply to any NetApp SAS disk shelf. A generic disk shelf is shown as two IOMs side-by-side (no power supplies are shown). HBAs (1, 2, 3 etc.) are now referred to as slots (1, 2, 3 etc.) Introduces the new rule for referring to onboard SAS ports as belonging to slot 0 and the 2040 system exception - the single onboard port is 0a. Introduces the new rule that for singlecontroller 20xx storage systems, the controller is in Slot B (the bottom slot); therefore, the controller is cabled as if it were controller 2/B. If a second controller is added to make this an HA pair configuration, the new controller is cabled as controller 1/A. The term active/active is updated to HA pair throughout. The term Multipath is updated to Multipath HA.

Appendix B: Document Update Record

43

44

Document Update Record

Index
A
ACP cabling procedure 28 ACP capability 26 audience v worksheet procedure 10 conventions formatting v

F C
cabling ACP examples 36 procedure 28 verifying ACP connections 31 cabling SAS examples 18 rules 4 verfiying SAS connections 14 worksheet 12 formatting conventions v

S
special messages vi

T
terminology 39

Index

45

46

Index