DVC TAC Time 2016 Mar Nexus9k3k

Nexus 9000 & Nexus 3000
TAC Time 2016
March 2016
Introduction-What is Nexus9000/3000 Family
Nexus 9000/9300 Series Switches Nexus 3000/3100 Series Switches
Nexus9504/Nexus9508/Nexus9516 N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396
2
9500 Field Upgradeable Units (FRU)
• 9500 has the following modular components which can upgraded or
replaced in the field ®
Nexus 9508 Front View ®
Nexus 9508 Rear View
• Supervisor
• Fabric Module
• Line Card
• System Controller
• Fan Tray
• Power Supply
• The Supervisor, System controller ,Fabric Module and LC have OBFL

(On-Board Failure Logging) for failure analysis
3
Nexus 9500 Platform Line Card
Line Cards
FM6 FM5 FM4 FM3 FM2 FM1
• I/O module with Merchant and
Merchant+ ASIC HG
MUX1
HG
MUX4
HG
MUX2
HG
MUX5
HG
MUX3
HG
MUX6
• Have Various Forwarding Tables

• L2 Mac Table And L3 Host Table
• ACL and Buffers for Queuing 012
3
456
MN 7Port
8 9 10
11
012
3
456
MN 7Port
8 9 10
11
Northstar 1 Northstar 2
ASIC Name 0-
MF Port
3- 6-8 9- 0-
MF Port
9-
NFE=Network Forwarding Engine-Trident 2(T2)

2 5 11
HG 2 11
ALE=Application Leaf Engine-North Star(NS) 7-

5
2-
Warpcor
0 29
31- 26-
24
7-
5
26-
24
T2
e N9K-X9564PQ T2
-Donner
Note: Internal ports are called as Hi-Gig/HG ports 40G
QSFP
10G SFP+ Ports FPFPFPFP
F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F 49 50 51 52
P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P P
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 2425 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48
4
Nexus 9500 Fabric Module
Fabrics Modules
• Interconnect Line Card slots
• Installed at the rear of the chassis
• Leverages Broadcom Trident II ASICs
• Max 1.92 Tbps per line card slot (6 Fabric Cards)
• 960 Gbps per line card slot (3 Fabric Cards)
• All Fabric Cards are active and carry traffic
• Fan Tray requires Fabric Card to be present in even slot
Trident II Trident II
ASIC-NFE ASIC-NFE
32 x 40G 32 x 40G
Hi-Gig2 Hi-Gig2
5
Nexus9500 Series Line Card Summary
Information X9600 Series Line X9500 Series Line X9400 Series
Cards Cards
ASIC Technology Merchant only Merchant+ Merchant only
N9K-X9636PQ N9K-X9564PX •N9K-X9432PQ
N9K-X9564TX •N9K-X9464PX
N9K-X9536PQ •N9K-X9464TX
Number of ASIC 3 T2 2 T2 + 2 NS 2 T2 40 gig 32 Ports

2 T2 + 2 NS 1 T2 48 1/10 gig , 4
2 T2 + 2 NS QSFP
Non Blocking Non Blocking Line rate > 200 byte

packet
Buffer Size 36 MB 104 MB 12 MB with one T2
24 MB with two T2
6
All Product shipping in FY16
Top Of The Rack Switch Family
Today Portfolio Extension

Refresh: Nexus 9300EX Fixed
Nexus 9300 Fixed (NX-OS / ACI)
(NX-OS / ACI) • ACI
• Scale (Route Table)
• Buffer
Exclusively Cisco Silicon • Features
Cisco ASICs + Merchant
T2&NS New: Nexus 9200 Fixed +
(NX-OS)
• Port Density
• Scale (Route Table)
Exclusively Cisco Silicon • Buffer
• Features
N3000 Fixed Series
N3000 fixed switches
Nexus 3132Q/3172P
+
• 25/50/100G
• VXLAN
Merchant Silicon T+ Merchant Silicon-T+
Merchant Silicon T2
© 2013-2014 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
Nexus 9300 Series Switch Summary
N9396TX/P N93128TX N9372TX N9372TX N9372PX
X
NFE (BCM T2) 1 1 2 1 1
ALE ( GEM-1 NS GEM-1 NS No GEM-1 No GEM -1 No GEM- 1
NorthStar)/GEM Donner Donner Donner
Oversubscribed No 1.5:1 No No No
Line Rate Yes Yes (packets > Yes Yes
194-Bytes)
QoS Classes 8 4 8 4 4
Buffer (MB) 36 (12*3) 104 24 (12*2) 104 (12*2+40*2) 104 (12*2+40*2)
(12*2+40*2)
8
General Recommendation for New and Existing
Deployments-Nexus9000
• Software Recommendation
Platform Series Recommended Release
Cisco Nexus 9500 6.1(2)I3(5), 7.0(3)I1(3), or 7.0(3)I2(2a)*
Cisco Nexus 9300 6.1(2)I3(5), 7.0(3)I1(3), or 7.0(3)I2(2a)*
Cisco NX-OS 7.0(3)I2(x) is the long-lived release train for the Cisco Nexus 9000 Series switches.
End-of-Sale and End-of-Life Announcements for Software Releases 6.1(2)I3(x), 7.0(3)IX1(x) and 7.0.3I1(x)
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/recommended_releaseb_Minimum_and_Recommended_Cisco_NXO
S_Releases_for_Cisco_Nexus_9000_Series_Switches.html
• Verified Scale limits for different features and protocol for each release
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-x/scalability/guide_34/b_Cisco_Nexus_9000_Series_
NXOS_Verified_Scalability_Guide_612I34/b_Cisco_Nexus_9000_Series_NXOS_Verified_Scalability_Guide_612I34_chapter_01.html
9
General Recommendation for New and Existing
Deployments-Nexus3000
• Software Recommendation
Platform Series Recommended Release
Cisco Nexus 3000 and Cisco 6.0(2)U6(5a) or 7.0(3)I2(2a)*
Nexus 3100
Cisco NX-OS 7.0(3)I2(x) is the long-lived release train for the Cisco Nexus 3000 Series switches.
End-of-Sale & End-of-Life Announcement for N3000/N3100 Software Releases 5.0.3Ux and 6.0.2U1x
http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/recommended_release/b_Minimum_and_Recommended_Cisco_NX-
OS_Releases_for_Cisco_Nexus_3000_Series_Switches.html
• Verified Scale limits for different features and protocol for each release
http:http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/scalability/7x/b_Nexus3k_Verified_Scalability_7x.html
10
Release Timelines – Nexus 9000 Standalone
Jun’14 Jul’14
Jan’15 Apr’15
Ash2.2a Ash2.2b
Bronte 1.1 Bronte1.2
Ash2.1 Ash2.2 Ash3.1 Camden Dublin
Mar’14 Apr’14 Sept’14 Sept’15 Feb’16
Ash2.3 N3K Train
Jun’14
Shipped
To be Shipped
Cisco NX-OS Release 7.0(3)I2(1) Overview For Nexus3000
Upgrade Matrix
From To Limitation Recommended Procedure
6.0(2)U6(3) 7.0(3)I2(1) None Install all and fast reload are the only upgrade methods supported
because of a BIOS upgrade requirement.
Store the pre-Release, 6.0(2)U6(3)’s configuration file
6.0(2)U6(2) or 7.0(3)I2(1) First, upgrade to Install all and fast reload are the only upgrade methods supported
earlier Cisco NX-OS because of a BIOS upgrade requirement.
Release 6.0(2)U6(3).
• install all nxos bootflash:nxos.7.0.3.I2.2a.bin Limitations-Not Supported Features

• A single image binary is now used for booting N3000 platforms. • Auto-Config
• Kickstart and system images are no longer used. • FEX
Example: • Network address translation (NAT)
• Port profiles
boot nxos <single_image_binary>
• NX-API REST
fast-reload nxos <single_image_binary> • PTP (limitation only N3100 switches)
• VXLAN BGP eVPN control plane
Downgrading to Release 6.0(2)U6(3) from Release 7.0(3)I2(1)
• Downgrading from Release 7.0(3)I2(1) to Release 6.0(2)U6(3) with

configuration is not officially supported via install all.
• To downgrade to Release 6.0(2)U6(3) from Release 7.0(3)I2(1) with

configuration, two reloads are required
To avoid multiple reloads, the no-save option is supported –
1. Enter the command: write erase

2. Enter the command: write erase boot
3. Enter the command: copy <Release 6.0(2)U6(3) -config> startup-config
4. Enter the command: install all kickstart <img.kick> system <img.sys> no-save bios-force
5. The ASCII-replay of Release 6.0(2)U6(3) configuration brings the switch up with the desired
configuration and Release 6.0(2)U6(3) image.
ACL TCAM TABLE
Characteristic
• Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks
• Egress ACL: 1K TCAM entries - 4x 256 banks
• Each ACL type needs its own dedicated bank/banks
• IPv4, IPv6 or MAC each needs dedicated bank/banks
• MAC-ACL IPv6 & any QOS needs double-width entries, which means needs at least 2 banks
• VACL is programmed symmetrically in both egress and ingress ACL

Interface Ingress ACL Egress ACL
Type
SVI TCAM Shared TCAM Not shared
L3 TCAM Shared TCAM Shared

14
ACL TCAM Default Region and Carving
• TCAM Banks will first get assigned to Feature which has largest region.
• Next TCAM Bank will get assigned to Feature which need double Width.
• TCAM Carving requires Line Card/TOR reload to take effect
• To read current TCAM allocation

N9K#Show system internal access-lists global
• To reconfigure TCAM Region

N9K(config)hardware access-list tcam <feature name> <size>
15
TCAM for QoS policy
There are 4 types of policy-maps. Banks
• Control-plane --- for control plane policing (CoPP)
• Network-qos
• QoS --- (classification, marking, policing)
• Queuing --- egress queuing and scheduling
Only type “QoS” needs TCAM resource.

QoS policy can be applied to Layer-3 interfaces, switch ports, VLANs.
Based on where the policy is applied, and which classifier is used, one of the following QoS
TCAM region need to be carved.
L3 QoS Port QoS VLAN QoS

IPV4 L3 QoS [l3qos] IPV4 Port QoS [qos] IPV4 VLAN QoS [vqos]
IPV6 L3 QoS [ipv6-l3qos] IPV6 Port QoS [ipv6-qos] IPV6 VLAN QoS [ipv6-vqos]
MAC L3 QoS [mac-l3qos] MAC Port QoS [mac-qos] MAC VLAN QoS [mac-vqos]
IPV4 L3 QoS Lite [l3qos-lite] IPV4 Port QoS Lite [qos-lite] IPV4 VLAN QoS Lite [vqos-lite]
If using DSCP/COS as qualifier, don’t need to carve for IPv6. The region for IPv4 will be
used for both IPv4 and IPv6
TCAM Carving for QoS Policy on FEX Ports
• VLAN QoS policy applies to FEX ports in the VLAN without using FEX TCAM regions. Only
need the switch-level VLAN QoS regions.
• For port QoS policy on FEX ports, one or more of the following TCAM regions need to be
carved based on the classifiers:
FEX QoS
FEX IPV4 Port QoS [fex-qos]
FEX IPV6 Port QoS [fex-ipv6-qos]
FEX MAC Port QoS [fex-mac-qos]
FEX IPV4 Port QoS Lite [fex-qos-lite]
• When using DSCP/COS as the classifier, can use “fex-qos” or “fex-qos-lite” for both IPv4
and IPv6 port QoS policy on FEX ports.
TCAM Carving for QoS Policy on ALE/ALE-2 Ports
12 x 40 GE QSFP+ (on GEM N9K-M12PQ) 6 x 40 GE QSFP+
ALE ALE-2
12x 42 Gbps 6 x 42 Gbps

NFE NFE
1/10GE 1/10GE
Ethernet Ethernet
• N9396 with N9K-M6PQ GEM • N9372TX /PX

• N9396PX/TX with N9K-M12PQ GEM
• N93128 with N9K-M6PQ GEM • N9332PQ
• N93128 with N9K-M12PQ GEM
• N93120
 Security ACL policy is enforced on T2.

 QoS policy on NFE 1/10/40GE ports are enforced on T2
 QoS policy on ALE/ALE-2 40GE ports need to be enforced on both T2 and ALE/ALE-2
 In addition to the TCAM on T2, there is a TCAM on ALE/ALE-2 which need to be carved for QoS policy on
ALE/ALE-2 port. In addition to the QoS regions on T2, one of the following QoS regions on ALE/ALE-2 need to be
carved based on the qos policy type and classifier type:
n9396-vtep-1# sh hardware access-list tcam region | in ns
NS IPV4 Port QoS [ns-qos] size = 0
NS IPV6 Port QoS [ns-ipv6-qos] size = 0
NS MAC Port QoS [ns-mac-qos] size = 0
NS IPV4 VLAN QoS [ns-vqos] size = 0
NS IPV6 VLAN QoS [ns-ipv6-vqos] size = 0
NS MAC VLAN QoS [ns-mac-vqos] size = 0
NS IPV4 L3 QoS [ns-l3qos] size = 256
NS IPV6 L3 QoS [ns-ipv6-l3qos] size = 0
NS MAC L3 QoS [ns-mac-l3qos] size = 0
QoS vs QoS-Lite
 QoS TCAM regions are double width to support confirm/violate policer statistics in traffic policing.
 IPv4 QoS-lite TCAM regions reduce the size to single width
 QoS vs QoS-lite:
 No difference for classification, marking functions.
 No differences in the policing function itself, but QoS provide both confirm and violate statistics
while QoS-lite only provide violate statistics.
 Either QoS or QoS-lite TCAM region can be enabled.
IPv4 QoS-Lite Regions

Feature Purpose Region Name
Layer 3 QoS QoS policy applied on Layer 3 interfaces. IPV4: l3qos-lite
Port QoS QoS policy applied on Layer 2 interfaces. IPV4: qos-lite
VLAN QoS QoS policy applied on VLAN. IPV4: vqos-lite
FEX QoS QoS policy applied on FEX interfaces. IPV4: fex-qos-lite
TCAM sharing for QoS policy
 By default TCAM sharing among interfaces or VLANs isn’t enabled. When multiple interfaces or
VLANs have the same QoS policy, one copy of the QoS policy is programed for each interface or
VLAN.
 TCAM sharing can be enabled by applying the QoS policy under interfaces or VLANs with the no-stats
option. The configuration command is “service-policy type qos input my-policy no-stats”.
Nexus 3100 ACL
What has changed from N3000
Total Ingress ACL TCAM is 4096 compared to 2048 of N3000. But Egress ACL TCAM is same, 1024
entries
Default ACL TCAM carving size has changed at ingress due to increased total TCAM size
ARP ACL doesn't need separate ACL TCAM region, SUP TCAM be used for the ARP ACL
Buffer And Queuing-T2-Nexus9000
• T2 has 12 Mbytes of
Buffer shared by all ports
for all Traffic
• Shared buffer divided Into Control
Control and default service
pool if module is T2 only Control
Shared Shared OOBFC
• Shared buffer divided into Buffer Buffer
Control, default and 12 MB Default 12 MB
OOBFC service Pool if Default
Module is T2 and NS
based
Module with T2 only Module with T2 And NS
OOBFC: Out of band flow control unicast service pool
22
Buffer And Queuing-North Star-Nexus9000
GEM 4x 40GE QSFP+ Uplinks

• North Star has 40 Mbytes of Buffer
10 MB 20 MB • Divided in to Three Pool
Buffer
NorthStar Buffer
ASIC 1 • Control , SPAN , Default
10 MB
Buffer
12 x 40G
Hi-Gig2
Control
Trident II SPAN
ASIC Shared
Buffer
12 x 40G Default
Ethernet
Front Panel 48x 1GE/10GE Ports
23
Nexus 3100
Static/Dynamic Buffer
A pool of 12.2 MB Buffer (T2Base) space is divided up among Ingress per port & Egress per port/queue and
Dynamically shared buffer-Nexus3000 T+ base has 9MB of Shared buffer.
Total Buffer Cell Size
12.2M 208 Byte Cells
(59076 Cells) The first Cell of any packet uses 64 Bytes for internal header. Packets larger than 144 Bytes require more
cells.
Buffer Details Buffer N3132 N3172

Partition Partition
Ingress reserved 45cells (9360bytes) Total Ingress 299.5KB 673.9KB
per port Reserved
Total Egress 1.171MB 2.635MB

Egress reserved per (8 UC + 8 MC Queues)* 11cells = 176 cells(36608bytes) Reserved
port/queue
Total Shared 10.8MB (88%) 8.97MB

Shared 52004/43164 (N3132/N3172) Cells Dynamically Shared (73%)
Instant Buffer Usage Stats
Buffer polling interval for 7.0
N9K#show hardware internal buffer info pkt-stats mod 1 Release is 500msecs
INSTANCE: 0
---------------------------------------------------------- • Instant Buffer utilization per queue
per port
Output Shared Service Pool Buffer Utilization (in cells)
• One cell represents 208 bytes
SP-0 SP-1 SP-2 SP-3
----------------------------------------------------------- Show hardware internal buffer info pkt-
stats input mod 1
Total Instant Usage 4474 0 89 2939
Remaining Instant • SP-3-Dedicted resource for Control
Usage 25466 0 14255 3405 Traffic
Peak/Max Cells Used 4821 0 327 3060 • SP-0-Resource for Locally Switched
Switch Cell Count 29940 0 14344 6344 Unicast ,Multicast and SPAN
---------------------------------------------------------- • SP-2 Extended Output queue for
show hardware internal ns buffer info pkt-stats Unicast using buffers from North
Star
25
Broadcom Unified Forwarding Table
T2 has the following Unified Forwarding Table:
SUPPORTED COMBINATIONS
Mode L2 L3 Host LPM
0 288K 16K 16K
1 224K 56K 16K
2 160K 90K 16K
3 98K 122K 16K
4 32K 16K 128K
26
Nexus 3100 Capabilities
Flexible Forwarding Table
Dedicated L2 Dedicated Host Dedicated

Table Shared Forwarding Table
256K (4 Banks of 64K ) Table LPM Table
32K 16K 16K
5 Different Forwarding Table Modes . Only Mode 0 at FCS
Mode L2 Table L3 Host Table ALPM Size L3 LPM TCAM
0 288K 16K 0 16K
1 224K 56K 0 16K
2 160K 88K 0 16K
3 96K 120K 0 16K
4
© 2013-2014 Cisco and/or its affiliates. All rights reserved. 32K 16K 128K N/ACisco Confidential 27
Routing Mode for Nexus9500
LPM Routing Mode Broadcom T2 Mode Cli Command
Default System routing mode 3 (For Line card)
4 (For Fabric Module)
Max-host routing mode 2--Line Card- V6 in LPM System routing max-mode host
3--For Fabric Module
Nonhierarchical routing 3--For Line Card System routing non-hierarchical
mode 4--With max-l3-mode Option [max-l3-mode]
option For Line card
No Routes on Fabric Module
64-bit ALPM routing mode Sub mode of mod 4 for System routing mode
Fabric modules hierarchical 64b-alpm
show hardware internal forwarding table utilization mod 1

Max Host Route Entries (shared v4/v6):16384 Non hierarchical
Max LPM Table Entries : 131072 routing mod
show hardware internal forwarding table utilization mod 21
Max Host Route Entries (shared v4/v6): 0
Max LPM Table Entries :0
28
Verify Consistency Between Software and Hardware Table
Table CLI
Physical Interface show consistency-checker link-state
Port-Channel show consistency-checker membership port-channels

Membership
Mac Address Table show consistency-checker l2
Vlan Membership show consistency-checker membership vlan
L3 interface-LIF L3 interface-LIF programming –Logical Interface for Routing

programming
For RIB and FIB show consistency-checker forwarding ipv4 unicast
29
Consistency Checkers-Mac address Table
N9K# show consistency-checker l2 module 1
Consistency check: PASSED
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen, + - primary entry using vPC Peer-Link,
(T) - True, (F) - False
Missing entries in the HW MAC Table
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
Extra and Discrepant entries in the HW MAC Table
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
30
Consistency Checker –Unicast Forwarding
N9K#test consistency-checker forwarding
Consistency check started.
N9K# show consistency-checker forwarding ipv4 unicast module 1
IPV4 Consistency check (in progress): table_id(0x1) slot(1)
Elapsed time : 8257 ms
N9K# show consistency-checker forwarding ipv4 unicast module 1
IPV4 Consistency check : table_id(0x1) slot(1)
Execution time : 13244 ms ()
No inconsistent adjacencies.
No inconsistent routes.
Consistency-Checker: PASS for 1
31
Troubleshooting Toolkit
• Ethanalyzer-To Analyze the traffic sent and received by CPU

• TCP Dump-Accessing Bash provide Linux tools
• ELAM- Elam Allows to capture single packet based on Trigger
• Packet Tracer-Helps to Trace the packet inside Switch
• Flex Counter-count Next hop Adjacency stats
• ERSPAN-Encapsulated Remote SPAN(ERSPAN)
• Consistency Checkers-Several Table check
32
Bash Support !!!! And Broadcom ASIC shell Access !!!!
• Goes beyond what standard CLI can provide
• Customers demand more capabilities/freedom Creativity
• Feature: bash-shell
• User Role: dev-ops or network-admin or vdc-admin*
• Strongly recommended: Some experience with shell/Linux-Use with extreme care
• Access is provided to each and every instance of the T2 ASIC
• No additional license is required to access the bcm-shell
• Permitted by default role network-admin
• Accounting log available for BCM activity
33
BCM Access some Examples hg0 hg11 hg0 hg11
T2 T2
Instance 0 Instance 1
N9K# bcm-shell mod 6 "show unit" QSPF QSPF

FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP FP
01 02 03 04 05Ports
06 07 08 09 10 11 12 Ports
13 14 15 16 17 18 19 20 21 22 23 24
Unit 0 chip BCM56852_A2 (current)
Xe0 Xe11 Xe0 Xe11
Unit 1 chip BCM56852_A2
Eth1/1 Eth1/12 Eth1/13 Eth1/24
N9K# bcm-shell mod 6 "ps" | in 19
xe19 up 1G FD SW Yes Disable None FA XGMII 1582
N9K# show accounting log | last 2
Mon Apr 20 08:31:52 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "show unit" (SUCCESS)
Mon Apr 20 08:32:14 2015:type=update:id=console0:user=admin:cmd=bcm-shell
module 6 "ps" | in 19 (SUCCESS)
34
FEX/Switch Support Matrix - NX-OS Mode
No FEX support on ALE Ports
TOR / Fixed Switches:
Modular Switches
Best Practices and Common Pitfall
• We do not support Lacp fast timers. Lacp fast configuration should be avoided in VPC peer-
link.
• Loopguard configuration on VPC Peer-link is not recommended.
• We recommend that you use the strict default CoPP policy initially and then later modify the
CoPP policies based on the data center and application requirements.
• Copp should not be disabled. When CoPP is removed, the system will by default limit traffic
to 50pps. Disabling of CoPP will not be allowed starting 7.0(3)I2(1x).
• TCAM Bank chaining or Bank-Mapping is not possible on Nexus 9000.
• Upgrading from 7.0(3)1(2), 7.0(3)I1(3), or 7.0(3)I1(3a) to 7.0(3)I3(1) requires installing a

patch and then upgrading with install all. Failing to follow this requirement requires console
access to recover. The patch is needed for Nexus 9504, 9508, and 9516 platforms. The
Cisco Nexus 93xx Series and ToR switches do not require this patch.
Important Caveats
 Qos Clasification is not supported for VXlan traffcic in network-to-access direction
 The Qos buffer-boost feature is not applicable for VXLAN traffic
 ACL and QOS for VXLAN traffic in the network-to-access direction is not supported.
 RP(Rendezvous point) on a VTEP device is not supported.
 VxLAN tunnels cannot have more than one underlay next hop on a given underlay port.
 VXLAN EVPN ingress replication is supported on Cisco Nexus Series 9300 Series switches.
It is not supported on Cisco Nexus Series 9500 Series switches
 ARP suppression is an enhancement provided by the MP-BGP EVPN control plane to
reduce network flooding caused by broadcast traffic from ARP requests.
 ISSU is not supported when switches has FEX, VXLAN, NAT, and Segment Routing
configured. Upgrade will be disruptive when configuration for these features are present.
Critical Bugs
 CSCux04393 - N9K stops responding to ARP request or packets sourced from the N9K
 CSCux79367 - N9k-Dhcp Offer/Ack endless loop in double side VPC scenario
 CSCuy49227 - N9K - Loops BFD echo packet on Spanning-tree Blocked Port
 CSCuy34791 – upgrade fails when free space in /tmp folder is below threshold
 Software issues should not be misinterpreted as HW failure.
 Always insist TAC troubleshooting before requesting RMA.
 For example, below two issues are software defects and not a hardware failure.
CSCuy42027 - BootFlash diagnostic failure on LC/FM/SC
CSCuy49570 - N9k - Fex 2300 report fan failure intermittently
Tech Zone
 https://techzone.cisco.com/
 Discussions show up in CSE’s work queue
 Topic searchable
 Big push to move documents here and
eventually move some docs to CCO
Internal Documentation
 N9K Training portal

http://quakeroats.cisco.com/CustAdv/ts/ielc/tst/el/online/courses/nexus_9k_portal/
 N9K NPI
http://quakeroats.cisco.com/CustAdv/ts/ielc/tst/tstraining/catalog/index.shtml?tab1filter=NPI%20E-
Learning&nomsg=1&crs=npi
 N9K FAQ
http://nexus9kaci.cisco.com/product-line/faq
 N9k Troubleshooting guide

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/6-
x/troubleshooting/guide/b_Cisco_Nexus_9000_Series_NX-
OS_Troubleshooting_Guide/b_Cisco_Standalone_Series_NX-
OS_Troubleshooting_Guide_chapter_01.html
Thank You For providing this opportunity
Our Team
Rajesh Gatti Somu Jayaraman
Clayton Sullivan Karthick Murugan
Shridhar V. Dhodapkar

DVC TAC Time 2016 Mar Nexus9k3k

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DVC TAC Time 2016 Mar Nexus9k3k

Uploaded by

Copyright:

Available Formats

Nexus 9000 & Nexus 3000

TAC Time 2016

Nexus 9000/9300 Series Switches Nexus 3000/3100 Series Switches

Nexus9504/Nexus9508/Nexus9516 N9K-C9332PQ N9K-C9372PX N9K-C9372TX N9K-C9396

• The Supervisor, System controller ,Fabric Module and LC have OBFL

• Have Various Forwarding Tables

NFE=Network Forwarding Engine-Trident 2(T2)

ALE=Application Leaf Engine-North Star(NS) 7-

Number of ASIC 3 T2 2 T2 + 2 NS 2 T2 40 gig 32 Ports

Non Blocking Non Blocking Line rate > 200 byte

Today Portfolio Extension

Ash2.1 Ash2.2 Ash3.1 Camden Dublin

Mar’14 Apr’14 Sept’14 Sept’15 Feb’16

Ash2.3 N3K Train

• install all nxos bootflash:nxos.7.0.3.I2.2a.bin Limitations-Not Supported Features

• Downgrading from Release 7.0(3)I2(1) to Release 6.0(2)U6(3) with

• To downgrade to Release 6.0(2)U6(3) from Release 7.0(3)I2(1) with

1. Enter the command: write erase

• Ingress ACL: 4K TCAM entries - 4x 512 banks + 8x 256 banks

• Egress ACL: 1K TCAM entries - 4x 256 banks

• Each ACL type needs its own dedicated bank/banks

• IPv4, IPv6 or MAC each needs dedicated bank/banks

• VACL is programmed symmetrically in both egress and ingress ACL

L3 TCAM Shared TCAM Shared

• TCAM Carving requires Line Card/TOR reload to take effect

• To read current TCAM allocation

• To reconfigure TCAM Region

Only type “QoS” needs TCAM resource.

L3 QoS Port QoS VLAN QoS

12x 42 Gbps 6 x 42 Gbps

• N9396 with N9K-M6PQ GEM • N9372TX /PX

 Security ACL policy is enforced on T2.

IPv4 QoS-Lite Regions

OOBFC: Out of band flow control unicast service pool

GEM 4x 40GE QSFP+ Uplinks

Front Panel 48x 1GE/10GE Ports

Buffer Details Buffer N3132 N3172

Total Egress 1.171MB 2.635MB

Total Shared 10.8MB (88%) 8.97MB

0 288K 16K 16K

1 224K 56K 16K

2 160K 90K 16K

3 98K 122K 16K

4 32K 16K 128K

Dedicated L2 Dedicated Host Dedicated

5 Different Forwarding Table Modes . Only Mode 0 at FCS

Mode L2 Table L3 Host Table ALPM Size L3 LPM TCAM

0 288K 16K 0 16K

1 224K 56K 0 16K

2 160K 88K 0 16K

3 96K 120K 0 16K

show hardware internal forwarding table utilization mod 1

Port-Channel show consistency-checker membership port-channels

Vlan Membership show consistency-checker membership vlan

L3 interface-LIF L3 interface-LIF programming –Logical Interface for Routing

• Ethanalyzer-To Analyze the traffic sent and received by CPU

N9K# bcm-shell mod 6 "show unit" QSPF QSPF

• Upgrading from 7.0(3)1(2), 7.0(3)I1(3), or 7.0(3)I1(3a) to 7.0(3)I3(1) requires installing a

 The Qos buffer-boost feature is not applicable for VXLAN traffic

 RP(Rendezvous point) on a VTEP device is not supported.

 CSCux79367 - N9k-Dhcp Offer/Ack endless loop in double side VPC scenario

 CSCuy49227 - N9K - Loops BFD echo packet on Spanning-tree Blocked Port

 Software issues should not be misinterpreted as HW failure.

 Always insist TAC troubleshooting before requesting RMA.