ProCurve Networking

LAN Aggregation Through Switch Meshing

Technical White paper

Introduction ............................................................................................................... 3 Understanding Switch Meshing ...................................................................................... 3 Creating Meshing Domains ........................................................................................ 5 Types of Meshing Domains ........................................................................................ 6 Meshed and Non-meshed Ports .................................................................................. 8 Different Speeds for Links ......................................................................................... 8 Easy Configuration ................................................................................................... 8 Connecting Multiple Meshing Domains ......................................................................... 8 Preventing Broadcast Storms ........................................................................................ 9 Locating Unknown Destinations ..................................................................................... 9 Using STP with Switch Meshing ................................................................................... 10 802.1Q VLANs and Meshed Switches ............................................................................ 11 Additional Guidelines for Meshing Domains ................................................................... 11 Enabling Protocols in the Meshing Domain ................................................................. 12 Backward Compatibility with Older Switches............................................................... 12 Conclusion ............................................................................................................... 12 Glossary of Terms ..................................................................................................... 13

Introduction
Because companies rely on their network to perform critical business tasks, that network must be fast, and it must be reliable. However, companies do not have an unlimited IT budget; they cannot add an infinite number of switches and high-speed cables to keep pace with users demand for bandwidth and to eliminate a single point of failure on the network. ProCurve Networking offers a LAN aggregation technology that helps companies as diverse as the Denver Regional Transportation District; GMA Network, Inc.; and Venetian Resort-HotelCasino maximize their investment in switches and cabling. Called switch meshing, this technology:

Provides significantly better bandwidth utilization than either Spanning Tree Protocol (STP) or
standard port trunking (which is also called port aggregation) Eliminates a single point of failure Provides quick failover if a switch or link becomes unavailable Is easy to set up Supports flexible configurations

Improves network performance by reducing congestion and load balancing traffic

ProCurve Networking has patented this unique technology, which is available on the ProCurve Switch 3400cl, 5300xl, and 6400cl Series as well as on older ProCurve switches: 1600M, 2400M, 2424M, 4000M, and 8000M. (The older switches have some limitations, which are explained later in this white paper.) This white paper provides an overview of switch meshing and describes how you can configure it for different types of network environments.

Understanding Switch Meshing

In switch meshing, multiple switches are redundantly linked together to form a meshing domain (see Figure 1). Like STP, switch meshing eliminates network loops by detecting redundant links and identifying the best path for traffic. When the meshing domain is established, the switches in that domain use the meshing protocol to gather information about the available paths and to determine the best path between switches. To select the best path, the meshed switches use the following criteria:

Outbound queue depth, or the current outbound load factor, for any given outbound port in a
possible path

Port speed, based on factors such as 10 Mbps, 100 Mbps, 1000 Mbps (or 1 Gbps), 10 Gbps,
full-duplex, or half-duplex

Inbound queue depth for any destination switch in a possible path Increased packet drops, indicating an overloaded port or switch
In Figure 1, for example, the workstations attached to Switch 3 need to access the servers attached to Switch 2. Based on the meshing criteria, the switches in this meshing domain identify the best path as follows: they transmit the data from Switch 3 to Switch 1, which forwards it to Switch 2. Switch 2, in turn, sends the data to its destinationone of the servers.

Figure 1. Identifying the Best Paths Through a Meshing Domain

Unlike STP, however, switch meshing does not permanently block the unused path between two switches. Within a meshing domain, all the paths between switches remain open, and switches can distribute traffic across these available paths as needed to maintain the same latency from path to path. Most of the criteria used to identify the best path are based on network conditions. The outbound queue depth, inbound queue depth, and packet drops all indicate which switches and ports are handling the most traffic. As you would expect, the values for these criteria change over time. To load balance traffic, the meshed switches periodically check this information, identify the best paths based on current network conditions, and then communicate this information to all the other switches in the meshing domain. The switches use the current path information to assign paths on a per-connection basis. When a new connection is established between a source address and a destination address, the traffic between the pair stays on the assigned path until the connection ages out or the path experiences a break. When the two devices later send new traffic, the switches use the best path at that time. Because network conditions may have changed, the best path may not be the same. For example, due to an increase in traffic, the meshed switches may detect packet drops because a port is overloaded. In this case, the switches will send the new traffic through a different path than was used previously. In the meshing domain shown in Figure 1, for example, the links between Switches 1 and 2 handle a high volume of traffic and occasionally become saturated. When a workstation attached to Switch 3 sends data to a server attached to Switch 2, Switch 3 sends the data to Switch 4, rather than to Switch 1. Switch 4 then sends the data to Switch 2 for delivery to its final destination (see Figure 2).

Figure 2. Identifying Alternate Paths Through a Meshing Domain

Many companies use port trunking to increase the available bandwidth between two switches that handle a high volume of traffic. Although port trunking can load balance traffic between the two switches, the increase in network performance is limited to one cable segment. Switch meshing, on the other hand, can load balance traffic over multiple switches, significantly increasing the speed at which data can be transmitted between two devices. Switch meshing enables companies to use their existing bandwidth efficiently, while still protecting against link or switch failure. If a link in the meshing domain fails, switch meshing is designed for fast convergence. Because redundant links remain open, meshed switches can select an alternate path in less than one second. Users are unaware that a problem occurred because they have access to their mission-critical applications.

Creating Meshing Domains

Before you implement switch meshing, you must evaluate your companys network: specifically, you must identify bottlenecks and busy or saturated links, and pinpoint single points of failure. As you evaluate your companys network, you should ask questions such as:

Which switches and links handle the most traffic? Have users complained of network performance problems? If so, you must identify where the
slowdown is occurring. If your company has a help desk, the help desk technicians may record such complaints. If complaints are not tracked, however, you may want to survey users who can at least provide anecdotal information, which you can use to begin your evaluation. redundant links to load balance traffic?

If you are using STP, have bottlenecks developed that could be relieved if you could use

 Which switches and links provide services for the entire company? For example, if an edge

switch went down, that failure would affect a group of users. However, if a core switch went down, that failure would affect the entire company. Although both failures would reduce your companys productivity, losing network services for the entire company would obviously be more costly than losing network services for a group of users. your company is a call center, the software program that representatives use to gather information about customers is critical to the operation of the company and must be available at all times.

Which switches and links deliver mission-critical services to the company? For example, if

During your evaluation, you may want to perform some simple tests during peak operations to determine latency on different links. You can use the ping command to determine the latency between two endpoints. When you send a ping to a remote device, the results show the roundtrip time in milliseconds, which will give you a basic idea of the latency when traffic is transmitted between two devices.

Types of Meshing Domains

After you identify the areas that need additional bandwidth or redundant links, you can determine which type of meshing domain you need to implement. There are two types of meshing domains:

Fully connected meshing domain Partially connected meshing domain

In a fully connected meshing domain, every switch is directly connected to every other switch in the domain (see Figure 3). This type of meshing domain provides the highest degree of availability because it eliminates every single point of failure between the switches. You can configure multiple meshed ports between each switch to provide higher bandwidth.

Figure 3. Fully Connected Meshing Domain

You may want to implement a fully connected meshing domain for core network switches that provide connectivity to critical services such as applications, the Internet, or email. Then, if one link goes down, users can still access the services they need to perform their jobs. In a partially connected meshing domain, only some switches are directly connected to other switches. For example, in Figure 4, a meshed backbone connects Switches 1 and 2. Switches 3, 4, 5, 6, 7, and 8 are connected to switches 1 and 2. However, Switches 3, 4, 5, 6, 7, and 8 are not directly connected to each other. You may want to implement a partially connected meshing domain if it includes both core and edge switches and it is not vital that the edge switches be directly connected to each other.

Figure 4. Partially Connected Meshing Domain

The type of meshing domain that you implement determines the number of switches allowed in the domain. If you establish a fully connected meshing domain, the domain can contain a maximum of five switches. A partially connected meshing domain, on the other hand, can include a maximum of 12 switches. For best performance, however, ProCurve Networking recommends that a partially connected meshing domain contain eight switches. The shortest path between any two nodes in the meshing domain can include no more than five hops. A path of six or more meshed hops between two nodes is unusable. In most meshing topologies, however, a shorter path is normally available, so switches will use paths of five hops or fewer through the meshing domain.

Meshed and Non-meshed Ports

A meshed switch can have some ports in the meshing domain and some ports outside the meshing domain. You enable meshing for point-to-point links between two switches that are part of a meshing domain. You do not enable meshing for ports that connect to:

Devices such as workstations, servers, or printers Switches that are outside the meshing domain Hubs
If all the ports on a switch are enabled for meshing, the switch is called a meshed core switch. If some ports are meshed and others are not, the switch is called a meshed edge switch. (These designations refer to the switchs role in the meshing domain, not its role in the network itself.) Most switches in a meshing domain will be meshed edge switches. You can enable up to 24 meshed ports on a switch, and all of the meshed ports on a given switch belong to the same meshing domain. If a switch port is not configured for meshing, you should not connect it to a meshed port on another switch. If you connect a meshed port to a non-meshed port, the meshed port will shut down. For this reason, meshing domains do not allow the following:

Hub links between meshed switch links Switches that are not configured for switch meshing

Different Speeds for Links

Unlike trunked (or aggregated) ports, redundant links in a meshing domain can be of different types and speeds. You can connect two switches with any combination of 10 Mbps, 100 Mbps, 1 Gbps, or 10 Gbps ports. For example, you might connect two switches through a 100 Mbps port and a 1 Gbps port. This flexibility enables you to use your companys existing cables and switch capabilities.

Easy Configuration
When your company receives additional budget to upgrade cabling or add switches to its network, switch meshing makes it easy to make these changes. Configuring switch meshing is a simple process: you use the mesh command to add ports to the meshing domain. Likewise, you can use the no mesh command to remove ports from the meshing domain. After adding or removing a meshed port on a switch, you must reboot the switch for changes to take effect. You can avoid repeated system disruptions by configuring switch meshing on all ports before you implement the meshing domain. If a meshed port detects a non-meshed port on the opposite end of a point-to-point connection, the link is blocked. As you bring up switch meshing on various switches, some meshed ports may be blocked temporarily. These conditions should clear themselves after you configure all the switches for meshing and then reboot them. To reduce the effect of blocked ports while the meshing domain is established, you can configure meshing either before installing the meshed switches in the network or before connecting the cables between the meshed ports.

Connecting Multiple Meshing Domains

Depending on your network, you may want to create multiple meshing domains. For example, you may want to create a meshing domain that encompasses the core switches on the network. You may also want to create a separate meshing domain for the edge switches that provide connectivity for departments such as customer service and accounting. To connect two or more meshing domains, you can use either a non-meshed switch or a nonmeshed link. In Figure 5, for example, a non-meshed link connects two switches in two different meshing domains. The ports that connect these two meshing domains are not enabled for meshing.

Figure 5. Connecting Multiple Meshing Domains

Preventing Broadcast Storms

Although unicast traffic can be routed across alternate paths in a meshing domain, broadcast and multicast traffic must be handled differently. To prevent broadcast or multicast storms, each switch in a meshing domain should receive only one copy of a broadcast or multicast packet. When the meshing domain is established, each switch identifies a broadcast path. (Broadcast and multicast traffic entering the meshing domain from different edge switches will probably take different paths.) Unless a link or a switch fails, the meshed switch will maintain this broadcast path. When a meshed edge switch receives a broadcast through a non-meshed port, it floods the broadcast out all its other non-meshed ports. However, it sends the broadcast out only the meshed ports that are part of the broadcast path. As a result, only one copy of the broadcast reaches the other meshed edge switches for broadcast out their non-meshed ports. Meshed core switches send the broadcast only through ports (or paths) that link to separate meshed edge switches. That is, meshed core switches send the broadcast only on the broadcast path. Handling broadcast and multicast traffic in this way helps keep the latency for these packets to each switch as low as possible.

Locating Unknown Destinations

Within a meshing domain, switches exchange address information from their switch address tables, so that they can transmit packets across the best path to their ultimate destination. If a switch receives a unicast packet with an unknown destination, that switch does not broadcast the packet across the meshing domain. To learn the destination for the unicast packet, the switch sends a query to all the switches in the meshing domain. These switches then send 802.2 test packets through their non-meshed ports. After learning the destination for the unicast packet, meshed switches can forward subsequent packets with the same destination address. To reduce the number of unknown destination packets, you can configure the switches to retain device addresses for longer periods. For IP networks, you can also speed up the discovery process by assigning switches an IP address.

Using STP with Switch Meshing

Although switch meshing provides distinct advantages over STP, you may still want to run STP on your companys network. For example, you might enable STP to eliminate redundant loops outside the meshing domain. You might also enable STP if you use one meshed link and one non-meshed link to connect two switches. You can use switch meshing in conjunction with STP. Within the meshing domain, the switch meshing protocol identifies redundant links and the best paths for trafficwith the added benefit of using redundant links as needed to load balance traffic (see Figure 6). Outside the meshing domain, STP manages redundant links. STP interprets the meshing domain as a single link and manages any redundant links that are established through non-meshed ports on meshed edge switches

Figure 6. Using STP with Switch Meshing

10

In Figure 6, STP may temporarily block a meshed link because STP interprets the cost on an external trunked link to be less than the cost on a meshed link. If this condition occurs, the meshed switch with the blocked link will automatically increase the cost on the external (nonmeshed) link so that STP will block the external link and unblock the meshed link. This process typically resolves itself in approximately 30 seconds. If you use STP with a meshing domain, all the switches in the domain should run the same version of STPone of the following:

STP (802.1D) RSTP (802.1w) MSTP (802.1s)

If the meshing domain includes a ProCurve Switch 1600M, 2400M, 2424M, 4000M, or 8000M, you must use STP (802.1D) because these switches do not support other versions of STP.

802.1Q VLANs and Meshed Switches

If your companys network includes virtual LANs (VLANs) and you want to use STP, you may want to use either Per VLAN Spanning Tree (PVST) or MSTP (802.1s). If you configure PVST and MSTP correctly, you can better utilize your existing network bandwidth because these protocols can establish different network paths for different VLANs. For example, VLAN 10 may use a different root bridge and network links than VLAN 11 does. However, because you must run multiple instances of Spanning Tree, configuring and troubleshooting such an environment can be complicated. In contrast, configuring switch meshing for a multi-VLAN environment is simple: you configure all the static VLANs on each meshed switch (even if no ports on the switch are assigned to a particular VLAN). When you enable switch meshing for a port, that port automatically becomes a member of all the static VLANs created on the switch. Because all meshed ports are members of all VLANs, they can forward traffic for any VLAN. If you want non-meshed ports on a meshed edge switch to handle certain VLAN traffic, you must manually assign those ports to the VLAN. If a non-meshed port is not a member of a particular VLAN, it cannot forward traffic originating in that VLAN to non-meshed devices. Likewise, the connecting port on the non-meshed switch must belong to the same VLAN, or it cannot receive the traffic. (It is necessary to use a router to communicate between VLANs.) If you have configured VLANs on your companys network, you may also be using GARP VLAN Registration Protocol (GVRP), which supports the creation of dynamic VLANs. A switch port that runs GVRP advertises the static VLANs for which it has been configured. Other GVRP ports can then join those VLANs as necessary for extending the VLANs throughout the network. If one switch in the meshing domain runs GVRP, then all switches in the domain must run it. Otherwise, the meshed switches cannot forward dynamic VLAN traffic. If GVRP is enabled, meshed ports join dynamic VLANs in the same way that non-meshed ports join dynamic VLANs. (The ProCurve 1600M, 2400M, 2424M, 4000M, and 8000M switches do not support GVRP. If a meshing domain includes any of these switches, you must disable GVRP on all switches in the domain.)

Additional Guidelines for Meshing Domains

All switches in the meshing domain must support ProCurve Networkings switch-meshing protocol. In addition, all meshed switches from the same product family must run the same switch software version. For example, if you update the software version on one ProCurve 5300xl Series switch, you must update the software version on any other ProCurve 5300xl Series switch in the meshing domain. (Whether or not you implement switch meshing, ProCurve Networking recommends that you always use the most recent software version available for the switches in your companys network.) When you enable meshing by assigning ports to a meshing domain, you must disable the switchs routing features (IP routing, Routing Information Protocol [RIP], and Open Shortest Path First [OSPF]). The switch-meshing protocol handles routing functions for the switch. If you are using the ProCurve Switch 3400cl or 6400cl Series in a meshing domain, you must also disable stacking on the switch. Stacking is not supported with switch meshing. (Stacking enables a group of up to 16 switches to share a single IP address and broadcast domain.)
11

Enabling Protocols in the Meshing Domain

When configuring a meshing domain, you should ensure that the switches in the domain are configured in the same way. For example, you should make a list of the protocols running on each switch in the domain because the switches in the meshing domain must be configured in the same way. As mentioned earlier, if you enable GVRP on one switch in the meshing domain, you must enable GVRP on the other switches in the meshing domain. In addition, Internet Group Management Protocol (IGMP), or IP multicast, must run on all or none of the switches in a meshing domain. Like trunked ports, the meshing domain appears as a single port to IGMP. With meshing domains, however, IGMP and multicast traffic can be sent over several links in the domain in the same manner as broadcast packets. This same rule applies to Link Layer Discovery Protocol (LLDP). If one switch in the meshing domain is running LLDP, all the switches in the domain must run the same protocol.

Backward Compatibility with Older Switches

As mentioned earlier, if your company has older ProCurve switches (such as the 1600M, 2400M, 2424M, 4000M, or 8000M), you can use these switches in the same meshing domain that includes ProCurve Switches 3400cl, 5300xl, and 6400cl Series. When creating a meshing domain that includes the older switches, however, you must configure the ProCurve Switches 3400cl, 5300xl, and 6400cl Series to use the mesh backward compatibility mode. In addition, the older switches do not allow different switches to use the same media access control (MAC) addresses to connect to a host, even when the connections are through different VLANs. If the meshing domain includes multiple switches that use the same MAC addresses, you cannot add an older switch to the meshing domain. Likewise, you cannot merge a meshing domain that includes duplicate MAC addresses with a meshing domain that includes an older switch.

Conclusion
With switch meshing, your company can make every switch, every port, and every cable count. You can provide high availability while improving network performance. Although STP provides failover capabilities, it does not improve network performance. Instead, redundant links remain idle even when active links become saturated. With switch meshing, on the other hand, switches can use redundant links to offload traffic from overused ports and saturated links, speeding up the transfer of data across the network. Keeping redundant links open has an additional benefit: failovers are quick. If a switch or a link becomes unavailable, switch meshing can rapidly reroute traffic. In addition, switch meshing is easy to configureeven for multi-VLAN environments. And because switch meshing supports flexible configurations, you can use the capabilities of your existing switches. You can connect the switches in a meshing domain through any combination of port and cable speeds.

12

Glossary of Terms
GVRPGARP VLAN Registration Protocol is an application of the Generic Attribute Registration Protocol (GARP). GVRP enables a switch to dynamically create 802.1Q-compliant VLANs on links that are connected to other devices running GVRP. Meshing domainA group of meshed switch ports that exchange meshing protocol packets. Paths between these ports can have multiple redundant links without creating broadcast storms. Meshed edge switchA switch that has some ports configured for switch meshing (and therefore in the switch meshing domain) and some ports not configured for switch meshing (and therefore outside the domain). Meshed core switchA switch that has all ports configured for switch meshing. Meshed portA port that is configured to exchange meshing protocol packets. Non-meshed portA port that is not configured to exchange meshing protocol packets. VLANsA Virtual LAN (VLAN) is comprised of multiple ports operating as members of the same subnet (broadcast domain). Ports on multiple devices can belong to the same VLAN, and traffic moving between ports in the same VLAN is bridged (or switched). A static VLAN is an 802.1Q-compliant VLAN configured with one or more ports that remain members, regardless of traffic usage. A dynamic VLAN is an 802.1Q-compliant VLAN membership that the switch temporarily creates on a port to provide a link to another port in the same VLAN on another device.

13

To find out more about ProCurve Networking products and solutions, visit our web site at
www.procurve.com

2005 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. XXXX-XXXXEN, 10/2005