Deploying Dell Networking MXL and M IOA switches ... - Dell Community

Deploying Dell Networking MXL and M IOA switches into a Cisco Nexus Network VLT and Spanning Tree configuration examples for Dell MXL and M I/O Aggregator modular switches connected to Cisco Nexus 5000 series switches Dell Networking Solutions Engineering May 2016

A Dell EMC Deployment Guide

Revisions Date

Revision

Description

Authors

May 2016

2.0

Revised document throughout. Added VLT feature and M IOA platform.

Jim Slaughter, Dennis Dadey, Davis Smith

Dec 2012

1.1

Original document titled "Deploying the Dell Force10 MXL into a Cisco Nexus Network Environment"

Jason Pearce, Manjesh Siddamurthy, Kevin Horton

Copyright © [2012-2016] Dell Inc. or its subsidiaries. All Rights Reserved. Except as stated below, no part of this document may be reproduced, distributed or transmitted in any form or by any means, without express permission of Dell. You may distribute this document within your company or organization only, without alteration of its contents. THIS DOCUMENT IS PROVIDED “AS-IS”, AND WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED. IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE SPECIFICALLY DISCLAIMED. PRODUCT WARRANTIES APPLICABLE TO THE DELL PRODUCTS DESCRIBED IN THIS DOCUMENT MAY BE FOUND AT: http://www.dell.com/learn/us/en/vn/terms-of-sale-commercial-and-public-sector-warranties Performance of network reference architectures discussed in this document may vary with differing deployment conditions, network loads, and the like. Third party products may be included in reference architectures for the convenience of the reader. Inclusion of such third party products does not necessarily constitute Dell’s recommendation of those products. Please consult your Dell representative for additional information. Trademarks used in this text: Dell™, the Dell logo, Dell Boomi™, PowerEdge™, PowerVault™, PowerConnect™, OpenManage™, EqualLogic™, Compellent™, KACE™, FlexAddress™, Force10™ and Vostro™ are trademarks of Dell Inc. EMC VNX®, and EMC Unisphere® are registered trademarks of Dell. Other Dell trademarks may be used in this document. Cisco Nexus®, Cisco MDS®, Cisco NX-0S®, and other Cisco Catalyst® are registered trademarks of Cisco System Inc. Intel®, Pentium®, Xeon®, Core® and Celeron® are registered trademarks of Intel Corporation in the U.S. and other countries. AMD® is a registered trademark and AMD Opteron™, AMD Phenom™ and AMD Sempron™ are trademarks of Advanced Micro Devices, Inc. Microsoft®, Windows®, Windows Server®, Internet Explorer®, MS-DOS®, Windows Vista® and Active Directory® are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries. Red Hat® and Red Hat® Enterprise Linux® are registered trademarks of Red Hat, Inc. in the United States and/or other countries. Novell® and SUSE® are registered trademarks of Novell Inc. in the United States and other countries. Oracle® is a registered trademark of Oracle Corporation and/or its affiliates. VMware®, Virtual SMP®, vMotion®, vCenter® and vSphere® are registered trademarks or trademarks of VMware, Inc. in the United States or other countries. IBM® is a registered trademark of International Business Machines Corporation. Broadcom® and NetXtreme® are registered trademarks of QLogic is a registered trademark of QLogic Corporation. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and/or names or their products and are the property of their respective owners. Dell disclaims proprietary interest in the marks and names of others.

2

Deploying Dell Networking MXL and M IOA switches into a Cisco Nexus Network | version 2.0

Table of contents Revisions.............................................................................................................................................................................2 1

Introduction ...................................................................................................................................................................5

2

Hardware ......................................................................................................................................................................7 2.1

Dell PowerEdge M1000e ....................................................................................................................................7

2.2

M1000e MXL and M I/O Aggregator blade switches ..........................................................................................7

2.2.1 Dell Networking MXL 10/40Gb Ethernet Switch .................................................................................................8 2.2.2 Dell PowerEdge M I/O Aggregator .....................................................................................................................8 2.2.3 FlexIO Expansion Modules .................................................................................................................................9 3

Scenario One – VLT Environment..............................................................................................................................10 3.1

VLT Overview ...................................................................................................................................................10

3.2

Spanning Tree with VLT ...................................................................................................................................11

3.3

VLT Interoperability with Cisco Nexus vPC ......................................................................................................12

3.3.1 Dell Networking VLT Components ...................................................................................................................12 3.3.2 Cisco Nexus vPC Components ........................................................................................................................13 3.4

VLT and vPC Configuration ..............................................................................................................................13

3.4.1 MXL VLT Configuration ....................................................................................................................................15 3.4.2 M IOA VLT Configuration .................................................................................................................................18 3.4.3 Nexus vPC Configuration .................................................................................................................................20 3.4.4 Server Network Adapter Configuration .............................................................................................................23 3.5

Commands to verify VLT and vPC ...................................................................................................................23

3.5.1 MXL / M IOA commands and output ................................................................................................................23 3.5.2 Nexus commands and output ...........................................................................................................................28 4

Scenario Two – Non-VLT Environment ......................................................................................................................35 4.1

Overview ...........................................................................................................................................................35

4.2

Configuration ....................................................................................................................................................36

4.3

PVST+ Configuration ........................................................................................................................................36

4.3.1 MXL Configuration ............................................................................................................................................38 4.3.2 Nexus Configuration .........................................................................................................................................41 4.3.3 Server Network Adapter Configuration .............................................................................................................43 4.3.4 Commands to verify PVST+ .............................................................................................................................43 4.4

3

MSTP Configuration .........................................................................................................................................50


4.4.1 MXL Configuration ............................................................................................................................................52 4.4.2 Nexus Configuration .........................................................................................................................................55 4.4.3 Server Network Adapter Configuration .............................................................................................................57 4.4.4 Commands to verify MSTP ...............................................................................................................................57 A

References .................................................................................................................................................................64

B

Components used in this guide ..................................................................................................................................65

C

Terminology ................................................................................................................................................................66

D

Resetting the MXL or M IOA to factory defaults .........................................................................................................68

E

MXL and M IOA Port Mapping ...................................................................................................................................69

F

Modifying Spanning Tree Port Cost and Priority ........................................................................................................70

G

Support and Feedback ...............................................................................................................................................71

About Dell EMC ................................................................................................................................................................71

4


1

Introduction Dell Networking provides customers with the most efficient use of current networking equipment at the lowest cost, while still providing great new technologies focused around the explosive data growth in the industry. Application demands have driven the need for increased bandwidth, lower latency and converged infrastructure in today’s networks. Dell EMC’s portfolio covers these key areas to provide the best in service and customer experience. Today’s businesses find it difficult to keep pace with the changing networking and enterprise landscape. With limited resources, they must support a variety of devices that provide key business functions, deliver reliable and flexible IT services, and provide discernible cost savings.

VRTX WAN

Remote/Branch Office

Internet

Dell Campus Networking Dell Data Center Networking

Campus

Storage Network

Dell Servers Dell Network Controllers and Security

Remote Data Centers

Public Cloud

Dell Storage

Data Center

Networking architecture overview The Dell PowerEdge M1000e chassis provides a revolutionary approach to shared infrastructure for enterprise computing. It combines the density and innovative agility of hyper-scale computing with the easeof-use and efficiency of advanced management technologies. The M1000e chassis provides a straightforward, modular approach to infrastructure and introduces a more practical way to manage the complex IT needs of businesses. The M1000e chassis has the capacity for up to six modular switches including the Dell Networking MXL and M I/O Aggregator (M IOA). This deployment guide provides examples of connecting MXL or M IOA switches to upstream Cisco Nexus 5000 series switches in a fault-tolerant network.

5


1.1

Typographical conventions This document uses the following typographical conventions:

6

Monospace text

Command Line Interface (CLI) examples

Bold monospace text

Commands entered at the CLI prompt

Underlined monospace text

CLI examples that word wrap (text is entered as a single command)

Italic monospace text

Variables in CLI examples


2

Hardware This section describes the hardware used to validate the topologies covered in this deployment guide. Refer to Appendix B for software versions used.

2.1

Dell PowerEdge M1000e The Dell PowerEdge M1000e modular blade enclosure, see Figure 2, provides an extremely reliable and efficient platform for building any IT infrastructure. The design of the M1000e enclosure reduces data center sprawl and IT complexity. It delivers one of the most energy efficient, flexible and manageable blade server implementations on the market. The PowerEdge M1000e supports server modules, network, storage, and cluster interconnect modules (switches and pass-through modules), power supplies, fans, and integrated keyboard-video-mouse (KVM) and Chassis Management Controller (CMC). The PowerEdge M1000e uses redundant and hot-pluggable components throughout to provide maximum uptime. Virtually unlimited in scalability, the PowerEdge M1000e chassis provides ultimate flexibility in server processor and chipset architectures. The M1000e infrastructure supports Intel and AMD server architectures simultaneously, while cutting-edge mechanical, electrical and software interface definitions enable multigenerational server support and expansion. For more information about the Dell PowerEdge M1000e, visit http://www.dell.com/us/business/p/poweredge-m1000e/pd.

Dell PowerEdge M1000e modular blade enclosure

2.2

M1000e MXL and M I/O Aggregator blade switches M1000e blade environments expand quickly and easily. Dell EMC’s complete, scale-on-demand MXL and M I/O Aggregator (M IOA) switches with FlexIO expansion modules support three fully redundant fabrics.

7


2.2.1

Dell Networking MXL 10/40Gb Ethernet Switch Figure 3 shows the Dell Networking MXL, a multilayer blade switch. It has two fixed 40GbE ports on the base module and support for two optional FlexIO Expansion Modules. It provides thirty-two 10GbE internal ports to server blades in the M1000e chassis. The MXL runs version 9 of the Dell Networking Operating System, providing switching, bridging and routing functionality for transmitting data, storage and server traffic.

Expansion Slot 1

Expansion Slot 0

Fixed 40GbE QSFP+ Ports

Dell Networking MXL

2.2.2

Dell PowerEdge M I/O Aggregator Figure 4 shows the Dell PowerEdge M IOA, a preconfigured, low-touch, Layer 2 blade switch. It has two fixed 40GbE ports on the base module and support for two optional FlexIO Expansion Modules. It provides thirtytwo 10GbE internal ports to server blades in the M1000e chassis. The M IOA runs version 9 of the Dell Networking Operating System, providing switching and bridging functionality for transmitting data, storage and server traffic.

I/O Bay 1 Expansion Slot 1

Expansion Slot 0

Fixed 40GbE QSFP+ Ports

Dell PowerEdge M I/0 Aggregator with one FlexIO Expansion Module installed

8


2.2.3

FlexIO Expansion Modules The Dell MXL and M IOA support a combination of four types of FlexIO Expansion Modules as shown in Figure 5. Supported modules include the following:    

Four-port Fibre Channel 8Gb module Four-port 10Gb Base-T FlexIO module (only one 10Gbase-T module can be used per switch) Four-port 10Gb SFP+ FlexIO module Two-port 40Gb QSFP+ FlexIO module

FlexIO expansion modules Environments this document describes can use any supported combination of Ethernet expansion modules (10 GB Base-T, 10Gb SFP+, or 40 GB QSFP+) and the two fixed, QSFP+ ports.

9


3

Scenario One – VLT Environment

3.1

VLT Overview Virtual Link Trunking (VLT) allows physical links between two switching devices to appear as a single virtual link to other networking devices. VLT reduces the role of spanning tree protocols by allowing link aggregation group (LAG) terminations on two separate switches, and by supporting a loop-free topology. VLT provides Layer 2 multipathing, creating redundancy through increased bandwidth, enabling multiple parallel paths between nodes and load-balancing traffic where alternative paths exist. In a non-VLT environment utilizing the spanning tree protocol, redundancy involves blocked ports as shown in Figure 6.

Dell Switch

Nexus Switch

Dell Switch

Nexus Switch

Server

Non-VLT environment with spanning tree blocked ports

10


In a VLT environment, all paths are active, utilizing all available links and switches to their fullest potential as shown in Figure 7. This increases available bandwidth and performance.

Dell Switch

Nexus Switch

Dell Switch

Nexus Switch

Server

VLT Environment with all paths active

3.2

Spanning Tree with VLT As a precautionary measure, Dell EMC recommends running the Rapid Spanning Tree Protocol (RSTP) on MXLs in VLT environments. (See the note below for M IOA.) Because VLT environments are loop-free, simultaneously running spanning tree is optional though considered a best practice in case of switch misconfiguration or improperly connected cables. In a properly configured VLT environment, there will be no ports blocked by spanning tree. Cisco enables Rapid per Vlan Spanning Tree Plus (RPVST+), its implementation of RSTP, on Nexus by default. Notes:  M IOA does not support spanning tree protocols. The M IOA configuration described later in this guide places the system in a preconfigured state called "VLT mode." On the M IOA, VLT mode places all external uplink ports in the same port channel, which cannot be modified. This eliminates the possibility of loops in VLT mode on the M IOA.  You can use the Per-VLAN Spanning Tree Plus (PVST+) protocol instead of RSTP on the MXL. Either protocol will interoperate with Nexus RPVST+ in a VLT environment.

11


3.3

VLT Interoperability with Cisco Nexus vPC In this environment, Dell Networking VLT runs on a pair of MXL or M IOA switches. It interoperates with a pair of Nexus 5000 series switches running Cisco virtual Port Channel (vPC), as shown in Figure 8. Since all paths are active, the result is a highly fault-tolerant environment that makes full use of the throughput capabilities of all switches and links.

VLT Domain

vPC Domain

MXL or MIOA

Nexus 5000

VLTi

vPC Peer Link

MXL or MIOA

Nexus 5000

PowerEdge Blade Server

Dell Networking VLT interoperability with Cisco Nexus vPC Note: The Dell Networking Switch pair in the VLT domain consists of either two MXLs or two M IOAs. VLT does not support mixing one MXL and one M IOA in the same domain.

3.3.1

Dell Networking VLT Components The following are key components of a Dell VLT network: Virtual link trunk (VLT): The combined port channel between an attached device and the VLT peer switches. In Figure 9, Port Channel (Po) 1, Po 2, and Po 128 are VLTs. VLT peer device: One of a pair of devices that are connected with the special port channel known as the VLT interconnect. In Figure 9, the two MXLs or M IOAs are VLT peer devices. VLT interconnect (VLTi): The link used to synchronize states between the VLT peer switches. Both ends must be on 10G or 40G interfaces. In Figure 9, Po 127 is the VLTi. VLT backup link: The backup link monitors the vitality of VLT peer switches. The backup link sends periodic keep alive messages between the VLT peer switches. The backup link is normally on the management network. VLT domain: This domain includes both VLT peer devices, the VLT interconnect and all of the port channels in the VLT connected to attached devices.

12


3.3.2

Cisco Nexus vPC Components The following are key components of a Cisco vPC network: Virtual Port Channel (vPC): The combined port channel between the vPC peer devices and the downstream device. In Figure 9, Po 200 and Po 129 are vPCs. vPC peer device: One of a pair of devices connected with the special port channel known as the vPC peer link. In Figure 9, the two Nexus 5000 switches are vPC peer devices. vPC peer link: The link used to synchronize states between the vPC peer devices. In Figure 9, Po 55 is the vPC peer link. vPC member port: Interfaces that belong to the vPCs. vPC peer-keepalive link: The peer-keepalive link monitors the vitality of a vPC peer Cisco Nexus device. The peer-keepalive link sends configurable, periodic keepalive messages between vPC peer devices. The management network normally provides this connection. vPC domain: The domain that includes both vPC peer devices, the vPC peer-keepalive link, and all of the port channels in the vPC connected to the downstream devices.

3.4

VLT and vPC Configuration This section provides detailed configuration steps to build the VLT/vPC network shown in Figure 9. The cloud shown in the diagram represents the upstream network.

VLT Domain 1 LACP NIC Team Po 1

Blade Server 1 VLAN 10

Te0/1

vPC Domain 1

Po 128 Po 129

Te0/41

E1/1

MXL or MIOA Te0/43

E1/3

Po 200

Nexus 5000 E1/9

Te0/2 Fo0/33


E1/7 E1/8

Fo0/37

Po 127

Po 55

Fo0/33 Fo0/37

E1/7 E1/8

Te0/1 Te0/2 LACP NIC Team Po 2

MXL or MIOA

Te0/43 Te0/41

E1/3 E1/1

E1/9

Nexus 5000

VLT-vPC network diagram

13


Blade Server 1 is in VLAN 10 and Blade Server 2 is in VLAN 20. Blade servers 1 and 2 reside in server slots 1 and 2 of the M1000e chassis respectively. The Dell switches reside in slots A1 and A2 in the back of the M1000e chassis. The configuration sections that follow use the same port numbers and port channel numbers shown in Figure 9. For more information on port mapping of MXL and M IOA ports, see Appendix E. The out-of-band management network is not shown in Figure 9 to minimize clutter. The Dell switches connect to the management network via the CMC network port on the M1000e chassis. Each Nexus switch has a dedicated management Ethernet port connected to the management network. In addition to switch administration, the management network is used by the switches to send VLT/vPC peer-keepalive messages to the management IP address of the peer. Proper configuration of the management interfaces is important and is included in the configuration sections that follow. Two sections, MXL VLT Configuration and M IOA VLT Configuration, provide the Dell Networking switch configuration. The Nexus vPC Configuration section is the same in this environment regardless of whether it is connected to a pair of MXLs or a pair of M IOAs. The configuration text is formatted so it can be cut and pasted into a plain text editor (Microsoft Notepad, Notepad++, for example) or directly into a switch console session. Notes:  The configuration steps were developed with the switches starting at their default configurations. This may be helpful if there is an existing configuration on a switch that conflicts with this environment.  To reset Dell MXL or M IOA switches to factory defaults, step-by-step instructions are provided in Appendix D. The Cisco Nexus 5000 series switches were reset by using the #write erase command followed by #reload. When prompted after reload, "Power On Auto Provisioning" was not used, the admin password was configured, and the Nexus basic configuration dialog was not used. Refer to the Nexus system documentation for more information.  When an MXL or M IOA switch is connected to a Cisco Nexus switch, a number of DCBX (Data Center Bridging Exchange) error messages may be displayed on the serial console. To suppress these messages, either configure DCBX on the switches (beyond the scope of this guide) or simply disable DCB on the Dell Networking switches. To do this, use the command: (conf)#no dcb enable

14


3.4.1

MXL VLT Configuration MXL-A1

15

MXL-A2

Set the hostname, enable RSTP, and configure the out-of-band management address. This example uses a static management IP address. DHCP may be used instead.

Set the hostname, enable RSTP, and configure the out-of-band management address. This example uses a static management IP address. DHCP may be used instead.

enable configure hostname MXL-A1 protocol spanning-tree rstp no disable exit

enable configure hostname MXL-A2 protocol spanning-tree rstp no disable exit

interface ManagementEthernet 0/0 ip address 172.25.112.91/24


If prompted - Proceed with Static IP [confirm yes/no]: Y Configure the management route.


exit management route 0.0.0.0/0 172.25.112.254


Configure the VLTi LAG (without LACP) and the VLTi LAG interfaces.

Configure the VLTi LAG (without LACP) and the VLTi LAG interfaces.

interface Port-channel 127 description "VLTi LAG to MXL-A2" channel-member fortyGigE 0/33,37 no shutdown exit

interface Port-channel 127 description "VLTi LAG to MXL-A1" channel-member fortyGigE 0/33,37 no shutdown exit

interface range fortyGigE 0/33, fortyGigE 0/37 description "VLTi link to MXL-A2" no shutdown exit

interface range fortyGigE 0/33, fortyGigE 0/37 description "VLTi link to MXL-A1" no shutdown exit

Configure the same VLT domain # on both peers. Set the backup (peer-keepalive) address to the management IP of MXL-A2. Set this switch as the VLT primary peer, configure the same VLT domain MAC address on both peers, and configure the unique unit id.

Configure the same VLT domain # on both peers. Set the backup (peer-keepalive) address to the management IP of MXL-A1. Set this switch as the VLT secondary peer, configure the same VLT domain MAC address


MXL-A1

MXL-A2 on both peers, and configure the unique unit id.

16

vlt domain 1 peer-link port-channel 127 back-up destination 172.25.112.92 primary-priority 1 system-mac mac-address 00:aa:bb:cc:dd:ee unit-id 0 exit

vlt domain 1 peer-link port-channel 127 back-up destination 172.25.112.91 primary-priority 2 system-mac mac-address 00:aa:bb:cc:dd:ee unit-id 1 exit

Create the downstream (server-facing) VLT portchannel interfaces and configure ports. Repeat these commands as needed for each server in the M1000e chassis.

Create the downstream (server-facing) VLT portchannel interfaces and configure ports. Repeat these commands as needed for each server in the M1000e chassis.

interface Port-channel 1 description "VLT LAG to Blade Server Slot 1" switchport vlt-peer-lag port-channel 1 spanning-tree rstp edge-port no shutdown exit



interface Port-channel 2 description "VLT LAG to Blade Server Slot 2" switchport vlt-peer-lag port-channel 2 spanning-tree rstp edge-port no shutdown e exit

interface TenGigabitEthernet 0/1 description "Link to Blade Server Slot 1" no switchport no portmode hybrid port-channel-protocol LACP port-channel 1 mode active exit no shutdown exit

interface TenGigabitEthernet 0/1 description "Link to Blade Server Slot 1" no switchport no portmode hybrid port-channel-protocol LACP port-channel 1 mode active exit no shutdown exit

interface TenGigabitEthernet 0/2

interface TenGigabitEthernet 0/2


17

MXL-A1

MXL-A2

description "Link to Blade Server Slot 2" no switchport no portmode hybrid port-channel-protocol LACP port-channel 2 mode active exit no shutdown exit

description "Link to Blade Server Slot 2" no switchport no portmode hybrid port-channel-protocol LACP port-channel 2 mode active exit no shutdown exit

Create the upstream VLT portchannel interface and configure ports.

Create the upstream VLT portchannel interface and configure ports.

interface Port-channel 128 description "VLT LAG to N5K" portmode hybrid switchport vlt-peer-lag port-channel 128 no shutdown exit

interface Port-channel 128 description "VLT LAG to N5K" portmode hybrid switchport vlt-peer-lag port-channel 128 no shutdown exit

interface range TenGigabitEthernet 0/41, TenGigabitEthernet 0/43 description "Link to N5K" no switchport port-channel-protocol LACP port-channel 128 mode active exit no shutdown

interface range TenGigabitEthernet 0/41, TenGigabitEthernet 0/43 description "Link to N5K" no switchport port-channel-protocol LACP port-channel 128 mode active exit no shutdown

Create VLANs 10 and 20 and tag the port channels in the VLANs.

Create VLANs 10 and 20 and tag the port channels in the VLANs.

interface vlan 10 tagged port-channel 1,127-128 no shutdown exit

interface vlan 10 tagged port-channel 1,127-128 no shutdown exit

interface vlan 20 tagged port-channel 2,127-128 no shutdown end

interface vlan 20 tagged port-channel 2,127-128 no shutdown end

Save the configuration.


write

write


MXL configuration notes:  Specification of a VLT system-mac mac-address is optional but Dell EMC recommends this to minimize traffic disruption if a VLT failover occurs. This address is used as the system ID in LACP PDUs, and must be the same on both VLT peers. If a VLT system-mac mac-address is not configured, the MAC address of the VLT primary peer is used.  The system automatically assigns a unique unit ID (0 or 1) to each peer switch upon creation of the VLT domain. Optionally, the unit ID can be manually configured as shown above to minimize the time required for the VLT system to converge when one peer switch reboots.  It is a best practice to configure spanning tree first when a spanning tree protocol is used with VLT,  Use a static LAG on the VLT interconnect (port channel 127 in this example). All other VLT LAGs should use LACP.  The portmode hybrid command allows an Ethernet interface to carry multiple tagged VLANs and a single untagged (also called a native) VLAN simultaneously. If a given port is only expected to carry tagged VLANs or a single untagged VLAN, the portmode hybrid command is not required. In this environment, RTSP communicates with Nexus RPVST+ using the untagged default VLAN (VLAN 1). Since port channel 128 also carries tagged VLANs 10 and 20, port channel 128 is put in hybrid mode. See the Nexus vPC Configuration section to configure the upstream switches.

3.4.2

M IOA VLT Configuration The benefit of the M IOA is ease-of-configuration with VLT mode. VLT mode is a low-touch mode with automated settings and targeted at server administrators. The only items that require configuration in VLT mode are VLANs and the management IP address (if DHCP is not used). VLT mode preconfigures the two fixed 40 Gb ports (Fo 0/33 and Fo 0/37) as the VLT interconnects. They must be used for this purpose. All other external ports are configured for 10 Gb and are added to uplink port channel 128. Note: Before putting the M IOA in VLT mode, restore it to factory defaults as detailed in Appendix D. MIOA-A1

18

MIOA-A2

Set the switch to VLT mode.

Set the switch to VLT mode.

enable config stack-unit 0 iom-mode vlt exit reload

enable config stack-unit 0 iom-mode vlt exit reload

After reloading, set the hostname and configure the out-of-band management address. This example uses a static management IP address. DHCP may be used instead.

After reloading, set the hostname and configure the out-of-band management address. This example uses a static management IP address. DHCP may be used instead.

enable configure

enable configure


MIOA-A1

MIOA-A2

hostname MIOA-A1 interface ManagementEthernet 0/0 ip address 172.25.112.91/24

hostname MIOA-A2 interface ManagementEthernet 0/0 ip address 172.25.112.92/24





Create VLANs 10 and 20 by tagging the server ports in the VLANs.

Create VLANs 10 and 20 by tagging the server ports in the VLANs.

interface TenGigabitEthernet 0/1 vlan tagged 10 interface TenGigabitEthernet 0/2 vlan tagged 20 end

interface TenGigabitEthernet 0/1 vlan tagged 10 interface TenGigabitEthernet 0/2 e vlan tagged 20 end



write

write

M IOA configuration notes:  To verify the system is in VLT mode, run the command show system stack-unit 0 iom-mode. The "Boot-Mode" field must be "vlt".  The system creates the server-facing port channels automatically on the M IOA after LACP configuration on the blade server network adapters is complete as described in Server Network Adapter Configuration.  By default, network ports on the blade servers installed in the M1000e chassis are down until the uplink port channel (po 128) becomes operational on the M IOA. This is due to a feature called Uplink Failure Detection (UFD) whereby, when upstream connectivity fails, the M IOA disables the downstream links. See the M IOA configuration guide (link in Appendix A) for more information on UFD.

19


3.4.3

Nexus vPC Configuration This section describes vPC configuration on the Cisco Nexus 5000 series switches. This section is the same regardless of whether the Nexus switches are connected to a pair of MXLs or M IOAs. N5K-1

20

N5K-2

Enable LACP and vPC features. Create VLANs 10 and 20. Configure this switch as the primary spanning tree root bridge for the native VLAN (VLAN 1) and VLANs 10 and 20.

Enable LACP and vPC features. Create VLANs 10 and 20. Configure this switch as the secondary spanning tree root bridge for the native VLAN (VLAN 1) and VLANs 10 and 20.

configure feature lacp feature vpc vlan 10,20 spanning-tree vlan 1,10,20 root primary

configure feature lacp feature vpc vlan 10,20 spanning-tree vlan 1,10,20 root secondary

Configure the hostname, the management interface address and the management route.


hostname N5K-1 interface mgmt 0 ip address 172.25.115.37/24 vrf member management no shutdown exit vrf context management ip route 0.0.0.0/0 172.25.115.254 exit


Create the VPC domain. Assign a role priority of 1 to make this switch the vPC primary. Specify the management IP of switch N5K-1 for the peer-keepalive link.

Create the VPC domain. Assign a role priority of 65535 to make this switch the vPC secondary. Specify the management IP of switch N5K-2 for the peer-keepalive link.

vpc domain 1 role priority 1 peer-keepalive dest 172.25.115.36 exit

vpc domain 1 role priority 65535 peer-keepalive dest 172.25.115.37 exit

Configure the port channel and port channel members for the vPC peer link.

Configure the port channel and port channel members for the vPC peer link.


21

N5K-1

N5K-2

interface port-channel 55 description "vPC Peer-Link to N5K-2" switchport mode trunk switchport trunk allowed vlan 1,10,20 no shutdown vpc peer-link exit

interface port-channel 55 description "vPC Peer-Link to N5K-1" switchport mode trunk switchport trunk allowed vlan 1,10,20 no shutdown vpc peer-link exit

interface ethernet 1/7-8 description "Link to N5K-2" switchport mode trunk switchport trunk allowed vlan 1,10,20 channel-group 55 mode active no shutdown exit

interface ethernet 1/7-8 description "Link to N5K-1" switchport mode trunk switchport trunk allowed vlan 1,10,20 channel-group 55 mode active no shutdown exit

Configure the port channel and port channel member links for downstream connectivity to the Dell Networking switches.

Configure the port channel and port channel member links for downstream connectivity to the Dell Networking switches.

interface port-channel 129 description "vPC to Dell" switchport mode trunk switchport trunk allowed vlan 1,10,20 vpc 129 no shutdown exit

interface port-channel 129 description "vPC to Dell" switchport mode trunk switchport trunk allowed vlan 1,10,20 vpc 129 no shutdown exit

interface ethernet 1/1, ethernet 1/3 description "Link to Dell" switchport mode trunk switchport trunk allowed vlan 1,10,20 channel-group 129 mode active no shutdown exit

interface ethernet 1/1, ethernet 1/3 description "Link to Dell" switchport mode trunk switchport trunk allowed vlan 1,10,20 channel-group 129 mode active no shutdown exit

Configure the port channel and port channel members for upstream connectivity to the external network.

Configure the port channel and port channel members for upstream connectivity to the external network


N5K-1

N5K-2

interface port-channel 200 description "vPC to External Network" switchport mode trunk switchport trunk allowed vlan 10,20 vpc 200 no shutdown exit

interface port-channel 200 description "vPC to External Network" switchport mode trunk switchport trunk allowed vlan 10,20 vpc 200 no shutdown exit

interface ethernet 1/9 description "Link to External Network" switchport mode trunk switchport trunk allowed vlan 10,20 channel-group 200 mode active no shutdown end

interface ethernet 1/9 description "Link to External Network" switchport mode trunk switchport trunk allowed vlan 10,20 channel-group 200 mode active no shutdown end

Save the configuration

Save the configuration

copy running-config startup-config


Nexus configuration notes:  It is a best practice to make the spanning tree root bridge as close as possible to the network core. In this environment, the Nexus switches will serve as the spanning tree root primary and secondary switches. On Nexus, the “primary” command above sets the priority to 24576, and the “secondary” command above sets the priority to 28672. The MXL switches each have a default priority of 32768. The switch with the lowest priority becomes the root bridge, so N5K-1 becomes the root bridge for the network and N5K-2 becomes the alternate.  This configuration uses the native VLAN (VLAN 1) for spanning-tree communication between the Nexus switches running RPVST+ and the MXL switch running RSTP. If the configuration uses M IOA instead of MXL, VLAN 1 does not need to be allowed on port channel 129 since the M IOA does not use spanning tree.  The peer-keepalive address is the management address of the vPC peer switch. This setting allows keepalive heartbeat packets to flow through the management network.

22


3.4.4

Server Network Adapter Configuration In this VLT environment, the server adapter ports are configured as LACP NIC teams. On Blade Server 1, the team is tagged in VLAN 10. On Blade Server 2, the team is tagged in VLAN 20. Note: Refer to the network adapter or operating system documentation for NIC teaming instructions. The network adapter software may refer to LACP as 802.3ad or Dynamic Link Aggregation. For an overview of NIC teaming in Windows 2012 R2, see http://technet.microsoft.com/en-us/library/hh831648.aspx. The network adapter sees the MXL pair or M IOA pair as a single switch. For a network adapter with two 10 Gb ports, the LACP NIC team has 20 Gbps bandwidth with both ports active.

3.5

Commands to verify VLT and vPC

3.5.1

MXL / M IOA commands and output Verify the configuration on the Dell Networking switches using the commands listed in Table 1: Dell Networking show commands Command

Output

show running-config vlt

Displays the running VLT domain configuration

show vlt brief

Displays the role (Primary or Secondary), and the status of the ICL Link, Heartbeat and VLT Peer

show vlt detail

Shows the VLT LAG information for both the local and VLT peer switch (The LAG Status must be UP in both the Local and Peer columns for the LAG to be functional.)

show vlt mismatch

Shows configuration inconsistencies between the VLT peers that may result in VLT errors

show mac-address-table

Displays the MAC address table Addresses learned from the VLT peer are marked with (N)

show spanning-tree rstp brief

VLT port channels are identified and are in the forwarding state

Note: MXL and M IOA command output is similar. To avoid duplicating all output, the command output examples that follow change MXL and M IOA switch names to DELL-A1 and DELL-A2, respectively.

3.5.1.1

show running-config vlt This command displays the VLT domain configuration settings. The back-up destination is the management IP address of the peer switch. The system-mac mac-address, if configured, must be identical on both switches.

23


DELL-A1#show running-config vlt ! vlt domain 1 peer-link port-channel 127 back-up destination 172.25.112.92 primary-priority 1 system-mac mac-address 00:aa:bb:cc:dd:ee unit-id 0 DELL-A2#show running-config vlt ! vlt domain 1 peer-link port-channel 127 back-up destination 172.25.112.91 primary-priority 2 system-mac mac-address 00:aa:bb:cc:dd:ee unit-id 1

3.5.1.2

show vlt brief The ICL Link Status, Heat Beat Status, and VLT Peer Status must all be up. One switch is primary and the other secondary. DELL-A1#show vlt brief VLT Domain Brief -----------------Domain ID: Role: Role Priority: ICL Link Status: HeartBeat Status: VLT Peer Status: Local Unit Id: Version: Local System MAC address: Remote System MAC address: Configured System MAC address: Remote system version: Delay-Restore timer: Delay-Restore Abort Threshold: Peer-Routing : Peer-Routing-Timeout timer: Multicast peer-routing timeout:

24

1 Primary 1 Up Up Up 0 6(6) 00:1e:c9:f1:04:ba 00:1e:c9:f1:03:92 00:aa:bb:cc:dd:ee 6(6) 90 seconds 60 seconds Disabled 0 seconds 150 seconds


DELL-A2#show vlt brief VLT Domain Brief -----------------Domain ID: Role: Role Priority: ICL Link Status: HeartBeat Status: VLT Peer Status: Local Unit Id: Version: Local System MAC address: Remote System MAC address: Configured System MAC address: Remote system version: Delay-Restore timer: Delay-Restore Abort Threshold: Peer-Routing : Peer-Routing-Timeout timer: Multicast peer-routing timeout:

3.5.1.3

1 Secondary 2 Up Up Up 1 6(6) 00:1e:c9:f1:03:92 00:1e:c9:f1:04:ba 00:aa:bb:cc:dd:ee 6(6) 90 seconds 60 seconds Disabled 0 seconds 150 seconds

show vlt detail The Local and Peer VLT LAGs are all up when all connected devices are up. DELL-A1#show vlt detail Local LAG Id Peer LAG Id ------------ ----------1 1 2 2 128 128

Local Status -----------UP UP UP

Peer Status ----------UP UP UP

Active VLANs ------------10 20 1, 10, 20

Notes: Output is identical on switch A2.

3.5.1.4

show vlt mismatch There should be no output to this command on either switch. If there is, resolve the mismatch. Mismatch examples include incompatible VLT configuration settings, VLAN differences, different switch operating system versions and spanning-tree inconsistencies. DELL-A1#show vlt mismatch DELL-A1#

Note: Output is identical on switch A2.

25


3.5.1.5

show mac-address-table Displays the MAC address table. Addresses learned from the VLT peer are marked with (N). DELL-A1#show mac-address-table Codes: *N - VLT Peer Synced MAC *I - Internal MAC Address used for Inter Process Communication VlanId Mac Address Type Interface 1 00:26:0b:f1:f0:81 Dynamic Po 128 1 00:2a:6a:f7:5b:68 Dynamic Po 128 1 00:2a:6a:f7:5b:6a Dynamic (N) Po 128 1 00:2a:6a:f7:80:48 Dynamic (N) Po 128 1 00:2a:6a:f7:80:4a Dynamic Po 128 10 00:10:18:d6:b3:e0 Dynamic Po 128 10 00:26:0b:f1:f0:81 Dynamic Po 128 10 e0:db:55:11:7b:44 Dynamic (N) Po 1 20 00:10:18:d6:b3:e0 Dynamic Po 128 20 00:26:0b:f1:f0:81 Dynamic Po 128 20 bc:30:5b:ef:d8:e0 Dynamic Po 2

State Active Active Active Active Active Active Active Active Active Active Active

DELL-A2#show mac-address-table Codes: *N - VLT Peer Synced MAC *I - Internal MAC Address used for Inter Process Communication VlanId Mac Address Type Interface 1 00:26:0b:f1:f0:81 Dynamic (N) Po 128 1 00:2a:6a:f7:5b:68 Dynamic (N) Po 128 1 00:2a:6a:f7:5b:6a Dynamic Po 128 1 00:2a:6a:f7:80:48 Dynamic Po 128 1 00:2a:6a:f7:80:4a Dynamic (N) Po 128 10 00:10:18:d6:b3:e0 Dynamic (N) Po 128 10 00:26:0b:f1:f0:81 Dynamic (N) Po 128 10 e0:db:55:11:7b:44 Dynamic Po 1 20 00:10:18:d6:b3:e0 Dynamic (N) Po 128 20 00:26:0b:f1:f0:81 Dynamic (N) Po 128 20 bc:30:5b:ef:d8:e0 Dynamic (N) Po 2

26


State Active Active Active Active Active Active Active Active Active Active Active

3.5.1.6

show spanning-tree rstp brief Note: M IOA does not support this command. The Root ID must match the MAC address of the Nexus switch configured as the spanning tree root bridge. RSTP runs in this environment as a precautionary measure only. If any ports are blocked (“BLK” in the "Sts" column), there is likely a configuration or cable connection issue causing a loop that has been blocked by spanning tree. The VLT or VLTi port channels will be identified as VLT or VLTi and should all be in the forwarding state (“FWD” in the "Sts" column) on both Dell switches. Ports that are administratively shutdown will appear as "DIS." MXL-A1#show spanning-tree rstp brief Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24577, Address 002a.6af7.5bbc Root Bridge hello time 2, max age 20, forward delay 15 Bridge ID Priority 32768, Address 001e.c9f1.04ba Configured hello time 2, max age 20, forward delay 15 Bpdu filter disabled globally Interface Name PortID ---------- -------Po 1 128.2 Po 2 128.3 Po 127 128.128 Po 128 128.129 Te 0/2 128.131 | (output truncated)

Prio ---128 128 128 128 128

Cost ------1800 1800 600 1400 2000

Interface Name Role PortID ---------- ------ -------Po 1 Desg 128.2 Po 2 Desg 128.3 Po 127 Desg 128.128 Po 128 Root 128.129 Te 0/2 Dis 128.131 | (output truncated)

Prio ---128 128 128 128 128

Sts ----------FWD(vlt) FWD(vlt) FWD(vltI) FWD(vlt) DIS

Cost ------1800 1800 600 1400 2000

Cost ------1400 1400 1400 1400 1400

Sts ----------FWD FWD FWD FWD DIS

Designated Bridge ID -------------------32768 001e.c9f1.04ba 32768 001e.c9f1.04ba 32768 001e.c9f1.04ba 24577 0023.04ee.be01 32768 001e.c9f1.04ba

Cost ------1400 1400 1400 1400 1400

MXL-A2#sh spanning-tree rstp brief Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24577, Address 002a.6af7.5bbc Root Bridge hello time 2, max age 20, forward delay 15 Bridge ID Priority 32768, Address 001e.c9f1.0392 Configured hello time 2, max age 20, forward delay 15 Bpdu filter disabled globally

27


Link-type --------(vlt) P2P (vlt) P2P (vltI)P2P (vlt) P2P P2P

PortID -------128.2 128.3 128.128 144.128 128.131

Edge ---Yes Yes No No No

Bpdu Filter -----No No No No No

Interface Name PortID ---------- -------Po 1 128.2 Po 2 128.3 Po 127 128.128 Po 128 128.129 Te 0/2 128.131 | (output truncated)

Prio ---128 128 128 128 128

Cost ------1800 1800 600 1400 2000

Interface Name Role PortID ---------- ------ -------Po 1 Desg 128.2 Po 2 Desg 128.3 Po 127 Root 128.128 Po 128 Root 128.129 Te 0/2 Dis 128.131 | (output truncated)

3.5.2

Prio ---128 128 128 128 128

Sts ----------FWD(vlt) FWD(vlt) FWD(vltI) FWD(vlt) DIS

Cost ------1800 1800 600 1400 2000

Cost ------2000 2000 2000 2000 2000

Sts ----------FWD FWD FWD FWD DIS

Designated Bridge ID -------------------32768 001e.c9f1.0392 32768 001e.c9f1.0392 32768 001e.c9f1.04ba 32768 001e.c9f1.0392 32768 001e.c9f1.0392

Cost ------2000 2000 2000 2000 2000

Link-type --------(vlt) P2P (vlt) P2P (vltI)P2P (vlt) P2P P2P

PortID -------128.2 128.3 128.128 128.129 128.131

Edge ---Yes Yes No No No

Bpdu Filter -----No No No No No

Nexus commands and output Verify the configuration on the Nexus switches using the commands listed in Table 2: Nexus show commands

3.5.2.1

Command

Output

show feature

Reports whether LACP and vPC are enabled

show running-config vpc

Displays vPC domain configuration

show vpc

Displays vPC peer and LAG status

show vpc consistencyparameters

Displays any configuration inconsistencies between the peers that may need to be corrected

show spanning-tree

vPC port channels are identified and are in the forwarding state

show feature LACP and vPC features must be enabled for this environment. Other features shown are optional. N5K-1# show feature | grep enabled lacp 1 enabled lldp 1 enabled sshServer 1 enabled telnetServer 1 enabled vpc 1 enabled

28


Note: Switch 2 not shown, output is identical.

3.5.2.2

show running-config vpc This command displays the vPC domain and port channel configuration settings. The peer-keepalive destination is the management IP address of the peer switch. N5K-1# show running-config vpc !Command: show running-config vpc !Time: Tue Apr 12 17:53:10 2016 version 7.0(5)N1(1) feature vpc vpc domain 1 role priority 1 peer-keepalive destination 172.25.115.36 delay restore 150 interface port-channel55 vpc peer-link interface port-channel129 vpc 129 interface port-channel200 vpc 200 N5K-2# show running-config vpc !Command: show running-config vpc !Time: Tue Apr 12 17:54:30 2016 version 7.0(5)N1(1) feature vpc vpc domain 1 role priority 65535 peer-keepalive destination 172.25.115.37 delay restore 150 interface port-channel55 vpc peer-link interface port-channel129 vpc 129

29


interface port-channel200 vpc 200

3.5.2.3

show vpc This command displays the vpc status. Peer status and vPC keep-alive status must be as shown. The consistency status fields should all show “success”. If not, see the show vpc consistency-parameters command. N5K-1# show vpc Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id Peer status vPC keep-alive status Configuration consistency status Per-vlan consistency status Type-2 consistency status vPC role Number of vPCs configured Peer Gateway Dual-active excluded VLANs Graceful Consistency Check Auto-recovery status

: : : : : : : : : : : :

1 peer adjacency formed ok peer is alive success success success primary 2 Disabled Enabled Enabled (timeout = 240 seconds)

vPC Peer-link status --------------------------------------------------------------------id Port Status Active vlans ---------- -------------------------------------------------1 Po55 up 1,10,20 vPC status ---------------------------------------------------------------------------id Port Status Consistency Reason Active vlans ------ ----------- ------ ----------- -------------------------- ----------129 Po129 up success success 1,10,20 200 Po200 up success success 10,20 N5K-2# show vpc Legend: (*) - local vPC is down, forwarding via vPC peer-link vPC domain id Peer status vPC keep-alive status Configuration consistency status

30

: : : :

1 peer adjacency formed ok peer is alive success


Per-vlan consistency status Type-2 consistency status vPC role Number of vPCs configured Peer Gateway Dual-active excluded VLANs Graceful Consistency Check Auto-recovery status

: : : : : : : :

success success secondary 2 Disabled Enabled Enabled (timeout = 240 seconds)

vPC Peer-link status --------------------------------------------------------------------id Port Status Active vlans ---------- -------------------------------------------------1 Po55 up 1,10,20 vPC status ---------------------------------------------------------------------------id Port Status Consistency Reason Active vlans ------ ----------- ------ ----------- -------------------------- ----------129 Po129 up success success 1,10,20 200 Po200 up success success 10,20

3.5.2.4

show vpc consistency-parameters This command pinpoints inconsistencies between the vPC peers. Depending on the severity of the misconfiguration, vPC may either warn the user (Type-2 misconfiguration) or suspend the Port Channel (Type-1 misconfiguration). In the specific case of a VLAN mismatch, only the VLAN that differs between the vPC member ports is suspended on the vPC Port Channels. N5K-1# sh vpc consistency-parameters ? global Global Parameters interface Specify interface vlans Vlans vpc Virtual Port Channel configuration N5K-1# sh vpc consistency-parameters global Legend: Type 1 : vPC will be suspended in case of mismatch

31

Name ------------QoS

Type ---2

Network QoS (MTU) Network Qos (Pause) Input Queuing (Bandwidth) Input Queuing (Absolute

2 2 2 2

Local Value ---------------------([], [], [], [], [], []) (1538, 0, 0, 0, 0, 0) (F, F, F, F, F, F) (100, 0, 0, 0, 0, 0) (F, F, F, F, F, F)


Peer Value ----------------------([], [], [], [], [], []) (1538, 0, 0, 0, 0, 0) (F, F, F, F, F, F) (100, 0, 0, 0, 0, 0) (F, F, F, F, F, F)

Priority) Output Queuing (Bandwidth) Output Queuing (Absolute Priority) Vlan to Vn-segment Map STP Mode STP Disabled STP MST Region Name STP MST Region Revision STP MST Region Instance to VLAN Mapping STP Loopguard STP Bridge Assurance STP Port Type, Edge BPDUFilter, Edge BPDUGuard STP MST Simulate PVST IGMP Snooping Group-Limit Allowed VLANs Local suspended VLANs

2 2

(100, 0, 0, 0, 0, 0) (F, F, F, F, F, F)

(100, 0, 0, 0, 0, 0) (F, F, F, F, F, F)

1 1 1 1 1 1

No Relevant Maps Rapid-PVST None "" 0

No Relevant Maps Rapid-PVST None "" 0

1 1 1

Disabled Enabled Normal, Disabled, Disabled Enabled 4000 1,10,20 -

Disabled Enabled Normal, Disabled, Disabled Enabled 4000 1,10,20 -

1 2 -

Note: Output is identical on switch N5K-2.

3.5.2.5

show spanning-tree The Root ID must match the MAC address of the Nexus switch configured as the spanning tree root bridge for each VLAN. RPVST+ runs in this environment as a precautionary measure only. If any ports are blocked (“BLK” in the Sts column) in this topology, there is likely a configuration or cable connection issue. All vPC/vPC peer-link port channels (po 55, 129, 200) are identified as vPC and in an active (forwarding) state on both Nexus switches. N5K-1# show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 24577 Address 002a.6af7.5bbc This bridge is the root Hello Time 2 sec Max Age 20 sec Bridge ID

Priority Address Hello Time

Interface ---------------Po55 Po129

32

Role ---Desg Desg

Forward Delay 15 sec

24577 (priority 24576 sys-id-ext 1) 002a.6af7.5bbc 2 sec Max Age 20 sec Forward Delay 15 sec

Sts --FWD FWD

Cost --------1 1

Prio.Nbr -------128.4150 128.4224

Type -------------------------------(vPC peer-link) Network P2p (vPC) P2p


VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 24586 Address 002a.6af7.5bbc This bridge is the root Hello Time 2 sec Max Age 20 sec Bridge ID


Interface ---------------Po55 Po129 Po200

Role ---Desg Desg Desg


Sts --FWD FWD FWD

Cost --------1 1 1

Prio.Nbr -------128.4150 128.4224 128.4295

Type -------------------------------(vPC peer-link) Network P2p (vPC) P2p (vPC) P2p

VLAN0020 Spanning tree enabled protocol rstp Root ID Priority 24596 Address 002a.6af7.5bbc This bridge is the root Hello Time 2 sec Max Age 20 sec Bridge ID



Role ---Desg Desg Desg




Sts --FWD FWD FWD

Cost --------1 1 1

Prio.Nbr -------128.4150 128.4224 128.4295


N5K-2# show spanning-tree VLAN0001 Spanning tree enabled protocol rstp Root ID Priority 24577 Address 002a.6af7.5bbc Cost 1 Port 4150 (port-channel55) Hello Time 2 sec Max Age 20 sec Bridge ID

33

Priority Address


28673 (priority 28672 sys-id-ext 1) 002a.6af7.8081


Hello Time Interface ---------------Po55 Po129

Role ---Root Desg

2

Sts --FWD FWD

sec

Max Age 20 sec

Cost --------1 1

Prio.Nbr -------128.4150 128.4224

Type -------------------------------(vPC peer-link) Network P2p (vPC) P2p

VLAN0010 Spanning tree enabled protocol rstp Root ID Priority 24586 Address 002a.6af7.5bbc Cost 1 Port 4150 (port-channel55) Hello Time 2 sec Max Age 20 sec Bridge ID



Role ---Root Desg Desg

Sts --FWD FWD FWD



34

Role ---Root Desg Desg


28682 (priority 28672 sys-id-ext 10) 002a.6af7.8081 2 sec Max Age 20 sec Forward Delay 15 sec Cost --------1 1 1

Prio.Nbr -------128.4150 128.4224 128.4295


VLAN0020 Spanning tree enabled protocol rstp Root ID Priority 24596 Address 002a.6af7.5bbc Cost 1 Port 4150 (port-channel55) Hello Time 2 sec Max Age 20 sec Bridge ID



28692 (priority 28672 sys-id-ext 20) 002a.6af7.8081 2 sec Max Age 20 sec Forward Delay 15 sec

Sts --FWD FWD FWD

Cost --------1 1 1

Prio.Nbr -------128.4150 128.4224 128.4295



4

Scenario Two – Non-VLT Environment This section presents an alternative to VLT for legacy Dell Networking MXL installations. It relies on the spanning tree protocol to protect against loops for redundant connections to Nexus switch pairs. Reliance on spanning tree for redundancy as presented here is not the preferred solution. Dell EMC recommends using the loop-free VLT environment presented in Scenario One – VLT Environment for MXL and M IOA switches. Note: The Dell Networking M IOA does not support spanning tree protocols. Therefore, this section does not include M IOA.

4.1

Overview Without VLT (or vPC on Nexus), redundant paths can form loops that must be blocked by a spanning tree protocol as shown in Figure 10.

MXL-A1

N5K-1

MXL-A2

N5K-2

Blade Server

Redundant links blocked by spanning tree in a fully meshed topology If an active link fails or is shut down, the spanning tree protocol enables fault tolerance by placing a blocked link in the forwarding state until the issue is resolved. Per VLAN Spanning Tree Plus (PVST+) and Multiple Spanning Tree Protocol (MSTP) allow for configuration of spanning tree on a per-VLAN basis. This allows network administrators to run different spanning tree instances and better utilize network bandwidth. Paths blocked for one VLAN can be made active for a different VLAN (or a group of VLANs in the case of MSTP).

35


4.2

Configuration The switch configuration examples are divided into two sections based on the spanning tree protocol used: PVST+ Configuration and MSTP Configuration. The configuration text is formatted so it can be cut and pasted into a plain text editor (Microsoft Notepad, Notepad++, for example) or directly into a switch console session. The configuration steps were developed with the switches starting at their default configurations. This may be helpful if there is an existing configuration on a switch that conflicts with this environment. Notes:  To reset Dell MXL switches to factory defaults, step-by-step instructions are provided in Appendix D. The Cisco Nexus 5000 series switches were reset by using the #write erase command followed by #reload. When prompted after reload, "Power on Auto Provisioning" was not used, the admin password was configured, and the Nexus basic configuration dialog was not used. Refer to the Nexus system documentation for more information.  When an MXL switch is connected to a Cisco Nexus switch, a number of DCBX (Data Center Bridging Exchange) error messages may be displayed on the serial console. To suppress these messages, either configure DCBX on the switches (beyond the scope of this guide) or disable DCB on the Dell Networking switches. To do this, use the command: (conf)#no dcb enable

4.3

PVST+ Configuration Per-VLAN Spanning Tree Plus (PVST+) is an implementation of the spanning-tree protocol where a separate spanning tree runs inside each VLAN. The Cisco Nexus implementation of this protocol is called Rapid Per VLAN Spanning Tree Plus (RPVST+) and is enabled on Nexus switches by default. This section provides detailed configuration steps to build the network running PVST+ shown in Figure 11 and Figure 12 . The cloud in the diagrams represents the upstream network. The network has two tagged VLANs: VLAN 30 and VLAN 40. The network adapter on Blade Server 1 is tagged in VLAN 30 and the network adapter on Blade Server 2 is tagged in VLAN 40. Blade servers 1 and 2 reside in slots 1 and 2, respectively, of the M1000e chassis. The MXLs reside in slots A1 and A2 in the back of the M1000e chassis.

36


PVST+ VLAN 30

VLAN 30 Root Bridge

Po101

Te0/1

Te0/41

E1/1

Te0/42

E1/2

MXL-A1

E1/3

N5K-1

Te0/43


Fo0/33

Po102

Te Fo0/37 0/44

E1/9

E1/4

Po 127 Fo0/33 Fo0/37

Te0/1

MXL-A2

E1/7 E1/8

Po 55 E1/7 E1/8

Te 0/43

Po103

E1/3 E1/9 E1/4

Te0/44 Te0/41

E1/1

Te0/42

E1/2

N5K-2 Spanning Tree Blocked Interface

Po104

Interfaces blocked by PVST+ on VLAN 30

PVST+ VLAN 40 Po101

Te0/2

Te0/41

E1/1

Te0/42

E1/2

MXL-A1

E1/3

N5K-1

Te0/43


Fo0/33

Te Fo0/37 0/44

Po102

E1/9 E1/4

Po 127 Fo0/33 Fo0/37

Te0/2

MXL-A2

E1/7 E1/8

Po 55 Te 0/43

E1/7 E1/8 Po103

E1/3

E1/9 E1/4

Te0/44 Te0/41

E1/1

Te0/42

E1/2


Po104 VLAN 40 Root Bridge

Interfaces blocked by PVST+ on VLAN 40 Configuration of switch NK5-1 as VLAN 30's root bridge and switch NK5-2 as VLAN 40's root bridge blocks different port channels for each VLAN. Port channels 101 and 103 forward traffic on VLAN 30 and port channels 102 and 104 forward traffic on VLAN 40. This allows for better utilization of the available links.

37


The configuration sections that follow use the same port numbers and port channel numbers shown in Figure 11 and Figure 12. For more information on port mapping of MXL ports, see Appendix E.

4.3.1

MXL Configuration MXL-A1

MXL-A2

Set the hostname and configure the out-of-band management address.


enable configure hostname MXL-A1

enable configure hostname MXL-A2



If prompted - Proceed with Static IP [confirm yes/no]: y Configure the management route.

If prompted - Proceed with Static IP [confirm yes/no]: y Configure the management route.



Enable PVST+ and globally disable BPDUs on edge ports.

Enable PVST+ and globally disable BPDUs on edge ports.

protocol spanning-tree pvst no disable edge-port bpdufilter default exit Configure the internal server-facing interfaces and make them edge ports.

Configure the internal server-facing interfaces and make them edge ports.

interface range tengigabitethernet 0/1, tengigabitethernet 0/2 no portmode hybrid switchport no shutdown spanning-tree pvst edge-port exit

interface range tengigabitethernet 0/1, tengigabitethernet 0/2 no portmode hybrid switchport no shutdown spanning-tree pvst edge-port exit

Configure the remaining interfaces and port channels.


interface range tengigabitethernet 0/41-42


port-channel-protocol LACP port-channel 101 mode active exit

38

protocol spanning-tree pvst no disable edge-port bpdufilter default exit

port-channel-protocol LACP port-channel 104 mode active exit


MXL-A1 no shutdown exit interface range tengigabitethernet 0/43-44

no shutdown exit interface range tengigabitethernet 0/43-44

port-channel-protocol LACP port-channel 102 mode active exit no shutdown exit


interface range fortyGigE 0/33,fortyGigE 0/37 port-channel-protocol LACP port-channel 127 mode active exit no shutdown exit


interface Port-channel 101 switchport no shutdown exit





39

MXL-A2


Tag the interfaces in VLANs 30 and 40.

Tag the interfaces in VLANs 30 and 40.

int vlan 30 tagged tengigabitethernet 0/1 tagged port-channel 101-102,127 no shutdown exit


int vlan 40 tagged tengigabitethernet 0/2

int vlan 40 tagged tengigabitethernet 0/2


MXL-A1

MXL-A2

tagged port-channel 101-102,127 no shutdown exit end

tagged port-channel 103-104,127 no shutdown exit end



write

write

MXL PVST+ configuration notes:  The portmode hybrid command allows an Ethernet interface to carry multiple tagged VLANs and a single untagged (native) VLAN. If a given port is only expected to carry tagged VLANs or a single untagged VLAN, the portmode hybrid command may be omitted from the configuration. In this environment, both VLANs (30 and 40) are tagged so the hybrid port state is not used.  Dell Networking systems do not expect a PVST+ BPDU on an untagged port. If this situation occurs, the Dell Networking OS places the port in an Error-Disable state. This behavior might result in the network not converging. To prevent the system from executing this action, use the no spanning-tree pvst err-disable cause invalid-pvst-bpdu command on each untagged port connected to another switch.  Only run the spanning-tree pvst edge-port command on ports connecting to servers or other end nodes and never on ports that connect to other switches. This command designates a port as an edge of the spanning tree (only switches participate in spanning tree). This enables it to begin forwarding traffic as soon as its link is active. This happens many seconds before the spanning-tree protocol would otherwise allow it to forward traffic. MXL ports te 0/1-32 can safely be made edge ports. As fixed internal ports, they can only be connected to server blades or storage devices inside the M1000e chassis. The optional command edge-port bpdufilter default stops transmission of BPDUs on edge ports.

40


4.3.2

Nexus Configuration N5K-1

41

N5K-2

Enable the LACP feature.


configure feature lacp






Ensure RPVST+ is enabled. Create the VLANs. Configure spanning tree settings.

Ensure RPVST+ is enabled. Create the VLANs. Configure spanning tree settings.

spanning-tree vlan 30,40 spanning-tree spanning-tree spanning-tree secondary

spanning-tree vlan 30,40 spanning-tree spanning-tree secondary spanning-tree

mode rapid-pvst pathcost method long vlan 30 root primary vlan 40 root

mode rapid-pvst pathcost method long vlan 30 root vlan 40 root primary

Configure the port channels and interfaces.


interface port-channel 101 switchport mode trunk switchport trunk allowed vlan 30,40 exit


interface port-channel 103 switchport mode trunk



N5K-1

42

N5K-2

switchport trunk allowed vlan 30,40 exit

switchport trunk allowed vlan 30,40 exit



interface ethernet 1/1-2 switchport mode trunk switchport trunk allowed vlan 30,40 channel-group 101 mode active exit






interface Ethernet1/9 switchport mode trunk switchport trunk allowed vlan 30,40 exit end

interface Ethernet1/9 switchport mode trunk switchport trunk allowed vlan 30,40 exit end






Nexus configuration notes  Nexus switches enable RPVST+ by default. The example shows the spanning-tree mode rapidpvst command in case the Nexus is not at default settings. To see current spanning tree settings on Nexus, run the command show running-config spanning-tree all.  The Dell Networking OS implementation of PVST+ uses IEEE 802.1s costs as the default port costs. For a more consistent path cost calculation with MXL, run the spanning-tree pathcost method long command on the Nexus switches.  It is a best practice to make the root bridge as close as possible to the network core. In this environment, the Nexus switches serve as the spanning tree root primary / alternate switches. On Nexus, the “primary” command above sets the priority to 24576, and the “secondary” command above sets the priority to 28672. The Dell switches have a default priority of 32768. The switch with the lowest priority becomes the root bridge.

4.3.3

Server Network Adapter Configuration In this topology, the server adapter ports are configured as load balancing NIC teams. Traffic is load balanced across the ports, and the team provides fault tolerance in case of a link failure. On Blade Server 1, the NIC team is tagged in VLAN 30. On Blade Server 2, the NIC team is tagged in VLAN 40. Note: Refer to the network adapter or operating system documentation for NIC teaming instructions. For an overview of NIC teaming in Windows 2012 R2, see http://technet.microsoft.com/enus/library/hh831648.aspx. Network adapters cannot be configured as LACP teams in this environment. To use LACP NIC teams, use VLT as covered in Scenario One – VLT Environment.

4.3.4

Commands to verify PVST+ For this environment, the spanning tree algorithm blocked ports appropriately based on the selection of different root bridges. If further adjustments are desired, additional interface parameters (port cost and port priority) can be adjusted to increase or decrease the probability that a port becomes a forwarding port. For more information, see Appendix F for Dell switches and your Nexus system documentation. In the command output, the Root ID must match the MAC address of the Nexus switch configured as the spanning tree root bridge for the VLAN. Blocked interfaces will be marked as “BLK” in the "Sts" column and forwarding interfaces will be marked as "FWD".

43


4.3.4.1

Spanning tree commands and output – PVST VLAN 30 PVST+ VLAN 30

VLAN 30 Root Bridge

Po101 Te0/41

E1/1

Te0/42

E1/2

MXL-A1

Te0/1

E1/3

N5K-1

Te0/43


Fo0/33

Te Fo0/37 0/44

E1/9

Po102

E1/4

Po 127

Po 55

Fo0/33 Fo0/37

Te0/1

E1/7 E1/8

Te 0/43

MXL-A2

E1/7 E1/8 Po103

Te0/44

Te0/41

Port Channels 101 and 103 are Forwarding

E1/3 E1/9 E1/4

N5K-2

Spanning Tree Blocked Interface

E1/1 E1/2

VLAN 30 blocked interfaces Verify PVST+ on VLAN 30 using the following commands: MXL-A1#show spanning-tree pvst vlan 30 brief VLAN 30 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24606, Address 002a.6af7.5bbc Root Bridge hello time 2, max age 20, forward delay 15 Bridge ID Priority 32768, Address 001e.c9f1.04ba Configured hello time 2, max age 20, forward delay 15 Bpdu filter enabled globally Interface Name PortID -------- -------Po 101 128.102 Po 102 128.103 Po 127 128.128 Te 0/1 128.130

Prio ---128 128 128 128

Interface Name ---------Po 101 Po 102 Po 127 Te 0/1

PortID -------128.102 128.103 128.128 128.130

Role -----Root Altr Altr Desg

Cost -----1800 1800 600 2000

Designated Sts Cost Bridge ID PortID ---------- ------- -------------------- -------FWD 1800 24606 002a.6af7.5bbc 144.100 BLK 1800 28702 002a.6af7.8081 144.101 BLK 1800 32768 001e.c9f1.0392 128.128 FWD 1800 32768 001e.c9f1.04ba 128.130

Prio ---128 128 128 128

Cost ------1800 1800 600 2000

Sts ----------FWD BLK BLK FWD

Cost ------1800 1800 1800 1800

MXL-A2#show spanning-tree pvst vlan 30 brief

44


Link-type --------P2P P2P P2P P2P

Edge ---No No No Yes

BpduFilter ---------No No No No

VLAN 30 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24606, Address 002a.6af7.5bbc Root Bridge hello time 2, max age 20, forward delay 15 Bridge ID Priority 32768, Address 001e.c9f1.0392 Configured hello time 2, max age 20, forward delay 15 Bpdu filter enabled globally Interface Name ---------Po 103 Po 104 Po 127 Te 0/1

PortID -------128.104 128.105 128.128 128.130


Role -----Root Altr Desg Desg

Prio ---128 128 128 128

Cost -----1800 1800 600 2000

PortID -------128.104 128.105 128.128 128.130

Prio ---128 128 128 128

Sts ----------FWD BLK FWD FWD

Cost ------1800 1800 600 2000

Cost ------1800 1800 1800 1800


Designated Bridge ID PortID -------------------- -------24606 002a.6af7.5bbc 144.102 28702 002a.6af7.8081 144.103 32768 001e.c9f1.0392 128.128 32768 001e.c9f1.0392 128.130

Cost ------1800 1800 1800 1800




N5K-1# show spanning-tree vlan 30 VLAN0030 Spanning tree enabled protocol rstp Root ID Priority 24606 Address 002a.6af7.5bbc This bridge is the root Hello Time 2 sec Max Age 20 sec Bridge ID


Interface ---------------Po55 Po101 Po103 Eth1/9

45

Role ---Desg Desg Desg Desg



Sts --FWD FWD FWD FWD

Cost --------1000 1000 1000 2000

Prio.Nbr -------128.4150 128.4196 128.4198 128.137

Type -------------------------------P2p P2p P2p P2p


N5K-2# show spanning-tree vlan 30 VLAN0030 Spanning tree enabled protocol rstp Root ID Priority 24606 Address 002a.6af7.5bbc Cost 1000 Port 4150 (port-channel55) Hello Time 2 sec Max Age 20 sec Bridge ID



46

Role ---Root Desg Desg Desg




Cost --------1000 1000 1000 2000

Prio.Nbr -------128.4150 128.4197 128.4199 128.137

Type -------------------------------P2p P2p P2p P2p


4.3.4.2

Spanning tree commands and output – PVST VLAN 40 PVST+ VLAN 40 Po101

Te0/41

E1/1

Te0/42

E1/2

MXL-A1

Te0/2

E1/3

N5K-1

Te0/43


Fo0/33

Te Fo0/37 0/44

E1/9

Po102

E1/4

Po 127

Po 55

Fo0/33 Fo0/37

MXL-A2

Te0/2


E1/7 E1/8

E1/7 E1/8

Te 0/43

E1/3

Po103

E1/9 E1/4

Te0/44 Te0/41

E1/1

Te0/42

E1/2


Po104 VLAN 40 Root Bridge

VLAN 40 blocked interfaces Verify PVST+ on VLAN 40 using the following commands: MXL-A1#show spanning-tree pvst vlan 40 brief VLAN 40 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24616, Address 002a.6af7.8081 Root Bridge hello time 2, max age 20, forward delay 15 Bridge ID Priority 32768, Address 001e.c9f1.04ba Configured hello time 2, max age 20, forward delay 15 Bpdu filter enabled globally

47


PortID -------128.102 128.103 128.128 128.140

Interface Name ---------Po 101 Po 102

Role -----Altr Root

Prio ---128 128 128 128

Cost -----1800 1800 600 2000

PortID -------128.102 128.103

Prio ---128 128

Sts ----------BLK FWD BLK FWD

Cost ------1800 1800

Cost ------1800 1800 1800 1800

Sts ----------BLK FWD

Designated Bridge ID PortID -------------------- -------28712 002a.6af7.5bbc 144.100 24616 002a.6af7.8081 144.101 32768 001e.c9f1.0392 128.128 32768 001e.c9f1.04ba 128.140

Cost ------1800 1800


Link-type --------P2P P2P

Edge ---No No

BpduFilter ---------No No

Po 127 Te 0/2

Altr Desg

128.128 128.140

128 128

600 2000

BLK FWD

1800 1800

P2P P2P

No Yes

No No

MXL-A2#show spanning-tree pvst vlan 40 brief VLAN 40 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24616, Address 002a.6af7.8081 Root Bridge hello time 2, max age 20, forward delay 15 Bridge ID Priority 32768, Address 001e.c9f1.0392 Configured hello time 2, max age 20, forward delay 15 Bpdu filter enabled globally Interface Name ---------Po 103 Po 104 Po 127 Te 0/2

PortID -------128.104 128.105 128.128 128.140


Role -----Altr Root Desg Desg

Prio ---128 128 128 128

Cost -----1800 1800 600 2000

PortID -------128.104 128.105 128.128 128.140

Prio ---128 128 128 128

Sts ----------BLK FWD FWD FWD

Cost ------1800 1800 600 2000

Cost ------1800 1800 1800 1800


Designated Bridge ID PortID -------------------- -------28712 002a.6af7.5bbc 144.102 24616 002a.6af7.8081 144.103 32768 001e.c9f1.0392 128.128 32768 001e.c9f1.0392 128.140

Cost ------1800 1800 1800 1800




N5K-1# show spanning-tree vlan 40 VLAN0040 Spanning tree enabled protocol rstp Root ID Priority 24616 Address 002a.6af7.8081 Cost 1000 Port 4150 (port-channel55) Hello Time 2 sec Max Age 20 sec Bridge ID



48





Cost --------1000 1000 1000 2000

Prio.Nbr -------128.4150 128.4196 128.4198 128.137

Type -------------------------------P2p P2p P2p P2p


N5K-2# show spanning-tree vlan 40 VLAN0040 Spanning tree enabled protocol rstp Root ID Priority 24616 Address 002a.6af7.8081 This bridge is the root Hello Time 2 sec Max Age 20 sec Bridge ID



49





Cost --------1000 1000 1000 2000

Prio.Nbr -------128.4150 128.4197 128.4199 128.137

Type -------------------------------P2p P2p P2p P2p


4.4

MSTP Configuration Multiple Spanning Tree Protocol (MSTP), specified in IEEE 802.1s and later merged into 802.1Q-2005, is a RSTP-based spanning tree variation. It can be an improvement to PVST+ in some environments. PVST+ allows one spanning tree instance for each VLAN. The 1:1 approach of PVST+ may not be suitable for a large number of VLANs, because each spanning tree instance consumes bandwidth and processing resources. Like PVST+, MSTP allows multiple spanning tree instances. However, MSTP allows the mapping of multiple VLANs to one spanning tree instance to reduce the total number of required spanning tree instances. With MSTP enabled, MST 0 includes all 4094 VLANs by default on the Dell and Nexus switches. Administrators can then group and move VLANs into new instances as needed. This section provides detailed configuration steps to build the network running MSTP shown in Figure 15 and Figure 16. The cloud in the diagrams represents the upstream network. This topology is a simplified example that uses three VLANS: 30, 40, and 50. VLANs 30 and 40 are configured in MST instance 1, and VLAN 50 is configured in MST instance 2. The network adapter on Blade Server 1 is tagged in VLANs 30 and 40 and the network adapter on Blade Server 2 is tagged in VLAN 50. Blade servers 1 and 2 reside in slots 1 and 2, respectively, of the M1000e chassis. The MXLs reside in slots A1 and A2 in the back of the M1000e chassis.

50


MST Instance 1 VLANs 30 & 40

MST 1 Root Bridge

Po101

Te0/1

Te0/41

E1/1

Te0/42

E1/2

MXL-A1

E1/3

N5K-1

Te0/43

Blade Server 1 VLAN 30 & 40

Fo0/33

Te Fo0/37 0/44

Po102

E1/9

E1/4

Po 127 Fo0/33 Fo0/37

Te0/1

MXL-A2

E1/7 E1/8

Po 55 E1/7 E1/8

Te 0/43

Po103

E1/3 E1/9 E1/4

Te0/44 Te0/41

E1/1

Te0/42

E1/2


Po104

Interfaces blocked in MST instance 1

MST Instance 2 VLAN 50 Po101

Te0/2

Te0/41

E1/1

Te0/42

E1/2

MXL-A1

E1/3

N5K-1

Te0/43


Fo0/33

Te Fo0/37 0/44

Po102

E1/9 E1/4

Po 127 Fo0/33 Fo0/37

Te0/2

MXL-A2

E1/7 E1/8

Po 55 Te 0/43

E1/7 E1/8 Po103

E1/3

E1/9 E1/4

Te0/44 Te0/41

E1/1

Te0/42

E1/2


Po104 MST 2 Root Bridge

Interfaces blocked in MST instance 2

51


4.4.1

MXL Configuration MXL-A1 Set the hostname and configure the out-of-band management address.


enable configure hostname MXL-A1 interface ManagementEthernet 0/0 ip address 172.25.112.91/24

enable configure hostname MXL-A2 interface ManagementEthernet 0/0 ip address 172.25.112.92/24

If prompted - Proceed with Static IP

If prompted - Proceed with Static IP

[confirm yes/no]: y Configure the management route.

[confirm yes/no]: y Configure the management route.



Enable MSTP, disable BPDU’s on edge ports, place VLANs 30 and 40 in MST instance 1 and VLAN 50 in MST instance 2.

Enable MSTP, disable BPDU’s on edge ports, place VLANs 30 and 40 in MST instance 1 and VLAN 50 in MST instance 2.

protocol spanning-tree mstp no disable edge-port bpdufilter default msti 1 vlan 30,40 msti 2 vlan 50 exit

protocol spanning-tree mstp no disable edge-port bpdufilter default msti 1 vlan 30,40 msti 2 vlan 50 exit



interface range tengigabitethernet 0/1, tengigabitethernet 0/2 switchport no shutdown spanning-tree mstp edge-port exit

interface range tengigabitethernet 0/1, tengigabitethernet 0/2 switchport no shutdown spanning-tree mstp edge-port exit





port-channel-protocol LACP port-channel 101 mode active exit no shutdown

52

MXL-A2

port-channel-protocol LACP port-channel 104 mode active exit no shutdown


MXL-A1 exit interface range tengigabitethernet 0/43-44

exit interface range tengigabitethernet 0/43-44





interface Port-channel 101 description LINK TO N5K-1 switchport no shutdown exit




interface Port-channel 127 description LINK TO MXL-A2 switchport no shutdown exit

53

MXL-A2

interface Port-channel 127 description LINK TO MXL-A1 switchport no shutdown exit

Tag the interfaces in VLANs 30, 40 and 50.

Tag the interfaces in VLANs 30, 40 and 50.




MXL-A1

MXL-A2



int vlan 50 tagged tengigabitethernet 0/2 tagged port-channel 101-102,127 no shutdown exit end

int vlan 50 tagged tengigabitethernet 0/2 tagged port-channel 103-104,127 no shutdown exit end



write

write

MXL MSTP configuration notes:  The portmode hybrid command allows an Ethernet interface to carry multiple tagged VLANs and a single untagged (native) VLAN. If a given port is only expected to carry tagged VLANs or a single untagged VLAN, the portmode hybrid command may be omitted from the configuration. In this environment, all VLANs are tagged so the hybrid port state is not used.  Only run the spanning-tree mstp edge-port command on ports connecting to servers or other end nodes and never on ports that connect to other switches. This command designates a port as an edge of the spanning tree (only switches participate in spanning tree). This enables it to begin forwarding traffic as soon as its link is active. This happens many seconds before the spanning-tree protocol would otherwise allow it to forward traffic. MXL ports te 0/1-32 can safely be made edge ports. As fixed internal ports, they can only be connected to server blades or storage devices inside the M1000e chassis. The optional command edge-port bpdufilter default stops transmission of BPDUs on edge ports.

54


4.4.2

Nexus Configuration N5K-1

55

N5K-2









Enable MSTP. Create the VLANs. Configure spanning tree settings.

Enable MSTP. Create the VLANs. Configure spanning tree settings.

spanning-tree mode mst vlan 30,40,50 spanning-tree pathcost method long spanning-tree mst 1 root primary spanning-tree mst 2 root secondary spanning-tree mst configuration instance 1 vlan 30,40 instance 2 vlan 50

spanning-tree mode mst vlan 30,40,50 spanning-tree pathcost method long spanning-tree mst 2 root primary spanning-tree mst 1 root secondary spanning-tree mst configuration instance 1 vlan 30,40 instance 2 vlan 50



interface port-channel 101 switchport mode trunk switchport trunk allowed vlan 30,40,50 exit





N5K-1

56

N5K-2

switchport trunk allowed vlan 30,40,50 exit

switchport trunk allowed vlan 30,40,50 exit



interface ethernet 1/1-2 switchport mode trunk switchport trunk allowed vlan 30,40,50 channel-group 101 mode active exit






interface Ethernet1/9 switchport mode trunk switchport trunk allowed vlan 30,40,50 exit

interface Ethernet1/9 switchport mode trunk switchport trunk allowed vlan 30,40,50 exit






Nexus configuration notes:  The Dell Networking OS implementation of MSTP uses IEEE 802.1s costs as the default port costs. For a more consistent path cost calculation with MXL, run the spanning-tree pathcost method long command on the Nexus switches.  It is a best practice to make the root bridge as close as possible to the network core. In this environment, the nexus switches serve as the spanning tree root primary / alternate switches. On Nexus, the “primary” command above sets the priority to 24576 and the “secondary” command above sets the priority to 28672. Dell switches have a default priority of 32768. The switch with the lowest priority becomes the root bridge.

General MSTP configuration notes:  In addition to the MST instance number, MSTP allows for the configuration of name and revision values. All three of these values must match on all connected switches to form an MSTP region. On Nexus and MXL, the default values for the name (null) and revision (0) match, which makes configuration of these fields unnecessary. With other 3rd party switches, these fields may require configuration in order to match.  All bridges in the MSTP region must have the same VLAN-to-instance mapping.

4.4.3

Server Network Adapter Configuration In this topology, the server adapter ports are configured as load balancing NIC teams. Traffic is load balanced across the ports, and the team provides fault tolerance in case of a link failure. On Blade Server 1, the NIC team is tagged in VLANs 30 and 40. On Blade Server 2, the NIC team is tagged in VLAN 50. Note: Refer to the network adapter or operating system documentation for NIC teaming instructions. For an overview of NIC teaming in Windows 2012 R2, see http://technet.microsoft.com/enus/library/hh831648.aspx. Network adapters cannot be configured as LACP teams in this environment. To use LACP NIC teams, use VLT as covered in Scenario One – VLT Environment.

4.4.4

Commands to verify MSTP For this environment, the MSTP algorithm blocks interfaces appropriately based on the selection of different root bridges for each MST instance. If further adjustments are desired, interface parameters (port cost and port priority) can be adjusted to increase or decrease the probability that the system puts a particular interface in the forwarding state. For more information on modifying port cost and priority, see Appendix F for Dell switches and Nexus documentation for your system. Run the following commands to verify the MST instance-to-VLAN mapping is consistent across all four switches in the topology.

57


Run the following commands on each MXL switch: MXL-A1#show spanning-tree mst configuration Revision: 0 MSTI VID 1 30,40 2 50

Note: Output is identical on MXL-A2. The MXL does not include the default instance (MSTI 0) in the output above. Run the following command to confirm the instance-to-VLAN mapping for MSTI 0: MXL-A1#sh spanning-tree msti 0 brief MSTI 0 VLANs mapped 1-29, 31-39, 41-49, 51-4094 | (output truncated) Note: Output is identical on MXL-A2.

Run the following command on each Nexus switch: N5K-1# show spanning-tree mst configuration Name [] Revision 0 Instances configured 3 Instance Vlans mapped -------- --------------------------------------------------------------------0 1-29,31-39,41-49,51-4094 1 30,40 2 50 -------------------------------------------------------------------------------

Note: Output is identical on N5K-2.

Notes: Consistency among all switches in the topology is important.  The MST instance numbers and mapped VLANs must match on all four switches.  Switch configuration does not include the name value. Therefore, it appears as [] on Nexus and is not included in the output on MXL (unless a name is configured). The default revision number is 0 which appears in the command output. If a name or revision number is configured, the value must match on all four switches.

58


4.4.4.1

Spanning tree commands and output for MST 1 MST Instance 1 VLANs 30 & 40

MST 1 Root Bridge

Po101

Te0/1

Te0/41

E1/1

Te0/42

E1/2

MXL-A1

E1/3

N5K-1

Te0/43

Blade Server 1 VLAN 30 & 40

Fo0/33

Te Fo0/37 0/44

E1/9

Po102

E1/4

Po 127

Po 55

Fo0/33 Fo0/37


Te0/1

E1/7 E1/8

MXL-A2

Te 0/43

E1/7 E1/8 Po103

E1/3 E1/9 E1/4

Te0/44 Te0/41

E1/1

Te0/42

E1/2

N5K-2

Po104

Spanning Tree Blocked Interface

MST instance 1 blocked interfaces In the command output, the Root ID must match the MAC address of the Nexus switch configured as the spanning tree root bridge for MST instance 1. Blocked interfaces will be marked as “BLK” in the "Sts" column and forwarding interfaces will be marked as "FWD". MXL-A1#show spanning-tree msti 1 brief MSTI 1 VLANs mapped 30, 40 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24576, Address 002a.6af7.5bbc Root Bridge hello time 2, max age 20, forward delay 15, max hops 19 Bridge ID Priority 32768, Address 001e.c9f1.04ba Configured hello time 2, max age 20, forward delay 15, max hops 20 Bpdu filter enabled globally Interface Name ---------Po 101 Po 102 Po 127 Te 0/1

PortID -------128.102 128.103 128.128 128.130

Prio ---128 128 128 128

Cost ------1800 1800 600 2000

Sts ----------FWD BLK BLK FWD

Cost ------1800 1800 1800 1800

Designated Bridge ID -------------------24576 002a.6af7.5bbc 28672 002a.6af7.8081 32768 001e.c9f1.0392 32768 001e.c9f1.04ba

Interface

59


PortID -------144.100 144.101 128.128 128.130 Bpdu

Name ---------Po 101 Po 102 Po 127 Te 0/1

Role -----Root Altr Altr Desg

PortID -------128.102 128.103 128.128 128.130

Prio ---128 128 128 128

Cost ------1800 1800 600 2000

Sts --------FWD BLK BLK FWD

Cost ------1800 1800 1800 1800



Filter -----No No No No

Boundary -------No No No No

MXL-A2#show spanning-tree msti 1 brief MSTI 1 VLANs mapped 30, 40 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24576, Address 002a.6af7.5bbc Root Bridge hello time 2, max age 20, forward delay 15, max hops 19 Bridge ID Priority 32768, Address 001e.c9f1.0392 Configured hello time 2, max age 20, forward delay 15, max hops 20 Bpdu filter enabled globally Interface Name ---------Po 103 Po 104 Po 127 Te 0/1

PortID -------128.104 128.105 128.128 128.130


Role -----Root Altr Desg Desg

Prio ---128 128 128 128

Cost ------1800 1800 600 2000

PortID -------128.104 128.105 128.128 128.130

Prio ---128 128 128 128


Cost ------1800 1800 600 2000

Cost ------1800 1800 1800 1800

Sts ---------FWD BLK FWD FWD

Designated Bridge ID -------------------24576 002a.6af7.5bbc 28672 002a.6af7.8081 32768 001e.c9f1.0392 32768 001e.c9f1.0392

Cost ------1800 1800 1800 1800



PortID -------144.102 144.103 128.128 128.130 Bpdu Filter -----No No No No

Boundary -------No No No No

N5K-1# show spanning-tree mst 1 ##### MST1 Bridge Root

vlans mapped: 30,40 address 002a.6af7.5bbc this switch for MST1


60



Cost --------1000 1000 1000 2000

priority

Prio.Nbr -------128.4150 128.4196 128.4198 128.137

24577 (24576 sysid 1)

Type -------------------------------P2p P2p P2p P2p



vlans mapped: 30,40 address 002a.6af7.8081 address 002a.6af7.5bbc port Po55


4.4.4.2



Cost --------1000 1000 1000 2000

priority priority cost Prio.Nbr -------128.4150 128.4197 128.4199 128.137

28673 (28672 sysid 1) 24577 (24576 sysid 1) 1000 rem hops 19

Type -------------------------------P2p P2p P2p P2p

Spanning tree commands and output for MST 2 MST Instance 2 VLAN 50 Po101

Te0/2

Te0/41

E1/1

Te0/42

E1/2

MXL-A1

E1/3

N5K-1

Te0/43


Fo0/33

Te Fo0/37 0/44

Po102

E1/9 E1/4

Po 127 Fo0/33 Fo0/37

Te0/2


MXL-A2

E1/7 E1/8

Po 55 Te 0/43

E1/7 E1/8 Po103

E1/3

E1/9 E1/4

Te0/44 Te0/41

E1/1

Te0/42

E1/2


Po104 MST 2 Root Bridge

MST instance 2 blocked interfaces

In the command output, the Root ID must match the MAC address of the Nexus switch configured as the spanning tree root bridge for MST instance 2. Blocked interfaces will be marked as “BLK” in the "Sts" column and forwarding interfaces will be marked as "FWD".

61


MXL-A1#sh spanning-tree msti 2 brief MSTI 2 VLANs mapped 50 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24576, Address 002a.6af7.8081 Root Bridge hello time 2, max age 20, forward delay 15, max hops 19 Bridge ID Priority 32768, Address 001e.c9f1.04ba Configured hello time 2, max age 20, forward delay 15, max hops 20 Bpdu filter enabled globally Interface Name ---------Po 101 Po 102 Po 127 Te 0/1

PortID -------128.102 128.103 128.128 128.130


Role -----Altr Root Altr Desg

Prio ---128 128 128 128

Cost ------1800 1800 600 2000

PortID -------128.102 128.103 128.128 128.130

Prio ---128 128 128 128

Sts ----------BLK FWD BLK FWD

Cost ------1800 1800 600 2000

Cost ------1800 1800 1800 1800

Designated Bridge ID -------------------28672 002a.6af7.5bbc 24576 002a.6af7.8081 32768 001e.c9f1.0392 32768 001e.c9f1.04ba

PortID -------144.100 144.101 128.128 128.130

Bpdu Sts Cost Link-type Edge Filter Boundary ---------- ------- --------- ---- ------ -------BLK 1800 P2P No No No FWD 1800 P2P No No No BLK 1800 P2P No No No FWD 1800 P2P Yes No No

MXL-A2#show spanning-tree msti 2 brief MSTI 2 VLANs mapped 50 Executing IEEE compatible Spanning Tree Protocol Root ID Priority 24576, Address 002a.6af7.8081 Root Bridge hello time 2, max age 20, forward delay 15, max hops 19 Bridge ID Priority 32768, Address 001e.c9f1.0392 Configured hello time 2, max age 20, forward delay 15, max hops 20 Bpdu filter enabled globally

62


PortID -------128.104 128.105 128.128 128.130


Role -----Altr Root Desg Desg

Prio ---128 128 128 128

Cost ------1800 1800 600 2000

PortID -------128.104 128.105 128.128 128.130

Prio ---128 128 128 128


Cost ------1800 1800 600 2000

Cost ------1800 1800 1800 1800

Designated Bridge ID -------------------28672 002a.6af7.5bbc 24576 002a.6af7.8081 32768 001e.c9f1.0392 32768 001e.c9f1.0392

PortID -------144.102 144.103 128.128 128.130

Bpdu Sts Cost Link-type Edge Filter Boundary ---------- ------- --------- ---- ------ -------BLK 1800 P2P No No No FWD 1800 P2P No No No FWD 1800 P2P No No No FWD 1800 P2P Yes No No



vlans mapped: 50 address 002a.6af7.5bbc address 002a.6af7.8081 port Po55




Cost --------1000 1000 1000 2000

priority priority cost Prio.Nbr -------128.4150 128.4196 128.4198 128.137

28674 (28672 sysid 2) 24578 (24576 sysid 2) 1000 rem hops 19

Type -------------------------------P2p P2p P2p P2p


vlans mapped: 50 address 002a.6af7.8081 this switch for MST2


63



Cost --------1000 1000 1000 2000

priority

Prio.Nbr -------128.4150 128.4197 128.4199 128.137

24578 (24576 sysid 2)

Type -------------------------------P2p P2p P2p P2p


A

References Dell Virtual Link Trunking Reference Architecture 2.0 Dell Networking Configuration Guide for the MXL 10/40GbE Switch I/O Module 9.10(0.0) Dell PowerEdge Configuration Guide for the M I/O Aggregator 9.10(0.0) Cisco NX-OS Software Virtual PortChannel: Fundamental Concepts 5.0 (Cisco Inc.) Cisco Nexus 5600 Series NX-OS Interfaces Configuration Guide, Release 7.x

64


B

Components used in this guide The software versions and licenses used while developing this guide are shown below.

65

Switch

Software Version

Licenses

Dell Networking MXL

Dell Networking OS 9.10(0.0)

Not Applicable

Dell PowerEdge M IOA

Dell Networking OS 9.10(0.0)

Not Applicable

Cisco Nexus 5672UP

Cisco NX-OS 7.0(5)N1(1)

LAN_BASE_SERVICES_PKG cisco 1.0 ENTERPRISE_PKG cisco 1.0

Cisco Nexus 5548

Cisco NX-OS 7.0(8)N1(1)

LAN_BASE_SERVICES_PKG cisco 1.0 ENTERPRISE_PKG cisco 1.0


C

Terminology CMC (Chassis Management Controller): The embedded management interface of the Dell PowerEdge™ M1000e blade server chassis. Among other functions, the CMC provides network and console access to installed IO modules including the Dell Networking MXL and M IOA switches. DHCP (Dynamic Host Configuration Protocol): A network management protocol used to dynamically assign network settings to devices connected to a network. Fault Tolerance: Enables a network to continue operating properly in the event of a failure of some of its components. LACP (Link Aggregation Control Protocol): LACP is used to control the bundling of several physical ports together to form a single logical channel. LACP allows a network device to negotiate an automatic bundling of links by sending LACP packets to the peer (which is a directly connected device that also implements LACP). LAG (Link Aggregation Group): Two or more network links bundled together to function as a single link. LAGs provide increased bandwidth, redundancy and load balancing. MSTP: (Multiple Spanning-Tree Protocol): A modified version of the rapid spanning tree protocol that carries multiple spanning tree instances within its rapid spanning-tree protocol packet. Spanning-tree instance 0 is assigned to the common rapid spanning-tree instance and additional instances above 0 may be configured. For each spanning-tree instance, a root switch is elected and unique active and backup links can be chosen providing potentially unique traffic paths on the network per instance. Each VLAN can then be assigned to a spanning-tree instance allowing active traffic on separate VLANs to potentially utilize separate paths across the network. In order for interconnected switches to participate together in more than the common spanning-tree instance, they must have the same MSTP configuration. MSTP was originally defined in the IEEE 802.1s standard and is included in IEEE 802.1Q standard. NIC Teaming: NIC Teaming allows multiple network interfaces on a computer to be placed into a team to combine bandwidth or provide traffic failover to prevent connectivity loss in the event of a network component failure. Out-of-Band: An out-of-band interface provides management connectivity to a device without participating in or relying on a device’s in-band (normal-use) data interfaces. Port Channel: See LAG. PVST+ (Per-VLAN Spanning-Tree Plus): A vendor-specific implementation of the spanning tree protocol that maintains separate instances for each VLAN, passing that instance’s protocol frames within the VLAN it manages. RSTP (Rapid Spanning-Tree Protocol): A standards-based modified version of the basic spanning tree protocol that allows for much faster convergence times of spanning tree instances and provides for special administratively-assigned port states that improve behavior in certain circumstances. RSTP is defined in IEEE 802.1w.

66


STP (Spanning Tree Protocol): A family of layer-2 management protocols used by Ethernet bridges or switches to establish a loop-free forwarding topology. Protocols include the original STP (IEEE 802.1d), RSTP, MSTP, and PVST+. Ethernet can easily forward traffic in endless loops if cabled or configured incorrectly. The spanning-tree protocols provide a standard way for an interconnected set of switches to only use links that will not cause traffic flows to loop. (Loops will rapidly saturate all available bandwidth). In a spanning tree, a single bridge is elected the root bridge—either by lowest assigned priority or by having the lowest MAC address. Once a root bridge is elected, every other bridge keeps its one link with the lowest path cost to the root bridge active. Links with redundant paths are then blocked by switches that don’t have the lowest path cost for that link. As a result, with spanning tree each non-root bridge effectively has only one active link between it and the root bridge and the topology of these unblocked links draws a tree to the root bridge. Note: VLT terminology is in section 3.3.1 and vPC terminology is in section 3.3.2.

67


D

Resetting the MXL or M IOA to factory defaults To restore the Dell Networking MXL or M IOA to its factory default settings, perform the following steps via a serial console connection: Dell#restore factory-defaults stack-unit 0 clear-all Proceed with factory settings? Confirm [yes/no]:yes The system reloads with factory default settings. When the system is done with the boot process, the following message is displayed: This device is in Bare Metal Provisioning (BMP) mode. BMP is attempting to download an image, configuration file or boot script using DHCP. To continue with the standard manual interactive mode, it is necessary to abort BMP. Press A to abort BMP now. Press C to continue with BMP. Press L to toggle BMP syslog and console messages. Press S to display the BMP status. [A/C/L/S]:A

(“A” is entered at the above prompt since we are not using the BMP feature.) % Warning: The bmp process will stop ... Dell>enable Dell#

The system is now ready for configuration from its default state.

68


E

MXL and M IOA Port Mapping Figure 19 shows the port mapping for the Dell Networking MXLs and M IOAs installed in the M1000e chassis. External port mapping, detailed in the figure below, is dependent on the FlexIO Expansion module(s) installed in each MXL or M IOA. Mapping of server-facing internal ports (numbered 1-32) is detailed in the attachment titled M1000e_Internal_Port_Mapping_1.1.pdf

Internal 10/1 Gb

Fixed QSFP Ports

Expansion Slot 0

Expansion Slot 1

Dell Networking MXL and Dell PowerEdge M I/O Aggregator – Port Mapping

MXL and M IOA port mapping for the M1000e blade enclosure

69


F

Modifying Spanning Tree Port Cost and Priority You can adjust two interface parameters (port cost and port priority) to increase or decrease the probability that a port becomes a forwarding port on Dell Networking switches.  

Port cost — a value that is based on the interface type. The greater the port cost, the less likely the port is selected to be a forwarding port. Port priority — influences the likelihood that a port is selected to be a forwarding port in case that several ports have the same port cost.

The following table lists the default values for port cost by interface. Interface Type 1000-Mb/s Ethernet interface 10-Gigabit Ethernet interface 40-Gigabit Ethernet interface Port Channel with one 10-Gigabit Ethernet interface Port Channel with one 40-Gigabit Ethernet interface Port Channel with two 10-Gigabit Ethernet interfaces Port Channel with two 40-Gigabit Ethernet interfaces

Port Cost Default Value 20000 2000 1400 2000 1400 1800 600

To change the port cost or port priority of an interface, use the following commands. Change the port cost of an interface. In INTERFACE mode:  PVST: spanning-tree pvst vlan vlan-id cost cost-value  MSTP: spanning-tree msti msti-number cost cost-value The cost value range is from 1 to 2000000. Change the port priority of an interface. In INTERFACE mode:  

PVST: spanning-tree pvst vlan vlan-id priority priority-value MSTP: spanning-tree msti msti-number priority priority-value

The priority value is from 0 to 240, in increments of 16. The default is 128. To view the current values for these interface parameters, use the show config command from INTERFACE mode.

70


G

Support and Feedback Contacting Technical Support Support Contact Information

Web: http://Support.Dell.com/ Telephone: USA: 1-800-945-3355

Feedback for this document We encourage readers to provide feedback on the quality and usefulness of this deployment guide by sending an email to [email protected]

About Dell EMC Dell EMC is a worldwide leader in data center and campus solutions, which includes the manufacturing and distribution of servers, network switches, storage devices, personal computers, and related hardware and software. For more information on these and other products, please visit the Dell EMC website at http://www.dell.com.

71
