VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus

VMware 10GE QoS Design Deep Dive with Cisco UCS & Nexus 10GE Networking Brad Hedlund, CCIE #5530 Data Center Architect Cisco Systems [email protected] BRAD HEDLUND .com

v 2.5

- About the Author Brad Hedlund Technical Solutions Architect, Data Center Cisco Systems, Inc. CCIE #5530 http://bradhedlund.com/about/

Blog: http://bradhedlund.com Twitter: http://twitter.com/bradhedlund E-mail: [email protected]

Comments welcome.

BRAD HEDLUND .com

Cisco UCS Networking Overview SAN A

ETH 1

ETH 2

SAN B

MGMT

MGMT

6100 Fabric A

6100 Fabric B

F E X

Chassis 1

F E X

F E X

Chassis 2

F E X

A

CNA B200

B

A

CNA CNA B250

B

BRAD HEDLUND .com

Emulex/Qlogic CNA FEX

FEX

Fabric B

Fabric A

10GE

10GE

CNA

vHBA

vNIC

vNIC

vHBA

vhba0

vmnic0

vmnic1

vhba1

vSwitch / Nexus 1000V BRAD HEDLUND .com

Emulex/Qlogic CNA – Logical View Fabric Interconnect B

Fabric Interconnect A vfc 1

vEth 1

vEth 2

FEX

vfc 2

FEX

CNA

vhba0

vmnic0

vmnic1


vhba1

Service Profile / Fabric Selection

BRAD HEDLUND .com

Cisco VIC “Palo” FEX

FEX

Fabric B

Fabric A

10GE

10GE

Cisco VIC

vHBA

vNIC

vNIC

vNIC

vNIC

vNIC

vNIC

vNIC

vNIC

vHBA

vhba 0

vmnic 0

vmnic 1

vmnic 2

vmnic 3

vmnic 4

vmnic 5

vmnic 6

vmnic 7

vhba 1

vSwitch / vDS / Nexus 1000V BRAD HEDLUND .com

58

Cisco VIC QoS: Example FEX

FEX

Fabric B

Fabric A

Cisco VIC 10GE

Strict Priority

7

10GE

TX

TX

Bandwidth Scheduler

VIC Mgmt

10%

0

40%

2

1

3

10%

6

0

1

2

3

10%

4

VIC Mgmt

40%

5

6

(8) COS Queues

No Drop

RR

Limit TX 1GE

vNIC 1

40%

10%

No Drop

RR

7

Bandwidth Scheduler

40%

5

4

Strict Priority

Limit TX 1GE

vNIC 2

COS 1

vHBA 1

vNIC 3

vNIC 4

COS 3

COS 4

COS 5

vNIC 5

vNIC 6

COS 1

BRAD HEDLUND .com

vHBA 2

vNIC 7

vNIC 8

COS 3

COS 4

COS 5

58

Cisco VIC – Logical View Fabric Interconnect B


vEth 1

vEth 2

vEth 3

vEth 4

vEth 5

FEX A

vEth 6

vEth 7

vEth 8

FEX B Cisco VIC

vhba 0

vmnic 0

vmnic 1

vmnic 2

vmnic 3

vmnic 4

vmnic 5


vmnic 6

vmnic 7

vhba 1

vfc 2

I/O Consolidation with 10GE VM Data

vmnic 0

VMotion

vmnic 1

CONCERN: Bandwidth Availability 10GE

MGMT

IP Storage

vmnic 2

vmnic 3

BRAD HEDLUND .com

SOLUTIONS: 1. MAX bw per NIC Rate Limits 2. MIN bw per NIC QoS 3. Do Nothing Let it rip

Network Switch

I/O Consolidation with 10GE

CONCERN: Bandwidth Availability

VM Data VMotion MGMT IP Storage

vmnic 1

SOLUTIONS: 1. MAX bw per Class Rate Limits 2. MIN bw per Class QoS 10GE 3. Do Nothing Let it rip

BRAD HEDLUND .com

Network Switch

MAX Bandwidth – Rate Limits 10GE VM Data

vmnic 0

2.5G

VMotion

vmnic 1

2.5G

vmnic 2

2.5G

vmnic 3

2.5G

MGMT

IP Storage

Network Switch

 Sum of Limits must not exceed physical link speed  Inefficient use of unused BW This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/ BRAD HEDLUND .com

MIN Bandwidth – QoS 10GE VM Data

vmnic 0

VMotion

vmnic 1

Network Switch

vmnic 2

MGMT

IP Storage

vmnic 3

 Minimum BW guarantee with no Maximum  Efficient: unused BW available to other NICs This slide animated @ http://bradhedlund.com/2010/08/16/cisco-ucs-qos-vs-hp-virtual-connect-rate-limiting/ BRAD HEDLUND .com

MIN Bandwidth – QoS Nexus 1000V Classifies Traffic

Cisco VIC Enforces Policy 10GE

UCS 6100 Nexus 5000 Enforces Policy

VM Data

VMotion Network Switch

vmnic 1

MGMT

IP Storage

BRAD HEDLUND .com

Traffic Considerations Required • Guest VM Data • Management – VMKernel – Service Console

• vMotion • Central Storage Access

Optional • Fault Tolerance (FT) • NFS • iSCSI • FCoE • Realtime – Video/Voice, Market Data

BRAD HEDLUND .com

Cisco UCS + VIC Design (1): Required Traffic Fabric Interconnect B


vEth 2

vEth 1

vhba 0

vmnic 0

vEth 6

vEth 3

vmnic 2

vmnic 4

vmnic 1

vmnic 3

vEth 7

vEth 8

vmnic 5

vfc 2

vhba 1

vSwitch or vDS

MGMT

VM DATA

vMotion

FC

Primary

vmnic1

vmnic2

vmnic5

MP

Backup

vmnic0

vmnic3

vmnic4

MP

Min BW

10% 1GE Max

40%

10%

40%

Class

Bronze COS 1

Best Effort COS 0 BRAD HEDLUND .com

Silver COS 2

COS 3

UCS Manager QoS System Class Design #1

• Assign each Class of Service a minimum bandwidth weighting • System wide policy

BRAD HEDLUND .com

Apply QoS Policy to vNIC • Create QoS Policy • Assign Policy to vNIC

BRAD HEDLUND .com

Cisco UCS + VIC Design (2): Req + 1 Opt Fabric Interconnect B


vEth 1

vhba 0

vEth 2

vmnic 0

vEth 3

vmnic 2

vEth 4

vmnic 4

vEth 5

vmnic 6

vmnic 1

vEth 6

vmnic 3

vEth 7

vmnic 5

vmnic 7

vEth 8

vfc 2

vhba 1

vSwitch or vDS

MGMT

VM DATA

vMotion

NFS

FC

Primary

vmnic1

vmnic2

vmnic5

vmnic7

MP

Backup

vmnic0

vmnic3

vmnic4

vmnic6

MP

Min BW

10% 1GE Max 30%

10%

10%

40%

Class

Bronze COS 1

Silver COS 2

Gold COS 4

COS 3

Best Effort COS 0 BRAD HEDLUND .com


• Assign each Class of Service a minimum bandwidth weighting • System wide policy BRAD HEDLUND .com

Cisco UCS + VIC Design (3): Req + 2 Opt + Realtime Fabric Interconnect B


vhba 0

vEth 1

vmnic 0

vEth 2

vEth 3

vmnic 2

vmnic 4

vEth 4

vEth 5

vmnic 6

vmnic 8

vEth 6

vEth 7

vEth 8

vmnic 1

vmnic 3

vmnic 5

vEth 9

vmnic 7

vEth 10

vmnic 9

vfc 2

vhba 1

vSwitch or vDS

MGMT

vMotion

Realtime

FT

VM DATA

NFS

FC

Primary

vmnic0

vmnic1

vmnic2

vmnic5

vmnic6

vmnic9

MP

Backup

vmnic1

vmnic0

vmnic3

vmnic4

vmnic7

vmnic8

MP

Min BW

1GE Max

10%

10%

20%

10%

10%

40%

Class

Bronze COS 1

Bronze COS 1

Plat COS 5 Gold COS 4 BRAD HEDLUND .com

Best Effort COS 0

Silver COS 2

COS 3


• “Platinum” Class for Realtime traffic • Lossless Ethernet and Multicast optimized • “Silver” and “Bronze” for vMotion & NFS with Jumbo Frames BRAD HEDLUND .com

Design #4: (2) 10GE NICs with Nexus 1000V Cisco UCS or Nexus 5000 Fabric Interconnect A / Nexus 5000 #1 vfc 1

Fabric Interconnect B / Nexus 5000 #2 vfc 2

vEth 7

vEth 4

QoS Enforcement

Cisco VIC or Other adapter Cisco UCS server or non-Cisco vhba 0

vmnic 0

QoS Enforcement (if Cisco VIC)

vmnic 1

QoS Enforcement (if N1K v1.4)

NEXUS 1000V

MGMT

vMotion

QoS Classification

Realtime

FT

VM DATA

NFS

vhba 1

FC

Primary

vmnic0

vmnic1

vmnic0

vmnic1

vmnic0

vmnic1

MP

Backup

vmnic1

vmnic0

vmnic1

vmnic0

vmnic1

vmnic0

MP

Min BW

1GE Max

10%

10%

20%

10%

10%

40%

Class

Bronze COS 1

Bronze COS 1


Best Effort COS 0

Silver COS 2

COS 3

vNIC QoS Policy for Design #4

• Preserve and use COS markings from Nexus 1000V • Same QoS System Class settings as Design #3

BRAD HEDLUND .com

Design #5: (2) 10GE NICs, no Cisco VIC, no N1K Nexus 5000 Nexus 5000 #2

Nexus 5000 #1 vfc 1

Eth 1/1

Eth 1/1

QoS Enforcement

vfc 2

QoS Classification

Cisco VIC or Other adapter Cisco C-Series or non-Cisco vhba 0

vmnic 0

vmnic 1

VMware NetIOC ? FCoE?

vSwitch or vDS

MGMT

vMotion

Realtime

FT

VM DATA

vhba 1

NFS

FC

Primary

vmnic0

vmnic1

vmnic0

vmnic1

vmnic0

vmnic1

MP

Backup

vmnic1

vmnic0

vmnic1

vmnic0

vmnic1

vmnic0

MP

Min BW

1GE Max

10%

10%

20%

10%

10%

40%

Class

Bronze COS 1

Bronze COS 1


Best Effort COS 0

Silver COS 2

COS 3

Narrative posted at: http://bradhedlund.com/?p=1650

BRAD HEDLUND .com