CISCO

Cisco UCS, Nexus, NetApp – FCoE End to End Config

I am sick of cables.  One of the most recent vSphere hosts I deployed had ELEVEN Gigabit Ethernet cables attached to it…and I actually have one that has Fourteen!  So, one of my goals when looking at deploying UCS was a setup that utilized 10gb Ethernet and FCoE everywhere. 

Well, it’s almost end to end.  It’s actually not possible to do FCoE end to end today, and you might read about the need for “FCoE Multi Hop”.  FCoE is as it is described, an implementation of Fibre Channel over Ethernet.  How it works is you basically establish an Ethernet infrastructure, and the FC (which is really just standard FC) runs over “Virtual FC” ports.  In order for this to be done completely end to end you need N_port Virtualization support on all devices in the chain.  This support exists today on the latest code for the Nexus switches, but not on the code for the UCS Fabric Interconnects (although I am told it is coming soon).  My temporary solution was to order my Fabric Interconnects with the expansion module that has Four 10gb Ethernet ports and Four 4gb Fibre Channel ports.

There are several models you can plug into the expansion slot on the FI’s but I chose this one for the following reasons:

·  It has more than enough FC ports for my immediate needs, and the extra 10gb ports mean it will continue to be useful when the FI software is updated to allow NPV.

·  Ports on addon modules include “Free Port Licenses”, so I will be able to add another couple of UCS Chassis down the road without paying the ridiculous port licence costs (the one thing about UCS I don’t like).

For now, this is what I’m describing.  Blue lines are 10gb Ethernet, Green are traditional Fibre Channel:

Hardware Breakdown

The config described below consists of the following hardware:

·  1 UCS Blade Chassis (Cisco 5108)

·  2 UCS Fabric Interconnects (Cisco 6120XP w/ 4x10GB & 4x4GB FC Expansion Module)

·  2 Cisco Nexus 5548UP Switches (with Layer3 Daughtercard)

·  1 NetApp 3210 Filer (with Dual Controllers)

What about VPC on the switches?

VPC is a great thing.  It lets you create a port-channel between a device and two separate switches, but from the device’s point of view the connections are going to the same device.  Lose a switch, nothing goes down.  We have the Nexus Switches connected in a VPC Peer Link (not shown in the diagram above).  The only challenge here is that a properly built Fibre Channel design calls for two completely separate fabrics; this is taken into account in the configs below.

Fabric Interconnect <-> Nexus (Ethernet) Config:

I have two 10gb links between each Fabric Interconnect and it’s corresponding Nexus Switch (no cross connections).  These are port-channelled in the traditional way to keep service in the event of port, transceiver, or cable loss.
On the UCS Side it’s easy to establish the port channel.  For each Fabric (A and B) just right click on Port Channels under the LAN tab and select Create Port Channel.  Give it a name and select your uplink ports.  When done, the UCS interface will look similar to the below:

On the Nexus side, the following configs are in place on both switches:

Pretty simple stuff.  When done, do a “show port-channel summary” command from both switches and you should see them up as so:
201   Po201(SU)   Eth      LACP      Eth1/17(P)   Eth1/18(P)

Fabric Interconnect <-> Nexus (Fibre Channel) Config:

On the UCS Side, you don’t actually have to do anything for the actual FC Uplinks.  UCS considers any traditional FC port an FC Uplink.  What you DO have to do is define your VSAN’s for those uplinks.  Although much of our Ethernet infrastructure is designed to allow packets to flow in multiple paths, with FC the mentality is you define 2 exact paths and if there is a failure somewhere along one of them, the whole path is just down (and therefore packets should be transmitted along the other path).  We do this in our FCoE setup by defining a different VSAN for Fabric A and Fabric B.  Since in this example the SAN is a NetApp 3210, I used VSAN3210 for Fabric A and VSAN3211 for Fabric B.
In your UCS Console, under the SAN Tab, open up SAN Cloud, then Fabric A.  Right click on VSANs and define VSAN3210 as shown:

Notice you have to define an FCoE VLAN as well?  We’ll get to that in a bit.  You can make this a different number if you want but it’s much easier to work with if you keep the numbers the same.  Make very sure that Default Zoning is set to Disabled, and that the fabric is set to Fabric A.
Now repeat this for Fabric B; using the number 3211 instead of 3210, and making sure to select the Fabric B radio button.