Cisco 3945, EtherSwitch, and UCSE

We’ve been working on network upgrades to our aging branch networking equipment and ran into some roadblocks when dealing with UCS-E series servers and EtherSwitch modules in a Cisco 3945E router. UCS-E is a fantastic piece of technology whereby an Intel-based x86 server can be slotted into a Cisco router, and EtherSwitch modules are full fledged switches (with optional PoE and Layer 3) that do the same. These types of modules make for an extremely dense and robust branch office environment.

The goal I set for the configuration of this equipment is to have the server’s CIMC and VMWare IP addresses in a SERVER vlan, which was also trunked to the two EtherSwitch modules.

The Problem

The internal ucse interfaces operate as routed ports, and the GigabitEthernet interfaces for the EtherSwitch modules operate as switchports. This makes things slightly more tricky than things would be otherwise as we’ve got to somehow bridge them into the same broadcast domain.

The Solution

By bridging the ucse interface and vlan interface together, we can place the UCS-E server into the same broadcast domain as the dot1q trunk we send to the EtherSwitch. In this guide, we will work towards this goal by creating a bridged virtual interface that connects the two together.

Configure VLAN and BVI

The first step is to configure the VLAN and BVI interfaces. A bridged virtual interface can be used to join two routed interfaces together:

ROUTER#conf t
ROUTER(config)#vlan 8
ROUTER(config-vlan)#name SERVER
ROUTER(config-vlan)#exit
ROUTER(config)#bridge irb
ROUTER(config)#int bvi 8
ROUTER(config-if)#description VLAN 8 ROUTED
ROUTER(config-if)#ip address 10.0.8.1 255.255.255.0
ROUTER(config-if)#exit
ROUTER(config)#bridge 8 protocol ieee
ROUTER(config)#bridge 8 route ip
ROUTER(config)#int vlan 8
ROUTER(config-if)#description VLAN 8 TRUNK
ROUTER(config-if)#bridge-group 8

The above configuration creates vlan 8, an interface vlan 8 (or Switched Virtual Interface as they are commonly called), and a bridge group called bvi 8. The brige-group command is used to join an interface to a BVI. The bridge n protocol ieee and bridge n route ip tells the Cisco router that this bridge is for IP traffic. The bridge irb command enabled Integrated Routing & Bridging. IRB must be configured before trying to create a BVI.

Configure UCS-E

The UCS-E server has two internal ports, but we’re only focusing on one in this post: the console port. This is the first interface listed in the slot where your UCS-E server is installed. Mine happens to be in slot 4:

ROUTER#conf t
ROUTER(config)#int ucse 4/0
ROUTER(config-if)#ip unnumbered BVI 8
ROUTER(config-if)#bridge-group 8
ROUTER(config-if)#imc ip address 10.0.8.2 255.255.255.0 default-gateway 10.0.8.1
ROUTER(config-if)#imc access-port shared-lom console

In the above configuration, we assign the router’s interface to the bridge-group 8, and give it the same IP address the BVI 8 interface has. This effectively places it in VLAN 8 because of our earlier configuration. We assign the server’s CIMC address, and tell it to use the built-in console port for both CIMC and server traffic.

Configure EtherSwitch

The EtherSwitch modules must have an IP address on the internal management interface. This interface is the first one listed on the router for the slot the EtherSwitch has been installed in. We will be using loopback 0 and the ip unnumbered command to make this happen as there is no need to do anything extra to get these to work. You may want to do something differently if the ability to SSH to each individual EtherSwitch is important to you, but here we’ll be connecting to the router and using the service-module command for the rest.

ROUTER#conf t
ROUTER(config)#int loopback 0
ROUTER(config-if)#ip address 10.0.0.1 255.255.255.255
ROUTER(config-if)#no shutdown

As you can see we’ve just created a loopback 0 and assigned it an IP. Configuring the EtherSwitch’s management interface is equally simplistic. I happened to have two of these, so the same thing was done for both (the second one was on gi 2/0):

ROUTER#conf t
ROUTER(config)#int gi 1/0
ROUTER(config-if)#ip unnumbered loopback 0
ROUTER(config-if)#no shutdown

Trunking the VLANs

Trunking is done exactly the same way it would be done on a traditional Catalyst switch: switchport mode trunk. Again, I did the same thing to both EtherSwitch modules’ second interface:

ROUTER#conf t
ROUTER(config)#int gi 1/1
ROUTER(config-if)#switchport mode trunk
ROUTER(config-if)#no shutdown

Now we’ll move on to the switches themselves. Again, the configuration is simple, but you’ll need to consult the Cisco documentation on which interface is the management and which is data. Ours happened to be gi 0/25, and I would guess that it would be gi 0/49 on a 48 port switch module:

ROUTER#service-module gi 1/0 session
Trying 10.0.0.1, 2066 ... Open
<enter>
<enter>
SWITCH>en
SWITCH#conf t
SWITCH(config)#vlan 8
SWITCH(config-vlan)#name SERVER
SWITCH(config-vlan)#exit
SWITCH(config)#int gi 0/25
SWITCH(config-if)#switchport trunk encap dot1q
SWITCH(config-if)#switchport mode trunk
SWITCH(config-if)#exit
SWITCH(config)#int range gi 0/1 - 24
SWITCH(config-if-range)#switchport mode access
SWITCH(config-if-range)#switchport access vlan 8
<Ctrl-Shift-6, x>
ROUTER#disconnect 
Closing connection to 10.2.0.1 [confirm]
<enter>
ROUTER#

In the above config, we enter into the EtherSwitch, setup vlan 8, configure the gi 0/25 interface to be a dot1q trunk, and then untag every interface to vlan 8. Note that Ctrl-Shift-6, x is what’s used to exit the switches’ console. Typing exit does exactly what it would if you were connected via a serial cable.

Conclusion

The bvi 8 interface is the thing doing all of the routing in this instance, which may seem odd to those who aren’t used to bridging in the Cisco world. This guide can also serve as a solid example of how to configure EtherSwitches- there isn’t a whole lot of information floating out there for that either.

Something to note here is that one probably shouldn’t just trunk a single VLAN to each EtherSwitch. Creating new vlans is just as simple as it is on Catalyst switches: vlan x and int vlan x.

I hope this article has helped you, and if you have any questions, please feel free to ask them down below!