We’ve been working on network upgrades to our aging branch networking equipment and ran into some roadblocks when dealing with UCS-E series servers and EtherSwitch modules in a Cisco 3945E router. UCS-E is a fantastic piece of technology whereby an Intel-based x86 server can be slotted into a Cisco router, and EtherSwitch modules are full fledged switches (with optional PoE and Layer 3) that do the same. These types of modules make for an extremely dense and robust branch office environment.
The goal I set for the configuration of this equipment is to have the server’s CIMC and VMWare IP addresses in a
SERVER vlan, which was also trunked to the two EtherSwitch modules.
ucse interfaces operate as routed ports, and the
GigabitEthernet interfaces for the EtherSwitch modules operate as switchports. This makes things slightly more tricky than things would be otherwise as we’ve got to somehow bridge them into the same broadcast domain.
By bridging the
ucse interface and
vlan interface together, we can place the UCS-E server into the same broadcast domain as the dot1q trunk we send to the EtherSwitch. In this guide, we will work towards this goal by creating a bridged virtual interface that connects the two together.
Configure VLAN and BVI
The first step is to configure the VLAN and BVI interfaces. A bridged virtual interface can be used to join two routed interfaces together:
ROUTER#conf t ROUTER(config)#vlan 8 ROUTER(config-vlan)#name SERVER ROUTER(config-vlan)#exit ROUTER(config)#bridge irb ROUTER(config)#int bvi 8 ROUTER(config-if)#description VLAN 8 ROUTED ROUTER(config-if)#ip address 10.0.8.1 255.255.255.0 ROUTER(config-if)#exit ROUTER(config)#bridge 8 protocol ieee ROUTER(config)#bridge 8 route ip ROUTER(config)#int vlan 8 ROUTER(config-if)#description VLAN 8 TRUNK ROUTER(config-if)#bridge-group 8
The above configuration creates
vlan 8, an
interface vlan 8 (or Switched Virtual Interface as they are commonly called), and a bridge group called
bvi 8. The
brige-group command is used to join an interface to a BVI. The
bridge n protocol ieee and
bridge n route ip tells the Cisco router that this bridge is for IP traffic. The
bridge irb command enabled Integrated Routing & Bridging. IRB must be configured before trying to create a BVI.
The UCS-E server has two internal ports, but we’re only focusing on one in this post: the
console port. This is the first interface listed in the slot where your UCS-E server is installed. Mine happens to be in slot 4:
ROUTER#conf t ROUTER(config)#int ucse 4/0 ROUTER(config-if)#ip unnumbered BVI 8 ROUTER(config-if)#bridge-group 8 ROUTER(config-if)#imc ip address 10.0.8.2 255.255.255.0 default-gateway 10.0.8.1 ROUTER(config-if)#imc access-port shared-lom console
In the above configuration, we assign the router’s interface to the
bridge-group 8, and give it the same IP address the
BVI 8 interface has. This effectively places it in
VLAN 8 because of our earlier configuration. We assign the server’s CIMC address, and tell it to use the built-in
console port for both CIMC and server traffic.
The EtherSwitch modules must have an IP address on the internal management interface. This interface is the first one listed on the router for the slot the EtherSwitch has been installed in. We will be using
loopback 0 and the
ip unnumbered command to make this happen as there is no need to do anything extra to get these to work. You may want to do something differently if the ability to SSH to each individual EtherSwitch is important to you, but here we’ll be connecting to the router and using the
service-module command for the rest.
ROUTER#conf t ROUTER(config)#int loopback 0 ROUTER(config-if)#ip address 10.0.0.1 255.255.255.255 ROUTER(config-if)#no shutdown
As you can see we’ve just created a
loopback 0 and assigned it an IP. Configuring the EtherSwitch’s management interface is equally simplistic. I happened to have two of these, so the same thing was done for both (the second one was on
ROUTER#conf t ROUTER(config)#int gi 1/0 ROUTER(config-if)#ip unnumbered loopback 0 ROUTER(config-if)#no shutdown
Trunking the VLANs
Trunking is done exactly the same way it would be done on a traditional Catalyst switch:
switchport mode trunk. Again, I did the same thing to both EtherSwitch modules’ second interface:
ROUTER#conf t ROUTER(config)#int gi 1/1 ROUTER(config-if)#switchport mode trunk ROUTER(config-if)#no shutdown
Now we’ll move on to the switches themselves. Again, the configuration is simple, but you’ll need to consult the Cisco documentation on which interface is the management and which is data. Ours happened to be
gi 0/25, and I would guess that it would be
gi 0/49 on a 48 port switch module:
ROUTER#service-module gi 1/0 session Trying 10.0.0.1, 2066 ... Open <enter> <enter> SWITCH>en SWITCH#conf t SWITCH(config)#vlan 8 SWITCH(config-vlan)#name SERVER SWITCH(config-vlan)#exit SWITCH(config)#int gi 0/25 SWITCH(config-if)#switchport trunk encap dot1q SWITCH(config-if)#switchport mode trunk SWITCH(config-if)#exit SWITCH(config)#int range gi 0/1 - 24 SWITCH(config-if-range)#switchport mode access SWITCH(config-if-range)#switchport access vlan 8 <Ctrl-Shift-6, x> ROUTER#disconnect Closing connection to 10.2.0.1 [confirm] <enter> ROUTER#
In the above config, we enter into the EtherSwitch, setup
vlan 8, configure the
gi 0/25 interface to be a dot1q trunk, and then untag every interface to
vlan 8. Note that
Ctrl-Shift-6, x is what’s used to exit the switches’ console. Typing
exit does exactly what it would if you were connected via a serial cable.
bvi 8 interface is the thing doing all of the routing in this instance, which may seem odd to those who aren’t used to bridging in the Cisco world. This guide can also serve as a solid example of how to configure EtherSwitches- there isn’t a whole lot of information floating out there for that either.
Something to note here is that one probably shouldn’t just trunk a single VLAN to each EtherSwitch. Creating new vlans is just as simple as it is on Catalyst switches:
vlan x and
int vlan x.
I hope this article has helped you, and if you have any questions, please feel free to ask them down below!