The Cisco install guide for the Nexus 1000v is a great reference, and I’ll help clarify and fill in the gaps the best I can. I won’t regurgitate too much of the steps in the install guide, so you will want to use the install guide for sure.

First let’s talk about the prerequisites ::

vCenter Server — In most cases, we have a data center of some sort with multiple ESX hosts running. Each ESX host has its own Virtual Machines running. So within our data center, it’s a good idea to define a cluster –or grouping of ESX hosts. It’s beneficial to group our ESX servers in a cluster because we can take advantage of some of the bells and whistles with VMware such as vMotion and high availability. vMotion is cool because if we need to take a server down for maintenance, we can drag and drop the VM to another ESX host. vMotion can also be a dynamic process if there is a unforeseen ESX server failure. So we define our data center, put our ESX servers in one or more clusters and have our VM’s running on each ESX server. The hierarchical structure is one of the benefits of vCenter because we now can centrally manage all of our ESX hosts from a single view. Instead of using the vSphere client to connect to each ESX server separately, we connect our vSphere client to the vCenter server which allows us to manage a whole data center of ESX hosts and their associated VM’s. Most importantly, we must be running vCenter server because that’s what the Nexus VSM communicates with directly.

Here’s and example screenshot ::

tromer-xpro = Server name hosting vCenter server
Data Center name = “Lab”
Cluster name = “POC”
ESX host = 192.168.1.105 (only have a single ESX hypervisor/server running in this example)
VM’s running on ESX host = Nexus VSM, and Nostalgia

You can use the following to run vCenter server :: XP pro sp2, Win 2k3 server, Win 2k8 server.

Go here to get vCenter Server as well as the ESXi 4 Hypervisor files. You will need to sign up with an email account not hosted by a free site like yahoo, etc. You can try it free for 60 days. The file name for vCenter Server should look something like this, “VMware-VIMSetup-all-4.0.0-162902.iso”. When you run it, you will get this screen ::

On the box you designate as the vCenter server, install “vCenter Server” and the ‘vCenter Update Manager”. I’ll explain the vCenter Update Manager later, but it makes your life a lot easier. You can also install the “vSphere Client” to whatever box you choose. This client is what you run to connect to either a single ESX host, or better yet, the vCenter Server. For the most part, you will be using it to connect to the vCenter server which is how I got the vCenter screenshot above.

ESXi 4 host– When you go to download vCenter, you should be able to download the ESX hypervisor as well. The file name is something like this, “VMware-VMvisor-Installer-4.0.0-171294.x86_64.iso”. The hypervisor is what gives you the ESX host for which you run your VM’s on. Ideally, you need at least 2, but you can start with 1 hypervisor if you are limited on hardware or resources to run a second. You will want a second to test moving a VM between ESX hosts (vMotion). You can connect to your ESX host IP address using the vSphere client, but honestly, I would do everything from the vCenter server since it manages all ESX hosts in your data center.

Nexus VSM– If you remember from my previous post, the VSM is essentially the Supervisor for the virtual switch chassis. It can exist as a virtual machine running on a ESX host, or a separate physical appliance. You can download it from Cisco here. In the screenshot above, you can see that I installed it on the one ESX hypervisor I have running so far. For demo purposes, installing the VSM as a VM is perfectly fine (this is valid for production too). You need a Cisco login to download this file. Now it probably makes sense to install the VSM on its own ESX host outside the cluster, but you don’t necessarily have to.

So following along the Cisco Install guide, you should be able to get the VSM up and running. Let’s stop and talk about installing and configuring the VSM. The install guide mentions having 3 virtual adapters running on your VSM. Think of it as a PC running with 3 network cards connecting it to the network.

So we define 3 separate port groups on the vswitch that’s connected to our VSM. You go to network configuration settings for the ESX host and define the port group and assign it to a VLAN.

See below ::

Once you have them defined, you can assign them to each of the 3 Network adapters you created for the VSM. Think of assigning them as a method of assigning a port profile or configuration to an interface. In this case, it’s an access port connecting to our VSM or VM running VSM software. The order in which you assign each port group (profile) to each adapter is important, so be sure to read the install guide carefully.

This is where you assign the port group or profile to each adapter ::

So at a high level, you are essentially doing this ::

Again, to make your brain not hurt as much, just think of the VSM as any old server with 3 network cards connecting to a switch with each on its own VLAN. Why separate them out into VLANs? Well, each VLAN or port group/interface has a purpose. The Mgmt Interface along with the VLAN it’s associated with is how you are going to telnet or SSH to the VSM. We telnet to it so we can configure the switch, just like we are used to doing. The Mgmt interface is also important because this is the interface the VSM will use to talk to the vCenter server. On the VSM or supervisor module, we can define things such as a data VLAN for all my Web servers. I define them using the CLI (more on that later) and once they are defined, they get pushed to the vCenter server. This is why the vCenter is so important, because the VSM is constantly connected to it. So we need to make sure the Mgmt interface has connectivity to the vCenter Server. To make your life easier, you can put the vCenter Server on the same VLAN you defined for the Mgmt interface on the VSM (I chose VLAN 1).

The “packet” and “control” VLANs are used between the VSM and the VEMs. Basically, the supervisor talks to each VEM over two channels. Some things it only does over the control link, while other things are done only over the packet channel. The VSM uses the packet vlan for such layer 2 protocols as CDP, LACP and IGMP. The control VLAN is basically what keeps the VEM and the VSM on the same page. If a new host is connected to the VEM or line card, then it will update the VSM with link status (“Hey, I have a new virtual interface online”). Netflow information from the VEM is also passed up to the VSM via the control link.

Make sure the switch port connected to your ESX host–running the VSM– is configured as a trunk port allowing whatever 3 VLANs you decide on. Technically, I don’t think you have to put them in separate VLANs, but let’s do it because it’s recommended by Cisco and it really makes things logically separate.

So something like this ::

So all of these VLANs you create get referenced again when you finally have the VSM up and running and you start it. At this point, you can run through the setup script on the Nexus 1000 VSM. It’s just like running “setup” on a Cisco IOS router–a bunch of prompts asking you for input. The install guide outlines this well, so you should be able to follow along with that.

Similar to a 6500 chassis, you can configure multiple VSM’s or Supervisor engines in an active/standby role. In my opinion, all you need is the standalone to start doing your testing but know that you can do it.

At this point, let’s do a sanity check to see what we have achieved so far.

-We should have a vCenter server up and running.
-We should have at least one ESX4 server up and running.
-We may also have the VSM running on the same ESX4 server.

Now if you have plenty of machines, then you probably have two or three ESX hosts running and one of them has the VSM running as a virtual machine. At this point, you should have pointed your vSphere client to your vCenter IP address and defined your Data Center, Cluster and added all your ESX hosts. You have installed the VSM and ran through the basic setup. You may have gotten far in the guide and connected the VSM to the vCenter server already. Just remember what VLANs you defined earlier because you’ll have to reference them.

So your setup may look like this ::

In the example above, we have the VSM running on one ESX box and another ESX box ready for a VM. In my screenshot earlier, you may have noticed I used a VM called “nostalgia”, this is a free VM you can download from the VM marketplace. It’s nice because it needs very little resources and can be a good candidate in the future for testing vMotion. So it’s just a easy way to get a VM up and running fast…

Notice that I have NO VEMs running in any of the ESX hosts yet. Both ESX hosts in the diagram still have the VMware vSwithes. You will eventually have to install the VEM software on each ESX host, and then basically add the Nexus VEM or interface line card to each ESX of your choosing. If you installed the “update manager” on the vCenter server, it should automatically download the VEM from VMware and install on the ESX host. If it’s installed, then that means you will be able to enable it on the ESX box. So right now when I connect to the VSM and do a “show mod” I don’t see any line cards or VEMs yet. You won’t see any until we associate a VEM with an ESX host.

Stick around because there’s a lot more to talk about. I will discuss more about the VEM side of things because this is really where I ran into problems and confusion.

Here’s something to leave you with. Were you wondering by chance if the Nexus VEM has to in fact totally replace the VMware Vswitch? Can both switches be on the same ESX box? How about this, “Can I have a VSM running on the same ESX host where I have a VEM? Just remember one thing, the VSM will always be connected to a VMware vSwitch (you can’t put the cart before the horse .

So if your like me, I decided to start with one ESX host and try to get the VSM running as well as another VM (Nostalgia). I did this all on one ESX box because I was short on RAM (more on the way!). Honestly, I had some trouble understanding this until I put it on paper. This can be a real face-melter until you actually work it out on paper…

Here’s a little tease for you of what a working setup can look like with a VSM and one VEM running on a single ESX host.

Yes, the VEM has two uplinks to the network. A little weird but I will discuss it in more detail in the near future. It’s part of that whole “keeping the channels separate” between the VSM and the VEM. Basically, the VMware data traffic is isolated from the control and packet traffic. Remember, the communication between VSM and VEM is crucial and should not be impacted by other traffic (data), so it’s a way of separating the two. More on these uplinks later…

Ted Romer
CCIE No. 21785

http://www.cisco.com/go/1000v

I decided I would give it a go and share my experiences and thoughts. Now I’m by no means an ESX expert, but I have recently started playing more and more with it. I have always been partial to regular VMware Workstation myself.

Background

I’m not going to go into too much detail here, but I wanted to give some background to add some context to the Nexus 1000v.

VMware has come out with this new Distributed Virtual Switching (DVS) term and Cisco has basically latched on with the introduction of the 1000v (as can other 3rd parties). Historically, we are used to using the integrated VMware virtual switch (vswitch) within the hypervisor in our ESX servers. You can basically think of the Nexus 1000v as a way of doing-away with the VMware vswitch and putting a Cisco virtual switch in its place.

So why replace a VMware switch with a Cisco switch? Let’s think about where we typically draw the imaginary line in ownership between our Server and Network teams. If I have an ESX hypervisor hosting 5 different virtual machines, they get connected internally with the vmware soft switch (vswitch). As a server admin, I have to create my port groups (or profiles) and assign them manually to each VM. I could use this to put each VM in its own VLAN for example. So the server guy is definitely forced to know a little bit about networking because he now has to manage the vswitch. As a network guy, I handle the port connecting to the physical ESX host. That’s going to be the “demarc” for me, as that’s pretty standard for how far my responsibility traditionally goes. If the server guy thinks there’s a problem with connectivity, I’ll verify all the way up to the port that connects to the server. At that point, I’ve verified everything on my end. Things are starting to change with virtualization…

Server virtualization using VMware or Blade enclosures have certainly blurred the lines a bit. We are now starting to see Network guys working more closely with the Server team– Server virtualization has contributed a lot to getting both teams to start talking again . So now the server guys have to tell me that they have 5 different VLANs behind this one virtual host, so I need to make sure I configure the port connecting to the physical server as a trunk (and allow vlans w-z). As you can imagine, lots of room for communication break-downs from both sides. I personally have experienced this when working in a data center environment. When bringing online a new Blade enclosure or VMware server, us network guys really need to know the details about what’s virtualized within that server. What I’ve noticed is that the Server guys don’t want to deal with all the vlan and policy creations for a given virtual machine, because that’s more of a network-centric item. I’m sure most server admins would like to just keep everything on one VLAN to keep is simple. Unfortunately, we sometimes need to have certain customizations at the edge ports connecting to our servers. It can also be a business requirement to make sure certain servers are protected with security policies (ACLs, VLANs, etc). I’ve had Server guys come up to me and tell me they want me to own the connection all the way down to the host, even if it’s virtual. If that’s the case, then we would now be looking into the vswitch level as our new “demarc” or logical divide between ownerships. I don’t see it as a problem, but we need something we can work with– something that’s familiar to us.

The vswitch isn’t bad, but if your looking for your typical CLI as a network guy your going to be dissapointed. While it does have some decent bells and whistles, it doesn’t nearly come close to what we can get from a Cisco switch for example. As a network guy, I can’t exactly manage the vswitch unless I log into the GUI to do so. I wouldn’t be happy about having to manage each vswitch running on every ESX host I have in a cluster. That sounds like a management-overhead-nightmare to me, and I can see why the Server guys don’t like it. We need a solution that will suit both parties and ultimately make our lives easier.

If I have 2 ESX hosts running in my cluster, I can move my virtual machines around between physical boxes using Vmotion. This is cool, but what about those port profiles I setup on ESX host A? I better make sure they exist in both places so that my virtual machine can retain its VLAN for example. This is something that the Server admins have to deal with. Keep in mind, there can be a lot more to a port profile than just a VLAN designation (consider other things like netflow, ACL’s, etc.). In an ideal world, they follow the VM independent of what physical ESX hypervisor it’s on.

The Nexus 1000v does some things to help us out ::

- Get rid of the management overhead of having to deal with separate virtual switches (one for each ESX host).

- Give the network guys something they are familiar with (assuming a Cisco background) and let them own the connection down to the host, virtualized or not.

- Help out the server guys when they need to move a VM to another physical server due to maintenance or problems with the existing server. And make it seamless!!! They don’t like having to reconfigure profiles on the new vswitch.

- Allow server guys to easily assign profiles to a server no matter what ESX host it resides on. Make it so they don’t mess with the network side so much.

- Give server and network guys insight into the virtualized environment. Give virtualized servers the same bells and whistles (ACL’s, netflow, Port spanning, Vlans, etc.) we can get on our physical servers connected to real switches. Help us bridge the virtualization gap!

What does it look like?

Think of the Nexus 1000v architecture as a virtualized modular switch. Think of a Cisco 6500 platform…

Here is the VEM at the ESX host level ::

Remember, each ESX host basically gets its own Ethernet module or VEM. The VEM handles all the forwarding-plane duties and doesn’t rely on the Supervisor (VSM) to forward traffic. Each VM that is attached to the VEM gets a unique Virtual Ethernet interface number (vEth). This vEth number will stay with this VM no matter which ESX host it gets moved to (make sure the new ESX host has a VEM too if using vmotion). The VEM’s have dedicated uplinks and vlans that are used to maintain a connection with the VSM. When I need to manage the Nexus 1000v, I login to the VSM (via mgmt IP address) and have a centralized management scheme. I can do a “show module” on the VSM, and see my supervisors (VSM active/standby) and every VEM (or module #) listed out that exists in each ESX hypervisor. It’s just like logging into a 6500 chassis and seeing all your modules and supervisors, it’s just logically separated.

Here’s another look at the logical connection from the VSM to each VEM ::

Here is what things looked like with the traditional Vmware vswitch ::

You can have two VSM instances which are basically your active and standby supervisor that you are used to seeing. A VEM or Ethernet module exists in each ESX host of your choosing. This will allow you to build a switch with 64 Ethernet modules (across 64 ESX hosts). Now that’s pretty cool when you think about it. There is no backplane or switch fabric to connect everything together like you have in a typical modular switch. So what has to happen is you create a few dedicated VLANs in your network so that the supervisors (VSM) can communicate with each of its modules. The Ethernet network is now the backplane that links the modules to the supervisors. The cool thing about this architecture (Similar to the 7k) is that the forwarding is handled by the line cards (or VEMs in this case). The Supervisor (or VSM) just takes care of the control responsibilities and allows you to manage the entire DVS or distributed virtual switch.

Stick around, I’ll have more information on what you need to get things going, and some help on the configuration side of things. I ran into a lot of road blocks personally. The documentation is out there, but there are some gaps here and there–especially for someone who’s not an ESX pro!. Lots more to say about the 1000v integration with Vcenter server as well.

Pretty amazing stuff imho…

Ted Romer
CCIE No. 21785

Summary

Now that the CCIE version 3 blueprint is being tested on, I have started working on my second CCIE in Voice. While the new blueprint is focused on CUCM version 7, it turns out this book adds value even if it’s based on version 6.

What worked

I wanted a good refresher so I could come up to speed on the latest improvements from version 6 of CUCM and higher. For me, the content in this book is great for understanding the latest features available in CUCM version 6. If you are already familiar with Call Manager than this book will definitely benefit you the most.

I felt the chapter on installation and upgrading was very well done. There are a lot of different versions you have to make sure you’re running before upgrading and this explains things well. It’s definitely going to be something you’ll need to refer to the upgrade guide for, but it’s a good place to start.

I would say this book covers all the basic areas of configuration in Call Manager that you will probably do to get a cluster up and running. I really liked the coverage of Presence support built into the CUCM. You should be able to get a basic Presence configuration working from reading this book. Presence is definitely one of those cool new features to play with.

The end of the book has some great examples on integrating Cisco Unity 5 and Video Advantage into the CUCM. You can use the chapter on Unity as a reference for creating the integration between Unity and CUCM.

Target Audience

This book is obviously geared towards the CCVP 642-446 exam, but it can be used for those preparing for the CCIE exam as well. If you just need a refresher (like me) on CUCM, then this is also a good read.

What didn’t work

You know, I’m kinda used to the “self-study” version of these books having walk-through labs in the end of each chapter vs. exam-like questions. Usually the “certification guides” have the exam questions after each chapter. If I remember right, the CUCM version 4 printing from a few years ago had some walk-through labs. I wouldn’t say there are a ton of walk through examples, rather screen shots of certain sections. If your expecting a book you can use to walk you through the basic configuration, you should reference the config guide on Cisco’s website. Not a big deal, but surprising for a “self-study”guide. Personally, I would read this to get yourself up to speed, then take a look at the actual version 6 SRND guide.

Ratings

Presentation (8/10)- Lots of tables/figures and example screenshots from the version 6 GUI. Plenty of visual aides. The examples given to explain a given topic make sense.

Content (7/10)- The book manages to come in at about 500 pages. Granted a small chunk of that are the exam questions at the end of each chapter, but not bad considering there are no detailed walk-through labs provided.

Practicality (6/10)- I would say if you have access to a CUCM version 6 cluster, you should be able to read each chapter and figure out how to configure the different areas within CUCM. I would recommend having access to a cluster so you can follow along. Not exactly a walk-through, but close…

Shelf Appeal (3/5)- Definitely worth keeping. There are a few chapters in the book that will prove to be a good reference. For example, the chapters on Presence and Unity are good ones to have to refer back to.

Value (3/5)- Probably going to cost you around $50 dollars online. Plan on buying Part 2 as well (to be reviewed in near future). I would have preferred both books bundled into one, but they are separate CCVP tests…

Overall (Great 8/10)- Overall a great read and I’m currently reading Part 2 of the series. After reading this book you should have the CUCM basics down, and a good idea how to get a basic CUCM config up and running.

Ted Romer
CCIE No. 21785

Turns out there is a field notice and I need the latest hardware version of the cards.

Open a TAC SR and get an RMA going. I figured it would be an EPLD upgrade issue, but I guess not.

Notice the version in the output from “show mod”

nexus7000# sho mod
Mod Ports Module-Type Model Status
— —– ——————————– —————— ————
1 32 10 Gbps Ethernet Module N7K-M132XP-12 ok
2 32 10 Gbps Ethernet Module N7K-M132XP-12 ok
3 32 10 Gbps Ethernet Module N7K-M132XP-12 ok
4 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok
5 0 Supervisor module-1X N7K-SUP1 active *

Mod Sw Hw
— ————– ——
1 4.1(5) 1.3
2 4.1(5) 1.3
3 4.1(5) 1.3
4 4.1(5) 1.0
5 4.1(5) 1.1

Source

NEW: Troubleshooting

Beginning October 18, 2009, the CCIE R&S lab exam will feature a two-hour troubleshooting section. Candidates will be presented with a series of trouble tickets for preconfigured networks and need to diagnose and resolve the network fault or faults. As with the configuration section, the network must be up and running for a candidate to receive credit. Candidates who finish the troubleshooting section early may proceed on to the configuration section, but they will not be allowed to go back to troubleshooting since their equipment will need to be reinitialized for the configuration portion.

As far as the exam topics go, it seems there are some additions ::

Source

- Performance routing (pfr)
- MPLS VPNs
- Optimization (Netflow, EEM)

I think this is all good stuff personally. MPLS VPNs are not really that hard. I have done some articles with config examples on this site for MPLS VPNs. They are quite fun.

The troubleshooting addition is going to be interesting it seems.

Performance routing (pfr) configurations can get really nasty and complex, but other than that not too much to worry about.