Cisco UC on UCS

One of our latest projects at work is moving from centrex to Cisco Unified Communications (UC). I was assigned to complete quite a few tasks for this project. One of the first tasks was getting our two UCS c240 rack mount servers going. Our purchase was part of a larger order, so our UCS rack mount servers are bare metal servers. No esxi or UC components were pre-installed. I originally wanted to install esxi on the cisco flexflash SD card, but then found out that UC on UCS doesn’t recommend that configuration. I also found out that Cisco recommends that 2 RAID 5 arrays should be created across the 16 drives that we have in our c240’s.

ucs 2u c240 server

If you want to stay on a Cisco tested reference configuration (TRC), then I would recommend that you check out this Cisco page for reference:

Cisco Virtual Internet Routing Lab – Up and Running…

I was finally able to fix the issue that I described having in my earlier Cisco VIRL article. My original bare metal box only had 3 NICs. VIRL requires that you have at least 5 NICs. If you don’t have 5 NIC’s, then you have to modify the /etc/virl.ini file with dummy interfaces. I did this earlier, but must have had a mistake in the config. I double checked the config and also ran the VIRL-rehost script that’s on the desktop when you login to VIRL. Running the script wasn’t in the VIRL doc steps, so I didn’t do this before. Running the script after modifying the virl.ini file with the dummy interfaces finally fixed my issue. The script modified the /etc/network/interfaces with the correct dummy interfaces. Here’s an example of what the script changed:

iface dummy1 inet static



post-up ip link set dummy1 promisc on

I setup an IOSv router and connected it to an L2 External (flat) network which connects back through one of my physical NICs. That connection then goes into a real cisco ws-c3560x 24 port switch. That switch is connected to my network and I assigned my PC another IP address on the 172.16.1.x network. I can now ssh into the IOSv router directly from my desktop.

virl flat network


Now that I have worked out all the bugs, I’m pretty impressed with the functionality that VIRL provides. Now I’m going to see how many routers I can throw at this box.

Ping you later,


Cisco VIRL setup

In my last post, I spoke about getting a cisco virl (virtual internet routing lab) server going. I started with a hyper-V installation, which wasn’t listed as being supported. I gave it a try anyways. What I came to find out is that hyper-V would not work with my setup because I couldn’t do nested virtualization. Cisco Virl runs KVM under the hood which needs native VT-d. I couldn’t get hyper-v to pass VT-d to the host, so that was a no go. I decided to wipe the drive and load the iso version of cisco virl directly on my box. After a few failed attempts, I finally got virl to run without giving an error when trying to license the software. You have to follow the virl install guide to the T. I accidentally didn’t put in the correct hostname that the guide said to use. That caused the initial installation of virl to give an error every time I tried to run the user workspace manager. After following the exact directions, I was able to get virl to load.

Now I’m running into one last error message:

state changed from BUILD to ERROR with message: No valid host was found. Exceeded max scheduling attempts 3 for instance

cisco virl

I’m digging into the logs in order to figure out what the issue is, I’m really close. Hopefully my next post will be an in depth review of cisco virl.

occupied, time to setup Cisco VIRL

I’ve been occupied with many other things going on lately especially with the holidays. Family, work, etc. I haven’t been able to sit and write, but I ran into something pretty cool that I wanted to share. Those of you who are always looking to expand your knowledge in networking, cisco just released VIRL or virtual internet routing lab. This is Cisco’s newest virtualization lab simulation tool. It reasonably priced as well.

I saw this as a perfect opportunity to build a homebrew cisco VIRL hyper-v server. Unfortunately hyper-v isn’t supported, but that didn’t stop me. I recently found a dell studio 435 with an i7-920 processor in the garbage down the street. The only issue it had was a dying power supply.

cisco VIRL home server

Once that was fixed, I started to install hyper-v 2012. To my dismay, I couldn’t get hyper-v loaded without the installation crashing. All settings looked to be good. Virtualization was enabled in the bios and all other recommended settings were set. I then realized that the bios version was pretty old and figured I try to do an update. Low and behold a bios firmware fixed the crashing issue. I slapped another broadcom pci-e nic and finished the install. Now I’m having issues getting the broadcom nic and the onboard nic to load correctly. After searching the Internet, I figured out how to manually add drivers in hyper-v core mode. I’m still running into some issues with NIC detection, so I think I may run esxi on another partition of the drive just to get VIRL running. Stay tuned….

You can now follow up on my cisco bare metal home build here.


Extreme Networks Midwest Roundtable

Around the Table

I recently attended the Extreme Networks Midwest Roundtable event. These types of small events are good to attend as they promote great technical discussions, company vision, and product road-maps. It also gives current customers a chance to speak up about what features we are looking for in new products. It also allows interested potential customers to interact with real world current customers.

Here are some thoughts that I captured from the guest speakers:

How much do we as administrators spend on just maintaining our current infrastructure? – Dan Dulac
How can we use IT to drive business outcomes? – Dan Dulac
Here’s a good one. Netflix sucks when I tried using it on xyz’s network, therefore xyz sucks. Brand perception is highly important. – Dan Dulac

These discussions lead into Mike Lebovitz talking about Extreme Networks “Purview”.
Purview is classified as an application intelligence device that allows you to see analytic data from your network.

purview layer 7 visibility

Extreme Analytics Layer 7 Visibility

Application visibility

In a nutshell, you get layer 7 visibility across your entire network. Granted you will need a specific line of Extreme Network switches to use it, but if you currently use Extreme Network’s gear I can see this as being another really useful tool. I’m still waiting to see if you will be able to enforce bandwidth/traffic shaping policies to edge ports based on L7 traffic. Some environments out there can benefit greatly about knowing what’s being used on their networks which ties into answering some of Dan Dulac’s questions above. We would benefit from the ability to shape our traffic in our current environment. We currently use an Allot Netenforcer which works great. However, this device sits and enforces at the border of our network and will need a periodic hardware refresh from time to time. If we could leverage Extreme Analytics with our existing Extreme Networks equipment and enforce closer to the edge ports that would be great.

Unified Extreme Networks Operating System

A few side notes to add were that Extreme Networks will be moving to a unified OS which will be the current Extreme OS or XOS. For those of you who don’t know, Extreme Networks recently purchased Enterasys. I’m excited to see the product merging that’s currently going on. The Enterasys dynamic policy (L2-L4 dynamic ACL) features will also be migrated to the Extreme line of equipment that can support it. Extreme Networks will also be building upon Enterasys Netsight, which is what we currently use to manage our network gear.

Overall the roundtable was a pretty neat event. It’s typically held in the fall around the downtown Milwaukee area for those who are interested in attending next year.

To cert or not to cert…

I have been pondering the thought of gaining some additional certifications for quite some time now. My good friend Shane recently asked what my thoughts were on certifications. I currently hold only two certificates myself. The first is Enterasys Switch Specialist and ITIL foundation certification. Both tests and training were funded by my employer. If your employer offers to pay for classes and for the test, then by all means do not pass up the opportunity.

I think the real question in everyone’s mind is how much will a certificate help me out? This thought may come to us when we are thinking about seeking different career opportunities or maybe when we receive additional job responsibilities. Now as far as discussing the first scenario, I have been on both the hiring/seeking spectrums within the IT industry. Certifications definitely help solidify choices. Although I truly believe that experience trumps certificates. Here’s why. For instance, you can have someone that barely knows networking. They have tinkered around with home networks and possibly the one switch/router at work. They go home and study for 6 weeks straight. They practice hammering away at the cisco cli. They take the CCNA test and pass it. I’m not dismissing that accomplishment. It’s a difficult task and deserves recognition. I don’t even have CCNA status. Here’s my point, if you don’t use it, you lose it. If that person doesn’t work with cisco every day, chances are that they will lose it. Which one would you hire? The guy with 10 years of reputable experience or a guy with 1 year of experience and their CCNA?

Now I know I’m probably getting criticism on that one, but hey, that’s just my opinion. Ok, now you’re asking why doesn’t the guy with 10 years of reputable experience just go out and get the CCNA? Valid question. If you work with cisco all day long, then that’s probably a good idea. If you’re not a cisco shop, you’ll have to study a little. Bottom line is value. If you’re starting to look for another job, then getting your CCNA won’t hurt. If you’re crunched for time with work, family, and other things like me and are content with where you’re at, then maybe you don’t want to spend the extra money, time, and pressures of test taking going after your CCNA right now.

This same theory can be applied to other certificates. Now if your job is ramping up your responsibilities, then it never hurts to ask them to pay for a boot camp course. It will benefit both yourself and your company. If you’re feeling froggy, study the material a little more and go take the test. You can’t lose at that point. Your new job responsibilities will get you the solid experience you need. If you fail the test the first time, don’t get discouraged. You’ll at least know where to brush up and you’ll continue to build your skills at work.

In the meantime, have fun learning. It never hurts to learn or push yourself with a challenge of obtaining a cert. If you can’t afford those expensive books like me, then try using safaribooks. There are also lots of trial VM solutions out there now that can help you create a nice lab environment relatively cheap (F5 Virtual Edition/brocade vyatta/cisco csr1000v). GNS3 is also a nice tool to have, just remember to buy yourself a cheap router from ebay with an iso image. I’m not going to condone any illegal practices. Have fun!!!

802.11 Wireless Channel Planning

Different Technology

I used to work with Motorola Canopy wireless gear back in the day. It was great ptmp wireless gear. The carrier to interference ratio on this gear was 3 dB, which meant that you could provide extremely reliable wireless links in high RF congested areas. Of course this was proprietary based equipment that didn’t come close to the contention based CDMA/CA 802.11 stuff. I was extremely spoiled when using the Motorola canopy line. It just worked.

802.11 Wireless Channel Planning

Why am I bringing up all this when talking about 802.11 wireless channel planning? Well it’s because of the preconceived notion that using all the 2.4Ghz 802.11 channels may be a good thing (including overlapping channels). At first, I though, just let the AP’s pick the best channel between 1-11 and that I would be good to go. Well that wasn’t the best solution as AP’s were selecting channels 1,3,4,7,10, etc. My thoughts were spurred on by this twitter discussion with wireless guru, Keith Parsons.

Javier S. “What’s better, SNR of 20db between two AP’s on channel 1 or SNR of 20db between AP on channel 1 and AP on channel 3?”
Keith P. “I’d go with the two on the same channel, given a choice. At least they’d ‘Play Nice’ with each other rather than ACI fighting.”
Javier S. “So is it CSMA/CA that works better at detection if interference is on same channel?”

Keith P. “It is the difference between how 802.11 deals with CCI vs ACI.”
Javier S. “So based on CCA, you would rather detect noise and back off rather than face possible data corruption from ACI.?”

The last question wasn’t answered. Basically I needed to do my homework and I would suggest you do the same. Check out the following link:

In the above article Andrew really dives into understanding how CDMA/CA works. What I gathered what that wifi can use carrier sense techniques in order to back off in order to avoid possible frame corruption. Keith’s statement of “AP’s on the same channel will likely play nice with each other” will allow carrier sense to do its job.


Ok, so I started doing some more research which lead me to my safaribooks account and checking out a CWNA (certified wireless network administrator) study guide book. The book defines CCI as co-channel interference or “unnecessary medium contention overhead that occurs because all the AP’s are on the same channel.” (Coleman and Westcott) Basically you have wireless devices following the rules of CDMA/CA.

Now ACI is defined as adjacent channel interference and is what you get when you use channels that overlap with one another. The only non-overlapping channels in 2.4Ghz are 1,6,11. When Keith suggests that he would rather see two AP’s on the same channel instead of two using overlapping channels such as 1 and 3, it’s because the 1 and 3 will give you ACI. ACI will cause re-transmits due to corruption of frames. You will rather want CDMA/CA to work as it should instead of facing re-transmits due to corrupt frames.


Where did this lead me? Well, I turned off the AP’s auto channel feature and went back to only using channels 1, 6, and 11. I hard set the channels myself and performed a site survey. My goal was to try to maximize the SNR between any two AP’s that my client could see that were on the same channel. At the end of the day, I’m seeing better performance especially since we just doubled the number of access points we used to have. I had to also play around with the minimum basic rate and power output levels as well in order to achieve maximum optimization. Hopefully this helps explain why proper channel planning is extremely important.


Here’s a busy, but healthy network using 1,6,11 (Shane, disregard my previous comments on your screenshot using only channels 1, 6, and 11) – Courtesy of Metageek Chanalyzer

Metageek wireless channel planning spectrum image


Healthy 2.4Ghz wireless channel planning. I’m working on 5Ghz as well. – Courtesy of Extreme Networks Oneview.

wireless channel planning 5Ghz building map

Wireless channel planning map

Wireless Channel Planning Resources

CWNA: Certified Wireless Network Administrator Official Study Guide: Exam PW0-105, 3rd Edition by David D. Coleman; David A. Westcott


ICX6610 Brocade Switch – Firmware

Ok, so I’m in the process of configuring our spare icx 6610 brocade switch with the same firmware and configuration of one of our main border WAN icx 6610’s that’s running BGP. If our router fails, then I will have a hot spare ready to go while I wait for an RMA. This requires that the space icx 6610 is running router firmware along with an icx 6610 advanced license. The icx 6610 has the ability to run in layer 2 switching mode only or layer 3 mode when the proper firmware is loaded. By default, I had two code versions loaded on the icx 6610. You can identify which code is which by looking at the flash file name. Running the show flash command will give you the following:

#show flash
Stack unit 1:
Compressed Pri Code size = 7189206, Version:07.4.00cT7f3 (FCXR07400c.bin)
Compressed Sec Code size = 8874046, Version:08.0.00aT7f3 (FCXR08000a.bin)
Compressed Boot-Monitor Image size = 370733, Version:07.3.03T7f5
Code Flash Free Space = 49020928

You want to look at the character after FCX in parentheses, I have (FCXR07400c.bin) and (FCXR08000a.bin). The fourth character is an R, which is for routing. If the letter was an S, then that would represent switching only. Right now, I have two different routing versions. Since this is a major rev., there was also a new boot code as well (07.3.03T7f5). Unfortunately, there’s not two slots in the flash to have two different boot codes. If I try to boot back to the 7.4 code, I won’t have the proper boot code that I had before. I will therefore copy another version 8 code to the primary flash space. To do that I will run the following command:

#copy tftp flash <ip-of-tftp-server> <flash-image> primary

In order to boot to the new flash (assuming you don’t need a new boot code), you can then run the following command:

#boot system flash <primary | secondary>

HINT: The default boot is set to boot from the primary location, so if you reboot again without running another command, it will boot back to the default location. If you want to always boot from the secondary flash location, run the following command in config t mode:

(config)#boot system flash secondary

Don’t forget to write mem.


Raspberry Pi network monitoring wifi Smokeping

I finally deployed my wireless raspberry pi network monitoring device at the edge of our wireless network. I installed the smokeping app on the raspberry pi with wheezy via apt-get. I already had a smokeping running on a Linux server, so I setup the raspberry pi to run smokeping in client mode.

Client mode allows the raspberry pi to pull the config from the smokeping server in order to know what tests to run. I setup some fping, DNS, and tcpping tests. This is a great way to test the client’s wireless experience (latency) on the other side of campus. I stuck the raspberry pi behind a bundle of cables to provide for additional attenuation to simulate a user in a worse case scenario. The AP was a few rooms away on a different floor as well. Here are some of the smokeping graphs:

raspberryPi Smokeping monitoring wifi latency

The line with the higher latency is the raspberry pi to google and the lower line is the latency from the smokeping server to google via fping in the picture above.

Here’s a longer term graph painted vi rrdtool by smokeping. There’s some packet loss going on over the wireless interface on the raspberry pi. I may have to move it to a better spot and see how the graphs look afterward. I also want to install iperf and maybe tshark as well.

RaspberryPi smokeping long term wifi latency

Here’s the command to get the raspberry pi running in client mode:

/usr/sbin/smokeping –master-url= –cache-dir=/var/smokeping/ –shared-secret=/var/smokeping/secret.txt

I also had to modify the permissions in the /tmp/smokeping-ms/data/ folder on the server in order to allow the rrds to be modified by apache.

If you’re looking for a small free solution or larger scale paid solution, take a look at I would recommend giving their NetBeez free tier model a try.


Another round of great IT web resources

Check out the new list of great IT web resources below. I’ve also started to use twitter to get involved with the network/wifi community. I don’t really like how twitter works, but that’s where lots of tech enthusiasts are going. The user interface on my nexus 4 android is not very intuitive, but once you figure it out, it’s a great resource. The good thing is that there’s not too much to figure out.

The hardest thing is getting followers, but hey, start posting worthy comments and you might catch a few followers.


IT Resource list: Chicago Network Operators Group You can find some great podcasts here. There’s an extensive list of tools in the armory section. Great info on SDN. Purdue Prof, great insight. Shane blogs very often. Worth the visit. A great wifi gem.

1 2 3 4 5 7