Latest Posts

F5 Forwarding IP VS fun…

I have been recently spending lots of time with our F5 Big IP 2000s. We have been working on deploying a new private network behind the F5 with nodes that admin’s would like to directly access from their secure admin workstations. Our current setup has the nodes behind the load balancers using basic Virtual servers that forward the traffic from external routable IP’s to internal non-routable IP’s. Therefore I would need to create multiple VS’s per each node that admins wanted to access. That would be a lot of VS’s.

There are a few other clever ways to get around this, well maybe not necessarily clever. The first is using a jump server. The admins would access one VS that would forward to one pool member with one node in it. Then they could then access the other private nodes from this “jump” server. The other option would be adding another NIC to the admin workstations and put that NIC on the same VLAN that the private nodes sit on. Both of these are not the greatest ideas.

I therefore convened with an F5 tech guru and passed this idea by him. Could I have a router with a routed interface within the F5 private VLAN that has the F5 private nodes? I could then take the private non-routable network and make it routable protected by an ACL on the router. The nodes would still point to the F5 for the default gateway. When an admin workstation would communicate to the node, it would send the traffic through the router; the router would then forward packets to the node. The issue then lies with the node sending the traffic back to the F5 because it’s the default gateway and that creates issues. I found out from the tech that there’s a way to get this to work by using an IP forwarding VS that listens on the F5 private VLAN.

You will first need to make sure that your current nodes are not in an SNAT, as SNATs along with an IP forwarding VS configured on the same interface don’t work, as the SNAT listens over the IP forwarding VS. Within the VS, the source network would be the private node network and the destination would be the network where the secure admin workstations sit, which would be accessible across the router that I placed on the private node VLAN. Now, in order to get the F5 to forward to the router on the private node VLAN instead of using its routing table, you have to create an iRule as the Forwarding (IP) resource within the VS. Here’s the iRule syntax:


if { [IP::addr [IP::client_addr] equals] }   {

nexthop internalvlannum



The is the internal private node network that sits on the F5. The address is the gateway of the router that’s sitting on the F5 private node network. This iRule forwards the traffic to the router instead of using the F5 to forward the traffic. You also have to assign a protocol profile (client)  Fast L4 profile to the forwarding (IP) VS as well with Loose initiate and loose close enabled to allow the tcp packets to flow correctly.

What happens in Mexico, saves Mexico

I just had an opportunity to go on my first mission’s trip to Playa Del Carmen in Mexico for a week with seven members of the church I attend. We were invited and led by Mission Explosion International. We planned to visit and help out a local Church of Christ in Playa Del Carmen. Our goals consisted of going out in the neighborhood to promote a clinic check that would be hosted in the local Church building. The clinic sessions included checking blood pressure and sugar levels along with hosting VBS (vacation bible school) days as well. We also planned to hand out Spanish Mission soccer balls.


We ended up doing 3 days of clinic work, 2 days of children outreach, one day of men’s and women’s bible study, and our Evangelical guide preached on Sunday. The trip was amazing. We had lots of fellowship, outreach, teaching, and preaching going on all lead by God and His spirit. If you ever have an opportunity to go on a mission’s trip, please pray diligently and seek God’s guidance in the matter. I found out that I was way more capable through Christ than I ever imagined. In the end, the culture, language, and the income levels may be different, but the people are no different than us in the fact that they need God and Jesus Christ just like we do.


Thank you to Rusty, Audrey, Jordan, and Kylee!!! I will never forget my first mission’s trip to Mexico.


Cisco UC on UCS

One of our latest projects at work is moving from centrex to Cisco Unified Communications (UC). I was assigned to complete quite a few tasks for this project. One of the first tasks was getting our two UCS c240 rack mount servers going. Our purchase was part of a larger order, so our UCS rack mount servers are bare metal servers. No esxi or UC components were pre-installed. I originally wanted to install esxi on the cisco flexflash SD card, but then found out that UC on UCS doesn’t recommend that configuration. I also found out that Cisco recommends that 2 RAID 5 arrays should be created across the 16 drives that we have in our c240’s.

ucs 2u c240 server

If you want to stay on a Cisco tested reference configuration (TRC), then I would recommend that you check out this Cisco page for reference:

Cisco Virtual Internet Routing Lab – Up and Running…

I was finally able to fix the issue that I described having in my earlier Cisco VIRL article. My original bare metal box only had 3 NICs. VIRL requires that you have at least 5 NICs. If you don’t have 5 NIC’s, then you have to modify the /etc/virl.ini file with dummy interfaces. I did this earlier, but must have had a mistake in the config. I double checked the config and also ran the VIRL-rehost script that’s on the desktop when you login to VIRL. Running the script wasn’t in the VIRL doc steps, so I didn’t do this before. Running the script after modifying the virl.ini file with the dummy interfaces finally fixed my issue. The script modified the /etc/network/interfaces with the correct dummy interfaces. Here’s an example of what the script changed:

iface dummy1 inet static



post-up ip link set dummy1 promisc on

I setup an IOSv router and connected it to an L2 External (flat) network which connects back through one of my physical NICs. That connection then goes into a real cisco ws-c3560x 24 port switch. That switch is connected to my network and I assigned my PC another IP address on the 172.16.1.x network. I can now ssh into the IOSv router directly from my desktop.

virl flat network


Now that I have worked out all the bugs, I’m pretty impressed with the functionality that VIRL provides. Now I’m going to see how many routers I can throw at this box.

Ping you later,


Cisco VIRL setup

In my last post, I spoke about getting a cisco virl (virtual internet routing lab) server going. I started with a hyper-V installation, which wasn’t listed as being supported. I gave it a try anyways. What I came to find out is that hyper-V would not work with my setup because I couldn’t do nested virtualization. Cisco Virl runs KVM under the hood which needs native VT-d. I couldn’t get hyper-v to pass VT-d to the host, so that was a no go. I decided to wipe the drive and load the iso version of cisco virl directly on my box. After a few failed attempts, I finally got virl to run without giving an error when trying to license the software. You have to follow the virl install guide to the T. I accidentally didn’t put in the correct hostname that the guide said to use. That caused the initial installation of virl to give an error every time I tried to run the user workspace manager. After following the exact directions, I was able to get virl to load.

Now I’m running into one last error message:

state changed from BUILD to ERROR with message: No valid host was found. Exceeded max scheduling attempts 3 for instance

cisco virl

I’m digging into the logs in order to figure out what the issue is, I’m really close. Hopefully my next post will be an in depth review of cisco virl.

occupied, time to setup Cisco VIRL

I’ve been occupied with many other things going on lately especially with the holidays. Family, work, etc. I haven’t been able to sit and write, but I ran into something pretty cool that I wanted to share. Those of you who are always looking to expand your knowledge in networking, cisco just released VIRL or virtual internet routing lab. This is Cisco’s newest virtualization lab simulation tool. It reasonably priced as well.

I saw this as a perfect opportunity to build a homebrew cisco VIRL hyper-v server. Unfortunately hyper-v isn’t supported, but that didn’t stop me. I recently found a dell studio 435 with an i7-920 processor in the garbage down the street. The only issue it had was a dying power supply.

cisco VIRL home server

Once that was fixed, I started to install hyper-v 2012. To my dismay, I couldn’t get hyper-v loaded without the installation crashing. All settings looked to be good. Virtualization was enabled in the bios and all other recommended settings were set. I then realized that the bios version was pretty old and figured I try to do an update. Low and behold a bios firmware fixed the crashing issue. I slapped another broadcom pci-e nic and finished the install. Now I’m having issues getting the broadcom nic and the onboard nic to load correctly. After searching the Internet, I figured out how to manually add drivers in hyper-v core mode. I’m still running into some issues with NIC detection, so I think I may run esxi on another partition of the drive just to get VIRL running. Stay tuned….

You can now follow up on my cisco bare metal home build here.


Extreme Networks Midwest Roundtable

I recently attended the Extreme Networks Midwest Roundtable event. These types of small events are good to attend as they promote great technical discussions, company vision, and product road-maps. It also gives current customers a chance to speak up about what features we are looking for in new products. It also allows interested potential customers to interact with real world current customers.

Here are some thoughts that I captured from the guest speakers:

How much do we as administrators spend on just maintaining our current infrastructure? – Dan Dulac
How can we use IT to drive business outcomes? – Dan Dulac
Here’s a good one. Netflix sucks when I tried using it on xyz’s network, therefore xyz sucks. Brand perception is highly important. – Dan Dulac

These discussions lead into Mike Lebovitz talking about Extreme Networks “Purview”.
Purview is classified as an application intelligence device that allows you to see analytic data from your network.

purview layer 7 visibility

Purview Layer 7 Visibility

In a nutshell, you get layer 7 visibility across your entire network. Granted you will need a specific line of Extreme Network switches to use it, but if you currently use Extreme Network’s gear I can see this as being another really useful tool. I’m still waiting to see if you will be able to enforce bandwidth/traffic shaping policies to edge ports based on L7 traffic. Some environments out there can benefit greatly about knowing what’s being used on their networks which ties into answering some of Dan Dulac’s questions above. We would benefit from the ability to shape our traffic in our current environment. We currently use an Allot Netenforcer which works great. However, this device sits and enforces at the border of our network and will need a periodic hardware refresh from time to time. If we could leverage Purview with our existing Extreme Networks equipment and enforce closer to the edge ports that would be great.

A few side notes to add were that Extreme Networks will be moving to a unified OS which will be the current Extreme OS or XOS. For those of you who don’t know, Extreme Networks recently purchased Enterasys. I’m excited to see the product merging that’s currently going on. The Enterasys dynamic policy (L2-L4 dynamic ACL) features will also be migrated to the Extreme line of equipment that can support it. Extreme Networks will also be building upon Enterasys Netsight, which is what we currently use to manage our network gear.

Overall the roundtable was a pretty neat event. It’s typically held in the fall around the downtown Milwaukee area for those who are interested in attending next year.

To cert or not to cert…

I have been pondering the thought of gaining some additional certifications for quite some time now. My good friend Shane recently asked what my thoughts were on certifications. I currently hold only two certificates myself. The first is Enterasys Switch Specialist and ITIL foundation certification. Both tests and training were funded by my employer. If your employer offers to pay for classes and for the test, then by all means do not pass up the opportunity.

I think the real question in everyone’s mind is how much will a certificate help me out? This thought may come to us when we are thinking about seeking different career opportunities or maybe when we receive additional job responsibilities. Now as far as discussing the first scenario, I have been on both the hiring/seeking spectrums within the IT industry. Certifications definitely help solidify choices. Although I truly believe that experience trumps certificates. Here’s why. For instance, you can have someone that barely knows networking. They have tinkered around with home networks and possibly the one switch/router at work. They go home and study for 6 weeks straight. They practice hammering away at the cisco cli. They take the CCNA test and pass it. I’m not dismissing that accomplishment. It’s a difficult task and deserves recognition. I don’t even have CCNA status. Here’s my point, if you don’t use it, you lose it. If that person doesn’t work with cisco every day, chances are that they will lose it. Which one would you hire? The guy with 10 years of reputable experience or a guy with 1 year of experience and their CCNA?

Now I know I’m probably getting criticism on that one, but hey, that’s just my opinion. Ok, now you’re asking why doesn’t the guy with 10 years of reputable experience just go out and get the CCNA? Valid question. If you work with cisco all day long, then that’s probably a good idea. If you’re not a cisco shop, you’ll have to study a little. Bottom line is value. If you’re starting to look for another job, then getting your CCNA won’t hurt. If you’re crunched for time with work, family, and other things like me and are content with where you’re at, then maybe you don’t want to spend the extra money, time, and pressures of test taking going after your CCNA right now.

This same theory can be applied to other certificates. Now if your job is ramping up your responsibilities, then it never hurts to ask them to pay for a boot camp course. It will benefit both yourself and your company. If you’re feeling froggy, study the material a little more and go take the test. You can’t lose at that point. Your new job responsibilities will get you the solid experience you need. If you fail the test the first time, don’t get discouraged. You’ll at least know where to brush up and you’ll continue to build your skills at work.

In the meantime, have fun learning. It never hurts to learn or push yourself with a challenge of obtaining a cert. If you can’t afford those expensive books like me, then try using safaribooks. There are also lots of trial VM solutions out there now that can help you create a nice lab environment relatively cheap (F5 Virtual Edition/brocade vyatta/cisco csr1000v). GNS3 is also a nice tool to have, just remember to buy yourself a cheap router from ebay with an iso image. I’m not going to condone any illegal practices. Have fun!!!

802.11 Wireless Channel Planning…

I used to work with Motorola Canopy wireless gear back in the day. It was great ptmp wireless gear. The carrier to interference ratio on this gear was 3 dB, which meant that you could provide extremely reliable wireless links in high RF congested areas. Of course this was proprietary based equipment that didn’t come close to the contention based CDMA/CA 802.11 stuff. I was extremely spoiled when using the motorola canopy line. It just worked.

Why am I bringing up all this when talking about 802.11 wireless channel planning? Well it’s because of the preconceived notion that using all the 2.4Ghz 802.11 channels may be a good thing (including overlapping channels). At first, I though, just let the AP’s pick the best channel between 1-11 and that I would be good to go. Well that wasn’t the best solution as AP’s were selecting channels 1,3,4,7,10, etc. My thoughts were spurred on by this twitter discussion with wireless guru, Keith Parsons.

Me to Keith “What’s better, SNR of 20db between two AP’s on channel 1 or SNR of 20db between AP on channel 1 and AP on channel 3?”
Keith “I’d go with the two on the same channel, given a choice. At least they’d ‘Play Nice’ with each other rather than ACI fighting.”
Me to Keith “So is it CSMA/CA that works better at detection if interference is on same channel?”

Keith “It is the difference between how 802.11 deals with CCI vs ACI.”
Me “So based on CCA, you would rather detect noise and back off rather than face possible data corruption from ACI.?”

The last question wasn’t answered. Basically I needed to do my homework and I would suggest you do the same. Check out the following link:

In the above article Andrew really dives into understanding how CDMA/CA works. What I gathered what that wifi can use carrier sense techniques in order to back off in order to avoid possible frame corruption. Keith’s statement of “AP’s on the same channel will likely play nice with each other” will allow carrier sense to do its job.

Ok, so I started doing some more research which lead me to my safaribooks account and checking out a CWNA (certified wireless network administrator) study guide book. The book defines CCI as co-channel interference or “unnecessary medium contention overhead that occurs because all the AP’s are on the same channel.” (Coleman and Westcott) Basically you have wireless devices following the rules of CDMA/CA.

Now ACI is defined as adjacent channel interference and is what you get when you use channels that overlap with one another. The only non-overlapping channels in 2.4Ghz are 1,6,11. When Keith suggests that he would rather see two AP’s on the same channel instead of two using overlapping channels such as 1 and 3, it’s because the 1 and 3 will give you ACI. ACI will cause re-transmits due to corruption of frames. You will rather want CDMA/CA to work as it should instead of facing re-transmits due to corrupt frames.

Where did this lead me? Well, I turned off the AP’s auto channel feature and went back to only using channels 1, 6, and 11. I hard set the channels myself and performed a site survey. My goal was to try to maximize the SNR between any two AP’s that my client could see that were on the same channel. At the end of the day, I’m seeing better performance especially since we just doubled the number of access points we used to have. I had to also play around with the minimum basic rate and power output levels as well in order to achieve maximum optimization.

Here’s a busy, but healthy network using 1,6,11 (Shane, disregard my previous comments on your screenshot using only channels 1, 6, and 11) – Courtesy of Metageek Chanalyzer

Metageek wireless channel planning spectrum image

Healthy 2.4Ghz wireless channel planning. I’m working on 5Ghz as well. – Courtesy of Extreme Networks Oneview.

wireless channel planning

Wireless channel planning map


CWNA: Certified Wireless Network Administrator Official Study Guide: Exam PW0-105, 3rd Edition by David D. Coleman; David A. Westcott


ICX6610 Brocade Switch – Firmware

Ok, so I’m in the process of configuring our spare icx 6610 brocade switch with the same firmware and configuration of one of our main border WAN icx 6610’s that’s running BGP. If our router fails, then I will have a hot spare ready to go while I wait for an RMA. This requires that the space icx 6610 is running router firmware along with an icx 6610 advanced license. The icx 6610 has the ability to run in layer 2 switching mode only or layer 3 mode when the proper firmware is loaded. By default, I had two code versions loaded on the icx 6610. You can identify which code is which by looking at the flash file name. Running the show flash command will give you the following:

#show flash
Stack unit 1:
Compressed Pri Code size = 7189206, Version:07.4.00cT7f3 (FCXR07400c.bin)
Compressed Sec Code size = 8874046, Version:08.0.00aT7f3 (FCXR08000a.bin)
Compressed Boot-Monitor Image size = 370733, Version:07.3.03T7f5
Code Flash Free Space = 49020928

You want to look at the character after FCX in parentheses, I have (FCXR07400c.bin) and (FCXR08000a.bin). The fourth character is an R, which is for routing. If the letter was an S, then that would represent switching only. Right now, I have two different routing versions. Since this is a major rev., there was also a new boot code as well (07.3.03T7f5). Unfortunately, there’s not two slots in the flash to have two different boot codes. If I try to boot back to the 7.4 code, I won’t have the proper boot code that I had before. I will therefore copy another version 8 code to the primary flash space. To do that I will run the following command:

#copy tftp flash <ip-of-tftp-server> <flash-image> primary

In order to boot to the new flash (assuming you don’t need a new boot code), you can then run the following command:

#boot system flash <primary | secondary>

HINT: The default boot is set to boot from the primary location, so if you reboot again without running another command, it will boot back to the default location. If you want to always boot from the secondary flash location, run the following command in config t mode:

(config)#boot system flash secondary

Don’t forget to write mem.


1 2 3 4 6