Best of Breed

I recently heard the term “best of breed” used when discussing network vendor selection. I was surprised by this answer because you don’t hear it too often.  The more I thought about it, why not “best of breed” selection? My time as a network and infrastructure supervisor has taught me that a data center environment can be full of different compute and storage vendor products. Our SAN environment consists of Pure, Tegile, EMC, and even QNAP. Each product has its place.  Pure serves the VDI environment, Tegile/EMC host production, and QNAP serves as a target for our Veeam backups. The team has also categorized and carved out each platform into tiered offerings.

On the other hand, network vendor selection tends to be biased.  Typically you’ll see one network vendor selected for the edge/access, distribution, and core. However, you will find a different wireless vendor from time to time.

I’ve seen many reasons for this, so I’ve compiled a list of the most popular reasons I’ve heard:

  • We would like to interact with only one vendor for purchases and support.
  • ABC vendor only works well with a particular management tool.
  • I only know vendor ABC, and we don’t have time to learn something new.
  • Did you hear that vendor ABC had an issue with XYZ product, I don’t want those problems.
  • Everyone else uses vendor ABC.
  • No other vendor supports my VOIP feature set.
  • You can’t do XYZ well or at all with any other vendor product.

I will say that there are a few use cases that keep you tied to a particular vendor that are listed above. However, what’s the harm at looking into something new or trying something different? We shouldn’t be afraid of learning new things.  I’ve done comparisons, demo’s, and have had to choose a different WAN router due to a lack of offerings before.  Management tools can become a difficult topic for discussion, but networking gear tends to stick with long time veteran SNMP for management, so that opens up possibilities.  Even Cumulus Networks, with all its automation buff, still provides legacy SNMP support. However, you wouldn’t want to use SNMP for automation with Cumulus as that would defeat their main business case of choosing their product.

So why not design around different use cases? If you need leaf/spine, don’t build the traditional access/distribution/core.  If you require a robust NAC solution, maybe Extreme Networks at the edge would be worth looking at instead of Cisco. If you require low latency, perhaps look into Arista or Mellanox.  If you’re in the service provider arena, Ciena may be worth looking into.

Being part of the networking field right now is a fun place to be. The variety of vendor selection creates some great competition and interesting niche feature sets.  Be sure to highly consider sticking with open standards if you travel down the “best of breed” road. Just remember to embrace openness and have fun. And yes, Cisco and Juniper have their places as well.

Extreme Networks – enabling a few things

Some of my most visited posts seem to be on brocade switching config, so I decided to put together our standard list of commands for some Extreme Networks switches we use. These commands can be used on the b5, c5, K series, 7100 series, and S Series Extreme Networks switches. Some commands are self explanatory, but for other’s I added a short description.

This command sets the vlan for the management ip set on the switch.

->set host vlan “vlanid”

->set ip address “ip address” mask “subnet mask” gateway “gateway ip”

We disable cdp on all edge ports without VOIP phones.

->set cdp state disable “port string”

We use the ciscodp command in order to set the tagged voice vlan on Cisco phones.

set ciscodp port vvid “voice vlan” “port string”

We manually configure a small set of vlans for each building, so gvrp isn’t necessary.

->set gvrp disable

->set igmpsnooping adminmode enable

->set igmpsnooping interfacemode “port string” enable

->set maclock enable

->set maclock enable “port string”

We limit the amount of mac’s that can be learned on the port and make it equal to the number of mac authentications we can do per port. Mac auth sessions are limited by switch model type.

->set maclock firstarrival “port string” “number”

->set macauthentication reauthentication enable “port string”

We set the maximum number of mac authentication sessions per port. This is limited based on switch model type.

->set multiauth port numusers 8 “port string”

You can do more than one number of port authentication types. By default we have mac auth, but you can also setup 802.1x auth as well. If you fail 802.1x auth, mac auth will be the next method of authentication.

->set multiauth precedence mac dot1x

->set port broadcast “port string” “pps threshold value”

We clear all the default snmp settings

->clear snmp access ro security-model v1

->clear snmp access ro security-model v2c

->clear snmp access public security-model v1

->clear snmp access public security-model v2c

->clear snmp community

->clear snmp group ro ro security-model v1

->clear snmp group ro ro security-model v2c

->clear snmp group public security-model v1

We setup every device with snmp v3 authentication.

->set snmp access public user “snmpusername” security-model usm

->set snmp user “snmpusername” authentication md5 “auth pass” encryption des privacy “priv pass”

->set snmp viewname All subtree 1

->set spantree spanguard enable

->set spantree adminedge “port string” true

->set ssh enable

->set telnet disable inbound

->set telnet disable outbound

->set webview disable

->set pot alias “port string” “alias”

This command enables POE on an edge port.

->set port inlinepower “port string” admin auto

->set port inlinepower “port string” admin off

Untagged vlan port setup.

->set port vlan “port string” “vlan id”

Tagged vlan port setup.

->set vlan egress “vlan id” “port string”

The one thing that could be better is the implementation of a command to apply the running config to the startup config. All these commands will be automatically applied and saved to the running configuration once entered.

Troubleshooting ARP/IGMP/Router CPU

We recently starting having issues with a building reporting that icmp stopped responding on a distribution router and some access switches behind the router. Some routing interfaces would respond, but the management VLAN interface wouldn’t. Further troubleshooting showed that the CPU processes on the router comprised of two Extreme Networks 7100 series switching running OSPF climbed up to 80/100% utilization. The “show logging buffer” revealed massive amounts of host-dos ARP attack events. The first thought was that a possible infected machine was creating an ARP storm. This would happen about twice a day for about a minute, but not at the exact same time. We tracked down the MAC addresses and removed the PC’s from the network. This didn’t seem to help, as another set of MAC’s addresses would show up in the logging buffer for host-dos ARP attack events the next day. We decided to start running a Wireshark packet capture. We could see the ARP storm along with some other IGMP traffic that would easily consume a Wireshark session, but we couldn’t identify the root cause.

After further investigation of the host-dos ARP logs, we noticed that the source interface should have been in STP blocking mode due to it being a redundant link to the access switch. My thought was that the massive ARP flooding could have been caused by a loop. Why would a loop occur? I then caught the CPU process table during the outage and I found that the IGMP process was consuming the router CPU. Could it be that the issue wasn’t due to an ARP storm, but the ARP storm was a secondary issue to something else going on? We decided to disable the redundant interfaces. This would take the possibility of a loop being created out of the picture. My thought was that the high CPU was causing dropped bpdu’s and the secondary link would go into forwarding on the access switch, but the router being CPU bound was still using the original link which caused the ARP storm/loop.

The issue continued with the redundant links disconnected, but now we weren’t seeing the host-dos ARP logs. Ok, so we knew we had high CPU utilization. We also knew it was the IGMP process. There was a slight traffic correlation on the routing interface before the CPU spike, so I enabled netflow on the upstream core router. Netflow started forwarding data to PRTG (Network monitoring utility). PRTG showed that the top talker was a newly built Landesk server. Now we were getting somewhere. Further research into Landesk revealed that the product uses multicast. My team decided to run another packet capture while booting up a lab and presto, the CPU started to spike on the router. The packet capture revealed a large number of multicast traffic classified to be used by Landesk. The multicast address was 239.83.100.109 along with UDP port destination of 33355 which was defined as Landesk “software distribution”. The flooding of multicast traffic seemed to be the culprit of the high router CPU utilization.

Happy troubleshooting,

@javi_isolis

Cisco Virtual Internet Routing Lab – Up and Running…

I was finally able to fix the issue that I described having in my earlier Cisco VIRL article. My original bare metal box only had 3 NICs. VIRL requires that you have at least 5 NICs. If you don’t have 5 NIC’s, then you have to modify the /etc/virl.ini file with dummy interfaces. I did this earlier, but must have had a mistake in the config. I double checked the config and also ran the VIRL-rehost script that’s on the desktop when you login to VIRL. Running the script wasn’t in the VIRL doc steps, so I didn’t do this before. Running the script after modifying the virl.ini file with the dummy interfaces finally fixed my issue. The script modified the /etc/network/interfaces with the correct dummy interfaces. Here’s an example of what the script changed:

iface dummy1 inet static

address 172.16.3.254/24

netmask 255.255.255.0

post-up ip link set dummy1 promisc on

I setup an IOSv router and connected it to an L2 External (flat) network which connects back through one of my physical NICs. That connection then goes into a real cisco ws-c3560x 24 port switch. That switch is connected to my network and I assigned my PC another IP address on the 172.16.1.x network. I can now ssh into the IOSv router directly from my desktop.

virl flat network

 

Now that I have worked out all the bugs, I’m pretty impressed with the functionality that VIRL provides. Now I’m going to see how many routers I can throw at this box.

Ping you later,

@javi_isolis

Cisco VIRL setup

In my last post, I spoke about getting a cisco virl (virtual internet routing lab) server going. I started with a hyper-V installation, which wasn’t listed as being supported. I gave it a try anyways. What I came to find out is that hyper-V would not work with my setup because I couldn’t do nested virtualization. Cisco Virl runs KVM under the hood which needs native VT-d. I couldn’t get hyper-v to pass VT-d to the host, so that was a no go. I decided to wipe the drive and load the iso version of cisco virl directly on my box. After a few failed attempts, I finally got virl to run without giving an error when trying to license the software. You have to follow the virl install guide to the T. I accidentally didn’t put in the correct hostname that the guide said to use. That caused the initial installation of virl to give an error every time I tried to run the user workspace manager. After following the exact directions, I was able to get virl to load.

Now I’m running into one last error message:

state changed from BUILD to ERROR with message: No valid host was found. Exceeded max scheduling attempts 3 for instance

cisco virl

I’m digging into the logs in order to figure out what the issue is, I’m really close. Hopefully my next post will be an in depth review of cisco virl.

occupied, time to setup Cisco VIRL

I’ve been occupied with many other things going on lately especially with the holidays. Family, work, etc. I haven’t been able to sit and write, but I ran into something pretty cool that I wanted to share. Those of you who are always looking to expand your knowledge in networking, cisco just released VIRL or virtual internet routing lab. This is Cisco’s newest virtualization lab simulation tool. It reasonably priced as well.

I saw this as a perfect opportunity to build a homebrew cisco VIRL hyper-v server. Unfortunately hyper-v isn’t supported, but that didn’t stop me. I recently found a dell studio 435 with an i7-920 processor in the garbage down the street. The only issue it had was a dying power supply.

cisco VIRL home server

Once that was fixed, I started to install hyper-v 2012. To my dismay, I couldn’t get hyper-v loaded without the installation crashing. All settings looked to be good. Virtualization was enabled in the bios and all other recommended settings were set. I then realized that the bios version was pretty old and figured I try to do an update. Low and behold a bios firmware fixed the crashing issue. I slapped another broadcom pci-e nic and finished the install. Now I’m having issues getting the broadcom nic and the onboard nic to load correctly. After searching the Internet, I figured out how to manually add drivers in hyper-v core mode. I’m still running into some issues with NIC detection, so I think I may run esxi on another partition of the drive just to get VIRL running. Stay tuned….

You can now follow up on my cisco bare metal home build here.

 

ICX6610 Brocade Switch – Firmware

Ok, so I’m in the process of configuring our spare icx 6610 brocade switch with the same firmware and configuration of one of our main border WAN icx 6610’s that’s running BGP. If our router fails, then I will have a hot spare ready to go while I wait for an RMA. This requires that the space icx 6610 is running router firmware along with an icx 6610 advanced license. The icx 6610 has the ability to run in layer 2 switching mode only or layer 3 mode when the proper firmware is loaded. By default, I had two code versions loaded on the icx 6610. You can identify which code is which by looking at the flash file name. Running the show flash command will give you the following:

#show flash
Stack unit 1:
Compressed Pri Code size = 7189206, Version:07.4.00cT7f3 (FCXR07400c.bin)
Compressed Sec Code size = 8874046, Version:08.0.00aT7f3 (FCXR08000a.bin)
Compressed Boot-Monitor Image size = 370733, Version:07.3.03T7f5
Code Flash Free Space = 49020928

You want to look at the character after FCX in parentheses, I have (FCXR07400c.bin) and (FCXR08000a.bin). The fourth character is an R, which is for routing. If the letter was an S, then that would represent switching only. Right now, I have two different routing versions. Since this is a major rev., there was also a new boot code as well (07.3.03T7f5). Unfortunately, there’s not two slots in the flash to have two different boot codes. If I try to boot back to the 7.4 code, I won’t have the proper boot code that I had before. I will therefore copy another version 8 code to the primary flash space. To do that I will run the following command:

#copy tftp flash <ip-of-tftp-server> <flash-image> primary

In order to boot to the new flash (assuming you don’t need a new boot code), you can then run the following command:

#boot system flash <primary | secondary>

HINT: The default boot is set to boot from the primary location, so if you reboot again without running another command, it will boot back to the default location. If you want to always boot from the secondary flash location, run the following command in config t mode:

(config)#boot system flash secondary

Don’t forget to write mem.

 

Cisco ASA 5505

asa-5505-front

I was recently asked by a friend to help out with a config for a cisco ASA 5505. An additional network and VLAN had been added on the ASA 5505 and we needed anyconnect VPN users to be able to access devices on that new network. I’m used to working with checkpoint firewalls, but most firewalls function in a similar fashion. I first logged onto the ASA using the ASDM gui. I checked the firewall rules and access looked to be permitted between the vpn network and the new network. I was a little stumped because it seemed that I needed a little more than just a firewall rule to allow both networks to communicate. I then consoled into the ASA and looked at the routing table. All looked good there. The ASA management interface was able to communicate with devices on both networks, so something else was missing. I then looked at the log files in the ASDM gui and was receiving the following message when trying to rdp into a server after connecting to the VPN:

Asymmetric NAT rules matched for forward and reverse denied due to NAT reverse path failure. An attempt to connect to a mapped host using its actual address was rejected.

I searched cisco for a solution and found a few vague possible solutions. I then pinged Network guru Shane to see if he could point me in the right direction. He hinted that I might have needed a “no nat statement”. Bingo, that worked. Here’s the command that I needed:

Nat(vlan1) 0 access-list vlan1_nat0_outbound

In the ASDM gui, its called a nat exempt statement. This rule makes sure that the internal traffic from the new network going to the VPN network would not get nat’d on its way out.

Since then, I opened up my safari books account back up for $9.99 a month which includes 5 book slots a month. The first thing I checked out was a Cisco Press ASA book. Safari books is a great resource for low cost e-learning. I highly recommend you give it a try. They don’t really advertise the $9.99 subscription, but if you sign up for a free trial, you can still choose the $9.99 pricing. I’m not 100 percent sure if they are still offering this to new users, but its worth checking out. As a network administrator, its important to know the fundamentals. Don’t get stuck in the, “I only know cisco or juniper” mode. If you know the fundamentals, then learning a new syntax and a few different config parameters is not the end of the world. After all you don’t want to limit yourself by putting a vendor name in front of your title, i.e. Cisco Network Administrator or Brocade Network Administrator. Be (emphasis added) the “Network Administrator”.

 

asa-5505-back

Enterasys 802.3ad link aggregation

Since we saved some cash by purchasing more 2×2 3705i Enterasys AP’s instead of 3×3 AP’s during our AP upgrades, we were able to purchase a few other items. We picked up three c5g Enterasys 48 port POE switches, gbics, and a few other parts. The first thing I did after we deployed 96 AP’s in our dorm rooms was setup 802.3ad link aggregation with my extra gbic’s from our current N3 chassis to our g3 series switch. Phase two will be to install another DFE blade in the N3 chassis and spread the link aggregation between two DFE blades.

On with the Enterasys Extreme Networks switch commands:

1.) Egress all the proper vlan’s you want trunked across the additional physical port. We will be setting up a lag.0.x port and at that point, the physical port egress no longer matters, but if the lag breaks down for some reason, then the physical port will have the correct vlan’s trunked. You could also ensure that the single port lag command is set, but again I like to have the extra safety precautions in place.

Example ->set vlan egress ->set vlan egress 20 ge.4.24

2.) Egress all the proper vlan’s on the lag port. Use the “show lacp” command to view the available lags and to make sure that lacp is globally enabled.

Example ->set vlan egress
->set vlan egress 20 lag.0.1

3.) Create a unique lacp admin key to statically set which ports will be joined to the lag

Example ->set lacp aadminkey
->set lacp aadminkey lag.0.1 20

4.) Set the aadminkey to the physical port

->set port lacp port ge.4.24 aadminkey 20

5.) Perform the same commands on the other switch that you will be connecting to. The aadminkey can be different on the other switch, but I like to try and use the same admin key on the opposite end if possible. Make sure you also have lacp enabled on the physical interfaces as well.

-> show port lacp port ge.4.24 status detail
wpid-IMG_20140109_143458643.jpg

Losing control

As server administrators continue migrating to virtualization, network admins lose control. I’m not talking about psychological control, but network resource and management control. Server admins probably feel a sense of freedom. They are probably saying, “Now I don’t have to go and bother those pesky network admins to fire up a new server.” This can decrease the provision time, but this can also cause a very adverse side effect. See, I’m a network administrator and I work with networks all day long. From time to time I dabble in ESX and I also manage and maintain a few Linux and windows servers. However, I’m by no means up to the task of daily server administration. I’m sure I can learn how to administer AD, mail, file shares, and print servers, but that’s not what I do on a daily basis. The same holds true for a server admin. I’m not saying they can’t figure out networking or do the basics, they just don’t do networking every day.

What that means, is that from time to time you end up with virtual switches not configured or optimized properly. Firewall rules are bypassed by server admins with ease. QOS settings are not configured properly. You get the point. You thought the BYOD network was bad, well the wild, wild, west has just infiltrated your server network infrastructure as well. You now have BYOS (bring your own server). How secure are those prebuilt OVA’s? Who really knows?

With all these thoughts and ideas in mind, what are the available options? I have currently been researching how we can regain control within these VM environments. Our current vendor Enterasys, now Extreme Networks provides a method to mac auth all devices seen on the switch port or lag that goes through a VM environment. This allows identification of VM’s with their NAC solution. The Enterasys switch can then apply dynamic policies to each frame coming across the switch port or lag. The default number of polices we can apply at one time is 8 on their S series switch. We would need a license to do 128 per port. Now maybe this is not the best strategy, but it’s one that I know of that can help. You can then create a default policy which blocks whatever you want based on rules up to L4. The server admin would then have to reach out to those good old network admins for correct policy enforcement. Enterasys even has a data center manager esx plugin that can be used to ease management. Now I don’t believe that this is the best solution for all environments, as it has downsides as well. MAC spoofing is one that comes to mind and this setup doesn’t come without cost.

Therefore, the next solution I’m looking into is open vswitch. This would act as a front end add-on piece in ESX as I understand. Other hypervisors already use open vswitch. Using openflow to control traffic qos/policy could be another avenue to maintain network harmony. I will continue my research and will post my findings….

1 2