I finally made it out to a CHI-NOG event, the Chicago network operators group. Experienced network engineers and architects put the group together to focus on all things network related. The yearly events concentrate on vendor-neutral topics and encourage other network enthusiasts to attend within the Chicago land region. This year’s gathering had more than a dozen sessions and a lineup with some excellent guest speakers. If you’re ever in the area and love networking with technology and people, I highly recommend you go. I attended quite a few of the sessions, but I’ll start with one of my favorites.
Rethinking BGP in the Data Center
Presented by Russ White
BGP the chosen EGP of the Internet has taken quite a hold in large-scale data centers across companies such as Facebook, Microsoft, LinkedIn, and Google. You can do all kinds of clever traffic engineering using BGP, but should it be the chosen IGP for data centers? The companies mentioned above are now looking into or are already deploying other technologies such as openR, openfabric, and firepath as a BGP replacement. Russ challenged BGP deployment complexity and talked about some of the most significant hurdles being delay and jitter within the hyperscale arena. Flooding also becomes an issue along with autoconfiguration of devices.
I think it’s important not to try and over complicate existing protocols to make them fit what we want. We need to become better engineers and try something different. That’s where white box switching and new protocols such as draft-white-openfabric come into play. White box allows for the deployment of newly developed routing protocols that are more appropriate for what we wish to accomplish. Automation is also critical for successful manageability. Russ talked about having a router or switch that you never have to configure or CLI into, a little tough to swallow for us network operators.
I couldn’t help but think about wireless controllers. When’s the last time you ever ssh’d into your wireless access points? We couldn’t imagine going back to individually configuring access points, what a nightmare! Centralized automated management for our switches and routers makes complete sense. Are we ready for the transition? The thought of what will happen to our existing jobs always comes up. However, I say we can then transition into working on solving other problems that we never had time to complete.
All the hype
White box switching seems to be all the networking hype. For some in-depth research, check out this podcast from packet pushers about ATT making its move into white box switching. Cisco is also committed to offering a decoupled version of IOS-XR from Cisco hardware to enable running their NOS on OCP (open compute project) compliant hardware aka “white box switching.” Fascinating stuff, but what’s the big deal? Well, I’m going to try and make a comparison.
A Lego comparison
I’m a huge adult fan of Lego (AFOL). I remember dumping old tin popcorn bins with Legos all over my bedroom floor as a child. I’m more organized today, but I can’t help tearing down and building new creations. Now imagine you have an advanced Lego technic set put together. You have gears that move, hinges that open and close, wheels turn, etc. Now imagine all those connecting pieces glued together. A nightmare for those AFOL’s who want to rebuild something special.
Picture that glued together Lego set as a networking switch or router. Sure you can plug and unplug a few items, configure features within the CLI, and even get some sweet stats via SNMP. However, your switch or router’s underlying code is static which you can’t change. You’re at the mercy of the vendors nicely glued together product. I’m not suggesting that’s necessarily a bad thing, but you get where I’m going. With white box switching, you finally get to be a bit more creative with your switch or router. You can unload the default network operating system and load up something completely different. You’ve just expanded your imagination beyond one vendor and their fixed code.
A modular future
Maybe we’ll start to see advanced hardware modularity for white box switching as well. You need more processing power; upgrade your CPU. You need more space for your NOS apps or massively large routing tables, then go ahead and add more RAM. Are you a Cisco or Cumulus fan, who cares, you choose what NOS to run. Now you’re building like an AFOL. The possibilities of customization that deliver high flexibility are endless.
EXOS in Nutanix CE
Now that I have my Nutanix CE lab setup, I wanted to get some of my virtual network operating systems installed within my home lab. One of the NOS’s I’ve been running is Extreme Networks virtual EXOS. My last EXOS-VM lived in Virtualbox and ESXi. Extreme Networks has a github page here with all the information you need to get started with running the VM within a Virtualbox or ESXi environment.
Issue and Solution with Nutanix
Following the EXOS installation guide using the downloadable iso and mimicking the Vmware/Virtualbox VM settings within Nutanix CE wouldn’t work. I kept receiving an issue with the disk not correctly detected while Continue reading »
Sorting Out HCI
Today’s Hyper-converged infrastructure (HCI) vendors have some exciting product offerings. HCI ultimately provides scalable and flexible storage along with coupling computing resources. Recently at work, our KPI metrics started to show that some of our SAN hardware was having issues keeping up with production workload. So we ended up looking at a few HCI vendors; Simplivity, Nutanix, Pivot 3, and Vmware VSAN. After our initial investigations, it became clear that we weren’t quite ready to step into HCI just yet. Our project scope explicitly called for storage performance. At the current junction, additional compute wasn’t necessary and not budgeted for the project. Since HCI solutions couple storage with computing costs, we ending up investing in an all-flash SAN solution.
However, I was very intrigued with the different HCI platforms. What interested me the most was the ability to scale storage using x86 based systems. During our Nutanix research, I came across their community edition. I decided to load Nutanix CE on my home virtualization server and give it a whirl. There are lots of other great sites with information on how to get the initial setup going, so I’ll focus more on some of my specific findings during my home lab testing.
My previous IT roles have revolved around the administration of different technologies, specifically networking technologies. However, I’ve always had the willingness to perform any other job functions as needed. That’s lead me to learn all types of new things such as tower climbing, billing, phone support, inventory tracking, training, and the list goes on. At my current employment, I started as a network administrator. I moved into a network supervisor position within three years, then was asked to serve as an interim supervisor for another area through a merger. I’m now the supervisor of networking and infrastructure.
Transitioning from a network administrator to a supervisor isn’t always a breeze. When you’ve spent lots of time administering systems, you become ingrained into build, Continue reading »
I recently heard the term “best of breed” used when discussing network vendor selection. I was surprised by this answer because you don’t hear it too often. The more I thought about it, why not “best of breed” selection? My time as a network and infrastructure supervisor has taught me that a data center environment can be full of different compute and storage vendor products. Our SAN environment consists of Pure, Tegile, EMC, and even QNAP. Each product has its place. Pure serves the VDI environment, Tegile/EMC host production, and QNAP serves as a target for our Veeam backups. The team has also categorized and carved out each platform into tiered offerings.
On the other hand, network vendor selection tends to be biased. Typically you’ll see one network vendor selected for the edge/access, distribution, and core. However, you will find a different wireless vendor from time to time.
Many reasons exist
A compilation of the most popular
- We would like to interact with only one vendor for purchases and support.
- ABC vendor only works well with a particular management tool.
- I only know vendor ABC, and we don’t have time to learn something new.
- Did you hear that vendor ABC had an issue with XYZ product, I don’t want those problems.
- Everyone else uses vendor ABC.
- No other vendor supports my VOIP feature set.
- You can’t do XYZ well or at all with any other vendor product.
I will say that there are a few use cases that keep you tied Continue reading »
Being able to track your work efficiently is a very useful skill. For years I completed my work but rarely tracked my work in a project management professional (PMP) sort of way. Sure, I’ve done the weekly reports, sticky notes, Outlook tasks, and Outlook calendar block scheduling, which are all useful. However, simple project management skills help create a consistent and straightforward approach to managing time, resources, and tasks. I’ve seen organizations take an all guns blazing PM approach to a nothing at all approach. Sometimes you’ll see IT subject matter experts resist PM due to the “I’m too busy” or “It takes too much time” statements, but in reality, basic fundamental project management is not that difficult.
Here’s an example of how you can model behavior and start to implement some basic PM skills. I recently had a team reach out to discuss a new project that would require infrastructure resources. The team pulled up a draft diagram, and we began our dialog. I started to ask peering questions, and the diagram began to transform. Once I was comfortable with understanding what we were trying to accomplish, I shared my screen with the team and opened up OneNote. I began typing each major task that needed to be completed and here’s Continue reading »
We recently upgraded some of our WAN link bandwidth capacity from 1Gbps to 10Gbps to decrease transfer rates of backups across our two data centers. Traffic between each site encrypts with Checkpoint physical open appliances. The upgrade to the WAN links involved installing 10Gbps Intel NICs in our Checkpoint open servers. Once all the pieces were in place, I started to test everything using iperf3.
My initial iperf3 TCP results showed a maximum capacity of around 650Mbps. Something seemed to be limiting my ability to push more traffic across data centers. I started looking at the primary site Checkpoint VPN open server gateway. Using top in expert mode, I found that Continue reading »
Extreme Networks Switching Commands
Some of my most visited posts seem to be on brocade switching configuration/commands, so I decided to put together our standard list of commands for some Extreme Networks switches we use. These commands can be used on the B5, C5, K series, 7100 series, and S Series Extreme Networks switches. These switches run the EOS network operating system. Extreme networks product line moving forward will be purely EXOS (ExtremeXOS operating system). Therefore the following commands will become legacy, but are still very useful to know since some of the EOS product line hasn’t reached EOL. Some commands are self explanatory, but for other’s I added a short description. Continue reading »
We recently starting having issues with a building reporting that icmp stopped responding on a distribution router and some access switches behind the router. Some routing interfaces would respond, but the management VLAN interface wouldn’t. Further troubleshooting showed that the CPU processes on the router comprised of two Extreme Networks 7100 series switching running OSPF climbed up to 80/100% utilization. The “show logging buffer” revealed massive amounts of host-dos ARP attack events. The first thought was that a possible infected machine was creating an ARP storm. Continue reading »