Latest Posts

Extreme Networks – enabling a few things

Some of my most visited posts seem to be on brocade switching config, so I decided to put together our standard list of commands for some Extreme Networks switches we use. These commands can be used on the b5, c5, K series, 7100 series, and S Series Extreme Networks switches. Some commands are self explanatory, but for other’s I added a short description.

This command sets the vlan for the management ip set on the switch.

->set host vlan “vlanid”

->set ip address “ip address” mask “subnet mask” gateway “gateway ip”

We disable cdp on all edge ports without VOIP phones.

->set cdp state disable “port string”

We use the ciscodp command in order to set the tagged voice vlan on Cisco phones.

set ciscodp port vvid “voice vlan” “port string”

We manually configure a small set of vlans for each building, so gvrp isn’t necessary.

->set gvrp disable

->set igmpsnooping adminmode enable

->set igmpsnooping interfacemode “port string” enable

->set maclock enable

->set maclock enable “port string”

We limit the amount of mac’s that can be learned on the port and make it equal to the number of mac authentications we can do per port. Mac auth sessions are limited by switch model type.

->set maclock firstarrival “port string” “number”

->set macauthentication reauthentication enable “port string”

We set the maximum number of mac authentication sessions per port. This is limited based on switch model type.

->set multiauth port numusers 8 “port string”

You can do more than one number of port authentication types. By default we have mac auth, but you can also setup 802.1x auth as well. If you fail 802.1x auth, mac auth will be the next method of authentication.

->set multiauth precedence mac dot1x

->set port broadcast “port string” “pps threshold value”

We clear all the default snmp settings

->clear snmp access ro security-model v1

->clear snmp access ro security-model v2c

->clear snmp access public security-model v1

->clear snmp access public security-model v2c

->clear snmp community

->clear snmp group ro ro security-model v1

->clear snmp group ro ro security-model v2c

->clear snmp group public security-model v1

We setup every device with snmp v3 authentication.

->set snmp access public user “snmpusername” security-model usm

->set snmp user “snmpusername” authentication md5 “auth pass” encryption des privacy “priv pass”

->set snmp viewname All subtree 1

->set spantree spanguard enable

->set spantree adminedge “port string” true

->set ssh enable

->set telnet disable inbound

->set telnet disable outbound

->set webview disable

->set pot alias “port string” “alias”

This command enables POE on an edge port.

->set port inlinepower “port string” admin auto

->set port inlinepower “port string” admin off

Untagged vlan port setup.

->set port vlan “port string” “vlan id”

Tagged vlan port setup.

->set vlan egress “vlan id” “port string”

The one thing that could be better is the implementation of a command to apply the running config to the startup config. All these commands will be automatically applied and saved to the running configuration once entered.

Continue reading »

Troubleshooting ARP/IGMP/Router CPU

We recently starting having issues with a building reporting that icmp stopped responding on a distribution router and some access switches behind the router. Some routing interfaces would respond, but the management VLAN interface wouldn’t. Further troubleshooting showed that the CPU processes on the router comprised of two Extreme Networks 7100 series switching running OSPF climbed up to 80/100% utilization. The “show logging buffer” revealed massive amounts of host-dos ARP attack events. The first thought was that a possible infected machine was creating an ARP storm. This would happen about twice a day for about a minute, but not at the exact same time. We tracked down the MAC addresses and removed the PC’s from the network. This didn’t seem to help, as another set of MAC’s addresses would show up in the logging buffer for host-dos ARP attack events the next day. We decided to start running a Wireshark packet capture. We could see the ARP storm along with some other IGMP traffic that would easily consume a Wireshark session, but we couldn’t identify the root cause.

After further investigation of the host-dos ARP logs, we noticed that the source interface should have been in STP blocking mode due to it being a redundant link to the access switch. My thought was that the massive ARP flooding could have been caused by a loop. Why would a loop occur? I then caught the CPU process table during the outage and I found that the IGMP process was consuming the router CPU. Could it be that the issue wasn’t due to an ARP storm, but the ARP storm was a secondary issue to something else going on? We decided to disable the redundant interfaces. This would take the possibility of a loop being created out of the picture. My thought was that the high CPU was causing dropped bpdu’s and the secondary link would go into forwarding on the access switch, but the router being CPU bound was still using the original link which caused the ARP storm/loop.

The issue continued with the redundant links disconnected, but now we weren’t seeing the host-dos ARP logs. Ok, so we knew we had high CPU utilization. We also knew it was the IGMP process. There was a slight traffic correlation on the routing interface before the CPU spike, so I enabled netflow on the upstream core router. Netflow started forwarding data to PRTG (Network monitoring utility). PRTG showed that the top talker was a newly built Landesk server. Now we were getting somewhere. Further research into Landesk revealed that the product uses multicast. My team decided to run another packet capture while booting up a lab and presto, the CPU started to spike on the router. The packet capture revealed a large number of multicast traffic classified to be used by Landesk. The multicast address was 239.83.100.109 along with UDP port destination of 33355 which was defined as Landesk “software distribution”. The flooding of multicast traffic seemed to be the culprit of the high router CPU utilization.

Happy troubleshooting,

@javi_isolis

Continue reading »

Some PHP snmp scripting

I was digging through some of my old notes and came across a few networking PHP scripts that I put together for some Proxim AP-4000 access points. I put this script and many others together to help manage these standalone access points before there were wireless controllers. This particular PHP script sets up a while loop to modify some snmp values to modify AP filters. The snmp values within this script can be modified to be used in changing other values as well. Your setup will require PHP installed along with the snmp package. Have fun.

<head>
<title>
AP-4000 Filter modification Script
</title>
</head>

<body>

<?php
//set the variable that will be the start number of the third octet within your IP range
$ip = 100;
//set your snmp RW password
$snmpRwPass = yoursnmppassword;

//setup the loop that will snmpset each AP mgmt IP address defined starting at your IP variable and completing before your max value
while ( $ip <= 111) {

//modifies snmp values of Proxim AP-4000 filters
snmpset(“192.168.1.$ip”,”$snmpRwPass”,”.1.3.6.1.4.1.11898.2.1.5.5.3.1.6.1″,”i”,”1″, “10”);

snmpset(“192.168.1.$ip”,”$snmpRwPass”,”.1.3.6.1.4.1.11898.2.1.5.5.3.1.6.2″,”i”,”1″, “10”);

snmpset(“192.168.1.$ip”,”$snmpRwPass”,”.1.3.6.1.4.1.11898.2.1.5.5.3.1.6.3″,”i”,”1″, “10”);

snmpset(“192.168.1.$ip”,”$snmpRwPass”,”.1.3.6.1.4.1.11898.2.1.5.5.1.0″, “i”,”1″, “10”);

//print output of each AP mgmt IP thats completed
echo “done with 192.168.1.”.$ip;
echo “<br>”;

//counter for increasing AP mgmt IP
$ip++;

}
?>
</body>
</html>

 

Continue reading »

Portable home lab virtualization server + gaming

I have a few PC’s that I use for testing, gaming, and other side projects. I wanted to pare down on a few systems, so I started looking into a portable VM home lab server setup that could potentially be used for testing at least four different VM’s and also allow for some decent gaming performance utilizing VM hardware GPU passthrough.

I first pondered on the Intel NUC Skull Canyon. It’s pretty portable, tough looking, and powerful, but it lacked the ability to easily install an external GPU and the ability to easily install a hypervisor when it first launched. It’s also pretty expensive and I was trying to stay around the $500-$600 range. I started looking at a few mini ITX cases and remembered coming across the ASRock M8 Mini ITX design in the past.

asrockm8portableserver

asrock m8 side view

I ended up finding the case, but it was a barebones only bundle that already included an older motherboard that didn’t support the type of passthrough that I was looking to utilize. The case reminded me of the old G4/5 Apple cases with the handles on each corner of the case. The handles on the corners make it a lot easier to carry.  I ended finding a discount open box from Newegg that made the purchase a little more bearable.

Here’s the home lab setup specifications and costs:

Case: ASRock M8 Mini ITX (included LGA 1150 mobo w/pwr supply) open box $186.69
Memory: G.SKILL Aegis 16GB (2 x 8GB) 288-Pin DDR4 SDRAM DDR4 2133 $52.42
CPU: Intel Xeon E3-1225 v5 SkyLake 3.3 GHz 8MB L3 Cache LGA 1151 $234.99
Motherboard: ASRock Server/Workstation MB-C236WSI $213.00

Total

$687.10

I ditched the 1150 motherboard and installed the Intel c236 chipset based LGA 1151 motherboard along with a Xeon Skylake based processor. My first attempt at installing Vmware exsi 6.0 was a failed attempt due to my inability to get anything working with passthrough. I tried multiple versions of Esxi, but I still couldn’t get my ATI 6770, usb, or sound passthrough working. I tried a few other graphics cards, but without sound, I threw the Esxi hypervisor out of the picture. I then decided to try an installation of Xenserver 7 and to my surprise, I was able to pass through all of my components. I did have to manually run some commands to get things going, but in the end I ended up with a VM that could possibly do some decent gaming.

ASrock MB-C236WSI asrock m8 with MB-C236WSI skylake xeon motherboardasrock m8 rear with MB-C236WSI skylake xeon motherboard

In order to get all the passthrough devices working, you may have to do some work within the CLI. I didn’t have to worry about setting up the GPU within the CLI as it was already listed as a passthrough device in the Xenserver GUI management interface.

In order to add other devices, first find your VM UUID once provisioned through the GUI manager. Then run the following command within the Xenserver CLI interface:

lspci -k | more

Find your pci devices. I was specifically looking for USB and sound. If you want to add multiple passthrough devices, you will have to run the next command once with both your pci devices listed within the command along with your VM UUID

xe vm-param-set other-config:pci=0/000:00:1f.3,0/000:00:14.0 uuid=a4f084ae-e8cf-144a-ac31-7bf456e333b5

Continue reading »

Why NetDevOps/NetOps will become important for Network Admins

Being a network administrator/engineer typically requires typing in ssh consoles to get things going. At some point, being able to automate tasks or being able to manipulate configurations based on a certain outcome will become necessary. I’ve gathered a few thoughts on real world views to network automation. The buzzword floating around for this topic is NetDevOps.

NetOps/NetDevOps(my definition): Network automation using code to run commands that would normally have to be typed in manually into each device. Example: Run code that can parse or write through configs, logs, and snmp values in order to take action on a specific outcome.

I won’t go into the details of the ins and outs of NetOps/NetDevOps and how to get started with coding. I’ve provided a list of links with information that other really smart people came up with.

Detailed Definitions:
https://cumulusnetworks.com/blog/netdevops-networking-methods-with-a-devops-mindset/
http://packetpushers.net/pull-my-strings-im-your-puppet-juniper-bringing-devops-to-networking/
Some examples:
https://www.nanog.org/sites/default/files/Carr_What_S_Netdevops_Why.pdf
Getting Started with coding:
https://www.nanog.org/sites/default/files/Swafford_Netops_Coding_101.pdf

Ok, now what can NetDevOps actually do for you network administrators out there? I started to create a list of items that NetDevOps could put a dent in. I don’t feel that you require a Google or Facebook sized infrastructure to take advantage of NetDevOps. My team and I currently manage around 120 switching/routing devices and we’re headed to add lots more. That’s no Google, so here’s my list:

  • Changing network admin passwords for devices when staffing changes can be very time consuming. Running some code that can SSH into switches and routers to update passwords and privileges  could be a very useful feature to have.
  • If you have routers that maintain the same ACLs or route maps, having to make changes can be a daunting task even if it’s only a dozen routers. Using code to automatically upload new changes to duplicate ACL’s and route maps across your routers will reduce time and human input errors.
  • I also have a few ideas about gathering wireless user data and plotting details on a google map to indicate AP’s that have a large volume of user connectivity. The map would provide visual information in real time that can help determine if you’re having a sticky client situation. You could also make some automated config changes to your AP’s power levels or with minimum basic rates to help the situation out.
  • You could manage bandwidth available across network links and trigger an automatic response to apply route maps to redirect traffic or apply QOS rules to your programming hearts content.

I’m sure there’s lots of other examples out there, please post others that you may have. I know this last one is heading down the openflow rabbit hole, but hey if you could do these types of things with your current equipment using a NetDevOps approach, why not?

Continue reading »

What does a Network Administrator do?

I wanted to share what a network administrator’s daily job duties, functions, and tasks may entail on a daily basis. For those new out there to the realm of IT, a network administrator typically interacts with the hardware/software components that transfer data to and from devices over a physical distance through some type of medium. Some of these devices include: personal computers, laptops, tablets, servers, switches, routers, firewalls, load balancers, wireless access points, and any other devices that rely on transmitting data. The components that are typically managed daily by a network administrator are switches, routers, wireless access points, DHCP/DNS servers, IP address provisioning, documenting/diagraming the network, monitoring bandwidth usage, and maintaining copper/fiber cable plants.

What does a network administrator do daily

Network Admin replacing Enterasys E1 switches.

The scope of what network administrators manage on a day to day basis typically depends on the IT organizational business structure. Organizations that are small or depend on a small IT work force may expect the network administrator to handle more than some of the items listed above. You may be the person who also manages/deploys servers, email systems, desktops, and a host of other things. This typically gives you a plethora of never ending projects that will keep you busy, which will hopefully make you a master of all disciplines. I don’t necessarily see this as a bad thing, but you may not have enough time in the day to architect the network the way you want it or in the best way it could be.

organized cat5 patch cords

Completed fresh installation of Extreme Networks C5k stack.

Some organizations may lump other responsibilities onto the Network Administrator. Telecom and networking is a combination I have seen before. Expect to work with telephony devices and services like VoIP, mass messaging, and voicemail services. Another combination is security and networking which would entail working on the network along with security devices/appliances and coming up with strategies on protecting data. Some networking devices have security features and functions embedded into them such as firewall/routing devices that perform IPS (Intrusion prevention system), IDS (intrusion detection system), spam filtering, and URL filtering/blocking which make networking and security devices easier to manage. I’m not promoting that organizations follow these models, but do expect to see some businesses operate in this fashion.

Larger IT organizations sometimes have the network administrator work specifically with networking devices/appliances. In this environment, you will have the opportunity to thrive by sharpening your skill set into becoming a master of networking. Mostly everything touches a network in today’s world, so always expect to be able to troubleshoot issues that can help other areas solve problems. Your mission should you choose to accept it, is to keep the network up and running efficiently. Just don’t forget to have fun and follow your passion. If you also have thoughts about obtaining certificates,check out this article by network guru Shane Killen, or check out my article on certificates. Please feel free to add other responsibilities that you may have heard that a network administrator does on a daily basis in the comments below.

Continue reading »

Checkpoint VPN MEP by default…

I started having issues that required the use of deploying another checkpoint VPN gateway. My team setup the new VM, installed Checkpoint Gaia, and completed the configuration for VPN. I created a new site in my windows checkpoint endpoint security client that pointed to the new DNS entry and off I went. I started to have issues being able to connect to the new VPN gateway after a few days, so I enabled logging in the checkpoint endpoint client. I discovered that my client was trying to connect to one of my original VPN gateways even though I didn’t have the original gateway defined in the VPN client. After a quick call to support, we found out that MEP (multiple entry point) was enabled by default on checkpoint VPN gateway’s that used the same encryption domain. I had to disable MEP, but couldn’t find any settings in the GUI.  The following KB article gives directions on how to disable MEP:

https://supportcenter.checkpoint.com/supportcenter/portal?eventSubmit_doGoviewsolutiondetails=&solutionid=sk78180

MEP wasn’t the desired configuration, but I could see its benefit of being enabled for a redundant VPN gateway setup. I may enable MEP in the future. Only time will tell.

Continue reading »

F5 Forwarding IP VS fun…

I have been recently spending lots of time with our F5 Big IP 2000s. We have been working on deploying a new private network behind the F5 with nodes that admin’s would like to directly access from their secure admin workstations. Our current setup has the nodes behind the load balancers using basic Virtual servers that forward the traffic from external routable IP’s to internal non-routable IP’s. Therefore I would need to create multiple VS’s per each node that admins wanted to access. That would be a lot of VS’s.

There are a few other clever ways to get around this, well maybe not necessarily clever. The first is using a jump server. The admins would access one VS that would forward to one pool member with one node in it. Then they could then access the other private nodes from this “jump” server. The other option would be adding another NIC to the admin workstations and put that NIC on the same VLAN that the private nodes sit on. Both of these are not the greatest ideas.

I therefore convened with an F5 tech guru and passed this idea by him. Could I have a router with a routed interface within the F5 private VLAN that has the F5 private nodes? I could then take the private non-routable network and make it routable protected by an ACL on the router. The nodes would still point to the F5 for the default gateway. When an admin workstation would communicate to the node, it would send the traffic through the router; the router would then forward packets to the node. The issue then lies with the node sending the traffic back to the F5 because it’s the default gateway and that creates issues. I found out from the tech that there’s a way to get this to work by using an IP forwarding VS that listens on the F5 private VLAN.

You will first need to make sure that your current nodes are not in an SNAT, as SNATs along with an IP forwarding VS configured on the same interface don’t work, as the SNAT listens over the IP forwarding VS. Within the VS, the source network would be the private node network and the destination would be the network where the secure admin workstations sit, which would be accessible across the router that I placed on the private node VLAN. Now, in order to get the F5 to forward to the router on the private node VLAN instead of using its routing table, you have to create an iRule as the Forwarding (IP) resource within the VS. Here’s the iRule syntax:

when CLIENT_ACCEPTED {

if { [IP::addr [IP::client_addr] equals 10.1.1.0/24] }   {

nexthop internalvlannum 10.1.1.254

}

}

The 10.1.1.0/24 is the internal private node network that sits on the F5. The 10.1.1.254 address is the gateway of the router that’s sitting on the F5 private node network. This iRule forwards the traffic to the router instead of using the F5 to forward the traffic. You also have to assign a protocol profile (client)  Fast L4 profile to the forwarding (IP) VS as well with Loose initiate and loose close enabled to allow the tcp packets to flow correctly.

Continue reading »

What happens in Mexico, saves Mexico

I just had an opportunity to go on my first mission’s trip to Playa Del Carmen in Mexico for a week with seven members of the church I attend. We were invited and led by Mission Explosion International. We planned to visit and help out a local Church of Christ in Playa Del Carmen. Our goals consisted of going out in the neighborhood to promote a clinic check that would be hosted in the local Church building. The clinic sessions included checking blood pressure and sugar levels along with hosting VBS (vacation bible school) days as well. We also planned to hand out Spanish Mission soccer balls.

mexico-mission-3

We ended up doing 3 days of clinic work, 2 days of children outreach, one day of men’s and women’s bible study, and our Evangelical guide preached on Sunday. The trip was amazing. We had lots of fellowship, outreach, teaching, and preaching going on all lead by God and His spirit. If you ever have an opportunity to go on a mission’s trip, please pray diligently and seek God’s guidance in the matter. I found out that I was way more capable through Christ than I ever imagined. In the end, the culture, language, and the income levels may be different, but the people are no different than us in the fact that they need God and Jesus Christ just like we do.

mexico-mission-2

Thank you to Rusty, Audrey, Jordan, and Kylee!!! I will never forget my first mission’s trip to Mexico.

@javi_isolis

Continue reading »

Cisco UC on UCS

One of our latest projects at work is moving from centrex to Cisco Unified Communications (UC). I was assigned to complete quite a few tasks for this project. One of the first tasks was getting our two UCS c240 rack mount servers going. Our purchase was part of a larger order, so our UCS rack mount servers are bare metal servers. No esxi or UC components were pre-installed. I originally wanted to install esxi on the cisco flexflash SD card, but then found out that UC on UCS doesn’t recommend that configuration. I also found out that Cisco recommends that 2 RAID 5 arrays should be created across the 16 drives that we have in our c240’s.

ucs 2u c240 server

If you want to stay on a Cisco tested reference configuration (TRC), then I would recommend that you check out this Cisco page for reference:

http://docwiki.cisco.com/wiki/UC_Virtualization_Supported_Hardware

Continue reading »
1 2 3 5