Tracking down problem devices on your network

If you've been a network administrator for very long, you've probably had to deal with a situation where you had to track down a problem user or machine.

Here's how that might have gone:

A situation emerged where you had to shut down a user on the network, and it needs to happen as soon as possible. It could be that your firewall logs caught her trying to hack into the HR server in the co-location facility; it might be that he caught a virus or worm that is causing network havoc; it might just be that the dirty SOB took your favorite parking spot and you're feeling froggy. Whichever the case, the task at hand is to locate them on the network and shut down their connectivity.

First, you'll need to figure out their IP address or conversely, if you have their IP address, you'll have to correlate that to a user. I don't know about you, but I prefer to know who I'm shutting down before I flip the switch. Since most users connect via DHCP, this will take a bit of work. Once you've found their IP address, you'll need to search thru the ARP tables on your routers (or layer 3 switches) to figure out which VLAN or subnet they're on and tie that IP address back to a MAC address. Then, you'll need to search through the CAM or MAC tables to find which port they're connected to and, finally, you can shutdown that port. As Colonel Ryan said on my favorite show, The Unit, "That's one working day that the government just took from me that I ain't ever gonna get back."

User tracking, or device tracking, is a specialized skill and requires constant updating as our LAN environments are constantly evolving. VLANs and VLAN trunking made this complicated. Wireless, 802.11x, networks made it even harder, and virtualization has thrown us a monkey wrench that we're still trying to figure out. 

As a best practice, we should all be keeping up-to-date network diagrams that detail location of user groups by department and by subnet. This will help you out in many types of troubleshooting scenarios besides this one, so stop making excuses and just do it.

In addition to keeping accurate and up-to-date documentation, I suggest investing in a tool that does automated user tracking. In many situations you'll need to not only track down users in real-time, but you may have scenarios that require you to track this information historically and to alert on future identification of these users on the network. This is practically impossible to do without an off-the-shelf tool unless you're really, really good at writing Perl scripts and you have a lot of free time on your hands.

Your ability to locate and solve this type of issue quickly can make you look either really good or really bad depending on how it goes. Case in point, I was working behind the scenes at a large tradeshow recently, and I was asked to give a group of executives a tour of the NOC. As I was demonstrating the network management tools, I started getting text messages from the system that we were having wireless issues. Turns out that one of the attendees for the show had a bad Wi-Fi card, and everywhere that user roamed it caused the wireless router to spike to 100% CPU and reboot, knocking off all of the other users. Because we'd deployed automated, user tracking tools we were able to identify the user, track down where he was, and lock him out of the network in less than 5 minutes. Now remember, all this happened while the executives were watching as it was during the tour. I ended up looking like a hero, and they all walked away impressed.

If I hadn't had the tools in place, I'd have basically turned my back to the executives and started blazing through CLI prompts until I'd forgotten all about my audience and eventually (hopefully) solved the issue. I don't know about you, but I much prefer the hero status.

How do you track down problem users in your environment? If you got orders to "find that user and shut down their port ASAP," how long do you think it would take you? Shoot me a comment and share your stories.

Flame on...

Follow me on Twitter

Josh Stephens is Head Geek and VP of Technology at SolarWinds, an IT management software company based in Austin, Texas. He shares network management best practices on SolarWinds’ GeekSpeak and thwack. Follow Josh on Twitter @sw_headgeek and SolarWinds @solarwinds.  


Copyright © 2012 IDG Communications, Inc.

It’s time to break the ChatGPT habit
Shop Tech Products at Amazon