School district gets an HP SDN makeover to address wireless growth, security problems

Faced with exponential growth in wireless devices and an increasingly digital curriculum, Jeff Dietsche, systems and infrastructure manager for the South Washington County Schools in Minnesota, decided his only hope was to deal with a single vendor and use SDN to streamline operations. Dietsche tells the tale to Network World Editor in Chief John Dix.

Let’s start with a little background on your environment and any recent changes that have come down the pike.

I’ve been with the district since 1991 and have seen a lot of growth and a lot of changes in that time, but I’ve never seen anything like the wireless revolution in the last four or five years.  In 2009 we had 1,000 wireless users, and in 2010 we had 2,000, and I thought -- "That’s a really, really big jump." Everybody was bringing in their own devices and it wasn’t clear how we were going to handle the growth, what policies we needed, etc.

At the time we maybe had a hundred access points and just enough bandwidth to support workstations plugged into wires, and all of a sudden our access points are filling up and we are running out of capacity. So I started to think, "How am I going to manage this given I’m the only guy here in this large school district with 18,000 students and 31 sites?"

I knew some local universities had standardized on a single-vendor for all of their switches, servers, workstations, laptops and everything because they were having such a hard time supporting a mix of vendors with all of the finger pointing, so I always kept that in the back of my mind.

We already had a 3Com VoIP phone system, and after HP bought 3Com they came out with a new architecture and new set of gear, so we went out to Palo Alto to talk to their product managers. We talked to the server people, the wireless people, the network people, the storage people, and it seemed like they had a good vision for the future, so it seemed like a good idea to standardize on HP.

What did your existing environment look like?

We had a little bit of Dell, a little bit of EMC, a little bit of Extreme, a little bit of 3Com. It was really whatever the best deal was at the time because school systems have such tight budgets. We just had to get by and be efficient, but that idea just wasn’t going to work when we saw this explosion in wireless. It was clear I wasn’t going to be able to handle it.

If your budget was so tight, how could you afford to swap everything to HP?

Our superintendents realized IT was struggling trying to manage all this stuff, and they had big plans to move toward a digitized curriculum using digitized textbooks and everything, so a $15 million bond referendum was passed that was just in time.

So when we went to visit HP we really had an agenda. What are we going to do for a blade server replacement? What kind of access points do we need? How are we going to grow this wireless system? And if we do go with HP, what do they have that would help us handle the growing security problems we were having?

What kind of security problems?

Back in 2010 the Internet and wireless, all this stuff was exploding on us. I was getting calls from some energy company in Great Britain saying, "You guys are attacking us." I was like, what? Turns out someone had clicked on a phishing message and gave out a password, so suddenly we’re sending tens of thousands of messages at this power plant in Great Britain.

And students were learning how to telnet into our switches and causing all kinds of trouble, so we got some quotes on how to address the security problem and most of them were for $1 million or more and required putting a box between every switch in every building. There was no way I could do that.

Yikes. So HP came to the rescue with what?

We ended up installing HP 2920 switches for a Proof of Concept in March, 2014, which support the OpenFlow 1.3 Software Defined Networking standard. When the POC was successful, we installed 150 of those switches and upgraded 210 older switches to HP 3800s.  

So that addressed your capacity problems and set you up to tackle the security issue?

Once we had HP switches capable of running OpenFlow 1.3 we added an HP SDN controller. We actually first ran it on our Dell blade server because we didn’t have our new HP c7000 installed yet. We also needed to upgrade our HP Intelligent Management Center. IMC is their single pane of glass dashboard that can see all of the wireless components, all the switches, anything that’s going on. It has alerts and alarms and backs up all of the configurations every week and can even do traffic analysis.

And for security, this is the honest-to-God truth: Once we understood how to install HP Net Protector with SDN we created a simple script in IMC and pushed it out to 400 switches. It probably took less than 15 minutes, and we had our entire district up and running for just a fraction of the cost of what that same type of security solution would have cost years ago.

What does Net Protector do for you?

Net Protector uses a reputation database in the cloud to check DNS requests. We get 22 million DNS requests per day, and Net Protector blocks about 200,000 at the port level each day. The Tipping Point database currently knows about 1.7 million malicious sites throughout the world, so, if Net Protector sees a request for one of those bad sites it just knocks it down right at the port.

You can raise or lower your scale in terms of what you want to block. Do you want to just block the best known bad sites, or do you want to block everything that might be even remotely bad? We can’t quite do that because teachers are using the Internet to such an extent that we don’t dare go too far with that. We have to stick to the well-known bad stuff because we’re afraid we would start hindering classrooms because of false positives.

So all the DNS requests from every single port, wired and wireless, flow back to the SDN controller on the VM farm on our c7000 blade server, and if a bad DNS request comes in it is blocked right at the port. It doesn’t even have a chance to get into my network because SDN is looking at it and saying, "That’s a bad one. Knock it down."

And that’s just like a miracle in my world. Wow, that’s cool. Even bad stuff happening on another station or between access points, they can’t affect each other. I can’t have a user at Point A affect the user at Point B because it can’t even get into the switch to say anything bad.

Now we’re getting ready to add the iBoss FireSphere app from the app store to address Advanced Persistent Threats. We were the ones who requested it from iBoss. So besides DNS requests getting knocked down, we’ll be able to analyze the packet contents for advanced persistent threats. So if it’s not in the reputation database, let’s say it’s something brand new that just doesn’t look right, iBoss APT is going to knock it down at the port going through the same SDN protocol.

So, do you think you are ahead of the curve now?

A year and a half ago I went up to the high school and sat in the media center and talked to some of the high school juniors and seniors. Some of the kids had the iPads the district gave them, some had laptops and smart phones, and many of the students were using all three devices at the same time. He’s texting his friends on his smart phone, then on his laptop he’s using his favorite word processor and on his iPad he’s running some classroom applications for his geography or his history because we’ve got digital textbooks running on these tablets.

It’s like we can’t go fast enough. We’ve got to plan now for more capacity. We’re doubling the number of Wireless Access Point controllers to increase wireless capacity for up to 2,000 access points for the 2015-2016 school year. Wireless is being used that much more every year. The amazing thing is that even though the client load is increasing, we don’t expect to have to add any additional SDN controllers.  That’s how cost effective this solution is. You can increase capacity on the wireless access controllers and number of  access points, but one SDN controller is able to handle the increased load.

We ended the year supporting 16,000 wireless devices. Now I think we’re approaching 23,000 or 24,000. We might be up close to 30,000 for the students.

Besides the security benefit, are you realizing any other SDN benefits?

Our sales engineer came recently and said the next phase is coming where we’ll be able to start implementing quality of service with SDN. That’s a really big goal because if you have massive amounts of audio and streaming video you’ve got to have quality of service to contain and control it.

Have you estimated what SDN has saved you in terms of man-hours when it comes to managing switches or configuring devices?

I guess the proof is in the pudding because I’m the only infrastructure manager. I have a pretty nice life right now because it’s almost like we’ve gone back to the mainframe age where you had a single vendor hardware architecture managed from a centralized location. It was really heaven back in the 80s.  It was so simple with that type of central standardized architecture. A top to bottom HP infrastructure with an HP IMC management console makes it feel a lot like operating a single mainframe. 

But a specific example: We’ve got card access systems that are all computer-based and if something happens to that system I’ll pull in a consultant and say, "We want to switch to a different VLAN configuration and I’ve got to do it across hundreds of switches at 31 locations, so create a script to do it, show me where it is and I’ll push the button tomorrow and implement it." With IMC you can do that.

Quite a story.

Yeah, it’s exciting.

This story, "School district gets an HP SDN makeover to address wireless growth, security problems" was originally published by Network World.