The Netgear router flaw post mortem -- plenty of blame to go around

In the recent Netgear router flaw, it's easy to blame Netgear for ignoring the initial report of the vulnerability. They have since admitted that it fell through the cracks. But there is plenty of blame to go around.

While Netgear owners are indebted to someone who goes by Acew0rm for finding the flaw, he appears to have dropped the ball. After notifying Netgear of the vulnerability on August 25, 2016 he walked away from the issue. His total effort in getting Netgear to acknowledge the problem was a single email message. I think he could have done more. When an email is not acknowledged, it's not much work to re-send it a second or, if needed, a third time. 

And, then there is CERT. Who? According to their website: "CERT is a part of the Software Engineering Institute (SEI), a federally funded research and development center (FFRDC) operated by Carnegie Mellon University."

On December 9, 2016, CERT publicized the router flaw which drew a ton of attention to the problem. I heard about it from an article that quoted CERT as the source of the story. When CERT went public, it was unclear which Netgear routers were vulnerable. And, since there was no work-around, CERT drew a ton of press with their suggestion to take Netgear routers off-line. The inevitable headlines followed

  • CERT Warns Users to Stop Using Two Netgear Router Models Due to Security Flaw
  • Unplug Your Easily Hijacked Netgear Routers Pronto
  • Stop using Netgear routers with unpatched security bug, experts warn
  • It may be time to turn off your router: Netgear confirms security vulnerability
  • A Ton of Popular Netgear Routers Are Exposed—With No Easy Fix

It was a typical hair-on-fire story even though the vulnerability was much harder to exploit compared the huge flaws abused by a Mirai variant that recently knocked Deutsche Telekom, TalkTalk and Eir customers off-line.

Did CERT do the right thing? In my opinion, no. I say this because CERT did not try to contact Netgear before running with the story.  

CERT has their rules for this sort of thing and they are hiding behind them. I would hide too, if I put up a billboard telling bad guys how to attack routers at a time when there was no defense. 

But, after multiple emails back and forth with two people at CERT, they are fine with it. Their position is that since the flaw was made pubic on December 7th at exploit-db.com, the cat was already out of the bag.

I am not familiar with the exploit-db.com website, but all the publicity here stemmed from the CERT notice. The exploit-db.com site generated no press interest at all.

CERT made a bad situation worse by publicizing the flaw when there was no work-around, without giving Netgear a chance to respond. 

This despite their website which says "Working with software vendors, we help resolve software vulnerabilities." Except when they don't.

And, a CERT blog by Garret Wassermann says " ... we help security researchers communicate with software vendors to resolve security issues." Except when they don't.

And the Vulnerability Disclosure Policy that they hide behind includes this: 

Q: Will you surprise vendors with announcements of vulnerabilities?
A: No. Prior to public disclosure, we'll make a good faith effort to inform vendors of our intentions.

Except when they don't

It also says "We will apprise any affected vendors of our publication plans and negotiate alternate publication schedules with the affected vendors when required."

Yet, in this case, CERT blind-sided Netgear. Why was it not required in this case to give Netgear a chance to get a handle on things? Maybe, with some warning, Netgear could have come up with a work-around to hold down the fort until the flaw was fully patched. Eventually, we got exactly that, but it came from a third party, Bas van Schaik.

I didn't see anything in the policy about publicizing a vulnerability simply because someone else already leaked the information, which, is the justification I was given by CERT.

All publishing is not the same. When CERT publishes, people notice. It's like advertising during the Super Bowl. 

And, all flaws are not the same. This particular vulnerability is somewhat hard to exploit, a victim has to be lured to a malicious web page.

If an ISP gave millions of vulnerable routers to their customers, then it would make sense for bad guys to target this flaw. But, that was not the case here. The initial flaw was found in one Netgear router, the R7000. It was not a dire situation.

Then too, the CERT advisory itself is flawed.

As I write this, it says "For users of models without a firmware fix, we recommend the following workarounds..." But there are no models without a firmware fix and there have not been for a while. For one thing, Netgear committed to fixing the problem in every vulnerable router. Then, they initially issued beta firmware before rolling out production firmware.

I am writing this on December 24th and, as of yesterday, all the vulnerable routers have new production firmware to fix this problem. 

The CERT advisory is also missing a test for the vulnerability. Many have been published and it's a simple thing to do. By testing their routers before and after a firmware update, Netgear owners can verify that the firmware did, in fact, fix the problem. And, if all Netgear owners tested their routers we may find a vulnerable model that the company missed. 

Finally, CERT never entertained the possibility of using a Guest network to block the problem. Guest Wi-Fi networks have different security profiles. It's possible that, depending on the options chosen, a Guest network would prevent a Wi-Fi user from accessing the router and thus block exploitation of the flaw. I mentioned this previously, but without access to a vulnerable Netgear router I can't test it. 

Back in March 2015, How-To Geek reviewed the NETGEAR Nighthawk X6 AC3200, a.k.a. the R8000, a vulnerable router. The review pointed out that the router has a Guest network option called "allow guests to see each other and access the local network." It is possible that blocking access to the local network would have kept people safe. But, we don't know because CERT let the opportunity for feedback slide.

The Netgear advisory also had a "not invented here" problem, never discussing Bas van Schaik's hack that killed the web interface. Does it really work? Does it impact anything else in the router? Netgear owners deserved answers they never got, especially since this was the only defense for a while. Or was it? Netgear too, never addressed the issue of using a Guest network as a defense mechanism. 

FIRMWARE UPDATING

Now that all the vulnerable routers have updated firmware available, the process of installing it shows itself to be sub-optimal. Last time, I griped about the manual nature of the firmware update which guarantees many router owners won't do it. But even within the realm of manual updates, the Netgear procedure is poor. 

Some routers can search for new firmware simply by clicking a button in their web interface. Judging by the Netgear instructions (see the R8000 for example), their routers can not. 

Another problem that some routers exhibit with firmware updates, is losing track of configuration changes. That is, installing new firmware may reset some changed options back to their default values. Netgear doesn't even know if their firmware does this or not. The first step in the update procedure is.

Write down all the settings which you changed from the default values, since you may need to re-enter them manually.

Better routers deal with this by letting you download a file with the current configuration. My favorite router, the Pepwave Surf SOHO not only offers this, it goes so far as to notify you every time you update the firmware that it would be a good idea to save the current configuration. 

DEFENSIVE STEPS

Finally, let me point out that this was far from the first router attack via a malicious web page, and there are defensive steps available. 

To begin with, change the default subnet. That is, rather than using 192.168.1.x, use 192.168.22.x. Any number between 5 and 250 should be fine. Also, don't make the router the first device on the network. For example, instead of assigning 192.168.22.1 to the router, make it 192.168.22.3.

If possible, change the TCP/IP port used for LAN side router access. For example, if you normally access the router from the LAN side with

http://1.2.3.4

this would mean using

http://1.2.3.4:9999

instead, where 9999 is the alternate port number. Good port numbers are between 3,000 and 65,000. Netgear routers do not offer alternate ports.

Finally, if it's offered, force router access over HTTPS rather than HTTP. 

FEEDBACK
Now that Computerworld, and all of parent company IDG's websites, have eliminated user comments, you can get in touch with me privately by email at my full name at Gmail. Public comments can be directed to me on twitter at @defensivecomput

Related:
Computerworld's IT Salary Survey 2017 results
Shop Tech Products at Amazon