The trouble with MPLS

High-end carrier services can mean high-end config issues (and risks)

Multisite and outsourced IT operations are making good use of Multiprotocol Label Switching (MPLS), but strange trouble is turning up more and more. Often in discussion with local network staffers, we come to the point when I ask about backhaul lines or internet service providers over which they presumably run a site-to-site virtual private network (VPN). They happily reply, "Oh, we have MPLS" and provide a network diagram consisting of a suitably inscrutable cloud.

Life is not so simple.  Increasingly, those IT infrastructures appear functional, but a simple scan turns up many times the number of hosts that ought to be visible. Finding rogue devices on a network is cause for a bit of alarm, but unknown subnets?

What's going on? MPLS is supposed to simplify wide-area networking with carrier-grade service, not increase the risks of exposing sensitive data. Finding one's network cross-connected with another organization is not something that can be dealt with tomorrow, and a serious address-space collision can put networks completely out of commission.

IT managers and technologists looking for a simple way to connect distant LANs turn to MPLS as a solution that has more currency and expandability than older offerings. The trouble is many of them make the decision to adopt MPLS without enough information.

Blue car, yellow car

As its name implies, MPLS embeds network switching or routing information into lower network layers. Like frame relay, it allows low-latency transit of network traffic between two distant points but leaves error handling up to the endpoints.  Like asynchronous transfer mode, it embeds transit information into lower network layers, but the variable-length packets of MPLS are more suited for encapsulating IP traffic than ATM's fixed-length cells.

By labeling traffic at a lower network level, less processing has to happen at each waypoint between source and destination. It's analogous to color-coding cars on the highway and allowing only blue cars to enter at the Los Angeles on ramp and exit at San Francisco and vice versa. Yellow cars might share the same road from San Diego to Santa Barbara, but they would enter or exit only on ramps flagged for yellow cars. 

The transit speed is unchanged -- MPLS doesn't make a 10Mbit/sec. link go to 11 -- but the entry, routing and exit decisions can be made much more simply and quickly. To send someone to San Francisco, you could give them a blue car, and the network (the road) would get him there without needing to get on and off the highway to ask for directions.

The devil's in the details. RFC 3031 defines MPLS, but it takes a subsequent half-dozen RFCs to cover the more drowsy topics of label distribution, handling, application and interfaces with other networks. MPLS configuration is, as others have noted, expensive and tedious, which is why the technology has been the domain of carriers for the better part of a decade.

It's misleading, then, when uninformed LAN administrators think an MPLS connection "just works." Simply plugging two network endpoints into a carrier's MPLS cloud may allow those networks to communicate, but without some attention, anyone else connected to the cloud may also be visible.

In more than one instance, an organization with just a couple of private IP, Class C subnets (192.168.1.x and 192.168.2.x, for example) discovered dozens of other subnets (such as 192.168.50.x) with thousands of live hosts. Its MPLS router was smart enough to sort out subnet mapping when the IP traffic from other sites was "popped" out of the MPLS cloud on the their side (avoiding wholesale IP collision), but without configuration to indicate otherwise, it was just one big happy network. 

Compliance and the cloud

What do you do when suddenly there are a lot more blue cars than you expected? And what about the other organizations wondering who's scanning their networks from a rogue host through their MPLS routers?  For any organization concerned about the Sarbanes-Oxley Act, the Gramm-Leach-Bliley Act, or the PCI (Payment Card Industry) data security standard, it's quite a mess.  

Who sorts out requirements for labeling before an MPLS network is provisioned?  Often no one, as it turns out. A carrier may provide a basic configuration that works, but an uninformed customer may unwittingly buy a low-end MPLS service with fast traffic handling but no isolation from the carrier's other customers. The carrier assumes that the client knows it's a semiprivate connection and will run a site-to-site VPN. Instead, the uninformed client opens the whole network to the cloud.

It's not that organizations shouldn't play with big kids' toys, but they need to be aware of their complexity. Traffic problems are not unique to MPLS, but its presentation and function make it easy to mistake for a VPN. A toaster oven is not the same thing as a modular food-heating unit, and a pair of MPLS routers are not plug-and-play access points.  

Three precautions

Anyone in the market ought to consider at least these three simple precautions when MPLS is in play:

  1. Ensure that responsibility for network traffic labeling is assigned. This means more than watching as the carrier's sales rep nods and says it's handled.  Just as access control lists between internal VLANs are necessary for meaningful segmentation, someone needs to define label assignment and distribution rules before data is forked over to the MPLS cloud.
  2. The contract or service-level agreement should include a description of suitability, not just performance metrics for the connection. This means the usual contractual disclaimer "We're not responsible for how you use this connection" ought to be extended or replaced with a bounding statement such as "suitable for the requirements specified in Exhibit A." That exhibit defines whether the endpoints are connected to VPN routers or a pile of unsegmented network spaghetti.
  3. Know what the options are for MPLS traffic engineering. Your situation may call for a dedicated MPLS cloud, heterogeneous options or another service altogether. Knowing your choices is the first step toward making a good one.

    Jon Espenschied has been at play in the security industry for enough years to become enthusiastic, blasé, cynical, jaded, content and enthusiastic again. He is currently a senior security consultant in Seattle, where his advice has been ignored by CEOs, auditors and sysadmins alike.

Computerworld's IT Salary Survey 2017 results
Shop Tech Products at Amazon