We are fortunate to live in an exciting time where multiple technological leaps are occurring. Specifically, I am thinking of the mobile industry transition from 4G to 5G, and the cross-industry IT paradigm shift to the Big Data approach. The 5G standards community is already planning to support the collection and transmission of massive amounts of data. This is one of its key requirements pillars in the area of supporting the IoT. What is left, however, is for the 5G community to ensure that the other component of Big Data, namely support for network and application intelligence, is also baked into the 5G architecture. Otherwise, 5G may become simply a pipe for Big Data passing between devices and the cloud infrastructure.
What is Big Data?
Big Data, of course, refers to data sets that are so large and complex that traditional database systems cannot handle them. Big Data also encompasses the techniques for data acquisition, storage, processing, analysis, querying and visualization. Basically, the ability to digest vast amounts of unstructured data in an automated fashion to derive useful insights and to drive automated feedback actions.
The simplest example of a traditional database tool is an Excel spreadsheet, which can interestingly handle up to one million rows and runs on your laptop. At the other end of the traditional database technology spectrum is Oracle DB, which runs on distributed grid computers and can scale to handle hundreds of millions of records. We can intuitively grasp that an Excel spreadsheet cannot handle Big Data, but why couldn’t we scale the underlying grid computer network running Oracle DB to handle Big Data?
The answer to this question lies in the fact that Oracle DB requires all its input data to be structured to fit into pre-defined alphanumerical records. However, in many cases, Big Data is composed of semi-structured or completely unstructured data (e.g., a mixture of text, videos, pictures, and sensor data) which cannot fit into the structure of traditional database systems. This is what makes Big Data systems different. This is where the application of new analysis techniques like machine learning is necessary. These methods allow computers to analyze and make decisions without being explicitly programmed for any particular structure of data.
How Big Data can improve network and application intelligence
As discussed IoT will be a key part of 5G, and how this unstructured data is used will be an important part of that story. Think, billions of sensors sending back all types of data (including video) to the network. The opportunity here is to use this data to improve the operation of the network itself, or to help the key new applications that will emerge in 5G work better. Big Data based intelligence, of course, refers to techniques to automatically improve network and application performance, recover from error conditions, and provide improved end user experience by analyzing context, location, etc.
We are technologically on the cusp of useful machine intelligence and completely self-managing systems, but we need to move quickly to ensure that these ideas are incorporated into the 5G architecture as much as they should be.
For example, the 3GPP standards organization has started doing important work in improving its Context Aware Engine (CAE) to handle Big Data type applications. Specifically, a northbound interface is being considered that allows the radio and core networks to stream data indicators into the 5G control plane that contains a CAE that can make better connectivity and flow management decisions for an individual subscriber or for the entire network.
On another front, the IETF standards organization is beginning to examine applying machine learning techniques to the internet. For example, analyzing network traffic to identify security threats, such as Denial of Service (DoS) attacks, and then triggering real-time corrective actions such as updating the routing protocols to dynamically, and temporarily, re-route traffic away from at risk parts of the network.
While I can see a few things going on here, I believe our industry could do a lot more, and specifically in the development of Big Data support interfaces between applications and the 5G network. For example, there will be many different types of IoT applications such as smart cities and smart grids all generating valuable intelligence that could be leveraged to make the network more reactive. It will be very useful if these different IoT applications can interact with the 5G network and, for example, request the edge processors of the 5G network to run certain Big Data analysis tasks and perform actions on their behalf. Part of this process might involve the IoT applications granting the right to the 5G network to analyze the Big Data that may be confidential and/or encrypted. This type of value added Big Data application interfaces to the network still needs to be fully defined at either 3GPP or IETF for maximum benefits.
How will it all work in 5G?
So how might an operator deploy and use Big Data as part of 5G? Let’s sketch out the basic outline here. First, it should be no great epiphany to observe that all operators are steadily migrating to so-called programmable networks enabled by NFV and SDN. This general purpose scalable fabric upon which 5G will be built can provide the compute and storage capacity to run the Big Data databases and analysis (e.g. Hadoop). This may be for the operator’s own network operation purposes or provided as a service to some IoT application that requests the 5G operator to do some critical Big Data processing on its behalf.
When required, the operators will even be able to do Big Data analysis at the edge of the network as envisioned in the Mobile Edge Computing (MEC) model. This will allow filtering and local processing near the source of the data. Another source of unstructured data for 5G will be through the large amount of video data expected to stream through the enhanced mobile broadband access that 5G will provide. An example of network intelligence, in this case, is if the 5G network can automatically detect a trending viral video then it may decide to dynamically bring on line more virtual network resources before the peak viewing period of the viral video and thus avoid network congestion.
5G requirements clearly call out the need to support the IoT and the Big Data it produces. This Big Data is inherently synergistic with other 5G trends of SDN, NFV and Edge Computing. However, there is still work to be done to ensure that the 5G architecture adequately supports all the necessary interfaces and procedures to ensure that the network can leverage this data for maximum benefits. Namely, network and application analysis and feedback actions based on the Big Data. Clear interface specifications are needed between the 5G network and applications that will emerge via the proliferation of the IoT. Only with these interfaces fully defined will we see the 5G programmable network reach its full potential.