Python gets a big data boost from DARPA
Continuum Analytics will extend the widely used NumPy library for distributed systems
IDG News Service - DARPA (the U.S. Defense Advanced Research Projects Agency) has awarded $3 million to software provider Continuum Analytics to help fund the development of Python's data processing and visualization capabilities for big data jobs.
The money will go toward developing new techniques for data analysis and for visually portraying large, multi-dimensional data sets. The work aims to extend beyond the capabilities offered by the NumPy and SciPy Python libraries, which are widely used by programmers for mathematical and scientific calculations, respectively.
More mathematically centered languages such as the R Statistical language might seem better suited for big-data number crunching, but Python offers an advantage of being easy to learn.
"Python is a very easy language to learn for non-programmers," said Peter Wang, president of Continuum Analytics. That's important because most big data analysts will probably not be programmers. If they can learn an easy language, they won't have to rely on an external software development group to complete their analysis, Wang said.
The work is part of DARPA's XData research program, a four-year, $100 million effort to give the Defense Department and other U.S. government agencies tools to work with large amounts of sensor data and other forms of big data.
For the XData project, DARPA awarded funding to about two dozen companies, including the University of Southern California, Stanford University and Lawrence Berkeley National Laboratory. The organizations are encouraged to use each other's technologies to further extend what can be done in big data, Wang said.
DARPA encouraged the funding recipients to release products based on their work and to release their code as open source, so the innovations can be widely used and supported outside of the military. The Defense Department is trying to avoid commissioning software that gets used only by the military, which may then become prohibitively time-consuming and expensive to update.
"With big data systems, you find new things you want to look at every week. You can't wait for that process any more," Wang said.
Headquartered in Austin, Continuum Analytics offers add-on products and services that help organizations use Python for data analysis. The company will use the DARPA money to continue development of a number of add-on technologies it has been working on, including Blaze, Numba and Bokeh, all of which provide advanced features not offered in Python itself.
At the PyData 2012 conference in New York last November, Continuum engineer Stephen Diehl discussed how Blaze would operate, describing the library as a potential successor to NumPy.
NumPy has limitations that Blaze seeks to correct, Diehl said. Most notably, NumPy only offers the ability to store a series of numbers as one continuous string of data. "It is a single buffer, a continuous block of memory. That may be OK for some uses, but the real world is more heterogenous," he said in a presentation.
This pilot fish is a contractor at a military base, working on some very cool fire-control systems for tanks. But when he spots something obviously wrong during a live-fire test, he can't get the firing-range commander's attention.
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Reduce federal infrastructure risk with compliance management and situational awareness
- IBM continuous monitoring and management solutions deliver real-time situational awareness to help federal agencies understand vulnerabilities, and protect the infrastructure.
- Improving IT Efficiencies: Four Advantages of Multi-Tenant Data Centers
- Increasing demands on IT are forcing organizations to rethink their data center options. For many organizations, that means turning to the flexibility afforded...
- Accelerating Cloud Deployment and Operations with Managed Services
- Companies that do not have sufficient in-house expertise to either deploy or maintain an IaaS cloud should turn to Managed Service Providers .
- Rethinking IT Operations in the Cloud
- This paper breaks down the challenges that often prevent the cloud from delivering the fast, flexible and affordable infrastructure companies seek - and...
- Gartner Magic Quadrant for Cloud-Enabled Managed Hosting, North America
- Cloud-enabled managed hosting brings cloudlike consumption and provisioning attributes to the traditional managed hosting market All Government IT White Papers
- NSS Labs & Cisco Present: Evaluating Leading Breach Detection Systems Today's constantly evolving advanced malware and APTs can evade point-in-time defenses to penetrate networks. Security professionals must evolve their strategy in lockstep to...
- Will the Real Endpoint Threat Detection and Response Please Stand Up? This webinar explores new technologies & process for protecting endpoints from advanced attackers as well as the innovations that are pushing the envelope...
- What should I look for in a Next Generation Firewall? SANS Provides Guidance With so many vendors claiming to have a Next Generation Firewall (NGFW), it can be difficult to tell what makes each one different....
- Why Are Customers Really Deploying an NGFW? It seems every IT Security expert is talking about the NGFW, but what are people really doing? This webcast covers 5 real-world customer...
- Charting Your Analytical Future - "Making predictive analytics part of your business processes" Webinar This session will show how predictive analytics can be used throughout the organization by anyone looking for answers and how organizations can make...
- All Government IT Webcasts