Python gets a big data boost from DARPA
Continuum Analytics will extend the widely used NumPy library for distributed systems
IDG News Service - DARPA (the U.S. Defense Advanced Research Projects Agency) has awarded $3 million to software provider Continuum Analytics to help fund the development of Python's data processing and visualization capabilities for big data jobs.
The money will go toward developing new techniques for data analysis and for visually portraying large, multi-dimensional data sets. The work aims to extend beyond the capabilities offered by the NumPy and SciPy Python libraries, which are widely used by programmers for mathematical and scientific calculations, respectively.
More mathematically centered languages such as the R Statistical language might seem better suited for big-data number crunching, but Python offers an advantage of being easy to learn.
"Python is a very easy language to learn for non-programmers," said Peter Wang, president of Continuum Analytics. That's important because most big data analysts will probably not be programmers. If they can learn an easy language, they won't have to rely on an external software development group to complete their analysis, Wang said.
The work is part of DARPA's XData research program, a four-year, $100 million effort to give the Defense Department and other U.S. government agencies tools to work with large amounts of sensor data and other forms of big data.
For the XData project, DARPA awarded funding to about two dozen companies, including the University of Southern California, Stanford University and Lawrence Berkeley National Laboratory. The organizations are encouraged to use each other's technologies to further extend what can be done in big data, Wang said.
DARPA encouraged the funding recipients to release products based on their work and to release their code as open source, so the innovations can be widely used and supported outside of the military. The Defense Department is trying to avoid commissioning software that gets used only by the military, which may then become prohibitively time-consuming and expensive to update.
"With big data systems, you find new things you want to look at every week. You can't wait for that process any more," Wang said.
Headquartered in Austin, Continuum Analytics offers add-on products and services that help organizations use Python for data analysis. The company will use the DARPA money to continue development of a number of add-on technologies it has been working on, including Blaze, Numba and Bokeh, all of which provide advanced features not offered in Python itself.
At the PyData 2012 conference in New York last November, Continuum engineer Stephen Diehl discussed how Blaze would operate, describing the library as a potential successor to NumPy.
NumPy has limitations that Blaze seeks to correct, Diehl said. Most notably, NumPy only offers the ability to store a series of numbers as one continuous string of data. "It is a single buffer, a continuous block of memory. That may be OK for some uses, but the real world is more heterogenous," he said in a presentation.
- 18 Hot IT Certifications for 2014
- CIOs Opting for IT Contractors Over Hiring Full-Time Staff
- 12 Best Free iOS 7 Holiday Shopping Apps
- For CMOs Big Data Can Lead to Big Profits
- Slideshow: 5 ways to lock down your mobile device
- Slideshow: 10 mistakes companies make after a data breach
- How to rob a bank: A social engineering walk through
- Which smartphone is the most secure?
If you think getting it right from day one is always what matters, you probably haven't been following technology too closely.
- IT Certification Study Tips
- Register for this Computerworld Insider Study Tip guide and gain access to hundreds of premium content articles, cheat sheets, product reviews and more.
- Mitigating DDoS Attacks with F5 Technology
- This document examines various DDoS attack methods and the application of specific ADC technologies to block attacks in the DDoS threat spectrum while...
- The DDoS Threat Spectrum
- Bolstered by favorable economics, today's global botnets are using distributed denial-of-service (DDoS) attacks to target firewalls, web services, and applications, often simultaneously.
- Defending Against Denial of Service Attacks
- By utilizing end-user interviews, this whitepaper explores a deeper understanding of DDoS defense plans and reveals the knowledge gaps around the Denial of...
- Strategic Solutions for Government IT
- This paper outlines why F5 is the optimum partner to help achieve the levels of security, performance and availability that are vital to...
- The Six Main Steps To Structured Analogy
- The role of structured analogy software is to automate the data extraction and processing work, provide visualization of the historical context forjudgments and... All Government IT White Papers
- Modernizing SAP environments with minimum risk - a path to Big Data Hear from top IDC analyst, Richard Villars, about the path you can start taking now to enable your organization to get the benefits...
- The Power of the Citrix Mobility Solution, XenMobile Does everything become a smartphone? Or does the smartphone begin to do everything? How can we afford to support BYOD? Rather, how can...
- BYOD Happens: How to Secure Mobility How to navigate the journey of securing mobility, including the BYOD corruption of IT, the top ten mobility strategies, and the mobility management...
- Fighting Fraud Videos: IBM Intelligent Investigation Manager Short videos about IBM Intelligent Investigation Manager (IIM) for Fraud. IIM optimizes the investigation of fraud for customers across many industries in both...
- IBM Intelligent Investigation Manager: Online Product Demo Intelligent Investigation Manager optimizes fraud investigation and analysis and it dynamically coordinates and reports on cases, provides analysis and visualization, and enables more...
- All Government IT Webcasts
Does your organization offer extensive benefits, cool perks, competitive salaries, opportunities for training and advancement? Then get it recognized!
Nominate your company or another deserving organization for Computerworld's 2014 Best Places to Work in IT list now through Dec. 12, 2013.