Yahoo working on Hadoop MapReduce 2
The new version will deliver better resource management
Todd Papaioannou, vice president of cloud architecture at Yahoo, told Computerworld this week that current iterations of Hadoop lack the ability to effectively manage resources across thousands of servers in a cluster.
So developers are working on improving utilization, scheduling and management of resources.
For example, the new architecture will include a global ResourceManager that will tracks server availability and scheduling invariants while a per-application ApplicationMaster runs inside the cluster and tracks the program semantics for a given job, Yahoo developer Arun Murthy wrote in a blog post.
Papaioannou said Yahoo contributed about 70% of the code for the current iteration of Hadoop and the Hadoop Distributed File System (HDFS).
Earlier this year, Yahoo dropped its own distribution of Hadoop and began working more closely with the Apache Hadoop community because it allows the open source community to help with development efforts, Papaioannou said.
Along with Apache, Hadoop uses an iteration of MapReduce, a programming technique that originated at Google, for building parallel programs. Running with Hadoop, MapReduce enables it to perform parallel batch processing.
"The next generation of HDFS will be more resilient, available and reliable," Papaioannou said. "We expect to put it all together in a release some time soon. That's an exercise of collaboration with rest of the development community."
Yahoo also just launched a new project called H Catalog, which is a table metadata management schema for Hadoop.
"That will help drive different use cases," he said. "It just went into Apache version last week."
Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian or subscribe to Lucas's RSS feed . His e-mail address is firstname.lastname@example.org.
Read more about Storage Software in Computerworld's Storage Software Topic Center.
- Best iPhone, iPad Business Apps for 2014
- 14 Tech Conventions You Should Attend in 2014
- 10 Desktop Apps to Power Your Windows PC
- How to Add New Job Skills Without Going Back to School
- Slideshow: 7 security mistakes people make with their mobile device
- iOS vs. Android: Which is more secure?
- 11 sure signs you've been hacked
- Simplify and Consolidate Data Protection for Better Business Results Learn about IBM® Tivoli® Storage Manager Operations Center, which provides advanced visualization, built-in analytics and integrated workflow automation features that leapfrog traditional backup...
- Gartner 2013 Magic Quadrant for Enterprise Backup/Recovery Software See why CommVault was positioned as the #1 leader in Gartner's 2013 Magic Quadrant for Enterprise Backup/Recovery software for the 3rd year in...
- Forrester Report: CommVault is a Leader in Enterprise Backup and Recovery In this report, Forrester takes a deep dive into the evaluation criteria, how CommVault is positioned and the features and functionality that make...
- Forrester Wave for Enterprise Backup and Recovery Read this report to see how CommVault continues to outpace its competitors and why Forrester positioned CommVault Simpana as the top backup and...
- Four Myths of High-Productivity App Dev Debunked Debunk the main myths surrounding high-productivity application development and how both platforms have overcome them.
On-Demand Webcast: 7 Reasons to Choose VoIP
Thinking about a new phone system for your business?
Be sure to watch this informative webcast. Steve Strauss, small business columnist for USA...
All Storage Software White Papers |