Yahoo working on Hadoop MapReduce 2
The new version will deliver better resource management
Todd Papaioannou, vice president of cloud architecture at Yahoo, told Computerworld this week that current iterations of Hadoop lack the ability to effectively manage resources across thousands of servers in a cluster.
So developers are working on improving utilization, scheduling and management of resources.
For example, the new architecture will include a global ResourceManager that will tracks server availability and scheduling invariants while a per-application ApplicationMaster runs inside the cluster and tracks the program semantics for a given job, Yahoo developer Arun Murthy wrote in a blog post.
Papaioannou said Yahoo contributed about 70% of the code for the current iteration of Hadoop and the Hadoop Distributed File System (HDFS).
Earlier this year, Yahoo dropped its own distribution of Hadoop and began working more closely with the Apache Hadoop community because it allows the open source community to help with development efforts, Papaioannou said.
Along with Apache, Hadoop uses an iteration of MapReduce, a programming technique that originated at Google, for building parallel programs. Running with Hadoop, MapReduce enables it to perform parallel batch processing.
"The next generation of HDFS will be more resilient, available and reliable," Papaioannou said. "We expect to put it all together in a release some time soon. That's an exercise of collaboration with rest of the development community."
Yahoo also just launched a new project called H Catalog, which is a table metadata management schema for Hadoop.
"That will help drive different use cases," he said. "It just went into Apache version last week."
Lucas Mearian covers storage, disaster recovery and business continuity, financial services infrastructure and health care IT for Computerworld. Follow Lucas on Twitter at @lucasmearian or subscribe to Lucas's RSS feed . His e-mail address is firstname.lastname@example.org.
Read more about Storage Software in Computerworld's Storage Software Topic Center.
- Are You Prepared for a Software Audit? Just the word "audit" is enough to make anyone shiver, and when it comes to a software audit, the reaction is no different....
- Top Reasons for Upgrading to Emulex Gen 5 Fibre Channel HBAs With Gen 5 Fibre Channel HBAs you can run more virtual machines and applications per server, cut your HBA installation and management time...
- Top 3 Myths about Big Data Security : Debunking common misconceptions about big data security Big data represents massive business possibilities and competitive advantage for organizations that are able to harness and use that information. But how are...
- Magic Quadrant for Data Masking Technology IBM is a leader in Gartner Inc's Magic Quadrant for Data Masking Technology. Read the full report to learn about IBM.
- The Foundation You Need to Build a Better Storage Infrastructure Watch this webcast to hear how you can maximize the economics of your data center by modifying your storage footprint and power usage...
- What should I look for in a Next Generation Firewall? SANS Provides Guidance With so many vendors claiming to have a Next Generation Firewall (NGFW), it can be difficult to tell what makes each one different.... All Storage Software White Papers | Webcasts