Hadoop skills are in high demand

Computerworld - NEW YORK -- The growing enterprise interest in Hadoop and related technologies is driving demand for professionals with big data skills.

Analysts and IT managers at the Hadoop World conference here this week repeatedly pointed to skills availability as one of the key challenges companies face in adopting Hadoop and said that those with the right skills could command healthy premiums.

One indication of just how limited that skills supply is: IT executives from JP Morgan Chase and EBay who delivered keynote addresses at the conference used the opportunity to recruit from the audience.

Hugh Williams, vice president of experience, search and platforms at EBay, told audience members that the auction site is recruiting Hadoop professionals and he invited those interested in exploring opportunities to speak with him.

Larry Feinsmith, managing director at JP Morgan Chase, who followed Williams, only half-jokingly told the audience that Chase was also hiring and would be willing to pay 10% more than EBay.

"Hadoop is the new data warehouse. It is the new source of data" within the enterprise, said James Kobielus, an analyst with Forrester Research. "There is a premium on people who know enough about the guts of Hadoop" to help companies take advantage of it, he said.

Hadoop allows companies to store and manage far larger volumes of structured and unstructured data than can be managed affordably by today's relational database management systems.

A growing number of companies have begun tapping the technology to store and analyze petabytes of data such as weblogs, click stream data and social media content to gain better insights about their customers and their business.

The increasing enterprise adoption is driving demand for people with advanced analytics skills, Kobielus said. That includes people with backgrounds in areas such as multivariate statistical analysis, data mining, predictive modeling, natural language processing, content analysis, text analysis and social network analysis, he said.

"Big data in the broader sense -- and Hadoop in particular -- is driving demand for people who have experience doing advanced analytics using newer approaches such as MapReduce and R for predictive and statistical modeling," he said. These are the data analysts or data scientists who will work with structured and unstructured data in Hadoop environments to deliver new insights and intelligence to the business, he said.

Interest in Hadoop is also creating demand for Hadoop platform management professionals, Kobielus said. Their job will be to implement Hadoop clusters, secure, manage and optimize them and to ensure that the cluster remains available for enterprise use. "These are the people who build out and optimize the platform" on which Hadoop applications run, he said.