It's no secret that big data professionals are hard to find. McKinsey & Co. predicts that by 2018 the U.S. could face a shortage of more than 1.5 million specialists needed to capture, store, manage and analyze big data -- which encompasses "the increasing volume and detail of information captured by enterprises, the rise of multimedia, social media and the Internet of Things."
Last spring the Obama administration launched a $200 million big data R&D initiative, part of which is aimed at expanding the workforce needed to develop and use big data technologies.
But employment experts do not expect the situation to get better anytime soon, a problem for companies trying to use big data for competitive advantage. How are they coping with the shortage of talent? Interviews with big data experts and employers found that organizations are using some, if not all, of these methods:
- Trying to find and hire a top data scientist, and putting that person in charge of building a big data team within the company.
- Working with current staff to identify those employees who already have some big data skills, nurturing and developing them through internal or external training, and creating new, higher-level positions for them.
- Taking advantage of new tools designed to enable employees to mine and manipulate big data without specialized training.
Usually, companies have to rely on the last two approaches because top data scientists are in very short supply.
"Everyone at this point has made peace with the fact that everybody out there who's good at this already has a great job and has no interest in leaving it whatsoever," says Josh Wills, senior director of data science at Cloudera. Wills himself can be considered something of a big data superstar -- he worked at Google before moving to Cloudera, which sells and supports big data management software.
Hiring a superstar
Even if a company could find a data scientist to hire, it probably could not find all the skills it needs in a single person.
He or she would need to know programming, be able to create statistical data models, have the appropriate business domain knowledge, understand the different big data platforms and how they work, and have excellent communications skills to explain what the data means in terms of the business.
"A blending of all these things is very rare," says Brian Hopkins, a principal analyst at Forrester Research. "It's probably unreasonable to expect that there will ever be a large cadre of those people."
It's hard to even know where to look for data scientists. Some companies look to graduate schools where they may find newly minted Ph.D.s in statistics, natural language processing or machine language. Others may try to lure scientists from other companies with big analytics programs, particularly search or social networking companies like Google and LinkedIn.