There is no doubt that big data has become a growth market, and is becoming one of the few on-premises projects seeing increased spending, as companies move less data-sensitive functions to the cloud.
In a report on big data growth, IDC said visual data discovery tools will grow 2.5 times faster than the rest of the business intelligence (BI) market and spending on cloud-based big data and analytics (BDA) solutions will grow three times faster than spending for on-premises solutions.
However, big data is and will continue to be hamstrung by a lack of talent for the coming years. IDC predicts that in the U.S. alone there will be 181,000 deep analytics roles in 2018 and five times that many positions requiring related skills in data management and interpretation. However, there won't be enough qualified applicants to fill those jobs.
And Gartner has said that by this year, big-data demand will create 4.4 million jobs globally but only one-third of those will be filled.
That's because big data analytics require skills beyond just using a dashboard to monitor data streams. You need experience in data science to set up your searches and parameters for designing filtering algorithms. You won't get those skills with a certification program; they require a master's degree or even a doctorate.
A survey by Burtch Works published in 2013 found nearly nine of 10 big data professionals have at least a master's if not a doctorate in a quantitative discipline such as statistics, applied mathematics, operations research or economics.
Another survey, this one from McKinsey Global Institute, predicted that by 2018, the U.S. could be dealing with a shortage of 1.5 million big data experts.
So, what do you do if you can't find a big data guru with an advanced degree? Here are four alternative approaches to finding, developing and retaining big data talent.
1. Start with people who know the business
"I agree with the assessments that [big data] skills are in incredibly short supply," says Nick Heudecker, research director for information management at Gartner. "Clients don't know what skills they need to begin with, let alone where to find them. There's a certain unawareness about what kinds of problems they will be facing and what analytical skills they will need to tackle them."
Companies often think they need someone with a Ph.D. in advanced data science or mathematics, but Heudecker said one alternative approach is to find someone who knows your business and teach them the analytics skills.
"Starting with an understanding of the business is more important than starting with an understanding of machine learning. You can teach people data manipulation and statistics or find someone with a degree who has some background in programming. You can source those people, give them additional training and move them into your big data or advanced analytics teams," he says.
2. Develop your own superstars
Min Xiao, a field engineering lead with big data software provider Tamr, says he has interviewed about 500 people in the last five years and recruited about 40 to 50. He agrees it's very hard to find the right big data analysts, but has his own method for finding talent.
"My trick has been to find people who are not a current superstar, but to find the potential and make them the superstar. I try to hire a lot of young people who haven't done the data scientist job but I can see they have the potential, or people on mid-level or senior level haven't done a data science role, but they have the potential and make them the superstar," he says.
+ ALSO: "Akamai CSO Takes a Creative Approach to Finding Security Pros" +
The potential he looks for is in education, both the degree and the school. He looks for people with majors in statistics, computer science and sometimes physics. A physics major might not be the first degree that comes to mind for analytics, but Xiao has said he's done well with those people.
"First of all, they are pretty smart if they have a degree in physics. They have training in mathematics, and in modern physics courses they also need to do a lot of programming. So they may not have formal computer science training, but in terms of computer skill required by a data scientist role, many of them are pretty good," he says.
The other area he looks at is schools with heavy emphasis on math and science, like MIT, Carnegie Mellon, Stanford, Brown and Johns Hopkins. "Certain schools have a reputation for setting the bar really high, so people who graduate from these schools can work hard and have the attitude to work hard," says Xiao.
3. Look for advanced Excel experts
Jason Chavarry, a manager in the people analytics department at The Hershey Company, finds big data talent in another unusual place: advanced Microsoft Excel users.
"Excel is the breeding ground. A lot of people get the abilities of big data from there and they tend to get asked to help with other jobs," he says.
Excel is an entry level conduit where people learn the basic functionality that is found in big data analytics, he added. "Everyone is used to a lot of the basic functionality. It's how you lay out a report or spreadsheet, what rules do you create. Excel goes across all of them. You can use it for base level stats, basic data analysis and visualization," he adds.
But Chavarry notes that you need different tools for different scale projects. For an analytics project on 5,000 rows of data, something like SAS or R would be overkill but Excel would be perfect. For 20 million rows, Excel would not be powerful enough. Then you need big data software and programming knowledge, but he is not wedded to one particular language.
"You're not really concerned about which one. If someone has the demonstrated ability to do one language, they have the ability to do the others. You look for that learning agility because they should be able to focus on the others," says Chavarry.
4. Grow your own
With talent scarce, the solution for most companies will be to grow their own talent farm. Ashley Stirrup, CMO with Talend, a big data software integration firm, says the company has gotten good results by establishing a mentorship program, pairing young talent with older experts.
"There is a whole other cast of skillsets for people who do the bridge between the business and these new technologies," says Stirrup. "Often, business people don't realize the potential of new technologies coming along and the tech people don't realize how to use it."
Unfortunately, keeping the talent can be difficult. Talend customers say that as they bring someone in and get them trained on new technologies, that person can find a job somewhere else for 50% more money, so they are having a hard time finding talent and a hard time keeping them after they are trained.
So how do you keep them, short of a binding contract that might poison the relationship with the employee? "The key things are showing them they can build their skillset and can be more valuable at your company. Also, you want to set expectations, not be looking at contracts," says Stirrup.
Xiao is encountering the same problem with talent plundering. He says Tamr tries to hire people with a team attitude and motivate them into finding value at the company. "When they see people like them on the team, they usually think this is a team I want to work with for the next few years. The market is very competitive and we have the sincere desire to make people successful or we won't attract great people," he says.
Heudecker also thinks companies should incentivize rather than shackle talent. "You may not need a team of Ph.D.s -- maybe just one person with a master's in statistics or computer science and an MBA to tell the story to the group is enough. Look at people's undergraduate degrees and if they are interested in the data story, build on that. Companies should offer incentive-based training and some way to insure the employee stays on for a while because these skills are in such demand," he says.
Eventually, Heudecker said, big data will become the new normal and the talent pool will expand. "If you look at the big-data infrastructure, it's very similar to the RDBMS market of the '80s. There were no apps back then, but people built them. The same will happen with big data," he says.
Patrizio is a freelance technology writer in Orange County, Calif. He can be reached at andy@andypatrizio.com.
This story, "4 ways to beat the big-data talent shortage" was originally published by Network World.