11. If the results look obvious, throw out the model.
An entertainment-based hospitality business wanted to know the best way to recover high-value, repeat customers who had stopped coming. Abbott Analytics developed a model that showed that 95% of the time most of those customers would come back.
"The patterns the model found were rather obvious for the most part. For example, customers who had been coming to the property monthly for several years but then stopped for a few months usually returned again" without any intervention, Abbott says.
The business quickly realized that it didn't need the model to predict what offers would get those customers back -- they expected to recover them anyway -- while the other 5% weren't likely to come back at all. "But models can be tremendously valuable if they identify who deviates from the obvious," Abbott says.
Rather than stop there, he suggested that they focus on the substantial number of high-value former customers who the model had predicted would return, but didn't. "Those were the anomalies, the ones to treat with a new program," Abbott says.
"Since we could predict with such high accuracy who would come back, someone who didn't come back was really an anomaly. These were the individuals for whom intervention was necessary."
But the business faced another problem: It didn't have any customer feedback on why they might have stopped coming and the models could not predict why the business had not recovered those customers. "They're going to have to come up with more data to identify the core cause of why they're not returning," Abbott says. Only then can the business start experimenting with emails and offers that address that reason.
12. Don't define clearly and precisely within the business context what the models are supposed to be doing.
Abbott once worked on a predictive model for a postal application that needed to predict the accuracy of bar codes it was reading. The catch: The calculation had to be made within 1/500 of a second so that an action could be taken as each document passed through the reader.
Abbott could have come up with an excellent algorithm, but it would have been useless if it couldn't produce the desired result in the timeline given. The model not only needed to make the prediction, but had to do so within a specific time frame - and that needed to be included in defining the model. So he had to make trade-offs in terms of the algorithms he could use. "The models had to be very simple so that they met the time budget, and that's typical in business," he says.
The model has to fit the business constraints, and those constraints need to be clearly spelled out in the design specification. Unfortunately, he adds, this kind of thinking often doesn't get taught in universities. "Too many people are just trying to build good models but have no idea how the model actually will be used," he says.
Bottom line: Failure is an option
If, after all of this, you think predictive analytics is too difficult, don't be afraid, consultants advise. Abbott explains the consultants' mindset: "You make mistakes along the way, you learn and you adjust," he says. It's worth the effort, he adds. "These algorithms look at data in ways humans can't and help to focus decision making in ways the business wouldn't be able to do otherwise."
"We get called a lot of times after people have tried and failed," says Elder. "It's really hard to do this right. But there's a lot more that people can get out of their data. And if you follow a few simple principles you can do well."
Robert L. Mitchell is a national correspondent for Computerworld. Follow him on Twitter at twitter.com/rmitch, or email him at rmitchell@computerworld.com.