Replay: The value of business simulation (Part 2)

In the previous installment, we looked at and discussed strategies for business simulation and the infrastructure needed to make such initiatives successful.  Now, we’re ready to discuss some practical examples of business simulation.  Imagine a mail order company selling products together with the necessary financing. Assume they’re considering replacing one of their credit risk models while at the same time trying to boost sales of a certain widget. Let’s further assume that their overall decision strategy to determine the best product to offer to a customer may be overruled in circumstances where the risk model determines that additional selling is not desirable because the underlying loan is too likely to default.

In the example, this company makes two changes to its in-production decision strategy. First, they replace the existing credit risk model with a new version. Second, they multiply the outcome of the propensity model for the widget by some factor greater than 1 to make it more likely to be prioritized as the product to offer. As the next step they want to apply this revised strategy to a selection of the recorded data. As stated above, every single product recommendation and every credit risk evaluation has been recorded. Because their mail order business sells fashion in addition to other products, it is sensitive to seasons. So to understand the new strategy’s effect during the summer they decide to apply the new strategy to last year’s interactions over the same period and study the deltas.

Slice and dice data

Once that slice of recorded data has been loaded, the company may take a sample from it.  With so many millions of interactions recorded, a large enough sample will be representative of all of them. They will then proceed to apply the revised strategies, with all its predictive propensity models, risk models, and rules, and look at the distribution of the results. How many more widgets will be sold? It’s possible to simulate this because the company is using propensity models to predict the likelihood of a customer accepting an offer for a widget. Thus, the change they made to boost the offer rate of the widget should see more (simulated) interactions where the widget is being offered and accepted by the customer. Unless, that is, widgets are expensive and it turns out the new risk model will reject more widget offers in favor of lower prioritized products (per the new strategy) that keep the company’s exposure within the desired bandwidth.

The company can thus study both metrics. How many widgets would we have sold if this had been the marketing strategy used during last year’s summer season?  And how many write-offs on the financing would have been the result of using the new risk strategy alongside the new, Go Widget, sales strategy? If the metrics show favorable improvements the new strategy can be taken into production. If not, the marketing and sales teams and their colleagues from the risk department can tweak their strategies and see if it makes the desired difference when applied to last year’s interactions.

Cause and effect

This simulation is not perfect. For instance, last year’s economy may have been worse than this year’s, allowing more customers to pay back their loans now. Unless some economic data is part of the credit risk strategy, the overall strategy may not be sensitive to it and the simulation will therefore miss it. And a causal chain of events will also be increasingly hard to predict. If the revised strategy would have offered product X instead of Y to a customer, the actual service interaction about a problem with product Y which is part of the recorded data didn’t actually happen. So while it’s quite possible to predict the one-time effects of a strategy change, simulating the downstream effects of those new outcomes quickly becomes less useful conjecture. There are other caveats as well, a bit too detailed to cover here. However, don’t compare this with a hypothetical oracle that can tell you exactly how your strategy will fare, compare it to the common practice of making changes and hope for the best.

The more explicit a company is around the decision strategies that govern its processes – customer processes or otherwise – the fewer surprises. And when those decisions are based on predictive analytics and carefully recorded data it becomes possible to simulate future business outcomes by replaying the past, and making the effect of changes, even in complex strategies, more predictable.

Computerworld's IT Salary Survey 2017 results
Shop Tech Products at Amazon