WorldPay drives Splunk out of IT and into customer facing business roles

WorldPay, the payments processing company, is driving the use of Splunk out of the IT department and into the customer-facing areas of the business, to give them tools that its technical lead is describing as “management porn”.

Splunk, a big-data company that analyses enterprise machine data in order to improve business efficiency, is holding its annual user conference in Las Vegas this week and Computerworld UK spoke to WorldPay to discuss how they are using the software to deliver benefits.

Darren Dance, Unix technical lead at WorldPay, began using Splunk's tools on a free licence basis, which only allows 500MB of data to go through the tools each day. However, the company’s usage has radically increased in recent months and is set to soar to one terabyte of data being pushed through Splunk each day, by the end of the year.

“Because we are building everything at a rate of knots, we started using Splunk to identify issues that we hadn't realised were there,” says Dance.

Splunk in the IT department

WorldPay was initially using Splunk to identify problems on its Unix estate and within its firewalls. For example, it has created a firewall dashboard that allows the network engineers to quickly identify where and how a connection has dropped by typing a few key phrases – such as source IP, destination IP, destination port, timeframe – into Splunk. Previously engineers had to go into the various log files of the various firewalls and find that information for themselves, which was inevitably wasting time.

“The networking guys have about 20 engineers and they have said that it has saved them 10 hours each week with just that dashboard alone. One of the guys said to me he'd struggle to do his job without it,” says Dance.

Elsewhere, Splunk identified that after WorldPay stripped out a lot of services within its Unix estate, in order to make it more secure, it caused a lot of issues. Dance says that everything was getting very 'chatty', because the team had stripped out DNS caching, which meant that every time a request was made the DNS servers had to open a connection, which was slowing down applications. The team has since added DNS caching, after Splunk identified the problem, to speed up processes.

Driving Splunk into the business

However, Dance recognises that a lot of the real benefits of Splunk will be through use cases outside of the IT department and within business areas, such as customer services.

“We put these tools in place in December/January time, but going forward we are going to push them into the customer space, helping the customer service guys make better use of this machine data. There is all this data that they don't understand and we have still got guys that see Splunk as an expensive log archiving tool,” says Dance.

“It's only once that they have that eureka moment that they realise that there's loads of useful stuff that we can get out of the tools and make the business more stable, make it more secure.”

One of the areas where Dance is rolling out Splunk is in customer service, in order to reduce the number of hours that the help desk spends resolving technical queries from merchants. For example, within the e-commerce gateway there is a risk score associated to each transaction, where checks are run on each transaction. So, if a transaction has a risk score of over 100 the transaction is declined. However, the merchant can override these settings and they can assign different values to those checks, but this can result in all of the transactions being declined.

With the merchant not realising what they have done, they will then ring WorldPay and troubleshoot with customer services, which previously has had to spend hours going through logs trying to figure out where the problem occurred.

“So what customer services are going to be able to do is type a merchant code into Splunk, it then goes through all the log files from all the relevant systems and plots a nice, pretty, line chart on the page that shows you where the service dropped and when the merchant changed the setting,” says Dance.

“You no longer have to make a phone call back to the merchant to explain what the problem is, you no longer have them continuously phoning into the call centre, and it's visual for customer services. It gives them every transaction that the merchant has made for the past 24 hours and allows them to definitively say what the problem is.”

How to get that all-important return on investment

Dance explained that Splunk is relatively simple to implement, involving the installation of an agent that scrapes the log files on any machine that has readable data. As a result, he has numerous plans for driving Splunk into the business.

“One idea that I've got is to take the data from the F5 load balancers, which will give me the source IPs of where all the transactions are coming from. I can then plot in real time with geo look-up straight onto Google Maps and see where the transactions are coming from,” he says.

“You can have that up on the wall and it's like management porn, it will be awesome. You can say, for example if you have a problem in a specific location, well at this time of day in this place we usually have X amount of transactions.”

WorldPay has delivered some formal training to its business users, funded by Splunk. Dance is keen to get the business users on board because he believes that's where he will be able to derive a strong return on investment (ROI).

“It's very easy to show an ROI with the business users, where we can show exactly how many hours per week we have saved," says Dance.

"Also, if it saves the time of an expensive engineer, because previously he or she would have to go and explain to a customer service person how to solve a problem, then you are getting an ROI from the fact that you can bring some of the technical tasks down into the lower paid, lower skilled roles within the company. But still get the same value."

Copyright © 2013 IDG Communications, Inc.

8 highly useful Slack bots for teams
Shop Tech Products at Amazon