Q&A: What happened when a customer support company upgraded some features to ChatGPT?

Fergal Reid, director of machine learning at Intercom, whose customer support software is used by 25,000 enterprises, just rolled out some new features for clients powered by ChatGPT. It turns out that there are still some things to get better.

1 2 Page 2
Page 2 of 2

"We have an internal [customer success] team that we use as like an alpha customer — Intercom has maybe 100 [customer success] reps. So, we would ship prototypes very quickly to them and get their feedback on the model. But we’re not using them to train the model. We’re just using them to as alpha testers to help identify problems and identify where it goes wrong.

"There is a lot of work to it. It’s very easy to come up with a compelling demo, but there is substantially more work to make it function in production. So, having a team of people who can look at it very early on — we don’t know if this is good or bad. Is this a toy? Will this help reps be more productive? Please tell us. That was really helpful. There were several other products we prototyped, but didn’t ship because they were toys."

Do you see this chatbot product eventually being shipped for end-user use — without a customer rep middleman, so to speak? "We’re investigating that at the moment. We’re not quite ready to share about that at the moment, but yeah, definitely this type of technology we think will soon be ready for use by end users. A lot of people are trying to work around these problems of hallucinations, and I think we’ve seen examples of them. Google had a launch recently, and at the launch it appeared their model presented something that was factually inaccurate, which disappointed a lot of people.

"So, everyone is having to figure out how to deal with this problem of occasional hallucinations. We are working very hard on that and we’re optimistic, but we don’t have anything else to share at this point."

How much time and effort has the upgraded AI-enabled features saved clients and their customer service reps? Has it cut their customer response times by a third, a half? "No. Probably not that high. I don’t have hard numbers because this is so new. We’ve got telemetry running against it, but it’s probably going to be another few weeks before we’ll have number. It’s a hard thing to measure.

"I’d say something like summarization can save around a minute or two minutes potentially on a 10- or 15-minute conversation. Something like that. That’s a soft interpretation of the feedback we’re getting, but also what we’re seeing. It’s definitely real and there’s a lot of excitement around it. Ever since going through a public beta, you can go on Twitter and find Intercom customers posting about how it’s saving them time. You don’t have to take our word for it.

"I also think everyone in this space has a challenge of keeping themselves honest at the moment. This technology is so obviously exciting, it’s very difficult to be sober about it and not over exaggerate. This is a machine to build compelling demos that don’t currently deliver real value. So, there’s a lot of work to do to understand how much real value is being delivered here. So, we’ll dig into that, but we wanted to ship early and have our customers tell us what they think of it.

"The reception has exceeded our expectations. There are a few features like summarization that are clearly valuable and then other [good] features like the ability to rephrase your text or make your text more friendly. Or another feature we have is the ability to write a short-hand version of your message and expand it out — customers have responded extremely strongly to those features."

So, you don't have any hard data on just how more efficient this makes a customer service representative? "Honestly, we need to look at our telemetry in a month or two and determine if they kept using it every time. I’m confident they will, but we need to check. I think this field overall is still working on the killer apps for this.

"We’ve got this amazing demo with ChatGPT, which has really gotten everyone to pay attention. But, there are some companies like Intercom that are determining how we can turn that from a toy into something with real business value. Then even at Intercom, we’re like, ‘We’ve shipped away these features and they’re cool. They seem like they would be valuable, but they’re not game-changing yet. They’re not making a customer rep twice as fast or three times as fast. How do we do that?'

"I think that’s the next wave of what we’re working on now, and those are longer development cycles. Those are not as simple as integrate quickly and get it out there. You’ve got to go really deep and understand the user problems and all the different facets of it and where did it fail. So, we’re working on that now. Probably a lot of our competitors and people in the industry are also working on the same problems and creating more valuable features.

"We had a pretty fast development cycle and got it out fast and got a lot of great customer feedback and that helps us decide where to go next. That’s the honest take on where things are at."

There’s so much hype around ChatGPT now. How do you deal with that while trying to temper customer expectations around your product? "Our actual strategy here is to try to be scrupulously honest about our expectations. We feel we can differentiate from the wave of hype by being honest."

How did you go about getting your customers to opt into using your new ChatGPT-powered bot? "At Intercom we have more than 25,000 customers. And, to a lot of those we say, ‘Hey, we’ve got something in beta. Would you like to opt into it?’ Some customers have just opted in and said in general they’re willing to use early software. And some customers won’t. Some customers are very risk averse enterprises, like a bank, and they don’t want to be part of the beta program.

"When we do have new software, we’ll send them a message to recruit for the beta. Our [project manager] for this set the campaign live and then had to pause it like after five minutes because her inbox was flooding with so much excitement about this. So, that’s what we did. We recruited a couple hundred customers to beta test this in mid-January. We said you have to click here to opt-in and use the API to process your data. Then customers did that and we turned the features on for them."

"Then we started looking at the telemetry the next day to see where people using this and is it working for them? Then, and this is how we normally run a beta at Intercom, we reached out to them and said, ‘Hey, can we get your feedback from this. We’d like to know if you found this valuable.’ And some customers were generous enough to give us actual quotes after a couple weeks that we featured on our blog.

"Again, we wanted to differentiate from all the hype. So many startups are creating just a landing page that’s just a thin skin over ChapGPT, and we were like we’ve actually got a product and these are actual customer quotes on our website to show it’s something real."

Copyright © 2023 IDG Communications, Inc.

1 2 Page 2
Page 2 of 2
It’s time to break the ChatGPT habit
Shop Tech Products at Amazon