Charlotte Ward: 0:13
Hello and welcome to episode two hundred and seventy-nine of the Customer Support Leaders Podcast. I’m Charlotte Ward. Today, welcome Rob Dwyer to talk about how AI may be overcooked in support. So, hello, dear listener. I have hit record on this conversation because my guest today said, Why have you not begun on the uh on the recording front? Isn’t that right, Rob?
Rob Dwyer: 0:45
That is right. Here we are having an amazing conversation about pasta sauce and pasta noodles and the theory behind them and our opinions on them. And we had lost it all. It was ephemeral and it disappeared out into the universe without ever being witnessed.
Charlotte Ward: 1:04
And how divergent our opinions were as well. You were all about the sauce, and I was just all about the sauce, and in two very different ways, right?
Rob Dwyer: 1:15
Maybe, maybe we need to have our own podcast called All About the Sauce, and we’ll just talk nonsense about sauce.
Charlotte Ward: 1:24
Why doesn’t that exist already? Why doesn’t that exist already? By the way, by the way, I’m gonna copyright this idea, but I was just talking to a friend of mine and we had an awesome idea for a YouTube channel, which was middle finger theater. This is I love it. I love it. This is so that’s mine. Nobody registered that domain. That’s mine. Um uh but anyway, that’s only I’m only mentioning that by virtue of it being the second good good kind of um social content idea that I’ve come up with today.
Rob Dwyer: 1:58
It seems like you could have a crossover episode of Middlefinger Theater where they talk about sauce.
Charlotte Ward: 2:07
I I need to do that. I need to do that. Can we just stop this conversation? Like, let’s just go and do that.
Rob Dwyer: 2:17
Well, I didn’t know if I was invited to middle finger theater. I wasn’t gonna be presumptuous.
Charlotte Ward: 2:24
I mean, it depends how often you deploy the middle finger as to whether you’re you’re qualified or not, frankly.
Rob Dwyer: 2:31
I think my uh close associates would tell you that I’m more likely to deploy verbal middle fingers than I am physical middle fingers.
Charlotte Ward: 2:42
Yeah, yeah. I I think many of my associates would say the same. Um, which is not true of my friend who I who who I am uh referencing anonymously in this conversation, who who, by all accounts, physically deploys the middle finger when needed.
Rob Dwyer: 3:01
Sometimes you just need to let people know how you feel, and that’s that’s a way to do it.
Charlotte Ward: 3:07
That is a way to do it. That is a way to do it. I mean, I’m British, of course, so I fall back to good old swearing uh every opportunity I possibly can, whether it’s uh social or professional or or indeed on the podcast. I’m not shy over the odd swear word here, but we’ll try and keep it, even if it might get a bit saucy, Rob. Did you see what I did there? We’ll we’ll keep it quickly. I’m not good.
Rob Dwyer: 3:30
I’m quite brilliant, brilliant, as they say on uh your side of the pond. Brilliant.
Charlotte Ward: 3:38
They do, they do say that among many other English words.
Rob Dwyer: 3:44
It’s uh you know what? It’s a language, a common language divided by an ocean, something like that.
Charlotte Ward: 3:50
Two nations divided by a common language, I think.
Rob Dwyer: 3:53
Yes, that’s it.
Charlotte Ward: 3:55
That’s the one it’s okay. You can come here for all of the uh all of the you know me validation.
Rob Dwyer: 4:01
Full me twice can get fooled again.
Charlotte Ward: 4:06
Yes, yes. That’s that’s from your side of the Atlantic. Uh let’s not let’s not delve into that one anymore. I feel like uh we’re gonna be here a very long time unless we actually say hello, um, and you get to do an introduction, which should have happened several minutes ago, but we were having such a good time. Um, thank you for joining me today, Rob. Would you like to introduce yourself?
Rob Dwyer: 4:30
I suppose I will I will do that. My name is Rob Dwyer, and I have two middle fingers. And they have not yet starred in anything, but they are looking for work. Um, the rest of me actually has two roles. I work for two companies here in the U.S., both of them based out of St. Louis, Missouri, even though I don’t live there any longer. I work for uh a BPO, contact center BPO called Customer Direct. And our sister company, HappyTo. Uh HappyTo’s kind of flagship product right now has conversational analytics that includes sentiment analysis and automated quality assurance. So these are the things that I do once upon a time. I was in the mortgage world in 2008 happened, and I was like, well, look at that. I guess I gotta find a new job. And uh somehow it it uh landed me here.
Charlotte Ward: 5:39
Hmm. Happy, happy, uh, happy um two.
Rob Dwyer: 5:44
Happy two.
Charlotte Ward: 5:45
Happy to that’s happy to happy to good, good, good. Um, yeah, what what’s the what is so now I’ve lost the English language entirely. You know, I guess I guess a happy landing, happy to landing, but uh, you know, uh a good uh a good outcome from dark times or something like that.
Rob Dwyer: 6:05
It it was it was it was the best of times, it was the worst of times. And I say that because um it it really was. I was uh in dire straits from a career standpoint, and really I thought I was with a company that I was gonna retire with and have a pension and you know be fancy schmancy, and all of a sudden I was like, I don’t know what I’m gonna do with the rest of my life. Uh, but I was also um just in uh the the brand new throes of a relationship that would eventually blossom into my marriage. And so uh, and it was a long distance relationship at the time. So I was both madly in love and going, I don’t know, I’m gonna make money.
Charlotte Ward: 6:59
Madly in love and deeply in panic or something.
Rob Dwyer: 7:02
Yes, exactly.
Charlotte Ward: 7:03
Deeply in love and mad panic. I I don’t know. Easy crossover, right? Um, so Rob, I think we better get down to brass tacks and uh decide what we’re actually talking about today. What are we talking about today?
Rob Dwyer: 7:16
I think what you said was we’ll figure something out. I believe that’s that’s what you said.
Charlotte Ward: 7:24
So that doesn’t sound like me at all. Fly by the seat of my pants zombie. Honestly, just turn up and hit record in the middle of a conversation about pasta. That is not how this goes.
Rob Dwyer: 7:36
And I’m so excited that that’s not how this goes, uh, or so you say, because honestly, I have never talked about pasta on a podcast before. And so I feel very privileged to have done so today.
Charlotte Ward: 7:53
It’s a first time for everything. Me too. So it’s it’s it’s uh yeah, pasta podcast first for me as well.
Rob Dwyer: 8:02
Maybe what we can talk about is how AI is maybe being overcooked.
Charlotte Ward: 8:11
I saw what you did there. I can tell I’m with a professional. You’re not flying. This is all prepped, isn’t it?
Rob Dwyer: 8:18
It’s absolutely not, but you know what? I’ve been thinking, you know, we have been working on this quality assurance product for a long time, and and we’re using uh generative AI as is apparently the entire planet. Um and uh we’ve learned some things along the way, and I I’ll give you some examples. There can be wild, wildly different uh levels of performance depending on the task based off of the model that you use. But you can’t just judge your performance based off of a few data points. You really need to get a decent amount of data points in before you can accurately judge uh how a model is going to perform at a specific task. And that can lead to uh some false hope where you go, oh, this is great. Yeah, and then a week later you’re like, what the hell is this? This is what is happening here? Um and so there’s a a constant exploration of what’s available. But I also think the other thing is focusing on what a really good use case is, what a maybe use case is, and what uh I like to call uh oh hell no use case is. Uh in the maybe even the near uh future. And this is coming from someone who works in the AI space. Like I am not an AI naysayer, I am not someone who says you shouldn’t be using this, but I do think there are some ethical questions that uh we’ve probably just broken right past, and maybe we’ll never be able to put that genie back in the bottle. But as far as deploying it, I think there are some good opportunities out there where it can be a positive. I do think a lot of companies are just throwing AI at the wall, throwing stuff at the wall and seeing what sticks.
Charlotte Ward: 11:16
Seeing if it’s cooked, right? Seeing it’s like spaghetti, seeing if it’s we went full circle.
Rob Dwyer: 11:21
Yeah. And I think a lot of it is very undercooked at this point. Very, very undercooked.
Charlotte Ward: 11:29
I have a uh I have a question for you which might be a bit a bit meta, uh, but I’m gonna ask it because it’s my show. Um, which is when did we start or when did we stop uh treating uh AI the same as any other tool we might buy in a business?
Rob Dwyer: 11:56
I think that when the zeitgeist turns a tool into this mad rush, it seems like a phenomena, right?
Charlotte Ward: 12:10
It becomes not about the problem, it becomes the uh about the technology.
Rob Dwyer: 12:18
So let’s think about and I I hate to use this example, but I’m going to use it anyway. Not all that long ago, everyone was like NFTs and blockchain are gonna change the world. Oh yeah. And then we had this thing where people were trading NFTs for these ridiculous amounts of money, and I think there was just this feeling like here’s this thing that’s a sea change and it’s going to change everything radically, and people just lost their minds, right? It’s kind of like being in love, right? You just you lose all rational thought and you do things that in hindsight you go, ah, did I really do that? And I think AI is very much in the same way. I think open AI was brilliant in their marketing of this thing they called ChatGPT and the different products that they had, just saying, Hey, it’s free, try it. Anyone can try it, and it just became this thing where I like I vividly remember, and yes, I’m guilty of it, like creating a poem in the style of Dr. Seuss about insert topic here, right? It’s really good at doing that, it’s really good. But that’s not a problem to solve. Like I wasn’t rolling around in bed at night thinking, how can I create content in the style of Dr. Seuss for LinkedIn on a consistent basis? That was that never went through my head. And I I think we all got like AI fever, just like a lot of people got NFT fever or blockchain fever. And we still haven’t come entirely down from that high yet, I don’t think.
Charlotte Ward: 14:19
I am disappointed that that’s not what was keeping you awake at night, to be honest. But uh it takes all sorts. I I that’s nothing, I think about nothing else at night uh about except how to create Dr. Zeus, you know, styled content.
Rob Dwyer: 14:36
I was just thinking about pasta. I was like, how am I gonna make a better sauce?
Charlotte Ward: 14:44
I think we we we can discuss that once we stop recording, because we don’t want to give away the family recipe, do we? Um yeah, and I think I think you’re right. I think that there is, you know, that there is somewhere between toy and tool, which is the kind of zeitgeist and fever pitch kind of like rabidness to use this technology. And I am and maybe that’s the arc, maybe it’s toy to technology to tool that we actually need to go over to get to a point where companies aren’t thinking I just need to use AI. They are they’re actually thinking again about the problems they need to solve.
Rob Dwyer: 15:32
Yeah, that will happen. And AI will absolutely solve real world problems. It’s solving real world problems today. I just think the expectation versus reality, there’s a there’s a pretty big gap. And I I will also say part of the gap comes along with a price tag. If your expectation is I can very inexpensively solve this problem with AI, you may find that it doesn’t work as well as as you thought it would, or this thing that works really well well when you scale it, all of a sudden the cost is significantly different than you anticipated.
Charlotte Ward: 16:24
Yeah, yeah. I I think that’s absolutely right. You said something um quite a while ago at the at the middle part of this discussion where you you talked about, I think it was just after the pasta, once we started getting into the into the kind of discussion around the models and and the quality of the output is dependent on the models you use. And you talked about how we can kind of get excited about early, early success, and um, and then everything goes off the rails a week later. Um as we go through this this arc of kind of I think I think I’m settling somewhere on this kind of toy to technology to tool arc in that middle point where we’re excited about the technology and we’re seeing things like that, you know, it it seems successful and then suddenly not. Um have you got some examples that you’d be willing to share? And I I’m cognizant of the the point where you said um just after that that there are some brilliant applications, some okay applications, and some things that custom companies really shouldn’t be doing just now. Um, some awful ideas. Um what have you seen that’s been, you know, um let’s say on the good to great idea scale, but but has gone off the rails that seemed successful and then went off the rails?
Rob Dwyer: 17:52
Yeah, I can tell you just share with you kind of what our experience has been. Obviously, one of the things that we’re doing is trying to automate quality at scale. So for I think everyone in your audience knows what I’m talking about, but just so we’re clear. In the contact center world or the support world, quality is the process of evaluating traditionally a small sample size of customer interactions to identify what we’re doing really well, where we have opportunities for improvement, and are we um meeting certain requirements that could be regulatory in nature or hard and fast requirement on behalf of a client? But that process of taking that to scale comes with some really interesting advantages. But as you start to do that, you find what we’ve traditionally done is we’ve taken this rubric and we’ve said, I am going to look at these different elements of how my agent handles an interaction with a customer and and evaluate them and give them some feedback based off of that. And we have found that large language models can be really good at that, but they can also surprise you in interesting ways and really struggle. So there are certain things that we’ve found that they’re awful at um in general, but uh to different degrees depending on the model. So let’s just start with counting. Large language models are awful at counting andor counting within context of a conversation between two people. So if I just want to know um how many times I used Your name, Charlotte, through the course of an interaction. Man, that is harder than you think it is. Now, uh, some might argue that that is not a terribly important thing to be measuring, and I am probably not going to put up much of a fight on that. But there are other things that I might want to measure from a quantitative standpoint within um the confines of that conversation. And the models, they’re prediction models, they’re not math models. Yeah, when we look at these models, because they’re prediction engines of language, it is akin to two kids in in primary school. I think that’s what you say across a prime primary school.
Charlotte Ward: 20:50
You would say elementary, right? Or something, let’s say like that.
Rob Dwyer: 20:54
Exactly. Grade school, if we’re, you know, being really colloquial. But if I’m in primary school and I’ve got one kid who’s really great at like English and writing and poetry, and one who’s really great at math, but they both struggle in the other. The model is the kid who’s really great at poetry and English and writing. And then struggles and Dr. Seuss, and then struggles when all of a sudden we want to do basic edition, like two plus two, and it’s like five. No, no, it’s not. So that’s kind of where things are today in that world. And so as you go through and you’re trying to, there are all different kinds of industries that we support. So if you’re trying to help people identify revenue opportunities, it really becomes a challenge to get this right to understand like how valuable was this particular interaction potentially. And those are the kinds of things that we have found like sometimes a model is just crap. Like it’s just you just gotta go, nope, never gonna work. I don’t care how many times I change the prompt, it is just not going to be good at this. And there are models that are good at some things and not good at other things, and so sometimes you just have to identify what are the strengths of this model. Is that enough for me to use it for this discrete task? And if so, great. And I think what people really want to do is they want to find the one model to rule them all. I got news for you. It’s probably not gonna happen, at least not at a price point that makes sense.
Charlotte Ward: 22:49
Yeah, that that that that I think is is the holy grail, isn’t it? The one the one model to rule them all, the one tool that does three or four different things for you without having to go and seek out different things. Um I I mean the quality one is interesting for me as well because there are other and you know, there’s a you’re the expert in the room here, just to be clear. Um, but I’m gonna I one other thing that I’ve observed is I I too got all excited about um about using AI as a you know in quality. Um and I ran it alongside our own human QA process, but not very long, Rob, not very long. And and the reason the the thing I struggled with is that for the type of support we do, which is very long-running tickets with lots of consultation, with lots of testing. This can you try that kind of exchanges? Send me the log file. Almost every single interaction has problem or error, or try that again. You know, it it it the model I feel is centered on some kind of average that is not an is not a um is it is is way in the center of a bell curve that I’m very far out on the edge of. Yeah, if that makes sense.
Rob Dwyer: 24:32
I think the I think the misunderstanding in general about these models is that they think. And I am I am of the opinion that they do not think and they are bad at logic. And they get worse the larger the context required is for it to really do a good job. So, in your use case, right, lots of back and forth, uh long duration, that’s where I would expect a model to break down and to struggle because it’s just not what it’s good at. If you want to ask a model about a discrete interaction and identify how well an agent stayed on topic or um exercised politeness or professionalism, these are things that a model can identify pretty readily, pretty easily, and give you good results. But we’ve also found, too, we’ve got some partners that what they’re looking for is very specific, very technical, um, and very uh when I say specific, specific to a particular industry, a highly regulated industry. And so you if you’re not uh using uh a rag model, which is retrieval augmented generation, which uh for those that don’t understand this, the easiest way to understand it is I’m gonna give you my knowledge base and I’m gonna pair my uh my LLM, my gen AI, but I’m gonna tell it, hey, make sure you go check all of everything you respond with, check the check the KB, right? Go review the KB and then come back. You can do some interesting things with that, and I do think that that is the future for a lot of industries. If if we do a really good job of curbing the hallucination and using rag models, it I think has a lot of potential. But if I’m not doing that, if I’m not going that far, then there are limits to what I can do. And I think everyone just needs to be realistic about where those limits are for their business, uh, their budget and their patience. And maybe temper their expectations, or maybe get excited and go, hey, I guess this maybe is more powerful than I gave it credit for, maybe in my business, but for this discrete application, it can do a lot of really cool things for me. But I I don’t think that’s where most business owners or businesses as a whole are. I don’t think they’re underestimating. I think they’re on the other side of the spectrum and overestimating. And to your point, when we talk about the you know, the one technology to rule them all, I think we’ve all experienced these large behemoth technology companies that buy all the solutions so they can be this one-stop shop for you, and they integrate it all into this Frankenstein’s monster type of application. Rarely do you go, I love everything about that. Because you’ve got these different code bases that were developed independently, and we’re just kind of shoehorning them in like it’ll fit if you just push hard enough, and the toes are like all curled up inside, and you’re like, see, it fits. And you’re like, Yeah, but I don’t want to walk very far in these shoes. I think most of us have been down that road.
Charlotte Ward: 28:45
Oh, we certainly have. Uh, you put me in mind of something that uh this is a slight, this is just a little anecdote for the sake of an anecdote. But um, when I worked, I did a year at Exxon when I was at university. I worked um in application support on site at one of their chemical test plants in the UK. Very, very interesting place to be in the mid-90s. Um, I worked with, I worked in the same little tiny cubbyhole of an office with the soft the software engineer who was developing the in-house, like, you know, homebrew application, which was managing inventory, blah, blah, blah. Anyway, um, I recall one and I used to support this application uh with all the chemists on site. Um, it didn’t behave very well. Uh, and and the way he described it to me was that this, and I’m I think I can remember the name of the application, but I’m not going to attempt it because someone will be listening and correct me. Um, but he said, yes, well, when we first built this thing, it was beautiful. It was, it was designed to be a ball that would roll and bounce and behave predictably in exactly the way you expected a brand new shiny bouncy ball to behave. Um, and over time we’ve just kind of stuck on all this extra functionality, all these different functions, all these different use cases, all these other bits and pieces that we’ve magpied from across, you know, other other parts of the industry, other other um other oil companies, etc. Uh, and now what we’ve got is kind of this this kind of there’s a ball still in there somewhere, but there’s these bits of gum and and nail sticking out and like all these lumpy bits. And basically now, if you try and roll the ball, it just kind of goes clunk, clunk, clunk and then disappears off into the corner. And that’s that’s what you’ve put me in mind of a bit. And that’s that’s exactly the experience of some of these big, big BMO companies, isn’t it? It’s exactly as you describe.
Rob Dwyer: 30:51
Yeah, and I think look, I get why these companies do this. It’s easier to gain market share, number one, by buying customers when I buy a solution and integrate it. And as I get you more, it’s it’s a way for me to get my hooks in you even more, right? You’re not just relying on me for this one service, you’re relying on me for 20 services. That makes it really hard to change providers, so it’s a way to create sticky customers. And look, every company wants sticky customers. That’s absolutely what you want. But it does create a not great experience at times, and I think I think some companies want to have vendors who are excellent at one or two things, and they will get them to talk to each other and work with each other, but that’s sometimes a better experience than having this one vendor that is like at most things, but it does most things, and you’re like, well, it sort of works, yeah, yeah, yeah.
Charlotte Ward: 32:17
I couldn’t agree more. I couldn’t agree more. Um, Rob, how did we end up here? I I feel like we we covered quite a lot from pasta to that weird ball that kind of doesn’t bounce and and roll where you expect it. And uh we threw some stuff at the walls, didn’t we?
Rob Dwyer: 32:34
Mm-hmm. Well, you can cover a lot when you’ve got uh lots of carbohydrates uh ready to go for a long journey. And so we started with carbs and we’ve been able to run a long race now.
Charlotte Ward: 32:49
That’s that is very true. That is very true. Uh I’m gonna I’m gonna uh go and just enjoy the rest of the the carbs for the evening though, and uh and and settle my settle my uh settle my digestive juices on on both that my my bowl of spaghetti in case anyone’s listening, and um who’s interested, and uh and everything we’ve talked about as well. I need to stew this over. Um super, super interesting meander through the the you know the the good, the bad and the ugly of uh of bringing AI in and like when to do it and uh and actually why you should do it or why you should not do it. And uh I I guess you know you used a word right near the end there which really struck with me, which was patience, right? I I think don’t don’t rush into it and uh uh take time to cogitate cogitate and uh digest. There we go.
Rob Dwyer: 33:52
I love it. I love it. Well, it’s always great talking with you, Charlotte. So thank you.
Charlotte Ward: 33:57
Thank you so much for joining me. Uh please come back. Please. Yeah I’d love to have you back. Of course. That’s it for today. Go to
