Charlotte Ward: 0:14
Hello and welcome to episode 278 of the Customer Support Leaders Podcast. I’m Charlotte Ward. Today, welcome back Cat Gains for episode five in a six-part series on incident management. I’d like to welcome back to the podcast today. Cat GainsCat, lovely to have you back again for I think what we figured out is part five, although the sixth episode of our mastering incident management series.
Kat Gaines: 0:50
Am I right? I think that’s how the math ended out. This is going to be the sixth episode. People are hearing technically the fifth section. So, you know, listeners, call it whatever you will.
Charlotte Ward: 1:03
Yeah, yeah. Um nice. Well, it’s uh it’s been a journey of unpacking um incidents in the running, but we’re talking about what happens next, aren’t we? All the fanfare and all all the all the firefighting is done. Everybody’s had a moment to take a breath. Um, and we’re talking about what happens next, right? What are we talking about today?
Kat Gaines: 1:28
We’re talking about after the incident, which if you’re listening, you might say, okay, well, the incident’s done, so we’re we’re done, right? That’s it. We’re good. And that is very wrong. Uh, there is so much to do after the incident itself. Ideally, with the primary objective being we want to learn something from what just happened to us. We want to take a beat and take a breather first and just give ourselves a minute to digest everything, just like what is it? They say don’t that old myth about don’t go swimming some certain amount of time after you’ve eaten. Don’t uh don’t go swimming into the deep end of reviewing right after. I’m not going to compare the incident to a meal because it’s not it’s not that satisfying when an incident happens, but right after things wrap up, you want to give yourself a little bit of time to just digest and process first, and then we we have some juice, juicy learnings usually to pick out from whatever just happened.
Charlotte Ward: 2:27
I feel there’s a weird kind of foody thing going on there, like they’re juicy and they’re like, you know, we don’t want to go do I dive in the deep end after we’ve that happens in conversations with me a lot.
Kat Gaines: 2:39
Yeah.
Charlotte Ward: 2:41
Hopefully, well, we’ve not made a meal of an incident. Hopefully, we’ve done it the right way. But yeah, yeah, maybe maybe that’s the maybe that’s the metaphor we were going for. Um, yeah, cool. So so first of all, take a beat. Everybody go get at least a cup of coffee, right?
Kat Gaines: 2:55
Yeah. For heaven’s sake, yeah. I think we have we have a temptation to sometimes dive into uh, and people will call them different things. So I think you and I will probably talk about it interchangeably during the conversation, but people call them postmortems or incident reviews, or sometimes you even have other terminology you prescribe to it. I think a lot of us have uh tried to start using incident review a little more because it’s a little more of a gentle word than post mortem, which sounds very intense, right?
Charlotte Ward: 3:24
I know one of our uh one of our customers likes to say, I don’t like calling them postmortems. Nobody died.
Kat Gaines: 3:30
Yeah, no one died. It’s not that big, which is fair. And so if you’re not calling them that, that’s okay. We might still let that slip out from just you know years of training in this conversation.
Charlotte Ward: 3:40
But I guess there’s um I guess there’s retrospective as well. That’s another one.
Kat Gaines: 3:44
Retros, that’s another one. Yeah, all kinds of things we can call them. There’s a temptation sometimes to say, oh, we gotta jump right in, we gotta take everything that we just learned and we gotta put into action right now, or we need to understand exactly how this happened. And there’s a lot of pressure, and we were talking about anxiety in previous episodes. There’s a lot of anxiety around that too, where again, you have a responsibility as a business to your customers and uh anyone else who is invested in understanding what’s happening in your business to explain how did we get here and how are we going to try and avoid this again? But you do still have to take a moment to take a breather, especially if it’s an incident that took a little bit of extra time and energy to work out. You need to give everyone a moment to say, okay, how did we get here? But first, you need to just make sure your mind is right, that you’re able to collect yourself a bit. And will I really advocate for folks stepping away completely after an incident? Whether it’s you need to go take a walk or something in that area.
Charlotte Ward: 4:49
How long is the right amount? I mean, I my gut feeling is that some things you can do after a quick walk, after a cup of coffee, but sometimes the really high stress ones, you need a day, actually. And I think there are internal factors driving how long you you take that that intermission for. And sometimes the second factors, I mean, if you’ve got customers clamoring for like the the the breakdown on how you’re gonna make sure this never happens again, you know, which some custom some environments I’ve been in uh can be like that, you know. Uh and others are just like you’ve got to update a status page later this week and it’s fine, you know. Um yeah, there’s there’s all sorts of factors that drive the when, isn’t there?
Kat Gaines: 5:33
Right. There are all kinds of factors. And I think that’s important to know that you you are probably not going to be able to escape those customers who are incredibly hungry for information right away, and especially high-profile customers for your business might be a little bit more pressurized just to get that information out. Uh, something I see as relatively uh standard or a pretty common choice for folks is anywhere within the three to five business day range to report the full breakdown of what happened in the incident. And of course, you are going to have customers who are going to say, I need to hear it today. I need to know after the incident wraps up. And so I was talking with someone else about this recently, where you can have both options. You can mitigate that hunger for immediate information. And you can also say, okay, we’re going to take time to fully discuss what happened here as a full report and bring you the full diagnostic report. But the the step to kind of mitigate that immediate bit of how do we know exactly what happened, or how do we at least have faith that you have an idea what happened as a customer right after an incident wraps up is to prepare some preliminary reporting for them. Throughout the incident, you’ve probably been discussing, and again, we’re discussing this series from the perspective of customer liaisons, right? And so this is someone who’s been listening very carefully throughout the whole incident. They’ve been managing communication, they’ve been noting what’s been happening. And so you could even allow them to draft this statement that can be part of their responsibility to say, here’s a very high-level overview of what we experienced, what we did, and you may or may not include the what are we going to do about it in the future step. If you have information that early and you’re sure it’s not going to change, go ahead and do that. But be cautious about doing so because that is one factor that will definitely update a little bit after internal discussion. But you can at least grab those other pieces of information, draft a statement, and then again bring it back to the rest of the incident team, like the incident commander, et cetera, and get that approved for your other customer-facing peers to share with anyone who’s super anxious and just needs some answers with the caveat that this is not the full context, and we’re going to have more breakdown for you in a couple of days. But here’s what we do know and what we can share right now, just to appease the trust that yes, we do know what’s going on here. We are we have not just been flailing this whole time.
Charlotte Ward: 8:07
And I think I think there’s a couple of really important points. One is that um the the maintenance of trust that your customers give you is important at these times of of high stress on both sides. Um and you should treat that as an opportunity to you know build and strengthen that trust. Um and therefore, in the moment, you should only impart um information to your customers that you are sure of and that you are sure will not change and that you are sure has to conjecture. Exactly, exactly. Because, you know, it’s just, I think if you if you are shooting from the hip, for want of a better way of putting it, right? We think it could be this, we think it’s that, you know, we’re gonna try. Even if you’re loading some caveats around certainty, I think customers just, you know, they’ve gone through a stressful moment as well. They don’t necessarily take on board any caveats that you might put in around expressing the level of certainty in that communication. So just say what you’re certain of, right? At that point.
Kat Gaines: 9:22
Yeah. And if you think about, you know, playing a game of telephone, for example, which we’ve all done at some point in our lives, where your original message is completely garbled by the time that you get to the other side. If you’re saying we think, then you’re going to have one of two things happen. That thing that you said you think is the case, but may or may not be completely certain of, is going to be taken completely as word and as the true fact around what happened. Andor as they pass it on, let’s say you’re talking to someone in your customer’s company, they pass it on to their leadership, it goes to their executive team. That may have taken on a different meaning by the time that it gets to them. Even if you’re passing on the facts, you can’t fully prevent that happening. But if you are passing on as close as possible to a message of complete certainty around what message you want your customers to hear around this, you’re going to reduce the risk in that eventually happening, in that eventuality.
Charlotte Ward: 10:22
Yeah, yeah, yeah, yeah, absolutely. Um, and so just in terms of that initial communication, the kind of things that you can be certain of, um, because you wrote them down during the incident. The the the time frame, we know when this thing started and when we absolutely we know what the symptoms were, hopefully. Or we’re gonna state what we know the symptoms to be, even if we’re we know some of them at least. We know some of them. Yeah, there might be uh unseen things which we will unveil over the the course of a retrospective, but we’re not gonna tell customers about those right now. Um, and the other thing we know with certainty is that we are gonna do a retrospective. So you can promise that, can’t you?
Kat Gaines: 11:04
Yeah, yeah. We know for sure that we’re that is our next step. We’re going to meet, we’re going to discuss everything that happened here, and we’re going to come back to you with more description around every piece of this thing so you can feel at ease with our analysis.
Charlotte Ward: 11:20
Absolutely, absolutely. So you’ve sent that first communication, then everyone’s gone and had a coffee or a walk in the park or a good sleep. Yes. What happens next?
Kat Gaines: 11:32
Well, then it’s time for a meeting. It’s time to meet and discuss it as a team. And so again, that could happen something typical we see is three to five business days. If sooner or later fits your business needs, that’s okay. Just make sure that it’s really fitting the needs of everyone in your team getting that breather. And on the other side, your customers and stakeholders getting timely communication. But get everyone together live to discuss. I think it can be really tempting to discuss these things asynchronously because we may do that in our normal working days, or that may have been even how we discuss some of the incident itself. But it is important to allow the space for people to have live conversation, discussion, to bring things up that others in the room may not have been aware of. And so as a customer liaison, you should be part of that meeting. And I think that’s the other temptation that happens too, that uh the rest of the incident team will say, ah, well, customer support did the communication during the call. They don’t need to be part of that meeting. But something we discussed early on in the series is that you are not just a recipient of data as a customer-facing team member, you are a source of data. You are hearing symptoms around what your customers faced during the incident. Again, that may not all be visible from the monitoring perspective. You have heard how that impacts your customers. You’ve heard how it broke either their experience of or trust in the product. You were probably able to see a little bit of information around kind of unknown pieces that were a struggle during the incident itself. And so you can bring that helpful signal in and you can highlight, for example, maybe we didn’t know to look for this type of issue during this type of incident. Maybe that’s something we can learn from and that gets pulled into the process later. And so if you’re not being included in these incident review meetings, that’s one thing to start asking for and updating process internally right away to say we do need someone from our team, ideally the person who was playing customer liaison during the incident itself. But if for some reason we can’t get that person a backup or leadership, but someone needs to represent the team and what they experienced from the customer perspective, but also the process perspective from their side too. Did it work? Did it function the way it was supposed to? Yeah. When we brought drafts for approval, did we get timely response or were other people butting in where they weren’t needed? Those types of things. I think that’s really just has to be reviewed too.
Charlotte Ward: 14:10
Yeah, and I think this is a really key point, isn’t it? That your retrospective is not just retrospecting, if that’s the word. Um, it’s not just retrospecting the technical incident landscape itself, the the landscape of the issue, the mitigation, and the final resolution. It’s the whole process. This is an opportunity as well to improve your incident management process itself.
Kat Gaines: 14:37
Yeah, exactly. You’re future-proofing the process so that the next person hopefully doesn’t encounter the same struggles. The same way we don’t want the next person on call to experience the same struggles with troubleshooting the issue itself. We don’t want the next person who’s running communications to have a very easy to remediate issue around the communication process or any part of the incident process.
Charlotte Ward: 15:01
Yeah, yeah, absolutely. Um, and one other thing um I think that um we forget is is, or at least I think it’s possible to forget, is that customer support or your customer liaison is also often part of the mitigation cycles, you know, and the testing cycle. So they actually do have input in from a technical perspective as well, quite often.
Kat Gaines: 15:24
Yeah, they really do. And it’s important for our folks who are not in support, who may be hearing this, to remember not to discount that, that your support team are still technical personnel. I think there is probably no other pet peeve I have that’s much larger than hearing about technical versus non-technical roles, which I understand is a state of being that it’s possible, but the way that it’s used, it’s almost weaponized in our industry a little bit to be able to say, oh, that person’s opinion isn’t important, their perspective isn’t going to help us, they’re not going to have anything to contribute here. And so it’s both realize the the nature of a support role in uh in a software company, in a technical support role is still often technically involved with the product. And even if you don’t think that’s the case, second guess yourself and how you’re using that terminology and what you’re losing by excluding that perspective.
Charlotte Ward: 16:21
Yeah, yeah. Um you know, I think uh if you don’t enter that room with confidence in the skills that you’re bringing, and the same is true in the middle of the incident as well, then there is every likelihood, not not guaranteed, but I think it’s a fair um how do I want to say this? Slightly politically. I think there’s a fair chance that the you know, to use that technical language, the technical people in the room will have a tendency to ride roughshod in some situations, I think, because you’re not saying, actually, I don’t have your skills, but I’m differently skilled. You know, I’m here because I’m bringing the customer voice or the product voice or whatever it is, you know, the something that is just not okay. I don’t have my head in code, but actually I understand this bit of the equation differently to you.
Kat Gaines: 17:17
Yeah.
Charlotte Ward: 17:17
You know, I think that’s right.
Kat Gaines: 17:19
And we need that, we need that perspective. We need that different, we need a voice outside of our own head to help us understand a full picture of something that happened. If we’re only ever narrating our version of what happened, we don’t hear that other experience from other parts of the organization. And we therefore have that blind spot where we can’t improve because at the end of the day, we’re unwilling to if we’re not looking for that information.
Charlotte Ward: 17:43
Well, we don’t see it, right? And I think it then become you then end up enter very similar territory to engineers who are writing their own test code. You know, it’s kind of you’re only testing for the expected all the time. And in a retrospective, if you’re not being challenged, you’re only drawing the same conclusions from your own perspective. You’re you’re you’re drawing conclusions from your own perspective that are um that kind of almost absolve your your part in it, you know, as a as as potentially a trigger, you know, for the incident, not at a personal level, but that bit of code you created, or that that alarm that, you know, rang, or, or you know, that that process that broke down or whatever the thing is, I think, I think you need, and this is the point of the retrospective ultimately, is to unify all of these views to find, you know, there’s there’s that parable, isn’t there, of the, or, or, or this kind of story of like the blind man and the elephant. And if you only see uh, you know, if you ask one person what this thing is, one blind man, what this thing is in front of them, if they’re stood by the leg, they’ll say it’s a tree. And if they’re stood by the trunk, they’ll say it’s a snake, you know, and and so on and so on. Um, but you need the whole perspective to understand what this elephant is. We’ve moved a long way from food analogies at this point, I think. So but you know, the the the rounded picture is important, isn’t it? And and yours is as valid a story in there as anyone else’s.
Kat Gaines: 19:17
Yes, yeah. I think that’s very important for everyone to remember.
Charlotte Ward: 19:22
So we’ve called the meeting, everyone’s come in, everyone’s got other things to do at this point. I think that should be recognized. We’ve all moved on, right? Nobody really wants to be there for a myriad of reasons. Uh so what so how do how do we conduct this meeting? Who conducts the meeting, I guess, is the first question.
Kat Gaines: 19:43
Yeah, so you’re absolutely right. This is interrupt work at its finest. Uh, no one expects to be here. No one is hoping to be here. Everyone wants to actually be working on things that continue to move their bottom line forward, but you’re gonna have everyone involved in the incident. Uh, in the meeting itself. And so ideally, the team who owns the piece of the product that was impacted is owning the postmortem or incident review process and the meeting. So they’re setting the agenda and walking through that meeting with everyone involved.
Charlotte Ward: 20:22
Uh so why why is it important that it’s that person or that group of people? And and the reason I ask that is that um in the process that I operate right now, it’s generally the incident commander who takes on sharing the retrospective. Is that not the right thing? Is that is it better if the product lead takes it on?
Kat Gaines: 20:45
It’s it’s gonna vary, right? So you can have your incident commander run it, or you can say this team is responsible for this part of the product. So they’re going to be most intimately familiar with what’s happening in that area, and therefore they should be running the meeting. Both approaches are completely valid. The incident commander just needs to make sure they’re pulling in that context from the owner of the impacted features or functionality as much as possible. And of course, they’ll be doing so, but often it helps to have that team write up the initial draft of the summary of what happened and then call the meeting, discuss it, and then start talking about what do we need to add, what do we need to take away, what have we missed in this process? Uh, and so it’s it’s not really the first time that you’re revisiting the issue itself. It’s sort of the secondary revisit after that write-up. And you’re coming in and saying, what did we miss so that we can, again, future-proof as well as we possibly can. Uh, so that’s one reason why we’d recommend having that team running it. Again, incident commander, completely valid choice. They just have to be sure to bring that in.
Charlotte Ward: 21:55
I I think it’s a really good point to um have as much of a write-up of the technical failure. Let’s call it a failure or an or an anomaly or something, you know, that this thing that went wrong at a very technical level, write it up in a way that is accurate but accessible for everyone that’s gonna be in the room because digestible, yeah. Digestible. Because once you’re in that meeting, they are there are gonna be people of differently skilled people. We won’t say non-technical people, there are gonna be differently skilled people in that room. And actually, it’s good practice to be able to write this up as an engineer in a way that you can explain it to everyone in that room, isn’t it? That give people time to digest, right?
Kat Gaines: 22:37
Yeah, and then no matter what, you’re going to have hopefully the full team in the room. So you’ll have the incident commander, whether they’re coordinating or not, uh, anyone who shadowed like a scribe or a deputy, anyone who again owned the service itself that was impacted, or anyone who played a key subject matter expertise role, anyone who is maybe an engineering or product lead for those impacted systems. And then, of course, as we’re discussing your customer liaison, you do have to be there and you do want to be part of this conversation as much as possible.
Charlotte Ward: 23:10
Because one of the outputs is more customer comms, isn’t it? So it’s really critical that you’re there. So so everyone’s in the room, everyone has hopefully had some write-up of a technical nature that they’ve understood. Um, how does the meeting run? Whoever’s running it, how does it run? What are what are we trying to get through in this time?
Kat Gaines: 23:35
We’re trying to get through a couple of different things. You want to make sure everyone’s on the same page about what happened at the end of the day. That is one of the key goals coming out of this, right? Uh, and you want to make sure that any information that wasn’t captured in that write-up was missed. So whether it’s technical or process, either failure or just things to remark on or notice or improve on later, those are the types of things that you’re going to be discussing. And so what you’re getting out of the conversation is really talking about what happened, getting a full narrative and recap. Uh, that should be hopefully relatively quick to pull together once you have that write-up. You shouldn’t be spending 20 to 30 minutes recapping, hopefully, uh what happened, but it should be a little bit quick. Everyone is on the same page. And then you want to talk about what is still happening, anything that’s been done so far. And you want to talk about next steps. Those are some of the key pieces you want to get to in the meeting itself. And then there are going to be the deeper dive pieces in each of those to again ensure that everyone’s voice is being represented, that everyone has a chance to add to something, or even to say, you know, I don’t think that’s completely accurate. I seem to remember it happening this way instead, or to ask questions about why we are pursuing these steps and maybe not others. And one of the key pieces of that conversation, as you can imagine, these sample questions and topics that I’m bringing up can get a little bit emotional for people. Uh it can be something that folks hold very close to their opinion of how they work. It can be something that people feel shame around, especially if there was human error involved in the genesis of the incident itself, right? And so something that you also have to be careful to practice and teach within your org is to try to run this process as a blameless process as much as possible. It takes it’s so hard. We have a whole we have operation guides around these things that write up the process piece of all of these topics we’ve been discussing. Uh, from I guess I would say primarily an engineering perspective, but the customer liaison and support perspective is also included within that. So I’m gonna correct myself and just say from an across the organizational perspective. And there is a whole section we have on blameless postmortems and how very difficult it is to do that. It is a very carefully acquired human skill because we have this just completely ingrained impulse to place blame for something that happened, to say, oh, well, that happened because of this. And that person made this happen. So it’s their fault.
Charlotte Ward: 26:33
And at this point, everyone is a little bit kind of wounded, aren’t they? Right? Yes, they are. Exactly.
Kat Gaines: 26:38
Like we were saying, yeah, yeah. Unplanned work. We didn’t want to be here, we didn’t want to be doing this, and so we have to fight that impulse and that tendency to just blame people and believe that every issue is because someone else did something wrong, to be able to say, okay, you know what? Things went wrong here. What enabled that person to make that mistake? What process was lacking, even if it’s as clear-cut as it of a human error as this is a bad deploy that caused an incident? Why was that person allowed to make that bad deploy? What process was not in place that that could have had another set of eyes on it? Is there something else we could have done systematically? And this isn’t to say that people are never at fault, but you need to be able to ask those how questions. How did we get here? And what put us in this place that over, I think the way that uh folks phrase this and the way that we phrased it in some of our documentation, instead of asking the who and why questions, which automatically shifts you back to blame, what allowed this to happen? How did we get here? What are the things about this human action that we’re unable by process or anything else? And really just bring the awareness that any single one of us could make these same mistakes given the right conditions. And so, what we’re looking for in this postmortem, again, we’re looking for how are we going to prevent the specific issue, but how are we future-proofing it? So, not just so Kat doesn’t make that same mistake again, because you know what, she’s at fault for this incident, it’s completely her, but anyone could actually make this mistake. Charlotte could make this mistake, absolutely anyone else could do it given the right conditions.
Charlotte Ward: 28:24
Yeah, yeah, yeah. How dare you? Honestly, yes. But right, we can. Any one of us can make mistakes, right? And I think that um I there’s a there’s one one one thing, one way I like to put this is that we should aim to fix the process and the systems, not the people. You know, and I think it’s you know, fix the systems, not the people, is just a really good way to think about it because it it focuses you outside of the people. And you touched on a couple of things there which are which are part of those systems, and it is part of like how do how do we put people in these situations where they can, you know, make mistakes or make even if it’s not like you know, there’s a difference between falling over and and taking the wrong step, if you see what I mean as well. Like you can make the wrong choices, and but even if a person makes the wrong choice, why are we putting people in the path of being able to make the wrong choice? And there’s an element of five whys here, isn’t there? Like, and I like to say, what’s the upstream? What’s upstream of this? What’s it’s always upstream, you know? It’s ultimately, for instance, that person picked the wrong playbook, or they’ve chose not to follow the playbook, or they followed the playbook but thought a step was irrelevant, and whatever it is, you know, why is this even a playbook in the first place? Why isn’t it a system? You know, that why isn’t there a software solution? So it’s kind of go go wind it back as far as you can. And sometimes you go back far enough and it almost seems like, well, that’s just a whole new piece of product we need. And that’s fine, right? That’s fine.
Kat Gaines: 30:07
You might identify that, and that’s one of your action steps that may come out of it. It may be a larger action step than you hoped for. But if that is the case, you have to be able to again, yeah, remove yourself from the blame and the individual human cause of how you got here, and again, just step zoom out and understand how did the process and tools allow this to happen.
Charlotte Ward: 30:32
So we’re we’re asking all the tough questions. So everyone’s in the room, nobody wants to be there. We’re we’ve we’ve done our homework as well because we’re committed to this process, even if we don’t want to be there. And now we’ve got all these tough questions flying around, which are you know, they are blameless, but we’re all feeling probably just a little bit defensive as well. You know, we’re we’re humans after all. Um so let’s say we we uh dial into the um the how in how do we get here? And we come up with our list of actions. Um out of this, we’re we’re getting a report at the end of the day. That’s part of the goal. The other part of so one goal is a report, and the other goal, if you like, is a set of actions to prevent this ever happening again. Is there anything else we’re looking for as outputs from this process?
Kat Gaines: 31:29
Yeah, I think you you want to ensure that your customers at the end of the day feel heard and represented in the process. They’re they’re not going to be part of it in the sense that they’re not going to be in the room when you’re having this meeting, right? But what you need to ensure that you’re getting out of it is that when you have that post-mortem or incident review report that you write up after this meeting happens, that they feel that the concerns they raised during the incident were addressed and that you had this conversation with protecting their trust and their relationship with you in mind. And so there’s a lot that, again, the customer-facing teams have to bring to that conversation to ensure that that is the end result. You have to ensure that again, you’re talking about the symptoms that customers experienced and that you’re also talking about the general faith in your business and your product. And it’s a little bit more, it can be less of a data-driven conversation, which I don’t love to veer toward, but talking about the sentiment that you’re getting from customers and realistically how you think this is going to impact how they’re interacting with you going forward. Is this going to raise tension when talking about this particular part of the product? Or, for example, if your support team wasn’t well equipped to respond during the incident, is that going to shake their faith in your support team for their ability to help them when something else like this is happening? And so you need to make sure that the output has all of those things addressed, especially if some of those process conversations with the customer-facing teams come up. And I’m I’m kind of harping on that because it’s something, again, that we tend to forget that these folks need to be in the room when these conversations are happening. And so we’re likely to make mistakes in this area as an industry where we are likely to forget how to communicate with customers or not provide our customer-facing teams with enough information early on. And that’s okay. Again, we’re being blameless here. It’s a common pit, uh just pit to fall into, but we need to show the adaptability and willingness to learn from those things and commit to getting better at them.
Charlotte Ward: 33:50
Yeah, yeah, absolutely. Um, and in all of this, in in all of this output from this meeting, equipping your customer teams to talk to customers about it is critical um in terms of just giving them the information. But yeah, help them translate it into customer-friendly language as well. Your your customer team can go a long way um in building and rebuilding that trust in the organization, in that bit of the product, or in support, or in you know, in any part of the customer touch points, right? But but like help them out with the language as well. They they’ll couch it in all the trust building stuff. But I think sometimes outputs from retros, unless you’ve got a really good coordinator, really good IC in there, or or whatever the whoever the leader of that meeting is, um, they can veer towards the engineering solutions. And so I think that, and this is it why also it’s important that you have customer liaison in the meeting, is not just to advocate for, but ensuring that the output or one output that you get is digestible, going back to that word, digestible by your support teams and by your customers.
Kat Gaines: 35:13
Right. That even if you need that engineering-focused explanation of what’s happening, because maybe your customers are engineers and so they need that breakdown of the technical detail. That’s okay. But you have to have the counterpart to it, understanding that some of your customers fall outside of that persona as well. And those people need to just as easily be able to understand what happened and why they can still trust you going forward as the people who understood the the technical breakdown.
Charlotte Ward: 35:43
Yeah, yeah, absolutely. So is this a nicely crafted, you know, six-page document that we’re delivering? How how how do we how do we actually get this report out there? And you know, what what are our communication options?
Kat Gaines: 36:00
Yeah, hopefully it’s not six pages. Uh if it is, that’s that’s a pretty hefty one. And hopefully that’s the rare circumstance. But yeah. We talked uh we talked earlier about internal versus external liaisons, and you’re gonna have some similar breakdown here. Uh how I’ve seen people do this a lot is again, the team who owned the incident itself will often write up the internal breakdown and distribute that, whether you’re distributing it through an email list or you’re documenting it on a wiki page somewhere, something else. Those folks will usually put that breakdown internally up for something. But then again, you’ve had the customer liaison in this meeting. So they’ve been thinking about crafting the external messaging and they should be using um a status page project product, excuse me, or something like that. Uh so for example, PagerDD has a status page product, so do a lot of other companies that will allow you to just post that messaging both during the incident and then after it’s wrapped up as well, with that full review of what happened. And you should be covering a couple of key areas. You should be covering what happened, that very straightforward breakdown of here is what we experienced, roughly when, how it probably impacted you as a customer. It should be diving a little bit further into uh more detail of what happened in the middle section of the document. So your first section is going to be more of a summary. Then you’re gonna have a little bit more detail following that. And then you’re going to have a learnings section, and you may call it what we’re doing going forward, uh, future proofing, learnings, any of these things are valid, but that’s where you tell customers here’s what to expect from us going forward. Here’s what we learned, here’s what we’re changing. You should have an identified a couple of action steps out of that meeting. And here is where we’re going to work to ensure that you can trust us again going forward. And you’ll want to close out that messaging. Be wary of excessive apologies, I’d say. Yeah. They tend to come off as less genuine the more and more you use them, right? Not that you shouldn’t feel remorseful for what happened or you shouldn’t want to apologize to your customers for what’s, you know, the classic phrasing, interruption to your service, those types of things. But there are two things that happen there. They feel less genuine uh when you overuse them, and the phrases that we all tend to lean on tend to sound very, very cliche after we use them a certain number of times.
Charlotte Ward: 38:35
We apologize for the inconvenience.
Kat Gaines: 38:37
Oh haven’t said. Those types of things. Actually, as um as we’re talking about this, one of the financial institutions that I put my trust in for my money is having an incident of a kind, and they’ve been using that kind of generic language. And of course, people are ripping them apart because this is not just an inconvenience for people. This is access to their funds, this is their money. It’s highly emotional and highly consequential for them if they can’t access it. And so you you just diminish your customers’ trust in you in that moment, right? And so when you close out an incident and when you close out a review like this, you want to be sure that whatever you’re saying is genuine. You can be remorseful, you can tell your customers that you are actually sorry about what happened, but you do want to be clear with them that you’re not just saying that in cliche, that you’re saying that because you do really mean it and because you have actionable steps going forward to prevent this happening again. You’re not just, oh, sorry for the inconvenience. Give them a genuine understanding that we know that this impacted you, we know you want to trust us, and we’re committed to doing that going forward. However, You phrase that, that’s the message you have to convey with your your closing statement.
Charlotte Ward: 40:05
And I think this is where it’s good to avoid too much macro language in some of these things. Like just, you know, if your last paragraph is entirely boilerplate, I’m never and and I I’ve read two of these, then I’m going to assume that most of the stuff as a customer. If I see when I see number three, I’m gonna take it with a big pinch of salt because you’re clearly just copying and pasting big. Exactly. So yeah, exactly. And I I think, you know, also um the context matters, I think. What goes on your report can be more formal, can be, I’m not saying impersonal, I’m not saying cliche, but it can be more formal, it can be uh more um structured, you know, it can be all of those things. But there are ways to put further information or more personal, the more personal touches into the line of sight of your customers that isn’t inside the framework of a status page or an instant report, you know, that right? It’s it’s a quick email following up afterwards from the customer success manager or from the support team that says, Hey, did you see this? You know, let me know if there’s anything we can do for you. You know, just little personal touches go a long way. And they don’t have to sit in the confines of the um it’s it’s actually sort of inappropriate that they do as well because they get a bit too personal to be on a status page or inside an instant report. But there’s plenty of opportunity to provide the personal touch, particularly for incidents that only affect a few customers, say, right?
Kat Gaines: 41:52
You do, yeah. You have to strike that balance. And so if you are a customer success manager, like you were mentioning, is a great example. If you’re delivering that messaging to one of your customers, whether you’re sending it to them while you’re on a phone call with them, or you’re just shooting over a quick email, you can give them that personal touch if you feel that the messaging isn’t there, or if it just wasn’t quite right in the tone of the report itself. So it’s about striking that balance and avoiding that cliche overuse language at all costs, but still ensuring that your customers know that they have a human being to connect with on the other side of it when they do have concerns and they’re not just being handed this document and then abandoned.
Charlotte Ward: 42:36
And I think, I think, yeah, right. And I I think that’s um, I think for me that’s one of the most important closing elements in these pages or reports is what can the customer do next if they have questions, if they want to talk to someone, because not every customer is going to be 100% satisfied by your perfectly crafted, retrospective, customer-facing, full stop period kind of language. This is it, we’re we’re done here, you know, and off you go. Like customers should be given and reassured that it’s okay to reach out. They should be given that opportunity, shouldn’t they? So I think for me, what what it’s important to say if you’ve got any further questions, don’t hesitate to whatever the thing is you want them to do, you know.
Kat Gaines: 43:23
You do, you have to give them that extra contact, and you have to remember too that just like we’re saying, we have to involve humans in the review process because you cannot you cannot represent every perspective just by having a couple of folks in there. So you do need the impacted humans internally in the review process. Think of that as the final step of including all the necessary humans in your review process, and it’s about remembering that customers are human too. And so they also have the opportunity to say, uh, you didn’t quite address this thing that came up for me, or I think you missed this part of it. This isn’t quite clear to me, or I don’t really know where to go from here. And that is another opportunity, even after you’ve closed out your review meeting, you’ve published the thing. That’s another opportunity to say, oh, we actually need to add that to the documentation because guess what? Someone else was able to point something out that we may have had blinders on to and we may have missed in the process.
Charlotte Ward: 44:19
Absolutely. Absolutely. Ensuring your customers feel heard is uh like end-to-end on the incident, right? Um, from uh first communication right to that status page or that report.
Kat Gaines: 44:31
100%.
Charlotte Ward: 44:32
Yeah, yeah. Um, thank you so much for talking me through instant reviews, retrospectives, post-mortems, even though nobody died. Um it’s been an absolute pleasure again, Kat. Um, I think we’ve got one more, at least one more, right? Um, so at least one more. At least one more. At least we that’s that’s all we’re committing to now. Yeah, so the shape that this this no longer a six-part series has taken. Uh yeah. Awesome. Okay, well, I look forward to our next conversation. And uh, thank you so much for joining me this time. Talk to you soon. Thank you. That’s it for today. Go to
