WEBVTT 1 00:00:14.480 --> 00:00:17.800 Hey, what's going on, everybody? I am the host of Adventures and 2 00:00:17.839 --> 00:00:22.079 DevOps and wait, yeah, that's the right channel name. Sorry. You 3 00:00:22.120 --> 00:00:26.480 know sometimes I get that mixed up, like almost every time I think, 4 00:00:26.519 --> 00:00:29.760 so if you're a frequent listener to the podcast, you know that I messed 5 00:00:29.800 --> 00:00:33.679 that up just about every time. So thanks for bearing with me on that. 6 00:00:34.439 --> 00:00:38.479 But today I'm not going to miss this part up because today I have 7 00:00:39.320 --> 00:00:44.520 as our guest of Drew Stokes. He's the senior manager of software engineering for 8 00:00:44.759 --> 00:00:49.240 Page Your Duty, and we're talking about incident management. And that's one of 9 00:00:49.240 --> 00:00:54.479 my favorite topics because it just goes so deep and it crosses so many disciplines 10 00:00:54.520 --> 00:01:00.960 across your infrastructure and your application teams and marketing and sales and the executive suite. 11 00:01:02.000 --> 00:01:06.079 Like depending on the level of the incident, you're just all overboard, 12 00:01:06.560 --> 00:01:08.280 all over the board with this. So Drew, welcome to the show. 13 00:01:08.319 --> 00:01:11.680 I'm excited to have you here. Thanks, I'm excited to be here. 14 00:01:11.719 --> 00:01:18.079 It's going to be too well right on, So tell us a little bit 15 00:01:18.120 --> 00:01:23.799 how you got into the field of incident management or incident response. Oh that's 16 00:01:23.799 --> 00:01:29.079 a good question. Okay, Yeah, So I've been in tech for a 17 00:01:29.079 --> 00:01:33.040 while, like most people here, I think it's been something like sixteen years, 18 00:01:33.840 --> 00:01:38.840 and I think originally I was kind of trying to figure out my way 19 00:01:38.920 --> 00:01:42.079 helping folks out with technology and networks, and then I got into front end 20 00:01:42.079 --> 00:01:46.719 development and moved into back end and then dropped into SRE and that's when I 21 00:01:46.799 --> 00:01:52.079 kind of really got familiar with not just the process of mitigating incidents, but 22 00:01:52.120 --> 00:01:56.000 actually managing them and trying to learn from them. So I did that for 23 00:01:56.040 --> 00:01:57.400 a while, and then I think for something like the last eight years, 24 00:01:57.439 --> 00:02:02.719 I've been primarily focused on people manager role. And there's a lot of ways 25 00:02:02.760 --> 00:02:07.319 in which you know, people managers are involved in incident management as well, 26 00:02:07.359 --> 00:02:13.240 both as stakeholders but also you know, facilitators and folks who are playing a 27 00:02:13.240 --> 00:02:16.479 supportive role for people who are responding. So kind of been in that space 28 00:02:17.639 --> 00:02:22.599 for a while now, and back in I think it was May of twenty 29 00:02:22.639 --> 00:02:27.360 twenty one, I joined a startup called Jelly, which was founded by Nora 30 00:02:27.479 --> 00:02:30.800 Jones, who's the author of the Chaos Engineering book and the founder of the 31 00:02:30.879 --> 00:02:35.759 Learning from Incidents community, and that was kind of where I really dropped into 32 00:02:36.840 --> 00:02:40.240 you know, incident management in general, but specifically this opportunity to kind of 33 00:02:40.280 --> 00:02:45.960 not just resolve incidents, mitigate the issues, but also to learn from them 34 00:02:45.960 --> 00:02:51.840 in order to improve future response and organizational performance. So there's a lot of 35 00:02:51.840 --> 00:02:53.680 really interesting ways to think about the space, and you mentioned at the beginning 36 00:02:53.719 --> 00:02:58.319 it's really important. And part of the reason is because it's so cross cutting, 37 00:02:58.400 --> 00:03:01.120 right, because incidents or a lens through which you can see the way 38 00:03:01.120 --> 00:03:06.520 that your organization and your people operate, and that applies to customer service, 39 00:03:06.599 --> 00:03:09.680 that that applies to executives and to the folks actually responding to the incidents. 40 00:03:09.680 --> 00:03:15.240 It's a really interesting space with a lot of opportunity, which you'll you'll hear 41 00:03:15.319 --> 00:03:19.479 that word a lot in this conversation. We refer to incidents as opportunities. 42 00:03:20.159 --> 00:03:23.080 Oh for sure they are because you know, one of the things that I 43 00:03:23.080 --> 00:03:25.800 think about a lot is just because we're in tech. You know, we've 44 00:03:27.120 --> 00:03:31.840 all done the Google search for is such and such service down because you're having 45 00:03:31.879 --> 00:03:36.319 problems and you're like, did I do something wrong? Or are they actually 46 00:03:36.360 --> 00:03:38.599 dead in the water right now? And I think that's like, to me, 47 00:03:38.680 --> 00:03:46.240 that's one of the hallmarks of highlighting that your incident response plan is really 48 00:03:46.280 --> 00:03:53.879 really well done whenever your customers know that you're having an incident incidents because you 49 00:03:54.120 --> 00:04:00.400 told them versus them discovering that something was broken. Yeah, there's a there's 50 00:04:00.439 --> 00:04:04.879 a level of well. So, so one interesting aspect here is you mentioned 51 00:04:04.919 --> 00:04:10.159 another cross cutting function there, right, which is you have internal stakeholders and 52 00:04:10.280 --> 00:04:14.479 external stakeholders for these types of things. But there's also this layer I think 53 00:04:14.479 --> 00:04:18.240 that you're referring to here of like operational excellence and observability. Right, do 54 00:04:18.279 --> 00:04:23.000 you know that the system's broken before someone tells you that the system is broken, 55 00:04:23.720 --> 00:04:28.199 And a lot of the ways in which you can improve that process is 56 00:04:28.199 --> 00:04:30.560 through the learning process after the incident. Right, So, if you have 57 00:04:30.600 --> 00:04:34.720 an incident, for example, where a customer reports an issue, looking at 58 00:04:34.759 --> 00:04:40.120 the details of that timeline and what actually happened can help you figure out where 59 00:04:40.199 --> 00:04:44.800 you need to add additional instrumentation or alerting, or how to adjust your team's 60 00:04:45.399 --> 00:04:49.439 processes, you know, your software development life cycle or your release process to 61 00:04:50.040 --> 00:04:56.600 better account for those kind of unpredictable behaviors in the system. So really interesting, 62 00:04:56.720 --> 00:05:00.600 like complicate, you know, when you're dealing with not just complex software 63 00:05:00.639 --> 00:05:05.720 systems, but also complex organizations and groups of people, right, really interesting 64 00:05:05.759 --> 00:05:11.399 opportunities to figure out how do we kind of iteratively approve improve our understanding of 65 00:05:11.399 --> 00:05:15.600 the system and our understanding of failure mode so that we can kind of inspire 66 00:05:15.720 --> 00:05:19.720 customer confidence and trust, right, letting them know that there's an issue before 67 00:05:20.120 --> 00:05:26.360 before they don't. Yeah, for sure. So you early early on in 68 00:05:26.360 --> 00:05:31.360 this you mentioned something I want to highlight, mitigating an incident versus managing an 69 00:05:31.439 --> 00:05:36.360 incident. Can you elaborate on the difference between those two, Yeah, that's 70 00:05:36.399 --> 00:05:42.720 a that's a great question. So there are a lot of different aspects of 71 00:05:42.759 --> 00:05:46.199 incident management in general, and I'll try to like decompose them in a way 72 00:05:46.240 --> 00:05:50.920 that makes sense here. So I think when you just reference detection, right, 73 00:05:51.000 --> 00:05:55.399 so there's a there's a phase there of like understanding whether or not there's 74 00:05:55.399 --> 00:05:58.360 an incident and trying to do something about it. And I think when we 75 00:05:58.399 --> 00:06:04.879 talk about managing incidents, what we're talking about is providing information and coordinating folks 76 00:06:04.920 --> 00:06:11.959 in incident response. Right. Mitigating an incident is doing something to address the 77 00:06:12.000 --> 00:06:15.839 issue and get the system back to a stable state or you know, performing 78 00:06:15.839 --> 00:06:20.120 in a way that's expected with regard to external stakeholders. But I think for 79 00:06:20.279 --> 00:06:27.279 us, managing an incident is really about investigating what's going on, getting the 80 00:06:27.360 --> 00:06:30.720 necessary folks with the subject matter expertise into the room to contribute to that, 81 00:06:31.279 --> 00:06:36.199 coordinating that group of people in you know, large organizations are really complex incidents. 82 00:06:36.199 --> 00:06:41.759 Sometimes you have multiple work streams of investigation within an incident, and then 83 00:06:41.800 --> 00:06:46.519 communicating status out to stakeholders, your customer success team, your executives in a 84 00:06:46.560 --> 00:06:50.839 way that allows them to stay informed but does not have them jump in and 85 00:06:50.879 --> 00:06:54.439 start, you know, trying to get involved in the process in a way 86 00:06:54.439 --> 00:07:00.439 that can you know, add additional complexity to the overall incident managed. So 87 00:07:00.120 --> 00:07:04.199 from my perspective, I think management is a lot more about the process of 88 00:07:04.199 --> 00:07:10.839 coordinating and communicating during an incident, and mitigation is about that moment when you've 89 00:07:10.920 --> 00:07:17.040 kind of identified and addressed the issue to stop whatever impact is associated with the 90 00:07:17.040 --> 00:07:21.000 incident, Right, that's your signal to your external stakeholders that we are in 91 00:07:21.000 --> 00:07:25.480 a stable state, we've seen things are good, and there are various other 92 00:07:25.480 --> 00:07:30.720 steps after that, But for me, that's the primary difference. Yeah, 93 00:07:30.920 --> 00:07:35.079 Yeah, I think that's really important for someone who's not done a lot of 94 00:07:35.120 --> 00:07:43.800 incident responses to understand that the management of it is equally important as the mitigating 95 00:07:43.839 --> 00:07:47.720 of it. And in many of the environments I've worked in, those are 96 00:07:47.720 --> 00:07:56.319 actually two key roles for any incident. You have the first responder who's trying 97 00:07:56.399 --> 00:08:00.240 to find the cause and restore the service. But then you know, alf 98 00:08:00.240 --> 00:08:07.600 what have your primary communications individual who is getting the information from that first responder 99 00:08:07.639 --> 00:08:11.000 and relaying it out and doing still in a way so that everyone feels like 100 00:08:11.079 --> 00:08:18.279 they're in touch with what's going on and they aren't going around the back door 101 00:08:18.000 --> 00:08:24.000 sending DMS to the first responder to get status updates. Yeah. Yeah. 102 00:08:24.079 --> 00:08:26.720 One thing we talk a lot about is kind of this this incident management maturity 103 00:08:26.759 --> 00:08:33.840 model, and we think about different buckets of you know, engineering teams or 104 00:08:33.919 --> 00:08:37.200 organizations with regard to kind of how they approach this. And I've been in 105 00:08:37.879 --> 00:08:43.360 you know, multiple layers of the lower maturity model, and sometimes it can 106 00:08:43.399 --> 00:08:46.759 be really difficult, yeah, to even understand who's doing what and who do 107 00:08:46.840 --> 00:08:48.480 I ask for an update? You know, I've got a customer who needs 108 00:08:48.480 --> 00:08:52.039 an update now, and we have an SLA in the contract, what's going 109 00:08:52.080 --> 00:08:56.399 on? It can be really difficult to even know who's doing that. And 110 00:08:56.440 --> 00:09:00.600 I think you find that in you know, incident response tooling like Jelly, 111 00:09:00.919 --> 00:09:05.039 those roles are actually codified in the process. You're assigning an incident commander, 112 00:09:05.440 --> 00:09:09.759 you're assigning a communications lead to try and take care of that external communication of 113 00:09:09.919 --> 00:09:13.200 here's the person to you know, connect with if you need an update, 114 00:09:13.559 --> 00:09:18.120 or here's the person responsible for managing this incident, so that if you join 115 00:09:18.159 --> 00:09:20.039 in, you can say, hey, I'm here and I know about X, 116 00:09:20.200 --> 00:09:24.320 you know, can I help that sort of thing? Right? And 117 00:09:24.399 --> 00:09:28.320 so that's one of the things that Jelly does for you. If you need 118 00:09:28.360 --> 00:09:33.399 to improve the majority, improve the maturity of your incident response playing, using 119 00:09:33.440 --> 00:09:37.519 something like Jelly can kind of help you say, hey, here are the 120 00:09:39.000 --> 00:09:43.679 here are the people and the processes you need in place, and provide a 121 00:09:43.720 --> 00:09:50.080 framework, right. Yeah. I think I think like every small organization goes 122 00:09:50.120 --> 00:09:52.840 through a phase where someone opens a Google doc and writes down a run book 123 00:09:52.840 --> 00:09:56.039 for how to run incidents, right, And so what we wanted to do 124 00:09:56.240 --> 00:10:00.159 is to provide some of that for you in a way that didn't get in 125 00:10:00.200 --> 00:10:03.240 your way. So we've got a bot in Slack right that you can use 126 00:10:03.279 --> 00:10:07.960 to declare incidents as sigence, stakeholders, set stages, communicate status, all 127 00:10:07.960 --> 00:10:11.159 that sort of stuff, so that you don't really have to go in and 128 00:10:11.240 --> 00:10:16.159 kind of trial and error that Google doc and try to get folks enrolled in 129 00:10:16.200 --> 00:10:20.679 the process. There's just a thing kind of nudging you along the way and 130 00:10:20.799 --> 00:10:24.840 helping to offload some of that cognitive burden. When you're in the middle of 131 00:10:24.919 --> 00:10:28.600 managing an incident, right or typically as an incident commander, you're thinking about 132 00:10:28.600 --> 00:10:31.519 a lot of things. Sometimes you're also trying to mitigate the incident. Right 133 00:10:31.559 --> 00:10:35.240 if it's two am, you may have a stretch of time where you're doing 134 00:10:35.279 --> 00:10:41.279 everything on your own. And so I think the more folks can find mechanisms 135 00:10:41.480 --> 00:10:48.120 and processes that help them reduce the number of things they're doing during management so 136 00:10:48.159 --> 00:10:52.720 they can focus on getting the right folks in the room and finding the means 137 00:10:52.759 --> 00:10:58.600 to mitigation, the more successful the response process becomes, which results in better 138 00:10:58.679 --> 00:11:03.320 data for your post incident analysis, and then you're you know, cross the 139 00:11:03.320 --> 00:11:07.519 incident learning over time. Yeah, it's one of those things that like we've 140 00:11:09.480 --> 00:11:15.840 we've all done incident response wrong enough time enough times that we we kind of 141 00:11:15.919 --> 00:11:20.240 know, So it's I think it's one of those things like you know, 142 00:11:20.279 --> 00:11:24.679 like in software engineering, like writing logs has been done for decades now, 143 00:11:24.759 --> 00:11:28.960 so you don't write your own logging engine. You just pull in a logging 144 00:11:30.039 --> 00:11:33.600 library because you don't need to reinvent that wheel. And I think incident response 145 00:11:33.679 --> 00:11:37.879 is one of those we don't need to reinvent this will we can just buy 146 00:11:37.919 --> 00:11:41.240 a wheel that's already built. Yeah, we've we've we actually have a couple 147 00:11:41.279 --> 00:11:46.639 of customers of Jelly who are trying to replace their wheels, right because you 148 00:11:46.679 --> 00:11:50.600 know, some of some of the large organizations who started this process ten years 149 00:11:50.600 --> 00:11:54.000 ago had to make their own I used to work at New Relic and we 150 00:11:54.080 --> 00:12:00.360 had a slock bot we called nerd bot, which was our incident response you 151 00:12:00.440 --> 00:12:03.639 know, facilitation tool. But there's a cost associated with those things, right, 152 00:12:03.679 --> 00:12:07.559 You have to maintain them over time. Oftentimes they kind of fall to 153 00:12:07.600 --> 00:12:11.360 the bottom of the priority stack, and so iterating on your internal process becomes 154 00:12:11.399 --> 00:12:15.120 really hard. And I think that's where if you go with something you know, 155 00:12:15.320 --> 00:12:18.200 like Jelly's incident a response spot, which is you know, fairly opinionated 156 00:12:18.200 --> 00:12:22.600 but narrow in scope, right, it's just here are the set of criteria 157 00:12:22.639 --> 00:12:28.799 that we use for this thing. With some customizable features like automation, then 158 00:12:28.840 --> 00:12:33.840 you don't have to kind of invent that wheel and then reinvent it iteratively for 159 00:12:33.919 --> 00:12:37.200 all time. And you also don't really have to, you know, answer 160 00:12:37.240 --> 00:12:41.639 a lot of those questions when your incidents become more complex. There's like different 161 00:12:41.679 --> 00:12:46.039 phases of your incident response process. When you're a five person team, you 162 00:12:46.120 --> 00:12:48.559 jump in a zoom call, right, and you fix it. When you're 163 00:12:48.639 --> 00:12:54.120 fifty people in a major incident room, it's a very different experience and requires 164 00:12:54.159 --> 00:13:00.679 a different set of skills and supporting tooling. So, yeah, cool, 165 00:13:00.799 --> 00:13:07.559 you mentioned a couple of times the post incident response plan, so elaborate on 166 00:13:07.600 --> 00:13:11.480 that a little bit for me. Yeah, this is another area where I 167 00:13:11.519 --> 00:13:16.799 think everyone kind of starts with a recognition that there's more that can be gleaned 168 00:13:16.840 --> 00:13:20.759 from these experiences. Right early on, you have an incident, you respond 169 00:13:20.799 --> 00:13:22.159 to it, you fix it, maybe you shoot an email off to folks 170 00:13:22.200 --> 00:13:24.639 saying what happened, and you know, here's what we're going to do to 171 00:13:24.639 --> 00:13:31.039 address in the future. But as your system complexity grows and as your organization 172 00:13:31.159 --> 00:13:35.120 grows, there are you know, many more opportunities to figure out how to 173 00:13:35.320 --> 00:13:41.720 change not just the system itself right to you know, write better logs or 174 00:13:41.840 --> 00:13:46.080 increase visibility into the system's behavior, but also to change how the organization is 175 00:13:46.080 --> 00:13:52.639 structured around those systems. Right. So, one anecdote I like to share 176 00:13:52.799 --> 00:13:56.480 is at my time in a previous company, we had this custom feature flag 177 00:13:56.559 --> 00:14:01.080 system that had been around for I don't know, it was like eight or 178 00:14:01.159 --> 00:14:03.440 nine years or something. Everybody wanted to get off of it. It wasn't 179 00:14:03.480 --> 00:14:07.360 great, and every time there was an incident with that system, someone from 180 00:14:07.399 --> 00:14:11.240 the network engineering team would be pulled in because they were one of the original 181 00:14:11.279 --> 00:14:15.080 authors. They had nothing to do with this system anymore, but no one 182 00:14:15.120 --> 00:14:20.039 else knew how it works. And so if you're just responding to and mitigating 183 00:14:20.080 --> 00:14:24.120 incidents and not looking any further, you don't see those types of organizational misalignment 184 00:14:24.240 --> 00:14:28.960 right where you've got a primary owner or subject matter expert that is, you 185 00:14:30.000 --> 00:14:31.840 know, accountable for a whole slew of things that have nothing to do with 186 00:14:31.879 --> 00:14:37.440 this foundational service that's critical for business function. If you've got a feature flag 187 00:14:37.480 --> 00:14:41.840 system in you know, a fourteen year old code base it's got to work. 188 00:14:43.120 --> 00:14:48.159 So I think when we talk about post incident learning, this is this 189 00:14:48.240 --> 00:14:52.080 is the next phase in maturity. Right, you figured out your response process, 190 00:14:52.200 --> 00:14:54.039 you know how to get the right folks in the room, you know 191 00:14:54.080 --> 00:14:58.240 how to move toward mitigation, and you're starting to capture some of the you 192 00:14:58.279 --> 00:15:01.200 know, follow ups that you want to take. Maybe we need more ossability. 193 00:15:01.399 --> 00:15:05.159 Maybe this library and our services out of date, and if we updata 194 00:15:05.240 --> 00:15:09.960 we'll get better performance. Like that, But it goes beyond some of those 195 00:15:11.000 --> 00:15:13.879 follow ups, and as you start to cultivate a process around this, and 196 00:15:13.919 --> 00:15:16.799 there's a lot of different ways that folks do this. You know they're refer 197 00:15:16.879 --> 00:15:22.240 to on this post mortems or learning reviews, or you know, sometimes you're 198 00:15:22.279 --> 00:15:26.519 just getting in a room and talking about the incident without the structure, you 199 00:15:26.559 --> 00:15:31.120 start to uncover all of these really interesting aspects of not only the responding team, 200 00:15:31.159 --> 00:15:33.639 but the organization overall. And so some of the things that we're most 201 00:15:33.720 --> 00:15:39.200 interested in learning is, you know, what did folks know when they responded 202 00:15:39.240 --> 00:15:41.879 to the incident and what did they not know? Right? What are the 203 00:15:41.919 --> 00:15:50.360 ways in which the folks involved communicated successfully and maybe not so much? How 204 00:15:50.399 --> 00:15:56.840 did the organization's processes contribute to or prevent aspects of a specific incident. It's 205 00:15:56.879 --> 00:16:00.519 all kinds of interesting stuff to dig into, and you can look at it 206 00:16:00.559 --> 00:16:04.200 from a bunch of different angles. So we have, you know, a 207 00:16:04.200 --> 00:16:10.360 lot of examples of our customers creating multiple investigations on an incident where a person 208 00:16:10.399 --> 00:16:14.480 A and person beat both investigate and then you see like where the differences are, 209 00:16:15.279 --> 00:16:18.600 and I think that turns up a lot of interesting stuff. We've taken 210 00:16:18.799 --> 00:16:23.000 the approach in Jelly of writing incident narratives, so you know, post learning, 211 00:16:23.039 --> 00:16:26.799 review, post mortem, whatever you want to call it. Our feeling 212 00:16:26.919 --> 00:16:33.120 is that incidents are stories and the way that people connect with information and learn 213 00:16:33.240 --> 00:16:36.679 is through storytelling. And so we've taken the approach that, you know, 214 00:16:37.000 --> 00:16:41.519 we want to provide folks with a tool to tell a story backed by evidence, 215 00:16:41.600 --> 00:16:45.080 right, what was actually said during the incident, what you know, 216 00:16:45.519 --> 00:16:48.519 metrics or data we were looking at, but to kind of nudge folks in 217 00:16:48.559 --> 00:16:55.679 the direction of sharing their perspective and their assertions about what it means. Right, 218 00:16:55.840 --> 00:16:59.639 when these two folks were talking, they were talking about different aspects of 219 00:16:59.679 --> 00:17:02.720 the system, and they didn't realize it what does that mean, right, 220 00:17:02.720 --> 00:17:08.079 what's the opportunity there to improve the incident management and the way that these teams 221 00:17:08.079 --> 00:17:14.599 are connected and communicating those sorts of things. Yeah, you see that a 222 00:17:14.599 --> 00:17:18.240 lot whenever you have people with different disciplines or different backgrounds, you know, 223 00:17:18.279 --> 00:17:25.519 a networking background versus a software engineering background. And I think that highlights one 224 00:17:25.599 --> 00:17:33.160 of the one of the arts of post incident response is creating those follow up 225 00:17:33.200 --> 00:17:41.000 items and getting those the right people engaged to recognize, prioritize, and address 226 00:17:41.160 --> 00:17:45.200 the things that you learned from that incident. Yeah, and you know that 227 00:17:45.400 --> 00:17:51.119 you mentioned like different disciplines. There are different different disciplines within the responding team, 228 00:17:51.200 --> 00:17:55.839 but there are also incidents provide this really unique opportunity to consider the different 229 00:17:55.839 --> 00:17:59.920 disciplines across an organization. Right, So for your major incidents, it's not 230 00:18:00.160 --> 00:18:03.720 just your you know, senior engineers from a specific team. It's also your 231 00:18:03.759 --> 00:18:08.480 customer support support folks on critical accounts. It's also your group leads and your 232 00:18:08.480 --> 00:18:15.960 executives. All of these people have different priorities and perspectives and understanding with regard 233 00:18:17.079 --> 00:18:19.319 to the impacted systems and the impact on the business. Right, if I'm 234 00:18:19.359 --> 00:18:22.880 responding to an incident, my goal is to make the chart go down, 235 00:18:23.440 --> 00:18:30.440 whereas my executive or salespeople's goal is to minimize the costs associated with customer impact. 236 00:18:30.559 --> 00:18:33.319 Right, We've got slas with our customers for uptime, and we need 237 00:18:33.359 --> 00:18:38.279 to keep that in line. And I think the different perspectives and priorities there 238 00:18:38.359 --> 00:18:42.599 result in that same kind of differing perspective that I mentioned earlier, where I 239 00:18:42.680 --> 00:18:47.480 may look at an incident and think it means one thing, but my group 240 00:18:47.599 --> 00:18:51.960 lead or you know, my sales associate may look at it and think another 241 00:18:52.039 --> 00:18:59.079 thing. And that opportunity with you know, incident narratives or post incident learning 242 00:18:59.160 --> 00:19:03.640 is to try and bridge that divide between those different perspectives and help everyone cultivate 243 00:19:03.640 --> 00:19:07.160 a shared understanding of what it means across those dimensions. Right, this is 244 00:19:07.200 --> 00:19:14.119 what this incident meant for business impact and process, for customer satisfaction, and 245 00:19:14.359 --> 00:19:18.440 for the you know, sustainability of our you know, critical services something like 246 00:19:18.440 --> 00:19:25.240 that. Yeah. I've even worked in organizations where it involved the marketing team 247 00:19:25.359 --> 00:19:30.880 because they were out scrolling Twitter, you know, catching tweet going on about 248 00:19:30.880 --> 00:19:34.599 the incident and responding those and trying to do trying to minimize the blast radius 249 00:19:34.640 --> 00:19:38.920 there. Yeah, this is a whole other aspect that's really interesting, which 250 00:19:38.960 --> 00:19:42.960 is like where do incidents come from? Right? Who says what an incident 251 00:19:44.160 --> 00:19:48.240 is? We've taken the approach that anyone can declare an incident. Some organizations 252 00:19:48.240 --> 00:19:52.359 we've worked with are very narrow in terms of who can declare them. But 253 00:19:52.359 --> 00:19:56.160 yeah, customer success marketing, you know, random person from the internet. 254 00:19:56.279 --> 00:20:02.519 There are all sources of potential incidents, you know, automation and observability, 255 00:20:02.519 --> 00:20:06.559 those sorts of things, and so it's you know, the the once you 256 00:20:06.599 --> 00:20:11.440 start thinking about this space and you start exploring ways of benefiting from these lenses 257 00:20:11.759 --> 00:20:18.720 on current state of systems and organizational process, you start to see like there 258 00:20:18.759 --> 00:20:25.359 are opportunities everywhere. Right at Jelly internally, we create incidents for things that 259 00:20:25.400 --> 00:20:29.079 are not incidents. If we have a release going out that we think might 260 00:20:29.119 --> 00:20:33.839 be you know, impactful to customers because it changes some aspect of the user 261 00:20:33.880 --> 00:20:37.599 experience, that's an incident. If we're trying to better understand database failover in 262 00:20:37.799 --> 00:20:41.799 RDS, for example, we run a game day as an incident, and 263 00:20:42.200 --> 00:20:47.480 doing that gives you this repository of information that you can use again to build 264 00:20:47.519 --> 00:20:51.519 that narrative and make those assertions about where are we and where do we want 265 00:20:51.519 --> 00:20:55.759 to be with regard to how we're operating and the health and stability of our 266 00:20:56.359 --> 00:21:00.279 systems. So that's a really interesting anecdote about marketing. I love when those 267 00:21:00.319 --> 00:21:04.240 things come in from places you don't expect, right, You just kind of 268 00:21:04.279 --> 00:21:07.519 get a message from someone that you haven't met before and they're like, hey, 269 00:21:07.559 --> 00:21:11.599 there's something going on yet we'd better declare Yeah, yeah, you see 270 00:21:12.039 --> 00:21:17.839 someone from marketing enter in one of the tech Slack channels and that this is 271 00:21:17.880 --> 00:21:23.319 not going to go well. So I think one of the cool one of 272 00:21:23.319 --> 00:21:30.839 the cool types of companies I like to work with fit the model of Jelly 273 00:21:30.920 --> 00:21:34.920 because you actually use your own product, you know, like when you build 274 00:21:34.960 --> 00:21:40.799 and release it, your team actually uses it to manage your own incidents. 275 00:21:40.839 --> 00:21:45.519 And I think that is really really cool because you get firsthand experience of what 276 00:21:45.559 --> 00:21:49.920 it's like to be your own customer, and you can understand what your customers 277 00:21:49.960 --> 00:21:56.240 are actually seeing when they're trying to use your tool. Yeah, one thing 278 00:21:56.240 --> 00:22:03.359 that was really interesting thing in the early days about working with our customers. 279 00:22:03.400 --> 00:22:06.519 It's interesting now as well. We'll have to talk about page duty at some 280 00:22:06.519 --> 00:22:10.519 point later. But one thing that was really interesting is that the customers that 281 00:22:10.559 --> 00:22:14.519 we work with are really passionate about their process and those opportunities to learn, 282 00:22:14.559 --> 00:22:18.839 and so we get to work really closely with them on you know, understanding 283 00:22:18.920 --> 00:22:22.079 their process and building tooling it works for them. We work with F five 284 00:22:22.200 --> 00:22:26.960 and Indeed and Honeycomb and Zendesk. These are like, you know, large 285 00:22:26.079 --> 00:22:30.920 influential organizations who are kind of at the cutting edge of this process. So 286 00:22:32.640 --> 00:22:37.759 there's this bi directional information share where you know, we can build features that 287 00:22:37.799 --> 00:22:41.039 support those organizations processes, but then we can also adopt some of those organizational 288 00:22:41.039 --> 00:22:45.119 processes because they make a lot of sense and they work well for us. 289 00:22:45.880 --> 00:22:51.759 I was we were doing a product demo for an important group of people the 290 00:22:51.799 --> 00:22:55.519 other day and we noticed some lag in one of our features and I actually 291 00:22:55.559 --> 00:23:00.160 declared an incident with Jelley about the performance of the incident was cons tol jelly, 292 00:23:00.720 --> 00:23:03.759 and we ran that in parallel during the demo, and it was there 293 00:23:03.799 --> 00:23:08.559 was this moment where I was just like, this is so cool running an 294 00:23:08.559 --> 00:23:11.880 incident with the tool that we're demoing to people, and there wasn't actually an 295 00:23:11.880 --> 00:23:15.880 issue. It was a Wi Fi lag you know thing, So everything was 296 00:23:15.920 --> 00:23:19.880 good and that's okay. That's also a learning opportunity. But yeah, it's 297 00:23:19.880 --> 00:23:26.519 been really exciting to kind of watch things evolve over time and be a you 298 00:23:26.559 --> 00:23:30.440 know, benefactor of that system as well as trying to evolve it for our 299 00:23:30.480 --> 00:23:36.759 customers and find that alignment across across orgs, which is really unique. Most 300 00:23:36.759 --> 00:23:41.200 of your incident response and post incident learning is within an org. We've had 301 00:23:41.200 --> 00:23:48.559 the unique opportunity to kind of extend that outward, so fun right on. 302 00:23:48.559 --> 00:23:52.119 One of the things I'm interested to get your opinion on is I over the 303 00:23:52.200 --> 00:23:59.279 years, I've developed the opinion that there's a difference between mitigating the issue and 304 00:23:59.400 --> 00:24:02.720 resolving issue. And I refer to that in in terms of, like during 305 00:24:02.759 --> 00:24:07.279 the incident, you know, you have you know, to say your API 306 00:24:07.359 --> 00:24:15.279 service is slow, it's okay during the incident to throw more servers at it. 307 00:24:15.839 --> 00:24:19.039 You know, we're going to we're going to mitigate the issue by adding 308 00:24:19.079 --> 00:24:23.079 more servers or adding more memory, or do something to make the symptoms of 309 00:24:23.119 --> 00:24:29.319 the problem go away. But then there's this like defining moment of okay, 310 00:24:30.079 --> 00:24:34.599 customer impact has been resolved, but now we've got to go back and find 311 00:24:34.839 --> 00:24:40.599 the root cause because adding the additional servers did not fix the issue that fixed 312 00:24:40.599 --> 00:24:45.039 the symptoms. And I'm interested to get your opinion on that. Yeah, 313 00:24:45.119 --> 00:24:48.000 it's a really good distinction that you're making there, and I think it has 314 00:24:48.039 --> 00:24:52.119 a lot to do with prioritization and understanding. Right. So oftentimes, especially 315 00:24:52.119 --> 00:24:57.680 in major incidents, there's a priority involved there to minimize customer impact, right, 316 00:24:57.680 --> 00:25:03.920 because customer impact means lost revenue. Incidents are expensive both in terms of 317 00:25:03.960 --> 00:25:07.519 time and you know, customer satisfaction and trust. And so I think there 318 00:25:07.519 --> 00:25:11.960 are kind of two ways in my experience that you mitigate before resolution. And 319 00:25:12.000 --> 00:25:18.200 the first i'm mentioning now is about minimizing the impact in favor of kind of 320 00:25:18.240 --> 00:25:22.480 getting things back on track. And so, like you said, throw some 321 00:25:22.519 --> 00:25:27.039 additional servers at the API and that'll address the symptom, but we still don't 322 00:25:27.119 --> 00:25:30.880 understand what's going on in the hood, right, And so I think the 323 00:25:30.920 --> 00:25:36.400 second reason, sometimes you can choose not to mitigate an issue. I've been 324 00:25:36.440 --> 00:25:41.720 in situations where we've had customer impact, but the priority of understanding what's going 325 00:25:41.759 --> 00:25:45.160 on has exceeded the priority of needing to address that impact, maybe because it's 326 00:25:45.200 --> 00:25:48.119 like, you know, one user at a customer rather than all customers in 327 00:25:48.160 --> 00:25:52.119 a major incident. And so that second bit I think is really interesting because 328 00:25:52.240 --> 00:25:59.920 you can use the incident and the the levers you can pull during the incident 329 00:26:00.039 --> 00:26:03.119 to create the conditions for learning while it's happening. Right, So if you 330 00:26:03.160 --> 00:26:06.920 mitigate the incident with the API, it means that you have an opportunity to 331 00:26:06.920 --> 00:26:10.480 explore what was actually going on. Maybe you isolate one of those servers and 332 00:26:10.519 --> 00:26:15.680 you start to dig into you know which function calls. If you've got distributed 333 00:26:15.720 --> 00:26:19.599 tracing, which is amazing, you know which specific function or endpoint is causing 334 00:26:19.680 --> 00:26:25.200 delay in the response, right, that's causing a delay across all responses, 335 00:26:26.039 --> 00:26:30.559 And you can kind of take advantage of that system state, which you know, 336 00:26:30.599 --> 00:26:33.000 if you reboot the servers, if you add a ton of them, 337 00:26:33.240 --> 00:26:37.160 those conditions go away and you lose your opportunity to understand what's going on. 338 00:26:37.200 --> 00:26:41.119 And so there's a lot of different ways to look at it. I think 339 00:26:41.200 --> 00:26:47.039 mitigation and resolution for folks outside of incident response, that's a mental framework for 340 00:26:47.160 --> 00:26:51.480 understanding are we good now and are we good for the long term? Right? 341 00:26:52.000 --> 00:26:56.400 But as a responder, those two events are really key in terms of 342 00:26:56.480 --> 00:27:02.000 communicating within the response group what our level of understanding and what priority decisions we're 343 00:27:02.039 --> 00:27:07.599 making with regard to customer impact or you know, system stability or what have 344 00:27:07.720 --> 00:27:14.400 you. Sometimes incidents are not resolved for days after you know the actual incident. 345 00:27:14.519 --> 00:27:18.799 I've especially for for large, complex incidents. Sometimes you just have to 346 00:27:18.839 --> 00:27:22.640 get things to a steady state and let them stay there until you have chance 347 00:27:22.680 --> 00:27:26.319 to enroll more folks or get a deeper understanding of what's going on. And 348 00:27:26.400 --> 00:27:29.359 sometimes those fixes are not things you can roll out, you know, as 349 00:27:29.720 --> 00:27:33.559 one hot fix. Sometimes they are major upgrades or major changes to kind of 350 00:27:33.599 --> 00:27:40.039 foundational business logic. So I'm glad you made that distinction because they're they're really 351 00:27:40.079 --> 00:27:42.319 important, and I think oftentimes folks outside of the incident are just like, 352 00:27:42.359 --> 00:27:47.559 when are we mitigated? When we mitigated? But you can't you can't lose 353 00:27:47.599 --> 00:27:51.759 sight of that, that time frame between mitigation and resolution, because that's where 354 00:27:51.799 --> 00:28:02.160 a lot of the you know, exploratory understanding comes out for sure. And 355 00:28:02.440 --> 00:28:08.240 one of the things that I try to insist on is that mitigating the issue, 356 00:28:08.599 --> 00:28:12.880 were allowed to make live changes in production, but the actual root cost 357 00:28:12.960 --> 00:28:18.440 fixed has to go through our normal development cycle of making the changes in DEV, 358 00:28:18.880 --> 00:28:22.799 pushing the changes to a staging environment, validating them, and then promoting 359 00:28:22.799 --> 00:28:27.160 those changes to production. So it has to follow that flow. Yeah, 360 00:28:27.240 --> 00:28:32.839 and that's that goes back to that prioritization opportunity. Right. So once you've 361 00:28:32.920 --> 00:28:37.039 kind of addressed the business impacting issue, then you've got to get back to 362 00:28:37.079 --> 00:28:40.960 your fundamentals, right, and your business processes and compliance and all of that. 363 00:28:41.519 --> 00:28:48.279 And so detangling those two things allows you to respond in a way that 364 00:28:48.319 --> 00:28:52.079 helps the business, and then address the issue in a way that helps the 365 00:28:52.119 --> 00:28:56.079 business, and do those in different ways, because especially when you're when you're 366 00:28:56.079 --> 00:28:59.119 further along in your maturity model, when you're a large organization, there's a 367 00:28:59.160 --> 00:29:03.480 lot of things that can hands stand in the way of quickly addressing an issue. 368 00:29:03.559 --> 00:29:07.559 Right. If you don't create a path for doing that, then incidents 369 00:29:07.599 --> 00:29:11.640 end up taking longer and having a lot more impact. So yeah, and 370 00:29:12.200 --> 00:29:15.559 the other thing we've learned in all of this is that every organization is different, 371 00:29:15.680 --> 00:29:22.079 Right. Some organizations have response processes that specifically call out different ways of 372 00:29:22.160 --> 00:29:29.160 mitigating impacting issues and different ways of capturing follow ups for those. Right, 373 00:29:29.240 --> 00:29:33.279 Sometimes the incident's not closed until you've resolved it, and sometimes it's closed at 374 00:29:33.279 --> 00:29:37.160 the point that it's mitigated and you've captured the follow ups you want to take 375 00:29:37.160 --> 00:29:41.680 action on. You know. As a result, sometimes folks keep talking about 376 00:29:41.680 --> 00:29:44.720 the incident after it's been closed and they want all of that for their post 377 00:29:44.799 --> 00:29:51.039 incident learning review as well. There's just so many different ways to tailor this 378 00:29:51.480 --> 00:29:56.759 whole incident management process to help an organization be more successful. Yeah, one 379 00:29:56.799 --> 00:30:00.920 of the places I worked years ago was is that a healthcare provider, and 380 00:30:00.960 --> 00:30:10.079 we did we provided medical services for hospitals across the US for trauma patients, 381 00:30:10.599 --> 00:30:15.319 and so every incident that we had, whenever we broke out an incident room, 382 00:30:15.400 --> 00:30:18.680 we actually had a person from our quality team who would join the call 383 00:30:18.720 --> 00:30:22.519 as well and let us know, like every five or ten minutes, how 384 00:30:22.519 --> 00:30:29.359 many patients across the United States couldn't receive life saving healthcare because our stuff was 385 00:30:29.359 --> 00:30:34.119 broken. And so we had a very unique incident response model there that doesn't 386 00:30:34.160 --> 00:30:37.799 really apply anywhere I've been since then, but there were still lessons that I've 387 00:30:37.839 --> 00:30:42.319 taken away from that, you know, number one is mitigate the issue as 388 00:30:42.400 --> 00:30:48.799 possible. Right, I'm so interested to hear how how did that information help 389 00:30:48.920 --> 00:30:59.160 or hinder mitigation for your teams. It really set the priority and kept us 390 00:30:59.200 --> 00:31:03.920 focused, you know, because as that number went up, you started to 391 00:31:03.000 --> 00:31:08.079 understand, you know, the impact that this was having. And this was 392 00:31:08.119 --> 00:31:15.359 not a other development team sucks or their network is terrible, or and many 393 00:31:15.599 --> 00:31:22.200 many of our incidents it was because of user error at one of the trauma 394 00:31:22.279 --> 00:31:26.480 centers. But it's still not okay to say, oh, well, they're 395 00:31:26.559 --> 00:31:29.119 just doing it wrong, because you have to realize at the same time, 396 00:31:29.720 --> 00:31:33.480 you know, while you're on the phone with that person, they're up on 397 00:31:33.400 --> 00:31:37.680 a table in the emergency room doing chess compressions on this patient. So they're 398 00:31:37.680 --> 00:31:41.559 going to give it their best shot, but they may not be the most 399 00:31:41.559 --> 00:31:45.240 attentive user at that time, and you just got to work with that. 400 00:31:47.079 --> 00:31:52.359 Yeah, you're you're highlighting like a perfect example of I think why we are 401 00:31:52.440 --> 00:31:57.359 so focused on post incident learning, and it's because the most important aspect of 402 00:31:57.400 --> 00:32:02.599 these complex technical systems that we're all building and maintaining are the people involved, 403 00:32:02.720 --> 00:32:07.559 Right, and when you're in an incident response room, a major incident room, 404 00:32:07.599 --> 00:32:10.279 whatever, and you've got someone reminding you of the impact, especially when 405 00:32:10.279 --> 00:32:15.000 that impact is you know, not just on dollars, but also on people's 406 00:32:15.440 --> 00:32:24.200 lives. You create the conditions for this like profound human creativity, right in 407 00:32:24.640 --> 00:32:29.160 terms of figuring out, you know, what can we do as a team 408 00:32:29.319 --> 00:32:32.039 to kind of we're back to the incident management space, what can we do 409 00:32:32.039 --> 00:32:35.839 as a team to kind of come up with a creative solution here and get 410 00:32:35.880 --> 00:32:38.359 us back to good, you know, temporarily. And I think if you're 411 00:32:38.400 --> 00:32:45.440 not reflecting on and talking about those moments in incident response and your you know, 412 00:32:45.759 --> 00:32:49.839 postings in a learning review or narrative review, whatever you call it, 413 00:32:50.480 --> 00:32:52.920 and you're missing out on all of those examples of the ways in which the 414 00:32:53.079 --> 00:32:58.039 people are helping support the system and keep things moving. You know, we 415 00:32:58.400 --> 00:33:04.359 hear a lot in tech and DevOps and elsewhere that like automation is the key 416 00:33:04.559 --> 00:33:09.799 to sustainability and more reliable systems. And there are things that we can automate, 417 00:33:10.039 --> 00:33:15.400 you know, especially assigning roles during incident management and response. But there's 418 00:33:15.440 --> 00:33:21.440 a lot of you know, human involvement tweaking the system and adding you know 419 00:33:21.559 --> 00:33:25.920 capacity, not you know, technical capacity in terms of number of network requests. 420 00:33:25.960 --> 00:33:30.880 You can handle things like that, but adding capacity in terms of the 421 00:33:30.880 --> 00:33:36.359 system's adaptability. And I just like, I would love to be a fly 422 00:33:36.480 --> 00:33:38.480 on the wall for one of those incidents that you mentioned, because I imagine 423 00:33:38.519 --> 00:33:44.000 folks really came together and came up with some creative solutions to find a way 424 00:33:44.000 --> 00:33:47.440 to mitigate those incidents and get things back to good so that they could figure 425 00:33:47.440 --> 00:33:50.799 out, you know, what the long term solutions were. That's such an 426 00:33:50.799 --> 00:33:55.880 exciting like space, Yeah, for sure, and it's you know, it 427 00:33:55.920 --> 00:34:02.200 was a majority of the role was communication. Like all of all the my 428 00:34:02.319 --> 00:34:09.360 coworkers there had exceptional technical skills, but their communication skills were just a plus 429 00:34:09.440 --> 00:34:14.400 one on top of that. And I think that's what made it work so 430 00:34:14.559 --> 00:34:16.559 well. And I still say that to this day. You know, DevOps 431 00:34:17.519 --> 00:34:22.119 is not a technical world. There's a technical comm component, but it really 432 00:34:22.199 --> 00:34:29.280 is communications in building the technical framework, but then communicating that out to your 433 00:34:29.320 --> 00:34:32.360 customers, the engineers that you support, and getting the feedback from them to 434 00:34:32.480 --> 00:34:37.840 understand what's the difference between what I built and what they thought I built. 435 00:34:38.519 --> 00:34:43.119 Yeah, it's It's really great when you have those folks who kind of know 436 00:34:43.280 --> 00:34:46.800 how to be in a critical situation and maintain you know, effective communication and 437 00:34:46.840 --> 00:34:52.360 find a solution to the issue. One thing we talk a lot about is 438 00:34:52.360 --> 00:34:54.880 like how do you scale that, how do you how do you externalize those 439 00:34:54.920 --> 00:35:00.119 skills? Oftentimes we find that the folks who are most effective and inti and 440 00:35:00.239 --> 00:35:04.280 don't have the capacity or time to help up level or train folks into that 441 00:35:04.920 --> 00:35:07.119 discipline. Right. It kind of requires a lot of different skills. You 442 00:35:07.159 --> 00:35:12.119 need a technical expertise, you need experience with the systems involved, and you 443 00:35:12.159 --> 00:35:17.920 need a good handle on like effective communication, not just for communicating the status 444 00:35:17.960 --> 00:35:22.119 of the incident, but also communicating with the folks that you are directing if 445 00:35:22.159 --> 00:35:27.559 you're in an incident commander role for example. There's another area where if you 446 00:35:27.599 --> 00:35:30.079 invest in learning from these things, you can create artifacts that folks pick up 447 00:35:30.079 --> 00:35:36.000 when they join the organization. Right in almost every large org I've been, 448 00:35:36.079 --> 00:35:42.960 there's a confluence space or Google drive folder is something full of post incident reviews. 449 00:35:43.440 --> 00:35:45.920 Sometimes I'll just go in and read those, right, and you start 450 00:35:45.960 --> 00:35:50.840 to learn, you know, who are the folks who demonstrate an ability to 451 00:35:50.920 --> 00:35:53.320 kind of respond to some of the most significant incidents and what are they doing, 452 00:35:53.920 --> 00:35:59.800 how are they doing that right? What skills or actions have they taken 453 00:35:59.840 --> 00:36:02.960 that stood out in the learning room review should I try and cultivate as a 454 00:36:04.000 --> 00:36:07.920 responder? And so that that can be a really interesting space too, is 455 00:36:08.440 --> 00:36:13.119 not just learning about the system and what things we can change to improve performance 456 00:36:13.159 --> 00:36:16.079 of at the time, but how are we leaving breadcrumbs for the new folks 457 00:36:16.119 --> 00:36:21.559 coming into the org who are growing into that discipline, because trial by fire 458 00:36:21.840 --> 00:36:25.639 during a major incident can be a really stressful, kind of terrifying experience, 459 00:36:27.119 --> 00:36:30.320 and so the more you can kind of give, you know, these these 460 00:36:30.480 --> 00:36:37.639 anecdotal or story based accounts of how things go in your organization, more comfortable 461 00:36:37.679 --> 00:36:42.320 folks and feel when they step into that role. Yeah, I think it's 462 00:36:42.320 --> 00:36:47.679 one of those areas where there's like a mentoring path there. And as I 463 00:36:47.719 --> 00:36:52.599 have gotten older and been doing this for a while, I've realized that that's 464 00:36:52.159 --> 00:36:58.719 that's a larger part of my job is sharing that that context because you can 465 00:36:58.760 --> 00:37:04.639 put the documentation, but then there's also like the unspoken or the unwritten part 466 00:37:04.679 --> 00:37:07.880 of that. You know, there's the mood, the field the context of 467 00:37:07.920 --> 00:37:14.000 the situation. And I think that's been a problem for you know, far 468 00:37:14.079 --> 00:37:17.159 beyond my lifetime, and the only way we've been successful at solving it now 469 00:37:17.639 --> 00:37:22.719 up to this point is just through that mentoring type role where you bring people 470 00:37:22.760 --> 00:37:28.000 in even though you know that they aren't ready to be the lead in this, 471 00:37:28.440 --> 00:37:32.280 you bring them in just so that they can can witness it and start 472 00:37:32.320 --> 00:37:38.079 making notes for themselves. Yeah, and that's where a process or a policy 473 00:37:38.159 --> 00:37:44.920 around incident response and incident learning that is based on transparency can be really helpful. 474 00:37:45.480 --> 00:37:49.719 Right, Sometimes you get a lot of folks joining the major incident room 475 00:37:49.760 --> 00:37:53.239 that are trying to contribute in ways that may not actually you know, help 476 00:37:53.360 --> 00:37:58.519 with mitigation. But a lot of times we find in large organizations that have 477 00:37:59.199 --> 00:38:02.599 you know, policies angle toward transparency, folks just joined to kind of understand 478 00:38:02.639 --> 00:38:07.840 and learn in the moment and also after the fact. So, you know, 479 00:38:07.920 --> 00:38:13.880 the the incident learning review calendar is always a place that I go and 480 00:38:13.920 --> 00:38:16.159 try to figure out, you know, which which of these incidents are going 481 00:38:16.199 --> 00:38:21.079 to be most helpful for me understanding the way this organization operates and the critical 482 00:38:21.119 --> 00:38:24.360 systems. Right in the past, role we had a COFCA platform that was 483 00:38:25.039 --> 00:38:29.280 you know, involved in a lot of incidents, not because the COFCA platform 484 00:38:29.360 --> 00:38:31.039 was a problem, but because everything was built around it, right, So 485 00:38:31.079 --> 00:38:35.280 every time there was an issue with any system, that kind of tied back 486 00:38:35.320 --> 00:38:37.760 to there. And that presents a really interesting lens for you know, how 487 00:38:37.760 --> 00:38:42.840 do these folks communicate with the low broader org and what changes are we making 488 00:38:42.920 --> 00:38:46.960 to shore up some of those critical dependencies, And you know, just being 489 00:38:46.960 --> 00:38:52.039 able to join a conversation about that, not having been involved in response or 490 00:38:52.079 --> 00:38:57.199 having anything to do with the teams involved, can be a really powerful opportunity 491 00:38:57.199 --> 00:39:00.079 for you to kind of learn about the team that you're working with and the 492 00:39:00.280 --> 00:39:05.519 underlying technologies. Especially for folks like me, it's been eight years since I 493 00:39:05.679 --> 00:39:08.840 was you know, maintaining those types of platforms and so picking up on some 494 00:39:08.920 --> 00:39:14.320 of that nuance so that I can support the folks who are around those systems 495 00:39:14.320 --> 00:39:16.920 can be really helpful. There's a line there, though, You've got to 496 00:39:16.920 --> 00:39:22.480 make sure that expectations are clear, right. If you're participating in something for 497 00:39:22.519 --> 00:39:28.400 the purpose of learning, you're kind of a sponge rather than someone who's bringing 498 00:39:28.519 --> 00:39:31.360 opinions, you know, not having understood the circumstances of the specific incident. 499 00:39:31.920 --> 00:39:37.400 So you need a healthy kind of culture and set of expectations around this. 500 00:39:37.559 --> 00:39:40.000 But I've seen a lot of orgs that do it well, and it is 501 00:39:40.599 --> 00:39:45.760 a game changer, you know, for for helping to provide you know, 502 00:39:46.599 --> 00:39:52.519 scalable mentorship and opportunities for folks to get a better understanding of the details. 503 00:39:52.760 --> 00:39:57.039 Yeah. One of the things you commented on that I think just can't be 504 00:39:57.119 --> 00:40:04.559 elaborated enough is transparen see. And I've worked in multiple places, and when 505 00:40:04.559 --> 00:40:09.440 I first started my career, it was it was in many instances a fireable 506 00:40:09.599 --> 00:40:14.920 event if you created an incident, and for that reason, people would try 507 00:40:14.920 --> 00:40:20.679 to hide and cover up their incidents, which led to no one learning from 508 00:40:20.719 --> 00:40:25.079 that. And these days, you know, I almost paraded around you know, 509 00:40:25.199 --> 00:40:31.199 hey, I broke this because there's a learning opportunity there. And I 510 00:40:31.199 --> 00:40:37.559 think it's really important to be open and to build the environment where people aren't 511 00:40:37.599 --> 00:40:42.320 afraid to say that they made mistakes, and even the dumb mistakes, we 512 00:40:42.400 --> 00:40:45.119 all do them, you know, you learn from it. And I actually, 513 00:40:45.159 --> 00:40:49.679 at some point in my career a boss of mine told me, and 514 00:40:49.800 --> 00:40:54.480 it's an anecdotal story, but it's still effective. Someone created an incident cost 515 00:40:54.519 --> 00:40:59.159 several hundred thousand dollars and said, oh am, I going to be fired 516 00:40:59.199 --> 00:41:02.440 now, and the responded, no, I just spent two hundred thousand dollars 517 00:41:02.480 --> 00:41:09.960 on your education. Why would I fire you now? Yeah? And this 518 00:41:10.000 --> 00:41:15.440 is where I think, like, it's really difficult to build trust, right, 519 00:41:15.480 --> 00:41:17.320 It's really easy to damage trust, it's really difficult to build it. 520 00:41:17.320 --> 00:41:22.599 And so if you're approaching your your incident management, you know, life cycle 521 00:41:22.679 --> 00:41:29.159 and process from the perspective of trying to support folks doing what they can to 522 00:41:29.199 --> 00:41:36.079 help the business be successful, you get a lot of really impactful contribution and 523 00:41:36.119 --> 00:41:39.079 collaboration with regard to you know, keeping systems healthy and things like that. 524 00:41:39.599 --> 00:41:45.199 But if you over index on you know, the measurable metrics. We're humans, 525 00:41:45.280 --> 00:41:49.800 right, every every human will gain a measure Right, you start to 526 00:41:49.840 --> 00:41:53.360 cultivate some of those types of environments where you know, what's the consequence of 527 00:41:53.400 --> 00:41:58.280 me doing the right thing here? Is is it going to reflect poorly on 528 00:41:58.360 --> 00:42:00.199 me? Is it going to cause an issue? And so, thankfully, 529 00:42:00.719 --> 00:42:06.079 I think every organization that the Jelly has worked with over the past two and 530 00:42:06.119 --> 00:42:10.000 a half years since I joined two and a half plus years, they've taken 531 00:42:10.000 --> 00:42:14.800 the approach that, yeah, these are these are blame aware learning reviews. 532 00:42:14.920 --> 00:42:17.920 Right. We know that folks make mistakes, that they don't have sufficient context 533 00:42:19.639 --> 00:42:23.400 in the moment, and that they can learn from those experiences and change their 534 00:42:23.400 --> 00:42:30.239 approach next time, versus this kind of you know, older model we'll say, 535 00:42:30.320 --> 00:42:37.800 of prioritizing the the the you know, public visibility of how things are 536 00:42:37.840 --> 00:42:39.519 going, and maybe like maybe we don't declare an incident for that one, 537 00:42:39.559 --> 00:42:43.800 we just try to fix it quickly. Early in my career, I was 538 00:42:44.400 --> 00:42:49.679 learning how to use Microsoft SEQL databases and we had a large share point site. 539 00:42:49.719 --> 00:42:54.800 It was another medical audit company, and I learned what drop database commands 540 00:42:54.840 --> 00:43:01.440 do, and I did the hire production database and fortunately I had enough experience 541 00:43:01.440 --> 00:43:06.159 to quickly restore it before anyone noticed. But that was an environment where I 542 00:43:06.199 --> 00:43:10.119 didn't feel comfortable, you know, broadcasting that I had just seen in the 543 00:43:10.199 --> 00:43:15.239 process of learning some new commands stopped the entire database. So yeah, it 544 00:43:15.239 --> 00:43:22.360 can be a tricky balance, but you know, some light is the best 545 00:43:22.360 --> 00:43:25.360 medicine, right. Transparency in these types of environments allow folks to do what's 546 00:43:25.440 --> 00:43:30.760 necessary to get things back to good And I think the more you can kind 547 00:43:30.800 --> 00:43:36.760 of socialize and demonstrate that transparency, the more effective your organization is going to 548 00:43:36.760 --> 00:43:39.840 be, and the more folks are going to want to contribute to that mission, 549 00:43:39.960 --> 00:43:45.360 whatever it is. Yeah, yeah, absolutely agreed. So let's talk 550 00:43:45.400 --> 00:43:51.719 a little bit about what's going on with Jelly these days. Yeah, So 551 00:43:51.880 --> 00:43:57.360 Jelly has been like the most interesting experience of my career. I think I 552 00:43:57.440 --> 00:44:02.239 mentioned I joined in twenty twenty one. I think it was Jelly was just 553 00:44:02.360 --> 00:44:07.199 a post incident analysis tool at that time. So we had this notion of 554 00:44:07.239 --> 00:44:12.599 building narratives and not much else, and we recognized that part of the post 555 00:44:12.679 --> 00:44:16.000 incident learning process involves having good data, and the way that you get good 556 00:44:16.079 --> 00:44:20.880 data is you get consistent in your process. And so we ended up building 557 00:44:20.880 --> 00:44:24.840 this incident response bot and we also went to the other end of the spectrum 558 00:44:24.840 --> 00:44:30.079 and started introducing features for cross incident analysis. And so this is, you 559 00:44:30.119 --> 00:44:32.840 know, after an incident, let's spend some time learning, but then how 560 00:44:32.840 --> 00:44:38.719 do we roll up those learnings into themes across incidents that help the organization make 561 00:44:38.800 --> 00:44:46.239 decisions around growing teams to support services or changing direction with regard to build versus 562 00:44:46.239 --> 00:44:51.960 buy those sorts of things. And so we've been working on a lot of 563 00:44:51.960 --> 00:44:57.000 cool stuff for the last two and a half years. And then in what 564 00:44:57.079 --> 00:45:01.440 was it, I think November seconds the public announcement that we were merging with 565 00:45:01.480 --> 00:45:09.400 Patrie Duty went out, which has been like really exciting and also a crying 566 00:45:09.440 --> 00:45:14.079 experience has been a month, right, And so page Duty is something like 567 00:45:15.360 --> 00:45:21.880 eleven hundred employees as of January of this year, we were twenty one. 568 00:45:22.360 --> 00:45:25.719 We're kind of in the process of figuring out how to bridge those two divides. 569 00:45:25.760 --> 00:45:30.280 And one thing that I'm really excited about is, you know, Jelly 570 00:45:30.360 --> 00:45:35.880 has spent a lot of time differentiating itself as a product in the postings and 571 00:45:36.039 --> 00:45:39.880 learning area, and I think we've brought a lot of kind of novel approaches 572 00:45:39.880 --> 00:45:45.559 and opinions students that response in general. Patri Duty has been doing this for 573 00:45:45.639 --> 00:45:51.079 fourteen plus years, right they and they created the category within which Jelry could 574 00:45:51.079 --> 00:45:54.519 become a company, which is pretty cool. And so what we're looking to 575 00:45:54.599 --> 00:46:01.280 do now is to take that practice, you know, post incident learning really 576 00:46:01.519 --> 00:46:06.719 get folks from the earlier phases of the maturity level where they're just doing incident 577 00:46:06.760 --> 00:46:09.760 response and maybe they're doing a post incident learning review on a Google doc, 578 00:46:10.159 --> 00:46:15.880 and bring them into the modern right and start creating incident narratives and doing learning 579 00:46:15.880 --> 00:46:21.239 reviews. Page of Duty has something like twenty seven thousand free and paid customers. 580 00:46:21.760 --> 00:46:24.800 There's a huge opportunity there to help folks understand a better way of kind 581 00:46:24.840 --> 00:46:31.159 of benefiting from all. So that's my focus right now is figuring out how 582 00:46:31.159 --> 00:46:37.719 do we bring those two worlds together while keeping an eye on preserving that kind 583 00:46:37.760 --> 00:46:45.760 of post incident learning tooling and opportunity. But yeah, a lot a lot 584 00:46:45.760 --> 00:46:49.760 of exciting stuff on the horizon we are. We are going into a new 585 00:46:49.880 --> 00:46:53.400 year, so I think things will look very different on the Page of Duty 586 00:46:53.440 --> 00:47:00.000 side and probably also improve on the Jelly side as well. It's going to 587 00:47:00.039 --> 00:47:04.840 be it's going to be really interesting. Yeah. I think it's a natural 588 00:47:04.880 --> 00:47:08.840 fit, you know, because Page your Duty is hands down a great tool 589 00:47:09.039 --> 00:47:19.079 for notifying people that there's something requesting their attention, but what you do after 590 00:47:19.159 --> 00:47:23.760 that is kind of up to you, and so it seems like a natural 591 00:47:23.800 --> 00:47:31.400 fit to just roll that right into into Jelly and and help help people like 592 00:47:31.480 --> 00:47:37.079 just from a business perspective, take this huge page your Duty customer base and 593 00:47:37.199 --> 00:47:42.159 just guide them into the thing that they thought they were doing all along. 594 00:47:43.639 --> 00:47:46.239 Yeah, one one focus for us has always been, you know, how 595 00:47:46.280 --> 00:47:52.920 can we improve the quality of our customers' postings and learning reviews, and how 596 00:47:52.920 --> 00:47:58.960 can we allow the folks conducting those investigations to focus on what matters. We've 597 00:47:58.960 --> 00:48:05.119 talked to organizations where, you know, there was one problem manager at a 598 00:48:05.119 --> 00:48:07.360 a company that used Microsoft teams, and part of their job was to go 599 00:48:07.440 --> 00:48:13.440 through every team's channel and find transcripts associated with an incident and put them in 600 00:48:13.480 --> 00:48:16.239 service. Now, nobody should be doing that, right, That's just toil. 601 00:48:16.440 --> 00:48:22.960 That's that's not productive. And so one thing I'm especially excited for with 602 00:48:22.000 --> 00:48:27.320 this partnership with page Duty is or this this acquisition by paye Duty, is 603 00:48:27.840 --> 00:48:30.079 they've got a ton of data, right, And so when you're building your 604 00:48:30.440 --> 00:48:34.800 post incident narratives, your timeline, and you're adding evidence and you're trying to 605 00:48:34.840 --> 00:48:38.760 help folks understand the details of an incident, the more data you have to 606 00:48:38.840 --> 00:48:43.719 substantiate those claims and those events that you're highlighting in the incident, the more 607 00:48:44.440 --> 00:48:49.079 folks can learn from you know, the not only the overall shape of the 608 00:48:49.119 --> 00:48:52.400 incident, but the systems involved and how they're used to understand you know, 609 00:48:52.440 --> 00:48:57.440 the underlying technology. And so there's an element there that's really exciting, which 610 00:48:57.480 --> 00:49:00.920 is just we have a lot more data to allow our our users to work 611 00:49:00.960 --> 00:49:07.239 with. But I also think, like I said, Page Duty has been 612 00:49:07.280 --> 00:49:10.400 known for a really long time, UH as kind of an industry leader in 613 00:49:10.519 --> 00:49:15.519 scheduling and alearning. Right Uh act and bail I got paged. I'm gonna 614 00:49:15.519 --> 00:49:19.559 go fix it. Uh. There is a better way, right, Like, 615 00:49:19.760 --> 00:49:24.559 there are ways to tie that process into the incident response process and the 616 00:49:24.599 --> 00:49:29.519 postings of the view and I think that's that's going to be our focus over 617 00:49:29.599 --> 00:49:32.119 the next you know, several months, is figuring out how do we give 618 00:49:35.639 --> 00:49:40.760 pager Duty more mechanisms for supporting responders throughout the entire incident management life cycle, 619 00:49:40.880 --> 00:49:45.440 not just the detection phase, which a lot of folks know and they're familiar 620 00:49:45.519 --> 00:49:51.039 with, but you know, Page Duty's full operations cloud, which most folks 621 00:49:51.119 --> 00:49:53.920 I've talked to don't even know exists. Uh. And and this is you 622 00:49:53.960 --> 00:50:00.440 know, the the AI automation for reducing noise to signal when it comes to 623 00:50:00.519 --> 00:50:06.440 events. This is all of the mechanisms around running actual incidents, and then 624 00:50:06.480 --> 00:50:10.599 this is the post incident as well. Pad has a feature today called post 625 00:50:10.639 --> 00:50:15.519 mortems, which is fairly straightforward. It's your your Google post mortem doc. 626 00:50:15.880 --> 00:50:20.880 But we think there's a lot of opportunity to not require that folks are going 627 00:50:20.920 --> 00:50:23.679 and creating these data sets manually, but just kind of provide that information so 628 00:50:23.719 --> 00:50:30.039 they can use it to better narratives that are living all the things. And 629 00:50:30.119 --> 00:50:31.039 yeah, I think I think it's a natural fit too. I mean, 630 00:50:31.079 --> 00:50:37.000 I've been using page of duty for basically my entire career, right and being 631 00:50:37.039 --> 00:50:42.599 able to bridge that gap between that paging and scheduling and then the things that 632 00:50:42.639 --> 00:50:45.760 I need to do to help my team be successful, it's going to be 633 00:50:45.159 --> 00:50:51.639 huge, you know, for from my experience. Yeah, I think having 634 00:50:51.719 --> 00:50:57.559 access to that data is going to lead to better collaboration after the fact, 635 00:50:57.559 --> 00:51:00.280 because that's for me, I've always struggled with that. You know, after 636 00:51:00.320 --> 00:51:04.960 the incident's over and you're trying to do the review of it, trying to 637 00:51:05.079 --> 00:51:09.000 remember what things happened in what order and remember all of those steps that you 638 00:51:09.079 --> 00:51:14.679 took. So if you've got something that can can prompt you with reminders and 639 00:51:15.000 --> 00:51:17.320 kind of pre populate that narrative for you. I think it's just going to 640 00:51:17.440 --> 00:51:23.360 lead to better, better results at the end. Yeah, there is nothing 641 00:51:23.440 --> 00:51:28.679 better than having a starting point when you are trying to investigate an incident. 642 00:51:28.760 --> 00:51:31.880 Right, If you open an empty Google doc, it's a hard time. 643 00:51:32.000 --> 00:51:37.119 But if you can start with you know, in Jelly today, you start 644 00:51:37.119 --> 00:51:39.719 with the incident transcript, all the conversation that happened in slack and data about 645 00:51:39.719 --> 00:51:45.039 who is involved, so much better than starting from nothing. And that's especially 646 00:51:45.079 --> 00:51:52.239 true when you know your incident response process uses multiple data sources like multiple incident 647 00:51:52.280 --> 00:51:57.000 response channels or your data dog charts or what have you. So we're not 648 00:51:57.320 --> 00:52:02.079 really looking to do the post incident narrative for you. We're looking to give 649 00:52:02.079 --> 00:52:07.400 you a point to start from because that saves time, it saves energy, 650 00:52:07.440 --> 00:52:12.320 and let's you focus on the things that only you can create within your post 651 00:52:12.360 --> 00:52:16.039 incident narrative. Right, the investigator is a conduit through which the folks who 652 00:52:16.079 --> 00:52:23.400 are involved in the organizational miscellany kind of come together into a coherent story about 653 00:52:23.400 --> 00:52:29.199 what happens and what it means. So we really want to like provide a 654 00:52:29.280 --> 00:52:34.840 foundation on top of which folks can have these conversations. And I think there's 655 00:52:34.840 --> 00:52:38.840 a lot of opportunity there with this kind of broader spectrum of data and integrations 656 00:52:40.199 --> 00:52:46.760 within customer's existing processes. Yeah, it reminds me a lot of like a 657 00:52:47.920 --> 00:52:53.800 there's a like a a people skill there. You put two people who don't 658 00:52:53.840 --> 00:53:00.239 know each other in a room and anything could possibly happen. They could strick 659 00:53:00.280 --> 00:53:02.239 up conversation, they could sit there in silence. You know, there's just 660 00:53:02.320 --> 00:53:07.880 no way to gauge it. But then if you give them a conversation starter, 661 00:53:07.800 --> 00:53:13.199 then you can sort of like guide the results from there. And I 662 00:53:13.239 --> 00:53:19.039 think I think that's the real value of what the post incident narrative does, 663 00:53:19.199 --> 00:53:23.440 is it's that conversation starter. Yeah, that I mean certainly for us, 664 00:53:23.639 --> 00:53:27.679 as you mentioned, like we use Jelly internally and we do our own learning 665 00:53:27.719 --> 00:53:32.800 reviews. I think the exercise of you know, mitigating the incident, putting 666 00:53:32.880 --> 00:53:37.039 together the learning review, those are valuable experiences for the folks involved. But 667 00:53:37.119 --> 00:53:40.880 getting everyone in the company, because we can do that at twenty one people 668 00:53:42.480 --> 00:53:45.440 into a room to talk about what happened, to ask questions to figure out 669 00:53:45.440 --> 00:53:49.119 what did you know? What did you not know? You know? What 670 00:53:49.199 --> 00:53:52.079 did I know? And I wasn't involved those those sorts of things. That's 671 00:53:52.119 --> 00:54:00.239 where you get really interesting kind of exponential increases and understand And it's not just 672 00:54:00.760 --> 00:54:04.760 the thing that most excites me about these learning reviews is it's not just the 673 00:54:04.840 --> 00:54:08.159 understanding of the technical or the organizational process. It's the understanding of each other. 674 00:54:08.480 --> 00:54:13.400 Right, How how I communicate in these environments? How you communicate what 675 00:54:13.440 --> 00:54:16.760 your expectations are, what sorts of things I need to be better about informing 676 00:54:16.840 --> 00:54:24.000 during response. It's a it's a retro right, and the software can't operate 677 00:54:24.039 --> 00:54:28.719 itself. And so if the people are working effectively together, then the software 678 00:54:28.800 --> 00:54:31.079 is working effectively and if they're not, then it's not. And I think 679 00:54:31.079 --> 00:54:36.119 that's that's one of the really big opportunities, especially for you know, the 680 00:54:36.280 --> 00:54:43.199 large complex organizations in novel economic environments, to figure out, you know, 681 00:54:43.280 --> 00:54:47.880 how do we improve our efficiency in our collaboration so that we can do what 682 00:54:49.320 --> 00:54:53.000 needs to be done? Really exciting, Oh, it is really exciting. 683 00:54:53.119 --> 00:54:59.039 I'm looking forward to seeing how this plays out for y'all. Yeah, I'll 684 00:54:59.079 --> 00:55:01.760 have to let you out there's you know, we're in a phase right now 685 00:55:01.800 --> 00:55:05.599 where there are too many good things for us to do, so we got 686 00:55:05.599 --> 00:55:10.440 to figure out the next best thing and focus on that. But yeah, 687 00:55:10.480 --> 00:55:15.760 that's that's the spot I want to be in. Endless opportunity ahead of us. 688 00:55:15.800 --> 00:55:20.800 We just got to figure out how we're going to get that to our 689 00:55:20.800 --> 00:55:27.719 customers as quickly as possible. Yeah, for sure. Yeah cool. So, 690 00:55:28.440 --> 00:55:30.719 anything else you'd like to share with us about incident response, Jelly, 691 00:55:30.960 --> 00:55:37.039 page of Duty, any topics at all. Yeah, if you're not already 692 00:55:37.119 --> 00:55:39.840 using Page of Duty, take a look. It's the best thing for paging 693 00:55:39.920 --> 00:55:43.519 that I've ever found. And if you want to give Jelly a try, 694 00:55:43.719 --> 00:55:45.840 there's a free trial on the site and we start you off with some pre 695 00:55:45.880 --> 00:55:51.280 built learning reviews so you can see what they look like. Start playing around 696 00:55:51.320 --> 00:55:54.199 in there, and if you have any questions, you know, I'm sure 697 00:55:54.280 --> 00:55:57.960 you'll be able to find me in the show notes here. But it's been 698 00:55:58.000 --> 00:56:00.599 really great to meet you, Will and thank you so much for opportunity to 699 00:56:00.679 --> 00:56:04.559 chat. No, thank you, it's been a great conversation. I've enjoyed 700 00:56:04.599 --> 00:56:07.760 it. And uh, if you're up for it, I would love to 701 00:56:07.400 --> 00:56:12.360 have you back on the show'd that'd be great? All right? So much? 702 00:56:13.000 --> 00:56:15.800 All right? Cool? Well, thanks for listening everyone, and I 703 00:56:15.840 --> 00:56:20.000 will see y'all next week. M