WEBVTT 1 00:00:05.679 --> 00:00:08.480 Hey everybody, and welcome to another episode of the Ruby 2 00:00:08.519 --> 00:00:12.480 Rokes podcast. This week on a panel, we have Luke Stutters. Hello, 3 00:00:12.640 --> 00:00:16.079 we have Dave Kamia. Hey everyone, I'm Charles Maxwood from 4 00:00:16.120 --> 00:00:17.039 dev chat dot TV. 5 00:00:17.359 --> 00:00:20.399 Quick shout out about most valuable dot depth. Go check 6 00:00:20.440 --> 00:00:22.920 it out. We have a special guest this week, and 7 00:00:22.960 --> 00:00:24.800 that is Paul Zeich. 8 00:00:25.199 --> 00:00:27.039 Zeich, well done, thank you. 9 00:00:27.920 --> 00:00:29.679 Now you're here from Checker. 10 00:00:29.760 --> 00:00:33.039 You gave a talk at rails komf about how you 11 00:00:33.119 --> 00:00:35.000 broke stuff or somebody broke stuff. 12 00:00:35.079 --> 00:00:36.600 Do you want to just kind of give us a 13 00:00:36.640 --> 00:00:37.759 quick intro to who you are. 14 00:00:37.679 --> 00:00:39.399 And what you do, and then we'll dive in and 15 00:00:39.439 --> 00:00:42.359 talk about what broke and how you've figured it out. 16 00:00:42.600 --> 00:00:42.960 Sure. 17 00:00:43.200 --> 00:00:46.920 So, I've been a software engineer for about ten years. Recently, 18 00:00:46.960 --> 00:00:49.640 in the last year or so, transitioned into an engineering 19 00:00:49.679 --> 00:00:53.000 management role. But I've worked at a number of different 20 00:00:53.560 --> 00:00:57.280 small startups. I and joined Checker in twenty seventeen when 21 00:00:57.280 --> 00:01:00.520 the company was at about one hundred employees thirty engineers. 22 00:01:00.679 --> 00:01:03.119 Contributed as an engineer for a couple of years to 23 00:01:03.200 --> 00:01:06.799 our team, and then have recently transitioned, like I said, 24 00:01:06.840 --> 00:01:09.799 into a jerry management role at the company. 25 00:01:09.959 --> 00:01:10.359 Very cool. 26 00:01:10.400 --> 00:01:12.840 I actually have a Checker T shirt in my closet 27 00:01:12.840 --> 00:01:15.719 that I never wear. It's check R for those that 28 00:01:15.799 --> 00:01:18.280 are listening and not reading it. Yeah, So why don't 29 00:01:18.280 --> 00:01:20.519 you kind of te us up for this as far 30 00:01:20.599 --> 00:01:24.280 as yeah, what happened, what broke? Yeah, give us a 31 00:01:24.319 --> 00:01:28.400 sort of a preliminary timeline and explain what Checker does 32 00:01:28.400 --> 00:01:29.159 and why that matters. 33 00:01:29.480 --> 00:01:29.799 Sure. 34 00:01:30.040 --> 00:01:34.239 So checker Checker was founded in twenty fourteen. Daniel and Jonathan, 35 00:01:34.519 --> 00:01:38.680 our founders, had worked in the on demand space another 36 00:01:38.760 --> 00:01:42.480 company and had discovered that is very difficult containing great 37 00:01:42.519 --> 00:01:46.359 background checks into their onboarding process. Background checks tend to 38 00:01:46.359 --> 00:01:49.480 be a very important final safety step for a lot 39 00:01:49.480 --> 00:01:52.120 of these companies to make sure that their platform is 40 00:01:52.159 --> 00:01:56.159 going to be safe and secure for their customers, and 41 00:01:56.239 --> 00:01:59.959 so in twenty fourteen they started an automated background check company. 42 00:02:00.239 --> 00:02:05.120 And initially the biggest selling point was that Checker abstracted 43 00:02:05.120 --> 00:02:08.280 away a lot of the complexity of background check process, 44 00:02:08.520 --> 00:02:13.639 collecting candidate information and then executing that flow and exposing 45 00:02:13.680 --> 00:02:17.199 that via an API that was developed in a sinatrap. 46 00:02:17.479 --> 00:02:21.439 And three years later, in twenty seventeen, I just joined 47 00:02:21.439 --> 00:02:25.039 about four or five months, four or five months before 48 00:02:25.280 --> 00:02:28.400 this particular incident happened. Fast forward to that point. We're 49 00:02:28.479 --> 00:02:31.800 running i'd say, a few million checks a year for 50 00:02:32.080 --> 00:02:36.000 a variety of different customers. Most of those customers use 51 00:02:36.039 --> 00:02:39.599 our API, like I said before, to manage that process, 52 00:02:39.639 --> 00:02:42.879 and they do most of the collection and interface with 53 00:02:42.919 --> 00:02:45.759 the candidate on their side in their own application. 54 00:02:46.120 --> 00:02:47.080 Oh that's interesting. 55 00:02:47.439 --> 00:02:50.199 Yeah, I think a lot of the background check portals 56 00:02:50.199 --> 00:02:54.240 that I've seen they yeah, they're like the fully baked 57 00:02:54.280 --> 00:02:57.400 portal instead of yeah, being a background service that somebody 58 00:02:57.439 --> 00:02:59.800 else can integrate into their own app. 59 00:03:00.120 --> 00:03:02.120 Did yeah exactly that ground check on me? 60 00:03:02.319 --> 00:03:05.680 Before this episode, I did not that there are a 61 00:03:05.719 --> 00:03:11.280 lot of very important guidelines in stipulations governed by the 62 00:03:11.280 --> 00:03:14.560 Federal Credit Reporting Act that make sure that you have 63 00:03:14.599 --> 00:03:17.319 to have a permissible purpose for running a background check. 64 00:03:17.439 --> 00:03:20.199 So in this case, most of our customers are using 65 00:03:20.680 --> 00:03:25.120 the permissible purpose around employment as the reason for actually 66 00:03:25.199 --> 00:03:25.840 running that check. 67 00:03:25.919 --> 00:03:28.240 Well, that's no fun, I know, right, I want to 68 00:03:28.240 --> 00:03:32.960 know everybody's dirty secrets. Interesting, So, yeah, why don't you 69 00:03:32.960 --> 00:03:35.759 tell us a little bit about what went down with 70 00:03:35.960 --> 00:03:36.759 the app? 71 00:03:36.879 --> 00:03:37.159 Right? 72 00:03:37.439 --> 00:03:41.639 So, like I said in twenty seventeen, Checker at this 73 00:03:41.719 --> 00:03:44.759 point was a pretty important component of a number of 74 00:03:44.759 --> 00:03:49.159 customers onboarding process. But we'd started off small and things 75 00:03:49.240 --> 00:03:51.840 grew quickly in a lot of ways. We were just 76 00:03:51.879 --> 00:03:55.000 trying to keep the lights on and scale the system 77 00:03:55.680 --> 00:03:58.759 along with our customers as they continued to grow. On 78 00:03:58.840 --> 00:04:01.280 demand is growing a lot this time as well. So 79 00:04:01.639 --> 00:04:06.360 in twenty seventeen, we were doing some fairly routine changes 80 00:04:06.400 --> 00:04:09.039 to a data model. I wasn't directly involved with that, 81 00:04:09.439 --> 00:04:12.919 but we were changing something from an eteger ID to 82 00:04:14.280 --> 00:04:17.160 UID and the references, and there were some backfills that 83 00:04:17.199 --> 00:04:21.079 needed to happen, And so an engineer executed a script 84 00:04:21.240 --> 00:04:26.120 on a Friday afternoon, which is always a great idea, 85 00:04:26.480 --> 00:04:29.920 and they executed a script at about four thirty pm, 86 00:04:30.120 --> 00:04:33.040 probably went and grabbed something, had a little happy hour, 87 00:04:33.079 --> 00:04:37.160 and then headed home. And about an hour later we 88 00:04:37.199 --> 00:04:41.680 started to receive a few various different pages to completely 89 00:04:41.839 --> 00:04:44.120 unrelated teams that didn't really know what was going on 90 00:04:44.160 --> 00:04:46.279 in terms of this backfill, and it didn't look like 91 00:04:46.319 --> 00:04:49.319 anything too serious. It was just an elevated number of 92 00:04:49.360 --> 00:04:52.879 exceptions in our client application that does some of the 93 00:04:52.920 --> 00:04:57.439 candidate PII collection, and so we just decided to that 94 00:04:57.560 --> 00:05:00.759 team decided to snooze that decided to just kind of 95 00:05:00.759 --> 00:05:01.240 ignore it. 96 00:05:01.319 --> 00:05:04.160 Yeah, so people that aren't aware. PII is an acronym 97 00:05:04.199 --> 00:05:08.480 for personally identifiable information and is usually protected by a law. 98 00:05:08.839 --> 00:05:09.959 Thank you anyway, go ahead. 99 00:05:10.319 --> 00:05:14.279 So come Saturday morning, this has been going on for 100 00:05:14.319 --> 00:05:18.279 about twelve hours now, this exception comes in again, and 101 00:05:18.680 --> 00:05:21.279 at that point someone on our team actually decided to 102 00:05:21.360 --> 00:05:25.160 escalate that and get more stakeholders involved. We had some 103 00:05:25.439 --> 00:05:28.319 variety of other issues going on. We just migrated from 104 00:05:28.639 --> 00:05:32.360 one deployment platform to Kubernetes, and so we had some 105 00:05:32.399 --> 00:05:35.439 issues getting onto the cluster. There are too many of 106 00:05:35.480 --> 00:05:36.879 us trying to get on at the same time, so 107 00:05:36.879 --> 00:05:39.639 we ended up all having to actually go into the 108 00:05:39.680 --> 00:05:43.000 office to the physical internet to finally get in debug 109 00:05:43.040 --> 00:05:46.319 the issue. So we had a couple of other confounding 110 00:05:46.480 --> 00:05:48.879 issues come up at the same time that made the 111 00:05:48.920 --> 00:05:52.560 process of response even worse. So finally, this is maybe 112 00:05:53.040 --> 00:05:55.399 ten o'clock in the morning, ten or eleven in the morning, 113 00:05:55.759 --> 00:05:58.560 we finally, after being able to take a look at that, 114 00:05:59.079 --> 00:06:01.959 identified what they issue was and we were starting We're 115 00:06:01.959 --> 00:06:05.279 responding to about fifty to sixty percent of the one 116 00:06:05.279 --> 00:06:08.120 of the most critical end points on our system, which 117 00:06:08.160 --> 00:06:11.279 is to actually create requests to make a report. So 118 00:06:11.439 --> 00:06:14.879 after you've collected the candidate's information, you say, please execute 119 00:06:14.879 --> 00:06:16.800 this report so we can get that back, and that's 120 00:06:16.800 --> 00:06:20.560 a synchronous request that you make using our API. And 121 00:06:20.720 --> 00:06:24.560 when that request was failing, it was failing about forty 122 00:06:24.600 --> 00:06:26.160 to fifty percent of the time with a four or 123 00:06:26.160 --> 00:06:29.439 four response, which isn't really expected. So at that point 124 00:06:29.480 --> 00:06:31.040 we were finally able to pin down the issue and 125 00:06:31.160 --> 00:06:34.279 it came back to this script, and it turned out 126 00:06:34.439 --> 00:06:37.319 that when you went to create this report, we would 127 00:06:37.360 --> 00:06:42.000 look for create these additional subobjects called screenings, and due 128 00:06:42.040 --> 00:06:44.360 to the script, we had actually created an issue where 129 00:06:44.480 --> 00:06:46.560 validation would cause. 130 00:06:46.079 --> 00:06:48.439 The reports to fail to create in this EDU. 131 00:06:48.399 --> 00:06:51.600 Case, So there's some confounding issues with the way that 132 00:06:51.639 --> 00:06:54.040 we had set up the data modeling to begin with 133 00:06:54.120 --> 00:06:57.399 that we were trying to work around and this acception happened. 134 00:06:57.600 --> 00:06:58.439 But when we. 135 00:06:58.399 --> 00:07:02.480 Finally fix the issue, that where we shifted more into 136 00:07:02.600 --> 00:07:05.439 what could what actually went wrong and what were the 137 00:07:05.480 --> 00:07:08.800 real issues that caused us this outage gotcha. 138 00:07:08.920 --> 00:07:11.560 So I'm curious as you work through this, what did 139 00:07:11.600 --> 00:07:13.399 you add to your workflow to make sure that this 140 00:07:13.560 --> 00:07:15.959 doesn't happen again, because I mean, some of it's going 141 00:07:16.040 --> 00:07:19.360 to be technical, right, it's testing or you know, maybe 142 00:07:19.360 --> 00:07:21.759 you set up a staging environment or something like that, 143 00:07:22.240 --> 00:07:24.720 and some of it is going to be hey, when 144 00:07:24.800 --> 00:07:27.399 this kind of alert comes up, do this thing right, 145 00:07:27.439 --> 00:07:29.800 because it sounded like you did have some early indication 146 00:07:29.879 --> 00:07:31.160 that this happened, right. 147 00:07:31.279 --> 00:07:33.560 So I think the first most important thing that we 148 00:07:33.600 --> 00:07:36.360 did was that are really from the beginning, we have 149 00:07:36.560 --> 00:07:39.959 had what we call a blame this culture. I think 150 00:07:39.959 --> 00:07:42.439 it's a common term now in the industry. But the 151 00:07:42.519 --> 00:07:46.800 idea there is to really focus on learning from issues, 152 00:07:46.920 --> 00:07:51.759 not trying to find who made the particular mistake, and 153 00:07:51.879 --> 00:07:54.959 trying to look at what processes you're missing and what 154 00:07:55.240 --> 00:07:57.439 changes you need to make in just your code base 155 00:07:57.480 --> 00:07:59.959 as well that would have prevented the problem from happening, 156 00:08:00.120 --> 00:08:03.319 so not trying to focus on the individual mistake. 157 00:08:03.079 --> 00:08:05.399 That was made. So as part of that, we did a. 158 00:08:05.399 --> 00:08:09.240 Post mortem doc and we went through and identified things 159 00:08:09.319 --> 00:08:12.879 like one we should really have like a dedicated script 160 00:08:13.120 --> 00:08:17.160 repository that goes through a code review process. So that's 161 00:08:17.319 --> 00:08:20.759 that's one thing we implemented, and we made some safeguards 162 00:08:20.800 --> 00:08:24.040 and to address this particular issue with the data models 163 00:08:24.319 --> 00:08:26.959 as well. But I think for everyone that the bigger 164 00:08:27.040 --> 00:08:30.120 issue was really the fact that we missed the outage 165 00:08:30.800 --> 00:08:34.320 for so long. And we did actually have some monitoring 166 00:08:34.320 --> 00:08:36.399 in place for this particular issue that would have that 167 00:08:36.399 --> 00:08:39.399 should have page for the downtime that we were experiencing 168 00:08:39.559 --> 00:08:43.279 for some report creation, But it turned out that our 169 00:08:43.320 --> 00:08:46.840 monitors were really just not set up in the most 170 00:08:46.840 --> 00:08:51.200 effective way to trigger for that particular type of outage 171 00:08:51.240 --> 00:08:54.600 in this case is a partial outage, and that's that 172 00:08:54.679 --> 00:08:57.960 requires a much more sensitive monitor in order for us 173 00:08:58.000 --> 00:09:01.639 to detect everything. We designed the porehand was much more 174 00:09:01.919 --> 00:09:04.559 targeted towards a complete failure of our system. 175 00:09:04.879 --> 00:09:07.679 And so was this something that could have been caught 176 00:09:07.759 --> 00:09:09.600 by automated tests. 177 00:09:09.879 --> 00:09:13.399 This particular issue most likely could not have been caught 178 00:09:13.440 --> 00:09:16.639 by an automated test because it was a it was 179 00:09:17.039 --> 00:09:20.679 so so outside of the norm of what we expected 180 00:09:20.840 --> 00:09:24.159 data to look like. So we had particularly I mean, 181 00:09:24.159 --> 00:09:26.799 we had of course unit tests for everything that we 182 00:09:26.799 --> 00:09:30.039 were running, and we had requests specs as well. We 183 00:09:30.159 --> 00:09:32.960 did not have like an end to end environment set 184 00:09:33.039 --> 00:09:35.120 up for like a staging environment where you could run 185 00:09:35.159 --> 00:09:36.159 these tests and to end. 186 00:09:36.679 --> 00:09:40.480 But again, the data in that this particular. 187 00:09:40.000 --> 00:09:43.919 Case was very old, and it was essentially doing a 188 00:09:44.000 --> 00:09:47.679 migration where that data was in a state that wasn't 189 00:09:47.759 --> 00:09:50.159 anywhere in our code base at this point, So I'm 190 00:09:50.200 --> 00:09:52.720 not sure we could have anticipated this particular issue. 191 00:09:52.799 --> 00:09:56.000 What was the Fordland? Did everyone like phone up and 192 00:09:56.000 --> 00:09:56.919 get really angry? 193 00:09:57.159 --> 00:09:59.000 Oh? From a customer's perpective. 194 00:09:58.799 --> 00:10:01.159 Yeah, yeah, this is the the best bit of outage 195 00:10:01.159 --> 00:10:03.799 stories is the kind of the human cost of whoever 196 00:10:03.840 --> 00:10:07.080 has to answer the phone the next week code drama. 197 00:10:06.720 --> 00:10:10.000 Right, That's that's always one of the especially as your 198 00:10:10.240 --> 00:10:13.559 as your application becomes more important to customers and what 199 00:10:13.600 --> 00:10:18.279 your service, the impact to customers is more and more extreme. 200 00:10:18.559 --> 00:10:21.240 And so in this case, piezas a Friday night, it 201 00:10:21.279 --> 00:10:24.000 wasn't something where a lot of our customers were actively 202 00:10:24.639 --> 00:10:28.159 monitoring on their end as well. Fortunately, we were able 203 00:10:28.200 --> 00:10:31.320 to see that retries were happening, and many of our 204 00:10:31.320 --> 00:10:35.720 customers use a retry fallback mechanism, so they were able 205 00:10:35.799 --> 00:10:39.000 to just allow those to run through. But this is 206 00:10:39.039 --> 00:10:43.120 particularly tricky in this case because there wasn't actually like 207 00:10:43.159 --> 00:10:47.559 a record idea for many of these these particular responses. Fortunately, 208 00:10:47.600 --> 00:10:50.600 we did have a. We do keep API logs, so 209 00:10:50.639 --> 00:10:53.080 we were able to see exactly which requests failed for 210 00:10:53.159 --> 00:10:55.559 each of our customers, and so we were able to 211 00:10:55.679 --> 00:10:58.720 then reach out to our customer success team and they 212 00:10:58.799 --> 00:11:01.159 were able to start to share the impact with each 213 00:11:01.159 --> 00:11:04.559 of those customers pretty quickly. I will say that we've 214 00:11:04.879 --> 00:11:07.759 done a lot of work to make our customer communication 215 00:11:07.840 --> 00:11:10.559 a lot, a lot more polished since then, and that's 216 00:11:10.559 --> 00:11:13.240 something that we're really focusing on now as well, and 217 00:11:13.360 --> 00:11:16.879 just being able to give more visibility customers sooner. And 218 00:11:16.960 --> 00:11:18.960 one of the most important things there is when it 219 00:11:18.960 --> 00:11:21.120 comes to monitoring, is that you really want to be 220 00:11:21.159 --> 00:11:23.720 able to find the issue and be able to start 221 00:11:23.759 --> 00:11:27.600 to investigate it before you You don't want a customer to 222 00:11:27.639 --> 00:11:30.600 identify it first. You should really understand what's happening in 223 00:11:30.639 --> 00:11:33.799 your system before anyone else detects that issue. 224 00:11:33.840 --> 00:11:37.120 And I guess, but this specific not this specific product, 225 00:11:37.159 --> 00:11:41.679 but kind of product where your customers are consuming your API. 226 00:11:42.200 --> 00:11:45.559 You're also at the mercy of their implementation too, so 227 00:11:46.240 --> 00:11:49.320 you know, making a kind of called against you, and 228 00:11:49.799 --> 00:11:52.720 if that call is failing, you know, you've got to 229 00:11:52.759 --> 00:11:55.159 hope that their system can cope with that as well. 230 00:11:55.600 --> 00:11:59.200 Exactly if some of these requests are happening in the browser, 231 00:11:59.480 --> 00:12:02.960 or we're not set up to automatically really try, that 232 00:12:02.960 --> 00:12:05.480 could be a much worse impact on the customers. 233 00:12:05.759 --> 00:12:08.120 Can we talk about the blameless culture for a bit. 234 00:12:08.679 --> 00:12:11.600 This is this is a new idea. And when I 235 00:12:11.679 --> 00:12:14.320 was managing engineering teams, I used to have what I 236 00:12:14.360 --> 00:12:17.279 called the finger of blame. So I used to around 237 00:12:17.320 --> 00:12:19.559 I would hold up my finger in the meeting and 238 00:12:19.600 --> 00:12:22.960 I'd introduce the finger as the finger of blame, and 239 00:12:23.000 --> 00:12:25.960 then we'd work out who the finger of blame should 240 00:12:25.960 --> 00:12:28.960 be pointing to. Now, more often than not, of course 241 00:12:29.000 --> 00:12:31.360 it was me. So the finger of blame was a 242 00:12:31.399 --> 00:12:34.600 double edged finger, but it was it was a kind 243 00:12:34.600 --> 00:12:36.960 of way of you know, people take it very seriously 244 00:12:37.039 --> 00:12:38.759 when they mess up the kind of stuff, so you 245 00:12:38.879 --> 00:12:41.120 kind of have to get your get your team back 246 00:12:41.200 --> 00:12:43.759 on board. So it's a way of kind of lightening 247 00:12:43.799 --> 00:12:49.159 the mood after after that week's disaster. But a blameless culture, 248 00:12:49.200 --> 00:12:51.960 as you said, is a kind of more more sophisticated 249 00:12:52.240 --> 00:12:55.480 way of doing it instead of pointing a jovial finger 250 00:12:55.759 --> 00:12:58.799 at the person who messed up. What's what does that 251 00:12:58.960 --> 00:13:01.120 look like? I mean, you know, do you just go 252 00:13:01.200 --> 00:13:03.960 around telling people it's not their ful or you know, 253 00:13:04.000 --> 00:13:07.279 how do you implement a blameless culture in what sounds 254 00:13:07.320 --> 00:13:09.039 like quite a big engineering team. 255 00:13:09.360 --> 00:13:11.679 I think I think it starts for us with it 256 00:13:11.720 --> 00:13:17.000 really started with our CTO Jonathan and co founder making 257 00:13:17.039 --> 00:13:20.519 that a priority from pretty much day one, basically from 258 00:13:20.519 --> 00:13:24.840 the beginning of our process. When we've had issues or incidents, 259 00:13:25.120 --> 00:13:28.000 we've done a post mortem doc, we've had a process 260 00:13:28.039 --> 00:13:32.960 around that, and it's always been very forward facing, very 261 00:13:33.080 --> 00:13:35.480 much about what could we have done better, what can 262 00:13:35.480 --> 00:13:37.960 we improve, what are the things we should be doing 263 00:13:38.399 --> 00:13:42.279 going forward. So I think having that first touch point 264 00:13:42.440 --> 00:13:45.000 and really having that emphasis from the beginning was really 265 00:13:45.039 --> 00:13:48.559 important and cascades down. I think as you're building out 266 00:13:48.639 --> 00:13:52.120 a bigger engineering team, that's critical is to be able 267 00:13:52.159 --> 00:13:55.200 to just continue to build keep that culture going. And 268 00:13:55.240 --> 00:13:58.320 I think that's that's something a challenge to continue doing. 269 00:13:58.559 --> 00:14:00.720 But I think as we grow and we've been able 270 00:14:00.799 --> 00:14:03.559 to do that so far, so I think that was 271 00:14:03.600 --> 00:14:06.000 step number one. I think a second piece of it 272 00:14:06.039 --> 00:14:08.600 is understanding and trying to understand when it's more of 273 00:14:08.600 --> 00:14:13.480 a process issue versus something that someone particularly did wrong. 274 00:14:13.600 --> 00:14:15.480 And I think a lot of the time. I think 275 00:14:15.519 --> 00:14:18.240 a lot of incidents do occur because you're trying to 276 00:14:18.639 --> 00:14:22.200 you're trying to make different prioritization decisions, and you're trying 277 00:14:22.240 --> 00:14:25.799 to make sure that you anticipate things in advance or 278 00:14:26.080 --> 00:14:31.000 failure ferialments, and sometimes you just miss those. And those 279 00:14:31.000 --> 00:14:34.399 are particular cases where I think the management team needs 280 00:14:34.440 --> 00:14:37.720 to really take responsibility for it. It's not an individual 281 00:14:38.000 --> 00:14:42.200 issue that caused that particular downtime or that that was 282 00:14:42.600 --> 00:14:45.080 necessarily that one piece of code, and so it could 283 00:14:45.080 --> 00:14:48.000 be just an example. Is I mean, this is an 284 00:14:48.000 --> 00:14:50.600 example I think actually where we had some technical debt. 285 00:14:50.600 --> 00:14:53.080 We were trying to clean it up, and that was 286 00:14:53.120 --> 00:14:55.080 a good thing, but I think we didn't necessarily have 287 00:14:55.240 --> 00:14:58.240 everything in place to be able to address that technical 288 00:14:58.320 --> 00:15:02.960 debt effectively, and that's not necessarily one engineer's responsibility to 289 00:15:03.039 --> 00:15:03.840 be out in front. 290 00:15:03.639 --> 00:15:05.360 Of Yeah, one thing I just want to add is 291 00:15:05.399 --> 00:15:08.519 that I like the blameless culture just from the sense 292 00:15:08.600 --> 00:15:13.000 of unless somebody is either malicious, which I have never 293 00:15:13.039 --> 00:15:16.440 ever ever encountered, or is chronically reckless, which I've also 294 00:15:16.559 --> 00:15:19.759 never encountered. Right, everybody is usually trying to pull along 295 00:15:20.440 --> 00:15:23.039 in the same way you know, if somebody has that issue, 296 00:15:23.080 --> 00:15:26.519 you identify it pretty fast and you usually are able 297 00:15:26.559 --> 00:15:30.600 to counter it before it becomes a real problem. But yeah, 298 00:15:31.159 --> 00:15:34.480 just to put that together then you know, yeah, the 299 00:15:34.519 --> 00:15:36.679 rest of it, it's, hey, look, we're on the same team. 300 00:15:36.679 --> 00:15:39.480 We're all trying to get the same place. So let's 301 00:15:39.519 --> 00:15:41.159 talk about how we can do this better so that 302 00:15:41.159 --> 00:15:44.080 doesn't happen again, because next time it might be me, right, 303 00:15:44.440 --> 00:15:48.279 that misses a critical step. And I don't want you 304 00:15:48.320 --> 00:15:51.480 while fingering me either. I mean, I want to learn 305 00:15:51.480 --> 00:15:53.960 from it, but I you know, we don't want people 306 00:15:54.399 --> 00:15:57.759 walking around in fear. Instead, if somebody screws up, we 307 00:15:57.759 --> 00:15:59.799 want them to come forward and say, hey, I might 308 00:15:59.840 --> 00:16:02.600 have this up before it becomes an issue next time. 309 00:16:02.960 --> 00:16:03.480 Absolutely. 310 00:16:03.480 --> 00:16:06.320 And I think one other thing to highlight here is 311 00:16:06.360 --> 00:16:09.360 that when you don't have a blameless culture, folks are 312 00:16:09.360 --> 00:16:12.159 going to be very afraid to speak out when they 313 00:16:12.159 --> 00:16:14.879 do soon an issue, whether it was there they think 314 00:16:15.159 --> 00:16:17.639 it was their mistake or someone else's, They're not going 315 00:16:17.720 --> 00:16:20.360 to want to escalate that issue and make sure that 316 00:16:20.360 --> 00:16:24.279 it gets attention necessarily. And so one of the best 317 00:16:24.320 --> 00:16:27.159 side effects of having a blameless culture is that you 318 00:16:27.240 --> 00:16:31.120 get really engaged response and everyone's going to work together 319 00:16:31.240 --> 00:16:33.679 to try to address the issue. I think that even 320 00:16:33.799 --> 00:16:39.240 cascades down to customer communication as well, because when you're 321 00:16:39.279 --> 00:16:42.399 really engaged in trying to do that, then you're doing 322 00:16:42.440 --> 00:16:44.360 the best thing for the customers as well, because you're 323 00:16:44.360 --> 00:16:48.000 trying to address these issues head on and not try 324 00:16:48.039 --> 00:16:49.919 to find ways to kind of smooth them out under 325 00:16:49.919 --> 00:16:50.399 the surface. 326 00:16:50.799 --> 00:16:53.960 Yeah, it also and this is important, and sometimes I 327 00:16:53.960 --> 00:16:56.879 think people hear this and they're going to go that 328 00:16:56.960 --> 00:16:59.600 sounds a little scary, But you want people to take 329 00:16:59.799 --> 00:17:03.240 chances sometimes, right, you want people to kind of take 330 00:17:03.279 --> 00:17:06.200 a shot at making things better. That opens it up 331 00:17:06.200 --> 00:17:09.000 to them to do that, right, it's oh, well, you know, 332 00:17:09.200 --> 00:17:12.039 I tried this tweak on the Jinkins file, or I 333 00:17:12.160 --> 00:17:15.279 tried this tweak on the Kubernetes setup, or I tried 334 00:17:15.279 --> 00:17:17.960 this tweak on this other thing, and a lot of 335 00:17:18.000 --> 00:17:20.079 times those things pay off. But if you don't give 336 00:17:20.119 --> 00:17:22.680 people the freedom to go for it, a lot of 337 00:17:22.680 --> 00:17:24.160 times you're going to miss out on a lot of 338 00:17:24.160 --> 00:17:27.400 those benefits. And again, as long as they're not being 339 00:17:27.480 --> 00:17:29.920 reckless about it, right, so they're taking the steps, they're 340 00:17:30.000 --> 00:17:31.960 verifying it on their own system and things like that. 341 00:17:32.640 --> 00:17:35.759 Then you benefit much much more from people being willing 342 00:17:35.799 --> 00:17:38.720 to take a shot. So yeah, so with the blameless culture, 343 00:17:38.759 --> 00:17:41.759 I'm curious. So you get together and you start identifying 344 00:17:41.759 --> 00:17:43.920 what the issue is. So what does that look like 345 00:17:44.000 --> 00:17:47.359 then as far as figuring out what's going on, because 346 00:17:47.359 --> 00:17:50.200 you're not pointing fingers, but you are looking for the 347 00:17:50.240 --> 00:17:52.160 commit that made the problem, right you are. 348 00:17:53.160 --> 00:17:55.440 I think at the end of the day, you're going 349 00:17:55.519 --> 00:17:57.880 to try to find the root cause. Right, You're going 350 00:17:57.960 --> 00:17:59.519 to look for that commit, You're going to look for 351 00:17:59.559 --> 00:18:02.440 the law. Maybe it was a script that was logged 352 00:18:02.440 --> 00:18:06.880 into your logging system, whatever it is, You're going to 353 00:18:06.920 --> 00:18:09.480 look for that and look for the root cause. So honestly, 354 00:18:09.480 --> 00:18:11.599 a lot of times, you know, maybe what caused the 355 00:18:11.640 --> 00:18:14.400 issue from whether if it was something that was specifically 356 00:18:14.440 --> 00:18:17.079 run by a specific person, and they probably feel a 357 00:18:17.119 --> 00:18:19.200 little bit of guilt there, but there's no reason. 358 00:18:18.920 --> 00:18:19.920 To lay on more there. 359 00:18:20.160 --> 00:18:22.880 And I think everyone, like you said, feels a lot 360 00:18:22.880 --> 00:18:26.759 of responsibility around the work that they're doing already, so 361 00:18:26.839 --> 00:18:30.160 there's no reason to overemphasize that. So what that looks 362 00:18:30.240 --> 00:18:34.799 like is typically the team that is impacted is really 363 00:18:34.839 --> 00:18:37.799 going to own that post mortem, and that's one way 364 00:18:37.960 --> 00:18:42.000 for you to feel like you're resolving the incident or 365 00:18:42.039 --> 00:18:45.160 that they issue that the cost incident. So this is 366 00:18:45.200 --> 00:18:47.440 a definitely become a different a bit of a different 367 00:18:47.440 --> 00:18:48.640 process as the team is growing. 368 00:18:48.680 --> 00:18:50.160 When we're at thirty. 369 00:18:50.039 --> 00:18:52.119 I think it's a little bit easier just to know 370 00:18:52.200 --> 00:18:55.240 exactly who should work on those types of mitigations. It 371 00:18:55.279 --> 00:18:59.279 doesn't typically it's pretty isolated to a specific team. As 372 00:18:59.359 --> 00:19:02.079 the team has grown, growing and the system is growing, 373 00:19:02.480 --> 00:19:05.480 that's definitely become more of a challenge because sometimes incidents 374 00:19:05.519 --> 00:19:08.960 happen because different issues that multiple teams have introduced, or 375 00:19:09.000 --> 00:19:11.359 maybe there's multiple teams that need to be involved in 376 00:19:11.400 --> 00:19:15.000 the mitigation and for that in that case, we've definitely 377 00:19:15.000 --> 00:19:17.720 been trying to involve our post mortem process and the 378 00:19:17.759 --> 00:19:21.200 action items. So we have a program manager that one 379 00:19:21.240 --> 00:19:24.920 of her responsibilities is specifically around making sure that we 380 00:19:24.960 --> 00:19:28.839 are coordinating some of those efforts and meeting some essays. 381 00:19:28.880 --> 00:19:34.599 So we had to some additional rules and coordination around 382 00:19:34.960 --> 00:19:38.079 the process as we've as we've started to grow, a 383 00:19:38.079 --> 00:19:41.200 lot of it was just on the individual teams initially, 384 00:19:41.240 --> 00:19:44.119 and now as we've grown again, there's more processes involved. 385 00:19:44.200 --> 00:19:46.039 I think that's pretty common thing that you have to 386 00:19:46.079 --> 00:19:47.559 introduce as teams grow. 387 00:19:48.000 --> 00:19:51.559 I will say that if you've got relatives who are 388 00:19:51.599 --> 00:19:56.039 in the medical profession, especially if they're pathologies, even the 389 00:19:56.160 --> 00:19:59.119 use of the term person posts more toem makes me 390 00:19:59.200 --> 00:20:03.359 uncomfortable because those are no fun at all. But yeah, 391 00:20:03.400 --> 00:20:07.240 it's also a word that we use. So yeah, it's oh, 392 00:20:07.759 --> 00:20:10.799 it just makes me. Oh, it's creepy. It's all zombies. 393 00:20:11.160 --> 00:20:11.519 I don't know. 394 00:20:11.680 --> 00:20:16.960 Yeah, the post mortem brings me flashbacks to episodes of 395 00:20:16.960 --> 00:20:19.400 The X Files in the nineties when Dana Scuddy was 396 00:20:19.440 --> 00:20:20.680 taking a Navy in apart. 397 00:20:20.960 --> 00:20:24.119 Yeah, but it does give you a little perspective too, right, 398 00:20:24.200 --> 00:20:27.440 because usually in our post mortems, we're talking about what 399 00:20:27.480 --> 00:20:30.440 went wrong with the system, not that somebody actually died 400 00:20:30.519 --> 00:20:31.720 because of this, Right, I. 401 00:20:31.799 --> 00:20:33.519