WEBVTT 1 00:00:05.040 --> 00:00:06.320 What's up born, How you doing. 2 00:00:07.160 --> 00:00:10.599 I'm doing I'm doing pretty well. Thanks, Thanks, well cool. 3 00:00:10.800 --> 00:00:14.039 I cut my intro a little shorter this week because 4 00:00:14.759 --> 00:00:17.079 I just kind of assume, and maybe this is a 5 00:00:17.079 --> 00:00:20.760 bad assumption on my part, that people know which podcasts 6 00:00:20.800 --> 00:00:22.800 are listening to because they had to click on it, 7 00:00:23.079 --> 00:00:25.800 and so I felt like it was kind of redundant 8 00:00:25.800 --> 00:00:29.600 to say, welcome to the Adventures in DevOps podcast. 9 00:00:30.280 --> 00:00:32.240 You just you just did it right. 10 00:00:32.320 --> 00:00:35.359 That was subtle though, right, that was clever. I'm gonna 11 00:00:35.479 --> 00:00:37.439 just pat myself on the back for that one. 12 00:00:37.600 --> 00:00:41.119 Yeah, I've got I've got actually an interesting fact that 13 00:00:41.159 --> 00:00:42.679 I can share, so you know, we can jump into 14 00:00:42.679 --> 00:00:47.159 that there was an OTP provider that actually changed hands 15 00:00:47.200 --> 00:00:51.240 similar to the XC vulnerability and compression and Linux not 16 00:00:51.280 --> 00:00:54.520 too long ago. And for those of you that did OTPs, 17 00:00:54.520 --> 00:00:56.240 that's one time passwords, so you can think. 18 00:00:56.119 --> 00:00:57.759 Of like an off path that you got installed in 19 00:00:57.799 --> 00:00:58.240 your phone. 20 00:00:58.880 --> 00:01:02.119 And it's sort of ridiculous that this even happened, because 21 00:01:02.200 --> 00:01:04.519 if you think about how bad it is for a 22 00:01:04.640 --> 00:01:08.200 open source library to get co opted by a militias attacker, 23 00:01:08.239 --> 00:01:11.319 to have an application on your phone that is also 24 00:01:11.400 --> 00:01:15.200 responsible for security for two factor off codes to change hands. 25 00:01:15.480 --> 00:01:19.560 That provider now has access to every single one of 26 00:01:19.599 --> 00:01:22.519 those users two factor off and it could be even 27 00:01:22.560 --> 00:01:25.799 primary factor if it comes to password resets and whatnot. 28 00:01:25.959 --> 00:01:29.480 So that's a great way of stealing credentials. And I 29 00:01:29.480 --> 00:01:31.000 don't think it's an attack factor that a lot of 30 00:01:31.040 --> 00:01:33.040 people think about. I think they, you know, it's like whatever, 31 00:01:33.079 --> 00:01:35.359 it just stores my two factor codes, doesn't really matter. 32 00:01:35.959 --> 00:01:38.879 But now there's actually a huge problem that could come 33 00:01:38.920 --> 00:01:39.599 up because of that. 34 00:01:40.359 --> 00:01:43.879 Nice I look forward to it. I'm just excited about that. 35 00:01:46.439 --> 00:01:48.359 I think everyone really has to switch over to weboth 36 00:01:48.400 --> 00:01:50.680 and that's the truth secret. And if you don't know 37 00:01:50.719 --> 00:01:52.359 what that is, come and talk to me after the show. 38 00:01:52.879 --> 00:01:55.439 I'm happy to give everyone an earful about that. 39 00:01:56.359 --> 00:01:57.840 Or can we just give up on it and just 40 00:01:57.879 --> 00:02:01.280 everyone uses password all lower case letters for their password. 41 00:02:01.640 --> 00:02:06.200 I mean there's some credibility to that approach. 42 00:02:05.879 --> 00:02:10.080 Right, there's a whole episode there. 43 00:02:11.840 --> 00:02:15.479 Maybe maybe, But today's episode, we're talking about one of 44 00:02:15.520 --> 00:02:19.960 my favorite topics, incident response and on call management. And 45 00:02:20.560 --> 00:02:25.800 to chat through that topic with us, we've got Felipe Jane, 46 00:02:25.960 --> 00:02:29.919 the CEO of pager Ley, joining us today. Felipe welcome. 47 00:02:31.560 --> 00:02:31.800 Thanks. 48 00:02:31.800 --> 00:02:35.080 So i'mon guys, happy to join on this and you 49 00:02:35.280 --> 00:02:41.120 have a biggest psion response and ons right on. 50 00:02:41.319 --> 00:02:41.599 Cool. 51 00:02:41.680 --> 00:02:47.280 I feel like incident response is a learned skill, you know, 52 00:02:47.439 --> 00:02:51.800 and it's learned on the job under pressure when everything's 53 00:02:51.840 --> 00:02:56.599 going to hell, And prior to your first incident, you 54 00:02:56.719 --> 00:02:59.680 never even thought through that this is where your life 55 00:02:59.719 --> 00:03:02.400 was going to lead. So how did you end up 56 00:03:03.039 --> 00:03:07.759 starting a company dedicated to this incident response? 57 00:03:09.639 --> 00:03:12.560 It began with like when I started out my first 58 00:03:13.240 --> 00:03:16.639 job basically, and I was part of Amazon and in 59 00:03:16.680 --> 00:03:19.360 the dtail page TAM, which is friendly one of the 60 00:03:19.479 --> 00:03:25.520 highest sort of pages in terms of traffic difficulsider, And 61 00:03:26.879 --> 00:03:29.199 that was like the first hand experience. I like, my 62 00:03:29.199 --> 00:03:32.520 manager put me on oncoholic within sort of months of joining, 63 00:03:33.000 --> 00:03:34.719 and he told me like, hey, this is like the 64 00:03:34.719 --> 00:03:38.199 best way to learn about things, And I kidnam it. 65 00:03:38.319 --> 00:03:42.120 That was some serious pressure, say in. 66 00:03:42.120 --> 00:03:48.000 The initial initial stuff and I and since then I've 67 00:03:48.039 --> 00:03:49.919 always knie, yeah, this is the best way to sort 68 00:03:49.960 --> 00:03:54.599 of learn things because you are absolutely learning each and 69 00:03:54.759 --> 00:03:55.960 every bit of things. 70 00:03:55.719 --> 00:03:57.599 In the shortest possible amount of time. 71 00:03:58.120 --> 00:04:01.599 So I think that that was my first I would 72 00:04:01.599 --> 00:04:04.439 say interaction with the incidents and the on cord world. 73 00:04:05.159 --> 00:04:09.039 Well, let's be realistic there. Your manager's thought process was, actually, 74 00:04:09.560 --> 00:04:11.319 if this guy's going to quit, I'm going to make 75 00:04:11.400 --> 00:04:13.719 him quit sooner rather than later, So I'm putting him 76 00:04:13.759 --> 00:04:14.199 on call. 77 00:04:14.319 --> 00:04:18.040 Right at the beginning, it could be. 78 00:04:18.000 --> 00:04:20.240 I think it was probably you know, just titting. 79 00:04:20.759 --> 00:04:24.600 This is how you get waters And yeah, it was 80 00:04:24.680 --> 00:04:27.439 pretty pretty sort of a new world. Before that, I 81 00:04:27.519 --> 00:04:31.480 always thought like, yeah, software development is more about building 82 00:04:31.480 --> 00:04:36.000 stuff and you know, maybe designing it. This part is 83 00:04:36.240 --> 00:04:40.000 truly what you see a management or managing your maintaining 84 00:04:40.040 --> 00:04:40.879 your product. 85 00:04:41.240 --> 00:04:44.399 And that was like the first kind of interaction I had. 86 00:04:47.199 --> 00:04:50.519 Right and cool. So then you went from Amazon. After 87 00:04:50.639 --> 00:04:53.839 that you went to Disney, right yeah. 88 00:04:54.000 --> 00:04:56.959 Yeah, so in Amazon it was pretty interesting. 89 00:04:57.279 --> 00:05:00.240 The dal page and like there are they It was 90 00:05:00.279 --> 00:05:03.600 sort of quite a few days, especially during Fine Days 91 00:05:03.600 --> 00:05:06.600 and cyber Mondays and the Christmas week is always like 92 00:05:06.839 --> 00:05:11.160 pretty high pressure stuff and I remember each and every 93 00:05:11.279 --> 00:05:14.360 one of them, like in terms of the events, like 94 00:05:14.519 --> 00:05:17.800 even if there's like a small blade, there's like I 95 00:05:17.839 --> 00:05:22.519 think so many teams on a single bridge in even 96 00:05:22.560 --> 00:05:25.160 in different locations, and everyone just giving their status up 97 00:05:25.240 --> 00:05:28.360 dates each and after time and it was like it 98 00:05:28.480 --> 00:05:30.399 was almost like i'd say, like a bar room. 99 00:05:30.879 --> 00:05:31.000 Uh. 100 00:05:31.240 --> 00:05:35.319 People I think have now started to tagg in on 101 00:05:35.439 --> 00:05:38.680 called rooms as barrooms. Like everyone givings the startus updates 102 00:05:38.720 --> 00:05:40.959 and see update and go. 103 00:05:40.959 --> 00:05:44.759 On for quite a few nights. So that was I 104 00:05:44.759 --> 00:05:47.800 think like pretty interesting in the Disney. 105 00:05:47.920 --> 00:05:51.120 It was on the other way around, like our major 106 00:05:51.160 --> 00:05:55.360 events in Disney for the live stream. So in India 107 00:05:55.399 --> 00:05:58.879 we had a cricket as the major school and we 108 00:05:59.000 --> 00:06:02.040 used to live stream cricket and we had around twenty 109 00:06:02.160 --> 00:06:03.920 million concurent. 110 00:06:03.680 --> 00:06:04.839 Viewers also at some point. 111 00:06:05.759 --> 00:06:10.040 And with that scale, each and every bit of system 112 00:06:10.279 --> 00:06:14.839 you know, from starting from where to CDN even to 113 00:06:14.879 --> 00:06:19.040 a load balancer, even to a small humanities file to 114 00:06:19.079 --> 00:06:22.959 even to maybe our cashing systems, everything gets tested a lot. 115 00:06:23.680 --> 00:06:28.720 So for us in Disney, uh, that was our major priority. 116 00:06:28.959 --> 00:06:33.120 How to you know, manage our on calls response instead 117 00:06:33.160 --> 00:06:37.160 of responding on calls during the live streams because we 118 00:06:37.199 --> 00:06:40.639 cannot even afford to go for a minute down because 119 00:06:40.680 --> 00:06:43.240 we know how much takes the life events. 120 00:06:42.839 --> 00:06:44.439 Are for the company. 121 00:06:44.839 --> 00:06:48.759 Well, especially when you're talking about like live streaming cricket 122 00:06:48.800 --> 00:06:51.279 to Indians, because y'all take that seriously. 123 00:06:52.120 --> 00:06:55.720 Yeah, yeah, yeah, we you know the you know, the 124 00:06:55.759 --> 00:06:59.319 broadcast contracts are for you know, billions of dollars and 125 00:06:59.639 --> 00:07:02.120 let's say we go for Doubt for a couple of minutes. 126 00:07:02.319 --> 00:07:04.920 We're really using losing money at every point in time, 127 00:07:05.519 --> 00:07:07.600 and we can see the Twitter. You know, we're just 128 00:07:07.639 --> 00:07:09.639 praending on trat real like your app is down on 129 00:07:09.759 --> 00:07:12.600 what's happening and what's not. So you need to be 130 00:07:12.759 --> 00:07:17.959 very very careful you know, how to respond publicly also, 131 00:07:18.040 --> 00:07:20.920 and how to you know, quickly bring things up in 132 00:07:20.959 --> 00:07:23.680 a way that it can last at least during the live. 133 00:07:24.839 --> 00:07:29.240 So so those were like I'd say, the most you know, 134 00:07:29.360 --> 00:07:33.279 the closest I can be a customer at that point 135 00:07:33.319 --> 00:07:37.040 of time, and the most engineer can be at the 136 00:07:37.160 --> 00:07:42.759 highest pressure point. So yeah, so that was my Disney 137 00:07:42.800 --> 00:07:47.079 stint and those that I've started out with patiently. 138 00:07:47.680 --> 00:07:50.680 Right. Yeah, So I think after that you're kind of 139 00:07:50.800 --> 00:07:55.279 committed at this point, you yeah, you're just your career 140 00:07:55.319 --> 00:07:59.600 path is now incident response after those two stands. 141 00:07:59.800 --> 00:08:00.600 Yeah yeah, yeah. 142 00:08:00.959 --> 00:08:06.199 So interestingly, I think, uh, the two companies had slightly 143 00:08:06.240 --> 00:08:10.720 a different way of handling response, maybe because. 144 00:08:10.480 --> 00:08:13.399 Of the company size or team sizes. 145 00:08:13.920 --> 00:08:19.079 But overall, I think that the concept way like everyone 146 00:08:19.160 --> 00:08:22.000 was leading to a single sort of a role where 147 00:08:22.040 --> 00:08:26.600 we wanted to reduce the same incidents again and at 148 00:08:26.680 --> 00:08:27.879 least that that's. 149 00:08:27.720 --> 00:08:28.839 Our primary rules. 150 00:08:29.279 --> 00:08:32.840 And I see at what I saw in the two things, 151 00:08:32.879 --> 00:08:36.639 like the primary part of the incident response is the process, 152 00:08:36.759 --> 00:08:38.519 Like how do you sort of you know, set up 153 00:08:38.559 --> 00:08:43.000 the processes, how do you enable your engineers to you know, 154 00:08:43.120 --> 00:08:44.399 follow these processes? 155 00:08:44.799 --> 00:08:45.240 Uh? 156 00:08:45.320 --> 00:08:48.759 And and assistant I think like as an engineer, as 157 00:08:48.759 --> 00:08:53.639 a developer, ah, that's why we call on call as 158 00:08:53.679 --> 00:08:57.039 an operational part. Nobody wants to you know, spend a 159 00:08:57.080 --> 00:08:59.759 lot of time on it. Like everyone wants to maybe code, 160 00:09:00.159 --> 00:09:05.399 develop features, maybe even design, architect even blog nowadays, but 161 00:09:05.879 --> 00:09:08.679 on call is the last part, Like everyone wants to 162 00:09:08.720 --> 00:09:11.720 spend time, and so most grinted work, especially if they're 163 00:09:11.799 --> 00:09:16.039 like you know, work after work related to burgs or incidents. 164 00:09:16.440 --> 00:09:20.919 So that's where I saw a lot of common patterns 165 00:09:21.840 --> 00:09:24.799 across you know, incident response, even. 166 00:09:24.879 --> 00:09:25.799 On call management. 167 00:09:26.200 --> 00:09:29.320 UH is like like we there can be a lot 168 00:09:29.360 --> 00:09:33.759 of tools or automations or even. 169 00:09:33.600 --> 00:09:36.000 Assistant agents which can help the engeneers. 170 00:09:36.120 --> 00:09:39.919 So that's why I kind of you know started started 171 00:09:39.919 --> 00:09:42.799 with ag which is helping the teams to assist the 172 00:09:42.840 --> 00:09:44.200 incident and management. 173 00:09:44.559 --> 00:09:47.360 Yeah, I think that's a solid point that's often overlooked. 174 00:09:47.840 --> 00:09:51.279 I work a lot with early stage startups, and it's 175 00:09:51.320 --> 00:09:54.360 a pattern I've seen over my career, like the biggest 176 00:09:54.440 --> 00:09:59.000 part of incident response happens before you ever have your 177 00:09:59.039 --> 00:10:01.679 first incident, but because you have to talk through like 178 00:10:01.879 --> 00:10:03.919 what are we going to do when this actually happens, 179 00:10:03.960 --> 00:10:06.639 who are we gonna bring on, how are we going 180 00:10:06.720 --> 00:10:11.159 to carry out communications? And so Yeah, I think that's 181 00:10:11.159 --> 00:10:13.080 a good solid point. I like the fact that you 182 00:10:13.159 --> 00:10:16.519 mentioned that there's multiple ways of doing that. You know, 183 00:10:16.559 --> 00:10:18.080 there's not one right way. 184 00:10:18.559 --> 00:10:22.039 Right right, So there, I would say, like the first 185 00:10:22.039 --> 00:10:26.080 part is like you need to sort of realize, yeah, 186 00:10:26.320 --> 00:10:29.159 the time has come in my organization that we need 187 00:10:29.200 --> 00:10:33.120 to have this set up. I what I've usually seen 188 00:10:33.159 --> 00:10:37.240 it all ninety percent times it comes down from the 189 00:10:37.320 --> 00:10:41.279 top leadership with if you have like cetos or epes 190 00:10:41.480 --> 00:10:45.240 depending on the organization size, If if those books have 191 00:10:45.320 --> 00:10:50.200 come from a place where incident response or oncal processes had. 192 00:10:50.039 --> 00:10:52.559 Been in place, they bring that culture into. 193 00:10:52.360 --> 00:10:55.080 The company because they they have realized value over the 194 00:10:55.120 --> 00:10:58.320 time of these processes in companies where they have not 195 00:10:58.559 --> 00:11:02.559 like usually or they take a longer time to realize, yeah, 196 00:11:02.600 --> 00:11:06.120 we need such processes. So that's the first part is 197 00:11:06.159 --> 00:11:09.879 to realize like, hey, these are the processes we need 198 00:11:09.960 --> 00:11:12.679 so that we can at least, you know, radiuce are 199 00:11:12.799 --> 00:11:15.159 on our issues in a longer. 200 00:11:14.879 --> 00:11:15.519 Frame of time. 201 00:11:16.200 --> 00:11:18.720 So that's the first part. The second part is them 202 00:11:18.799 --> 00:11:21.759 to set up that on called roster. So on the 203 00:11:21.919 --> 00:11:25.480 rosters is something like uh, now that is something that 204 00:11:25.519 --> 00:11:29.279 which is very dependent on our too work. Some emanations 205 00:11:29.399 --> 00:11:33.120 wants to have a centralized on call team which kind 206 00:11:33.159 --> 00:11:36.840 of handles everything like let's say, even if it's like 207 00:11:36.879 --> 00:11:39.000 a re issue, they do it, even if it's like 208 00:11:39.039 --> 00:11:43.279 an infra situation, they handle it. And people have you know, 209 00:11:43.320 --> 00:11:45.879 different ways of setting up maybe like one one person 210 00:11:46.480 --> 00:11:49.480 from each team or just one person every week, and 211 00:11:49.600 --> 00:11:54.440 they do some some omnations have theredd on called team 212 00:11:54.519 --> 00:11:58.559 for each of their separate teams, so uh, and they 213 00:11:58.679 --> 00:12:01.000 kind of rotated weekly by I becieve monthly man. 214 00:12:01.480 --> 00:12:02.519 So that's the other part. 215 00:12:02.639 --> 00:12:05.480 The second part is to set the oncle on roster 216 00:12:06.080 --> 00:12:11.080 and I think then pergnission takes some time until the 217 00:12:11.360 --> 00:12:13.519 oncle rosters get setted and. 218 00:12:13.440 --> 00:12:15.960 They start you know, debugging tickets, and. 219 00:12:16.039 --> 00:12:18.559 After some certain amount of time then they go into 220 00:12:18.600 --> 00:12:22.559 the setting up the incident response part, which is the 221 00:12:22.639 --> 00:12:25.600 post mortems as well as you know, figuring out Hey, like, 222 00:12:25.720 --> 00:12:29.919 these are the sort of our general kind of you know, 223 00:12:29.960 --> 00:12:32.399 steps we take to solve an issue. These are the 224 00:12:32.559 --> 00:12:36.320 certain workflows that we do. Now let's try to streamline 225 00:12:36.320 --> 00:12:38.879 this both in the response as well as in the 226 00:12:38.960 --> 00:12:41.679 post mortem process as on the post modern analysis. 227 00:12:42.080 --> 00:12:44.279 So that's how that's what. 228 00:12:44.120 --> 00:12:49.320 We have seen orgnisation go from step A to step 229 00:12:49.679 --> 00:12:51.399 to last parties of post models. 230 00:12:51.600 --> 00:12:55.039 You said something really interesting I think, which is I 231 00:12:55.039 --> 00:12:57.519 haven't worked at any company so far, and even my 232 00:12:57.639 --> 00:13:01.879 own authors here we have process. It isn't like we 233 00:13:01.919 --> 00:13:04.279 don't do anything when that happens. Maybe it's because we're 234 00:13:04.320 --> 00:13:06.639 a tech focused or a software focused company, and that's 235 00:13:06.639 --> 00:13:10.039 pretty much where I've worked. But the part that you 236 00:13:10.039 --> 00:13:12.080 said that was really interesting for me is that software 237 00:13:12.080 --> 00:13:15.879 engineers don't like on call and you know, I have 238 00:13:17.200 --> 00:13:19.200 I want to challenge that or like, you know, I want. 239 00:13:19.080 --> 00:13:20.559 To live in the world where it's not a problem. 240 00:13:20.600 --> 00:13:22.720 It's like, why why do people not like it so much, 241 00:13:22.720 --> 00:13:23.799 any thoughts about that. 242 00:13:24.240 --> 00:13:28.519 Because people as engineers or developers, they don't consider as 243 00:13:28.559 --> 00:13:32.320 part of the building process. We love to build, we 244 00:13:32.440 --> 00:13:35.480 love to you know, architect things, but once we have 245 00:13:35.559 --> 00:13:37.600 done that, then we don want to sort of you know, 246 00:13:37.720 --> 00:13:41.039 going and you know, fix out just one tiny part 247 00:13:41.080 --> 00:13:43.440 of it which is actually causing the major issue, but 248 00:13:43.679 --> 00:13:47.120 going there fixing part of it and probably you know, 249 00:13:47.279 --> 00:13:50.679 just taking a blame or so people kind of have 250 00:13:50.840 --> 00:13:54.440 that kind of bias. Also, Hey, my my product is 251 00:13:54.480 --> 00:13:55.360 like a bug three. 252 00:13:55.559 --> 00:13:58.399 My product is you know, super gid so so. 253 00:13:58.840 --> 00:14:01.600 And going there fixing this as well as you know, 254 00:14:01.679 --> 00:14:04.639 you already have a lot of other work going on 255 00:14:04.759 --> 00:14:08.559 the strains in this agile world. So that's where people 256 00:14:08.799 --> 00:14:11.120 don't defini sort of difference, don't want to spend a 257 00:14:11.120 --> 00:14:11.879 lot of time on it. 258 00:14:12.399 --> 00:14:15.799 So that's what you think. 259 00:14:15.799 --> 00:14:18.840 It's not like well prioritized or rewarded. Like if you 260 00:14:18.879 --> 00:14:20.879 do on call work, you're not rewarded for it. If 261 00:14:20.919 --> 00:14:23.360 you write buglass code, you're not rewarded for it. So 262 00:14:23.480 --> 00:14:26.559 you know, whatever I don't want to do it, it's 263 00:14:26.559 --> 00:14:28.120 going to happen, and then I have to pay the 264 00:14:28.279 --> 00:14:29.919 I have to pay the fine because of it, and 265 00:14:30.000 --> 00:14:31.240 I don't I don't get the benefit. 266 00:14:32.440 --> 00:14:37.279 Yeah, I think like benefits and all, like cans probably 267 00:14:37.279 --> 00:14:40.559 be sort of be defined by the engineting managers or 268 00:14:40.600 --> 00:14:42.919 team leads if they want to sort of reward or 269 00:14:42.960 --> 00:14:45.799 they want to highlight maybe you know, like if the 270 00:14:45.840 --> 00:14:48.759 person has solved these many decads and these incidents, or 271 00:14:49.120 --> 00:14:54.279 maybe find a better way of you know, rewarding rewarding uh, 272 00:14:54.600 --> 00:14:57.320 developers who actually solve a lot of incidents. 273 00:14:58.080 --> 00:14:59.120 But yeah, I think like in. 274 00:14:59.159 --> 00:15:02.000 General sense, like it's not part of the building, it's 275 00:15:02.159 --> 00:15:05.320 only maintain, but that's the major. 276 00:15:08.000 --> 00:15:09.919 Like, no, no, I totally got it. 277 00:15:10.399 --> 00:15:12.720 Yeah, I don't like on call because it's never my code, 278 00:15:12.759 --> 00:15:13.879 it's always something else. 279 00:15:14.159 --> 00:15:18.039 Yeah, but I mean that makes me think there's something Yeah, no, 280 00:15:18.080 --> 00:15:19.600 I totally get I mean I feel like there's something 281 00:15:19.600 --> 00:15:20.919 fundamentally broken there. 282 00:15:20.960 --> 00:15:23.679 Like I've seen that where I worked at one. 283 00:15:23.600 --> 00:15:27.039 Of the previous previous jobs, that was twenty fifty engineers 284 00:15:27.080 --> 00:15:30.519 that were all rotating through all the same on call schedule, 285 00:15:30.960 --> 00:15:33.720 as if somehow code just because it was all in 286 00:15:33.720 --> 00:15:37.039 the monolith, if if something was broken, I somehow would 287 00:15:37.039 --> 00:15:39.320 magically know what was going on and say I don't 288 00:15:39.360 --> 00:15:42.919 know products or logistics code when I had nothing to 289 00:15:42.960 --> 00:15:45.440 do with the development of it, Like I like it 290 00:15:45.519 --> 00:15:47.600 might as well be like some for it, like I 291 00:15:47.639 --> 00:15:51.120 don't know Aramaic or you know, uniform to me, like 292 00:15:51.159 --> 00:15:53.759 I have no idea what that was going on there 293 00:15:53.840 --> 00:15:56.240 at all, and yet somehow I have to come in 294 00:15:56.320 --> 00:15:59.080 and debug or find out what the problem. 295 00:15:58.879 --> 00:16:02.279 Was, right, Yeah, I think in Amazon this was a 296 00:16:02.360 --> 00:16:06.480 case like the entire I the de deal page was 297 00:16:07.000 --> 00:16:10.720 kind of at that point of time like that, and 298 00:16:10.240 --> 00:16:14.240 and that particular sort of service the page is like 299 00:16:14.320 --> 00:16:18.679 maintained by more than one fifty software engineers. So like 300 00:16:19.000 --> 00:16:21.080 most of the times you are debugging something that you 301 00:16:21.120 --> 00:16:24.320 have not ruled, so and you're evening, it's a three 302 00:16:24.480 --> 00:16:27.799 in the night, and you're already frustrated, like you don't 303 00:16:27.799 --> 00:16:31.000 know what's happening. So that's why I think some bit 304 00:16:31.039 --> 00:16:33.919 of uscision do come from. And that's why I think 305 00:16:34.000 --> 00:16:37.519 like good processes can sort of somehow mitigate some of 306 00:16:37.600 --> 00:16:41.600 the pain points, especially like institutions. 307 00:16:41.039 --> 00:16:45.399 Like was the expectation at three am that software engineers 308 00:16:45.399 --> 00:16:47.679 should be able to log on and identify the problem 309 00:16:47.720 --> 00:16:50.399 and push out a fix like that seems like there's 310 00:16:50.440 --> 00:16:54.080 something that would never actually have like actually worked out 311 00:16:54.120 --> 00:16:55.679 in practice. 312 00:16:55.200 --> 00:16:58.360 It did like something. So like it kind of depends 313 00:16:58.440 --> 00:17:01.440 so what kind ofs that you have in place. So 314 00:17:01.600 --> 00:17:04.440 one past one part is to maybe mitigate. Sometimes medication 315 00:17:04.519 --> 00:17:07.079 can be done just to give a revered last running. 316 00:17:07.279 --> 00:17:10.359 That's one medication people don't have. But that's even that 317 00:17:10.519 --> 00:17:14.400 is not done. You probably need to sort of page 318 00:17:14.759 --> 00:17:17.880 the person who has probably added that line of code 319 00:17:17.960 --> 00:17:20.839 and take hit from him or her, or you maybe 320 00:17:20.920 --> 00:17:23.000 have certain like a time of team or something like 321 00:17:23.079 --> 00:17:26.160 that which can sort of you know, uh orches straight 322 00:17:26.240 --> 00:17:29.880 and collaborate with a lot of different on calls for 323 00:17:30.000 --> 00:17:32.279 different developers. 324 00:17:31.839 --> 00:17:34.559 To sort of mitigate and put up put a patch 325 00:17:34.640 --> 00:17:35.359 or fix the issues. 326 00:17:35.759 --> 00:17:39.079 Right, So this kind of like depends on you know, 327 00:17:39.200 --> 00:17:43.240 how you have set up that inst response forrocess itself. 328 00:17:43.519 --> 00:17:46.759 Yeah, I think that's a really good distinction to make 329 00:17:46.839 --> 00:17:52.279 there that like during an incident, oftentimes the primary goal 330 00:17:52.519 --> 00:17:56.440 is to mitigate the problem, which is different than solving 331 00:17:56.559 --> 00:17:59.799 the root cause of the problem. So like if if 332 00:17:59.799 --> 00:18:03.000 we're run an outage or an incident, like we might 333 00:18:03.160 --> 00:18:09.680 mitigate the problem by launching you know, fifteen more Kubernetes 334 00:18:09.720 --> 00:18:13.400 pods with just insane amounts of memory, just so we 335 00:18:13.480 --> 00:18:16.000 can ride through the problem too, we're able to figure 336 00:18:16.039 --> 00:18:18.960 out the root cause and test that theory and then 337 00:18:19.000 --> 00:18:20.039 deploy a fix for it. 338 00:18:20.480 --> 00:18:23.000 So just so I got you right, Well, your strategy 339 00:18:23.119 --> 00:18:25.519 for incident management is turning it off and back on. 340 00:18:25.599 --> 00:18:29.640 Again absolutely three times. Always reboot three times. 341 00:18:31.559 --> 00:18:34.000 Yeah, I think like legging ice stream. 342 00:18:34.720 --> 00:18:38.480 You know, one of the major kind of our last 343 00:18:38.559 --> 00:18:41.160 heart was to just put the live stream on and 344 00:18:41.720 --> 00:18:44.480 don't maybe don't have a pay on or something like that. 345 00:18:44.680 --> 00:18:46.920 So that can be one even in. 346 00:18:48.839 --> 00:18:51.920 Different scenarios that you can sit like, okay, even if 347 00:18:52.000 --> 00:18:53.160 fixing is stacking time even. 348 00:18:53.039 --> 00:18:58.279 Immedi maybe do something. Maybe just put that I took there. 349 00:18:58.839 --> 00:19:02.519 There was a certain uh forcessiarity had in peace, so 350 00:19:02.839 --> 00:19:06.160 I just said, like, just increase the community spots and 351 00:19:06.799 --> 00:19:09.680 at least that your customer is not facing that issue 352 00:19:09.680 --> 00:19:12.799 for the movement, then they use use that time to 353 00:19:13.759 --> 00:19:14.880 to actually fix the issue. 354 00:19:15.200 --> 00:19:18.640 How do you decide what mitigation strategy makes the most sense, Like, 355 00:19:19.000 --> 00:19:20.960 if you like, I feel like we're in the case 356 00:19:21.000 --> 00:19:23.279 of the world now where we're going to automate whatever 357 00:19:23.400 --> 00:19:26.279 it is. So if we have some number of failures, 358 00:19:26.359 --> 00:19:28.799 Do we just immediately start deploying extra pods? Do we 359 00:19:28.960 --> 00:19:32.160 immediately try to roll back to a previous code version? 360 00:19:32.240 --> 00:19:35.240 Like can we even know upfront what the right approach 361 00:19:35.400 --> 00:19:37.200 is there to automate? Because the last thing I want, 362 00:19:37.240 --> 00:19:39.519 I feel like, is someone to get online and after 363 00:19:39.680 --> 00:19:41.799 half an hour be like, you know what, maybe we 364 00:19:41.920 --> 00:19:45.160 should deploy some pods with an insane amount of memory 365 00:19:45.200 --> 00:19:46.480 that will solve all of our problems. 366 00:19:47.359 --> 00:19:50.240 Yeah, yeah, it's interesting. 367 00:19:50.319 --> 00:19:55.319 It was like like as an engineer, like even if 368 00:19:55.400 --> 00:19:59.319 you have some similarity, you would know this is the issue. 369 00:19:59.519 --> 00:20:02.079 Let's see, if you're seeing them mentally, then part increase 370 00:20:02.359 --> 00:20:03.000 makes a sense. 371 00:20:03.440 --> 00:20:07.119 But let's say if there's a something else you're seeing 372 00:20:07.119 --> 00:20:10.319 a lot of five texis, probably the part increase might 373 00:20:10.359 --> 00:20:11.000 not make a sense. 374 00:20:11.000 --> 00:20:14.400 They're probably reverting to a previous version might make a sense. 375 00:20:15.079 --> 00:20:18.799 So I think you need to have some bit of humanity, 376 00:20:18.960 --> 00:20:21.680 you know, with whatever system that you are hiding it 377 00:20:22.279 --> 00:20:23.759 in case you're not, then it becomes. 378 00:20:23.559 --> 00:20:27.079 A huge challenges. Then you probably don't know you know 379 00:20:27.200 --> 00:20:31.279 what to do. So I think, like I think some 380 00:20:31.839 --> 00:20:35.440 hulmarities need at least to have that first modication. 381 00:20:35.559 --> 00:20:41.799 Still, I think that alludes to an entire skill set 382 00:20:42.000 --> 00:20:44.480 in software engineering of how to troubleshoot. 383 00:20:45.039 --> 00:20:45.160 You know. 384 00:20:45.240 --> 00:20:48.200 It's because like debugging when you're writing code is completely 385 00:20:48.319 --> 00:20:53.839 different I think than troubleshooting a live system and all 386 00:20:53.880 --> 00:20:56.839 of its different dependencies and trying to figure out where 387 00:20:56.920 --> 00:20:59.599 the potential problem might be and how do you how 388 00:20:59.640 --> 00:21:02.519 do you get some faith in that theory and then 389 00:21:03.319 --> 00:21:04.440 do something to mitigate it. 390 00:21:04.799 --> 00:21:05.000 Yeah. 391 00:21:05.200 --> 00:21:07.599 Yeah, I think like a couple of techniques that we 392 00:21:07.799 --> 00:21:10.079 have seen and we had to kind of deployed. One 393 00:21:10.240 --> 00:21:14.119 is maybe do like a shadow on care like you 394 00:21:14.279 --> 00:21:17.000 can you do on calls, like someone is handing the 395 00:21:17.079 --> 00:21:19.279 on care, but let's say if they have major incidents, 396 00:21:19.720 --> 00:21:22.759 you shadow them. So you know, these are tools like 397 00:21:22.920 --> 00:21:25.839 even even I've seen sometimes people don't even know like 398 00:21:26.000 --> 00:21:28.240 these are dashboards that you can refer to and which 399 00:21:28.240 --> 00:21:31.720 will probably help you more so, uh, reaching out to 400 00:21:32.079 --> 00:21:35.920 people shadowing them definitely hips, especially when whenever there's an 401 00:21:36.000 --> 00:21:38.640