WEBVTT 1 00:00:07.839 --> 00:00:10.000 So, Warren, what's been your experience with SRI. 2 00:00:10.720 --> 00:00:14.359 That's quite the way of just jumping into the episode, 3 00:00:14.400 --> 00:00:17.239 and I was so unprepared because I'm not even the 4 00:00:17.600 --> 00:00:20.160 guest for today's episode, So I don't I don't have 5 00:00:20.199 --> 00:00:21.399 an answer to that question. 6 00:00:21.199 --> 00:00:25.120 Honestly, Yeah, for sure, Jillian, And what about your thoughts 7 00:00:25.120 --> 00:00:27.280 on SRA. You're just like, you're a contract not just 8 00:00:27.320 --> 00:00:29.679 a contractor. You're a contractor. You just light stuff on 9 00:00:29.760 --> 00:00:31.960 fire and then if they want to put out you 10 00:00:32.000 --> 00:00:33.079 bill them for it, right. 11 00:00:33.479 --> 00:00:35.079 I mean really like, I just need to make sure 12 00:00:35.119 --> 00:00:38.320 data doesn't get deleted because that's like a problem that 13 00:00:38.399 --> 00:00:41.079 is that is a problem, Okay, And that's really all 14 00:00:41.079 --> 00:00:41.880 that I worry about. 15 00:00:42.920 --> 00:00:45.759 It's a security reliability because you can't be exploited if 16 00:00:45.759 --> 00:00:47.240 there's no data left. 17 00:00:47.759 --> 00:00:49.119 I mean, that's true. 18 00:00:49.679 --> 00:00:51.320 I will say that there is. There is a book 19 00:00:51.359 --> 00:00:55.479 out there published by one of the hyperscalers. I believe 20 00:00:56.560 --> 00:00:59.240 that attempts to define it in their own terms, but 21 00:00:59.320 --> 00:01:01.280 I feel like, like a lot of things they do, 22 00:01:01.479 --> 00:01:06.040 it's used as a introduction into the topic rather than 23 00:01:06.920 --> 00:01:09.879 used as experts. So I'm really interested to hear what 24 00:01:10.000 --> 00:01:13.680 our guests today will be sharing with us. 25 00:01:14.079 --> 00:01:17.239 Well, yeah, which is a great segue because clearly we 26 00:01:17.319 --> 00:01:23.040 need some expert opinions on SRE, and so we've got 27 00:01:23.200 --> 00:01:25.599 Amy Night with us. Amy, Welcome to the show. 28 00:01:26.640 --> 00:01:27.439 Happy to be here. 29 00:01:27.959 --> 00:01:30.879 I'm excited to have you here. So tell us a 30 00:01:30.920 --> 00:01:36.319 little bit about your background and how you got to SRE. 31 00:01:36.920 --> 00:01:37.159 Yeah. 32 00:01:37.280 --> 00:01:39.040 I love SRI. 33 00:01:40.319 --> 00:01:43.359 It's a little bit of a love hate relationship. It 34 00:01:43.400 --> 00:01:46.640 can be very stressful at times, but I also love it. 35 00:01:46.719 --> 00:01:50.480 I feel like it is kind of when I when 36 00:01:50.480 --> 00:01:52.519 I and this, I'll get into kind of like my 37 00:01:52.599 --> 00:01:56.840 path into SRE, and it just it really suits. I 38 00:01:56.879 --> 00:02:00.920 feel like my engineering brain perfectly and the high level 39 00:02:00.920 --> 00:02:05.599 there is. I come from non traditional background, so self taught, 40 00:02:05.719 --> 00:02:09.280 did the whole boot camp thing twelve years ago now, 41 00:02:09.439 --> 00:02:11.080 which is insane to me. 42 00:02:11.360 --> 00:02:14.639 I'm really old, or feel really old. 43 00:02:16.439 --> 00:02:19.120 Like I saw someone on Twitter the other day talk 44 00:02:19.120 --> 00:02:21.680 about like, what did you use before GET and I 45 00:02:21.800 --> 00:02:24.080 was like, oh, my gosh, I'm really old now because 46 00:02:24.080 --> 00:02:26.280 my first job we use material, which I guess is 47 00:02:26.360 --> 00:02:29.280 it but at the same time like it. 48 00:02:29.759 --> 00:02:30.039 Yeah. 49 00:02:30.199 --> 00:02:36.759 So But anyways, so backing up, and I used to 50 00:02:36.800 --> 00:02:39.840 do JavaScript jobber, so I kind of started off in 51 00:02:39.840 --> 00:02:43.879 the JavaScript Land, did node. 52 00:02:43.680 --> 00:02:46.439 And all the front end. 53 00:02:47.000 --> 00:02:48.639 I kind of focused heavily on front end for a 54 00:02:48.680 --> 00:02:53.759 little while, did note as well, and then I think 55 00:02:53.800 --> 00:02:58.800 it was like twenty eighteen twenty nineteen, I went to 56 00:02:58.960 --> 00:03:01.360 work at NPM, which was like my dream job at 57 00:03:01.360 --> 00:03:04.680 the time until they got acquired, and you know, it's 58 00:03:04.800 --> 00:03:06.319 just a little bit of a bummer because it was 59 00:03:06.400 --> 00:03:09.639 very short lived. But when I was there, we did 60 00:03:09.719 --> 00:03:13.360 cross functional teams and my manager at the time was like, 61 00:03:13.599 --> 00:03:17.280 you seem really interested in like this platform s re 62 00:03:17.439 --> 00:03:23.639 work infrastructure. We can probably hire for a web developer 63 00:03:24.039 --> 00:03:25.680 more easily than we can hire for that role. Do 64 00:03:25.680 --> 00:03:27.039 you want to go over to that team? And I 65 00:03:27.120 --> 00:03:28.879 was like, yes, sign me out, like you want to, 66 00:03:29.479 --> 00:03:32.319 you know, if I have the opportunity to learn a 67 00:03:32.360 --> 00:03:37.360 new skill on the job, Like yes. So I did 68 00:03:37.479 --> 00:03:41.680 SRE at MPM for a while, and then I did 69 00:03:41.879 --> 00:03:45.280 most of my sr rework at Paramount Global, which is 70 00:03:46.280 --> 00:03:51.520 like the streaming application, and did two Super Bowls there well, 71 00:03:52.759 --> 00:03:56.960 the last last year's Super Bowl. And so that's kind 72 00:03:56.960 --> 00:03:59.639 of my journey into SRE And why I say it 73 00:03:59.719 --> 00:04:02.919 is like I feel like SRI is really like my 74 00:04:02.919 --> 00:04:07.479 my perfect niche in engineering and that is because coming 75 00:04:07.479 --> 00:04:10.759 from this non traditional background, I've always been the like 76 00:04:11.159 --> 00:04:15.120 the programmer who is like, sure that new tech is 77 00:04:15.159 --> 00:04:17.720 really fun and interesting, but like, shouldn't that be something 78 00:04:17.720 --> 00:04:20.439 we're working on, like on the weekends on our own time, 79 00:04:20.560 --> 00:04:24.040 Like isn't our job? Like I hate saying this because 80 00:04:24.079 --> 00:04:27.000 people hate me, but at the end of the day, 81 00:04:27.199 --> 00:04:30.160 like I'm just like I feel very fortunate to have 82 00:04:30.199 --> 00:04:32.439 a career in tech, and in the back of my mind, 83 00:04:32.480 --> 00:04:35.439 I'm always like my job, whether I like it or not, 84 00:04:36.199 --> 00:04:39.000 Like if I'm working hours, my job is to make 85 00:04:39.040 --> 00:04:44.079 this company profitable. And while I love programming for fun 86 00:04:44.160 --> 00:04:47.160 on my own, like, I have a responsibility to the 87 00:04:47.199 --> 00:04:50.720 business to make make the business profitable, keep it alive. 88 00:04:50.720 --> 00:04:52.759 Work doing a lot of startups, so I. 89 00:04:52.720 --> 00:04:56.519 Understand this, and that's where SRI really fits in because 90 00:04:56.560 --> 00:04:59.240 it's like a lot of roles. Yes, you're accountable to 91 00:04:59.279 --> 00:05:01.360 the business, but I feel like in an SR role 92 00:05:02.480 --> 00:05:05.879 through we can get into like slas, slis SLOs, but 93 00:05:05.920 --> 00:05:10.519 the slas like you are accountable to the business stakeholders 94 00:05:10.560 --> 00:05:15.160 and that is your overarching goal. So basically, in a nutshell, 95 00:05:15.399 --> 00:05:19.360 SA site reliability engineering. I also call it like production 96 00:05:19.519 --> 00:05:24.360 readiness engineering. It is your task to keep the application 97 00:05:24.879 --> 00:05:30.759 running optimally for as as much up time as possible. 98 00:05:31.439 --> 00:05:34.839 So what I heard was that you at MPM had 99 00:05:35.600 --> 00:05:37.600 so much work and we're doing such a great job 100 00:05:37.600 --> 00:05:41.839 that they're like, here's more work. Be responsible for the 101 00:05:41.879 --> 00:05:45.480 success of the business and not just solving tickets and 102 00:05:45.560 --> 00:05:46.759 pushing out futures. 103 00:05:47.920 --> 00:05:51.399 No, but honestly, truly, like my manager there, he was 104 00:05:51.560 --> 00:05:54.399 absolutely amazing, as were a lot of the people that 105 00:05:54.439 --> 00:06:00.399 I worked with, absolutely brilliant, talented people. And I don't know, 106 00:06:00.519 --> 00:06:04.519 I'm like distributed systems nerd. Like on the weekends, I 107 00:06:04.680 --> 00:06:08.000 like to watch YouTube videos of distributed systems interviews, Like 108 00:06:08.040 --> 00:06:10.920 I don't know why, this is very interesting to me, 109 00:06:12.600 --> 00:06:16.000 but I've always like just gravitated towards that sort of thing. 110 00:06:16.199 --> 00:06:18.600 So you don't have to shame yourself for that. I 111 00:06:18.600 --> 00:06:22.000 think lots of us on the weekend, Yeah, we're watching 112 00:06:22.079 --> 00:06:24.639 interviews and other tech things. Yeah, for sure. 113 00:06:25.120 --> 00:06:27.240 Whenever there's like a big outage of like a big 114 00:06:27.279 --> 00:06:29.920 tech company, I'm always like reading the post mortems and 115 00:06:29.959 --> 00:06:31.800 trying to talk to the like talk about them with 116 00:06:31.879 --> 00:06:35.519 my husband, and he was he's just like he tries 117 00:06:35.560 --> 00:06:39.040 to act interested, but he's just good for He's like, 118 00:06:39.120 --> 00:06:42.120 I love how nerdy you are, Like, okay, I'll just 119 00:06:42.240 --> 00:06:43.560 talk to myself that. 120 00:06:45.160 --> 00:06:48.160 You've got a really I think that's a super interesting 121 00:06:48.279 --> 00:06:51.959 point where you talked about you were working for working 122 00:06:52.000 --> 00:06:55.439 out on JavaScript development and then you went over to SRE, 123 00:06:55.680 --> 00:06:59.319 And I think that's really important skill to have for 124 00:06:59.399 --> 00:07:03.199 that because, like as the SRE, like it's not just 125 00:07:03.279 --> 00:07:07.480 about the infrastructure or the data dog metrics or whatever, 126 00:07:07.519 --> 00:07:11.000 like a lot of times you've got to dig into 127 00:07:11.040 --> 00:07:15.680 the code and discover like what's actually going on here? 128 00:07:16.360 --> 00:07:20.519 And so how how do you feel like those web 129 00:07:20.519 --> 00:07:23.240 development skills set you up for success as an SRE. 130 00:07:23.680 --> 00:07:27.000 Yeah, that's a really good question, and I it's a 131 00:07:27.040 --> 00:07:29.839 common question I get from people who are also interested 132 00:07:29.879 --> 00:07:34.319 in SRI because and I also feel like, you know, 133 00:07:35.519 --> 00:07:40.360 backing up a little bit SRE. I know, Warren mentioned 134 00:07:40.399 --> 00:07:42.360 like the Google sr book, but I also feel like 135 00:07:42.399 --> 00:07:46.560 this focus on SRI really came about via micro services buzzword. 136 00:07:46.639 --> 00:07:48.079 I hate saying them, but. 137 00:07:48.040 --> 00:07:48.680 It is what it is. 138 00:07:51.360 --> 00:07:57.639 But so to kind of answer your question there, I 139 00:07:57.720 --> 00:08:01.959 feel like the teams that I've been on not necessarily MPM, 140 00:08:02.079 --> 00:08:07.000 but paramount specifically, they had a lot of like systems engineers. 141 00:08:07.040 --> 00:08:09.439 Like the team was very system engineer heavy, so a 142 00:08:09.439 --> 00:08:11.519 lot of these people who like worked in data centers 143 00:08:12.160 --> 00:08:15.040 and those are all obviously very valuable skills as an SR, 144 00:08:16.480 --> 00:08:20.720 but where they were really lacking was like the programming 145 00:08:20.759 --> 00:08:23.680 side of things application development side of things. And I 146 00:08:23.720 --> 00:08:25.959 feel like to have a really good SRA team, you 147 00:08:26.000 --> 00:08:28.879 do need a mixture of both and you can really 148 00:08:28.920 --> 00:08:32.679 like learn from each other. So in the SR, like 149 00:08:32.759 --> 00:08:35.159 in SR roles that I've been in, I am doing 150 00:08:35.200 --> 00:08:39.960 some programming nuts. I mean, you need to be able 151 00:08:39.960 --> 00:08:42.679 to debug the program. You're also doing some programming in 152 00:08:43.600 --> 00:08:48.240 you're doing like scripting. A lot of SR, I don't 153 00:08:48.440 --> 00:08:51.159 think people realize is a lot of like cashing at 154 00:08:51.159 --> 00:08:53.879 the edge and so you're doing a lot of custom 155 00:08:53.960 --> 00:08:57.600 logic that gets deployed to your CDNs and things like that. 156 00:08:58.320 --> 00:09:01.519 So it might not be as intensive as like building 157 00:09:01.559 --> 00:09:03.600 out a future and stuff like that, you still need 158 00:09:03.639 --> 00:09:07.240 those programming concepts to write that kind of code and 159 00:09:07.240 --> 00:09:11.279 then also to work with the developers because you know, 160 00:09:11.360 --> 00:09:13.559 for better or worse, there is like that's like the 161 00:09:14.519 --> 00:09:16.840 that's the telltale like thing that people say about SRI 162 00:09:17.039 --> 00:09:19.639 is like that push and pull between the engineering department 163 00:09:19.679 --> 00:09:21.679 because they have a vested interest in getting their stuff 164 00:09:21.720 --> 00:09:24.799 out to production and then SRI has a vested interest 165 00:09:25.000 --> 00:09:28.000 in keeping the system healthy and functioning. 166 00:09:28.879 --> 00:09:31.399 Yeah for sure. I mean you did mention that it 167 00:09:31.679 --> 00:09:34.159 could be different in different places. So like from your experience, 168 00:09:34.200 --> 00:09:37.440 what has being an SRI really met? Like is there 169 00:09:37.480 --> 00:09:39.799 a standard day? And I know that's a terrible question 170 00:09:39.879 --> 00:09:42.600 for asking you know, any sort of engineering technical person, 171 00:09:43.120 --> 00:09:45.519 but you know some some expectations around what the role 172 00:09:45.600 --> 00:09:48.120 is that you were doing both at MPM and Paramount. 173 00:09:48.639 --> 00:09:48.960 Yeah. 174 00:09:49.000 --> 00:09:52.039 So MPM obviously like a very it was a relatively 175 00:09:52.080 --> 00:09:56.759 small team. We just we had the SR team and 176 00:09:57.159 --> 00:09:58.759 there was a lot of like I said, it was 177 00:09:58.759 --> 00:10:01.799 cross functional, so we're with building out features, but a 178 00:10:01.879 --> 00:10:04.240 team was like a cross functional team where we had 179 00:10:04.240 --> 00:10:08.720 a platform engineer, we had SR infrastructure engineer, we had 180 00:10:08.840 --> 00:10:13.720 a application developer, and all working together to get something 181 00:10:13.799 --> 00:10:16.000 out to production. And like the SR side of things, 182 00:10:16.039 --> 00:10:19.639 there would be like I was saying, I talk about 183 00:10:19.759 --> 00:10:23.120 SR in terms of like production readiness, so making sure 184 00:10:23.200 --> 00:10:27.559 that this feature is ready for production, if there are 185 00:10:27.639 --> 00:10:30.799 like dashboards that need to be made, making sure that 186 00:10:30.840 --> 00:10:33.200 those are in place so that we can properly monitor 187 00:10:33.559 --> 00:10:37.960 when it goes to production, making sure that you have 188 00:10:38.240 --> 00:10:41.080 the proper resources for the machines that it's going to 189 00:10:41.080 --> 00:10:46.840 be running on, things like that. At Paramount, you know, 190 00:10:46.879 --> 00:10:50.240 this is a much larger MPM, has a very large scale, 191 00:10:50.240 --> 00:10:55.679 but it was extremely bursty. And when I say like 192 00:10:55.759 --> 00:11:00.279 extremely bursty, like the traffic would spike, but it would 193 00:11:00.279 --> 00:11:06.039 be very short lived. At Paramount, similar but not quite 194 00:11:06.039 --> 00:11:09.559 to that extent like birsty, as in like you're going 195 00:11:09.600 --> 00:11:11.799 to see a high rush of traffic when a large 196 00:11:11.799 --> 00:11:17.039 event kicks off, It'll stay relatively high, but it kind 197 00:11:17.080 --> 00:11:19.039 of like starts to taper off as the events go 198 00:11:19.159 --> 00:11:23.559 down and things like that. But Paramount SRA like a 199 00:11:24.480 --> 00:11:30.159 larger SR teams, it's usually segmented into different departments. So 200 00:11:30.240 --> 00:11:33.960 you have like an infrastructure focused department, you would have 201 00:11:34.000 --> 00:11:36.519 like a devsec ops because there's a lot of security 202 00:11:36.519 --> 00:11:39.279 that goes into SRE, and you would. 203 00:11:39.039 --> 00:11:41.080 Have like an observability monitoring team. 204 00:11:41.120 --> 00:11:44.879 I feel like observability and monitoring is where you hear 205 00:11:44.919 --> 00:11:48.200 a lot of like focus on SRE for sure, but 206 00:11:48.279 --> 00:11:51.159 it is definitely like segmented into different teams, and it 207 00:11:51.240 --> 00:11:53.399 is very different at different places, so. 208 00:11:53.360 --> 00:11:54.759 You spend all your time looking at graphs. 209 00:11:54.799 --> 00:11:58.360 That's what I when I was more on the infrastructure 210 00:11:58.399 --> 00:12:00.759 side of things. But yes, like we were heavily with 211 00:12:02.519 --> 00:12:05.840 like observability team to make sure that the dashboards are 212 00:12:05.879 --> 00:12:08.120 in place and they have the data that you need 213 00:12:08.159 --> 00:12:09.519 and everything sod that properly. 214 00:12:09.960 --> 00:12:13.240 That sounds like the go between, like when get the 215 00:12:13.240 --> 00:12:16.759 application metrics into some place that different team owns, and 216 00:12:16.799 --> 00:12:19.519 then when that team exposes a problem, you're then have 217 00:12:19.600 --> 00:12:21.919 to ferry that back to the application team to tell 218 00:12:21.960 --> 00:12:24.360 them about the issue that they caused in production. 219 00:12:25.879 --> 00:12:27.039 Yeah, and that's. 220 00:12:26.840 --> 00:12:30.799 Where like what Forrest was saying, it's complicated what. 221 00:12:32.679 --> 00:12:33.080 So? Uh? 222 00:12:33.200 --> 00:12:35.440 Will and I had a little joke going on before 223 00:12:35.519 --> 00:12:38.559 the stream started, we forgot to end the joke, and 224 00:12:39.120 --> 00:12:42.200 the joke the joke basically ended where I decided to 225 00:12:42.240 --> 00:12:46.039 change his moniker in the in the stream that uh 226 00:12:46.279 --> 00:12:49.799 that that amy only only you can stream only you 227 00:12:49.799 --> 00:12:51.679 can see no one else, like none of the none, 228 00:12:51.919 --> 00:12:54.360 none of the audience will be able to see that. Will. 229 00:12:54.440 --> 00:12:56.360 It's Will, I'm so sorry. 230 00:12:56.600 --> 00:13:02.360 No, it's hilarious. It's great because well played Warren, well played. 231 00:13:04.600 --> 00:13:05.799 You look like a forest. 232 00:13:05.879 --> 00:13:08.159 But well, that is how the joke started, is because 233 00:13:08.440 --> 00:13:10.399 like Warren is asking me something and I was like, dude, 234 00:13:10.399 --> 00:13:13.000 I don't know. I just feel like Forrest Gump because 235 00:13:13.039 --> 00:13:15.919 I just wander around places in my life and great 236 00:13:15.919 --> 00:13:17.759 shit happens. And then people were like, how did you 237 00:13:17.799 --> 00:13:19.320 do that? And like I don't know. I was just 238 00:13:19.360 --> 00:13:20.159 standing there. 239 00:13:21.519 --> 00:13:26.320 Okay, Will, Will, I'm so sorry. I feel you got 240 00:13:26.360 --> 00:13:29.720 me good, thank you. And now I've lost my train 241 00:13:29.759 --> 00:13:30.279 of thought. 242 00:13:30.120 --> 00:13:30.639 A little bit. 243 00:13:30.759 --> 00:13:34.399 But too, I think what I was about to say 244 00:13:34.639 --> 00:13:38.360 is like there's a fine line because obviously it doesn't 245 00:13:38.399 --> 00:13:41.399 make sense for the SRI to be the person that 246 00:13:42.080 --> 00:13:47.200 debugs the code, but in a perfect scenario, ultimately, like 247 00:13:47.279 --> 00:13:51.679 the sort of thing that probably should happen is SRI is. 248 00:13:51.759 --> 00:13:55.440 Usually sometimes you have like a knock team who's kind 249 00:13:55.440 --> 00:13:57.159 of like the first line of defense that does like 250 00:13:57.399 --> 00:13:59.720 a smoke test, like, okay, we got an alert, we 251 00:13:59.759 --> 00:14:00.519 got paged? 252 00:14:00.879 --> 00:14:03.200 Is the application? Can I reach it? 253 00:14:03.679 --> 00:14:03.759 Like? 254 00:14:03.799 --> 00:14:05.919 Can I just go to my browser and reach it? 255 00:14:06.000 --> 00:14:06.840 Like is it up? 256 00:14:07.320 --> 00:14:08.519 Is it hard down? 257 00:14:08.840 --> 00:14:10.320 Or is it is latency? 258 00:14:10.679 --> 00:14:13.039 You know, starting to go down that sort of thing, 259 00:14:13.080 --> 00:14:15.320 And the knock team is usually the first line of 260 00:14:15.320 --> 00:14:19.480 defense there. If the KNOP team deems it like okay, 261 00:14:19.600 --> 00:14:22.480 something is awry here, then they'll usually. 262 00:14:22.360 --> 00:14:23.399 Page SRI team. 263 00:14:24.240 --> 00:14:27.240 So SRI team kind of usually you'll have like playbooks 264 00:14:27.240 --> 00:14:29.240 in place to do like a sanity check of like 265 00:14:29.559 --> 00:14:32.080 jump on one of the Kopernettes pods, you know, kind 266 00:14:32.080 --> 00:14:36.559 of see what's going on? Are these pods airing something 267 00:14:36.600 --> 00:14:39.519 like that. But ultimately what should happen is there should 268 00:14:39.840 --> 00:14:42.639 in my opinion anyways like a close relationship with SRI 269 00:14:43.360 --> 00:14:48.000 and the development team that it's clear like we have 270 00:14:48.080 --> 00:14:51.240 an environment in this micro services system where we can 271 00:14:51.279 --> 00:14:56.000 do a safe roll back, and so what typically would 272 00:14:56.039 --> 00:14:59.639 happen at least in my opinion, my role like I 273 00:14:59.679 --> 00:15:03.759 would deem like okay, and I have a rough understanding 274 00:15:03.799 --> 00:15:06.200 of like what the feature is, what's the impact because 275 00:15:06.240 --> 00:15:10.080 there could be a business impact of rolling back. Ultimately 276 00:15:10.120 --> 00:15:13.840 we want to make that would when I say business impact, 277 00:15:13.960 --> 00:15:18.039 like you have some sort of like feature for Black 278 00:15:18.039 --> 00:15:22.080 Friday and it's peak traffic and this feature is like 279 00:15:22.759 --> 00:15:25.759 driving revenue, so there's like a but it's also like 280 00:15:25.840 --> 00:15:27.960 causing issues for customers, So you need to kind of 281 00:15:27.960 --> 00:15:29.799 make the decision. And this is where the stress gets 282 00:15:29.840 --> 00:15:34.440 high of like what's the financial trade off ultimately here 283 00:15:34.600 --> 00:15:37.639 like to this business person, like is it honestly like 284 00:15:37.679 --> 00:15:39.320 it will bubble all the way up like through the 285 00:15:39.440 --> 00:15:43.320 organization to like the CEO of the company, and you 286 00:15:43.399 --> 00:15:45.799 need to make the decision of like what's the what's 287 00:15:45.840 --> 00:15:49.360 the lesser of the financial impacts here? But long winded 288 00:15:49.399 --> 00:15:54.120 answer like sorre should have confidence from the engineering team 289 00:15:54.200 --> 00:15:55.919 that it's safe to roll back and do like an 290 00:15:55.960 --> 00:16:00.240 instant roll back. And it's also then helpful as an 291 00:16:00.320 --> 00:16:03.759 sary if like, let's say, a lot of times you 292 00:16:03.799 --> 00:16:07.200 will get paged in the middle of the night, and 293 00:16:07.919 --> 00:16:12.720 it's a large organization and sometimes sometimes you have great 294 00:16:12.759 --> 00:16:15.679 engineers on your team who you can escalate too, and 295 00:16:15.720 --> 00:16:19.080 they usually have like a paging rotation also, but a 296 00:16:19.080 --> 00:16:21.679 lot of times I'm not gonna lie, I was on 297 00:16:21.720 --> 00:16:24.679 an SR team and it's the middle of the night 298 00:16:24.759 --> 00:16:27.279 and I page, you know, I'm looking at this pod 299 00:16:27.320 --> 00:16:31.559 that's crashing and I can see that it's like actually 300 00:16:31.600 --> 00:16:34.960 like an error in the code, and I page the 301 00:16:35.000 --> 00:16:36.120 engineering team and. 302 00:16:37.840 --> 00:16:41.200 What it crickets. 303 00:16:40.200 --> 00:16:43.919 Just like a group project, Like it's the same Actually 304 00:16:44.039 --> 00:16:46.840 I have page again in twenty minutes and crickets. 305 00:16:46.879 --> 00:16:50.120 So I've been in a state where like I am 306 00:16:50.200 --> 00:16:52.720 just rolling pods all night. When I say rolling pods, 307 00:16:52.720 --> 00:16:56.200 like restarting pods over and over and over again until 308 00:16:56.240 --> 00:16:57.559 someone wakes up in the morning. 309 00:16:58.480 --> 00:17:01.519 Wait, so you're when you say knock like network operations center? 310 00:17:01.600 --> 00:17:01.799 Is that? 311 00:17:02.360 --> 00:17:05.720 Is that mostly to manage the physical hardware. 312 00:17:07.000 --> 00:17:09.559 It's hard for me to say I didn't work. They 313 00:17:09.559 --> 00:17:12.039 were kind of like our first line of defense at 314 00:17:12.039 --> 00:17:14.839 some of my jobs, and it works super closely with them. 315 00:17:15.240 --> 00:17:16.160 But yeah, those are. 316 00:17:16.119 --> 00:17:19.519 Typically the people that were like the hard like they 317 00:17:19.519 --> 00:17:22.920 were working in the data center, and you know, the 318 00:17:23.039 --> 00:17:26.119 organization obviously sees like value and kind of transitioning them 319 00:17:26.119 --> 00:17:27.519 to different roles and things like that. 320 00:17:27.640 --> 00:17:31.119 So yeah, so you then have to look at a 321 00:17:31.119 --> 00:17:34.680 lot of application specific metrics, so more on the software 322 00:17:34.720 --> 00:17:39.119 validation side, and then be that that ferry that bridge 323 00:17:39.160 --> 00:17:41.119 back from Okay, there's a problem at the hardware level 324 00:17:41.160 --> 00:17:43.799 that's getting escalated and having to find the application team 325 00:17:43.839 --> 00:17:46.160 and of course they have no on call rotation ever, 326 00:17:46.319 --> 00:17:49.880 because they're not accountable for keeping the systems up and 327 00:17:49.960 --> 00:17:50.720 running right. 328 00:17:50.640 --> 00:17:52.119 They're just to shift code. 329 00:17:53.240 --> 00:17:55.319 Yeah, just ship it and hand it over you know, 330 00:17:55.880 --> 00:17:59.640 but that's where you know close relationship also, and like 331 00:17:59.640 --> 00:18:01.599 I say saying like I keep going back to this 332 00:18:01.680 --> 00:18:06.920 like production readiness in it should be that before the 333 00:18:06.960 --> 00:18:10.440 application team hands over something like if saying like, work 334 00:18:10.480 --> 00:18:12.400 with the s RE team in advance to make sure 335 00:18:13.559 --> 00:18:17.720 that if it's a mission critical service in your micro 336 00:18:17.799 --> 00:18:21.200 service architecture, like that the dashboards are in place, you 337 00:18:21.279 --> 00:18:27.160 understand the dependencies like a when I say dependencies too, 338 00:18:27.200 --> 00:18:28.759 like a lot of things that come up in SRE. 339 00:18:29.759 --> 00:18:32.160 And that's why I like this is just I say, 340 00:18:32.319 --> 00:18:37.640 nerd out on distributed system stuff. You can see one 341 00:18:37.799 --> 00:18:44.079 application starting to show some latency, but the actual impact 342 00:18:44.839 --> 00:18:50.920 is more so like a service downstream, and there are 343 00:18:51.039 --> 00:18:56.000 different ways of like understanding obviously, like if this part 344 00:18:56.079 --> 00:18:59.839 of the system is having this percentage of latency, then 345 00:19:00.039 --> 00:19:04.519 I need to understand that these downstream services are getting 346 00:19:04.640 --> 00:19:07.559 hit by this other service X number of times it's 347 00:19:07.599 --> 00:19:10.799 going to start like having this kind of latency. So 348 00:19:10.880 --> 00:19:13.519 you really have to understand like the entire system and 349 00:19:13.640 --> 00:19:17.400 understand those dependencies. That's where it helps, Like that's where 350 00:19:17.480 --> 00:19:21.559 s RE really comes in because when like these application 351 00:19:21.599 --> 00:19:25.440 teams are kind of like siloed working on their specific service. 352 00:19:25.960 --> 00:19:29.799 The SR team is more responsible for understanding like the 353 00:19:29.880 --> 00:19:33.359 trace of a request through the system and understanding those 354 00:19:33.359 --> 00:19:34.440 downstream effects. 355 00:19:34.599 --> 00:19:37.039 Well, I mean, you just got into some really deep 356 00:19:37.079 --> 00:19:39.960 topics there, Like there's a huge complexity with like queuing 357 00:19:40.480 --> 00:19:44.079 theory and window based you know, back offs there as 358 00:19:44.119 --> 00:19:47.880 well as like the system effect like actual systems, thinking 359 00:19:48.000 --> 00:19:51.200 of how one thing triggers another thing, which triggers another thing. 360 00:19:51.240 --> 00:19:54.000 And I don't remember ever seeing a boot camp that 361 00:19:54.119 --> 00:19:57.319 would have gone anyone prepared for handling this. And you 362 00:19:57.440 --> 00:19:59.680 got through this, and now I have I understand you 363 00:19:59.759 --> 00:20:02.559 were still like being on the spot to actually make 364 00:20:02.640 --> 00:20:05.440 a business impactful decision for the organization. 365 00:20:06.079 --> 00:20:11.160 Yes, that's where it gets very stressful. And that's where 366 00:20:11.279 --> 00:20:13.440 I just I feel like in this kind of role, 367 00:20:13.720 --> 00:20:19.680 you really need someone who cares and yes, because otherwise 368 00:20:20.799 --> 00:20:23.799 the company's revenue is on the line, whether that be 369 00:20:24.839 --> 00:20:26.640 not even like I was saying an example of like 370 00:20:26.640 --> 00:20:28.720 the Black Friday issue, where like you have to make 371 00:20:28.759 --> 00:20:35.039 the call of is this shopping cart feature more important 372 00:20:35.079 --> 00:20:38.680 than the fact that X number of other users can't 373 00:20:38.720 --> 00:20:43.519 do this other thing that they need to do, versus 374 00:20:43.680 --> 00:20:47.279 even understanding like the slas and is there like a 375 00:20:47.319 --> 00:20:50.680 financial obligation that I need to pay my customers if 376 00:20:51.519 --> 00:20:53.119 X number of things start to go down? 377 00:20:53.519 --> 00:20:55.240 So did you have like some sort of dashboard that 378 00:20:55.279 --> 00:20:57.519 listed out every single feature and like how much it 379 00:20:57.559 --> 00:20:59.400 was supposed to make for the company, and like how 380 00:20:59.519 --> 00:21:02.279 much the shopping I mean, I feel like NPM and 381 00:21:02.279 --> 00:21:04.279 paramoun are a little bit weird because it's not really 382 00:21:04.319 --> 00:21:06.880 like end user shopping car scenarios. I mean, I don't 383 00:21:06.880 --> 00:21:10.640 really understand paramounts, you know, end user pricing model. I 384 00:21:10.680 --> 00:21:13.240 know there's like plus right, but other than that, So 385 00:21:13.279 --> 00:21:15.759 I feel like it's it's even harder to really understand 386 00:21:15.799 --> 00:21:18.039 what the long term impact is for each one of 387 00:21:18.039 --> 00:21:19.039 those things that's breaking. 388 00:21:19.319 --> 00:21:21.839 You know, it's very different obviously at different places, Like 389 00:21:23.039 --> 00:21:24.960 so I keep going back to the shopping cart example. 390 00:21:25.039 --> 00:21:28.799 Like I worked at different startups in like the retail industry, 391 00:21:28.799 --> 00:21:31.160 and although I wasn't on the SR team there specifically 392 00:21:31.480 --> 00:21:34.599 as more in like an architectural role, so like kind 393 00:21:34.640 --> 00:21:38.160 of tasked with understanding that I'm working with the demops team. 394 00:21:38.200 --> 00:21:43.319 But yeah, like in a perfect world, you would have 395 00:21:43.400 --> 00:21:47.200 dashboard self you make those decisions, but you don't have 396 00:21:47.319 --> 00:21:50.799 that so you just you have to make the best 397 00:21:50.839 --> 00:21:54.960 call based on your understanding and you know, escalate to 398 00:21:55.000 --> 00:21:59.039 the right people if you can. And that's where I 399 00:21:59.039 --> 00:22:03.119 should really get into, you know, is maybe like shift 400 00:22:03.119 --> 00:22:07.640 the conversation too to like strategies besides just like reactive strategies, 401 00:22:08.160 --> 00:22:10.559 because we've talked about like a lot of like reactive 402 00:22:10.759 --> 00:22:15.160 approaches to things. But part of what SRI is tasked 403 00:22:15.200 --> 00:22:18.039 with is like how do we make this system more 404 00:22:18.079 --> 00:22:20.559 resilient by nature of things we can do? 405 00:22:21.200 --> 00:22:23.160 And I feel like you're in the perfect position to 406 00:22:23.519 --> 00:22:26.440 think about and sort of ide eight on that. But 407 00:22:26.559 --> 00:22:28.240 don't you end up with like a lot of pushback 408 00:22:28.319 --> 00:22:30.559 because that's sort of like a shift left approach in 409 00:22:30.559 --> 00:22:34.160 the organization with like oh, you know application product teams, 410 00:22:34.160 --> 00:22:36.200 you know this thing that you're doing that has like 411 00:22:36.279 --> 00:22:38.960 a high likelyhood that there's going to be a production impact, 412 00:22:39.000 --> 00:22:40.720 maybe you should do something different. How do you get 413 00:22:40.720 --> 00:22:43.759 that actually across those teams so they can like start 414 00:22:43.799 --> 00:22:46.519 making those actual changes because it's like a mindset or 415 00:22:46.599 --> 00:22:47.599 culture shift for them. 416 00:22:48.119 --> 00:22:51.000 Kind Of where I'm going with this is more so 417 00:22:51.240 --> 00:22:54.440 on like the redundancy side of things, like SRI team 418 00:22:54.480 --> 00:22:57.079 making the decision and it's you have to work with 419 00:22:57.200 --> 00:23:02.440 like the engineering management leadership to understand and like, you know, 420 00:23:02.480 --> 00:23:05.480 if we're running in like a Kupernettese environment making that 421 00:23:05.839 --> 00:23:09.160 do we either do we do multi cloud? Do we 422 00:23:09.400 --> 00:23:14.480 obviously we do multi region those sorts of decisions, And 423 00:23:14.519 --> 00:23:18.359 then that's where I get into I don't think a 424 00:23:18.400 --> 00:23:21.599 lot of people necessarily realize, like I was saying, how 425 00:23:21.720 --> 00:23:27.359 valuable like content delivery is in this because just by 426 00:23:27.440 --> 00:23:30.039 nature of having as much as you can cashed and 427 00:23:30.079 --> 00:23:33.880 distributed outside of like before the request even gets to 428 00:23:34.640 --> 00:23:37.599 whether it's like your physical data center or or a cloud, 429 00:23:38.599 --> 00:23:42.720 how can we like offload some of that traffic? Because 430 00:23:42.759 --> 00:23:45.200 I feel like a lot of these systems are just 431 00:23:45.559 --> 00:23:49.359 the fallover without that. And I know we mentioned it 432 00:23:49.400 --> 00:23:51.079 like at the beginning of the episode, and maybe I'll 433 00:23:51.119 --> 00:23:54.119 like touch on it more at the end. But one 434 00:23:54.160 --> 00:23:56.440 thing that's been very interesting for me to like sit 435 00:23:56.480 --> 00:23:58.319 back and watch is like Jenny I is. 436 00:24:00.160 --> 00:24:03.519 Saying that word it is, it. 437 00:24:03.359 --> 00:24:06.160 Is what it is like twofold. 438 00:24:07.279 --> 00:24:11.200 First off, like my heart goes out to s on 439 00:24:11.279 --> 00:24:14.880 their sres who work in environments where they have like 440 00:24:15.000 --> 00:24:18.160 developers that are just what this whole vibe coding thing, Like, 441 00:24:19.400 --> 00:24:23.920 I don't think people realize how much like these lms 442 00:24:23.960 --> 00:24:28.400 are hallucinating and producing like content that is just garbage, 443 00:24:28.440 --> 00:24:31.200 absolute garbage soup. 444 00:24:31.240 --> 00:24:34.039 Poor one out for the SR supporting vibe coding is 445 00:24:34.079 --> 00:24:34.680 what you're saying. 446 00:24:35.200 --> 00:24:38.559 Yeah, we did have some previous guests on the show 447 00:24:38.640 --> 00:24:41.799 who were like swearing to us that their lms didn't 448 00:24:41.839 --> 00:24:42.759 ever hallucinate. 449 00:24:43.720 --> 00:24:46.440 I literally was like, I think I won't say the 450 00:24:46.599 --> 00:24:48.400 LM because I'm going to get myself in trouble. But 451 00:24:48.680 --> 00:24:51.119 I was like asking, like going back and forth with 452 00:24:51.160 --> 00:24:54.200 one of them earlier this week, like writing stuff, and 453 00:24:54.680 --> 00:24:56.880 I asked it to revise it, and it started like 454 00:24:57.000 --> 00:25:01.279 putting all these like statistics in, and I was like, I. 455 00:25:01.200 --> 00:25:06.160 Was using a product that allowed me to. 456 00:25:07.160 --> 00:25:11.480 Incorporate a lot of basically like a rag guy, so 457 00:25:11.519 --> 00:25:14.039 I could incorporate like a bunch of external sources. And 458 00:25:14.079 --> 00:25:17.039 I was like, there's no way those statistics are in 459 00:25:17.119 --> 00:25:21.519 those sources, like because I know the data that I added. 460 00:25:22.000 --> 00:25:24.680 And I asked it, like where did you get these statistics? 461 00:25:24.680 --> 00:25:26.599 And it's like, you're right, I made them up. I'm sorry. 462 00:25:29.680 --> 00:25:32.119 At least you've got a conclusive answer there, because We're 463 00:25:32.119 --> 00:25:35.440 seeing this a lot more with seoed articles today, where 464 00:25:35.519 --> 00:25:37.599 not only are they being generated by llms, but the 465 00:25:38.000 --> 00:25:41.720 links that they go to do not back up whatever 466 00:25:41.960 --> 00:25:43.559 is in the article, and like so even if a 467 00:25:43.640 --> 00:25:46.160 human is writing it, it's just some fluff piece. And 468 00:25:46.200 --> 00:25:48.559 if you go there, it's like, oh, sixty percent of 469 00:25:48.599 --> 00:25:51.640 attacks are from this particular source, like email is the 470 00:25:51.680 --> 00:25:53.839 source of all attacks and security. And you go and 471 00:25:53.839 --> 00:25:56.000 you click on the link and it's just some completely 472 00:25:56.039 --> 00:25:58.640 made up garbage that's also not researched by some other 473 00:25:58.640 --> 00:26:01.799 company that nothing to do with the topic whatsoever. So 474 00:26:01.880 --> 00:26:06.000 can you blame the probabilistic you know, statistical engine that's 475 00:26:06.079 --> 00:26:08.119 just predicting the next word that is also going to 476 00:26:08.200 --> 00:26:10.119 pull out links that have nothing to do with the 477 00:26:10.119 --> 00:26:10.960 topic whatsoever. 478 00:26:11.400 --> 00:26:12.160 Yes, yeah, you. 479 00:26:12.119 --> 00:26:14.440 Can't blame it, but it's like it's on us to. 480 00:26:16.319 --> 00:26:19.319 You know, vet it. But what I wanted to touch on that. 481 00:26:19.319 --> 00:26:21.519 Very briefly was just basically, like I read. 482 00:26:21.359 --> 00:26:26.119 Something this morning, like the percentage like there's a direct 483 00:26:26.119 --> 00:26:30.960 correlation between these companies that are allowing like more AI 484 00:26:31.319 --> 00:26:38.319 assistive like developer stuff and outages. So my heart, there's 485 00:26:38.599 --> 00:26:42.000 I'd have to like I think I screenshotted it on LinkedIn, 486 00:26:42.079 --> 00:26:45.200 but there was like it was a lot, and it's 487 00:26:45.279 --> 00:26:46.279 just in the back of my mind. 488 00:26:46.279 --> 00:26:47.359 So there's that side of things. 489 00:26:47.400 --> 00:26:49.680 And then there's also just like these like you know, 490 00:26:49.759 --> 00:26:53.400 open Ai and all these companies like pushing out it's 491 00:26:53.400 --> 00:26:56.519 like a race to get stuff out to production faster, 492 00:26:56.640 --> 00:27:00.599 and there's like that. I think last year just the 493 00:27:00.680 --> 00:27:02.319 number of outages, like every time I would go to 494 00:27:02.400 --> 00:27:05.400 chat tipt I think like almost like a week, I 495 00:27:05.400 --> 00:27:06.519 was like half the time is down. 496 00:27:08.400 --> 00:27:10.279 I think the jury is really out on the numbers. 497 00:27:10.279 --> 00:27:12.480 I think we've concluded two things, and whether or not 498 00:27:12.519 --> 00:27:14.880 people are willing to listen to those is a separate topic. 499 00:27:15.319 --> 00:27:20.200 You are trading quality or speed. That is, that is 500 00:27:20.240 --> 00:27:23.119 the trade off, And so you know, I think there 501 00:27:23.119 --> 00:27:24.920 are a bunch of there are a bunch of good 502 00:27:24.960 --> 00:27:27.920 articles recently by people who work at some of these 503 00:27:29.319 --> 00:27:31.079 You say you don't like the word jen Ai. I 504 00:27:31.160 --> 00:27:36.480 really hate the word agentic. That's that's mine companies that 505 00:27:36.759 --> 00:27:40.599 they themselves are not using their own LM to produce 506 00:27:40.640 --> 00:27:42.960 the code, right, And so you know, I see these 507 00:27:43.200 --> 00:27:45.160 CEOs say oh yeah and stand up and be like, oh, 508 00:27:45.200 --> 00:27:47.519 our company like twenty five percent of all the source 509 00:27:47.519 --> 00:27:49.759 code we write is by you know, some sort of 510 00:27:49.839 --> 00:27:52.319 LM And I'm like, what do you what are you 511 00:27:52.400 --> 00:27:54.200 going to do about that risk? You know you're gonna 512 00:27:54.839 --> 00:27:56.119 decrease that in the future. 513 00:27:56.559 --> 00:27:59.359 Yeah, I should say, like I'm not opposed to these tools, 514 00:27:59.359 --> 00:28:03.359 but they need to be used responsibly and for better 515 00:28:03.440 --> 00:28:06.640 or worse, like humans, Like it's our nature. Like that's 516 00:28:06.640 --> 00:28:09.400 what like the famous like programming quote, like we're lazy, 517 00:28:09.480 --> 00:28:13.039 Like a lazy person makes a good programmer because they 518 00:28:13.039 --> 00:28:14.920 want to automate things. So like there's a lot of 519 00:28:15.000 --> 00:28:19.119 value in these tools, but you hands down need to 520 00:28:19.240 --> 00:28:24.559 use them responsibly and be able to use them, like 521 00:28:24.920 --> 00:28:28.160 with actual knowledge behind it. Like I mentor a lot 522 00:28:28.160 --> 00:28:32.359 of developers and like you know, they're just they're wanting 523 00:28:32.359 --> 00:28:35.400 to get so done. And I'll see like this code 524 00:28:35.400 --> 00:28:37.519 that they wrote, and I'm like that that I know 525 00:28:37.519 --> 00:28:38.359 where you gotta from. 526 00:28:38.519 --> 00:28:40.519 That's not right. 527 00:28:41.920 --> 00:28:45.559 I don't know why it's telling you to like install 528 00:28:46.119 --> 00:28:47.319 Python packages in. 529 00:28:47.359 --> 00:28:49.960 Your own application, but this is not right. 530 00:28:52.359 --> 00:28:54.440 Yeah anyways, Yeah, but. 531 00:28:54.480 --> 00:28:56.480 On a side note, I'm really admired that you got 532 00:28:56.480 --> 00:28:59.720 those Python packages to work inside of no js so. 533 00:29:01.079 --> 00:29:03.240 Well. It just comes to the foreign function interface and 534 00:29:03.279 --> 00:29:05.640 you know, you just shell out to that, no, no 535 00:29:05.720 --> 00:29:08.519 big deal, and your application, you know, just multiple programming language. 536 00:29:08.519 --> 00:29:10.759 You know, what's what's the big deal? It's all it's 537 00:29:10.799 --> 00:29:11.680 all simply under the. 538 00:29:11.680 --> 00:29:14.880 Hood, exactly. It's all one since but doesn't matter. 539 00:29:15.079 --> 00:29:16.599 Right for now, for now. 540 00:29:20.880 --> 00:29:24.599 So I want to come back to talking about playbooks, 541 00:29:24.599 --> 00:29:27.039 because you mentioned those and I got excited. But I 542 00:29:27.039 --> 00:29:32.160 think like an important thing to hang the playbook conversation 543 00:29:32.359 --> 00:29:36.119 on is at talking about being an sr AT scale, 544 00:29:36.440 --> 00:29:40.200 because like being an sr AT a small company versus 545 00:29:40.200 --> 00:29:44.240 being an sr AT an enterprise, I think are quite 546 00:29:44.319 --> 00:29:50.240 possibly two completely different career fields. And and so how 547 00:29:50.279 --> 00:29:55.200 did you address like the scale of of applications you 548 00:29:55.200 --> 00:29:55.920 were dealing with. 549 00:29:56.519 --> 00:30:00.039 That's a good question too. I think it really so 550 00:30:00.119 --> 00:30:02.519 like talking about this like micro service system where a 551 00:30:02.599 --> 00:30:08.839 lot of people usually are at right now, typically everything 552 00:30:08.960 --> 00:30:12.279 is like a trade off, and that's like a handwaybe answer, 553 00:30:12.319 --> 00:30:15.240 but it's true. And so like when we go back 554 00:30:15.240 --> 00:30:18.880 to like the production readiness stuff, I think it's like 555 00:30:18.920 --> 00:30:24.319 helpful to understand like a like a process of like 556 00:30:24.400 --> 00:30:26.559 kind of rating these micro services. Like it's just like 557 00:30:26.599 --> 00:30:29.200 tier one tier two, tier three, Like like an a 558 00:30:29.400 --> 00:30:32.599 system is probably going to be like a tier one system, 559 00:30:32.680 --> 00:30:39.279 whereas if you have like a recommendations engine that's running 560 00:30:39.319 --> 00:30:42.079 things in the background, that's more of like a tier 561 00:30:42.119 --> 00:30:48.920 three system because critical functionality still works. Check out is 562 00:30:48.960 --> 00:30:51.599 going to be obviously a tier one system something like that. 563 00:30:52.359 --> 00:30:55.839 But there's a lot of things to consider, Like I 564 00:30:55.839 --> 00:30:59.640 feel like, especially like for people that are getting into 565 00:30:59.720 --> 00:31:05.200 this kind of role, it's important to understand like the 566 00:31:05.400 --> 00:31:09.799 traffic patterns, Like if you're working in like a small startup, 567 00:31:11.039 --> 00:31:13.519 you're accountable to your customers, like maybe just during like 568 00:31:13.640 --> 00:31:17.759 business hours, like a banking application or something like that 569 00:31:17.920 --> 00:31:20.839 is not going to be as critical as an application 570 00:31:20.920 --> 00:31:28.480 that's used internationally. Things like that, those are like does 571 00:31:28.519 --> 00:31:29.680 that kind of answer your question? 572 00:31:29.839 --> 00:31:32.240 Like things to consider, Yeah, for sure. 573 00:31:33.720 --> 00:31:37.559 And then tying it back to playbooks, like there's a 574 00:31:37.599 --> 00:31:41.000 certain level of scale where you just can't be familiar 575 00:31:41.039 --> 00:31:43.480 with all of these systems and you have to rely 576 00:31:43.680 --> 00:31:47.200 on playbooks. So what are some of the strategies that 577 00:31:47.319 --> 00:31:50.400 you've seen to making sure that you have an accurate, 578 00:31:50.480 --> 00:31:52.119 up to date playbook when you need it. 579 00:31:53.480 --> 00:31:54.759 Yeah, up to date. 580 00:31:56.079 --> 00:31:58.160 Everything is legacy as soon as you write it. Like 581 00:31:58.200 --> 00:32:00.559 as soon as you write something down, it is out 582 00:32:00.559 --> 00:32:03.160 of date. That code is already wrong as soon as 583 00:32:03.160 --> 00:32:03.799 it's released. 584 00:32:04.200 --> 00:32:06.519 That Yeah, So that's I'm going to go back to. 585 00:32:06.839 --> 00:32:13.119 Like building out an S OR team is you need 586 00:32:13.200 --> 00:32:17.880 technical skills obviously, but you really need people who take 587 00:32:18.000 --> 00:32:23.359 ownership and care. Like we're talking about the playbooks. Like 588 00:32:24.000 --> 00:32:27.200 I've definitely been at companies where I will get I'm 589 00:32:27.440 --> 00:32:28.319 brand new. 590 00:32:28.480 --> 00:32:29.279 I'll get Paige. 591 00:32:29.359 --> 00:32:32.160 I go to the playbook and I see that like 592 00:32:32.200 --> 00:32:35.839 the playbooks can get it hasn't been updated in five years, 593 00:32:38.160 --> 00:32:41.200 and I'm like, you know, shoot, crap. 594 00:32:41.519 --> 00:32:43.519 Like what do I do? Like I'm getting Paige. 595 00:32:44.839 --> 00:32:47.039 Usually, like when you're Paige, it will point you to 596 00:32:47.279 --> 00:32:52.079 the playbook and it's not there. So, you know, the 597 00:32:52.400 --> 00:32:56.440 best way that I've seen people tackle this is like ownership. 598 00:32:57.079 --> 00:33:01.839 Someone needs to own these playbooks. Somebody if a if 599 00:33:01.880 --> 00:33:03.759 A system is like a Tier one or a Tier 600 00:33:03.759 --> 00:33:06.480 two system, like this is part of this, like production 601 00:33:06.599 --> 00:33:10.400 readiness checklist. Like before before the SRI is okay with 602 00:33:10.480 --> 00:33:15.640 deploying this and like a lot of times, like company 603 00:33:15.720 --> 00:33:18.279 like the the SR team is they're not really the 604 00:33:18.319 --> 00:33:21.920 gatekeeper because a lot of these companies, like the developers, 605 00:33:21.920 --> 00:33:25.720 have the ability to ship themselves, but there are obviously 606 00:33:25.799 --> 00:33:30.039 consequences if they ship something and it crashes and and 607 00:33:30.200 --> 00:33:35.160 that doesn't completely come that that is like a shared ownership. 608 00:33:35.880 --> 00:33:39.400 But having there needs to be ownership of these playbooks, 609 00:33:39.400 --> 00:33:43.000 and that's usually done by the s RE team. Typically, 610 00:33:44.599 --> 00:33:49.039 you know, you will get the leadership knows when something 611 00:33:49.079 --> 00:33:53.680 is very critical, and good leadership will make sure that 612 00:33:53.799 --> 00:33:57.799 these teams are communicating on like a regular cadence. I'm 613 00:33:57.799 --> 00:34:01.920 big on like these regular cadences, not necessarily because meetings 614 00:34:02.000 --> 00:34:06.000 kind of suck, but they are important for communication. 615 00:34:06.440 --> 00:34:10.360 I like that you put the responsibility of the playbooks 616 00:34:10.360 --> 00:34:12.920 on the s RE team. There's actually a debate that 617 00:34:13.440 --> 00:34:16.199 my CEO and myself have been talking about a bunch 618 00:34:16.239 --> 00:34:19.639 where if it's an artifact that gets thrown over the wall, 619 00:34:19.800 --> 00:34:22.320 then it's not going like you can't it doesn't contain 620 00:34:22.400 --> 00:34:25.760 knowledge in a way it contains They're usually actionable things, 621 00:34:26.000 --> 00:34:28.960 and so since the knowledge is there, you won't know 622 00:34:29.000 --> 00:34:31.880 how to adapt to a playbook when you get to 623 00:34:31.960 --> 00:34:37.119 a new scenario that's not encountered before. Right, And most 624 00:34:37.159 --> 00:34:38.960 of the problems that we find in systems that we 625 00:34:39.000 --> 00:34:41.639 automate are only problems that we haven't seen before. 626 00:34:42.679 --> 00:34:46.199 And I will say too so when I say like 627 00:34:46.440 --> 00:34:50.719 having accountability and ownership, like that's where for better or worse, 628 00:34:50.800 --> 00:34:54.880 everybody hates on call schedules, but those on call schedules 629 00:34:55.039 --> 00:34:58.400 do serve as kind of like a backpropagation of like 630 00:34:58.599 --> 00:35:01.039 if somebody is on if you're on an on call 631 00:35:01.159 --> 00:35:04.039 rotation and you're getting paid frequently, like you have a 632 00:35:04.199 --> 00:35:08.519 vested interest in stopping that because if it gets out 633 00:35:08.519 --> 00:35:13.519 of hand, it's just a huge snowball of you know, obviously, 634 00:35:13.559 --> 00:35:17.519 like you you're starting to like your sleep is interrupted. 635 00:35:17.639 --> 00:35:20.679 So and for better or worse, like people will be like, oh, 636 00:35:20.760 --> 00:35:22.360 you know, you had a rough night, take it easy 637 00:35:22.400 --> 00:35:22.800 during the day. 638 00:35:22.880 --> 00:35:23.679 Yeah right, that's not. 639 00:35:26.119 --> 00:35:28.480 You're still getting up and doing your usual hours, but 640 00:35:28.559 --> 00:35:32.280 you're just like I'm dead. So there's a vested interest 641 00:35:32.360 --> 00:35:35.800 in in like sharing that ownership of the on call rotation. 642 00:35:35.960 --> 00:35:38.440 And if you're getting paid constantly, like you're going to 643 00:35:38.760 --> 00:35:41.840 want to make sure that your teammates have the playbooks 644 00:35:41.840 --> 00:35:44.440 in place. And you're like, I just had a huge 645 00:35:44.480 --> 00:35:47.679 outage and these are the steps that I took to 646 00:35:47.760 --> 00:35:50.320 remediate this these are the things that were helpful. So 647 00:35:50.760 --> 00:35:52.760 after my on call rotation, I'm going to go in 648 00:35:52.840 --> 00:35:55.400 and update the playbook and make sure that the steps 649 00:35:55.440 --> 00:35:57.559 that I took are there so that maybe there's like 650 00:35:57.599 --> 00:35:59.679 more junior members on the team they also know and 651 00:35:59.719 --> 00:36:02.639 things like that, because usually too and like these kind 652 00:36:02.679 --> 00:36:05.880 of environments that you'll have like multiple tiers there, so 653 00:36:05.920 --> 00:36:08.239 there's a knock. But then even within s RE, like 654 00:36:08.280 --> 00:36:11.039 I think it's helpful to have like multiple tiers of 655 00:36:11.119 --> 00:36:14.559 SRY support, so you have like more senior level sries 656 00:36:14.840 --> 00:36:18.920 and then more junior level esteres and like maybe maybe 657 00:36:19.000 --> 00:36:21.159 during the day you would page more junior SRY or 658 00:36:21.239 --> 00:36:23.559 something like that, whereas a senior is around to step 659 00:36:23.559 --> 00:36:25.480 in right away, but you want to have stuff in 660 00:36:25.559 --> 00:36:26.920 place for it makes sense. 661 00:36:27.000 --> 00:36:29.159 What I'm hearing is you're a huge proponent of these 662 00:36:29.159 --> 00:36:33.159 incident management tools that are generating playbooks dynamically on the 663 00:36:33.199 --> 00:36:38.039 fly using some sort of lum please a lot of so. 664 00:36:38.159 --> 00:36:39.119 Much reviewing them. 665 00:36:39.239 --> 00:36:43.119 That's fine by me, but like, yeah, these lms are helpful, 666 00:36:43.159 --> 00:36:46.000 but they are they are not the end all, Like 667 00:36:46.440 --> 00:36:47.320 that's not the end. 668 00:36:47.199 --> 00:36:47.639 Of the road. 669 00:36:48.199 --> 00:36:52.159 You just gave him my next startup idea, Warren, I 670 00:36:52.159 --> 00:36:54.039 mean you have to compete with some of our past guests, 671 00:36:54.119 --> 00:36:58.360 I think, who are already already promising h dynamic run 672 00:36:58.400 --> 00:37:00.519 books based off I mean, there's a thing I think 673 00:37:00.519 --> 00:37:02.679 Sentry does, and I think there's a couple other tools 674 00:37:02.679 --> 00:37:06.079 now that they have your source code or at least 675 00:37:06.320 --> 00:37:07.960 you know, and then they have the stack trays from 676 00:37:07.960 --> 00:37:09.880 the error and the actual error message, and they use 677 00:37:09.960 --> 00:37:12.840 that to automatically propose a pull request into your get 678 00:37:12.840 --> 00:37:15.400 repository that supposedly fixes the problem so that someone can 679 00:37:15.440 --> 00:37:16.280 review and improve it. 680 00:37:16.719 --> 00:37:19.079 And you've got to know that at three am, you 681 00:37:19.119 --> 00:37:20.519 are just going to click improve. 682 00:37:23.679 --> 00:37:25.760 Yeah, yeah, that's yeah. 683 00:37:25.840 --> 00:37:27.880 All those tools, Like I said, I think they're helpful, 684 00:37:27.960 --> 00:37:30.400 but they need to be used responsibly and with somebody 685 00:37:30.400 --> 00:37:33.639 who is actually like a human being with background that 686 00:37:33.760 --> 00:37:34.639 says yeah your name. 687 00:37:34.960 --> 00:37:38.519 Yeah. I think we're still discovering, like what the rules 688 00:37:38.519 --> 00:37:42.519 of responsibility are. Like I feel like we're learning quickly, 689 00:37:42.559 --> 00:37:44.960 but we're still learning, you know, some of those being 690 00:37:45.000 --> 00:37:49.239 like you have to give it very discrete tasks, you know, 691 00:37:49.639 --> 00:37:52.159 ask it to do one thing so that the result 692 00:37:52.199 --> 00:37:54.199 that comes back to you is something you can actually 693 00:37:54.760 --> 00:37:57.079 parse in your head. And you know, don't give it 694 00:37:57.159 --> 00:37:59.840 something that you don't know how to do, because then 695 00:37:59.880 --> 00:38:03.320 you have no way to check its work. Or another 696 00:38:03.360 --> 00:38:06.159 one that I use that I really like is treat 697 00:38:06.199 --> 00:38:10.840 it like your own personal intern or junior developer. You know, 698 00:38:10.880 --> 00:38:15.960 you wouldn't like hire an intern and then say, you know, hey, 699 00:38:16.400 --> 00:38:20.760 go architect this four tiered micro service for us. It'll 700 00:38:20.800 --> 00:38:21.760 be due by Friday. 701 00:38:22.079 --> 00:38:26.360 Oops. I mean you're onto something there, will I will 702 00:38:26.400 --> 00:38:29.719 say that maybe it's just forcing us to finally think 703 00:38:29.760 --> 00:38:35.119 about concretely how human should actually work together effectively. And 704 00:38:35.599 --> 00:38:37.280 I you know, in the last you know what is 705 00:38:37.320 --> 00:38:39.519 the year twenty twenty five, we haven't figured out how 706 00:38:39.519 --> 00:38:41.800 to do that correctly, and now we're throwing in a 707 00:38:41.840 --> 00:38:44.039 tool into the next which actually requires us to do 708 00:38:44.079 --> 00:38:46.679 that work. I'm going to guess it's going to be 709 00:38:46.679 --> 00:38:51.000 like another two hundred years before we got we got that, 710 00:38:51.280 --> 00:38:53.840 you know, actually figured out how to optimally communicate one 711 00:38:53.920 --> 00:38:54.320 human being. 712 00:38:54.639 --> 00:38:57.199 I'm telling you, I have a teenager, and like I 713 00:38:57.239 --> 00:38:59.199 would just like to point out that we have been 714 00:38:59.280 --> 00:39:01.800 peopling for millions of years and we still do not 715 00:39:01.960 --> 00:39:06.480 have a good way to make the teenagers not really 716 00:39:06.519 --> 00:39:10.039 really dumb Okay, like just it's never gonna happen. It's 717 00:39:10.039 --> 00:39:10.840 never gonna happen. 718 00:39:11.719 --> 00:39:14.639 Millions is a bit long. I'll give you like four 719 00:39:14.719 --> 00:39:16.239 hundred thousand or so. 720 00:39:16.559 --> 00:39:17.159 But I. 721 00:39:18.639 --> 00:39:22.119 Do definitely accept that argument that you know, but you know, 722 00:39:22.400 --> 00:39:25.320 acceleration of evolution, so maybe maybe we'll get there faster 723 00:39:25.400 --> 00:39:25.760 this time. 724 00:39:26.199 --> 00:39:31.599 Right, Moore's law for humans, that's a thing, right, Yeah, Yeah, I. 725 00:39:32.199 --> 00:39:35.599 Really appreciate Moore's law. What I don't appreciate is people 726 00:39:35.679 --> 00:39:37.679 saying that Moore's law applies to things that it doesn't 727 00:39:37.679 --> 00:39:42.199 apply to. I mean, I totally agree that the principle 728 00:39:42.719 --> 00:39:47.280 by what is it correlation doesn't necessarily imply causation. Although 729 00:39:47.559 --> 00:39:49.039 you know, you look at lots of things and they 730 00:39:49.039 --> 00:39:52.480 do double at you know, some rate. That's but people 731 00:39:52.480 --> 00:39:54.679 get more law wrong a lot. They apply to things 732 00:39:54.719 --> 00:39:57.000 that don't make sense. It is literally just the size 733 00:39:57.000 --> 00:40:00.400 of the transistors will reduce by half every eighteen months. 734 00:40:00.559 --> 00:40:02.920 That's that's it. That's the whole law. It doesn't apply 735 00:40:03.199 --> 00:40:06.519 like statistically to anything else. Now we do see doubling 736 00:40:06.880 --> 00:40:10.079 and size reduction and other things definitely, like dies on 737 00:40:10.119 --> 00:40:12.199 a chip and et cetera. But you know, there's like 738 00:40:12.199 --> 00:40:14.719 an interesting thing. Does it apply to these new plastic 739 00:40:14.800 --> 00:40:17.320 chips that I was China producing now that don't use 740 00:40:17.320 --> 00:40:21.400 silicon I don't remember the material and that are supposedly 741 00:40:21.480 --> 00:40:23.639 like half or a tenth of the size, you know, 742 00:40:23.719 --> 00:40:25.280 does Moore's law apply there too? 743 00:40:26.119 --> 00:40:26.840 I have no idea. 744 00:40:26.880 --> 00:40:30.719 I was just trying to make a joke, okay for us. 745 00:40:32.440 --> 00:40:35.159 I wanted to backtrack just like a little bit and 746 00:40:35.199 --> 00:40:37.840 point out Amy that I really like your kind of 747 00:40:38.159 --> 00:40:42.000 go gettedness, I guess with understanding actually what the business needs. 748 00:40:42.119 --> 00:40:45.599 Because I work with a lot of people and especially students, 749 00:40:45.639 --> 00:40:47.840 and they just have no idea, like no idea like 750 00:40:47.880 --> 00:40:49.320 what it is, like what is supposed to be the 751 00:40:49.400 --> 00:40:52.719 end result of this kind of like process that they're doing. 752 00:40:53.079 --> 00:40:55.239 And I think it's very important. I mean it's important 753 00:40:55.280 --> 00:40:57.760 from like work, but I think it's also very important 754 00:40:57.760 --> 00:41:00.199 from a career perspective as well, Like would you like 755 00:41:00.239 --> 00:41:04.159 to maybe get new jobs? Like you should? You should 756 00:41:04.239 --> 00:41:06.159 understand what's going on at least a little bit. 757 00:41:06.840 --> 00:41:08.920 Yeah, Like I said, I don't know. 758 00:41:09.039 --> 00:41:11.440 I just like still to this day feel very fortunate 759 00:41:11.480 --> 00:41:14.800 to be able to do something that I pretty much 760 00:41:14.880 --> 00:41:19.800 enjoy every single day. And I just I don't know, 761 00:41:20.480 --> 00:41:23.920 maybe I just just like too harsh growing up or something. 762 00:41:24.000 --> 00:41:27.159 But I'm just like I feel like if I'm getting 763 00:41:27.199 --> 00:41:32.000 a salary, then I should be. There is value in 764 00:41:32.079 --> 00:41:35.079 pushing back sometimes when you're like senior enough to have 765 00:41:35.119 --> 00:41:38.199 a level of expertise, like in a professional and respectful way. 766 00:41:39.039 --> 00:41:41.760 But yes, like, at the end of the day, our job, 767 00:41:41.840 --> 00:41:43.679 whether we like it or not, is to make this 768 00:41:43.760 --> 00:41:50.480 company profitable, whether that be features, making sure that users 769 00:41:50.800 --> 00:41:53.360 have confidence in the service. 770 00:41:53.519 --> 00:41:55.639 Yeah, sorry, stuff like that, They're. 771 00:41:55.440 --> 00:41:59.800 Going to say, despite leaderships and a strategy to the contrary, 772 00:42:01.199 --> 00:42:02.760 I am. I am sort of curious that you have 773 00:42:02.840 --> 00:42:08.000 any you know, personal strategies for gleaning better insight into 774 00:42:08.360 --> 00:42:11.719 how the business should work or how a feature that is, 775 00:42:11.800 --> 00:42:15.840 say technical, transforms into value for the business. Are you 776 00:42:15.920 --> 00:42:19.960 working with product managers or with someone on the marketing 777 00:42:19.960 --> 00:42:23.519 and sales team? Are you grabbing information from the internet? 778 00:42:23.559 --> 00:42:25.599 Like how are you actually figuring this out? 779 00:42:26.239 --> 00:42:30.679 Like what is and isn't valuable for the business? I 780 00:42:30.719 --> 00:42:32.440 guess it's for. 781 00:42:32.559 --> 00:42:36.320 Me personally over my career. It probably depends on the 782 00:42:36.320 --> 00:42:39.360 domain I'm working in. So I've worked in like very 783 00:42:39.440 --> 00:42:43.280 like domains of like the construction industry, Like I don't 784 00:42:43.320 --> 00:42:50.000 have expertise in what these people want obviously, Like I 785 00:42:50.039 --> 00:42:52.920 like my personality, I want to understand the domain that 786 00:42:52.960 --> 00:42:55.599 I'm working in so that I can try to better 787 00:42:55.679 --> 00:42:58.639 understand these types of things. But in that sort of scenario, 788 00:42:58.760 --> 00:43:00.599 like I am going to lean more on the product 789 00:43:00.639 --> 00:43:04.960 people now if it's like a dev focused company. You know, 790 00:43:05.159 --> 00:43:09.559 my own thoughts obviously, like doing this a while, like 791 00:43:09.639 --> 00:43:11.880 having friends that I work with and like hearing their 792 00:43:11.920 --> 00:43:17.320 stories that goes into like what isn't isn't valuable? Like 793 00:43:18.320 --> 00:43:22.679 I'm trying to think of like different scenarios like that 794 00:43:22.760 --> 00:43:27.719 I've been in. There was definitely one later on in 795 00:43:27.719 --> 00:43:32.400 my career where there was like this is like an 796 00:43:32.480 --> 00:43:37.519 SR type thing, so like the security team was extremely 797 00:43:37.559 --> 00:43:41.320 focused on like getting this tool out because they were 798 00:43:41.480 --> 00:43:48.079 getting paged based on things that were happening. But the 799 00:43:48.239 --> 00:43:55.559 tool that they had vetted, while it seemed good, it 800 00:43:55.719 --> 00:43:57.960 was a very like rush job to get it out 801 00:43:58.000 --> 00:44:02.159 in time. And so like on my side of things, 802 00:44:02.519 --> 00:44:04.639 like there was a lot of code that needed to 803 00:44:04.639 --> 00:44:08.519 be deployed to get form the infant being a little vague, 804 00:44:10.239 --> 00:44:13.719 so I'll try if I can. That's fair, I can clarify, 805 00:44:13.800 --> 00:44:16.480 I will, but at the end of the sorry it. 806 00:44:16.440 --> 00:44:19.320 Works, no, but it works because the more vague you are, 807 00:44:19.440 --> 00:44:22.679 the more trauma that listeners will have because it will 808 00:44:22.719 --> 00:44:25.800 relate to their situation. Like I have like three different 809 00:44:25.800 --> 00:44:27.719 traumas right now, just based off of what you said. 810 00:44:27.880 --> 00:44:30.840 You know, security teams pushing stuff out, you know, not 811 00:44:30.920 --> 00:44:33.719 knowing what to do, you know, not well vetted software 812 00:44:33.760 --> 00:44:37.280 that now everyone is dependent on, like you know, not 813 00:44:37.440 --> 00:44:38.599 a short number of times. 814 00:44:39.119 --> 00:44:43.199 Yeah, So like and I have, like in this particular scenario, 815 00:44:43.280 --> 00:44:46.400 like a really close relationship, like really got long well 816 00:44:46.559 --> 00:44:49.360 with the team that wanted to push this out. In 817 00:44:49.440 --> 00:44:51.280 the ser side of things, I keep saying, you have 818 00:44:51.320 --> 00:44:53.719 to understand like the tier one, Tier two, Tier three, 819 00:44:53.960 --> 00:44:56.719 and this was getting deployed to like tier one, So 820 00:44:56.760 --> 00:44:57.639 this is like getting. 821 00:44:57.400 --> 00:45:01.000 Deployed at the edge. So if this is wrong, like. 822 00:45:02.400 --> 00:45:04.800 There's no like the quest the request doesn't go through, 823 00:45:05.480 --> 00:45:08.000 like it doesn't even it's not even like a latency issue, 824 00:45:08.000 --> 00:45:11.760 like it's it's dead. Yeah, it's I don't know. I 825 00:45:12.000 --> 00:45:13.360 see Lauren like shaking his head. 826 00:45:13.440 --> 00:45:17.079 So I was gonna say past trauma, like actually will 827 00:45:17.079 --> 00:45:19.239 and I will, and I did an episode last year 828 00:45:19.599 --> 00:45:23.440 about how to handle scaling around the holidays, because you 829 00:45:23.480 --> 00:45:26.440 know it's a time, especially for companies that do e 830 00:45:26.440 --> 00:45:28.880 commerce et cetera, anything with orders, there is a huge 831 00:45:28.920 --> 00:45:31.400 spike and how to be prepared for that. And it's 832 00:45:31.400 --> 00:45:34.719 sort of interesting that if you break down the systems 833 00:45:34.719 --> 00:45:36.920 thinking model you brought it up, there is this perverse 834 00:45:36.960 --> 00:45:40.280 incentive where if you have a deadline, that you are 835 00:45:40.320 --> 00:45:43.360 more likely to rush to meet that deadline, which means 836 00:45:43.360 --> 00:45:46.000 that you cut more corners, which means the thing is 837 00:45:46.039 --> 00:45:48.159 more risky. And so if you create a deadline or 838 00:45:48.199 --> 00:45:51.639 a code freeze to prevent risky code from getting into production, 839 00:45:51.719 --> 00:45:55.039 you are actually encouraging the exact thing that you are 840 00:45:55.079 --> 00:45:55.840 trying to prevent. 841 00:45:56.719 --> 00:45:59.719 So and like also on that end, and this is 842 00:45:59.719 --> 00:46:02.280 where like it, I guess it does like you start 843 00:46:02.320 --> 00:46:07.159 to like traumatic events and you that is usually the case. 844 00:46:07.320 --> 00:46:11.239 I've also seen the case of we do a code 845 00:46:11.280 --> 00:46:16.719 freeze and a system doesn't get touched for six months, 846 00:46:16.760 --> 00:46:22.719 maybe even a year, and then you decide that you 847 00:46:23.079 --> 00:46:26.400 do want to deploy like a new feature, and boom 848 00:46:26.559 --> 00:46:30.039 it's down and hard down because nobody has touched it 849 00:46:30.079 --> 00:46:35.000 in a year. And so like this certain situation that 850 00:46:35.079 --> 00:46:37.559 I ran into months we had like a reindexing job 851 00:46:38.079 --> 00:46:41.639 and there was so much data to reindex that what 852 00:46:42.079 --> 00:46:44.960 that the machines that it had been running on could 853 00:46:45.039 --> 00:46:49.400 no longer support the reindex process. So we're like screwed 854 00:46:49.440 --> 00:46:54.920 at this point. So yes, so you have to wait, 855 00:46:55.920 --> 00:47:00.840 and there's trade offs in all regards, and as I sorry, 856 00:47:00.960 --> 00:47:03.719 like it's just really your job to try to understand 857 00:47:03.760 --> 00:47:06.880 like not even like what's the perfect scenario, but like 858 00:47:06.960 --> 00:47:10.440 what's the least of like the evils here? 859 00:47:10.599 --> 00:47:14.440 He dos actually is a really interesting guide for resilience engineering, 860 00:47:14.480 --> 00:47:16.239 and one of the things that they talk a lot 861 00:47:16.239 --> 00:47:19.840 about is what happens when there is like one or 862 00:47:19.880 --> 00:47:23.679 two points of failure, what actual new systems start coming 863 00:47:23.679 --> 00:47:25.760 into play, What does the request volume start to look 864 00:47:25.800 --> 00:47:30.159 like versus the nominal state? And you mentioned cash is 865 00:47:30.159 --> 00:47:31.280 at the edge, and I think one of the most 866 00:47:31.280 --> 00:47:34.199 common scenarios is why do you add a cash well 867 00:47:34.239 --> 00:47:36.480 because you're trying to reduce the load on your system, 868 00:47:37.039 --> 00:47:39.639 which means that you create the ability to create higher 869 00:47:39.679 --> 00:47:41.800 load on your system to hit the cash and as 870 00:47:41.840 --> 00:47:44.400 soon as the cash you know, drops or gets busted 871 00:47:44.400 --> 00:47:46.760 because you're using something like valky because no one's using 872 00:47:46.760 --> 00:47:50.840 Reddus anymore. Of course, which isn't designed to be persistent. 873 00:47:51.000 --> 00:47:53.000 It's going to drop at some point, which means you're 874 00:47:53.000 --> 00:47:54.960 going to get a lot of load to your system 875 00:47:55.079 --> 00:47:57.000 without going through the cash and it's going to cause 876 00:47:57.000 --> 00:47:58.400 a hard failure at that point, and how are you 877 00:47:58.440 --> 00:48:01.079 going to recover? And so it sounds great to add 878 00:48:01.119 --> 00:48:02.519 a cash in, but then you're gonna have to deal 879 00:48:02.559 --> 00:48:05.360 with this quite critical scenario where you are just instead 880 00:48:05.360 --> 00:48:08.360 of being able to handle any number requests, you're going 881 00:48:08.400 --> 00:48:09.400 to be able to handle nothing. 882 00:48:09.920 --> 00:48:12.800 Yeah, back to like what you're saying the distributed system stuff, 883 00:48:12.840 --> 00:48:17.159 Like I in my mind you brought up like single 884 00:48:17.199 --> 00:48:21.760 point of failure and it's just like the nature of 885 00:48:21.800 --> 00:48:23.639 how things work, like you're always going to have a 886 00:48:23.679 --> 00:48:26.079 single point of failure. And it's not so much like 887 00:48:26.119 --> 00:48:27.800 how do we get rid of this single point of failure, 888 00:48:27.800 --> 00:48:31.440 because I don't even think that's possible, because you're going 889 00:48:31.519 --> 00:48:33.599 to have like traffic management at some point, Like what 890 00:48:33.639 --> 00:48:38.280 if the traffic management fails? It's more so in my mind, 891 00:48:38.440 --> 00:48:43.320 like what is what system seems to be the most 892 00:48:43.360 --> 00:48:46.239 resilient and that's where we want to place this single 893 00:48:46.280 --> 00:48:49.719 point of failure. So like usually it is like the distribution, 894 00:48:50.039 --> 00:48:52.400 Like if you have like a cash distribution, that is 895 00:48:52.519 --> 00:48:56.280 usually the in my opinion, like your wisest single point 896 00:48:56.280 --> 00:48:56.760 of failure. 897 00:48:57.280 --> 00:48:59.599 This is an interesting way of phrasing and I haven't 898 00:48:59.639 --> 00:49:00.880 heard that perspective before. 899 00:49:01.599 --> 00:49:04.199 What's your on call rotation look like? 900 00:49:07.320 --> 00:49:10.840 So it's super different based on the company. I am 901 00:49:10.840 --> 00:49:16.559 a big fan of companies who are hopefully you work 902 00:49:16.599 --> 00:49:19.719 at a company that's large enough that they have engineers 903 00:49:19.760 --> 00:49:22.760 across the globe and so your on call rotation can 904 00:49:22.960 --> 00:49:27.320 fall within the hours that allow you to uh work 905 00:49:27.400 --> 00:49:32.639 optimally and not be up all night. But like I said, 906 00:49:32.880 --> 00:49:36.360 I also firmly believe like everybody should share in the 907 00:49:36.480 --> 00:49:39.679 on call rotation because it shares ownership and if things 908 00:49:39.719 --> 00:49:41.880 start to go awry, then you have a vested interest 909 00:49:41.880 --> 00:49:45.199 in making things better. But the on call rotation like 910 00:49:45.360 --> 00:49:48.079 is it? It is what it is, and you know 911 00:49:48.119 --> 00:49:49.639 it can be a little rough at the beginning, but 912 00:49:49.719 --> 00:49:53.559 it's a great way to learn honestly, like just dive 913 00:49:53.599 --> 00:49:57.760 in because there's nobody knows everything in the system. Like 914 00:49:57.960 --> 00:50:00.079 you're going to hit something that probably somebody has and 915 00:50:00.199 --> 00:50:00.679 hit before. 916 00:50:01.159 --> 00:50:04.199 When you say everyone, like, what's the size of the 917 00:50:04.440 --> 00:50:06.800 org uh that you're thinking about there? 918 00:50:07.760 --> 00:50:08.000 You know. 919 00:50:08.159 --> 00:50:10.920 I've been in teams where, like my own conrotation honestly 920 00:50:11.000 --> 00:50:13.760 was like every other week, which is a little rough. 921 00:50:13.880 --> 00:50:16.199 I've been on teams where it's I want to say, 922 00:50:16.280 --> 00:50:18.880 like maybe once a month or something like that. I 923 00:50:18.880 --> 00:50:24.159 feel like that's pretty reasonable. I think it's helpful to have, 924 00:50:24.239 --> 00:50:27.000 like I said, like the tears of support also to 925 00:50:27.400 --> 00:50:32.360 like hopefully it doesn't get to this point, but it's 926 00:50:32.400 --> 00:50:35.239 definitely a real thing. I've seen, like with myself and others, 927 00:50:35.280 --> 00:50:38.400 where you're on an issue for twenty four or forty 928 00:50:38.440 --> 00:50:41.400 hours straight and you need to hand that off because 929 00:50:41.440 --> 00:50:44.079 after a certain amount of time, like you're just your 930 00:50:44.079 --> 00:50:51.280 brain dead, or like you need to bathe, right, I 931 00:50:51.320 --> 00:50:53.480 mean I guess like if I think of like military 932 00:50:53.559 --> 00:50:55.679 or something like, you can go a long time without bathing. 933 00:50:55.719 --> 00:51:00.360 So maybe that's not the best example. Like you need to. 934 00:51:00.360 --> 00:51:02.960 Sleep, especially in a remote workforce culture. 935 00:51:04.079 --> 00:51:06.599 Yeah, like you need sleep to be able to like 936 00:51:07.159 --> 00:51:11.400 debug things, but there should usually I feel like in 937 00:51:11.519 --> 00:51:14.320 a proper on call rotation, like a handoff phase, if 938 00:51:14.320 --> 00:51:16.800 it's been like more than twenty four hours something. 939 00:51:16.519 --> 00:51:19.280 Like that, yeah, I will. I think follow the sun 940 00:51:19.440 --> 00:51:21.199 implies that you, like you hand it off to someone 941 00:51:21.239 --> 00:51:23.480 who's still awake at that time and not you. You 942 00:51:23.599 --> 00:51:25.840 just stay awake until the sun comes up. 943 00:51:27.679 --> 00:51:31.960 That that, in my experience, it typically works the best. 944 00:51:32.400 --> 00:51:34.280 So this is this may be a little bit of 945 00:51:34.320 --> 00:51:39.000 a tangent, but since you're from the JavaScript Java Jabba podcast, 946 00:51:39.079 --> 00:51:42.480 I have to ask you preference no JS, DNO or BOND. 947 00:51:43.239 --> 00:51:47.079 I'm old and boring, so I like nodes still No. 948 00:51:47.400 --> 00:51:51.079 I've talked to like the Dino team and like I 949 00:51:51.079 --> 00:51:52.400 I don't have anything against that. 950 00:51:52.519 --> 00:51:54.559 I just I'm old and boring. 951 00:51:54.280 --> 00:51:56.760 And I shouldn't say I owe Jos. 952 00:51:57.880 --> 00:52:00.079 I still like Pearl, so like, I agree. 953 00:52:00.360 --> 00:52:02.639 I think it's just fitting that she said I like Pearl, 954 00:52:02.679 --> 00:52:04.320 and the rest of the audio cut out. 955 00:52:04.119 --> 00:52:07.559 After that, The said part is like it's still recording 956 00:52:07.599 --> 00:52:10.079 for her, so we'll find out when the episode released 957 00:52:10.119 --> 00:52:11.519 was She actually said. 958 00:52:11.400 --> 00:52:13.320 It's only for this podcast, you guys that I have 959 00:52:13.360 --> 00:52:15.840 so many technical problems, Like I'm fine in all of 960 00:52:15.880 --> 00:52:16.760 my other meetings. 961 00:52:16.800 --> 00:52:20.639 It's just this'll be doing something right now. 962 00:52:21.000 --> 00:52:22.079 I believe you. Jillian. 963 00:52:22.519 --> 00:52:23.039 That's good. 964 00:52:23.320 --> 00:52:24.079 I have no comment. 965 00:52:26.360 --> 00:52:28.440 That's probably for the best quite possibly. 966 00:52:28.880 --> 00:52:31.639 How do you deal with the complexity of having to 967 00:52:31.679 --> 00:52:34.599 share so much knowledge with so many people? So the 968 00:52:34.639 --> 00:52:38.320 more people that are working in solution in an area 969 00:52:38.360 --> 00:52:40.920 and a team or means there's more things that are 970 00:52:40.960 --> 00:52:43.159 going to be created, which means there's more things everyone 971 00:52:43.199 --> 00:52:47.000 will have to be aware of to potentially deal with 972 00:52:47.039 --> 00:52:49.960 when an on call incident comes up. Is there a 973 00:52:50.000 --> 00:52:53.239 good metric or litmus test to figure out what the 974 00:52:53.280 --> 00:52:56.760 maximum amount of stuff is that a team can actually manage. 975 00:52:57.239 --> 00:53:00.159 Yeah, honestly, that is a very good question, and I 976 00:53:00.840 --> 00:53:02.840 feel like I don't have a good answer, but now 977 00:53:02.880 --> 00:53:05.760 that's something I want to go see if I can 978 00:53:05.880 --> 00:53:11.199 dig into more. But I also say, like how we 979 00:53:11.280 --> 00:53:15.760 have like these different like tiers of support. It's hard 980 00:53:15.880 --> 00:53:18.159 if you get into like a more senior role because 981 00:53:18.199 --> 00:53:21.440 stuff is going to bubble up to you, but it's 982 00:53:21.519 --> 00:53:25.000 usually helpful. Like in a large team, there's usually even 983 00:53:25.079 --> 00:53:28.039 like slas to the developer team. So even like a 984 00:53:28.079 --> 00:53:33.280 lower environment is going to like be held accountable to 985 00:53:33.280 --> 00:53:35.199 the business to keep a lower environment up in. 986 00:53:35.199 --> 00:53:36.119 A certain amount of time. 987 00:53:38.119 --> 00:53:41.920 Like sort of related to your question, like having those 988 00:53:41.960 --> 00:53:44.639 types of people like work in those environments and that 989 00:53:44.719 --> 00:53:47.400 helps distribute knowledge of like how these different systems work 990 00:53:47.480 --> 00:53:51.719 without being tasked with like the actual production environment, and 991 00:53:51.760 --> 00:53:55.920 then if they do well in those lower environments, and 992 00:53:55.960 --> 00:53:58.360 then they can kind of move on to a production environment, 993 00:53:58.400 --> 00:54:00.239 and that helps spread the nolle. 994 00:54:00.400 --> 00:54:04.840 Pray that's prey. I mean, I think there's a there's 995 00:54:04.840 --> 00:54:07.400 a corollary here where I often have to advise some 996 00:54:07.440 --> 00:54:10.239 companies on which is we have like five different mobile 997 00:54:10.239 --> 00:54:12.760 apps because you know, there's many different providers out there, 998 00:54:12.800 --> 00:54:17.000 and then there's a website and also web native version 999 00:54:17.159 --> 00:54:19.440 for your mobile app that isn't like an app version 1000 00:54:19.480 --> 00:54:22.320 but is responsive like and then there's back and that 1001 00:54:22.360 --> 00:54:24.920 goes with it. And the infrastructure is that all one team, 1002 00:54:25.079 --> 00:54:27.480 you know, so everyone is cross functional and understand what's 1003 00:54:27.480 --> 00:54:29.639 going on. Or do you have like ten teams that 1004 00:54:29.760 --> 00:54:32.039 each have the exact same functionality. And I feel like 1005 00:54:32.079 --> 00:54:35.880 the answer keeps on being well, they're both wrong and 1006 00:54:35.960 --> 00:54:38.000 there's no there's no good solution there, and you just 1007 00:54:38.000 --> 00:54:39.840 got to have to deal with the consequences. 1008 00:54:40.039 --> 00:54:40.719 It's awesome. 1009 00:54:40.840 --> 00:54:43.320 I mean, this is why I did startups for a 1010 00:54:43.360 --> 00:54:48.639 long time. But as I've gotten like further into my career, 1011 00:54:48.840 --> 00:54:53.239 I feel like it's hard to make an impact unless 1012 00:54:53.280 --> 00:54:55.800 you've been at the company for really for a good 1013 00:54:55.840 --> 00:54:58.760 amount of time because it the larger the code base 1014 00:54:58.840 --> 00:55:01.280 and like the domain than and you really do need 1015 00:55:01.280 --> 00:55:05.239 that time to understand it, whereas like a startup usually 1016 00:55:05.280 --> 00:55:08.760 like a smaller application, and you can understand it more. 1017 00:55:08.800 --> 00:55:11.960 But like a lot of these like larger companies, they 1018 00:55:11.960 --> 00:55:14.280 have like these huge systems and it takes time to 1019 00:55:14.480 --> 00:55:15.199 understand it. 1020 00:55:16.800 --> 00:55:19.840 That's a really interesting challenge. Then we see in the 1021 00:55:19.840 --> 00:55:24.480 tech industry the average tenure for engineers reducing over time. Yeah, 1022 00:55:24.599 --> 00:55:27.800 does that mean that we're actually more and more incapable 1023 00:55:27.880 --> 00:55:30.960 of building larger systems because people aren't staying at the 1024 00:55:30.960 --> 00:55:34.440 company to be able to diagnose and build more resilient systems. 1025 00:55:34.480 --> 00:55:36.519 Are we still attempting to do that and the systems 1026 00:55:36.559 --> 00:55:39.679 we're building just aren't resilient anymore? Or is this just 1027 00:55:39.800 --> 00:55:42.960 like you see whatever problems are close to you, and 1028 00:55:43.360 --> 00:55:46.920 larger companies with bigger solutions are doing a good job 1029 00:55:46.960 --> 00:55:51.760 at retaining experienced staff that have experience in their own 1030 00:55:51.760 --> 00:55:53.920 solution in order to help solve those challenges. 1031 00:55:54.360 --> 00:55:58.159 It's it's very tricky because yeah, there's like a trade 1032 00:55:58.199 --> 00:56:01.039 off both ways, Like you stay a company long enough, 1033 00:56:01.079 --> 00:56:03.400 like yes, you might kind of have some like siloed 1034 00:56:03.400 --> 00:56:08.159 information and start to not know how other people are 1035 00:56:08.159 --> 00:56:11.239 solving problems, and then that kind of can produce like 1036 00:56:11.320 --> 00:56:15.719 some blind spots in your knowledge. But at the same time, 1037 00:56:15.960 --> 00:56:18.719 like if you don't stay at a place long enough, 1038 00:56:19.719 --> 00:56:23.559 it can be hard to see like how this huge 1039 00:56:23.599 --> 00:56:24.880 system interacts and. 1040 00:56:24.920 --> 00:56:25.599 Be able to. 1041 00:56:27.039 --> 00:56:29.960 Really solve like these harder problems because it takes a 1042 00:56:29.960 --> 00:56:32.559 while to like get the whole domain, the whole system 1043 00:56:32.639 --> 00:56:37.119 in your head. It's a there's definitely like like a 1044 00:56:37.239 --> 00:56:40.199 vested interest I feel like in these companies and like retaining. 1045 00:56:39.800 --> 00:56:42.519 People, well, yeah, for sure. I mean there's the benefits 1046 00:56:42.519 --> 00:56:44.679 to them and not to the individual and a lot 1047 00:56:44.679 --> 00:56:46.360 of times, I mean, I guess, you know, maybe a 1048 00:56:46.360 --> 00:56:47.840 different way of phrasing this is if we look at 1049 00:56:47.880 --> 00:56:50.480 Dumbar's number and ORG is going to be at max 1050 00:56:50.519 --> 00:56:53.280 like one hundred and fifty people, how many years of 1051 00:56:53.360 --> 00:56:57.920 experience working in that company do we project someone would 1052 00:56:58.039 --> 00:57:00.199 a new hire would have to work before they or 1053 00:57:00.840 --> 00:57:03.360 at the point where they are delivering or understanding the 1054 00:57:03.400 --> 00:57:06.679 system sufficiently that one hundred and fifty person ORG has 1055 00:57:06.719 --> 00:57:08.119 created any thoughts on that. 1056 00:57:08.119 --> 00:57:12.000 That's an interesting question too, like how it kind of 1057 00:57:12.039 --> 00:57:15.599 like scales based on the size and the system and 1058 00:57:15.679 --> 00:57:16.519 how long you'd have. 1059 00:57:16.519 --> 00:57:19.159 To be there, but a personal preference, you know, like 1060 00:57:19.280 --> 00:57:21.360 oh yeah, once I'm there, you know, six months, I 1061 00:57:21.360 --> 00:57:24.880 feel like I'm now somewhat accountable for every system or 1062 00:57:25.119 --> 00:57:27.000 a year or two years or something like that. 1063 00:57:27.480 --> 00:57:32.440 It probably this doesn't answer your question, unfortunately, but I 1064 00:57:32.440 --> 00:57:35.440 would say like one of the things that could be 1065 00:57:35.519 --> 00:57:40.159 helpful is like if you start to like segment your 1066 00:57:40.159 --> 00:57:43.280 career working in a similar domain, so some of that 1067 00:57:43.400 --> 00:57:48.280 knowledge is transferable, like whether you're talking about like the 1068 00:57:48.360 --> 00:57:52.960 hyperscaler that you're working on, or if you're more focused 1069 00:57:53.000 --> 00:57:57.400 on like CDM distribution, and maybe that's valuable to other companies, 1070 00:57:57.400 --> 00:58:00.119 whereas like if it's a you know, if it's a 1071 00:58:00.920 --> 00:58:05.199 paylalled type of application, that maybe your CDN experience is 1072 00:58:05.239 --> 00:58:06.639 not as beneficial there. 1073 00:58:07.360 --> 00:58:09.159 I think also when you say one hundred and fifty 1074 00:58:09.199 --> 00:58:11.800 person or you refer into like the entire company or 1075 00:58:11.920 --> 00:58:13.480 just engineering. 1076 00:58:13.280 --> 00:58:15.679 I mean even a subset of engineering potentially, like I 1077 00:58:15.760 --> 00:58:17.960 just like at Dunbar's number, like your teams are going 1078 00:58:18.039 --> 00:58:20.280 to be like two pizza teams for instance, that are 1079 00:58:20.360 --> 00:58:22.519 cross functional. But then there are different types of teams 1080 00:58:22.519 --> 00:58:25.639 and different features within the product or product suite that 1081 00:58:25.679 --> 00:58:28.039 you're delivering, and that can only get up to one 1082 00:58:28.079 --> 00:58:30.400 hundred and fifty. Above that you need to create a 1083 00:58:30.440 --> 00:58:32.920 second org and have them work on a completely different 1084 00:58:32.960 --> 00:58:35.519 feature set with a completely different focus. You know, you 1085 00:58:35.519 --> 00:58:37.440 think of like how in a hyperscale or I'm going 1086 00:58:37.480 --> 00:58:40.599 to use Google Workspace as an example, like email, Gmail 1087 00:58:40.679 --> 00:58:44.000 and drive, right, I can imagine those being separate, completely 1088 00:58:44.039 --> 00:58:47.360 separate orgs for instance, But I would not expect one 1089 00:58:47.559 --> 00:58:51.280 org structure where it's fundamentally accountable for every single product 1090 00:58:51.639 --> 00:58:54.480 in Google Workspace. There's just too much stuff there. So 1091 00:58:55.280 --> 00:58:57.119 that's the part for me that makes the most sense, 1092 00:58:57.159 --> 00:58:58.760 that you can't have more than one hundred and fifty 1093 00:58:58.800 --> 00:59:01.519 people working on the same product because there's only so 1094 00:59:01.599 --> 00:59:04.119 much functional work there, and at a larger size you 1095 00:59:04.159 --> 00:59:07.639 have a communication breakdown. You can't pass the information effectively. 1096 00:59:07.679 --> 00:59:10.960 So that's just sort of the metrics on the math 1097 00:59:11.119 --> 00:59:12.239 that seem to work out. 1098 00:59:12.559 --> 00:59:15.000 Too many cooks in the kitchen, and like I think 1099 00:59:15.400 --> 00:59:17.599 for better where some people have like a vested interest 1100 00:59:17.639 --> 00:59:20.519 in like trying to offer value, but unfortunately some there's 1101 00:59:20.519 --> 00:59:21.519 the values not there. 1102 00:59:22.400 --> 00:59:24.480 Yeah, I mean even to your point of like, well, 1103 00:59:24.559 --> 00:59:26.960 let's have an on call rotation in everyone and say 1104 00:59:26.960 --> 00:59:29.719 the team in the ORG could participate in right, one 1105 00:59:29.760 --> 00:59:32.440 hundred and fifty people means you're on call for your 1106 00:59:32.480 --> 00:59:37.639 product once every three years, right, So that doesn't work out, right, 1107 00:59:37.679 --> 00:59:39.440 So you know if your August fifty people, then you 1108 00:59:39.480 --> 00:59:41.920 get to the okay, once a year, I'm on a call. Well, 1109 00:59:41.960 --> 00:59:44.679 that maybe works out potentially if you have a low 1110 00:59:44.800 --> 00:59:48.000 number of bugs. You deal with real time applications that 1111 00:59:48.239 --> 00:59:50.639 can never really be down, highly resilient, and you run 1112 00:59:51.159 --> 00:59:55.000 you're probably running fire drills during during working hours as 1113 00:59:55.039 --> 00:59:57.800 if there were an incident to know how to respond 1114 00:59:57.800 --> 01:00:00.880 and not just hopefully waiting and never seeing a problem. 1115 01:00:01.440 --> 01:00:04.760 I kind of like, I feel like computer systems like 1116 01:00:05.239 --> 01:00:07.760 mirror a lot of like real life. So whether it 1117 01:00:07.800 --> 01:00:10.920 be like I people who don't understand astory, they're kind 1118 01:00:10.960 --> 01:00:12.199 of like, what in the world do you do? I'm like, 1119 01:00:12.280 --> 01:00:17.519 I'm basically like an emergency room doctor, and like somebody 1120 01:00:17.559 --> 01:00:19.840 comes into the emergency room and my responsibility is like 1121 01:00:19.920 --> 01:00:23.719 keep them alive until the right person can come in, 1122 01:00:23.800 --> 01:00:27.239 and you know, get them into like critical condition, not. 1123 01:00:27.599 --> 01:00:31.280 Even not even the surge end, but you know, high level. 1124 01:00:31.400 --> 01:00:34.199 I understand how how patients work and the sets of 1125 01:00:34.239 --> 01:00:36.360 problems that can happen, and I can direct people on 1126 01:00:36.400 --> 01:00:37.159 the right places. 1127 01:00:37.320 --> 01:00:39.199 Yeah, the wet stuff on the inside and this dry 1128 01:00:39.239 --> 01:00:40.119 stuff on the outside. 1129 01:00:40.960 --> 01:00:42.159 I love it. 1130 01:00:44.480 --> 01:00:47.079 You said, you said nuclear physics. Well I don't think 1131 01:00:47.079 --> 01:00:52.360 you said E M T right right right, Yeah, true, true, true. 1132 01:00:52.559 --> 01:00:57.440 But I do spend not to just qualifies my opinion 1133 01:00:57.760 --> 01:00:59.599 by any means, but I do spend a lot of 1134 01:00:59.639 --> 01:01:04.000 time out in the wilderness, and like the wilderness, the 1135 01:01:04.039 --> 01:01:08.800 wilderness first aid stuff I've been exposed to is pretty 1136 01:01:08.840 --> 01:01:11.199 much that keep the wet stuff on the inside, dry 1137 01:01:11.199 --> 01:01:12.239 stuff on the outside. 1138 01:01:12.360 --> 01:01:19.519 Okay, so now I'm imagining bear grills. I don't out 1139 01:01:19.599 --> 01:01:22.400 back are you know, frequently known on the internet means 1140 01:01:22.400 --> 01:01:25.880 for drinking his own You're unnecessarily. 1141 01:01:27.440 --> 01:01:29.719 But the likes he got, that's the metric. 1142 01:01:31.119 --> 01:01:32.880 You gotta wonder what he was really going for. Was 1143 01:01:32.920 --> 01:01:35.639 he going for, you know, teaching survival skills or was it, 1144 01:01:35.719 --> 01:01:38.920 you know, just to become a paid influencer. 1145 01:01:39.639 --> 01:01:43.039 I'm pretty sure he's got a mortgage to pay. 1146 01:01:43.880 --> 01:01:48.360 Yeah, so. 1147 01:01:50.239 --> 01:01:54.320 Someone who's interested in pursuing a career in s E. 1148 01:01:55.320 --> 01:01:59.880 Give us like your top rundown pros and cons and 1149 01:02:00.960 --> 01:02:03.760 things you'd wish you things you wish you had known 1150 01:02:04.039 --> 01:02:05.440 before ending up here. 1151 01:02:05.559 --> 01:02:07.599 There's a lot of different paths that you can take 1152 01:02:07.639 --> 01:02:08.400 into s E. 1153 01:02:09.079 --> 01:02:09.599 One thing I. 1154 01:02:09.519 --> 01:02:11.559 Didn't really touch on, and I know we're close on time, 1155 01:02:11.679 --> 01:02:13.280 so I won't go into it too much is like 1156 01:02:13.360 --> 01:02:17.360 a performance engineer. That's very important for an s E 1157 01:02:17.440 --> 01:02:20.239 team to do performance testing all these kinds of things. 1158 01:02:20.559 --> 01:02:23.159 That's a huge part of s E. Is like resource allocation, 1159 01:02:24.039 --> 01:02:28.119 and you need performance testing in place to better make 1160 01:02:28.159 --> 01:02:32.639 those decisions. So performance engineering, like I can say, like 1161 01:02:32.639 --> 01:02:35.320 from my background, like coming in from application engineering, there's 1162 01:02:35.320 --> 01:02:38.719 a lot of value there. Could you understand like how 1163 01:02:38.760 --> 01:02:41.320 these micro how these microserver systems. 1164 01:02:40.920 --> 01:02:41.599 Like work together. 1165 01:02:42.599 --> 01:02:44.800 And then like the systems in Ino your background has 1166 01:02:44.840 --> 01:02:47.320 like an ability to debug at like the network level 1167 01:02:47.360 --> 01:02:50.000 really well and things like that, and all of those 1168 01:02:50.000 --> 01:02:54.039 things come in and offer value to an s R team. 1169 01:02:55.079 --> 01:02:59.840 I guess things I wish I would have known kind 1170 01:02:59.840 --> 01:03:01.800 of like we talked about the beginning of the episode, 1171 01:03:01.840 --> 01:03:06.280 I I'm like very much like go go go, and 1172 01:03:07.039 --> 01:03:09.960 like early in my career, everybody was like, you're gonna burn. 1173 01:03:09.840 --> 01:03:12.199 Out, like slow down. I was like impossible. 1174 01:03:15.119 --> 01:03:18.360 And I still say, like I didn't burn out because 1175 01:03:18.360 --> 01:03:22.400 I still absolutely love it. But I really did not 1176 01:03:23.000 --> 01:03:25.920 realize that like it actually did start to like the 1177 01:03:26.000 --> 01:03:29.199 lack of sleep just didn't start to kind of like 1178 01:03:29.239 --> 01:03:31.840 take a toll on my actual health a little bit. 1179 01:03:33.119 --> 01:03:37.920 So like lesson learned to myself is like, be be 1180 01:03:38.119 --> 01:03:42.880 cognizant of like your personality and like your you know what, 1181 01:03:43.000 --> 01:03:46.079 some of your weaknesses might be there, because like for me, 1182 01:03:46.440 --> 01:03:48.079 I absolutely love what I do and I want to 1183 01:03:48.079 --> 01:03:49.760 be able to do this for a very long time. 1184 01:03:49.920 --> 01:03:52.199 So I kind of came to this realization that I 1185 01:03:52.280 --> 01:03:56.400 was like whoa, like if you don't put it, and management, 1186 01:03:56.480 --> 01:03:58.880 like my manager was fine, like he would encourage me 1187 01:03:58.960 --> 01:04:02.039 to do more of it, not like my director like 1188 01:04:02.159 --> 01:04:05.039 same thing, like dude, like you you need to take 1189 01:04:05.079 --> 01:04:08.920 some time off. So like my advice would be like, 1190 01:04:08.960 --> 01:04:10.960 if you're more on that side of the spectrum, like 1191 01:04:11.079 --> 01:04:15.440 know yourself and allow yourself that time because it will 1192 01:04:15.559 --> 01:04:17.639 make you better in the long run. 1193 01:04:19.119 --> 01:04:20.360 Other things. 1194 01:04:22.559 --> 01:04:25.239 I would just say, like if it sounds interesting to you, 1195 01:04:25.280 --> 01:04:28.280 like don't shy away from it because it is I 1196 01:04:28.280 --> 01:04:32.719 feel like a huge there's like a huge surface of 1197 01:04:33.400 --> 01:04:38.880 areas you can tackle. But if I'm always like of 1198 01:04:38.960 --> 01:04:42.960 the mindset like anybody can learn anything obviously, like some 1199 01:04:43.000 --> 01:04:45.280 people are like more technically talented, they come in with 1200 01:04:45.320 --> 01:04:46.360 more experience. 1201 01:04:46.000 --> 01:04:46.719 Those sorts of things. 1202 01:04:46.719 --> 01:04:48.320 But if it's interesting to you, like and you have 1203 01:04:48.440 --> 01:04:49.119 the drive like. 1204 01:04:49.159 --> 01:04:49.760 Go for it. 1205 01:04:50.639 --> 01:04:53.079 Those would be my biggest like two takeaways and. 1206 01:04:53.079 --> 01:04:55.440 Through the power of MS now you now you two, 1207 01:04:55.519 --> 01:04:58.760 everyone can be an expert in whatever topic you know 1208 01:04:59.440 --> 01:05:03.800 you fancy. I mean listening to you talking to me 1209 01:05:03.840 --> 01:05:05.920 about this, it really makes me think. Like you say, 1210 01:05:06.039 --> 01:05:07.800 you really have to care a lot, But caring a 1211 01:05:07.840 --> 01:05:11.280 lot means understanding the business and gives you an edge 1212 01:05:11.320 --> 01:05:14.039 over everyone else who may not care as much. They 1213 01:05:14.079 --> 01:05:16.360 won't give them a drive to really dive in and 1214 01:05:16.400 --> 01:05:18.480 be able to make the right decision. But then there's 1215 01:05:18.519 --> 01:05:20.480 the question that comes up of like if you care 1216 01:05:20.559 --> 01:05:24.679 too much, you're liable to burn out from doing that, 1217 01:05:25.199 --> 01:05:27.719 and you really need to know yourself. And that's a 1218 01:05:27.840 --> 01:05:30.039 tall order I think for a lot of people. 1219 01:05:30.440 --> 01:05:33.639 Yeah, Like it was very it was a challenge for 1220 01:05:33.719 --> 01:05:38.159 me to be like you can't be the hero twenty 1221 01:05:38.199 --> 01:05:40.960 four to seven or like you are not going to 1222 01:05:41.000 --> 01:05:45.000 be able to like sustain this, but there's. 1223 01:05:44.599 --> 01:05:47.000 Just one more fire over there that you know how 1224 01:05:47.000 --> 01:05:47.559 to deal with. 1225 01:05:48.039 --> 01:05:50.800 Oh lord, Yeah, I mean I guess I would say too, 1226 01:05:50.840 --> 01:05:53.000 Like I'm probably at a place now where the thing 1227 01:05:53.039 --> 01:05:55.119 that I did really enjoy is like I did get 1228 01:05:55.119 --> 01:05:57.719 to serve in more of like an engineering manager role 1229 01:05:57.920 --> 01:06:01.119 on the SR team towards the end my last role. 1230 01:06:01.320 --> 01:06:05.800 And you know, everybody is like driven by different things, 1231 01:06:05.800 --> 01:06:07.519 but you can kind of see people in the org 1232 01:06:07.639 --> 01:06:11.840 that have a lot of potential and starting to kind 1233 01:06:11.880 --> 01:06:13.800 of like mentor them and hand things off to them 1234 01:06:13.840 --> 01:06:14.760 and things like that can. 1235 01:06:14.679 --> 01:06:15.920 Be very rewarding and. 1236 01:06:17.719 --> 01:06:20.079 Really like the best thing you can do is this 1237 01:06:20.199 --> 01:06:22.960 sounds like so hand baby, but like fort multiplier, Like 1238 01:06:23.360 --> 01:06:27.039 you know, whether you're an actual manager or in a 1239 01:06:27.119 --> 01:06:30.679 senior role or anything like that, like what can you 1240 01:06:30.760 --> 01:06:35.599 do to help the company and the teammates like fall 1241 01:06:35.639 --> 01:06:36.639 into the pit of success? 1242 01:06:36.920 --> 01:06:41.119 I like that analogy important leadership skill. 1243 01:06:41.239 --> 01:06:43.440 Learn to delegate, Right, you have to. 1244 01:06:43.400 --> 01:06:45.360 Be okay at a certain point in your career with 1245 01:06:45.400 --> 01:06:47.400 like not being the hero, because if you're in that 1246 01:06:47.519 --> 01:06:51.880 kind of role, like sometimes your value is more so 1247 01:06:52.079 --> 01:06:55.199 in enabling others and building out parts of the system 1248 01:06:55.239 --> 01:06:57.159 that like help others fall into the pit of success. 1249 01:06:57.199 --> 01:06:59.480 And if you kind of like are one of those 1250 01:06:59.480 --> 01:07:05.159 people that feels like they always need like the like recognition, like, 1251 01:07:05.760 --> 01:07:09.039 then like leadership type roles might not be the path 1252 01:07:09.119 --> 01:07:12.360 for you because a lot of the recognition is going 1253 01:07:12.400 --> 01:07:14.800 to be like hidden behind the scenes of enabling others. 1254 01:07:15.119 --> 01:07:17.719 Yeah, but it's warm, cuzzy. You like the girl and he. 1255 01:07:17.719 --> 01:07:18.360 Likes it, so. 1256 01:07:20.519 --> 01:07:22.840 I agree with you. But it's a it's a completely 1257 01:07:22.840 --> 01:07:25.639 different high probably not the best analogy in the world, 1258 01:07:25.679 --> 01:07:28.239 but that's the one that just sits in my head. 1259 01:07:28.840 --> 01:07:31.360 It's more so like I would I guess I would 1260 01:07:31.480 --> 01:07:35.440 almost equate it to like delayed gratification, Like you're not 1261 01:07:35.440 --> 01:07:38.320 going to get the like that a girl in the moment, 1262 01:07:38.440 --> 01:07:40.960 but it might come like two years later when somebody 1263 01:07:40.960 --> 01:07:42.840 that you managed comes back to you and be like, man, 1264 01:07:42.960 --> 01:07:45.800 like you taught me this thing and I used it 1265 01:07:45.840 --> 01:07:47.559 today and thank you, and you're. 1266 01:07:47.400 --> 01:07:49.440 Like really good. 1267 01:07:50.000 --> 01:07:54.360 Right. It's interesting that you say that, though, because that 1268 01:07:54.519 --> 01:07:57.679 actually means that some of the most enjoyment you get 1269 01:07:57.719 --> 01:08:00.280 out of it if that's the right word. Is actually 1270 01:08:00.639 --> 01:08:04.559 seeing others execute effectively and seeing back on what you've 1271 01:08:04.599 --> 01:08:09.800 taught them, which indirectly has no impact on the business 1272 01:08:09.880 --> 01:08:12.000 in a way, especially if they leave and go somewhere else. 1273 01:08:12.000 --> 01:08:15.760 So you know, there is this weird paradox there. 1274 01:08:16.439 --> 01:08:16.720 Yeah. 1275 01:08:16.800 --> 01:08:18.319 Yeah, yeah, it's a little of both. 1276 01:08:18.479 --> 01:08:18.800 Yeah. 1277 01:08:18.840 --> 01:08:21.720 For sure, it's helping the business, and hopefully you help 1278 01:08:21.800 --> 01:08:23.720 other people on the team that help the business or 1279 01:08:23.920 --> 01:08:26.399 just go on and do good things themselves. 1280 01:08:26.880 --> 01:08:28.479 I don't know if this is relevant, but this keeps 1281 01:08:28.520 --> 01:08:31.520 coming to my mind in this conversation. There's this SMBC 1282 01:08:31.680 --> 01:08:36.960 Saturday Morning Breakfast Cereal webcomic issue where Superman's like, how 1283 01:08:36.960 --> 01:08:41.159 can I deliver the most value possible? And you know, 1284 01:08:41.479 --> 01:08:42.800 the mayor of the city is like, well, you know 1285 01:08:42.840 --> 01:08:45.640 what actually saving people? Like you're pretty slow at that. 1286 01:08:45.760 --> 01:08:47.279 If we look at it, you know what would really 1287 01:08:47.319 --> 01:08:50.359 work is if we just had more energy and to 1288 01:08:50.399 --> 01:08:52.239 make this, you know, relevant for twenty twenty five, to 1289 01:08:52.319 --> 01:08:55.479 obviously power all of those lms to help people. And 1290 01:08:55.560 --> 01:08:58.399 so can you just turn this one giant crank really 1291 01:08:58.479 --> 01:09:03.760 fast as you can. That's just your whole life just there, 1292 01:09:03.920 --> 01:09:06.640 And like decades later, he's still turning the crack and like, 1293 01:09:06.680 --> 01:09:09.279 that's not a very rewarding experience, but that's what gives 1294 01:09:09.279 --> 01:09:13.920 the most value to UH civilization. And you know, one day, 1295 01:09:14.199 --> 01:09:17.960 the years later, the mayor wakes up and says, you 1296 01:09:17.960 --> 01:09:21.000 know what, actually, we don't need you anymore. We automated 1297 01:09:21.000 --> 01:09:24.159 your job away with nuclear fusion, and now we generate 1298 01:09:24.199 --> 01:09:27.199 a limited energy all the time, basically without doing anything. 1299 01:09:27.239 --> 01:09:28.159 You can just retire. 1300 01:09:30.119 --> 01:09:33.239 Well, hopefully that doesn't happen you kind of You're always 1301 01:09:33.239 --> 01:09:36.479 wanting to invent new ways to make yourself valuable. 1302 01:09:36.520 --> 01:09:38.079 That's part of being in tech. 1303 01:09:38.279 --> 01:09:40.760 I don't see it's running out anytime soon, which is 1304 01:09:40.840 --> 01:09:44.600 good because I've got I've got a mortgage to cover, So. 1305 01:09:46.039 --> 01:09:48.840 I don't know. I hate to have the dissenting opinion here. 1306 01:09:48.880 --> 01:09:52.479 I feel like there is just people making the same mistakes. 1307 01:09:52.800 --> 01:09:56.920 Yeah that's true too, Yeah, the same, the same underlying mistakes. 1308 01:09:57.000 --> 01:09:58.520 Yes, it's history repeating it. 1309 01:09:58.920 --> 01:10:01.399 Yes, so warning, if you could just compile a list 1310 01:10:01.439 --> 01:10:02.239 of those. 1311 01:10:02.760 --> 01:10:06.039 I tried, so I have tried to, you know, write 1312 01:10:06.079 --> 01:10:09.560 down so what happens frequently. I always thought I'd be 1313 01:10:09.600 --> 01:10:11.199 doing engineering, But what I do is I write blog 1314 01:10:11.239 --> 01:10:14.760 posts and I record podcast episodes, and I get triggered 1315 01:10:14.760 --> 01:10:17.479 sometimes while I'm doing other things in my life, talking 1316 01:10:17.479 --> 01:10:19.920 with my colleagues and whatnot, in the communities that I'm in, 1317 01:10:20.119 --> 01:10:22.279 so much so that I feel compelled to have to 1318 01:10:22.319 --> 01:10:24.680 write something. And so there is a list of things 1319 01:10:24.720 --> 01:10:29.319 out there of problems that I've seen and my recommendations 1320 01:10:29.359 --> 01:10:31.560 for it. But some of them are like really nuanced thing, 1321 01:10:31.680 --> 01:10:34.800 Like there's one particular little thing in security that everyone 1322 01:10:34.880 --> 01:10:37.000 seems to run into, and then there's like a forty 1323 01:10:37.079 --> 01:10:40.520 minute conference talk on it or a blog post on 1324 01:10:40.560 --> 01:10:42.119 it that I'm sure no one has ever read. 1325 01:10:44.199 --> 01:10:46.840 Well, it sounds like a really good point. Put all 1326 01:10:46.840 --> 01:10:50.680 those words together into one coherent sentence in the right order. Right, 1327 01:10:51.079 --> 01:10:53.159 That sounds like a really good time to move on picks, 1328 01:10:53.159 --> 01:10:57.000 and maybe you can point someone to some of those resources. 1329 01:10:57.199 --> 01:11:01.199 Uh, I'm not going to plug blog as my pick, 1330 01:11:01.359 --> 01:11:04.399 but I'm happy to put a link for this episode 1331 01:11:04.760 --> 01:11:07.560 for anyone that wants it. Actually, I'll find the Saturday 1332 01:11:07.560 --> 01:11:12.199 Morning Breakfast Cereal episode two and put that. But my 1333 01:11:12.279 --> 01:11:14.279 pick actually is going to be based of our conversation. 1334 01:11:14.359 --> 01:11:17.119 There is a book by Peter Senge called The Fifth Discipline, 1335 01:11:17.439 --> 01:11:20.319 and it's all about systems thinking. And there's this one 1336 01:11:20.359 --> 01:11:23.880 great example in the book about it's a canonical thing 1337 01:11:23.920 --> 01:11:26.800 that I believe it's taught at I think it's Harvard 1338 01:11:26.840 --> 01:11:30.640 Business School, but also at MIT in the economics theory. 1339 01:11:30.880 --> 01:11:34.920 And that's basically, there is a brewery that makes a 1340 01:11:34.960 --> 01:11:39.960 beer and they make enough for a distributor to sell 1341 01:11:39.960 --> 01:11:42.560 the beer to retailers for people to buy it, and 1342 01:11:42.960 --> 01:11:46.880 every retailer sells like your local store sells two cases 1343 01:11:46.960 --> 01:11:51.000 every week. And then in some random week, a new 1344 01:11:51.079 --> 01:11:54.439 pop song comes out and everyone loves the beer, and 1345 01:11:54.600 --> 01:11:58.000 because the band mentions the beer in the song, and 1346 01:11:58.039 --> 01:12:00.760 from that point on, every single retailer just increases the 1347 01:12:00.840 --> 01:12:04.840 number of beers the case they sell by one to three. 1348 01:12:05.319 --> 01:12:08.640 And this causes some ridiculous effects to happen throughout the 1349 01:12:08.680 --> 01:12:12.520 whole ecosystem. The distributors will fall over, the brewery will 1350 01:12:12.520 --> 01:12:15.319 get some massive orders, and basically what ends up happening 1351 01:12:15.319 --> 01:12:18.359 as lots of people will be fired for failing to 1352 01:12:18.399 --> 01:12:21.479 deliver effectively. And it's just such a small change to happen, 1353 01:12:21.520 --> 01:12:23.159 and you can see huge impacts, and I feel like 1354 01:12:23.199 --> 01:12:26.359 it's a really interesting case study of what happens over 1355 01:12:26.399 --> 01:12:29.800 and over again, and lots of organizations and relates out 1356 01:12:29.880 --> 01:12:32.439 to a lot what Amy was talking about in today's episodes. 1357 01:12:32.600 --> 01:12:34.439 Those relevant. 1358 01:12:34.960 --> 01:12:40.159 That's pretty cool. That's called the fifth dimension discipline, fifth discipline, gotcha, 1359 01:12:40.319 --> 01:12:44.600 re Peter sin get yeah? Right on, Jillian, what about you? 1360 01:12:44.680 --> 01:12:45.560 What'd you bring for a pick? 1361 01:12:45.760 --> 01:12:47.359 I don't know. I was going to pick. I'm giving 1362 01:12:47.359 --> 01:12:50.840 a talk next week on HPC, And now it occurs 1363 01:12:50.880 --> 01:12:52.199 to me that I don't know if that's going to 1364 01:12:52.239 --> 01:12:55.680 be like recorded and put any place, So that's maybe 1365 01:12:55.720 --> 01:12:59.680 not my pick. I don't know, like Horizons zero down 1366 01:12:59.800 --> 01:13:03.159 the video game I've been playing, and maybe the talk. 1367 01:13:03.239 --> 01:13:05.840 Maybe the talk too. If it's actually published and recorded, 1368 01:13:06.159 --> 01:13:06.680 we'll see. 1369 01:13:06.960 --> 01:13:09.880 You can always record it yourself on your laptop and 1370 01:13:09.920 --> 01:13:12.800 then you'll have the transcript, you can publish the slides, 1371 01:13:12.840 --> 01:13:15.199 and then you can you know, release it to all 1372 01:13:15.239 --> 01:13:16.520 your faithful audience there. 1373 01:13:16.880 --> 01:13:18.960 Yeah, that's true. I mean I could just help somebody 1374 01:13:19.000 --> 01:13:20.800 record it on my phone, right, Like, I don't know. 1375 01:13:21.119 --> 01:13:24.119 Okay, so we're gonna count on seeing your talk. 1376 01:13:24.000 --> 01:13:26.279 Then, yeah, sure, that's the point. It's supposed to be 1377 01:13:26.279 --> 01:13:30.520 self promotion, like, right, that is what I do. 1378 01:13:31.279 --> 01:13:33.439 Amy. I know you've done this before. What do you 1379 01:13:33.479 --> 01:13:34.640 have for a pick this week? 1380 01:13:35.000 --> 01:13:39.720 See I was debating if I should pick something technical 1381 01:13:39.880 --> 01:13:43.079 or non technical. I guess I will try to break 1382 01:13:43.079 --> 01:13:45.000 it up a little bit. I'm going to pick a 1383 01:13:45.000 --> 01:13:48.520 band that I am like absolutely obsessed with and I 1384 01:13:48.560 --> 01:13:51.159 can't shut up about it, And because other people won't 1385 01:13:51.159 --> 01:13:52.760 listen to me anymore about it, this is a great 1386 01:13:52.760 --> 01:13:57.119 opportunity to share it. My husband is really tired to 1387 01:13:57.159 --> 01:14:00.159 be talking about them and blasting them, so and that 1388 01:14:00.279 --> 01:14:02.119 is sleep token. I don't know if anybody who's ever 1389 01:14:02.159 --> 01:14:04.600 heard of it or heard of them, but I will 1390 01:14:04.680 --> 01:14:09.199 drop a link. I first heard about them like maybe 1391 01:14:09.319 --> 01:14:11.319 like two or three years ago, and they have a 1392 01:14:11.319 --> 01:14:14.600 bunch of new songs out this year, and I am, yeah, 1393 01:14:14.800 --> 01:14:18.039 it's when I've had like a long, stressful day at work. 1394 01:14:18.079 --> 01:14:19.760 I don't know why, Like I listened to them and 1395 01:14:19.840 --> 01:14:23.279 it's like peaceful to me to fall asleep toobe. So whatever, 1396 01:14:23.680 --> 01:14:25.079 maybe other people will like it too. 1397 01:14:25.239 --> 01:14:26.039 What's the genre? 1398 01:14:27.680 --> 01:14:28.520 That's the thing? 1399 01:14:28.760 --> 01:14:31.920 So they are like they're all over the place, like 1400 01:14:32.159 --> 01:14:36.600 I guess traditionally more like metal. They're almost like a 1401 01:14:36.640 --> 01:14:40.479 little bit like ed M, a little bit metal, like honestly, 1402 01:14:40.520 --> 01:14:43.159 like a tiny bit hip hop something like they're just 1403 01:14:43.199 --> 01:14:46.399 like you listen to your song and you're like, oh, 1404 01:14:46.520 --> 01:14:49.439 this is like really relaxing, and then they start screaming 1405 01:14:49.520 --> 01:14:51.600 and then there and then it's like back like a 1406 01:14:51.680 --> 01:14:54.359 hip hop's kind of vibe, and I'm just like, what's happening. 1407 01:14:56.119 --> 01:14:57.800 It's just like you're gonna say, like, yeah, it's like 1408 01:14:57.880 --> 01:15:01.640 it's like heiden or Dominate the Christmas Donkey. I just 1409 01:15:01.720 --> 01:15:03.680 use that music to falls leap that night. I'm just 1410 01:15:03.720 --> 01:15:07.720 like really like, not my personal choice, but uh, you know, 1411 01:15:07.720 --> 01:15:10.039 if it works for you, that's pretty interesting. 1412 01:15:10.079 --> 01:15:14.800 I like metal, so why I absolutely love them, So yeah, 1413 01:15:15.000 --> 01:15:16.479 that's that's going to be my pick. 1414 01:15:17.479 --> 01:15:19.600 That's awesome. I was just as you were saying it 1415 01:15:19.640 --> 01:15:21.760 and you're like, oh, it's so just what I want 1416 01:15:21.760 --> 01:15:23.920 to listen to after a long, stressful day. I was like, oh, 1417 01:15:24.000 --> 01:15:25.760 this would be so great if it's a metal band, 1418 01:15:25.880 --> 01:15:27.880 and then you're like a surprise, guess. 1419 01:15:27.600 --> 01:15:31.600 What I mean. They're not metal in the sense of 1420 01:15:31.680 --> 01:15:34.640 like like I really like this band Motionless and White. 1421 01:15:34.680 --> 01:15:36.960 If people have heard of them, they're not metal in 1422 01:15:36.960 --> 01:15:39.520 that sense. They're like a little more chill than that. 1423 01:15:39.680 --> 01:15:44.720 But yeah, right, on, I'll drop a link before. 1424 01:15:44.479 --> 01:15:45.039 We hang up. 1425 01:15:45.399 --> 01:15:50.880 Cool. Cool, all right, So my pick I switched it 1426 01:15:50.960 --> 01:15:54.720 since we started recording because it's now a relevant topic. 1427 01:15:55.359 --> 01:16:02.119 On Netflix, there is a series called The bear Grills 1428 01:16:02.199 --> 01:16:07.439 Celebrity Hunt, and so it's all of these celebrities who 1429 01:16:07.439 --> 01:16:10.960 are taken to Costa Rica and then they compete, and 1430 01:16:11.000 --> 01:16:13.600 the losers of the competition have to go into this 1431 01:16:13.840 --> 01:16:19.520 area that's like walled off and they have to survive 1432 01:16:19.760 --> 01:16:22.800 in there for one hour while bear Grills is hunting them. 1433 01:16:22.880 --> 01:16:26.520 So they have an hour to either escape the bear 1434 01:16:26.560 --> 01:16:30.479 pit or get caught by bear Grills. And it was 1435 01:16:30.520 --> 01:16:34.000 actually just super fun to watch. And I brought it 1436 01:16:34.079 --> 01:16:37.039 up at work saying, Hey, for our next off site, 1437 01:16:37.159 --> 01:16:39.079 we should go to the bear pit and have bear 1438 01:16:39.159 --> 01:16:40.720 Grills hunt us down. 1439 01:16:40.720 --> 01:16:43.800 Where bear grills will actually be well buttoned. 1440 01:16:44.680 --> 01:16:45.439 I'd be cool with that. 1441 01:16:45.560 --> 01:16:47.920 Yeah, I feel like that may create some sort of 1442 01:16:48.560 --> 01:16:51.640 psychological safety issues for coming back work later. 1443 01:16:52.079 --> 01:16:54.399 Yeah, there's probably a downside to that, but let's not 1444 01:16:54.439 --> 01:16:55.319 focus on that. 1445 01:16:56.720 --> 01:16:57.319 Makes me think. 1446 01:16:57.399 --> 01:17:00.199 My husband, Like, at first I thought I was like, 1447 01:17:00.199 --> 01:17:03.239 what are you watching? But he watches this like it's 1448 01:17:03.279 --> 01:17:05.439 probably like a famous YouTuber people probably know I can't 1449 01:17:05.479 --> 01:17:07.279 think of him, but like he's always out in the 1450 01:17:07.319 --> 01:17:10.279 wilderness like doing things, and my husband loves watching him. 1451 01:17:10.279 --> 01:17:12.439 And initially I was like, this is crazy, but now 1452 01:17:12.479 --> 01:17:13.279 I kind of like it too. 1453 01:17:14.119 --> 01:17:16.720 All Right, he's usually in Alaska. 1454 01:17:17.479 --> 01:17:19.439 There have been a lot of those episodes where my 1455 01:17:19.520 --> 01:17:22.479 kids were watching something like that's stupid. Why are you 1456 01:17:22.520 --> 01:17:24.359 watching wasting your time on that? You know, like two 1457 01:17:24.359 --> 01:17:26.600 hours later, I'm still sitting next to him watching it. 1458 01:17:27.239 --> 01:17:29.880 But anyway, Amy, thanks for being on the show. This 1459 01:17:30.079 --> 01:17:32.760 was a fun chat. Thank you for your insights and 1460 01:17:32.800 --> 01:17:37.039 your expertise and sharing your thoughts on sre Warren Chillian. 1461 01:17:37.159 --> 01:17:39.800 Thank you both for jumping in and co hosting today. 1462 01:17:39.840 --> 01:17:43.720 It's good to see both and for all our listeners, 1463 01:17:43.800 --> 01:17:45.039 thanks for listening.