1
00:00:07,839 --> 00:00:10,000
Speaker 1: So, Warren, what's been your experience with SRI.

2
00:00:10,720 --> 00:00:14,359
Speaker 2: That's quite the way of just jumping into the episode,

3
00:00:14,400 --> 00:00:17,239
and I was so unprepared because I'm not even the

4
00:00:17,600 --> 00:00:20,160
guest for today's episode, So I don't I don't have

5
00:00:20,199 --> 00:00:21,399
an answer to that question.

6
00:00:21,199 --> 00:00:25,120
Speaker 1: Honestly, Yeah, for sure, Jillian, And what about your thoughts

7
00:00:25,120 --> 00:00:27,280
on SRA. You're just like, you're a contract not just

8
00:00:27,320 --> 00:00:29,679
a contractor. You're a contractor. You just light stuff on

9
00:00:29,760 --> 00:00:31,960
fire and then if they want to put out you

10
00:00:32,000 --> 00:00:33,079
bill them for it, right.

11
00:00:33,479 --> 00:00:35,079
Speaker 3: I mean really like, I just need to make sure

12
00:00:35,119 --> 00:00:38,320
data doesn't get deleted because that's like a problem that

13
00:00:38,399 --> 00:00:41,079
is that is a problem, Okay, And that's really all

14
00:00:41,079 --> 00:00:41,880
that I worry about.

15
00:00:42,920 --> 00:00:45,759
Speaker 1: It's a security reliability because you can't be exploited if

16
00:00:45,759 --> 00:00:47,240
there's no data left.

17
00:00:47,759 --> 00:00:49,119
Speaker 3: I mean, that's true.

18
00:00:49,679 --> 00:00:51,320
Speaker 2: I will say that there is. There is a book

19
00:00:51,359 --> 00:00:55,479
out there published by one of the hyperscalers. I believe

20
00:00:56,560 --> 00:00:59,240
that attempts to define it in their own terms, but

21
00:00:59,320 --> 00:01:01,280
I feel like, like a lot of things they do,

22
00:01:01,479 --> 00:01:06,040
it's used as a introduction into the topic rather than

23
00:01:06,920 --> 00:01:09,879
used as experts. So I'm really interested to hear what

24
00:01:10,000 --> 00:01:13,680
our guests today will be sharing with us.

25
00:01:14,079 --> 00:01:17,239
Speaker 1: Well, yeah, which is a great segue because clearly we

26
00:01:17,319 --> 00:01:23,040
need some expert opinions on SRE, and so we've got

27
00:01:23,200 --> 00:01:25,599
Amy Night with us. Amy, Welcome to the show.

28
00:01:26,640 --> 00:01:27,439
Speaker 4: Happy to be here.

29
00:01:27,959 --> 00:01:30,879
Speaker 1: I'm excited to have you here. So tell us a

30
00:01:30,920 --> 00:01:36,319
little bit about your background and how you got to SRE.

31
00:01:36,920 --> 00:01:37,159
Speaker 5: Yeah.

32
00:01:37,280 --> 00:01:39,040
Speaker 6: I love SRI.

33
00:01:40,319 --> 00:01:43,359
Speaker 4: It's a little bit of a love hate relationship. It

34
00:01:43,400 --> 00:01:46,640
can be very stressful at times, but I also love it.

35
00:01:46,719 --> 00:01:50,480
I feel like it is kind of when I when

36
00:01:50,480 --> 00:01:52,519
I and this, I'll get into kind of like my

37
00:01:52,599 --> 00:01:56,840
path into SRE, and it just it really suits. I

38
00:01:56,879 --> 00:02:00,920
feel like my engineering brain perfectly and the high level

39
00:02:00,920 --> 00:02:05,599
there is. I come from non traditional background, so self taught,

40
00:02:05,719 --> 00:02:09,280
did the whole boot camp thing twelve years ago now,

41
00:02:09,439 --> 00:02:11,080
which is insane to me.

42
00:02:11,360 --> 00:02:14,639
Speaker 5: I'm really old, or feel really old.

43
00:02:16,439 --> 00:02:19,120
Speaker 4: Like I saw someone on Twitter the other day talk

44
00:02:19,120 --> 00:02:21,680
about like, what did you use before GET and I

45
00:02:21,800 --> 00:02:24,080
was like, oh, my gosh, I'm really old now because

46
00:02:24,080 --> 00:02:26,280
my first job we use material, which I guess is

47
00:02:26,360 --> 00:02:29,280
it but at the same time like it.

48
00:02:29,759 --> 00:02:30,039
Speaker 6: Yeah.

49
00:02:30,199 --> 00:02:36,759
Speaker 4: So But anyways, so backing up, and I used to

50
00:02:36,800 --> 00:02:39,840
do JavaScript jobber, so I kind of started off in

51
00:02:39,840 --> 00:02:43,879
the JavaScript Land, did node.

52
00:02:43,680 --> 00:02:46,439
Speaker 6: And all the front end.

53
00:02:47,000 --> 00:02:48,639
Speaker 4: I kind of focused heavily on front end for a

54
00:02:48,680 --> 00:02:53,759
little while, did note as well, and then I think

55
00:02:53,800 --> 00:02:58,800
it was like twenty eighteen twenty nineteen, I went to

56
00:02:58,960 --> 00:03:01,360
work at NPM, which was like my dream job at

57
00:03:01,360 --> 00:03:04,680
the time until they got acquired, and you know, it's

58
00:03:04,800 --> 00:03:06,319
just a little bit of a bummer because it was

59
00:03:06,400 --> 00:03:09,639
very short lived. But when I was there, we did

60
00:03:09,719 --> 00:03:13,360
cross functional teams and my manager at the time was like,

61
00:03:13,599 --> 00:03:17,280
you seem really interested in like this platform s re

62
00:03:17,439 --> 00:03:23,639
work infrastructure. We can probably hire for a web developer

63
00:03:24,039 --> 00:03:25,680
more easily than we can hire for that role. Do

64
00:03:25,680 --> 00:03:27,039
you want to go over to that team? And I

65
00:03:27,120 --> 00:03:28,879
was like, yes, sign me out, like you want to,

66
00:03:29,479 --> 00:03:32,319
you know, if I have the opportunity to learn a

67
00:03:32,360 --> 00:03:37,360
new skill on the job, Like yes. So I did

68
00:03:37,479 --> 00:03:41,680
SRE at MPM for a while, and then I did

69
00:03:41,879 --> 00:03:45,280
most of my sr rework at Paramount Global, which is

70
00:03:46,280 --> 00:03:51,520
like the streaming application, and did two Super Bowls there well,

71
00:03:52,759 --> 00:03:56,960
the last last year's Super Bowl. And so that's kind

72
00:03:56,960 --> 00:03:59,639
of my journey into SRE And why I say it

73
00:03:59,719 --> 00:04:02,919
is like I feel like SRI is really like my

74
00:04:02,919 --> 00:04:07,479
my perfect niche in engineering and that is because coming

75
00:04:07,479 --> 00:04:10,759
from this non traditional background, I've always been the like

76
00:04:11,159 --> 00:04:15,120
the programmer who is like, sure that new tech is

77
00:04:15,159 --> 00:04:17,720
really fun and interesting, but like, shouldn't that be something

78
00:04:17,720 --> 00:04:20,439
we're working on, like on the weekends on our own time,

79
00:04:20,560 --> 00:04:24,040
Like isn't our job? Like I hate saying this because

80
00:04:24,079 --> 00:04:27,000
people hate me, but at the end of the day,

81
00:04:27,199 --> 00:04:30,160
like I'm just like I feel very fortunate to have

82
00:04:30,199 --> 00:04:32,439
a career in tech, and in the back of my mind,

83
00:04:32,480 --> 00:04:35,439
I'm always like my job, whether I like it or not,

84
00:04:36,199 --> 00:04:39,000
Like if I'm working hours, my job is to make

85
00:04:39,040 --> 00:04:44,079
this company profitable. And while I love programming for fun

86
00:04:44,160 --> 00:04:47,160
on my own, like, I have a responsibility to the

87
00:04:47,199 --> 00:04:50,720
business to make make the business profitable, keep it alive.

88
00:04:50,720 --> 00:04:52,759
Speaker 6: Work doing a lot of startups, so I.

89
00:04:52,720 --> 00:04:56,519
Speaker 4: Understand this, and that's where SRI really fits in because

90
00:04:56,560 --> 00:04:59,240
it's like a lot of roles. Yes, you're accountable to

91
00:04:59,279 --> 00:05:01,360
the business, but I feel like in an SR role

92
00:05:02,480 --> 00:05:05,879
through we can get into like slas, slis SLOs, but

93
00:05:05,920 --> 00:05:10,519
the slas like you are accountable to the business stakeholders

94
00:05:10,560 --> 00:05:15,160
and that is your overarching goal. So basically, in a nutshell,

95
00:05:15,399 --> 00:05:19,360
SA site reliability engineering. I also call it like production

96
00:05:19,519 --> 00:05:24,360
readiness engineering. It is your task to keep the application

97
00:05:24,879 --> 00:05:30,759
running optimally for as as much up time as possible.

98
00:05:31,439 --> 00:05:34,839
Speaker 2: So what I heard was that you at MPM had

99
00:05:35,600 --> 00:05:37,600
so much work and we're doing such a great job

100
00:05:37,600 --> 00:05:41,839
that they're like, here's more work. Be responsible for the

101
00:05:41,879 --> 00:05:45,480
success of the business and not just solving tickets and

102
00:05:45,560 --> 00:05:46,759
pushing out futures.

103
00:05:47,920 --> 00:05:51,399
Speaker 4: No, but honestly, truly, like my manager there, he was

104
00:05:51,560 --> 00:05:54,399
absolutely amazing, as were a lot of the people that

105
00:05:54,439 --> 00:06:00,399
I worked with, absolutely brilliant, talented people. And I don't know,

106
00:06:00,519 --> 00:06:04,519
I'm like distributed systems nerd. Like on the weekends, I

107
00:06:04,680 --> 00:06:08,000
like to watch YouTube videos of distributed systems interviews, Like

108
00:06:08,040 --> 00:06:10,920
I don't know why, this is very interesting to me,

109
00:06:12,600 --> 00:06:16,000
but I've always like just gravitated towards that sort of thing.

110
00:06:16,199 --> 00:06:18,600
Speaker 2: So you don't have to shame yourself for that. I

111
00:06:18,600 --> 00:06:22,000
think lots of us on the weekend, Yeah, we're watching

112
00:06:22,079 --> 00:06:24,639
interviews and other tech things. Yeah, for sure.

113
00:06:25,120 --> 00:06:27,240
Speaker 4: Whenever there's like a big outage of like a big

114
00:06:27,279 --> 00:06:29,920
tech company, I'm always like reading the post mortems and

115
00:06:29,959 --> 00:06:31,800
trying to talk to the like talk about them with

116
00:06:31,879 --> 00:06:35,519
my husband, and he was he's just like he tries

117
00:06:35,560 --> 00:06:39,040
to act interested, but he's just good for He's like,

118
00:06:39,120 --> 00:06:42,120
I love how nerdy you are, Like, okay, I'll just

119
00:06:42,240 --> 00:06:43,560
talk to myself that.

120
00:06:45,160 --> 00:06:48,160
Speaker 1: You've got a really I think that's a super interesting

121
00:06:48,279 --> 00:06:51,959
point where you talked about you were working for working

122
00:06:52,000 --> 00:06:55,439
out on JavaScript development and then you went over to SRE,

123
00:06:55,680 --> 00:06:59,319
And I think that's really important skill to have for

124
00:06:59,399 --> 00:07:03,199
that because, like as the SRE, like it's not just

125
00:07:03,279 --> 00:07:07,480
about the infrastructure or the data dog metrics or whatever,

126
00:07:07,519 --> 00:07:11,000
like a lot of times you've got to dig into

127
00:07:11,040 --> 00:07:15,680
the code and discover like what's actually going on here?

128
00:07:16,360 --> 00:07:20,519
And so how how do you feel like those web

129
00:07:20,519 --> 00:07:23,240
development skills set you up for success as an SRE.

130
00:07:23,680 --> 00:07:27,000
Speaker 4: Yeah, that's a really good question, and I it's a

131
00:07:27,040 --> 00:07:29,839
common question I get from people who are also interested

132
00:07:29,879 --> 00:07:34,319
in SRI because and I also feel like, you know,

133
00:07:35,519 --> 00:07:40,360
backing up a little bit SRE. I know, Warren mentioned

134
00:07:40,399 --> 00:07:42,360
like the Google sr book, but I also feel like

135
00:07:42,399 --> 00:07:46,560
this focus on SRI really came about via micro services buzzword.

136
00:07:46,639 --> 00:07:48,079
I hate saying them, but.

137
00:07:48,040 --> 00:07:48,680
Speaker 1: It is what it is.

138
00:07:51,360 --> 00:07:57,639
Speaker 4: But so to kind of answer your question there, I

139
00:07:57,720 --> 00:08:01,959
feel like the teams that I've been on not necessarily MPM,

140
00:08:02,079 --> 00:08:07,000
but paramount specifically, they had a lot of like systems engineers.

141
00:08:07,040 --> 00:08:09,439
Like the team was very system engineer heavy, so a

142
00:08:09,439 --> 00:08:11,519
lot of these people who like worked in data centers

143
00:08:12,160 --> 00:08:15,040
and those are all obviously very valuable skills as an SR,

144
00:08:16,480 --> 00:08:20,720
but where they were really lacking was like the programming

145
00:08:20,759 --> 00:08:23,680
side of things application development side of things. And I

146
00:08:23,720 --> 00:08:25,959
feel like to have a really good SRA team, you

147
00:08:26,000 --> 00:08:28,879
do need a mixture of both and you can really

148
00:08:28,920 --> 00:08:32,679
like learn from each other. So in the SR, like

149
00:08:32,759 --> 00:08:35,159
in SR roles that I've been in, I am doing

150
00:08:35,200 --> 00:08:39,960
some programming nuts. I mean, you need to be able

151
00:08:39,960 --> 00:08:42,679
to debug the program. You're also doing some programming in

152
00:08:43,600 --> 00:08:48,240
you're doing like scripting. A lot of SR, I don't

153
00:08:48,440 --> 00:08:51,159
think people realize is a lot of like cashing at

154
00:08:51,159 --> 00:08:53,879
the edge and so you're doing a lot of custom

155
00:08:53,960 --> 00:08:57,600
logic that gets deployed to your CDNs and things like that.

156
00:08:58,320 --> 00:09:01,519
So it might not be as intensive as like building

157
00:09:01,559 --> 00:09:03,600
out a future and stuff like that, you still need

158
00:09:03,639 --> 00:09:07,240
those programming concepts to write that kind of code and

159
00:09:07,240 --> 00:09:11,279
then also to work with the developers because you know,

160
00:09:11,360 --> 00:09:13,559
for better or worse, there is like that's like the

161
00:09:14,519 --> 00:09:16,840
that's the telltale like thing that people say about SRI

162
00:09:17,039 --> 00:09:19,639
is like that push and pull between the engineering department

163
00:09:19,679 --> 00:09:21,679
because they have a vested interest in getting their stuff

164
00:09:21,720 --> 00:09:24,799
out to production and then SRI has a vested interest

165
00:09:25,000 --> 00:09:28,000
in keeping the system healthy and functioning.

166
00:09:28,879 --> 00:09:31,399
Speaker 2: Yeah for sure. I mean you did mention that it

167
00:09:31,679 --> 00:09:34,159
could be different in different places. So like from your experience,

168
00:09:34,200 --> 00:09:37,440
what has being an SRI really met? Like is there

169
00:09:37,480 --> 00:09:39,799
a standard day? And I know that's a terrible question

170
00:09:39,879 --> 00:09:42,600
for asking you know, any sort of engineering technical person,

171
00:09:43,120 --> 00:09:45,519
but you know some some expectations around what the role

172
00:09:45,600 --> 00:09:48,120
is that you were doing both at MPM and Paramount.

173
00:09:48,639 --> 00:09:48,960
Speaker 6: Yeah.

174
00:09:49,000 --> 00:09:52,039
Speaker 4: So MPM obviously like a very it was a relatively

175
00:09:52,080 --> 00:09:56,759
small team. We just we had the SR team and

176
00:09:57,159 --> 00:09:58,759
there was a lot of like I said, it was

177
00:09:58,759 --> 00:10:01,799
cross functional, so we're with building out features, but a

178
00:10:01,879 --> 00:10:04,240
team was like a cross functional team where we had

179
00:10:04,240 --> 00:10:08,720
a platform engineer, we had SR infrastructure engineer, we had

180
00:10:08,840 --> 00:10:13,720
a application developer, and all working together to get something

181
00:10:13,799 --> 00:10:16,000
out to production. And like the SR side of things,

182
00:10:16,039 --> 00:10:19,639
there would be like I was saying, I talk about

183
00:10:19,759 --> 00:10:23,120
SR in terms of like production readiness, so making sure

184
00:10:23,200 --> 00:10:27,559
that this feature is ready for production, if there are

185
00:10:27,639 --> 00:10:30,799
like dashboards that need to be made, making sure that

186
00:10:30,840 --> 00:10:33,200
those are in place so that we can properly monitor

187
00:10:33,559 --> 00:10:37,960
when it goes to production, making sure that you have

188
00:10:38,240 --> 00:10:41,080
the proper resources for the machines that it's going to

189
00:10:41,080 --> 00:10:46,840
be running on, things like that. At Paramount, you know,

190
00:10:46,879 --> 00:10:50,240
this is a much larger MPM, has a very large scale,

191
00:10:50,240 --> 00:10:55,679
but it was extremely bursty. And when I say like

192
00:10:55,759 --> 00:11:00,279
extremely bursty, like the traffic would spike, but it would

193
00:11:00,279 --> 00:11:06,039
be very short lived. At Paramount, similar but not quite

194
00:11:06,039 --> 00:11:09,559
to that extent like birsty, as in like you're going

195
00:11:09,600 --> 00:11:11,799
to see a high rush of traffic when a large

196
00:11:11,799 --> 00:11:17,039
event kicks off, It'll stay relatively high, but it kind

197
00:11:17,080 --> 00:11:19,039
of like starts to taper off as the events go

198
00:11:19,159 --> 00:11:23,559
down and things like that. But Paramount SRA like a

199
00:11:24,480 --> 00:11:30,159
larger SR teams, it's usually segmented into different departments. So

200
00:11:30,240 --> 00:11:33,960
you have like an infrastructure focused department, you would have

201
00:11:34,000 --> 00:11:36,519
like a devsec ops because there's a lot of security

202
00:11:36,519 --> 00:11:39,279
that goes into SRE, and you would.

203
00:11:39,039 --> 00:11:41,080
Speaker 6: Have like an observability monitoring team.

204
00:11:41,120 --> 00:11:44,879
Speaker 4: I feel like observability and monitoring is where you hear

205
00:11:44,919 --> 00:11:48,200
a lot of like focus on SRE for sure, but

206
00:11:48,279 --> 00:11:51,159
it is definitely like segmented into different teams, and it

207
00:11:51,240 --> 00:11:53,399
is very different at different places, so.

208
00:11:53,360 --> 00:11:54,759
Speaker 2: You spend all your time looking at graphs.

209
00:11:54,799 --> 00:11:58,360
Speaker 4: That's what I when I was more on the infrastructure

210
00:11:58,399 --> 00:12:00,759
side of things. But yes, like we were heavily with

211
00:12:02,519 --> 00:12:05,840
like observability team to make sure that the dashboards are

212
00:12:05,879 --> 00:12:08,120
in place and they have the data that you need

213
00:12:08,159 --> 00:12:09,519
and everything sod that properly.

214
00:12:09,960 --> 00:12:13,240
Speaker 2: That sounds like the go between, like when get the

215
00:12:13,240 --> 00:12:16,759
application metrics into some place that different team owns, and

216
00:12:16,799 --> 00:12:19,519
then when that team exposes a problem, you're then have

217
00:12:19,600 --> 00:12:21,919
to ferry that back to the application team to tell

218
00:12:21,960 --> 00:12:24,360
them about the issue that they caused in production.

219
00:12:25,879 --> 00:12:27,039
Speaker 5: Yeah, and that's.

220
00:12:26,840 --> 00:12:30,799
Speaker 4: Where like what Forrest was saying, it's complicated what.

221
00:12:32,679 --> 00:12:33,080
Speaker 3: So? Uh?

222
00:12:33,200 --> 00:12:35,440
Speaker 2: Will and I had a little joke going on before

223
00:12:35,519 --> 00:12:38,559
the stream started, we forgot to end the joke, and

224
00:12:39,120 --> 00:12:42,200
the joke the joke basically ended where I decided to

225
00:12:42,240 --> 00:12:46,039
change his moniker in the in the stream that uh

226
00:12:46,279 --> 00:12:49,799
that that amy only only you can stream only you

227
00:12:49,799 --> 00:12:51,679
can see no one else, like none of the none,

228
00:12:51,919 --> 00:12:54,360
none of the audience will be able to see that. Will.

229
00:12:54,440 --> 00:12:56,360
Speaker 5: It's Will, I'm so sorry.

230
00:12:56,600 --> 00:13:02,360
Speaker 1: No, it's hilarious. It's great because well played Warren, well played.

231
00:13:04,600 --> 00:13:05,799
Speaker 6: You look like a forest.

232
00:13:05,879 --> 00:13:08,159
Speaker 1: But well, that is how the joke started, is because

233
00:13:08,440 --> 00:13:10,399
like Warren is asking me something and I was like, dude,

234
00:13:10,399 --> 00:13:13,000
I don't know. I just feel like Forrest Gump because

235
00:13:13,039 --> 00:13:15,919
I just wander around places in my life and great

236
00:13:15,919 --> 00:13:17,759
shit happens. And then people were like, how did you

237
00:13:17,799 --> 00:13:19,320
do that? And like I don't know. I was just

238
00:13:19,360 --> 00:13:20,159
standing there.

239
00:13:21,519 --> 00:13:26,320
Speaker 4: Okay, Will, Will, I'm so sorry. I feel you got

240
00:13:26,360 --> 00:13:29,720
me good, thank you. And now I've lost my train

241
00:13:29,759 --> 00:13:30,279
of thought.

242
00:13:30,120 --> 00:13:30,639
Speaker 6: A little bit.

243
00:13:30,759 --> 00:13:34,399
Speaker 4: But too, I think what I was about to say

244
00:13:34,639 --> 00:13:38,360
is like there's a fine line because obviously it doesn't

245
00:13:38,399 --> 00:13:41,399
make sense for the SRI to be the person that

246
00:13:42,080 --> 00:13:47,200
debugs the code, but in a perfect scenario, ultimately, like

247
00:13:47,279 --> 00:13:51,679
the sort of thing that probably should happen is SRI is.

248
00:13:51,759 --> 00:13:55,440
Usually sometimes you have like a knock team who's kind

249
00:13:55,440 --> 00:13:57,159
of like the first line of defense that does like

250
00:13:57,399 --> 00:13:59,720
a smoke test, like, okay, we got an alert, we

251
00:13:59,759 --> 00:14:00,519
got paged?

252
00:14:00,879 --> 00:14:03,200
Speaker 6: Is the application? Can I reach it?

253
00:14:03,679 --> 00:14:03,759
Speaker 4: Like?

254
00:14:03,799 --> 00:14:05,919
Speaker 5: Can I just go to my browser and reach it?

255
00:14:06,000 --> 00:14:06,840
Speaker 6: Like is it up?

256
00:14:07,320 --> 00:14:08,519
Speaker 5: Is it hard down?

257
00:14:08,840 --> 00:14:10,320
Speaker 6: Or is it is latency?

258
00:14:10,679 --> 00:14:13,039
Speaker 4: You know, starting to go down that sort of thing,

259
00:14:13,080 --> 00:14:15,320
And the knock team is usually the first line of

260
00:14:15,320 --> 00:14:19,480
defense there. If the KNOP team deems it like okay,

261
00:14:19,600 --> 00:14:22,480
something is awry here, then they'll usually.

262
00:14:22,360 --> 00:14:23,399
Speaker 6: Page SRI team.

263
00:14:24,240 --> 00:14:27,240
Speaker 4: So SRI team kind of usually you'll have like playbooks

264
00:14:27,240 --> 00:14:29,240
in place to do like a sanity check of like

265
00:14:29,559 --> 00:14:32,080
jump on one of the Kopernettes pods, you know, kind

266
00:14:32,080 --> 00:14:36,559
of see what's going on? Are these pods airing something

267
00:14:36,600 --> 00:14:39,519
like that. But ultimately what should happen is there should

268
00:14:39,840 --> 00:14:42,639
in my opinion anyways like a close relationship with SRI

269
00:14:43,360 --> 00:14:48,000
and the development team that it's clear like we have

270
00:14:48,080 --> 00:14:51,240
an environment in this micro services system where we can

271
00:14:51,279 --> 00:14:56,000
do a safe roll back, and so what typically would

272
00:14:56,039 --> 00:14:59,639
happen at least in my opinion, my role like I

273
00:14:59,679 --> 00:15:03,759
would deem like okay, and I have a rough understanding

274
00:15:03,799 --> 00:15:06,200
of like what the feature is, what's the impact because

275
00:15:06,240 --> 00:15:10,080
there could be a business impact of rolling back. Ultimately

276
00:15:10,120 --> 00:15:13,840
we want to make that would when I say business impact,

277
00:15:13,960 --> 00:15:18,039
like you have some sort of like feature for Black

278
00:15:18,039 --> 00:15:22,080
Friday and it's peak traffic and this feature is like

279
00:15:22,759 --> 00:15:25,759
driving revenue, so there's like a but it's also like

280
00:15:25,840 --> 00:15:27,960
causing issues for customers, So you need to kind of

281
00:15:27,960 --> 00:15:29,799
make the decision. And this is where the stress gets

282
00:15:29,840 --> 00:15:34,440
high of like what's the financial trade off ultimately here

283
00:15:34,600 --> 00:15:37,639
like to this business person, like is it honestly like

284
00:15:37,679 --> 00:15:39,320
it will bubble all the way up like through the

285
00:15:39,440 --> 00:15:43,320
organization to like the CEO of the company, and you

286
00:15:43,399 --> 00:15:45,799
need to make the decision of like what's the what's

287
00:15:45,840 --> 00:15:49,360
the lesser of the financial impacts here? But long winded

288
00:15:49,399 --> 00:15:54,120
answer like sorre should have confidence from the engineering team

289
00:15:54,200 --> 00:15:55,919
that it's safe to roll back and do like an

290
00:15:55,960 --> 00:16:00,240
instant roll back. And it's also then helpful as an

291
00:16:00,320 --> 00:16:03,759
sary if like, let's say, a lot of times you

292
00:16:03,799 --> 00:16:07,200
will get paged in the middle of the night, and

293
00:16:07,919 --> 00:16:12,720
it's a large organization and sometimes sometimes you have great

294
00:16:12,759 --> 00:16:15,679
engineers on your team who you can escalate too, and

295
00:16:15,720 --> 00:16:19,080
they usually have like a paging rotation also, but a

296
00:16:19,080 --> 00:16:21,679
lot of times I'm not gonna lie, I was on

297
00:16:21,720 --> 00:16:24,679
an SR team and it's the middle of the night

298
00:16:24,759 --> 00:16:27,279
and I page, you know, I'm looking at this pod

299
00:16:27,320 --> 00:16:31,559
that's crashing and I can see that it's like actually

300
00:16:31,600 --> 00:16:34,960
like an error in the code, and I page the

301
00:16:35,000 --> 00:16:36,120
engineering team and.

302
00:16:37,840 --> 00:16:41,200
Speaker 6: What it crickets.

303
00:16:40,200 --> 00:16:43,919
Speaker 3: Just like a group project, Like it's the same Actually

304
00:16:44,039 --> 00:16:46,840
I have page again in twenty minutes and crickets.

305
00:16:46,879 --> 00:16:50,120
Speaker 4: So I've been in a state where like I am

306
00:16:50,200 --> 00:16:52,720
just rolling pods all night. When I say rolling pods,

307
00:16:52,720 --> 00:16:56,200
like restarting pods over and over and over again until

308
00:16:56,240 --> 00:16:57,559
someone wakes up in the morning.

309
00:16:58,480 --> 00:17:01,519
Speaker 2: Wait, so you're when you say knock like network operations center?

310
00:17:01,600 --> 00:17:01,799
Speaker 1: Is that?

311
00:17:02,360 --> 00:17:05,720
Speaker 2: Is that mostly to manage the physical hardware.

312
00:17:07,000 --> 00:17:09,559
Speaker 4: It's hard for me to say I didn't work. They

313
00:17:09,559 --> 00:17:12,039
were kind of like our first line of defense at

314
00:17:12,039 --> 00:17:14,839
some of my jobs, and it works super closely with them.

315
00:17:15,240 --> 00:17:16,160
Speaker 6: But yeah, those are.

316
00:17:16,119 --> 00:17:19,519
Speaker 4: Typically the people that were like the hard like they

317
00:17:19,519 --> 00:17:22,920
were working in the data center, and you know, the

318
00:17:23,039 --> 00:17:26,119
organization obviously sees like value and kind of transitioning them

319
00:17:26,119 --> 00:17:27,519
to different roles and things like that.

320
00:17:27,640 --> 00:17:31,119
Speaker 2: So yeah, so you then have to look at a

321
00:17:31,119 --> 00:17:34,680
lot of application specific metrics, so more on the software

322
00:17:34,720 --> 00:17:39,119
validation side, and then be that that ferry that bridge

323
00:17:39,160 --> 00:17:41,119
back from Okay, there's a problem at the hardware level

324
00:17:41,160 --> 00:17:43,799
that's getting escalated and having to find the application team

325
00:17:43,839 --> 00:17:46,160
and of course they have no on call rotation ever,

326
00:17:46,319 --> 00:17:49,880
because they're not accountable for keeping the systems up and

327
00:17:49,960 --> 00:17:50,720
running right.

328
00:17:50,640 --> 00:17:52,119
Speaker 1: They're just to shift code.

329
00:17:53,240 --> 00:17:55,319
Speaker 4: Yeah, just ship it and hand it over you know,

330
00:17:55,880 --> 00:17:59,640
but that's where you know close relationship also, and like

331
00:17:59,640 --> 00:18:01,599
I say saying like I keep going back to this

332
00:18:01,680 --> 00:18:06,920
like production readiness in it should be that before the

333
00:18:06,960 --> 00:18:10,440
application team hands over something like if saying like, work

334
00:18:10,480 --> 00:18:12,400
with the s RE team in advance to make sure

335
00:18:13,559 --> 00:18:17,720
that if it's a mission critical service in your micro

336
00:18:17,799 --> 00:18:21,200
service architecture, like that the dashboards are in place, you

337
00:18:21,279 --> 00:18:27,160
understand the dependencies like a when I say dependencies too,

338
00:18:27,200 --> 00:18:28,759
like a lot of things that come up in SRE.

339
00:18:29,759 --> 00:18:32,160
And that's why I like this is just I say,

340
00:18:32,319 --> 00:18:37,640
nerd out on distributed system stuff. You can see one

341
00:18:37,799 --> 00:18:44,079
application starting to show some latency, but the actual impact

342
00:18:44,839 --> 00:18:50,920
is more so like a service downstream, and there are

343
00:18:51,039 --> 00:18:56,000
different ways of like understanding obviously, like if this part

344
00:18:56,079 --> 00:18:59,839
of the system is having this percentage of latency, then

345
00:19:00,039 --> 00:19:04,519
I need to understand that these downstream services are getting

346
00:19:04,640 --> 00:19:07,559
hit by this other service X number of times it's

347
00:19:07,599 --> 00:19:10,799
going to start like having this kind of latency. So

348
00:19:10,880 --> 00:19:13,519
you really have to understand like the entire system and

349
00:19:13,640 --> 00:19:17,400
understand those dependencies. That's where it helps, Like that's where

350
00:19:17,480 --> 00:19:21,559
s RE really comes in because when like these application

351
00:19:21,599 --> 00:19:25,440
teams are kind of like siloed working on their specific service.

352
00:19:25,960 --> 00:19:29,799
The SR team is more responsible for understanding like the

353
00:19:29,880 --> 00:19:33,359
trace of a request through the system and understanding those

354
00:19:33,359 --> 00:19:34,440
downstream effects.

355
00:19:34,599 --> 00:19:37,039
Speaker 2: Well, I mean, you just got into some really deep

356
00:19:37,079 --> 00:19:39,960
topics there, Like there's a huge complexity with like queuing

357
00:19:40,480 --> 00:19:44,079
theory and window based you know, back offs there as

358
00:19:44,119 --> 00:19:47,880
well as like the system effect like actual systems, thinking

359
00:19:48,000 --> 00:19:51,200
of how one thing triggers another thing, which triggers another thing.

360
00:19:51,240 --> 00:19:54,000
And I don't remember ever seeing a boot camp that

361
00:19:54,119 --> 00:19:57,319
would have gone anyone prepared for handling this. And you

362
00:19:57,440 --> 00:19:59,680
got through this, and now I have I understand you

363
00:19:59,759 --> 00:20:02,559
were still like being on the spot to actually make

364
00:20:02,640 --> 00:20:05,440
a business impactful decision for the organization.

365
00:20:06,079 --> 00:20:11,160
Speaker 4: Yes, that's where it gets very stressful. And that's where

366
00:20:11,279 --> 00:20:13,440
I just I feel like in this kind of role,

367
00:20:13,720 --> 00:20:19,680
you really need someone who cares and yes, because otherwise

368
00:20:20,799 --> 00:20:23,799
the company's revenue is on the line, whether that be

369
00:20:24,839 --> 00:20:26,640
not even like I was saying an example of like

370
00:20:26,640 --> 00:20:28,720
the Black Friday issue, where like you have to make

371
00:20:28,759 --> 00:20:35,039
the call of is this shopping cart feature more important

372
00:20:35,079 --> 00:20:38,680
than the fact that X number of other users can't

373
00:20:38,720 --> 00:20:43,519
do this other thing that they need to do, versus

374
00:20:43,680 --> 00:20:47,279
even understanding like the slas and is there like a

375
00:20:47,319 --> 00:20:50,680
financial obligation that I need to pay my customers if

376
00:20:51,519 --> 00:20:53,119
X number of things start to go down?

377
00:20:53,519 --> 00:20:55,240
Speaker 2: So did you have like some sort of dashboard that

378
00:20:55,279 --> 00:20:57,519
listed out every single feature and like how much it

379
00:20:57,559 --> 00:20:59,400
was supposed to make for the company, and like how

380
00:20:59,519 --> 00:21:02,279
much the shopping I mean, I feel like NPM and

381
00:21:02,279 --> 00:21:04,279
paramoun are a little bit weird because it's not really

382
00:21:04,319 --> 00:21:06,880
like end user shopping car scenarios. I mean, I don't

383
00:21:06,880 --> 00:21:10,640
really understand paramounts, you know, end user pricing model. I

384
00:21:10,680 --> 00:21:13,240
know there's like plus right, but other than that, So

385
00:21:13,279 --> 00:21:15,759
I feel like it's it's even harder to really understand

386
00:21:15,799 --> 00:21:18,039
what the long term impact is for each one of

387
00:21:18,039 --> 00:21:19,039
those things that's breaking.

388
00:21:19,319 --> 00:21:21,839
Speaker 4: You know, it's very different obviously at different places, Like

389
00:21:23,039 --> 00:21:24,960
so I keep going back to the shopping cart example.

390
00:21:25,039 --> 00:21:28,799
Like I worked at different startups in like the retail industry,

391
00:21:28,799 --> 00:21:31,160
and although I wasn't on the SR team there specifically

392
00:21:31,480 --> 00:21:34,599
as more in like an architectural role, so like kind

393
00:21:34,640 --> 00:21:38,160
of tasked with understanding that I'm working with the demops team.

394
00:21:38,200 --> 00:21:43,319
But yeah, like in a perfect world, you would have

395
00:21:43,400 --> 00:21:47,200
dashboard self you make those decisions, but you don't have

396
00:21:47,319 --> 00:21:50,799
that so you just you have to make the best

397
00:21:50,839 --> 00:21:54,960
call based on your understanding and you know, escalate to

398
00:21:55,000 --> 00:21:59,039
the right people if you can. And that's where I

399
00:21:59,039 --> 00:22:03,119
should really get into, you know, is maybe like shift

400
00:22:03,119 --> 00:22:07,640
the conversation too to like strategies besides just like reactive strategies,

401
00:22:08,160 --> 00:22:10,559
because we've talked about like a lot of like reactive

402
00:22:10,759 --> 00:22:15,160
approaches to things. But part of what SRI is tasked

403
00:22:15,200 --> 00:22:18,039
with is like how do we make this system more

404
00:22:18,079 --> 00:22:20,559
resilient by nature of things we can do?

405
00:22:21,200 --> 00:22:23,160
Speaker 2: And I feel like you're in the perfect position to

406
00:22:23,519 --> 00:22:26,440
think about and sort of ide eight on that. But

407
00:22:26,559 --> 00:22:28,240
don't you end up with like a lot of pushback

408
00:22:28,319 --> 00:22:30,559
because that's sort of like a shift left approach in

409
00:22:30,559 --> 00:22:34,160
the organization with like oh, you know application product teams,

410
00:22:34,160 --> 00:22:36,200
you know this thing that you're doing that has like

411
00:22:36,279 --> 00:22:38,960
a high likelyhood that there's going to be a production impact,

412
00:22:39,000 --> 00:22:40,720
maybe you should do something different. How do you get

413
00:22:40,720 --> 00:22:43,759
that actually across those teams so they can like start

414
00:22:43,799 --> 00:22:46,519
making those actual changes because it's like a mindset or

415
00:22:46,599 --> 00:22:47,599
culture shift for them.

416
00:22:48,119 --> 00:22:51,000
Speaker 4: Kind Of where I'm going with this is more so

417
00:22:51,240 --> 00:22:54,440
on like the redundancy side of things, like SRI team

418
00:22:54,480 --> 00:22:57,079
making the decision and it's you have to work with

419
00:22:57,200 --> 00:23:02,440
like the engineering management leadership to understand and like, you know,

420
00:23:02,480 --> 00:23:05,480
if we're running in like a Kupernettese environment making that

421
00:23:05,839 --> 00:23:09,160
do we either do we do multi cloud? Do we

422
00:23:09,400 --> 00:23:14,480
obviously we do multi region those sorts of decisions, And

423
00:23:14,519 --> 00:23:18,359
then that's where I get into I don't think a

424
00:23:18,400 --> 00:23:21,599
lot of people necessarily realize, like I was saying, how

425
00:23:21,720 --> 00:23:27,359
valuable like content delivery is in this because just by

426
00:23:27,440 --> 00:23:30,039
nature of having as much as you can cashed and

427
00:23:30,079 --> 00:23:33,880
distributed outside of like before the request even gets to

428
00:23:34,640 --> 00:23:37,599
whether it's like your physical data center or or a cloud,

429
00:23:38,599 --> 00:23:42,720
how can we like offload some of that traffic? Because

430
00:23:42,759 --> 00:23:45,200
I feel like a lot of these systems are just

431
00:23:45,559 --> 00:23:49,359
the fallover without that. And I know we mentioned it

432
00:23:49,400 --> 00:23:51,079
like at the beginning of the episode, and maybe I'll

433
00:23:51,119 --> 00:23:54,119
like touch on it more at the end. But one

434
00:23:54,160 --> 00:23:56,440
thing that's been very interesting for me to like sit

435
00:23:56,480 --> 00:23:58,319
back and watch is like Jenny I is.

436
00:24:00,160 --> 00:24:03,519
Speaker 6: Saying that word it is, it.

437
00:24:03,359 --> 00:24:06,160
Speaker 5: Is what it is like twofold.

438
00:24:07,279 --> 00:24:11,200
Speaker 4: First off, like my heart goes out to s on

439
00:24:11,279 --> 00:24:14,880
their sres who work in environments where they have like

440
00:24:15,000 --> 00:24:18,160
developers that are just what this whole vibe coding thing, Like,

441
00:24:19,400 --> 00:24:23,920
I don't think people realize how much like these lms

442
00:24:23,960 --> 00:24:28,400
are hallucinating and producing like content that is just garbage,

443
00:24:28,440 --> 00:24:31,200
absolute garbage soup.

444
00:24:31,240 --> 00:24:34,039
Speaker 1: Poor one out for the SR supporting vibe coding is

445
00:24:34,079 --> 00:24:34,680
what you're saying.

446
00:24:35,200 --> 00:24:38,559
Speaker 2: Yeah, we did have some previous guests on the show

447
00:24:38,640 --> 00:24:41,799
who were like swearing to us that their lms didn't

448
00:24:41,839 --> 00:24:42,759
ever hallucinate.

449
00:24:43,720 --> 00:24:46,440
Speaker 4: I literally was like, I think I won't say the

450
00:24:46,599 --> 00:24:48,400
LM because I'm going to get myself in trouble. But

451
00:24:48,680 --> 00:24:51,119
I was like asking, like going back and forth with

452
00:24:51,160 --> 00:24:54,200
one of them earlier this week, like writing stuff, and

453
00:24:54,680 --> 00:24:56,880
I asked it to revise it, and it started like

454
00:24:57,000 --> 00:25:01,279
putting all these like statistics in, and I was like, I.

455
00:25:01,200 --> 00:25:06,160
Speaker 5: Was using a product that allowed me to.

456
00:25:07,160 --> 00:25:11,480
Speaker 4: Incorporate a lot of basically like a rag guy, so

457
00:25:11,519 --> 00:25:14,039
I could incorporate like a bunch of external sources. And

458
00:25:14,079 --> 00:25:17,039
I was like, there's no way those statistics are in

459
00:25:17,119 --> 00:25:21,519
those sources, like because I know the data that I added.

460
00:25:22,000 --> 00:25:24,680
And I asked it, like where did you get these statistics?

461
00:25:24,680 --> 00:25:26,599
And it's like, you're right, I made them up. I'm sorry.

462
00:25:29,680 --> 00:25:32,119
Speaker 2: At least you've got a conclusive answer there, because We're

463
00:25:32,119 --> 00:25:35,440
seeing this a lot more with seoed articles today, where

464
00:25:35,519 --> 00:25:37,599
not only are they being generated by llms, but the

465
00:25:38,000 --> 00:25:41,720
links that they go to do not back up whatever

466
00:25:41,960 --> 00:25:43,559
is in the article, and like so even if a

467
00:25:43,640 --> 00:25:46,160
human is writing it, it's just some fluff piece. And

468
00:25:46,200 --> 00:25:48,559
if you go there, it's like, oh, sixty percent of

469
00:25:48,599 --> 00:25:51,640
attacks are from this particular source, like email is the

470
00:25:51,680 --> 00:25:53,839
source of all attacks and security. And you go and

471
00:25:53,839 --> 00:25:56,000
you click on the link and it's just some completely

472
00:25:56,039 --> 00:25:58,640
made up garbage that's also not researched by some other

473
00:25:58,640 --> 00:26:01,799
company that nothing to do with the topic whatsoever. So

474
00:26:01,880 --> 00:26:06,000
can you blame the probabilistic you know, statistical engine that's

475
00:26:06,079 --> 00:26:08,119
just predicting the next word that is also going to

476
00:26:08,200 --> 00:26:10,119
pull out links that have nothing to do with the

477
00:26:10,119 --> 00:26:10,960
topic whatsoever.

478
00:26:11,400 --> 00:26:12,160
Speaker 6: Yes, yeah, you.

479
00:26:12,119 --> 00:26:14,440
Speaker 5: Can't blame it, but it's like it's on us to.

480
00:26:16,319 --> 00:26:19,319
Speaker 4: You know, vet it. But what I wanted to touch on that.

481
00:26:19,319 --> 00:26:21,519
Speaker 5: Very briefly was just basically, like I read.

482
00:26:21,359 --> 00:26:26,119
Speaker 4: Something this morning, like the percentage like there's a direct

483
00:26:26,119 --> 00:26:30,960
correlation between these companies that are allowing like more AI

484
00:26:31,319 --> 00:26:38,319
assistive like developer stuff and outages. So my heart, there's

485
00:26:38,599 --> 00:26:42,000
I'd have to like I think I screenshotted it on LinkedIn,

486
00:26:42,079 --> 00:26:45,200
but there was like it was a lot, and it's

487
00:26:45,279 --> 00:26:46,279
just in the back of my mind.

488
00:26:46,279 --> 00:26:47,359
Speaker 6: So there's that side of things.

489
00:26:47,400 --> 00:26:49,680
Speaker 4: And then there's also just like these like you know,

490
00:26:49,759 --> 00:26:53,400
open Ai and all these companies like pushing out it's

491
00:26:53,400 --> 00:26:56,519
like a race to get stuff out to production faster,

492
00:26:56,640 --> 00:27:00,599
and there's like that. I think last year just the

493
00:27:00,680 --> 00:27:02,319
number of outages, like every time I would go to

494
00:27:02,400 --> 00:27:05,400
chat tipt I think like almost like a week, I

495
00:27:05,400 --> 00:27:06,519
was like half the time is down.

496
00:27:08,400 --> 00:27:10,279
Speaker 2: I think the jury is really out on the numbers.

497
00:27:10,279 --> 00:27:12,480
I think we've concluded two things, and whether or not

498
00:27:12,519 --> 00:27:14,880
people are willing to listen to those is a separate topic.

499
00:27:15,319 --> 00:27:20,200
You are trading quality or speed. That is, that is

500
00:27:20,240 --> 00:27:23,119
the trade off, And so you know, I think there

501
00:27:23,119 --> 00:27:24,920
are a bunch of there are a bunch of good

502
00:27:24,960 --> 00:27:27,920
articles recently by people who work at some of these

503
00:27:29,319 --> 00:27:31,079
You say you don't like the word jen Ai. I

504
00:27:31,160 --> 00:27:36,480
really hate the word agentic. That's that's mine companies that

505
00:27:36,759 --> 00:27:40,599
they themselves are not using their own LM to produce

506
00:27:40,640 --> 00:27:42,960
the code, right, And so you know, I see these

507
00:27:43,200 --> 00:27:45,160
CEOs say oh yeah and stand up and be like, oh,

508
00:27:45,200 --> 00:27:47,519
our company like twenty five percent of all the source

509
00:27:47,519 --> 00:27:49,759
code we write is by you know, some sort of

510
00:27:49,839 --> 00:27:52,319
LM And I'm like, what do you what are you

511
00:27:52,400 --> 00:27:54,200
going to do about that risk? You know you're gonna

512
00:27:54,839 --> 00:27:56,119
decrease that in the future.

513
00:27:56,559 --> 00:27:59,359
Speaker 4: Yeah, I should say, like I'm not opposed to these tools,

514
00:27:59,359 --> 00:28:03,359
but they need to be used responsibly and for better

515
00:28:03,440 --> 00:28:06,640
or worse, like humans, Like it's our nature. Like that's

516
00:28:06,640 --> 00:28:09,400
what like the famous like programming quote, like we're lazy,

517
00:28:09,480 --> 00:28:13,039
Like a lazy person makes a good programmer because they

518
00:28:13,039 --> 00:28:14,920
want to automate things. So like there's a lot of

519
00:28:15,000 --> 00:28:19,119
value in these tools, but you hands down need to

520
00:28:19,240 --> 00:28:24,559
use them responsibly and be able to use them, like

521
00:28:24,920 --> 00:28:28,160
with actual knowledge behind it. Like I mentor a lot

522
00:28:28,160 --> 00:28:32,359
of developers and like you know, they're just they're wanting

523
00:28:32,359 --> 00:28:35,400
to get so done. And I'll see like this code

524
00:28:35,400 --> 00:28:37,519
that they wrote, and I'm like that that I know

525
00:28:37,519 --> 00:28:38,359
where you gotta from.

526
00:28:38,519 --> 00:28:40,519
Speaker 6: That's not right.

527
00:28:41,920 --> 00:28:45,559
Speaker 4: I don't know why it's telling you to like install

528
00:28:46,119 --> 00:28:47,319
Python packages in.

529
00:28:47,359 --> 00:28:49,960
Speaker 6: Your own application, but this is not right.

530
00:28:52,359 --> 00:28:54,440
Speaker 4: Yeah anyways, Yeah, but.

531
00:28:54,480 --> 00:28:56,480
Speaker 1: On a side note, I'm really admired that you got

532
00:28:56,480 --> 00:28:59,720
those Python packages to work inside of no js so.

533
00:29:01,079 --> 00:29:03,240
Speaker 2: Well. It just comes to the foreign function interface and

534
00:29:03,279 --> 00:29:05,640
you know, you just shell out to that, no, no

535
00:29:05,720 --> 00:29:08,519
big deal, and your application, you know, just multiple programming language.

536
00:29:08,519 --> 00:29:10,759
You know, what's what's the big deal? It's all it's

537
00:29:10,799 --> 00:29:11,680
all simply under the.

538
00:29:11,680 --> 00:29:14,880
Speaker 6: Hood, exactly. It's all one since but doesn't matter.

539
00:29:15,079 --> 00:29:16,599
Speaker 2: Right for now, for now.

540
00:29:20,880 --> 00:29:24,599
Speaker 1: So I want to come back to talking about playbooks,

541
00:29:24,599 --> 00:29:27,039
because you mentioned those and I got excited. But I

542
00:29:27,039 --> 00:29:32,160
think like an important thing to hang the playbook conversation

543
00:29:32,359 --> 00:29:36,119
on is at talking about being an sr AT scale,

544
00:29:36,440 --> 00:29:40,200
because like being an sr AT a small company versus

545
00:29:40,200 --> 00:29:44,240
being an sr AT an enterprise, I think are quite

546
00:29:44,319 --> 00:29:50,240
possibly two completely different career fields. And and so how

547
00:29:50,279 --> 00:29:55,200
did you address like the scale of of applications you

548
00:29:55,200 --> 00:29:55,920
were dealing with.

549
00:29:56,519 --> 00:30:00,039
Speaker 4: That's a good question too. I think it really so

550
00:30:00,119 --> 00:30:02,519
like talking about this like micro service system where a

551
00:30:02,599 --> 00:30:08,839
lot of people usually are at right now, typically everything

552
00:30:08,960 --> 00:30:12,279
is like a trade off, and that's like a handwaybe answer,

553
00:30:12,319 --> 00:30:15,240
but it's true. And so like when we go back

554
00:30:15,240 --> 00:30:18,880
to like the production readiness stuff, I think it's like

555
00:30:18,920 --> 00:30:24,319
helpful to understand like a like a process of like

556
00:30:24,400 --> 00:30:26,559
kind of rating these micro services. Like it's just like

557
00:30:26,599 --> 00:30:29,200
tier one tier two, tier three, Like like an a

558
00:30:29,400 --> 00:30:32,599
system is probably going to be like a tier one system,

559
00:30:32,680 --> 00:30:39,279
whereas if you have like a recommendations engine that's running

560
00:30:39,319 --> 00:30:42,079
things in the background, that's more of like a tier

561
00:30:42,119 --> 00:30:48,920
three system because critical functionality still works. Check out is

562
00:30:48,960 --> 00:30:51,599
going to be obviously a tier one system something like that.

563
00:30:52,359 --> 00:30:55,839
But there's a lot of things to consider, Like I

564
00:30:55,839 --> 00:30:59,640
feel like, especially like for people that are getting into

565
00:30:59,720 --> 00:31:05,200
this kind of role, it's important to understand like the

566
00:31:05,400 --> 00:31:09,799
traffic patterns, Like if you're working in like a small startup,

567
00:31:11,039 --> 00:31:13,519
you're accountable to your customers, like maybe just during like

568
00:31:13,640 --> 00:31:17,759
business hours, like a banking application or something like that

569
00:31:17,920 --> 00:31:20,839
is not going to be as critical as an application

570
00:31:20,920 --> 00:31:28,480
that's used internationally. Things like that, those are like does

571
00:31:28,519 --> 00:31:29,680
that kind of answer your question?

572
00:31:29,839 --> 00:31:32,240
Speaker 6: Like things to consider, Yeah, for sure.

573
00:31:33,720 --> 00:31:37,559
Speaker 1: And then tying it back to playbooks, like there's a

574
00:31:37,599 --> 00:31:41,000
certain level of scale where you just can't be familiar

575
00:31:41,039 --> 00:31:43,480
with all of these systems and you have to rely

576
00:31:43,680 --> 00:31:47,200
on playbooks. So what are some of the strategies that

577
00:31:47,319 --> 00:31:50,400
you've seen to making sure that you have an accurate,

578
00:31:50,480 --> 00:31:52,119
up to date playbook when you need it.

579
00:31:53,480 --> 00:31:54,759
Speaker 6: Yeah, up to date.

580
00:31:56,079 --> 00:31:58,160
Speaker 2: Everything is legacy as soon as you write it. Like

581
00:31:58,200 --> 00:32:00,559
as soon as you write something down, it is out

582
00:32:00,559 --> 00:32:03,160
of date. That code is already wrong as soon as

583
00:32:03,160 --> 00:32:03,799
it's released.

584
00:32:04,200 --> 00:32:06,519
Speaker 4: That Yeah, So that's I'm going to go back to.

585
00:32:06,839 --> 00:32:13,119
Like building out an S OR team is you need

586
00:32:13,200 --> 00:32:17,880
technical skills obviously, but you really need people who take

587
00:32:18,000 --> 00:32:23,359
ownership and care. Like we're talking about the playbooks. Like

588
00:32:24,000 --> 00:32:27,200
I've definitely been at companies where I will get I'm

589
00:32:27,440 --> 00:32:28,319
brand new.

590
00:32:28,480 --> 00:32:29,279
Speaker 6: I'll get Paige.

591
00:32:29,359 --> 00:32:32,160
Speaker 4: I go to the playbook and I see that like

592
00:32:32,200 --> 00:32:35,839
the playbooks can get it hasn't been updated in five years,

593
00:32:38,160 --> 00:32:41,200
and I'm like, you know, shoot, crap.

594
00:32:41,519 --> 00:32:43,519
Speaker 6: Like what do I do? Like I'm getting Paige.

595
00:32:44,839 --> 00:32:47,039
Speaker 4: Usually, like when you're Paige, it will point you to

596
00:32:47,279 --> 00:32:52,079
the playbook and it's not there. So, you know, the

597
00:32:52,400 --> 00:32:56,440
best way that I've seen people tackle this is like ownership.

598
00:32:57,079 --> 00:33:01,839
Someone needs to own these playbooks. Somebody if a if

599
00:33:01,880 --> 00:33:03,759
A system is like a Tier one or a Tier

600
00:33:03,759 --> 00:33:06,480
two system, like this is part of this, like production

601
00:33:06,599 --> 00:33:10,400
readiness checklist. Like before before the SRI is okay with

602
00:33:10,480 --> 00:33:15,640
deploying this and like a lot of times, like company

603
00:33:15,720 --> 00:33:18,279
like the the SR team is they're not really the

604
00:33:18,319 --> 00:33:21,920
gatekeeper because a lot of these companies, like the developers,

605
00:33:21,920 --> 00:33:25,720
have the ability to ship themselves, but there are obviously

606
00:33:25,799 --> 00:33:30,039
consequences if they ship something and it crashes and and

607
00:33:30,200 --> 00:33:35,160
that doesn't completely come that that is like a shared ownership.

608
00:33:35,880 --> 00:33:39,400
But having there needs to be ownership of these playbooks,

609
00:33:39,400 --> 00:33:43,000
and that's usually done by the s RE team. Typically,

610
00:33:44,599 --> 00:33:49,039
you know, you will get the leadership knows when something

611
00:33:49,079 --> 00:33:53,680
is very critical, and good leadership will make sure that

612
00:33:53,799 --> 00:33:57,799
these teams are communicating on like a regular cadence. I'm

613
00:33:57,799 --> 00:34:01,920
big on like these regular cadences, not necessarily because meetings

614
00:34:02,000 --> 00:34:06,000
kind of suck, but they are important for communication.

615
00:34:06,440 --> 00:34:10,360
Speaker 2: I like that you put the responsibility of the playbooks

616
00:34:10,360 --> 00:34:12,920
on the s RE team. There's actually a debate that

617
00:34:13,440 --> 00:34:16,199
my CEO and myself have been talking about a bunch

618
00:34:16,239 --> 00:34:19,639
where if it's an artifact that gets thrown over the wall,

619
00:34:19,800 --> 00:34:22,320
then it's not going like you can't it doesn't contain

620
00:34:22,400 --> 00:34:25,760
knowledge in a way it contains They're usually actionable things,

621
00:34:26,000 --> 00:34:28,960
and so since the knowledge is there, you won't know

622
00:34:29,000 --> 00:34:31,880
how to adapt to a playbook when you get to

623
00:34:31,960 --> 00:34:37,119
a new scenario that's not encountered before. Right, And most

624
00:34:37,159 --> 00:34:38,960
of the problems that we find in systems that we

625
00:34:39,000 --> 00:34:41,639
automate are only problems that we haven't seen before.

626
00:34:42,679 --> 00:34:46,199
Speaker 4: And I will say too so when I say like

627
00:34:46,440 --> 00:34:50,719
having accountability and ownership, like that's where for better or worse,

628
00:34:50,800 --> 00:34:54,880
everybody hates on call schedules, but those on call schedules

629
00:34:55,039 --> 00:34:58,400
do serve as kind of like a backpropagation of like

630
00:34:58,599 --> 00:35:01,039
if somebody is on if you're on an on call

631
00:35:01,159 --> 00:35:04,039
rotation and you're getting paid frequently, like you have a

632
00:35:04,199 --> 00:35:08,519
vested interest in stopping that because if it gets out

633
00:35:08,519 --> 00:35:13,519
of hand, it's just a huge snowball of you know, obviously,

634
00:35:13,559 --> 00:35:17,519
like you you're starting to like your sleep is interrupted.

635
00:35:17,639 --> 00:35:20,679
So and for better or worse, like people will be like, oh,

636
00:35:20,760 --> 00:35:22,360
you know, you had a rough night, take it easy

637
00:35:22,400 --> 00:35:22,800
during the day.

638
00:35:22,880 --> 00:35:23,679
Speaker 6: Yeah right, that's not.

639
00:35:26,119 --> 00:35:28,480
Speaker 4: You're still getting up and doing your usual hours, but

640
00:35:28,559 --> 00:35:32,280
you're just like I'm dead. So there's a vested interest

641
00:35:32,360 --> 00:35:35,800
in in like sharing that ownership of the on call rotation.

642
00:35:35,960 --> 00:35:38,440
And if you're getting paid constantly, like you're going to

643
00:35:38,760 --> 00:35:41,840
want to make sure that your teammates have the playbooks

644
00:35:41,840 --> 00:35:44,440
in place. And you're like, I just had a huge

645
00:35:44,480 --> 00:35:47,679
outage and these are the steps that I took to

646
00:35:47,760 --> 00:35:50,320
remediate this these are the things that were helpful. So

647
00:35:50,760 --> 00:35:52,760
after my on call rotation, I'm going to go in

648
00:35:52,840 --> 00:35:55,400
and update the playbook and make sure that the steps

649
00:35:55,440 --> 00:35:57,559
that I took are there so that maybe there's like

650
00:35:57,599 --> 00:35:59,679
more junior members on the team they also know and

651
00:35:59,719 --> 00:36:02,639
things like that, because usually too and like these kind

652
00:36:02,679 --> 00:36:05,880
of environments that you'll have like multiple tiers there, so

653
00:36:05,920 --> 00:36:08,239
there's a knock. But then even within s RE, like

654
00:36:08,280 --> 00:36:11,039
I think it's helpful to have like multiple tiers of

655
00:36:11,119 --> 00:36:14,559
SRY support, so you have like more senior level sries

656
00:36:14,840 --> 00:36:18,920
and then more junior level esteres and like maybe maybe

657
00:36:19,000 --> 00:36:21,159
during the day you would page more junior SRY or

658
00:36:21,239 --> 00:36:23,559
something like that, whereas a senior is around to step

659
00:36:23,559 --> 00:36:25,480
in right away, but you want to have stuff in

660
00:36:25,559 --> 00:36:26,920
place for it makes sense.

661
00:36:27,000 --> 00:36:29,159
Speaker 2: What I'm hearing is you're a huge proponent of these

662
00:36:29,159 --> 00:36:33,159
incident management tools that are generating playbooks dynamically on the

663
00:36:33,199 --> 00:36:38,039
fly using some sort of lum please a lot of so.

664
00:36:38,159 --> 00:36:39,119
Speaker 6: Much reviewing them.

665
00:36:39,239 --> 00:36:43,119
Speaker 4: That's fine by me, but like, yeah, these lms are helpful,

666
00:36:43,159 --> 00:36:46,000
but they are they are not the end all, Like

667
00:36:46,440 --> 00:36:47,320
that's not the end.

668
00:36:47,199 --> 00:36:47,639
Speaker 5: Of the road.

669
00:36:48,199 --> 00:36:52,159
Speaker 1: You just gave him my next startup idea, Warren, I

670
00:36:52,159 --> 00:36:54,039
mean you have to compete with some of our past guests,

671
00:36:54,119 --> 00:36:58,360
I think, who are already already promising h dynamic run

672
00:36:58,400 --> 00:37:00,519
books based off I mean, there's a thing I think

673
00:37:00,519 --> 00:37:02,679
Sentry does, and I think there's a couple other tools

674
00:37:02,679 --> 00:37:06,079
now that they have your source code or at least

675
00:37:06,320 --> 00:37:07,960
you know, and then they have the stack trays from

676
00:37:07,960 --> 00:37:09,880
the error and the actual error message, and they use

677
00:37:09,960 --> 00:37:12,840
that to automatically propose a pull request into your get

678
00:37:12,840 --> 00:37:15,400
repository that supposedly fixes the problem so that someone can

679
00:37:15,440 --> 00:37:16,280
review and improve it.

680
00:37:16,719 --> 00:37:19,079
Speaker 2: And you've got to know that at three am, you

681
00:37:19,119 --> 00:37:20,519
are just going to click improve.

682
00:37:23,679 --> 00:37:25,760
Speaker 6: Yeah, yeah, that's yeah.

683
00:37:25,840 --> 00:37:27,880
Speaker 4: All those tools, Like I said, I think they're helpful,

684
00:37:27,960 --> 00:37:30,400
but they need to be used responsibly and with somebody

685
00:37:30,400 --> 00:37:33,639
who is actually like a human being with background that

686
00:37:33,760 --> 00:37:34,639
says yeah your name.

687
00:37:34,960 --> 00:37:38,519
Speaker 1: Yeah. I think we're still discovering, like what the rules

688
00:37:38,519 --> 00:37:42,519
of responsibility are. Like I feel like we're learning quickly,

689
00:37:42,559 --> 00:37:44,960
but we're still learning, you know, some of those being

690
00:37:45,000 --> 00:37:49,239
like you have to give it very discrete tasks, you know,

691
00:37:49,639 --> 00:37:52,159
ask it to do one thing so that the result

692
00:37:52,199 --> 00:37:54,199
that comes back to you is something you can actually

693
00:37:54,760 --> 00:37:57,079
parse in your head. And you know, don't give it

694
00:37:57,159 --> 00:37:59,840
something that you don't know how to do, because then

695
00:37:59,880 --> 00:38:03,320
you have no way to check its work. Or another

696
00:38:03,360 --> 00:38:06,159
one that I use that I really like is treat

697
00:38:06,199 --> 00:38:10,840
it like your own personal intern or junior developer. You know,

698
00:38:10,880 --> 00:38:15,960
you wouldn't like hire an intern and then say, you know, hey,

699
00:38:16,400 --> 00:38:20,760
go architect this four tiered micro service for us. It'll

700
00:38:20,800 --> 00:38:21,760
be due by Friday.

701
00:38:22,079 --> 00:38:26,360
Speaker 2: Oops. I mean you're onto something there, will I will

702
00:38:26,400 --> 00:38:29,719
say that maybe it's just forcing us to finally think

703
00:38:29,760 --> 00:38:35,119
about concretely how human should actually work together effectively. And

704
00:38:35,599 --> 00:38:37,280
I you know, in the last you know what is

705
00:38:37,320 --> 00:38:39,519
the year twenty twenty five, we haven't figured out how

706
00:38:39,519 --> 00:38:41,800
to do that correctly, and now we're throwing in a

707
00:38:41,840 --> 00:38:44,039
tool into the next which actually requires us to do

708
00:38:44,079 --> 00:38:46,679
that work. I'm going to guess it's going to be

709
00:38:46,679 --> 00:38:51,000
like another two hundred years before we got we got that,

710
00:38:51,280 --> 00:38:53,840
you know, actually figured out how to optimally communicate one

711
00:38:53,920 --> 00:38:54,320
human being.

712
00:38:54,639 --> 00:38:57,199
Speaker 3: I'm telling you, I have a teenager, and like I

713
00:38:57,239 --> 00:38:59,199
would just like to point out that we have been

714
00:38:59,280 --> 00:39:01,800
peopling for millions of years and we still do not

715
00:39:01,960 --> 00:39:06,480
have a good way to make the teenagers not really

716
00:39:06,519 --> 00:39:10,039
really dumb Okay, like just it's never gonna happen. It's

717
00:39:10,039 --> 00:39:10,840
never gonna happen.

718
00:39:11,719 --> 00:39:14,639
Speaker 2: Millions is a bit long. I'll give you like four

719
00:39:14,719 --> 00:39:16,239
hundred thousand or so.

720
00:39:16,559 --> 00:39:17,159
Speaker 3: But I.

721
00:39:18,639 --> 00:39:22,119
Speaker 2: Do definitely accept that argument that you know, but you know,

722
00:39:22,400 --> 00:39:25,320
acceleration of evolution, so maybe maybe we'll get there faster

723
00:39:25,400 --> 00:39:25,760
this time.

724
00:39:26,199 --> 00:39:31,599
Speaker 1: Right, Moore's law for humans, that's a thing, right, Yeah, Yeah, I.

725
00:39:32,199 --> 00:39:35,599
Speaker 2: Really appreciate Moore's law. What I don't appreciate is people

726
00:39:35,679 --> 00:39:37,679
saying that Moore's law applies to things that it doesn't

727
00:39:37,679 --> 00:39:42,199
apply to. I mean, I totally agree that the principle

728
00:39:42,719 --> 00:39:47,280
by what is it correlation doesn't necessarily imply causation. Although

729
00:39:47,559 --> 00:39:49,039
you know, you look at lots of things and they

730
00:39:49,039 --> 00:39:52,480
do double at you know, some rate. That's but people

731
00:39:52,480 --> 00:39:54,679
get more law wrong a lot. They apply to things

732
00:39:54,719 --> 00:39:57,000
that don't make sense. It is literally just the size

733
00:39:57,000 --> 00:40:00,400
of the transistors will reduce by half every eighteen months.

734
00:40:00,559 --> 00:40:02,920
That's that's it. That's the whole law. It doesn't apply

735
00:40:03,199 --> 00:40:06,519
like statistically to anything else. Now we do see doubling

736
00:40:06,880 --> 00:40:10,079
and size reduction and other things definitely, like dies on

737
00:40:10,119 --> 00:40:12,199
a chip and et cetera. But you know, there's like

738
00:40:12,199 --> 00:40:14,719
an interesting thing. Does it apply to these new plastic

739
00:40:14,800 --> 00:40:17,320
chips that I was China producing now that don't use

740
00:40:17,320 --> 00:40:21,400
silicon I don't remember the material and that are supposedly

741
00:40:21,480 --> 00:40:23,639
like half or a tenth of the size, you know,

742
00:40:23,719 --> 00:40:25,280
does Moore's law apply there too?

743
00:40:26,119 --> 00:40:26,840
Speaker 1: I have no idea.

744
00:40:26,880 --> 00:40:30,719
Speaker 2: I was just trying to make a joke, okay for us.

745
00:40:32,440 --> 00:40:35,159
Speaker 3: I wanted to backtrack just like a little bit and

746
00:40:35,199 --> 00:40:37,840
point out Amy that I really like your kind of

747
00:40:38,159 --> 00:40:42,000
go gettedness, I guess with understanding actually what the business needs.

748
00:40:42,119 --> 00:40:45,599
Because I work with a lot of people and especially students,

749
00:40:45,639 --> 00:40:47,840
and they just have no idea, like no idea like

750
00:40:47,880 --> 00:40:49,320
what it is, like what is supposed to be the

751
00:40:49,400 --> 00:40:52,719
end result of this kind of like process that they're doing.

752
00:40:53,079 --> 00:40:55,239
And I think it's very important. I mean it's important

753
00:40:55,280 --> 00:40:57,760
from like work, but I think it's also very important

754
00:40:57,760 --> 00:41:00,199
from a career perspective as well, Like would you like

755
00:41:00,239 --> 00:41:04,159
to maybe get new jobs? Like you should? You should

756
00:41:04,239 --> 00:41:06,159
understand what's going on at least a little bit.

757
00:41:06,840 --> 00:41:08,920
Speaker 5: Yeah, Like I said, I don't know.

758
00:41:09,039 --> 00:41:11,440
Speaker 4: I just like still to this day feel very fortunate

759
00:41:11,480 --> 00:41:14,800
to be able to do something that I pretty much

760
00:41:14,880 --> 00:41:19,800
enjoy every single day. And I just I don't know,

761
00:41:20,480 --> 00:41:23,920
maybe I just just like too harsh growing up or something.

762
00:41:24,000 --> 00:41:27,159
But I'm just like I feel like if I'm getting

763
00:41:27,199 --> 00:41:32,000
a salary, then I should be. There is value in

764
00:41:32,079 --> 00:41:35,079
pushing back sometimes when you're like senior enough to have

765
00:41:35,119 --> 00:41:38,199
a level of expertise, like in a professional and respectful way.

766
00:41:39,039 --> 00:41:41,760
But yes, like, at the end of the day, our job,

767
00:41:41,840 --> 00:41:43,679
whether we like it or not, is to make this

768
00:41:43,760 --> 00:41:50,480
company profitable, whether that be features, making sure that users

769
00:41:50,800 --> 00:41:53,360
have confidence in the service.

770
00:41:53,519 --> 00:41:55,639
Speaker 6: Yeah, sorry, stuff like that, They're.

771
00:41:55,440 --> 00:41:59,800
Speaker 2: Going to say, despite leaderships and a strategy to the contrary,

772
00:42:01,199 --> 00:42:02,760
I am. I am sort of curious that you have

773
00:42:02,840 --> 00:42:08,000
any you know, personal strategies for gleaning better insight into

774
00:42:08,360 --> 00:42:11,719
how the business should work or how a feature that is,

775
00:42:11,800 --> 00:42:15,840
say technical, transforms into value for the business. Are you

776
00:42:15,920 --> 00:42:19,960
working with product managers or with someone on the marketing

777
00:42:19,960 --> 00:42:23,519
and sales team? Are you grabbing information from the internet?

778
00:42:23,559 --> 00:42:25,599
Like how are you actually figuring this out?

779
00:42:26,239 --> 00:42:30,679
Speaker 5: Like what is and isn't valuable for the business? I

780
00:42:30,719 --> 00:42:32,440
guess it's for.

781
00:42:32,559 --> 00:42:36,320
Speaker 4: Me personally over my career. It probably depends on the

782
00:42:36,320 --> 00:42:39,360
domain I'm working in. So I've worked in like very

783
00:42:39,440 --> 00:42:43,280
like domains of like the construction industry, Like I don't

784
00:42:43,320 --> 00:42:50,000
have expertise in what these people want obviously, Like I

785
00:42:50,039 --> 00:42:52,920
like my personality, I want to understand the domain that

786
00:42:52,960 --> 00:42:55,599
I'm working in so that I can try to better

787
00:42:55,679 --> 00:42:58,639
understand these types of things. But in that sort of scenario,

788
00:42:58,760 --> 00:43:00,599
like I am going to lean more on the product

789
00:43:00,639 --> 00:43:04,960
people now if it's like a dev focused company. You know,

790
00:43:05,159 --> 00:43:09,559
my own thoughts obviously, like doing this a while, like

791
00:43:09,639 --> 00:43:11,880
having friends that I work with and like hearing their

792
00:43:11,920 --> 00:43:17,320
stories that goes into like what isn't isn't valuable? Like

793
00:43:18,320 --> 00:43:22,679
I'm trying to think of like different scenarios like that

794
00:43:22,760 --> 00:43:27,719
I've been in. There was definitely one later on in

795
00:43:27,719 --> 00:43:32,400
my career where there was like this is like an

796
00:43:32,480 --> 00:43:37,519
SR type thing, so like the security team was extremely

797
00:43:37,559 --> 00:43:41,320
focused on like getting this tool out because they were

798
00:43:41,480 --> 00:43:48,079
getting paged based on things that were happening. But the

799
00:43:48,239 --> 00:43:55,559
tool that they had vetted, while it seemed good, it

800
00:43:55,719 --> 00:43:57,960
was a very like rush job to get it out

801
00:43:58,000 --> 00:44:02,159
in time. And so like on my side of things,

802
00:44:02,519 --> 00:44:04,639
like there was a lot of code that needed to

803
00:44:04,639 --> 00:44:08,519
be deployed to get form the infant being a little vague,

804
00:44:10,239 --> 00:44:13,719
so I'll try if I can. That's fair, I can clarify,

805
00:44:13,800 --> 00:44:16,480
I will, but at the end of the sorry it.

806
00:44:16,440 --> 00:44:19,320
Speaker 2: Works, no, but it works because the more vague you are,

807
00:44:19,440 --> 00:44:22,679
the more trauma that listeners will have because it will

808
00:44:22,719 --> 00:44:25,800
relate to their situation. Like I have like three different

809
00:44:25,800 --> 00:44:27,719
traumas right now, just based off of what you said.

810
00:44:27,880 --> 00:44:30,840
You know, security teams pushing stuff out, you know, not

811
00:44:30,920 --> 00:44:33,719
knowing what to do, you know, not well vetted software

812
00:44:33,760 --> 00:44:37,280
that now everyone is dependent on, like you know, not

813
00:44:37,440 --> 00:44:38,599
a short number of times.

814
00:44:39,119 --> 00:44:43,199
Speaker 4: Yeah, So like and I have, like in this particular scenario,

815
00:44:43,280 --> 00:44:46,400
like a really close relationship, like really got long well

816
00:44:46,559 --> 00:44:49,360
with the team that wanted to push this out. In

817
00:44:49,440 --> 00:44:51,280
the ser side of things, I keep saying, you have

818
00:44:51,320 --> 00:44:53,719
to understand like the tier one, Tier two, Tier three,

819
00:44:53,960 --> 00:44:56,719
and this was getting deployed to like tier one, So

820
00:44:56,760 --> 00:44:57,639
this is like getting.

821
00:44:57,400 --> 00:45:01,000
Speaker 6: Deployed at the edge. So if this is wrong, like.

822
00:45:02,400 --> 00:45:04,800
Speaker 4: There's no like the quest the request doesn't go through,

823
00:45:05,480 --> 00:45:08,000
like it doesn't even it's not even like a latency issue,

824
00:45:08,000 --> 00:45:11,760
like it's it's dead. Yeah, it's I don't know. I

825
00:45:12,000 --> 00:45:13,360
see Lauren like shaking his head.

826
00:45:13,440 --> 00:45:17,079
Speaker 2: So I was gonna say past trauma, like actually will

827
00:45:17,079 --> 00:45:19,239
and I will, and I did an episode last year

828
00:45:19,599 --> 00:45:23,440
about how to handle scaling around the holidays, because you

829
00:45:23,480 --> 00:45:26,440
know it's a time, especially for companies that do e

830
00:45:26,440 --> 00:45:28,880
commerce et cetera, anything with orders, there is a huge

831
00:45:28,920 --> 00:45:31,400
spike and how to be prepared for that. And it's

832
00:45:31,400 --> 00:45:34,719
sort of interesting that if you break down the systems

833
00:45:34,719 --> 00:45:36,920
thinking model you brought it up, there is this perverse

834
00:45:36,960 --> 00:45:40,280
incentive where if you have a deadline, that you are

835
00:45:40,320 --> 00:45:43,360
more likely to rush to meet that deadline, which means

836
00:45:43,360 --> 00:45:46,000
that you cut more corners, which means the thing is

837
00:45:46,039 --> 00:45:48,159
more risky. And so if you create a deadline or

838
00:45:48,199 --> 00:45:51,639
a code freeze to prevent risky code from getting into production,

839
00:45:51,719 --> 00:45:55,039
you are actually encouraging the exact thing that you are

840
00:45:55,079 --> 00:45:55,840
trying to prevent.

841
00:45:56,719 --> 00:45:59,719
Speaker 4: So and like also on that end, and this is

842
00:45:59,719 --> 00:46:02,280
where like it, I guess it does like you start

843
00:46:02,320 --> 00:46:07,159
to like traumatic events and you that is usually the case.

844
00:46:07,320 --> 00:46:11,239
I've also seen the case of we do a code

845
00:46:11,280 --> 00:46:16,719
freeze and a system doesn't get touched for six months,

846
00:46:16,760 --> 00:46:22,719
maybe even a year, and then you decide that you

847
00:46:23,079 --> 00:46:26,400
do want to deploy like a new feature, and boom

848
00:46:26,559 --> 00:46:30,039
it's down and hard down because nobody has touched it

849
00:46:30,079 --> 00:46:35,000
in a year. And so like this certain situation that

850
00:46:35,079 --> 00:46:37,559
I ran into months we had like a reindexing job

851
00:46:38,079 --> 00:46:41,639
and there was so much data to reindex that what

852
00:46:42,079 --> 00:46:44,960
that the machines that it had been running on could

853
00:46:45,039 --> 00:46:49,400
no longer support the reindex process. So we're like screwed

854
00:46:49,440 --> 00:46:54,920
at this point. So yes, so you have to wait,

855
00:46:55,920 --> 00:47:00,840
and there's trade offs in all regards, and as I sorry,

856
00:47:00,960 --> 00:47:03,719
like it's just really your job to try to understand

857
00:47:03,760 --> 00:47:06,880
like not even like what's the perfect scenario, but like

858
00:47:06,960 --> 00:47:10,440
what's the least of like the evils here?

859
00:47:10,599 --> 00:47:14,440
Speaker 2: He dos actually is a really interesting guide for resilience engineering,

860
00:47:14,480 --> 00:47:16,239
and one of the things that they talk a lot

861
00:47:16,239 --> 00:47:19,840
about is what happens when there is like one or

862
00:47:19,880 --> 00:47:23,679
two points of failure, what actual new systems start coming

863
00:47:23,679 --> 00:47:25,760
into play, What does the request volume start to look

864
00:47:25,800 --> 00:47:30,159
like versus the nominal state? And you mentioned cash is

865
00:47:30,159 --> 00:47:31,280
at the edge, and I think one of the most

866
00:47:31,280 --> 00:47:34,199
common scenarios is why do you add a cash well

867
00:47:34,239 --> 00:47:36,480
because you're trying to reduce the load on your system,

868
00:47:37,039 --> 00:47:39,639
which means that you create the ability to create higher

869
00:47:39,679 --> 00:47:41,800
load on your system to hit the cash and as

870
00:47:41,840 --> 00:47:44,400
soon as the cash you know, drops or gets busted

871
00:47:44,400 --> 00:47:46,760
because you're using something like valky because no one's using

872
00:47:46,760 --> 00:47:50,840
Reddus anymore. Of course, which isn't designed to be persistent.

873
00:47:51,000 --> 00:47:53,000
It's going to drop at some point, which means you're

874
00:47:53,000 --> 00:47:54,960
going to get a lot of load to your system

875
00:47:55,079 --> 00:47:57,000
without going through the cash and it's going to cause

876
00:47:57,000 --> 00:47:58,400
a hard failure at that point, and how are you

877
00:47:58,440 --> 00:48:01,079
going to recover? And so it sounds great to add

878
00:48:01,119 --> 00:48:02,519
a cash in, but then you're gonna have to deal

879
00:48:02,559 --> 00:48:05,360
with this quite critical scenario where you are just instead

880
00:48:05,360 --> 00:48:08,360
of being able to handle any number requests, you're going

881
00:48:08,400 --> 00:48:09,400
to be able to handle nothing.

882
00:48:09,920 --> 00:48:12,800
Speaker 4: Yeah, back to like what you're saying the distributed system stuff,

883
00:48:12,840 --> 00:48:17,159
Like I in my mind you brought up like single

884
00:48:17,199 --> 00:48:21,760
point of failure and it's just like the nature of

885
00:48:21,800 --> 00:48:23,639
how things work, like you're always going to have a

886
00:48:23,679 --> 00:48:26,079
single point of failure. And it's not so much like

887
00:48:26,119 --> 00:48:27,800
how do we get rid of this single point of failure,

888
00:48:27,800 --> 00:48:31,440
because I don't even think that's possible, because you're going

889
00:48:31,519 --> 00:48:33,599
to have like traffic management at some point, Like what

890
00:48:33,639 --> 00:48:38,280
if the traffic management fails? It's more so in my mind,

891
00:48:38,440 --> 00:48:43,320
like what is what system seems to be the most

892
00:48:43,360 --> 00:48:46,239
resilient and that's where we want to place this single

893
00:48:46,280 --> 00:48:49,719
point of failure. So like usually it is like the distribution,

894
00:48:50,039 --> 00:48:52,400
Like if you have like a cash distribution, that is

895
00:48:52,519 --> 00:48:56,280
usually the in my opinion, like your wisest single point

896
00:48:56,280 --> 00:48:56,760
of failure.

897
00:48:57,280 --> 00:48:59,599
Speaker 2: This is an interesting way of phrasing and I haven't

898
00:48:59,639 --> 00:49:00,880
heard that perspective before.

899
00:49:01,599 --> 00:49:04,199
Speaker 1: What's your on call rotation look like?

900
00:49:07,320 --> 00:49:10,840
Speaker 4: So it's super different based on the company. I am

901
00:49:10,840 --> 00:49:16,559
a big fan of companies who are hopefully you work

902
00:49:16,599 --> 00:49:19,719
at a company that's large enough that they have engineers

903
00:49:19,760 --> 00:49:22,760
across the globe and so your on call rotation can

904
00:49:22,960 --> 00:49:27,320
fall within the hours that allow you to uh work

905
00:49:27,400 --> 00:49:32,639
optimally and not be up all night. But like I said,

906
00:49:32,880 --> 00:49:36,360
I also firmly believe like everybody should share in the

907
00:49:36,480 --> 00:49:39,679
on call rotation because it shares ownership and if things

908
00:49:39,719 --> 00:49:41,880
start to go awry, then you have a vested interest

909
00:49:41,880 --> 00:49:45,199
in making things better. But the on call rotation like

910
00:49:45,360 --> 00:49:48,079
is it? It is what it is, and you know

911
00:49:48,119 --> 00:49:49,639
it can be a little rough at the beginning, but

912
00:49:49,719 --> 00:49:53,559
it's a great way to learn honestly, like just dive

913
00:49:53,599 --> 00:49:57,760
in because there's nobody knows everything in the system. Like

914
00:49:57,960 --> 00:50:00,079
you're going to hit something that probably somebody has and

915
00:50:00,199 --> 00:50:00,679
hit before.

916
00:50:01,159 --> 00:50:04,199
Speaker 2: When you say everyone, like, what's the size of the

917
00:50:04,440 --> 00:50:06,800
org uh that you're thinking about there?

918
00:50:07,760 --> 00:50:08,000
Speaker 5: You know.

919
00:50:08,159 --> 00:50:10,920
Speaker 4: I've been in teams where, like my own conrotation honestly

920
00:50:11,000 --> 00:50:13,760
was like every other week, which is a little rough.

921
00:50:13,880 --> 00:50:16,199
I've been on teams where it's I want to say,

922
00:50:16,280 --> 00:50:18,880
like maybe once a month or something like that. I

923
00:50:18,880 --> 00:50:24,159
feel like that's pretty reasonable. I think it's helpful to have,

924
00:50:24,239 --> 00:50:27,000
like I said, like the tears of support also to

925
00:50:27,400 --> 00:50:32,360
like hopefully it doesn't get to this point, but it's

926
00:50:32,400 --> 00:50:35,239
definitely a real thing. I've seen, like with myself and others,

927
00:50:35,280 --> 00:50:38,400
where you're on an issue for twenty four or forty

928
00:50:38,440 --> 00:50:41,400
hours straight and you need to hand that off because

929
00:50:41,440 --> 00:50:44,079
after a certain amount of time, like you're just your

930
00:50:44,079 --> 00:50:51,280
brain dead, or like you need to bathe, right, I

931
00:50:51,320 --> 00:50:53,480
mean I guess like if I think of like military

932
00:50:53,559 --> 00:50:55,679
or something like, you can go a long time without bathing.

933
00:50:55,719 --> 00:51:00,360
Speaker 6: So maybe that's not the best example. Like you need to.

934
00:51:00,360 --> 00:51:02,960
Speaker 1: Sleep, especially in a remote workforce culture.

935
00:51:04,079 --> 00:51:06,599
Speaker 4: Yeah, like you need sleep to be able to like

936
00:51:07,159 --> 00:51:11,400
debug things, but there should usually I feel like in

937
00:51:11,519 --> 00:51:14,320
a proper on call rotation, like a handoff phase, if

938
00:51:14,320 --> 00:51:16,800
it's been like more than twenty four hours something.

939
00:51:16,519 --> 00:51:19,280
Speaker 2: Like that, yeah, I will. I think follow the sun

940
00:51:19,440 --> 00:51:21,199
implies that you, like you hand it off to someone

941
00:51:21,239 --> 00:51:23,480
who's still awake at that time and not you. You

942
00:51:23,599 --> 00:51:25,840
just stay awake until the sun comes up.

943
00:51:27,679 --> 00:51:31,960
Speaker 4: That that, in my experience, it typically works the best.

944
00:51:32,400 --> 00:51:34,280
Speaker 2: So this is this may be a little bit of

945
00:51:34,320 --> 00:51:39,000
a tangent, but since you're from the JavaScript Java Jabba podcast,

946
00:51:39,079 --> 00:51:42,480
I have to ask you preference no JS, DNO or BOND.

947
00:51:43,239 --> 00:51:47,079
Speaker 4: I'm old and boring, so I like nodes still No.

948
00:51:47,400 --> 00:51:51,079
I've talked to like the Dino team and like I

949
00:51:51,079 --> 00:51:52,400
I don't have anything against that.

950
00:51:52,519 --> 00:51:54,559
Speaker 6: I just I'm old and boring.

951
00:51:54,280 --> 00:51:56,760
Speaker 2: And I shouldn't say I owe Jos.

952
00:51:57,880 --> 00:52:00,079
Speaker 3: I still like Pearl, so like, I agree.

953
00:52:00,360 --> 00:52:02,639
Speaker 1: I think it's just fitting that she said I like Pearl,

954
00:52:02,679 --> 00:52:04,320
and the rest of the audio cut out.

955
00:52:04,119 --> 00:52:07,559
Speaker 2: After that, The said part is like it's still recording

956
00:52:07,599 --> 00:52:10,079
for her, so we'll find out when the episode released

957
00:52:10,119 --> 00:52:11,519
was She actually said.

958
00:52:11,400 --> 00:52:13,320
Speaker 3: It's only for this podcast, you guys that I have

959
00:52:13,360 --> 00:52:15,840
so many technical problems, Like I'm fine in all of

960
00:52:15,880 --> 00:52:16,760
my other meetings.

961
00:52:16,800 --> 00:52:20,639
Speaker 4: It's just this'll be doing something right now.

962
00:52:21,000 --> 00:52:22,079
Speaker 2: I believe you. Jillian.

963
00:52:22,519 --> 00:52:23,039
Speaker 3: That's good.

964
00:52:23,320 --> 00:52:24,079
Speaker 1: I have no comment.

965
00:52:26,360 --> 00:52:28,440
Speaker 3: That's probably for the best quite possibly.

966
00:52:28,880 --> 00:52:31,639
Speaker 2: How do you deal with the complexity of having to

967
00:52:31,679 --> 00:52:34,599
share so much knowledge with so many people? So the

968
00:52:34,639 --> 00:52:38,320
more people that are working in solution in an area

969
00:52:38,360 --> 00:52:40,920
and a team or means there's more things that are

970
00:52:40,960 --> 00:52:43,159
going to be created, which means there's more things everyone

971
00:52:43,199 --> 00:52:47,000
will have to be aware of to potentially deal with

972
00:52:47,039 --> 00:52:49,960
when an on call incident comes up. Is there a

973
00:52:50,000 --> 00:52:53,239
good metric or litmus test to figure out what the

974
00:52:53,280 --> 00:52:56,760
maximum amount of stuff is that a team can actually manage.

975
00:52:57,239 --> 00:53:00,159
Speaker 4: Yeah, honestly, that is a very good question, and I

976
00:53:00,840 --> 00:53:02,840
feel like I don't have a good answer, but now

977
00:53:02,880 --> 00:53:05,760
that's something I want to go see if I can

978
00:53:05,880 --> 00:53:11,199
dig into more. But I also say, like how we

979
00:53:11,280 --> 00:53:15,760
have like these different like tiers of support. It's hard

980
00:53:15,880 --> 00:53:18,159
if you get into like a more senior role because

981
00:53:18,199 --> 00:53:21,440
stuff is going to bubble up to you, but it's

982
00:53:21,519 --> 00:53:25,000
usually helpful. Like in a large team, there's usually even

983
00:53:25,079 --> 00:53:28,039
like slas to the developer team. So even like a

984
00:53:28,079 --> 00:53:33,280
lower environment is going to like be held accountable to

985
00:53:33,280 --> 00:53:35,199
the business to keep a lower environment up in.

986
00:53:35,199 --> 00:53:36,119
Speaker 6: A certain amount of time.

987
00:53:38,119 --> 00:53:41,920
Speaker 4: Like sort of related to your question, like having those

988
00:53:41,960 --> 00:53:44,639
types of people like work in those environments and that

989
00:53:44,719 --> 00:53:47,400
helps distribute knowledge of like how these different systems work

990
00:53:47,480 --> 00:53:51,719
without being tasked with like the actual production environment, and

991
00:53:51,760 --> 00:53:55,920
then if they do well in those lower environments, and

992
00:53:55,960 --> 00:53:58,360
then they can kind of move on to a production environment,

993
00:53:58,400 --> 00:54:00,239
and that helps spread the nolle.

994
00:54:00,400 --> 00:54:04,840
Speaker 2: Pray that's prey. I mean, I think there's a there's

995
00:54:04,840 --> 00:54:07,400
a corollary here where I often have to advise some

996
00:54:07,440 --> 00:54:10,239
companies on which is we have like five different mobile

997
00:54:10,239 --> 00:54:12,760
apps because you know, there's many different providers out there,

998
00:54:12,800 --> 00:54:17,000
and then there's a website and also web native version

999
00:54:17,159 --> 00:54:19,440
for your mobile app that isn't like an app version

1000
00:54:19,480 --> 00:54:22,320
but is responsive like and then there's back and that

1001
00:54:22,360 --> 00:54:24,920
goes with it. And the infrastructure is that all one team,

1002
00:54:25,079 --> 00:54:27,480
you know, so everyone is cross functional and understand what's

1003
00:54:27,480 --> 00:54:29,639
going on. Or do you have like ten teams that

1004
00:54:29,760 --> 00:54:32,039
each have the exact same functionality. And I feel like

1005
00:54:32,079 --> 00:54:35,880
the answer keeps on being well, they're both wrong and

1006
00:54:35,960 --> 00:54:38,000
there's no there's no good solution there, and you just

1007
00:54:38,000 --> 00:54:39,840
got to have to deal with the consequences.

1008
00:54:40,039 --> 00:54:40,719
Speaker 6: It's awesome.

1009
00:54:40,840 --> 00:54:43,320
Speaker 4: I mean, this is why I did startups for a

1010
00:54:43,360 --> 00:54:48,639
long time. But as I've gotten like further into my career,

1011
00:54:48,840 --> 00:54:53,239
I feel like it's hard to make an impact unless

1012
00:54:53,280 --> 00:54:55,800
you've been at the company for really for a good

1013
00:54:55,840 --> 00:54:58,760
amount of time because it the larger the code base

1014
00:54:58,840 --> 00:55:01,280
and like the domain than and you really do need

1015
00:55:01,280 --> 00:55:05,239
that time to understand it, whereas like a startup usually

1016
00:55:05,280 --> 00:55:08,760
like a smaller application, and you can understand it more.

1017
00:55:08,800 --> 00:55:11,960
But like a lot of these like larger companies, they

1018
00:55:11,960 --> 00:55:14,280
have like these huge systems and it takes time to

1019
00:55:14,480 --> 00:55:15,199
understand it.

1020
00:55:16,800 --> 00:55:19,840
Speaker 2: That's a really interesting challenge. Then we see in the

1021
00:55:19,840 --> 00:55:24,480
tech industry the average tenure for engineers reducing over time. Yeah,

1022
00:55:24,599 --> 00:55:27,800
does that mean that we're actually more and more incapable

1023
00:55:27,880 --> 00:55:30,960
of building larger systems because people aren't staying at the

1024
00:55:30,960 --> 00:55:34,440
company to be able to diagnose and build more resilient systems.

1025
00:55:34,480 --> 00:55:36,519
Are we still attempting to do that and the systems

1026
00:55:36,559 --> 00:55:39,679
we're building just aren't resilient anymore? Or is this just

1027
00:55:39,800 --> 00:55:42,960
like you see whatever problems are close to you, and

1028
00:55:43,360 --> 00:55:46,920
larger companies with bigger solutions are doing a good job

1029
00:55:46,960 --> 00:55:51,760
at retaining experienced staff that have experience in their own

1030
00:55:51,760 --> 00:55:53,920
solution in order to help solve those challenges.

1031
00:55:54,360 --> 00:55:58,159
Speaker 4: It's it's very tricky because yeah, there's like a trade

1032
00:55:58,199 --> 00:56:01,039
off both ways, Like you stay a company long enough,

1033
00:56:01,079 --> 00:56:03,400
like yes, you might kind of have some like siloed

1034
00:56:03,400 --> 00:56:08,159
information and start to not know how other people are

1035
00:56:08,159 --> 00:56:11,239
solving problems, and then that kind of can produce like

1036
00:56:11,320 --> 00:56:15,719
some blind spots in your knowledge. But at the same time,

1037
00:56:15,960 --> 00:56:18,719
like if you don't stay at a place long enough,

1038
00:56:19,719 --> 00:56:23,559
it can be hard to see like how this huge

1039
00:56:23,599 --> 00:56:24,880
system interacts and.

1040
00:56:24,920 --> 00:56:25,599
Speaker 6: Be able to.

1041
00:56:27,039 --> 00:56:29,960
Speaker 4: Really solve like these harder problems because it takes a

1042
00:56:29,960 --> 00:56:32,559
while to like get the whole domain, the whole system

1043
00:56:32,639 --> 00:56:37,119
in your head. It's a there's definitely like like a

1044
00:56:37,239 --> 00:56:40,199
vested interest I feel like in these companies and like retaining.

1045
00:56:39,800 --> 00:56:42,519
Speaker 2: People, well, yeah, for sure. I mean there's the benefits

1046
00:56:42,519 --> 00:56:44,679
to them and not to the individual and a lot

1047
00:56:44,679 --> 00:56:46,360
of times, I mean, I guess, you know, maybe a

1048
00:56:46,360 --> 00:56:47,840
different way of phrasing this is if we look at

1049
00:56:47,880 --> 00:56:50,480
Dumbar's number and ORG is going to be at max

1050
00:56:50,519 --> 00:56:53,280
like one hundred and fifty people, how many years of

1051
00:56:53,360 --> 00:56:57,920
experience working in that company do we project someone would

1052
00:56:58,039 --> 00:57:00,199
a new hire would have to work before they or

1053
00:57:00,840 --> 00:57:03,360
at the point where they are delivering or understanding the

1054
00:57:03,400 --> 00:57:06,679
system sufficiently that one hundred and fifty person ORG has

1055
00:57:06,719 --> 00:57:08,119
created any thoughts on that.

1056
00:57:08,119 --> 00:57:12,000
Speaker 4: That's an interesting question too, like how it kind of

1057
00:57:12,039 --> 00:57:15,599
like scales based on the size and the system and

1058
00:57:15,679 --> 00:57:16,519
how long you'd have.

1059
00:57:16,519 --> 00:57:19,159
Speaker 2: To be there, but a personal preference, you know, like

1060
00:57:19,280 --> 00:57:21,360
oh yeah, once I'm there, you know, six months, I

1061
00:57:21,360 --> 00:57:24,880
feel like I'm now somewhat accountable for every system or

1062
00:57:25,119 --> 00:57:27,000
a year or two years or something like that.

1063
00:57:27,480 --> 00:57:32,440
Speaker 4: It probably this doesn't answer your question, unfortunately, but I

1064
00:57:32,440 --> 00:57:35,440
would say like one of the things that could be

1065
00:57:35,519 --> 00:57:40,159
helpful is like if you start to like segment your

1066
00:57:40,159 --> 00:57:43,280
career working in a similar domain, so some of that

1067
00:57:43,400 --> 00:57:48,280
knowledge is transferable, like whether you're talking about like the

1068
00:57:48,360 --> 00:57:52,960
hyperscaler that you're working on, or if you're more focused

1069
00:57:53,000 --> 00:57:57,400
on like CDM distribution, and maybe that's valuable to other companies,

1070
00:57:57,400 --> 00:58:00,119
whereas like if it's a you know, if it's a

1071
00:58:00,920 --> 00:58:05,199
paylalled type of application, that maybe your CDN experience is

1072
00:58:05,239 --> 00:58:06,639
not as beneficial there.

1073
00:58:07,360 --> 00:58:09,159
Speaker 1: I think also when you say one hundred and fifty

1074
00:58:09,199 --> 00:58:11,800
person or you refer into like the entire company or

1075
00:58:11,920 --> 00:58:13,480
just engineering.

1076
00:58:13,280 --> 00:58:15,679
Speaker 2: I mean even a subset of engineering potentially, like I

1077
00:58:15,760 --> 00:58:17,960
just like at Dunbar's number, like your teams are going

1078
00:58:18,039 --> 00:58:20,280
to be like two pizza teams for instance, that are

1079
00:58:20,360 --> 00:58:22,519
cross functional. But then there are different types of teams

1080
00:58:22,519 --> 00:58:25,639
and different features within the product or product suite that

1081
00:58:25,679 --> 00:58:28,039
you're delivering, and that can only get up to one

1082
00:58:28,079 --> 00:58:30,400
hundred and fifty. Above that you need to create a

1083
00:58:30,440 --> 00:58:32,920
second org and have them work on a completely different

1084
00:58:32,960 --> 00:58:35,519
feature set with a completely different focus. You know, you

1085
00:58:35,519 --> 00:58:37,440
think of like how in a hyperscale or I'm going

1086
00:58:37,480 --> 00:58:40,599
to use Google Workspace as an example, like email, Gmail

1087
00:58:40,679 --> 00:58:44,000
and drive, right, I can imagine those being separate, completely

1088
00:58:44,039 --> 00:58:47,360
separate orgs for instance, But I would not expect one

1089
00:58:47,559 --> 00:58:51,280
org structure where it's fundamentally accountable for every single product

1090
00:58:51,639 --> 00:58:54,480
in Google Workspace. There's just too much stuff there. So

1091
00:58:55,280 --> 00:58:57,119
that's the part for me that makes the most sense,

1092
00:58:57,159 --> 00:58:58,760
that you can't have more than one hundred and fifty

1093
00:58:58,800 --> 00:59:01,519
people working on the same product because there's only so

1094
00:59:01,599 --> 00:59:04,119
much functional work there, and at a larger size you

1095
00:59:04,159 --> 00:59:07,639
have a communication breakdown. You can't pass the information effectively.

1096
00:59:07,679 --> 00:59:10,960
So that's just sort of the metrics on the math

1097
00:59:11,119 --> 00:59:12,239
that seem to work out.

1098
00:59:12,559 --> 00:59:15,000
Speaker 4: Too many cooks in the kitchen, and like I think

1099
00:59:15,400 --> 00:59:17,599
for better where some people have like a vested interest

1100
00:59:17,639 --> 00:59:20,519
in like trying to offer value, but unfortunately some there's

1101
00:59:20,519 --> 00:59:21,519
the values not there.

1102
00:59:22,400 --> 00:59:24,480
Speaker 2: Yeah, I mean even to your point of like, well,

1103
00:59:24,559 --> 00:59:26,960
let's have an on call rotation in everyone and say

1104
00:59:26,960 --> 00:59:29,719
the team in the ORG could participate in right, one

1105
00:59:29,760 --> 00:59:32,440
hundred and fifty people means you're on call for your

1106
00:59:32,480 --> 00:59:37,639
product once every three years, right, So that doesn't work out, right,

1107
00:59:37,679 --> 00:59:39,440
So you know if your August fifty people, then you

1108
00:59:39,480 --> 00:59:41,920
get to the okay, once a year, I'm on a call. Well,

1109
00:59:41,960 --> 00:59:44,679
that maybe works out potentially if you have a low

1110
00:59:44,800 --> 00:59:48,000
number of bugs. You deal with real time applications that

1111
00:59:48,239 --> 00:59:50,639
can never really be down, highly resilient, and you run

1112
00:59:51,159 --> 00:59:55,000
you're probably running fire drills during during working hours as

1113
00:59:55,039 --> 00:59:57,800
if there were an incident to know how to respond

1114
00:59:57,800 --> 01:00:00,880
and not just hopefully waiting and never seeing a problem.

1115
01:00:01,440 --> 01:00:04,760
Speaker 4: I kind of like, I feel like computer systems like

1116
01:00:05,239 --> 01:00:07,760
mirror a lot of like real life. So whether it

1117
01:00:07,800 --> 01:00:10,920
be like I people who don't understand astory, they're kind

1118
01:00:10,960 --> 01:00:12,199
of like, what in the world do you do? I'm like,

1119
01:00:12,280 --> 01:00:17,519
I'm basically like an emergency room doctor, and like somebody

1120
01:00:17,559 --> 01:00:19,840
comes into the emergency room and my responsibility is like

1121
01:00:19,920 --> 01:00:23,719
keep them alive until the right person can come in,

1122
01:00:23,800 --> 01:00:27,239
and you know, get them into like critical condition, not.

1123
01:00:27,599 --> 01:00:31,280
Speaker 2: Even not even the surge end, but you know, high level.

1124
01:00:31,400 --> 01:00:34,199
I understand how how patients work and the sets of

1125
01:00:34,239 --> 01:00:36,360
problems that can happen, and I can direct people on

1126
01:00:36,400 --> 01:00:37,159
the right places.

1127
01:00:37,320 --> 01:00:39,199
Speaker 1: Yeah, the wet stuff on the inside and this dry

1128
01:00:39,239 --> 01:00:40,119
stuff on the outside.

1129
01:00:40,960 --> 01:00:42,159
Speaker 5: I love it.

1130
01:00:44,480 --> 01:00:47,079
Speaker 2: You said, you said nuclear physics. Well I don't think

1131
01:00:47,079 --> 01:00:52,360
you said E M T right right right, Yeah, true, true, true.

1132
01:00:52,559 --> 01:00:57,440
Speaker 1: But I do spend not to just qualifies my opinion

1133
01:00:57,760 --> 01:00:59,599
by any means, but I do spend a lot of

1134
01:00:59,639 --> 01:01:04,000
time out in the wilderness, and like the wilderness, the

1135
01:01:04,039 --> 01:01:08,800
wilderness first aid stuff I've been exposed to is pretty

1136
01:01:08,840 --> 01:01:11,199
much that keep the wet stuff on the inside, dry

1137
01:01:11,199 --> 01:01:12,239
stuff on the outside.

1138
01:01:12,360 --> 01:01:19,519
Speaker 2: Okay, so now I'm imagining bear grills. I don't out

1139
01:01:19,599 --> 01:01:22,400
back are you know, frequently known on the internet means

1140
01:01:22,400 --> 01:01:25,880
for drinking his own You're unnecessarily.

1141
01:01:27,440 --> 01:01:29,719
Speaker 1: But the likes he got, that's the metric.

1142
01:01:31,119 --> 01:01:32,880
Speaker 2: You gotta wonder what he was really going for. Was

1143
01:01:32,920 --> 01:01:35,639
he going for, you know, teaching survival skills or was it,

1144
01:01:35,719 --> 01:01:38,920
you know, just to become a paid influencer.

1145
01:01:39,639 --> 01:01:43,039
Speaker 1: I'm pretty sure he's got a mortgage to pay.

1146
01:01:43,880 --> 01:01:48,360
Speaker 6: Yeah, so.

1147
01:01:50,239 --> 01:01:54,320
Speaker 1: Someone who's interested in pursuing a career in s E.

1148
01:01:55,320 --> 01:01:59,880
Give us like your top rundown pros and cons and

1149
01:02:00,960 --> 01:02:03,760
things you'd wish you things you wish you had known

1150
01:02:04,039 --> 01:02:05,440
before ending up here.

1151
01:02:05,559 --> 01:02:07,599
Speaker 5: There's a lot of different paths that you can take

1152
01:02:07,639 --> 01:02:08,400
into s E.

1153
01:02:09,079 --> 01:02:09,599
Speaker 6: One thing I.

1154
01:02:09,519 --> 01:02:11,559
Speaker 4: Didn't really touch on, and I know we're close on time,

1155
01:02:11,679 --> 01:02:13,280
so I won't go into it too much is like

1156
01:02:13,360 --> 01:02:17,360
a performance engineer. That's very important for an s E

1157
01:02:17,440 --> 01:02:20,239
team to do performance testing all these kinds of things.

1158
01:02:20,559 --> 01:02:23,159
That's a huge part of s E. Is like resource allocation,

1159
01:02:24,039 --> 01:02:28,119
and you need performance testing in place to better make

1160
01:02:28,159 --> 01:02:32,639
those decisions. So performance engineering, like I can say, like

1161
01:02:32,639 --> 01:02:35,320
from my background, like coming in from application engineering, there's

1162
01:02:35,320 --> 01:02:38,719
a lot of value there. Could you understand like how

1163
01:02:38,760 --> 01:02:41,320
these micro how these microserver systems.

1164
01:02:40,920 --> 01:02:41,599
Speaker 6: Like work together.

1165
01:02:42,599 --> 01:02:44,800
Speaker 4: And then like the systems in Ino your background has

1166
01:02:44,840 --> 01:02:47,320
like an ability to debug at like the network level

1167
01:02:47,360 --> 01:02:50,000
really well and things like that, and all of those

1168
01:02:50,000 --> 01:02:54,039
things come in and offer value to an s R team.

1169
01:02:55,079 --> 01:02:59,840
I guess things I wish I would have known kind

1170
01:02:59,840 --> 01:03:01,800
of like we talked about the beginning of the episode,

1171
01:03:01,840 --> 01:03:06,280
I I'm like very much like go go go, and

1172
01:03:07,039 --> 01:03:09,960
like early in my career, everybody was like, you're gonna burn.

1173
01:03:09,840 --> 01:03:12,199
Speaker 6: Out, like slow down. I was like impossible.

1174
01:03:15,119 --> 01:03:18,360
Speaker 4: And I still say, like I didn't burn out because

1175
01:03:18,360 --> 01:03:22,400
I still absolutely love it. But I really did not

1176
01:03:23,000 --> 01:03:25,920
realize that like it actually did start to like the

1177
01:03:26,000 --> 01:03:29,199
lack of sleep just didn't start to kind of like

1178
01:03:29,239 --> 01:03:31,840
take a toll on my actual health a little bit.

1179
01:03:33,119 --> 01:03:37,920
So like lesson learned to myself is like, be be

1180
01:03:38,119 --> 01:03:42,880
cognizant of like your personality and like your you know what,

1181
01:03:43,000 --> 01:03:46,079
some of your weaknesses might be there, because like for me,

1182
01:03:46,440 --> 01:03:48,079
I absolutely love what I do and I want to

1183
01:03:48,079 --> 01:03:49,760
be able to do this for a very long time.

1184
01:03:49,920 --> 01:03:52,199
So I kind of came to this realization that I

1185
01:03:52,280 --> 01:03:56,400
was like whoa, like if you don't put it, and management,

1186
01:03:56,480 --> 01:03:58,880
like my manager was fine, like he would encourage me

1187
01:03:58,960 --> 01:04:02,039
to do more of it, not like my director like

1188
01:04:02,159 --> 01:04:05,039
same thing, like dude, like you you need to take

1189
01:04:05,079 --> 01:04:08,920
some time off. So like my advice would be like,

1190
01:04:08,960 --> 01:04:10,960
if you're more on that side of the spectrum, like

1191
01:04:11,079 --> 01:04:15,440
know yourself and allow yourself that time because it will

1192
01:04:15,559 --> 01:04:17,639
make you better in the long run.

1193
01:04:19,119 --> 01:04:20,360
Speaker 6: Other things.

1194
01:04:22,559 --> 01:04:25,239
Speaker 4: I would just say, like if it sounds interesting to you,

1195
01:04:25,280 --> 01:04:28,280
like don't shy away from it because it is I

1196
01:04:28,280 --> 01:04:32,719
feel like a huge there's like a huge surface of

1197
01:04:33,400 --> 01:04:38,880
areas you can tackle. But if I'm always like of

1198
01:04:38,960 --> 01:04:42,960
the mindset like anybody can learn anything obviously, like some

1199
01:04:43,000 --> 01:04:45,280
people are like more technically talented, they come in with

1200
01:04:45,320 --> 01:04:46,360
more experience.

1201
01:04:46,000 --> 01:04:46,719
Speaker 6: Those sorts of things.

1202
01:04:46,719 --> 01:04:48,320
Speaker 4: But if it's interesting to you, like and you have

1203
01:04:48,440 --> 01:04:49,119
the drive like.

1204
01:04:49,159 --> 01:04:49,760
Speaker 6: Go for it.

1205
01:04:50,639 --> 01:04:53,079
Speaker 4: Those would be my biggest like two takeaways and.

1206
01:04:53,079 --> 01:04:55,440
Speaker 2: Through the power of MS now you now you two,

1207
01:04:55,519 --> 01:04:58,760
everyone can be an expert in whatever topic you know

1208
01:04:59,440 --> 01:05:03,800
you fancy. I mean listening to you talking to me

1209
01:05:03,840 --> 01:05:05,920
about this, it really makes me think. Like you say,

1210
01:05:06,039 --> 01:05:07,800
you really have to care a lot, But caring a

1211
01:05:07,840 --> 01:05:11,280
lot means understanding the business and gives you an edge

1212
01:05:11,320 --> 01:05:14,039
over everyone else who may not care as much. They

1213
01:05:14,079 --> 01:05:16,360
won't give them a drive to really dive in and

1214
01:05:16,400 --> 01:05:18,480
be able to make the right decision. But then there's

1215
01:05:18,519 --> 01:05:20,480
the question that comes up of like if you care

1216
01:05:20,559 --> 01:05:24,679
too much, you're liable to burn out from doing that,

1217
01:05:25,199 --> 01:05:27,719
and you really need to know yourself. And that's a

1218
01:05:27,840 --> 01:05:30,039
tall order I think for a lot of people.

1219
01:05:30,440 --> 01:05:33,639
Speaker 4: Yeah, Like it was very it was a challenge for

1220
01:05:33,719 --> 01:05:38,159
me to be like you can't be the hero twenty

1221
01:05:38,199 --> 01:05:40,960
four to seven or like you are not going to

1222
01:05:41,000 --> 01:05:45,000
be able to like sustain this, but there's.

1223
01:05:44,599 --> 01:05:47,000
Speaker 2: Just one more fire over there that you know how

1224
01:05:47,000 --> 01:05:47,559
to deal with.

1225
01:05:48,039 --> 01:05:50,800
Speaker 4: Oh lord, Yeah, I mean I guess I would say too,

1226
01:05:50,840 --> 01:05:53,000
Like I'm probably at a place now where the thing

1227
01:05:53,039 --> 01:05:55,119
that I did really enjoy is like I did get

1228
01:05:55,119 --> 01:05:57,719
to serve in more of like an engineering manager role

1229
01:05:57,920 --> 01:06:01,119
on the SR team towards the end my last role.

1230
01:06:01,320 --> 01:06:05,800
And you know, everybody is like driven by different things,

1231
01:06:05,800 --> 01:06:07,519
but you can kind of see people in the org

1232
01:06:07,639 --> 01:06:11,840
that have a lot of potential and starting to kind

1233
01:06:11,880 --> 01:06:13,800
of like mentor them and hand things off to them

1234
01:06:13,840 --> 01:06:14,760
and things like that can.

1235
01:06:14,679 --> 01:06:15,920
Speaker 6: Be very rewarding and.

1236
01:06:17,719 --> 01:06:20,079
Speaker 4: Really like the best thing you can do is this

1237
01:06:20,199 --> 01:06:22,960
sounds like so hand baby, but like fort multiplier, Like

1238
01:06:23,360 --> 01:06:27,039
you know, whether you're an actual manager or in a

1239
01:06:27,119 --> 01:06:30,679
senior role or anything like that, like what can you

1240
01:06:30,760 --> 01:06:35,599
do to help the company and the teammates like fall

1241
01:06:35,639 --> 01:06:36,639
into the pit of success?

1242
01:06:36,920 --> 01:06:41,119
Speaker 1: I like that analogy important leadership skill.

1243
01:06:41,239 --> 01:06:43,440
Speaker 2: Learn to delegate, Right, you have to.

1244
01:06:43,400 --> 01:06:45,360
Speaker 4: Be okay at a certain point in your career with

1245
01:06:45,400 --> 01:06:47,400
like not being the hero, because if you're in that

1246
01:06:47,519 --> 01:06:51,880
kind of role, like sometimes your value is more so

1247
01:06:52,079 --> 01:06:55,199
in enabling others and building out parts of the system

1248
01:06:55,239 --> 01:06:57,159
that like help others fall into the pit of success.

1249
01:06:57,199 --> 01:06:59,480
And if you kind of like are one of those

1250
01:06:59,480 --> 01:07:05,159
people that feels like they always need like the like recognition, like,

1251
01:07:05,760 --> 01:07:09,039
then like leadership type roles might not be the path

1252
01:07:09,119 --> 01:07:12,360
for you because a lot of the recognition is going

1253
01:07:12,400 --> 01:07:14,800
to be like hidden behind the scenes of enabling others.

1254
01:07:15,119 --> 01:07:17,719
Speaker 5: Yeah, but it's warm, cuzzy. You like the girl and he.

1255
01:07:17,719 --> 01:07:18,360
Speaker 6: Likes it, so.

1256
01:07:20,519 --> 01:07:22,840
Speaker 1: I agree with you. But it's a it's a completely

1257
01:07:22,840 --> 01:07:25,639
different high probably not the best analogy in the world,

1258
01:07:25,679 --> 01:07:28,239
but that's the one that just sits in my head.

1259
01:07:28,840 --> 01:07:31,360
Speaker 4: It's more so like I would I guess I would

1260
01:07:31,480 --> 01:07:35,440
almost equate it to like delayed gratification, Like you're not

1261
01:07:35,440 --> 01:07:38,320
going to get the like that a girl in the moment,

1262
01:07:38,440 --> 01:07:40,960
but it might come like two years later when somebody

1263
01:07:40,960 --> 01:07:42,840
that you managed comes back to you and be like, man,

1264
01:07:42,960 --> 01:07:45,800
like you taught me this thing and I used it

1265
01:07:45,840 --> 01:07:47,559
today and thank you, and you're.

1266
01:07:47,400 --> 01:07:49,440
Speaker 6: Like really good.

1267
01:07:50,000 --> 01:07:54,360
Speaker 2: Right. It's interesting that you say that, though, because that

1268
01:07:54,519 --> 01:07:57,679
actually means that some of the most enjoyment you get

1269
01:07:57,719 --> 01:08:00,280
out of it if that's the right word. Is actually

1270
01:08:00,639 --> 01:08:04,559
seeing others execute effectively and seeing back on what you've

1271
01:08:04,599 --> 01:08:09,800
taught them, which indirectly has no impact on the business

1272
01:08:09,880 --> 01:08:12,000
in a way, especially if they leave and go somewhere else.

1273
01:08:12,000 --> 01:08:15,760
So you know, there is this weird paradox there.

1274
01:08:16,439 --> 01:08:16,720
Speaker 6: Yeah.

1275
01:08:16,800 --> 01:08:18,319
Speaker 5: Yeah, yeah, it's a little of both.

1276
01:08:18,479 --> 01:08:18,800
Speaker 6: Yeah.

1277
01:08:18,840 --> 01:08:21,720
Speaker 4: For sure, it's helping the business, and hopefully you help

1278
01:08:21,800 --> 01:08:23,720
other people on the team that help the business or

1279
01:08:23,920 --> 01:08:26,399
just go on and do good things themselves.

1280
01:08:26,880 --> 01:08:28,479
Speaker 2: I don't know if this is relevant, but this keeps

1281
01:08:28,520 --> 01:08:31,520
coming to my mind in this conversation. There's this SMBC

1282
01:08:31,680 --> 01:08:36,960
Saturday Morning Breakfast Cereal webcomic issue where Superman's like, how

1283
01:08:36,960 --> 01:08:41,159
can I deliver the most value possible? And you know,

1284
01:08:41,479 --> 01:08:42,800
the mayor of the city is like, well, you know

1285
01:08:42,840 --> 01:08:45,640
what actually saving people? Like you're pretty slow at that.

1286
01:08:45,760 --> 01:08:47,279
If we look at it, you know what would really

1287
01:08:47,319 --> 01:08:50,359
work is if we just had more energy and to

1288
01:08:50,399 --> 01:08:52,239
make this, you know, relevant for twenty twenty five, to

1289
01:08:52,319 --> 01:08:55,479
obviously power all of those lms to help people. And

1290
01:08:55,560 --> 01:08:58,399
so can you just turn this one giant crank really

1291
01:08:58,479 --> 01:09:03,760
fast as you can. That's just your whole life just there,

1292
01:09:03,920 --> 01:09:06,640
And like decades later, he's still turning the crack and like,

1293
01:09:06,680 --> 01:09:09,279
that's not a very rewarding experience, but that's what gives

1294
01:09:09,279 --> 01:09:13,920
the most value to UH civilization. And you know, one day,

1295
01:09:14,199 --> 01:09:17,960
the years later, the mayor wakes up and says, you

1296
01:09:17,960 --> 01:09:21,000
know what, actually, we don't need you anymore. We automated

1297
01:09:21,000 --> 01:09:24,159
your job away with nuclear fusion, and now we generate

1298
01:09:24,199 --> 01:09:27,199
a limited energy all the time, basically without doing anything.

1299
01:09:27,239 --> 01:09:28,159
You can just retire.

1300
01:09:30,119 --> 01:09:33,239
Speaker 4: Well, hopefully that doesn't happen you kind of You're always

1301
01:09:33,239 --> 01:09:36,479
wanting to invent new ways to make yourself valuable.

1302
01:09:36,520 --> 01:09:38,079
Speaker 6: That's part of being in tech.

1303
01:09:38,279 --> 01:09:40,760
Speaker 1: I don't see it's running out anytime soon, which is

1304
01:09:40,840 --> 01:09:44,600
good because I've got I've got a mortgage to cover, So.

1305
01:09:46,039 --> 01:09:48,840
Speaker 2: I don't know. I hate to have the dissenting opinion here.

1306
01:09:48,880 --> 01:09:52,479
I feel like there is just people making the same mistakes.

1307
01:09:52,800 --> 01:09:56,920
Speaker 4: Yeah that's true too, Yeah, the same, the same underlying mistakes.

1308
01:09:57,000 --> 01:09:58,520
Speaker 6: Yes, it's history repeating it.

1309
01:09:58,920 --> 01:10:01,399
Speaker 1: Yes, so warning, if you could just compile a list

1310
01:10:01,439 --> 01:10:02,239
of those.

1311
01:10:02,760 --> 01:10:06,039
Speaker 2: I tried, so I have tried to, you know, write

1312
01:10:06,079 --> 01:10:09,560
down so what happens frequently. I always thought I'd be

1313
01:10:09,600 --> 01:10:11,199
doing engineering, But what I do is I write blog

1314
01:10:11,239 --> 01:10:14,760
posts and I record podcast episodes, and I get triggered

1315
01:10:14,760 --> 01:10:17,479
sometimes while I'm doing other things in my life, talking

1316
01:10:17,479 --> 01:10:19,920
with my colleagues and whatnot, in the communities that I'm in,

1317
01:10:20,119 --> 01:10:22,279
so much so that I feel compelled to have to

1318
01:10:22,319 --> 01:10:24,680
write something. And so there is a list of things

1319
01:10:24,720 --> 01:10:29,319
out there of problems that I've seen and my recommendations

1320
01:10:29,359 --> 01:10:31,560
for it. But some of them are like really nuanced thing,

1321
01:10:31,680 --> 01:10:34,800
Like there's one particular little thing in security that everyone

1322
01:10:34,880 --> 01:10:37,000
seems to run into, and then there's like a forty

1323
01:10:37,079 --> 01:10:40,520
minute conference talk on it or a blog post on

1324
01:10:40,560 --> 01:10:42,119
it that I'm sure no one has ever read.

1325
01:10:44,199 --> 01:10:46,840
Speaker 1: Well, it sounds like a really good point. Put all

1326
01:10:46,840 --> 01:10:50,680
those words together into one coherent sentence in the right order. Right,

1327
01:10:51,079 --> 01:10:53,159
That sounds like a really good time to move on picks,

1328
01:10:53,159 --> 01:10:57,000
and maybe you can point someone to some of those resources.

1329
01:10:57,199 --> 01:11:01,199
Speaker 2: Uh, I'm not going to plug blog as my pick,

1330
01:11:01,359 --> 01:11:04,399
but I'm happy to put a link for this episode

1331
01:11:04,760 --> 01:11:07,560
for anyone that wants it. Actually, I'll find the Saturday

1332
01:11:07,560 --> 01:11:12,199
Morning Breakfast Cereal episode two and put that. But my

1333
01:11:12,279 --> 01:11:14,279
pick actually is going to be based of our conversation.

1334
01:11:14,359 --> 01:11:17,119
There is a book by Peter Senge called The Fifth Discipline,

1335
01:11:17,439 --> 01:11:20,319
and it's all about systems thinking. And there's this one

1336
01:11:20,359 --> 01:11:23,880
great example in the book about it's a canonical thing

1337
01:11:23,920 --> 01:11:26,800
that I believe it's taught at I think it's Harvard

1338
01:11:26,840 --> 01:11:30,640
Business School, but also at MIT in the economics theory.

1339
01:11:30,880 --> 01:11:34,920
And that's basically, there is a brewery that makes a

1340
01:11:34,960 --> 01:11:39,960
beer and they make enough for a distributor to sell

1341
01:11:39,960 --> 01:11:42,560
the beer to retailers for people to buy it, and

1342
01:11:42,960 --> 01:11:46,880
every retailer sells like your local store sells two cases

1343
01:11:46,960 --> 01:11:51,000
every week. And then in some random week, a new

1344
01:11:51,079 --> 01:11:54,439
pop song comes out and everyone loves the beer, and

1345
01:11:54,600 --> 01:11:58,000
because the band mentions the beer in the song, and

1346
01:11:58,039 --> 01:12:00,760
from that point on, every single retailer just increases the

1347
01:12:00,840 --> 01:12:04,840
number of beers the case they sell by one to three.

1348
01:12:05,319 --> 01:12:08,640
And this causes some ridiculous effects to happen throughout the

1349
01:12:08,680 --> 01:12:12,520
whole ecosystem. The distributors will fall over, the brewery will

1350
01:12:12,520 --> 01:12:15,319
get some massive orders, and basically what ends up happening

1351
01:12:15,319 --> 01:12:18,359
as lots of people will be fired for failing to

1352
01:12:18,399 --> 01:12:21,479
deliver effectively. And it's just such a small change to happen,

1353
01:12:21,520 --> 01:12:23,159
and you can see huge impacts, and I feel like

1354
01:12:23,199 --> 01:12:26,359
it's a really interesting case study of what happens over

1355
01:12:26,399 --> 01:12:29,800
and over again, and lots of organizations and relates out

1356
01:12:29,880 --> 01:12:32,439
to a lot what Amy was talking about in today's episodes.

1357
01:12:32,600 --> 01:12:34,439
Those relevant.

1358
01:12:34,960 --> 01:12:40,159
Speaker 1: That's pretty cool. That's called the fifth dimension discipline, fifth discipline, gotcha,

1359
01:12:40,319 --> 01:12:44,600
re Peter sin get yeah? Right on, Jillian, what about you?

1360
01:12:44,680 --> 01:12:45,560
What'd you bring for a pick?

1361
01:12:45,760 --> 01:12:47,359
Speaker 3: I don't know. I was going to pick. I'm giving

1362
01:12:47,359 --> 01:12:50,840
a talk next week on HPC, And now it occurs

1363
01:12:50,880 --> 01:12:52,199
to me that I don't know if that's going to

1364
01:12:52,239 --> 01:12:55,680
be like recorded and put any place, So that's maybe

1365
01:12:55,720 --> 01:12:59,680
not my pick. I don't know, like Horizons zero down

1366
01:12:59,800 --> 01:13:03,159
the video game I've been playing, and maybe the talk.

1367
01:13:03,239 --> 01:13:05,840
Maybe the talk too. If it's actually published and recorded,

1368
01:13:06,159 --> 01:13:06,680
we'll see.

1369
01:13:06,960 --> 01:13:09,880
Speaker 2: You can always record it yourself on your laptop and

1370
01:13:09,920 --> 01:13:12,800
then you'll have the transcript, you can publish the slides,

1371
01:13:12,840 --> 01:13:15,199
and then you can you know, release it to all

1372
01:13:15,239 --> 01:13:16,520
your faithful audience there.

1373
01:13:16,880 --> 01:13:18,960
Speaker 3: Yeah, that's true. I mean I could just help somebody

1374
01:13:19,000 --> 01:13:20,800
record it on my phone, right, Like, I don't know.

1375
01:13:21,119 --> 01:13:24,119
Speaker 1: Okay, so we're gonna count on seeing your talk.

1376
01:13:24,000 --> 01:13:26,279
Speaker 3: Then, yeah, sure, that's the point. It's supposed to be

1377
01:13:26,279 --> 01:13:30,520
self promotion, like, right, that is what I do.

1378
01:13:31,279 --> 01:13:33,439
Speaker 1: Amy. I know you've done this before. What do you

1379
01:13:33,479 --> 01:13:34,640
have for a pick this week?

1380
01:13:35,000 --> 01:13:39,720
Speaker 4: See I was debating if I should pick something technical

1381
01:13:39,880 --> 01:13:43,079
or non technical. I guess I will try to break

1382
01:13:43,079 --> 01:13:45,000
it up a little bit. I'm going to pick a

1383
01:13:45,000 --> 01:13:48,520
band that I am like absolutely obsessed with and I

1384
01:13:48,560 --> 01:13:51,159
can't shut up about it, And because other people won't

1385
01:13:51,159 --> 01:13:52,760
listen to me anymore about it, this is a great

1386
01:13:52,760 --> 01:13:57,119
opportunity to share it. My husband is really tired to

1387
01:13:57,159 --> 01:14:00,159
be talking about them and blasting them, so and that

1388
01:14:00,279 --> 01:14:02,119
is sleep token. I don't know if anybody who's ever

1389
01:14:02,159 --> 01:14:04,600
heard of it or heard of them, but I will

1390
01:14:04,680 --> 01:14:09,199
drop a link. I first heard about them like maybe

1391
01:14:09,319 --> 01:14:11,319
like two or three years ago, and they have a

1392
01:14:11,319 --> 01:14:14,600
bunch of new songs out this year, and I am, yeah,

1393
01:14:14,800 --> 01:14:18,039
it's when I've had like a long, stressful day at work.

1394
01:14:18,079 --> 01:14:19,760
I don't know why, Like I listened to them and

1395
01:14:19,840 --> 01:14:23,279
it's like peaceful to me to fall asleep toobe. So whatever,

1396
01:14:23,680 --> 01:14:25,079
maybe other people will like it too.

1397
01:14:25,239 --> 01:14:26,039
Speaker 2: What's the genre?

1398
01:14:27,680 --> 01:14:28,520
Speaker 6: That's the thing?

1399
01:14:28,760 --> 01:14:31,920
Speaker 4: So they are like they're all over the place, like

1400
01:14:32,159 --> 01:14:36,600
I guess traditionally more like metal. They're almost like a

1401
01:14:36,640 --> 01:14:40,479
little bit like ed M, a little bit metal, like honestly,

1402
01:14:40,520 --> 01:14:43,159
like a tiny bit hip hop something like they're just

1403
01:14:43,199 --> 01:14:46,399
like you listen to your song and you're like, oh,

1404
01:14:46,520 --> 01:14:49,439
this is like really relaxing, and then they start screaming

1405
01:14:49,520 --> 01:14:51,600
and then there and then it's like back like a

1406
01:14:51,680 --> 01:14:54,359
hip hop's kind of vibe, and I'm just like, what's happening.

1407
01:14:56,119 --> 01:14:57,800
Speaker 2: It's just like you're gonna say, like, yeah, it's like

1408
01:14:57,880 --> 01:15:01,640
it's like heiden or Dominate the Christmas Donkey. I just

1409
01:15:01,720 --> 01:15:03,680
use that music to falls leap that night. I'm just

1410
01:15:03,720 --> 01:15:07,720
like really like, not my personal choice, but uh, you know,

1411
01:15:07,720 --> 01:15:10,039
if it works for you, that's pretty interesting.

1412
01:15:10,079 --> 01:15:14,800
Speaker 4: I like metal, so why I absolutely love them, So yeah,

1413
01:15:15,000 --> 01:15:16,479
that's that's going to be my pick.

1414
01:15:17,479 --> 01:15:19,600
Speaker 1: That's awesome. I was just as you were saying it

1415
01:15:19,640 --> 01:15:21,760
and you're like, oh, it's so just what I want

1416
01:15:21,760 --> 01:15:23,920
to listen to after a long, stressful day. I was like, oh,

1417
01:15:24,000 --> 01:15:25,760
this would be so great if it's a metal band,

1418
01:15:25,880 --> 01:15:27,880
and then you're like a surprise, guess.

1419
01:15:27,600 --> 01:15:31,600
Speaker 4: What I mean. They're not metal in the sense of

1420
01:15:31,680 --> 01:15:34,640
like like I really like this band Motionless and White.

1421
01:15:34,680 --> 01:15:36,960
If people have heard of them, they're not metal in

1422
01:15:36,960 --> 01:15:39,520
that sense. They're like a little more chill than that.

1423
01:15:39,680 --> 01:15:44,720
But yeah, right, on, I'll drop a link before.

1424
01:15:44,479 --> 01:15:45,039
Speaker 6: We hang up.

1425
01:15:45,399 --> 01:15:50,880
Speaker 1: Cool. Cool, all right, So my pick I switched it

1426
01:15:50,960 --> 01:15:54,720
since we started recording because it's now a relevant topic.

1427
01:15:55,359 --> 01:16:02,119
On Netflix, there is a series called The bear Grills

1428
01:16:02,199 --> 01:16:07,439
Celebrity Hunt, and so it's all of these celebrities who

1429
01:16:07,439 --> 01:16:10,960
are taken to Costa Rica and then they compete, and

1430
01:16:11,000 --> 01:16:13,600
the losers of the competition have to go into this

1431
01:16:13,840 --> 01:16:19,520
area that's like walled off and they have to survive

1432
01:16:19,760 --> 01:16:22,800
in there for one hour while bear Grills is hunting them.

1433
01:16:22,880 --> 01:16:26,520
So they have an hour to either escape the bear

1434
01:16:26,560 --> 01:16:30,479
pit or get caught by bear Grills. And it was

1435
01:16:30,520 --> 01:16:34,000
actually just super fun to watch. And I brought it

1436
01:16:34,079 --> 01:16:37,039
up at work saying, Hey, for our next off site,

1437
01:16:37,159 --> 01:16:39,079
we should go to the bear pit and have bear

1438
01:16:39,159 --> 01:16:40,720
Grills hunt us down.

1439
01:16:40,720 --> 01:16:43,800
Speaker 2: Where bear grills will actually be well buttoned.

1440
01:16:44,680 --> 01:16:45,439
Speaker 1: I'd be cool with that.

1441
01:16:45,560 --> 01:16:47,920
Speaker 2: Yeah, I feel like that may create some sort of

1442
01:16:48,560 --> 01:16:51,640
psychological safety issues for coming back work later.

1443
01:16:52,079 --> 01:16:54,399
Speaker 1: Yeah, there's probably a downside to that, but let's not

1444
01:16:54,439 --> 01:16:55,319
focus on that.

1445
01:16:56,720 --> 01:16:57,319
Speaker 5: Makes me think.

1446
01:16:57,399 --> 01:17:00,199
Speaker 4: My husband, Like, at first I thought I was like,

1447
01:17:00,199 --> 01:17:03,239
what are you watching? But he watches this like it's

1448
01:17:03,279 --> 01:17:05,439
probably like a famous YouTuber people probably know I can't

1449
01:17:05,479 --> 01:17:07,279
think of him, but like he's always out in the

1450
01:17:07,319 --> 01:17:10,279
wilderness like doing things, and my husband loves watching him.

1451
01:17:10,279 --> 01:17:12,439
And initially I was like, this is crazy, but now

1452
01:17:12,479 --> 01:17:13,279
I kind of like it too.

1453
01:17:14,119 --> 01:17:16,720
Speaker 5: All Right, he's usually in Alaska.

1454
01:17:17,479 --> 01:17:19,439
Speaker 1: There have been a lot of those episodes where my

1455
01:17:19,520 --> 01:17:22,479
kids were watching something like that's stupid. Why are you

1456
01:17:22,520 --> 01:17:24,359
watching wasting your time on that? You know, like two

1457
01:17:24,359 --> 01:17:26,600
hours later, I'm still sitting next to him watching it.

1458
01:17:27,239 --> 01:17:29,880
But anyway, Amy, thanks for being on the show. This

1459
01:17:30,079 --> 01:17:32,760
was a fun chat. Thank you for your insights and

1460
01:17:32,800 --> 01:17:37,039
your expertise and sharing your thoughts on sre Warren Chillian.

1461
01:17:37,159 --> 01:17:39,800
Thank you both for jumping in and co hosting today.

1462
01:17:39,840 --> 01:17:43,720
It's good to see both and for all our listeners,

1463
01:17:43,800 --> 01:17:45,039
thanks for listening.