1
00:00:14,919 --> 00:00:21,120
On, y'all, welcome to another
episode of Adventures and dev Ops. I'm

2
00:00:21,160 --> 00:00:25,960
your host, Will Button joining me
in the studio, my co host back

3
00:00:26,039 --> 00:00:30,760
on a streak making Tom Brady look
like a slacker, Warren Parade. Welcome

4
00:00:30,800 --> 00:00:36,880
Warren, and thanks for letting me
come back for an of my hopes up

5
00:00:36,920 --> 00:00:40,320
that it will keep on going.
I'm Warren, I'm the CTO of author

6
00:00:40,359 --> 00:00:43,880
has just to reintroduce myself. Yeah, I mean, I like how this

7
00:00:43,960 --> 00:00:49,119
is going so far, and I
have no plans to coward my way out

8
00:00:50,079 --> 00:00:54,439
of the future right on. I'm
excited to hear that, because otherwise it's

9
00:00:54,479 --> 00:01:00,679
just me and the guest and always
lead to trouble. Speaking guests joining us

10
00:01:00,719 --> 00:01:08,719
today Pete Fritchman sre infrastructure staff infrastructure
engineer over at Observe, Inc. And

11
00:01:10,599 --> 00:01:17,719
Pete has joined us today to talk
about observability in those pesky internal applications that

12
00:01:17,760 --> 00:01:23,799
we all have. And I don't
know, it might be a love hate

13
00:01:23,879 --> 00:01:26,680
relationship there, but Pete, welcome
to the show. Hey, thank you

14
00:01:26,719 --> 00:01:32,280
for having me right on. So
tell us a little bit about your background,

15
00:01:32,280 --> 00:01:37,319
because you have You've got the staff
engineer title, which takes a while

16
00:01:37,439 --> 00:01:42,760
to get to and is also I
think a little bit uncommon on the infrastructure

17
00:01:42,840 --> 00:01:47,879
side. You know, it's pretty
common on the software engineering side, but

18
00:01:48,560 --> 00:01:53,040
I think it's a little more rare
to see staff engineers on the infrastructure side.

19
00:01:53,040 --> 00:01:57,439
So tell us how you got to
that point. Sure, yeah,

20
00:01:57,480 --> 00:02:00,879
I mean I've been doing this for
a long time. I had in the

21
00:02:00,959 --> 00:02:04,200
computers as a kid, which I
guess is more common these days, maybe

22
00:02:04,319 --> 00:02:07,680
less so in the nineties. But
I knew early on as a kid that

23
00:02:07,879 --> 00:02:12,680
I wanted to do computer stuff,
and I wasn't exactly sure what computer stuff

24
00:02:12,840 --> 00:02:16,560
was. Something with programming. I
enjoyed that kind of you know, make

25
00:02:16,599 --> 00:02:22,159
the computer do what I want aspect
of things. I was really lucky to

26
00:02:22,240 --> 00:02:27,199
land an internship at my local ISP
in like eighth grade, the summer after

27
00:02:27,240 --> 00:02:30,039
eighth grade, before ninth grade.
Oh no, I had a great mentor

28
00:02:30,080 --> 00:02:34,879
there, and I haven't kind of. I had done a mentorship like my

29
00:02:34,960 --> 00:02:38,599
seventh grade summer doing Linux stuff,
and I bought a laptop and ran Linux

30
00:02:38,639 --> 00:02:43,560
on it. And running Linux on
a laptop is a great way to become

31
00:02:43,599 --> 00:02:45,759
one with Linux, you know,
hate it, love it, whatever.

32
00:02:46,759 --> 00:02:52,039
And working at this ISP and having
a really great mentor George kind of like

33
00:02:52,439 --> 00:02:54,199
I realized, Okay, CIS and
Men is the thing I'd like to do,

34
00:02:54,240 --> 00:03:01,919
Like I enjoy debugging these problems and
building things and writing automation. So,

35
00:03:02,120 --> 00:03:05,439
you know, I did the high
school thing. I worked all through

36
00:03:05,520 --> 00:03:09,680
high school. I'm in a workaholic
forever you for better or worse. I

37
00:03:09,800 --> 00:03:14,759
worked at this ISP all through high
school. Went to college for a year.

38
00:03:15,840 --> 00:03:19,639
Wasn't my thing. It was fun
socially, but the school part was

39
00:03:19,719 --> 00:03:23,319
just not. I don't know,
I just lacked the focus. I just

40
00:03:23,360 --> 00:03:29,080
really wanted to work right. So
I had contributed a bunch to FreeBSD in

41
00:03:29,080 --> 00:03:30,879
my high school days. I was
a ports committer and ports are like the

42
00:03:30,879 --> 00:03:37,400
packages in previous D. And through
that I landed a job and a really

43
00:03:37,400 --> 00:03:44,240
awesome group at FedEx doing system administration
on everything Internet facing. And it was

44
00:03:44,240 --> 00:03:46,439
a small group, and looking back, like that was the group to be

45
00:03:46,479 --> 00:03:52,560
in at FedEx for doing Unixie stuff. They were definitely ahead of their time.

46
00:03:53,319 --> 00:03:54,800
Everything was automated. They wrote their
own tools, and I thought this

47
00:03:54,879 --> 00:03:58,719
was very normal for like a two
thousand and two shop, which I you

48
00:03:58,719 --> 00:04:01,560
know, now we know maybe was
so the whole automation. First thing has

49
00:04:01,599 --> 00:04:06,120
always just kind of been how else
would you do it? Kind of thinking,

50
00:04:06,879 --> 00:04:11,919
and then I was lucky enough to
land and that's every gig at Google

51
00:04:11,960 --> 00:04:15,280
after that in two thousand and five, and you know, they were like,

52
00:04:15,319 --> 00:04:16,439
hey, we should automate things and
I was like, well, yeah,

53
00:04:16,439 --> 00:04:20,079
how else would you do it?
Right? And then just from there

54
00:04:20,120 --> 00:04:25,959
it's been a whirlwind of startups.
And I tried the banking world for a

55
00:04:25,959 --> 00:04:29,600
little bit, had some fun there, has some not fun there. I

56
00:04:29,759 --> 00:04:32,800
ultimately decided to go back to the
startup world because I think that's my true

57
00:04:33,879 --> 00:04:36,199
you know, that's where the for
me, that's where the most fun is

58
00:04:36,759 --> 00:04:39,879
the most fun. I had a
bank was kind of a startup in a

59
00:04:39,920 --> 00:04:45,199
bank, and that's very hard to
find, oh for sure. So yeah,

60
00:04:45,240 --> 00:04:49,279
cool, Yeah, I mean there's
definitely a huge It's it's almost completely

61
00:04:49,319 --> 00:04:57,439
two separate, completely careers working at
large enterprise organizations versus startups, Like you

62
00:04:57,480 --> 00:05:00,879
have to have two different mental models
to be successful at each of those.

63
00:05:01,079 --> 00:05:03,959
It's almost two different skill sets.
I mean the base technical skill set of

64
00:05:03,959 --> 00:05:09,759
course, and then you know at
a startup you kind of have to pick

65
00:05:09,839 --> 00:05:11,759
up the pieces and lead the way
with what you have. And then at

66
00:05:11,759 --> 00:05:14,360
a big enterprise you have all the
resources, but you also have all the

67
00:05:14,399 --> 00:05:17,439
politics, and so you have to
you know, as much as you know,

68
00:05:17,720 --> 00:05:20,439
everyone I know hates that you have
to play. You have to make

69
00:05:20,480 --> 00:05:27,800
friends in other organizations and figure out
how to influence people, and it is

70
00:05:27,839 --> 00:05:30,439
hard and stress. I enjoy the
stress of tech, and the stress of

71
00:05:30,439 --> 00:05:36,399
that is just it's a lot.
Yeah, it's a lot of political engineering

72
00:05:36,720 --> 00:05:47,040
versus softer engineering laying right, for
sure. So when you started work at

73
00:05:47,079 --> 00:05:51,120
the ISP, like are we talking
back? Like dial up modem is P.

74
00:05:51,800 --> 00:05:55,680
Yeah. I sat in the after
room. I had a I had

75
00:05:55,720 --> 00:05:58,959
like PM two e's next to me
and those are quiet at least, and

76
00:05:59,000 --> 00:06:00,879
then I had a rack of like
fifty six k's that were just you know,

77
00:06:00,920 --> 00:06:03,319
I'm pretty sure I could whistle my
way into a fourteen and four connection.

78
00:06:03,560 --> 00:06:09,000
Back then, we were like the
regional ISP, So we did t

79
00:06:09,160 --> 00:06:14,160
ones for businesses and they're there.
Big thing was they would put the I

80
00:06:14,199 --> 00:06:15,160
forget what Cisco it was, but
there was something where you can take the

81
00:06:15,160 --> 00:06:16,879
t one and like, oh,
I want to take some of my parents

82
00:06:16,920 --> 00:06:19,360
and use them as phone lines,
and some of them as data and that

83
00:06:19,439 --> 00:06:25,199
was like a revolutionary in ninety seven, so that yeah, for sure.

84
00:06:25,279 --> 00:06:28,360
Yeah. I was working in the
telco industry right around then and nice.

85
00:06:28,920 --> 00:06:31,240
Yeah, and then we got you
know, d slams and until DSL thing

86
00:06:31,360 --> 00:06:33,920
came through, and it was just
it was a fun way to get a

87
00:06:33,959 --> 00:06:36,319
lot of exposure to a lot of
different things. I got to go to

88
00:06:36,399 --> 00:06:40,240
pops and put stuff in, I
got to build Unix boxes, I got

89
00:06:40,279 --> 00:06:42,839
to kind of do the whole gamut
of things. So you got the benefit

90
00:06:42,839 --> 00:06:46,120
of dealing with Y two. Yeah, you know like it. Yeah,

91
00:06:46,199 --> 00:06:50,279
it was. It was such a
non event that people at work weren't even

92
00:06:50,279 --> 00:06:53,720
worried. You know, everyone was
very much a realist there. They were

93
00:06:53,759 --> 00:06:56,519
like, every whatever, everything breaks, we just live in the world or

94
00:06:56,560 --> 00:07:00,319
nothing works, and you know,
we passed ourselfware. What more is there

95
00:07:00,360 --> 00:07:02,720
to do. I knew people.
I didn't know them like very well or

96
00:07:02,720 --> 00:07:08,600
personally, but I knew people pre
Y two k that the area I lived

97
00:07:08,600 --> 00:07:13,560
in at the time, they built
underground bunkers and stopped them and they were

98
00:07:13,600 --> 00:07:17,480
going like late December they were going
underground and saying we're not coming out for

99
00:07:17,879 --> 00:07:23,399
ten years or something, and I've
never seen any of those people again,

100
00:07:23,519 --> 00:07:28,199
so I'm like insanely curious. Were
they of January? Are they still in

101
00:07:28,720 --> 00:07:33,079
Yeah? Yeah, yeah. I
didn't to have like the you know,

102
00:07:33,480 --> 00:07:35,639
I didn't know Kobal. I mean, there was a whole you know,

103
00:07:36,000 --> 00:07:41,600
crew of people that were just making
insane money and projects. I'm just watching

104
00:07:41,680 --> 00:07:44,240
ka prep needed or not. I
mean, I feel like we'll hit that

105
00:07:44,279 --> 00:07:46,519
again in twenty thirty eight with thirty
two bit. I t like that that

106
00:07:46,560 --> 00:07:49,680
one is actually a little scary.
I hope to be retired by that,

107
00:07:50,519 --> 00:07:59,480
right, no doubt. That seems
like too much cool. So so talking

108
00:07:59,560 --> 00:08:05,079
about internal platforms and observability, Yeah, like that's that was whenever this came

109
00:08:05,120 --> 00:08:09,720
across as our topic, I was
like, you know, that's that's just

110
00:08:09,839 --> 00:08:15,240
brilliant because all of these little internal
apps, and sometimes not little internal apps,

111
00:08:15,279 --> 00:08:20,959
but like the things that the company
uses to make decisions about are we

112
00:08:20,040 --> 00:08:26,279
doing the right thing or not?
They often are just like little pet projects.

113
00:08:26,360 --> 00:08:33,240
So what's what's your experience there and
how do you get those recognized as

114
00:08:33,320 --> 00:08:37,879
the valuable assets that they are?
Yeah, Well, often it happens on

115
00:08:37,960 --> 00:08:43,159
its own, and but it's the
worst case of there's a catastrophic failure at

116
00:08:43,200 --> 00:08:46,000
the worst possible time, and everyone
goes, oh, that was really important.

117
00:08:46,679 --> 00:08:52,919
I worked somewhere at a public company, and the system that did the

118
00:08:52,159 --> 00:08:56,480
closing the books every day, you
know, like very barring, right,

119
00:08:56,000 --> 00:08:58,600
Well, it wasn't working and it
was taking you know, thirty hours to

120
00:08:58,639 --> 00:09:01,919
close twenty four hours worth of books, and obviously that doesn't work so well.

121
00:09:03,600 --> 00:09:05,600
And they kind of didn't notice it
or take any action until, like

122
00:09:05,759 --> 00:09:09,159
they were a week before they had
to do some sec filing quarterly thing.

123
00:09:09,200 --> 00:09:11,879
I don't know all the details,
but it became, you know, suddenly

124
00:09:11,960 --> 00:09:15,919
on Friday afternoon, it was we
need all hands on deck to fix this.

125
00:09:16,000 --> 00:09:24,200
But it's everything from that to not
just necessarily apps, but like internal

126
00:09:24,200 --> 00:09:28,600
infrastructure, right CEI and developer experience, and everyone's got these you know,

127
00:09:28,159 --> 00:09:31,879
shell script infrastructure that dev's run to
run the local cluster on their laptop,

128
00:09:31,919 --> 00:09:35,440
and there's you know, any company
you can probably sit down and find a

129
00:09:35,440 --> 00:09:39,759
million of them. I just think
they're perpetually under you know. The sexy

130
00:09:39,759 --> 00:09:43,720
part is production, right, people
want to build SLOs for users and show

131
00:09:43,720 --> 00:09:48,399
awesome graphs and look at this great
incident management process. We have. Then

132
00:09:48,399 --> 00:09:52,679
they have this like Jenkins instance,
it's barely holding on internally, you know

133
00:09:52,799 --> 00:09:54,399
that everyone hates but no one really
talks about because it's just, oh,

134
00:09:54,440 --> 00:09:58,919
yeah, that's life, you know, that whole side of things just need

135
00:10:01,000 --> 00:10:07,000
more love. Yeah, And so
I think part of that, you know,

136
00:10:07,159 --> 00:10:11,799
like you mentioned there, especially with
like around Jenkins and things, it's

137
00:10:11,879 --> 00:10:18,519
people. It's like people who build
tools to solve specific problems that they're having,

138
00:10:18,159 --> 00:10:24,039
and then it almost has like this
organic growth of other people see it

139
00:10:24,080 --> 00:10:28,200
and I'm like, oh yeah,
I need to use that too, and

140
00:10:28,240 --> 00:10:33,720
then it grows in its role in
the company. So how do you identify

141
00:10:33,879 --> 00:10:39,919
those before that catastrophic event? Yeah, well, you know a lot if

142
00:10:39,960 --> 00:10:41,720
you think about So I do a
lot of thinking about like how to compare

143
00:10:41,759 --> 00:10:45,559
it to production. So in production, if you say I want to launch

144
00:10:45,600 --> 00:10:48,240
this new micro service or I want
to consume this two AWS service, you

145
00:10:48,279 --> 00:10:52,120
write the sign bocs, you have
launch review. I mean hopefully right,

146
00:10:52,320 --> 00:10:56,519
ideally you have these things, you
have this whole like rigmarole and process for

147
00:10:56,559 --> 00:10:58,320
it. But eternally it's like,
oh, I'm going to fire up a

148
00:10:58,360 --> 00:11:00,799
EC too and run this new tool
I downloaded hey, two weeks later,

149
00:11:00,840 --> 00:11:05,159
it's important you have to apply that
same same principle. And part of it

150
00:11:05,200 --> 00:11:05,879
can be a staffing thing, right, Like, you know, you might

151
00:11:05,919 --> 00:11:09,519
have this team of a million people
working on PROD and then you got the

152
00:11:09,559 --> 00:11:15,000
two you know, the IT guys
working on infrastructure stuff that doesn't work very

153
00:11:15,000 --> 00:11:18,120
well at all. Yeah, So
I mean, I think you have to

154
00:11:18,159 --> 00:11:24,320
put some process to it, and
I mean I think that I'm a big

155
00:11:24,360 --> 00:11:28,960
fan of the whole infrastructure management,
incident management, post mortem process. Like,

156
00:11:28,000 --> 00:11:31,600
I think that that is a great
way to drive, you know,

157
00:11:31,639 --> 00:11:33,480
out of just happen. And we
all have to accept that nothing's one hundred

158
00:11:33,480 --> 00:11:37,559
percent, but you should get the
most out of every adage. So when

159
00:11:37,559 --> 00:11:43,200
your internal tool does blow up and
cause a you know, company visible,

160
00:11:43,279 --> 00:11:45,559
everyone is like, hey, I
don't really know what this was, but

161
00:11:45,600 --> 00:11:48,960
we couldn't do business for a day. Make the most of that, right,

162
00:11:48,080 --> 00:11:52,000
Hey, this thing broke. By
the way, there's five other services

163
00:11:52,000 --> 00:11:56,039
we've identified that are in the exact
same state and could all blow up tomorrow

164
00:11:56,039 --> 00:11:58,240
and we'd be on the same call. And so that's your that's your you

165
00:11:58,240 --> 00:12:03,240
know, entrance point to getting everyone
to have attention on that thing. And

166
00:12:03,240 --> 00:12:05,120
it's unfortunate that, you know,
things have to fail first sometimes, but

167
00:12:07,039 --> 00:12:09,879
that's how it goes when you're especially
at a startup where you're kind of have

168
00:12:09,919 --> 00:12:15,480
conflicting priorities. Oh for sure.
Yeah, And I think it's a really

169
00:12:15,559 --> 00:12:18,120
good good point there is, like
outages happen, so let's just make the

170
00:12:18,120 --> 00:12:22,679
most out of that learning experience.
Absolutely, which is that's been a cultural

171
00:12:22,799 --> 00:12:28,639
change for us as an industry over
the last I don't know, i'd say

172
00:12:28,639 --> 00:12:33,559
ten to fifteen years. I feel
like Etsy published that blame you know,

173
00:12:33,600 --> 00:12:37,519
the famous blameless postmortems. It feels
like that's ages ago now, but it's

174
00:12:37,519 --> 00:12:41,799
still so relevant every day. Yeah. Yeah, things broke, who cares

175
00:12:41,840 --> 00:12:43,639
how they broke. Let's just have
it not break in the same way again,

176
00:12:45,480 --> 00:12:46,840
right, And if you can do
that, that to me is like

177
00:12:46,879 --> 00:12:50,399
the health of a necessary team,
right, it's not. Yeah, I

178
00:12:50,440 --> 00:12:52,679
really hate places that measure health by
oh there were four outages last quarter,

179
00:12:54,240 --> 00:12:58,240
okay, like where they repeat root
causes, Yes, okay, maybe there's

180
00:12:58,279 --> 00:13:01,639
a real problem. But if they
weren't repeat root causes and you're solving root

181
00:13:01,639 --> 00:13:07,480
causes and writing good postmortems, O
just are great. Yeah, yeah,

182
00:13:07,480 --> 00:13:09,000
for sure. And I think it's
I think that's one of the things that

183
00:13:09,720 --> 00:13:16,759
it's really hard to convince people of
is outages aren't as bad as you think

184
00:13:16,879 --> 00:13:24,000
because where most of us work like
we can recover and and I say that

185
00:13:24,039 --> 00:13:30,000
coming from a background where sometimes when
we had outages, people's lives were at

186
00:13:30,039 --> 00:13:33,399
stake. And so if you can
walk out of an outage and say no

187
00:13:33,440 --> 00:13:39,200
one died, like we're going to
be all right. Yeah. Well,

188
00:13:39,200 --> 00:13:43,759
plus it ties into SLIS SLS air
budgets, right, people a lot of

189
00:13:43,799 --> 00:13:48,200
times before the SLO concept became very
popular, like it's five minutes of downtime

190
00:13:48,200 --> 00:13:52,679
bad? What does that mean?
They have this air budget and you can

191
00:13:52,720 --> 00:13:54,519
go, well, you know,
we're a three nine service committed to we

192
00:13:54,559 --> 00:13:58,840
have forty four and a half minutes
a month. Okay, five minutes isn't

193
00:13:58,879 --> 00:14:01,039
great, but it's the fine.
You know we can. Yeah, but

194
00:14:01,080 --> 00:14:05,080
if you have you know, hey, I've gone past it and we're writing

195
00:14:05,159 --> 00:14:07,240
checks or you know, giving SLA
credits like okay, now you know we

196
00:14:07,279 --> 00:14:15,080
have this quantifiable not how people feel
number to talk about things and yeah,

197
00:14:15,120 --> 00:14:18,399
so do you for internal apps,
do you take them that far where you

198
00:14:18,519 --> 00:14:24,000
give them assign them slis and SLOs
and slas for performance. I've been starting

199
00:14:24,000 --> 00:14:28,000
to so this is actually my first, my first gig out of many where

200
00:14:28,000 --> 00:14:31,120
I've really kind of focused on internal
stuff. I've often been the problem of,

201
00:14:31,159 --> 00:14:33,679
you know, the person focusing on
proud and you know, on a

202
00:14:33,679 --> 00:14:37,440
different team that now my team is
kind of you know, wearing all the

203
00:14:37,480 --> 00:14:43,240
hats thing here, and yeah,
so I'm we're putting SLOs on internal services

204
00:14:43,240 --> 00:14:46,759
and internal workflows and trying to treat
them with the same we have that in

205
00:14:46,759 --> 00:14:50,840
production, of course, and we're
trying to give it the same, uh,

206
00:14:50,519 --> 00:14:56,720
the same level of you know,
thought and execution. I've seen a

207
00:14:56,799 --> 00:15:03,200
huge shift in the industry over We're
all like where some teams maybe they're called

208
00:15:03,200 --> 00:15:07,240
platform teams, have sort of been
excluded from thinking about what a product is

209
00:15:07,440 --> 00:15:11,159
or how they do product management or
even product ownership. And I feel like

210
00:15:11,159 --> 00:15:15,919
that's really been turning around, first
with the DevOps movement and now just I'll

211
00:15:15,919 --> 00:15:18,720
look at how we build services,
micro services all all together and no one

212
00:15:18,799 --> 00:15:24,679
sort of excluded or have a different
process of internal teams. Doesn't matter.

213
00:15:24,799 --> 00:15:28,600
You're still offering a real service to
someone. Just happens to be your customers

214
00:15:28,679 --> 00:15:31,759
or within the same customer or same
organization. That's exactly how I sell it

215
00:15:31,840 --> 00:15:35,120
is. Yeah, you still have
customers, they just happen to be coworkers,

216
00:15:35,960 --> 00:15:39,480
right. Yeah, you can still
do the same stuff. You can

217
00:15:39,480 --> 00:15:41,679
build user journeys. You can figure
out what does a customer expect, what

218
00:15:41,720 --> 00:15:45,080
makes a customer angry. It's a
lot easier to figure it out because you

219
00:15:45,120 --> 00:15:48,080
can ask them and slack instead of
guessing, like, man, what are

220
00:15:48,120 --> 00:15:50,039
my customers? What's the threshold at
which my customer complains? You can just

221
00:15:50,120 --> 00:15:52,600
straight up ask them when they work
for you know, I don't think they

222
00:15:52,679 --> 00:15:56,879
necessarily have good, like nice answers
though, Like you're a customer, you

223
00:15:56,919 --> 00:16:00,600
find there's a variety of personalities,
right, people, the moment of Jenkins

224
00:16:00,679 --> 00:16:03,759
job doesn't passes in four minutes.
They're like, ah, Jenkins is graph

225
00:16:03,799 --> 00:16:07,240
It's terrible, but at least you
can get it. You know. That's

226
00:16:07,240 --> 00:16:10,080
how customers are. You know,
are in general right you have so you

227
00:16:10,080 --> 00:16:12,399
get the whole gamut of emotions.
But you can tell when there's an outage.

228
00:16:12,440 --> 00:16:15,480
You know, how loud people are. Internal people will tend to be

229
00:16:15,519 --> 00:16:18,519
louder internally that in your customer I
mean, that's what I mean, Like

230
00:16:18,600 --> 00:16:21,200
you could definitely like it's still a
problem in some regard, but like how

231
00:16:21,240 --> 00:16:25,320
do you temper the difference of perspective? Like I feel like users on the

232
00:16:25,360 --> 00:16:27,960
outside are more quiet in a lot
of ways, like it's difficult to pull

233
00:16:29,000 --> 00:16:32,039
information out of them, and internally, as you mentioned, you know,

234
00:16:32,120 --> 00:16:34,519
it's like everyone screaming as soon as
something goes even a little bit. I

235
00:16:34,559 --> 00:16:38,559
think that if you can show them
that a there are graphs, right,

236
00:16:38,600 --> 00:16:41,360
like there should always be a graph, you know, like generically I work

237
00:16:41,360 --> 00:16:44,639
for AGRAD that always sad, where's
the graphere's the graph? And it was

238
00:16:44,639 --> 00:16:45,919
annoying at first, but you're like, okay, maybe there should always be

239
00:16:45,960 --> 00:16:51,240
a graph so people know that You're, hey, people are actually watching Jenkins.

240
00:16:51,240 --> 00:16:55,320
People are actually rather computers are watching
Jenkins, not people, right,

241
00:16:56,440 --> 00:16:57,759
And if you show like, hey, we had this outage and yes it

242
00:16:57,799 --> 00:17:00,480
was a really terrible day and no
one shift any God, but here's this

243
00:17:00,519 --> 00:17:06,599
great postpartum everyone can read and that's
got a reasonable trigger and root cause and

244
00:17:06,640 --> 00:17:10,000
follow ups and timeline and we're actually
closing the follow ups. Like it's very

245
00:17:10,039 --> 00:17:12,759
visible and I feel that should be
the same way with customer outages. It's

246
00:17:12,839 --> 00:17:18,640
very visible, right, nothing,
there shouldn't be anything to hype. Yeah,

247
00:17:18,759 --> 00:17:22,079
So one idea I've been working on
because I work a lot with startups

248
00:17:22,440 --> 00:17:29,640
and like, the whole thing about
a startup is odds are the product that

249
00:17:29,720 --> 00:17:32,599
you launch is not going to be
the product that you're successful with, and

250
00:17:32,640 --> 00:17:34,680
you're going to try a lot of
different things before you become successful as a

251
00:17:34,680 --> 00:17:41,160
company. So how do we measure
that so we don't spend any more time

252
00:17:41,279 --> 00:17:45,799
than it's absolutely necessary working on the
wrong problem. And so I've been working

253
00:17:45,799 --> 00:17:48,680
on this idea of success criteria,
like what does it take to make this

254
00:17:48,799 --> 00:17:55,400
application successful? And a lot of
my background is in the infrastructure for mobile

255
00:17:55,400 --> 00:17:57,759
apps, and for mobile apps,
you can measure it as you know,

256
00:17:57,799 --> 00:18:03,880
we need ten thousand monthly active users
spending forty five minutes per week in the

257
00:18:04,000 --> 00:18:10,039
app or something like that, measure
engagement. Yeah, yeah, yeah,

258
00:18:10,119 --> 00:18:14,000
So like because then if you if
you tie in, like you know,

259
00:18:14,039 --> 00:18:17,920
our our total cost of acquisition to
get a new user as x amount of

260
00:18:17,920 --> 00:18:22,319
dollars, and our cost of infrastructure
is x amount of dollars per user.

261
00:18:22,400 --> 00:18:25,559
You know, you can do some
pretty simple math there to find out how

262
00:18:25,559 --> 00:18:30,240
many users you need for this to
be a profitable product. And so trying

263
00:18:30,279 --> 00:18:34,599
to figure that out and get that
in the early stages of an application so

264
00:18:34,720 --> 00:18:41,960
we know when to either double down
on this application or shelve it as soon

265
00:18:42,000 --> 00:18:47,240
as possible. I'm wondering if like
that same thing applies to internal tools,

266
00:18:47,240 --> 00:18:51,319
and how do you how do you
define what that looks like? Yeah,

267
00:18:51,359 --> 00:18:53,359
that's a good question. And I
think I had like a good you know,

268
00:18:53,440 --> 00:19:02,240
where we deployed the like a code
analysis tool, and yeah, I

269
00:19:02,240 --> 00:19:04,400
don't it's a good I don't know
how to quantify it for stuff like that.

270
00:19:04,559 --> 00:19:08,039
But again, we have some of
the very active developers that will give

271
00:19:08,079 --> 00:19:11,319
feedback of like, hey, this
then gave me a bad analysis and which

272
00:19:11,359 --> 00:19:12,240
we can and make it better.
It's like, okay, that means they're

273
00:19:12,240 --> 00:19:17,640
actually looking at it and you know, making code health better. And I

274
00:19:17,640 --> 00:19:19,799
don't know, I think the like, I think you're on the right track

275
00:19:19,880 --> 00:19:23,480
with you know, having a number
like mus and that kind of thing.

276
00:19:25,119 --> 00:19:32,400
It's tough internally, Yeah, yeah, well, and it's interesting because yeah,

277
00:19:32,480 --> 00:19:34,279
well, I think there's two classes
of internal products, right there's internal

278
00:19:34,319 --> 00:19:38,039
products that people are very opinionated on. Code analysis is one, right,

279
00:19:38,039 --> 00:19:41,200
Like, I think this gives me
bad code analysis. I don't like that.

280
00:19:41,920 --> 00:19:48,160
But CI right, I think largely
people don't actually care what CI product

281
00:19:48,160 --> 00:19:52,440
you're running. They care do my
prs get approvals fast? Is master green?

282
00:19:53,599 --> 00:19:56,720
The deployments happen at the pace that
we think they should at our company,

283
00:19:57,079 --> 00:20:00,240
and if that works, who cares
what the product is? Right,

284
00:20:00,279 --> 00:20:02,920
they're like, And so for stuff
like that, you can just be super

285
00:20:02,960 --> 00:20:04,240
results oriented, right, Like,
hey, is are the s lives?

286
00:20:04,319 --> 00:20:08,119
These our solos we're trying to go
for we're making them, we're not making

287
00:20:08,160 --> 00:20:15,279
them. We're a d inking shop
now, and we're considering alternatives. And

288
00:20:15,519 --> 00:20:17,839
of course, you know, you
ask people what we should. There's a

289
00:20:17,920 --> 00:20:21,680
million things, but no one actually
cares what we run, right, you

290
00:20:21,680 --> 00:20:25,200
know, if they can declare their
jobs in some declarative way that's not too

291
00:20:25,279 --> 00:20:30,720
terrible and it works, everyone's happy. Yeah, I'll tell you what'll make

292
00:20:30,759 --> 00:20:34,119
them care. If they have to
be the ones to convert your drinking jobs

293
00:20:34,200 --> 00:20:37,599
to the new platform, then a
lot of people will be like, ah,

294
00:20:37,680 --> 00:20:42,559
you know, Chinkin's is probably okay. There's a lot of GitHub actions

295
00:20:42,640 --> 00:20:45,759
talk because a lot of people have
done stuff there. So I mean part

296
00:20:45,759 --> 00:20:48,160
of you know, part of that
is that's the politicals or well, the

297
00:20:48,720 --> 00:20:52,640
human side of it is, Hey, maybe it's if all of the products

298
00:20:52,720 --> 00:20:55,279
can do the thing, maybe we
pick the one people have the most experience

299
00:20:55,319 --> 00:20:57,480
with just to make life easier in
the transition. That's a good point.

300
00:20:57,640 --> 00:21:02,279
I mean, will you ask question, which is, how do we know

301
00:21:02,400 --> 00:21:06,559
our thing is going to be successful
at a startup level? And if we

302
00:21:06,599 --> 00:21:11,559
take the perceived public metrics on ninety
percent fail are what a companies do with

303
00:21:11,920 --> 00:21:15,720
teams that are part of that ninety
percent? Do they let them go?

304
00:21:17,400 --> 00:21:22,440
Do they reposition them? I think
that's a huge struggle and it's scary to

305
00:21:22,559 --> 00:21:27,720
know that there are some metrics associated
with your successful even within a larger company

306
00:21:29,279 --> 00:21:33,920
that you don't have job security in. Yeah, I can tell you just

307
00:21:33,960 --> 00:21:42,200
in my experience with startups, that's
like when you identify the success failed criteria

308
00:21:42,759 --> 00:21:48,519
answers that question because if you can
identify it early, then you take those

309
00:21:48,559 --> 00:21:52,640
people and you put them on what
your next idea is. But in many

310
00:21:52,720 --> 00:21:56,720
cases, if you wait too late
to identify that this product is not going

311
00:21:56,759 --> 00:22:00,079
to be successful. You've already been
bleeding too much cash for church along that

312
00:22:00,200 --> 00:22:04,960
you've got to salvage what's left,
and the first thing that goes when you're

313
00:22:04,960 --> 00:22:08,279
trying to cut costs is your staff. I wonder if there's a lesson to

314
00:22:08,279 --> 00:22:12,079
be learned that can be pulled to
internal apps. Though in the startup market,

315
00:22:12,200 --> 00:22:17,880
there's this idea of letters of intent, right getting signatures from potential customers

316
00:22:17,920 --> 00:22:22,000
even before you built anything based off
of that idea, and maybe maybe there's

317
00:22:22,000 --> 00:22:27,200
an idea here of how to transition
this to even internal teams. Yeah,

318
00:22:27,519 --> 00:22:32,240
I think, yeah, I like
the idea of formalizing it. I mean,

319
00:22:32,240 --> 00:22:34,480
I think what I often do for
these kind of things is like you

320
00:22:34,559 --> 00:22:37,079
find, uh, you know,
hey, you might have ten teams and

321
00:22:37,160 --> 00:22:41,480
six teams really care about this thing
you're building because they're big use of it.

322
00:22:41,480 --> 00:22:44,359
You find like a champion on each
team that is, you know,

323
00:22:44,400 --> 00:22:48,440
opinionated and is willing to talk to
you about what's good and bad. And

324
00:22:48,519 --> 00:22:49,920
maybe not a letter of intent,
but you write a design doc and then

325
00:22:49,960 --> 00:22:52,240
you make sure that they've read it
and give you feedback, not just to

326
00:22:52,319 --> 00:22:55,200
check, but you know, if
anyone reads a design doc and has no

327
00:22:55,279 --> 00:23:00,400
comments, they didn't actually read it. Like I've never met an engineer without

328
00:23:00,400 --> 00:23:03,079
an opinion on a design. So
so you get you know, you get

329
00:23:03,119 --> 00:23:07,240
feedback from people on these teams,
and it's maybe not as formal as a

330
00:23:07,279 --> 00:23:10,400
letter of intent, but you kind
of have the you know, social buy

331
00:23:10,400 --> 00:23:14,559
in, right, Yeah. Yeah, And that was one of the things

332
00:23:14,559 --> 00:23:19,119
that we whenever I worked at Active, I started with them super early and

333
00:23:21,240 --> 00:23:26,119
we just accidentally got this right.
But we built this platform for all the

334
00:23:26,160 --> 00:23:34,079
engineers to build and deploy their software, and the way we built it,

335
00:23:34,119 --> 00:23:40,640
we we had such great collaboration between
an infrastructure team and an engineering team that

336
00:23:40,839 --> 00:23:47,359
anytime the engineering team wanted to expand
the capabilities of that platform tool, most

337
00:23:47,400 --> 00:23:49,839
of the time their request came in
the form of a poor request to just

338
00:23:49,920 --> 00:23:55,839
add that feature to it. Yeah, and I've never been able to duplicate

339
00:23:55,880 --> 00:24:00,440
that since then, but that had
such a significant impact on how I think

340
00:24:00,440 --> 00:24:03,799
about platforms that that's my goal every
day. Yeah. I like that.

341
00:24:03,799 --> 00:24:08,200
It's kind of the open source approach, right, Yeah, Yeah, that's

342
00:24:08,240 --> 00:24:11,039
I mean, I think a lot
of that is the buy in, right,

343
00:24:11,039 --> 00:24:14,440
because if you're using a platform or
a language that no one else wants

344
00:24:14,559 --> 00:24:18,119
or knows or likes. They're not
going to learn rusk to send you a

345
00:24:18,160 --> 00:24:19,799
poll review to fix a thing they
don't like. But if it's already in

346
00:24:19,880 --> 00:24:22,759
go and the rest of your codes
and go, yeah, I just show

347
00:24:22,839 --> 00:24:27,759
up one morning. Maybe there's a
corollary here as well to the total economy

348
00:24:27,799 --> 00:24:33,000
in the startup world, where you
know your customers are somewhere along the spectrum

349
00:24:33,079 --> 00:24:37,640
of innovators, early adopters, early
minority, all the way to laggards,

350
00:24:37,680 --> 00:24:38,880
and it's really who are you talking
to? You know, what does the

351
00:24:38,880 --> 00:24:42,240
rest of your organization look like?
Because I feel like, well, you

352
00:24:42,240 --> 00:24:48,839
would need innovators and early adopters there
who not only need that functionality but have

353
00:24:48,880 --> 00:24:52,000
a huge stake or care about how
it's implemented, and not just people who

354
00:24:52,039 --> 00:24:57,880
think it's table stakes or just belongs
there or have opinions that they just want

355
00:24:57,880 --> 00:25:03,680
it done their way. Yeah,
yeah, true, yeah, And that's

356
00:25:03,920 --> 00:25:06,039
I think you hit that. I
think you hit it right on the head

357
00:25:06,079 --> 00:25:14,480
there that early stage startups require that
innovation mindset. For sure. They're not

358
00:25:14,480 --> 00:25:18,319
going to get very far if only
a few people are thinking about how to

359
00:25:18,319 --> 00:25:21,680
push what do you bring their product
is forward. Yeah, I really a

360
00:25:21,680 --> 00:25:25,359
great and I want to clarify that
there. You know, there's there's a

361
00:25:25,400 --> 00:25:30,039
certain type of person who just wants
to show up for work and do their

362
00:25:30,039 --> 00:25:33,799
assigned job and they want to do
that for twenty or thirty years till they

363
00:25:33,799 --> 00:25:38,240
retire. And there's absolutely nothing wrong
with that, just that an early stage

364
00:25:38,240 --> 00:25:44,160
startup is not the right environment for
you to be successful. Yeah, you

365
00:25:44,240 --> 00:25:45,960
definitely see that. You can find
out the bigger companies. You have these

366
00:25:45,960 --> 00:25:49,200
classes of employees, right, there
are some people that you can tell either

367
00:25:49,240 --> 00:25:53,279
came from startups or are going to
go to startups afterwards. And then like

368
00:25:53,279 --> 00:25:56,880
you said, there are some people
that just you know, close tickets and

369
00:25:56,920 --> 00:26:00,640
do the work. And that's yeah, completely fine. Yeah, I sometimes

370
00:26:00,640 --> 00:26:06,599
wish that I could be that person. It'd be a lot lessful. And

371
00:26:06,680 --> 00:26:07,559
yeah, I don't know, maybe
I will be in five or day.

372
00:26:07,559 --> 00:26:12,960
Who knows, right, people change, But yeah, So let's talk a

373
00:26:12,960 --> 00:26:18,480
little bit about the types of internal
platforms, because we've talked about you know,

374
00:26:18,480 --> 00:26:26,480
like CICD, what other tools are
out there that we should be looking

375
00:26:26,599 --> 00:26:33,400
for as internal tools. One that
pops up immediately in my mind only because

376
00:26:33,440 --> 00:26:37,680
I've been bitten by this one at
every single company I've ever worked at for

377
00:26:37,720 --> 00:26:45,440
the last three decades is the internal
data analytics team. Like those guys have

378
00:26:45,279 --> 00:26:52,960
monster infrastructures and are just doing what
it takes to generate the reports that the

379
00:26:53,000 --> 00:26:56,680
business wants to see. And whenever
you get a hold of it, like,

380
00:26:56,759 --> 00:27:00,960
oh wow, I'm not even really
certain how this thing is working.

381
00:27:00,839 --> 00:27:04,400
So what are there to break?
Internal tools are out there. Well,

382
00:27:04,480 --> 00:27:08,000
some people run their own vcs,
you know, get get lab on prem

383
00:27:08,880 --> 00:27:12,759
h we run Garrett so that's its
own you know, Carrien feeding and that's

384
00:27:12,759 --> 00:27:18,880
the loos. I think one thing
people actually miss is the monitoring infrastructure themselves.

385
00:27:18,880 --> 00:27:23,119
You know, if you run Prometheus
and Elastic or if you don't have

386
00:27:23,160 --> 00:27:26,920
a vendor, right, and even
if you have a vendor, you have

387
00:27:26,000 --> 00:27:30,119
some component that is shipping metrics to
them, right, you have to monitor

388
00:27:30,160 --> 00:27:33,839
that. If that stuff all breaks, you're flying blind and you need to

389
00:27:33,880 --> 00:27:40,799
know it. I think that often
goes neglected, right. I worked somewhere

390
00:27:40,839 --> 00:27:42,720
where someone came to us once and
they were like, you know, they

391
00:27:42,799 --> 00:27:45,960
run a really noisy on call rotation, which is its own problem, and

392
00:27:47,039 --> 00:27:48,839
they knew things were broken because they
had I haven't gotten a page duty in

393
00:27:48,920 --> 00:27:52,359
ninety minutes, and that was their
escalation and I was like, well,

394
00:27:52,440 --> 00:27:55,400
it's actually broken, and this is
so sad that this is how we found

395
00:27:55,400 --> 00:27:59,039
out about the problem. On many
levels, right one, we didn't know

396
00:27:59,160 --> 00:28:03,440
too. You expect to be paged
at least once every ninety minutes, but

397
00:28:03,559 --> 00:28:07,119
it's it's truly I think the monitoring
infrastructure, it seems like it should be

398
00:28:07,119 --> 00:28:11,400
obvious, but it's not necessarily obvious
that do you think that's Do you think

399
00:28:11,400 --> 00:28:17,759
that's a mess from like existing observability
tools that they aren't able to have internal

400
00:28:17,799 --> 00:28:21,640
metrics on what the expectation is on
getting logs, Like I feel like we've

401
00:28:21,720 --> 00:28:25,480
used cloud Watch for a while and
one of the things it does in AWS

402
00:28:25,559 --> 00:28:30,759
is you can alert on missing data. Well that's a hard like in Prometheus,

403
00:28:30,759 --> 00:28:33,880
it's very hard to alert on missing
data right end of alerting on like

404
00:28:33,920 --> 00:28:37,200
the uptime series being zero. But
then what if you have a problem that

405
00:28:37,240 --> 00:28:41,079
generates your targets and there is no
uptime series right, Like there are a

406
00:28:41,119 --> 00:28:45,119
lot of these different things. Yeah, I think that it's not Yeah,

407
00:28:45,160 --> 00:28:47,960
I don't think it's first class of
you know, watch the watchers. I

408
00:28:48,039 --> 00:28:49,759
think the data is there, though, and you have to kind of do

409
00:28:49,880 --> 00:28:56,359
it. I worked somewhere where we
had a relatively large Prometheus and Grafana infrastructure

410
00:28:56,400 --> 00:29:00,440
for metrics and a really large Blunk
infrastructure for law, and both of them

411
00:29:00,440 --> 00:29:03,480
had their problems and they're run much
two different teams. And we got together

412
00:29:03,519 --> 00:29:07,480
and said, hey, let's make
a deal. Right, We'll write a

413
00:29:07,519 --> 00:29:11,720
Probert for Splunk and have an export
metrics and monitors Spunk and Prometheus. You

414
00:29:11,759 --> 00:29:15,039
write a prober for Prometheus and have
it writ logs and monitor and Splunk And

415
00:29:15,079 --> 00:29:18,160
that's not perfect, but you make
with what you've got. And that was

416
00:29:18,200 --> 00:29:22,839
a company where it was a large
place that was going to be hard to

417
00:29:22,880 --> 00:29:26,039
bring in a third party tool to
help us, and you know, but

418
00:29:26,240 --> 00:29:30,200
we may do with it. And
it was super successful. We found lots

419
00:29:30,240 --> 00:29:32,240
of problems with you know, if
they both died at the same time,

420
00:29:32,279 --> 00:29:33,599
Okay, yeah, that's the end
of the world. But the world is

421
00:29:33,599 --> 00:29:37,799
probably already ending if they both died. So do you think there are third

422
00:29:37,799 --> 00:29:41,839
party vendors that actually solve this in
a reasonable way? Well, I mean,

423
00:29:41,839 --> 00:29:45,640
I think it's hard to say,
right, It depends what you're looking.

424
00:29:45,640 --> 00:29:48,519
If you're using a third party vendor
for monitoring, you know, you

425
00:29:48,559 --> 00:29:53,799
need to look at your metrics of
shipping. Is your data there with them?

426
00:29:55,000 --> 00:29:57,119
Right? So you have to set
something up in their system to do

427
00:29:57,200 --> 00:30:00,880
it. But then you do you
want to back up? Uh? You

428
00:30:00,880 --> 00:30:03,920
know? Do you want to know
if they're up? You may have to

429
00:30:03,000 --> 00:30:08,200
run a little side monitoring infrastructure too
to watch them because it might not be

430
00:30:08,200 --> 00:30:11,359
anything you can do about it,
but you may want to at least be

431
00:30:11,400 --> 00:30:14,359
aware that, Hey, the thing
that normally sends me alerts is not going

432
00:30:14,440 --> 00:30:18,559
to send me alerts. Maybe we
should all be go back to the knock

433
00:30:18,680 --> 00:30:21,799
days and you know, stare at
some things for a couple of hours.

434
00:30:22,839 --> 00:30:23,799
I mean, that's exactly what I
don't want to have to think about,

435
00:30:23,799 --> 00:30:27,319
Like I don't have to think about
my vendor being down in some way that

436
00:30:27,480 --> 00:30:30,599
requires me to monitor them. Like
I feel like, you know, if

437
00:30:30,640 --> 00:30:33,240
I'm paying money out for that,
I should I should be getting that by

438
00:30:33,319 --> 00:30:37,160
default. I don't know. Maybe
I think that's just my pessimistic brain.

439
00:30:37,599 --> 00:30:44,559
Yeah, for sure, everything will
break. Everything will break, So whether

440
00:30:44,599 --> 00:30:47,880
you're building it or not, it's
going to break and you either want to

441
00:30:47,880 --> 00:30:49,519
know about it or you know,
if you think something doesn't break, you're

442
00:30:49,599 --> 00:30:52,359
just not measuring it. Yeah,
that's kind of mine. Yeah, yeah,

443
00:30:52,480 --> 00:30:56,160
I have two examples where I think
that's a really like a lot of

444
00:30:56,160 --> 00:31:03,160
effort has gone into that Core Logics
as a logging and monitoring platform, and

445
00:31:03,200 --> 00:31:07,559
they have anomaly detection, and so
if you'll if you have a service that

446
00:31:08,200 --> 00:31:12,400
normally spits out, you know,
one thousand log entries per hour, if

447
00:31:12,440 --> 00:31:18,839
it stops, it identifies that as
an anomaly, and we'll trigger that and

448
00:31:18,839 --> 00:31:23,319
say, hey, nothing is like
technically alerted, but something's changed here.

449
00:31:25,000 --> 00:31:29,920
And then another one that we just
recently started using as a tool for monitoring

450
00:31:29,960 --> 00:31:36,559
our infrastructure spin called clouds zero,
and it has a really cool anomaly detection

451
00:31:36,720 --> 00:31:40,640
as well, says, hey,
this project, their current spend rate has

452
00:31:40,680 --> 00:31:45,039
been this, but today it changed
to this, which is you know,

453
00:31:45,079 --> 00:31:49,000
not necessarily a problem, but cool
that it acknowledges that and you're like,

454
00:31:49,079 --> 00:31:56,440
yeah, why did that change?
I think the correlation analysis like that is

455
00:31:56,480 --> 00:32:00,400
really good for not necessarily paging someone
itto a, but reports and debugging like

456
00:32:00,440 --> 00:32:06,039
hey, I have a problem started
two days ago, what else changed two

457
00:32:06,119 --> 00:32:07,720
days ago? Right, Hey,
you also started spending less money on this.

458
00:32:07,839 --> 00:32:10,960
It's like, yeah, she doesn't
mean causation, but I'm certainly going

459
00:32:12,000 --> 00:32:15,039
to look there first. Now.
I think people underestimate that really though,

460
00:32:15,079 --> 00:32:19,119
like just looking out, when does
the problem start? What else happened there?

461
00:32:19,160 --> 00:32:22,200
And I feel like it's so obvious
to say that, but it's usually

462
00:32:22,200 --> 00:32:24,480
one of the first things that's missed. Yeah, for sure. It's been

463
00:32:24,519 --> 00:32:28,240
like a pet project of mine for
a very long time. I have a

464
00:32:28,279 --> 00:32:31,640
super stale, busted repo trying to
do this. Back when, uh remember

465
00:32:31,720 --> 00:32:36,720
CEP continuous event processing, it was
all the hype and like maybe it wasn't

466
00:32:36,720 --> 00:32:39,519
all the hype, but it was
somewhat popular in the late two thousands,

467
00:32:39,839 --> 00:32:45,720
I was trying to do the uh
whole Winters forecasting, Like the thing that's

468
00:32:45,720 --> 00:32:47,480
built in is that you know already
tool added that thing again ages ago,

469
00:32:47,680 --> 00:32:52,519
dating myself. You could, you
know, put the prediction line in front

470
00:32:52,559 --> 00:32:54,039
of your graph, And I always
thought like, yeah, let's just do

471
00:32:54,079 --> 00:32:58,039
that, like let's predict every time
series. Like it may not be great

472
00:32:58,039 --> 00:33:00,079
signal and I'm not going to look
at it unless something's broke, but hey,

473
00:33:00,160 --> 00:33:02,880
I've got twenty thousand metrics. I'm
not going to look at all of

474
00:33:02,920 --> 00:33:07,319
them, right, So when the
site car starts getting slow, it'd be

475
00:33:07,359 --> 00:33:10,000
really interesting if I could also see, hey, i'll wait, time on

476
00:33:10,039 --> 00:33:14,240
the database server went up. It's
okay. Are we sending queries that a

477
00:33:14,319 --> 00:33:16,839
disc go bad? Oh? Hey, look errors on this disk spindle started

478
00:33:16,880 --> 00:33:22,000
going up too, Like maybe there's
something here. And I think it solves

479
00:33:22,200 --> 00:33:29,160
another monitoring problem that people still make
is people monitor root causes and not what

480
00:33:29,200 --> 00:33:31,799
they actually want out of a system, right, like sending me an alert

481
00:33:31,799 --> 00:33:35,599
that my CPU is high at too
in the morning, Like, cool,

482
00:33:35,599 --> 00:33:38,039
we're using the CPU we paid for, like comes up back to bed right,

483
00:33:38,640 --> 00:33:43,240
awesome capacity planning. But if it's
making the site slow, page me

484
00:33:43,279 --> 00:33:45,960
and say the site is slow,
right, don't use me and say don't

485
00:33:45,000 --> 00:33:47,640
page me and say job is doing
GC pauses? Well, yeah, you're

486
00:33:47,680 --> 00:33:52,759
running travel That's what it does all
day. If they're too long, you're

487
00:33:52,759 --> 00:33:55,440
going to fail a latency monitor and
page me on the thing that matters.

488
00:33:55,440 --> 00:33:59,880
So like, I want these root
causes to be the correlation engine to kind

489
00:33:59,880 --> 00:34:02,119
of help me figure out. Okay, well eighty thousand things go wrong,

490
00:34:02,160 --> 00:34:05,839
and if you put an alert on
all of those. You're back at that

491
00:34:05,880 --> 00:34:10,519
team that notices you don't get paid
for ninety minutes. Right, Yeah,

492
00:34:10,519 --> 00:34:14,559
if you look at every team that
has page fatigue, this is exactly the

493
00:34:14,559 --> 00:34:16,880
problem. Yeah, most people don't
even know why they get paid for ninety

494
00:34:16,880 --> 00:34:20,559
percent of the things. It's organically
grown over the life of the team.

495
00:34:20,880 --> 00:34:23,920
Hey, this one time, you
know, the database backups did this thing

496
00:34:24,039 --> 00:34:27,199
terribly, and now we page on
it. And then you have so many

497
00:34:27,239 --> 00:34:30,159
of those it's like, well,
there's eight thousand alerts and most of them

498
00:34:30,199 --> 00:34:34,760
resolve themselves, and we just thrug
our shoulders. And I don't think it

499
00:34:34,800 --> 00:34:37,880
starts from a good place. Like
I think there's this idea that just having

500
00:34:37,000 --> 00:34:42,480
dashboards is valuable in itself, and
it sort of propagates from there. It's

501
00:34:42,480 --> 00:34:44,280
like, oh, you know,
what are all the metrics we could be

502
00:34:44,320 --> 00:34:49,079
collecting, and let's show so.
And sometimes there is an organization that thinks

503
00:34:49,119 --> 00:34:52,239
that there is some inherent value,
like oh, this is how many requests

504
00:34:52,280 --> 00:34:55,199
we're getting, Let's use it to
get more headcount because we have to support

505
00:34:55,239 --> 00:35:00,159
such complex things. Or I remember
a previous company that thought it was really

506
00:35:00,159 --> 00:35:05,119
cool to have a geolocation dashboard of
where things were happening all over the world

507
00:35:05,360 --> 00:35:07,320
related to their thing, And I'm
like, you know what, I did

508
00:35:07,440 --> 00:35:09,519
need to drive that with real data. I could have just randomly pinged a

509
00:35:09,559 --> 00:35:13,960
spot on the world on a flat, you know, two dimensional map and

510
00:35:14,000 --> 00:35:15,559
be like, look it's happening.
I didn't need to know the lights coming

511
00:35:15,679 --> 00:35:20,559
up all over Yeah, for sure. It's just so totally unnecessary. And

512
00:35:20,599 --> 00:35:24,480
so I've been on this maybe personal
vendetta here to make them be actionable.

513
00:35:24,760 --> 00:35:29,119
So you know, what is the
business impact? But more than that,

514
00:35:29,199 --> 00:35:31,400
you know what will you do with
that information? Is the site being slow?

515
00:35:31,440 --> 00:35:36,280
Will you actually take an action here
to make it faster? Or is

516
00:35:36,280 --> 00:35:38,760
there like a run book or something
that we can go and actually execute on.

517
00:35:38,880 --> 00:35:42,960
Well that's another interesting part of it
all is like the term run book

518
00:35:42,960 --> 00:35:46,440
has been so ruined by so many
places. Like if your run book is

519
00:35:46,480 --> 00:35:52,280
log into the server and run the
script, like yeah, it's just not

520
00:35:52,320 --> 00:35:55,039
automated, and like I think our
run book should be these are the places

521
00:35:55,039 --> 00:35:58,960
you should look, like, if
it's ever a thing we know that breaks,

522
00:35:59,239 --> 00:36:01,840
fix it. Like, yeah,
I talked to someone once that was

523
00:36:01,840 --> 00:36:05,760
saying they were very proud. I
mean I felt that or they built this

524
00:36:05,800 --> 00:36:07,960
great thing and like we had this
service the crashes all the time, and

525
00:36:07,960 --> 00:36:13,039
we built this great thing that automatically
restarts it. I was like, okay,

526
00:36:14,320 --> 00:36:16,679
I see right. How about collecting
stack traces and sending it to developers

527
00:36:16,719 --> 00:36:21,679
and fixing the reason the service crashes? All like, yes, you should

528
00:36:21,719 --> 00:36:24,119
restart the service when it crashes,
no argument, But like, that is

529
00:36:24,119 --> 00:36:27,840
not the end of your journey.
That is the first ten percent of your

530
00:36:27,920 --> 00:36:30,519
journey, right, you know,
I you know, I love that because

531
00:36:30,519 --> 00:36:32,880
it actually happened at one of the
previous companies I was in this. The

532
00:36:32,960 --> 00:36:38,000
name of the service was the Service
Monitor Monitor, and it did actually do

533
00:36:38,079 --> 00:36:43,039
this. The root cause, though, and maybe you've got some infinite wisdom

534
00:36:43,079 --> 00:36:49,039
here, is they were using a
library for math operations that had a memory

535
00:36:49,079 --> 00:36:52,920
leak in it, and so this
would have required actually contacting a third party

536
00:36:52,920 --> 00:36:57,960
company to get their open source software
actually fixed, which I think was under

537
00:36:57,960 --> 00:37:04,519
a proprietor use license. So sometimes
you have to do some ridiculous things.

538
00:37:05,119 --> 00:37:07,320
Yeah, I mean, I guess
if you're in a world where you have

539
00:37:07,400 --> 00:37:10,800
to use a vendor library you can't
fix, you may just have to you

540
00:37:10,840 --> 00:37:15,519
know, live with the workarounds and
acknowledge the terror. Yeah, no,

541
00:37:15,639 --> 00:37:20,119
for sure. The problem is when
you start expending that to every problem that

542
00:37:20,199 --> 00:37:23,880
looks similar, rather than knowing that
it's the right answer. Right the Nightly

543
00:37:23,920 --> 00:37:31,239
Jenkins restart. Yeah, well go
ahead. I would say, like,

544
00:37:31,280 --> 00:37:36,519
you know, a lot of some
bank places I've worked, we're like,

545
00:37:36,559 --> 00:37:39,039
hey, we should just you know, we're twenty four by five point five,

546
00:37:39,159 --> 00:37:42,920
right, you know the Japan and
US markets and they're closed, you

547
00:37:42,920 --> 00:37:47,559
know, Friday night. Let's just
reboot everything every Saturday. I'm like why,

548
00:37:47,679 --> 00:37:51,320
Like why not? And I was
like, I think I asked the

549
00:37:51,400 --> 00:37:55,599
question first, like like yeah,
we should. We should reboot things when

550
00:37:55,639 --> 00:38:00,000
we upgrade the kernel, and we
should use that time to apply upgrade.

551
00:38:00,159 --> 00:38:02,599
But just doing a bunch of stuff
every Saturday because we can because there's no

552
00:38:02,679 --> 00:38:07,840
services. It's like it feels like
we're just making busy work and you know,

553
00:38:08,079 --> 00:38:10,800
finding things to break on a Saturday
and ruin our weekends. Yeah,

554
00:38:10,800 --> 00:38:15,320
but there's also really hard arguments to
argue against, even though you know there's

555
00:38:15,360 --> 00:38:23,239
somehow fundamentally wrong. Yeah, we
didn't do it. Regress, Yeah,

556
00:38:23,239 --> 00:38:27,239
someone you know, there was a
change set out to put a reboot in

557
00:38:27,239 --> 00:38:31,480
a crown job, and I was
like, that is no, that's something

558
00:38:31,559 --> 00:38:35,480
is going to be terrible there,
and I don't want to that's how to

559
00:38:35,559 --> 00:38:38,800
ruin a weekend one on one.
Yeah, totally. And it was already

560
00:38:38,840 --> 00:38:43,800
like that because I don't know.
I think twenty four by seven is a

561
00:38:43,880 --> 00:38:47,280
way better environment to have fundamentally good
operations than twenty four or five five.

562
00:38:47,280 --> 00:38:51,360
There's just too many bad habits in
twenty four or five five. Oh,

563
00:38:51,599 --> 00:38:53,920
we can always restart this by turning
off the database and running you're turning off

564
00:38:53,960 --> 00:38:57,599
the clients and running an alter.
Eventually you're going to have to run an

565
00:38:57,599 --> 00:39:00,880
alter mid week during trading. Yeah, you're not gonna like it, but

566
00:39:00,880 --> 00:39:02,320
it's gonna happen, and you're gonna
have to take an out us to do

567
00:39:02,320 --> 00:39:05,599
it because you haven't figured out how
to do it. I mean, there's

568
00:39:05,639 --> 00:39:08,000
two obvious failure modes from now,
which are, well, what happens if

569
00:39:08,039 --> 00:39:13,320
something changes on the restart, like
you know, a new upgrade or something

570
00:39:13,400 --> 00:39:17,280
right now you're triggering that at an
unpredictable time, or just straight database replication

571
00:39:17,440 --> 00:39:21,760
crashing and losing out on whatever what
was in the journal at that moment,

572
00:39:22,639 --> 00:39:25,079
that's not the thing you want to
actually have happened. Yeah, do it

573
00:39:25,119 --> 00:39:28,719
in your lab to figure out how
to deal with it, but maybe not

574
00:39:28,760 --> 00:39:37,000
proud. Yeah. So You've mentioned
Jenkins and Garrett, and so I feel

575
00:39:37,039 --> 00:39:40,519
like I'm picking up on a trend
here that you're running a lot of services

576
00:39:40,679 --> 00:39:45,239
in house, whereas other companies may
choose to use SaaS providers for those Do

577
00:39:45,280 --> 00:39:50,159
you have a particular opinion on that? Well, I mean I feel like

578
00:39:50,360 --> 00:39:54,360
my whole career, I've been at
all angles of the build versus by debate.

579
00:39:54,800 --> 00:39:59,719
It was funny some of the banks
I've worked that you really any of

580
00:39:59,719 --> 00:40:04,440
the big orgs you see, like
the historical cio CTOs whoever they are,

581
00:40:04,760 --> 00:40:08,039
or one wile will get hired and
there we have to build everything. Then

582
00:40:08,239 --> 00:40:10,760
then you can you know, you
have ten years of this right, and

583
00:40:10,760 --> 00:40:14,400
then you look back and you have
all these disparate things of like, oh,

584
00:40:14,440 --> 00:40:17,239
this must have been built during the
build era of two thousand and nine.

585
00:40:17,679 --> 00:40:22,880
I mean, I think the answer
is, really it depends on the

586
00:40:22,920 --> 00:40:25,920
staff and what their expertise is,
and should you run Jenkins in house?

587
00:40:25,920 --> 00:40:29,599
What can you run Jenkins in house? Right? Like do you have are

588
00:40:29,639 --> 00:40:34,480
you going to dedicate the resources to
do it? Right? But just outsourcing

589
00:40:34,519 --> 00:40:37,639
something isn't always as easy as just
paying somebody, right we mentioned before,

590
00:40:37,639 --> 00:40:43,360
it's porting stuff there, it's operationalizing
it, and actually, like taking taking

591
00:40:43,360 --> 00:40:47,800
advantage of a SaaS service to do
provide value is sometimes just as hard.

592
00:40:49,119 --> 00:40:52,559
The challenge isn't running the infrastructure,
it's using the infrastructure effectively. Right,

593
00:40:52,599 --> 00:40:58,800
Having Jenkins doesn't help. Having jobs
that do meaningful things are important, and

594
00:40:58,880 --> 00:41:01,280
you have that problem in any CI
infrastructure. So I think some of the

595
00:41:01,320 --> 00:41:07,400
things that doesn't matter, you know, observability, right, it's really hard

596
00:41:07,480 --> 00:41:12,920
to run high availability observability. It's
also I'm mouthful to say so, I

597
00:41:12,960 --> 00:41:15,039
mean, like I'm a fan of
I mean worker reservability company, but I

598
00:41:15,039 --> 00:41:17,719
mean I'm a fan of outsourcing some
of that stuff. I think there's a

599
00:41:17,719 --> 00:41:22,239
certain scale where you run it internally, I think very few people are at

600
00:41:22,239 --> 00:41:29,760
that scale. Yeah, very few
people are actually running for four nines proper,

601
00:41:30,360 --> 00:41:32,719
Like that's I don't know if it's
a triple digit number, but it's

602
00:41:32,719 --> 00:41:36,119
a small amount of companies and actually
a lot of people think they're doing it,

603
00:41:36,159 --> 00:41:38,320
but not a lot of people actually
do it, right, It's interesting.

604
00:41:38,320 --> 00:41:42,360
I mean, we're we're five nines
on our core competency service, which

605
00:41:42,400 --> 00:41:45,440
is like identifying stuff, but yeah, it's huge. We're not we're not

606
00:41:45,599 --> 00:41:50,719
running the monitoring observability stuff ourselves like
we are, like we've found we're using

607
00:41:50,719 --> 00:41:52,760
our cloud provider or actually we're still
in the process of trying to find a

608
00:41:52,840 --> 00:41:57,079
vendor that actually works with us.
And I think that's going to fuel my

609
00:41:57,199 --> 00:42:00,280
next question. I'm curious whether or
not you see companies get the build versus

610
00:42:00,280 --> 00:42:05,719
buy decision, right. I know
that they put a lot of effort into

611
00:42:05,840 --> 00:42:09,079
comparing vendors, but then I feel
like it's sort of there's this gap on

612
00:42:09,679 --> 00:42:15,920
actually being able to correctly identify what
the total cost of ownership is if they

613
00:42:15,920 --> 00:42:19,840
do actually go and build or run
something themselves. You know that that is

614
00:42:19,840 --> 00:42:22,360
a tough one, right. TCO
is so hard because it's easy to do

615
00:42:22,440 --> 00:42:25,360
the this is my AWS bill,
this is the vendor bill. How do

616
00:42:25,360 --> 00:42:30,599
you quantify the adages and the people
and the stress and the upgrades? And

617
00:42:30,159 --> 00:42:32,199
yeah, I haven't seen a good
answer for that. I mean, of

618
00:42:32,239 --> 00:42:36,360
course many you know, any vendor
will try to do that for you because

619
00:42:36,400 --> 00:42:39,079
they of course want to help.
Yeah, good, but that one is

620
00:42:39,119 --> 00:42:43,639
tough, I mean, but seeing
it done right. I think the most

621
00:42:43,679 --> 00:42:47,320
common mistake people make when they're doing
the evaluation is writing a requirement stock is

622
00:42:47,320 --> 00:42:52,639
surprisingly hard. Yeah, like writing
a requirement stock for CI and don't use

623
00:42:52,679 --> 00:42:58,239
a single product name right right.
People say, like, my requirement is

624
00:42:58,280 --> 00:43:00,199
an Envoyd proxy. It's like,
there's absolut no way. If your requirement

625
00:43:00,239 --> 00:43:04,760
is an on Boid proxy, what
have you done? Right? If your

626
00:43:04,760 --> 00:43:07,760
requirement is a thing that speaks XDS, maybe envoys the only answer. But

627
00:43:07,840 --> 00:43:12,679
write down the thing you want,
the outcome you want. This is the

628
00:43:12,719 --> 00:43:16,159
same root cause versus slow thing,
Like it's a it's the x Y problem

629
00:43:16,199 --> 00:43:19,679
as well. I don't know if
yeah yeah, people say I want to

630
00:43:19,679 --> 00:43:22,559
do this thing is like, what
are you really trying to accomplish? And

631
00:43:22,599 --> 00:43:25,320
then let's figure out this is what
we could build and accomplish that. This

632
00:43:25,360 --> 00:43:29,360
is what we can buy and accomplish
that or not accomplish that, and maybe

633
00:43:29,400 --> 00:43:31,440
the decision gets more obvious, but
it's just so hard to frame it in

634
00:43:31,519 --> 00:43:36,840
terms of completely agnostic to the tool, the thing you want to do.

635
00:43:37,400 --> 00:43:38,760
You get to be the bad guy. You you have to say, well,

636
00:43:39,119 --> 00:43:42,199
you were going to do an easy
job, which is just pick a

637
00:43:42,239 --> 00:43:45,000
tool. Uh, and now you're
forcing them to go back to the drawing

638
00:43:45,039 --> 00:43:49,840
board like well, why you know, really really look right? Why do

639
00:43:49,880 --> 00:43:54,000
you want your builds to succeed?
Yeah? Exactly, But it's a good

640
00:43:54,079 --> 00:43:57,239
question, like in the CI case, like why do you want your bills

641
00:43:57,280 --> 00:44:00,400
to succeed? Because I want to
merge code faster and ship code to production

642
00:44:00,480 --> 00:44:04,599
faster. Okay, so your metric
is time from commit to development, time

643
00:44:04,599 --> 00:44:07,199
from commit to production. Okay,
we came up with really great metrics from

644
00:44:07,199 --> 00:44:14,480
asking guard. I think it usually
it seems kind of like, I don't

645
00:44:14,480 --> 00:44:17,119
know, weird at first, and
you know, contrived, but I think

646
00:44:17,159 --> 00:44:20,840
it actually leads to like, oh, okay, yeah that makes sense.

647
00:44:21,079 --> 00:44:23,639
No, I'm totally what do Yeah, it makes me think that there's like

648
00:44:23,719 --> 00:44:28,000
some Freudian stuff here where I just
want to sit here with a pipe and

649
00:44:28,039 --> 00:44:34,039
go. But why do you want
your bill to succeed? Well? Is

650
00:44:34,039 --> 00:44:37,719
it because of unresolved issues with your
mother that you feel your build business succeed?

651
00:44:38,360 --> 00:44:44,400
I mean when you pull individuals into
an organization, their personal values do

652
00:44:44,679 --> 00:44:50,679
impact what that organization drives as important
and sometimes I've known software engineers to be

653
00:44:51,039 --> 00:44:55,159
uh, quite illogical and driven by
their emotional state to you know, it

654
00:44:55,199 --> 00:44:59,400
has to be like this, it's
so much better, and you know,

655
00:44:59,440 --> 00:45:02,760
you joke, but there is something
there. No, that's definitely I mean,

656
00:45:02,800 --> 00:45:13,719
I'm probably guilty of that. Yeah. So you work for a observability

657
00:45:13,760 --> 00:45:17,360
company and you're observing your internal tools. Do you use your own product for

658
00:45:17,480 --> 00:45:22,119
that? We do? Yeah,
Yeah, trying to dog food all the

659
00:45:22,159 --> 00:45:29,400
time and say we measure SLOs internally. One cool thing we've done is we

660
00:45:29,480 --> 00:45:31,159
have a bunch of you know,
it's not great, like you know,

661
00:45:31,239 --> 00:45:36,639
some some shell scripts stuff that maybe
shouldn't be shell scripts, but everyone's got

662
00:45:36,760 --> 00:45:40,119
a pile of that somewhere and it's
pretty important in the normal workflow. And

663
00:45:40,159 --> 00:45:45,039
it was having a lot of problems, and uh, you know, we

664
00:45:45,119 --> 00:45:47,239
just found a better way to measure
like what is the success of people that

665
00:45:47,360 --> 00:45:51,159
say I want to run a local
cluster? How often does that fail?

666
00:45:52,039 --> 00:45:57,840
Turns out it was failing a lot
more than we thought. And then the

667
00:45:57,840 --> 00:45:59,960
buggy. It's really hard because you're
like, okay, well could you go

668
00:46:00,159 --> 00:46:02,320
put a set dash X in this
file? And run it again in the

669
00:46:02,360 --> 00:46:07,280
mouth, and no one wants to
like, no one wants to do that,

670
00:46:07,400 --> 00:46:08,239
right. People just want to say, hey, I think broken,

671
00:46:08,360 --> 00:46:12,519
I can't you just fix it?
Right? So we built all of this

672
00:46:12,639 --> 00:46:15,559
in even to our shell right,
so like we have like a tracing view

673
00:46:15,599 --> 00:46:20,079
I think of micro service tracing.
Like there's a request I D and you

674
00:46:20,159 --> 00:46:22,280
know it hits tny services, Like
you run this one shell script that actually

675
00:46:22,320 --> 00:46:27,000
is running like you know, ninety
three shell scripts underneath it for better or

676
00:46:27,079 --> 00:46:30,239
worse or worse, but it's happening, and we have like a trace graph

677
00:46:30,280 --> 00:46:34,360
of you ran this and like so
the people come and say, man,

678
00:46:34,400 --> 00:46:37,280
it's took nine minutes to start this
thing and it normally takes six minutes.

679
00:46:38,159 --> 00:46:42,800
This sucks. Why is this?
So we can go in, look up

680
00:46:42,840 --> 00:46:45,480
their user name, find with a
nine minute run, click it, look

681
00:46:45,559 --> 00:46:49,000
at the trace, and go,
oh, yeah, there was this bug

682
00:46:49,079 --> 00:46:53,280
pulling from ECR or whatever it is, right, and just doing that helped

683
00:46:53,320 --> 00:46:57,800
us pin down. It turns out
all these problems with the tool were like

684
00:46:57,840 --> 00:47:00,599
systemic to one or two things being
wrong, you know, poor assumptions being

685
00:47:00,599 --> 00:47:04,880
made, and we fix those and
we're kind of off to the races on

686
00:47:04,920 --> 00:47:08,079
it, and now we're building a
prober for it, so we'll have like

687
00:47:08,119 --> 00:47:10,039
a graph. You know, again, everything has a graph, right,

688
00:47:10,079 --> 00:47:14,559
So the same way we have an
solo for ingesting an observation in a certain

689
00:47:14,599 --> 00:47:17,079
amount of time, we'll have an
sol for Hey, this thing can create

690
00:47:17,119 --> 00:47:22,000
a environment locally and it happens in
less than this amount of time. And

691
00:47:22,000 --> 00:47:24,960
then we'll have variants of it,
like does it work for people that run

692
00:47:24,960 --> 00:47:28,519
it over and over again on their
box? What about a new engineer that

693
00:47:28,519 --> 00:47:30,320
logs into a fresh box and runs
it, because that's always a different you

694
00:47:30,400 --> 00:47:35,719
know, there's some you know,
that's where always the goblins live in these

695
00:47:35,719 --> 00:47:37,760
things. Well, my terraform works
fine? Oh man, I destroyed it.

696
00:47:37,800 --> 00:47:40,239
Have to run it from scratch,
you know, does that work?

697
00:47:40,480 --> 00:47:45,760
So kind of trying to measure all
those different things from that using our tool.

698
00:47:46,760 --> 00:47:51,559
Is there like an eBPF integration here
that you plan on utilizing in the

699
00:47:51,599 --> 00:47:55,679
future to understand what the requests are
or how the script is running fundamentally on

700
00:47:55,719 --> 00:47:59,039
the machine. Yeah, I think
I think we're looking at that, and

701
00:47:59,119 --> 00:48:01,039
I haven't dealt with it too much
myself, but I think that that is

702
00:48:01,119 --> 00:48:05,679
kind of the ultimate for this,
I would I think marry the two right

703
00:48:06,119 --> 00:48:08,039
VPF for the raw like just show
me everything that's running and helped me and

704
00:48:08,079 --> 00:48:15,119
then my injected data. So yeah, I think that is in the future

705
00:48:15,639 --> 00:48:20,159
for your internal tools, Like once
you identify them, you're like, Okay,

706
00:48:20,440 --> 00:48:23,639
we need to bring this up to
being like a part of our we

707
00:48:23,639 --> 00:48:28,199
need to treat it like it's part
of our core infrastructure. How do you

708
00:48:29,719 --> 00:48:35,800
socialize that across engineering so that everyone
knows that this is a supported tool.

709
00:48:35,840 --> 00:48:38,519
This is a preferred path if you
were thinking about going and building something on

710
00:48:38,559 --> 00:48:42,559
your own for this one use case, we already have you covered here.

711
00:48:42,760 --> 00:48:49,079
How do you communicate that email to
the whole list? Pending your Slack message?

712
00:48:49,079 --> 00:48:52,800
What could you go wrong? Right? Channel? Baby? You can

713
00:48:52,880 --> 00:48:55,920
write documentation all day, but like
I think we all know how much our

714
00:48:55,920 --> 00:49:01,239
documentation gets read, right, I
think I think the way to do it

715
00:49:01,320 --> 00:49:06,119
is to this you know, goes
on the layer aight problem we're talking about

716
00:49:06,119 --> 00:49:08,400
earlier, but like have to build
a good relationship with all these teams and

717
00:49:08,440 --> 00:49:13,639
have them come to you sooner in
the process, like the sooner infrastructure and

718
00:49:13,800 --> 00:49:17,599
s E. Folks can be involved
in anything, like before code is written

719
00:49:17,639 --> 00:49:21,239
would be super ideal, so that
them come to you and say, hey,

720
00:49:21,280 --> 00:49:23,079
I wrote this really cool thing,
but it uses mago dB. It's

721
00:49:23,079 --> 00:49:27,719
like, we don't do that here, right, Like maybe there's a great

722
00:49:27,760 --> 00:49:30,000
reason that does that, but you
know, but it's much harder once they've

723
00:49:30,000 --> 00:49:32,719
written code, right, Like,
you know, it's the poker pot committed

724
00:49:32,760 --> 00:49:36,320
things like while I call for the
fluster on the flop, I'm putting all

725
00:49:36,320 --> 00:49:37,519
my money in on the tourna matter
what. It's like, let's do the

726
00:49:37,559 --> 00:49:42,639
math. Not great, get to
them, you know, pre flop,

727
00:49:42,880 --> 00:49:45,719
right, Like should you even be
in this pot with mango dB? Right?

728
00:49:45,760 --> 00:49:50,920
Maybe? Yeah, but that's not
easy to do, right, You

729
00:49:51,000 --> 00:49:54,360
have to There's not a technical answer
there. That's just build relationships, make

730
00:49:54,400 --> 00:50:00,159
your team available, make people you
know, had interactions people like you know.

731
00:50:00,599 --> 00:50:02,199
I think it was back to the
sort of product management thing earlier that

732
00:50:02,239 --> 00:50:07,199
we were talking about, where if
you are at the point where you need

733
00:50:07,239 --> 00:50:09,440
to tell people about the thing that
you're working on, like maybe you didn't

734
00:50:09,440 --> 00:50:15,000
approach the situation necessarily in the best
way rather than driving it from how you

735
00:50:15,039 --> 00:50:16,920
would a startup, which is okay, you know, what do our customers

736
00:50:17,000 --> 00:50:20,840
pain points look like? And you
know, as users, what do they

737
00:50:20,840 --> 00:50:22,360
want? And they're coming to us
and saying, hey, when are you

738
00:50:22,400 --> 00:50:27,280
done with this thing that we asked
for? And the implementation details are of

739
00:50:27,320 --> 00:50:30,880
course what you're picking because you know
that best. But fundamentally it's just a

740
00:50:30,920 --> 00:50:35,760
matter of pinging them on whatever RSS
feed that they're looking at. Yeah.

741
00:50:35,880 --> 00:50:40,079
I worked at a large company before
and we had an infrastructure PM and it

742
00:50:40,159 --> 00:50:45,360
was amazing. It was just so
it felt like it was easy mode,

743
00:50:45,400 --> 00:50:47,679
right, Like someone else is going
to go gather requirements I don't have.

744
00:50:49,320 --> 00:50:52,039
Yeah, I just I'll go do
some work. That's that's fine. And

745
00:50:52,039 --> 00:50:54,960
then they show up, you know, prioritize on a list. It's like,

746
00:50:55,000 --> 00:50:59,480
well this is this is awesome,
this is what it's like on the

747
00:50:59,480 --> 00:51:01,760
other side, right, you just
need to plug that into chat GPT and

748
00:51:01,880 --> 00:51:04,920
get the answer out as well,
and then you can just you know,

749
00:51:04,960 --> 00:51:10,239
stop doing all the work altogether.
What do my engineers want? That's interesting?

750
00:51:10,960 --> 00:51:15,239
Get a slack bot that uses chat
GPT to act as a PM role.

751
00:51:16,920 --> 00:51:20,360
I was just the other way around. Have the pms just answer,

752
00:51:20,480 --> 00:51:22,079
like answer the question of what they
want to have built, and then it

753
00:51:22,119 --> 00:51:27,960
will automatically build it for them.
This is what the hive mind Internet wants

754
00:51:28,000 --> 00:51:31,920
to build. Probably not so bad
if everyone wants to. I don't think

755
00:51:31,920 --> 00:51:37,679
you'll get to five nines. I
think it's more like five two's will be

756
00:51:37,719 --> 00:51:45,079
in the front in there somewhere if
you carry it out to enough decimal places.

757
00:51:45,119 --> 00:51:51,639
There's some nines. You know,
non repeating imager and doesn't really good

758
00:51:54,480 --> 00:51:59,079
cool, So what else should we
be thinking about? For internal tools?

759
00:52:00,320 --> 00:52:05,280
That's your big takeaway piece of advice. Apply all the same riggor Like you

760
00:52:05,280 --> 00:52:10,079
know, when PROD breaks, you
go and declare an incident using your incident

761
00:52:10,119 --> 00:52:15,039
management tool. You have a communications
role, you have you have the whole

762
00:52:15,039 --> 00:52:16,519
thing or you got to run,
but hopefully if you don't, you should,

763
00:52:17,360 --> 00:52:21,119
And then when you're done, you
write a post mortem. You might

764
00:52:21,159 --> 00:52:23,119
even publish. The customers publish it
internally, right, Like why shouldn't an

765
00:52:23,119 --> 00:52:27,559
engineer be able to read about why
Jenkins broke? If production breaks, you're

766
00:52:27,559 --> 00:52:30,119
going to file a bunch of follow
ups. You're going to prioritize it over

767
00:52:30,199 --> 00:52:32,159
other works because we don't want PROD
to break again. Same thing for internal

768
00:52:32,199 --> 00:52:37,039
tools, Like it's really I think
that's the big takeaway is there's really not

769
00:52:37,159 --> 00:52:40,639
much of a difference. I mean, and people, well, if PROD

770
00:52:40,719 --> 00:52:44,639
is down, our customers that pay
us can't do work. Okay, great,

771
00:52:45,039 --> 00:52:47,760
if CI is down, how are
you going to ship a fix when

772
00:52:47,880 --> 00:52:51,880
PROD is down? Right? Like
you're like, oh, I brought this,

773
00:52:51,960 --> 00:52:54,760
I wrote this really great infrastructure where
changes can only go through CI and

774
00:52:54,800 --> 00:52:59,760
not handmade. That's a great thing. But if your CI isn't as of

775
00:53:00,039 --> 00:53:02,559
the ball has production three nine CI
four nine is production. You can only

776
00:53:02,599 --> 00:53:07,079
make changes through CI. You know, not a math major. But we're

777
00:53:07,079 --> 00:53:10,559
going to have a problem here forty
minutes out of the year. You know

778
00:53:10,559 --> 00:53:15,119
it's going to be an issue.
So I think that treating it, I

779
00:53:15,159 --> 00:53:19,960
think people just underestimate how important that
stuff really is and the impact it can

780
00:53:19,960 --> 00:53:22,239
have. You don't have to wait
till it. You know. The worst

781
00:53:22,239 --> 00:53:28,840
case is production is having a problem. You're monitoring is broken and you can't

782
00:53:28,840 --> 00:53:30,719
see it. Your CI is broken
and you can't ship a fix for it,

783
00:53:31,079 --> 00:53:35,519
whether that fixes can FIG or code, I mean, and then that's

784
00:53:35,519 --> 00:53:38,119
a really terrible postportum to have to
send out to a customer of well,

785
00:53:38,159 --> 00:53:42,760
we knew what was wrong, but
we had to fix. We couldn't build

786
00:53:42,800 --> 00:53:46,119
the dockor image that had to fix
because you know our two thousand and four

787
00:53:46,159 --> 00:53:51,280
era Jenkins decided it's a crash and
I'll start it. Yet, Yeah,

788
00:53:51,400 --> 00:53:53,079
that's not It was not a good
look for anybody. I mean, you

789
00:53:53,119 --> 00:53:58,000
identified it. If there's value in
doing this activity for some of your services

790
00:53:58,079 --> 00:54:00,039
because of what the users look like, then there's probably value in doing it

791
00:54:00,079 --> 00:54:04,400
for other services that just happen to
be internal. I think that's a big

792
00:54:04,440 --> 00:54:07,280
part of post mortems, right,
Like if you find a production problem where

793
00:54:07,440 --> 00:54:10,440
oh, hey, we had this
bug in our database connection pool and we

794
00:54:10,480 --> 00:54:14,440
had this reconnection issue and this thing
happened. Where else do you have connection

795
00:54:14,480 --> 00:54:16,519
pools? Right? That should be
the logical question. You ask the same

796
00:54:16,519 --> 00:54:21,440
thing internally, Right, we had
this neglected service. Where else what else

797
00:54:21,599 --> 00:54:24,880
is flying under the radar that's going
to bite us? And those are the

798
00:54:24,880 --> 00:54:28,599
tough post you know, some of
the post mortem actually like fix this bug

799
00:54:28,639 --> 00:54:30,960
with a reproduced case. Okay,
that's like a day of work, right,

800
00:54:30,239 --> 00:54:32,360
you know, when you're done because
the test passes, it's easy.

801
00:54:34,000 --> 00:54:38,199
These are much more ominous, like
project the follow ups. But if you

802
00:54:38,199 --> 00:54:43,199
don't do them, you're going to
pay the price. Is there like some

803
00:54:43,360 --> 00:54:47,599
obvious pitfall that a lot of companies
or maybe even everyone seems to get wrong

804
00:54:47,800 --> 00:54:52,920
in this area sort of besides the
stuff we've been talking about, well in

805
00:54:52,960 --> 00:54:59,840
post mortems, people are extraordinarily bad
at distinction between root causes and triggers.

806
00:55:00,920 --> 00:55:04,440
Right, Pete type this command and
took the site down? Right, root

807
00:55:04,480 --> 00:55:09,320
cause is not Pete sucks? Right, that's right? Like the root cause

808
00:55:09,440 --> 00:55:13,599
is why was there a command to
where are the where are the seatbelts?

809
00:55:13,639 --> 00:55:15,440
Where are the gates? Where is
the code review? Where's all that stuff?

810
00:55:15,480 --> 00:55:19,920
Or you have to really really I
think the five wise thing is interesting.

811
00:55:19,920 --> 00:55:22,360
I don't think you actually have to
write down why five times and a

812
00:55:22,440 --> 00:55:27,159
documented filled out I think that's a
little bit you know, the meaning and

813
00:55:27,239 --> 00:55:30,000
not great, but the philosophy of
like, really come to the root cause

814
00:55:30,039 --> 00:55:36,440
of the problem, right, Like
root causes aren't well? Aws had an

815
00:55:36,480 --> 00:55:39,239
availability zone die? Like, what
can we do? You can run more

816
00:55:39,280 --> 00:55:42,679
than one availability zone? You can
do this, You can do that right,

817
00:55:42,760 --> 00:55:45,599
like, and maybe you choose not
to at this point, but you

818
00:55:45,599 --> 00:55:50,239
should at least identify it and say
Okay, like we know the root cause,

819
00:55:50,320 --> 00:55:53,000
and we've chosen that this is a
risk, and this is why we're

820
00:55:53,000 --> 00:55:57,599
a three nine service or a four
nine service. And maybe someday will make

821
00:55:57,599 --> 00:55:59,920
it better, maybe we won't,
but at least being honest with yourself about

822
00:55:59,960 --> 00:56:04,800
it. Ah, that's huge right
there. I want to highlight that because

823
00:56:06,760 --> 00:56:08,800
like, just because you identify the
root cause doesn't mean you have to do

824
00:56:08,840 --> 00:56:16,119
anything about it. Because I've seen
multiple instances where companies build infrastructure that is

825
00:56:16,280 --> 00:56:22,960
far beyond their budget and their actual
requirements because they're focused on that. And

826
00:56:22,360 --> 00:56:29,719
the analogy I like to use is
like, whenever I go to work every

827
00:56:29,800 --> 00:56:35,480
day, the fastest, most efficient
way for me to get there is buying

828
00:56:35,519 --> 00:56:39,800
my own jet copter, But my
budget really says I should stick with my

829
00:56:39,840 --> 00:56:44,320
eighty seven Toyota Corolla, you know, and so you have like balance those

830
00:56:44,320 --> 00:56:49,159
two things, right, right,
Maybe the fix is leave ten minutes earlier

831
00:56:49,239 --> 00:56:52,920
instead instead of mind Yeah exactly.
Yeah. Well I think also it comes

832
00:56:52,920 --> 00:56:55,920
back to SLOs of like, Okay, we had this big problem, but

833
00:56:57,119 --> 00:57:00,360
ay was it a big problem?
You know? Like it's hard back to

834
00:57:00,360 --> 00:57:02,440
the emotion thing, right, Like
if a big customer is impacted, it

835
00:57:02,760 --> 00:57:07,239
gets a lot more priority, but
you have to quantify at the end of

836
00:57:07,239 --> 00:57:08,960
the day, like, hey,
this is how many nines we have.

837
00:57:09,079 --> 00:57:14,320
This is our error budget. We
used it during this incident. Maybe that's

838
00:57:14,360 --> 00:57:17,239
not good, but that's this is
par for the course, and we don't

839
00:57:17,320 --> 00:57:22,440
need to suddenly become multi cloud,
multi region, you know, all the

840
00:57:22,480 --> 00:57:25,840
things load balancing, complexity, because
that's the other thing is looking at.

841
00:57:25,960 --> 00:57:30,920
You know, you can add nines
with complexity, but complexity can also reduce

842
00:57:30,000 --> 00:57:35,920
nnes. And you have to be
careful over engineering and response to things.

843
00:57:35,960 --> 00:57:38,760
And I hate to use the term
like you know this is always the escalator.

844
00:57:38,880 --> 00:57:42,119
Well, if there's an act of
God that you know, this managed

845
00:57:42,119 --> 00:57:44,360
service goes away, it's like,
well, what's an act of God?

846
00:57:44,440 --> 00:57:47,079
Is that any going down? I
don't know, that's just expected, right,

847
00:57:49,239 --> 00:57:52,320
Is it a tornado hit the Virginia
area and all of the US's to

848
00:57:52,320 --> 00:57:53,639
one one away? Okay, there's
an act of God we don't have to

849
00:57:53,639 --> 00:57:58,480
plan for. But if you're trying
to be a five nine service, you're

850
00:57:58,480 --> 00:58:00,480
not running in one region anyway,
So you know, it kind of all

851
00:58:00,559 --> 00:58:04,440
has to that has to make sense
together. You know, the whole story

852
00:58:04,440 --> 00:58:07,360
has to just kind of flow.
And that's where I think some people get

853
00:58:07,400 --> 00:58:10,320
off. They they write an SLO, but they don't have a story to

854
00:58:10,360 --> 00:58:15,920
back it, or they write a
really complex story where they don't need for

855
00:58:15,000 --> 00:58:22,719
an solo that's simpler. Yeah,
point for sure. But it's an art

856
00:58:22,840 --> 00:58:25,840
so it's you know, there's no
right or wrong answer. I always say

857
00:58:25,960 --> 00:58:30,880
solo everything is so technical and solos
or this like walfty, what is right?

858
00:58:30,920 --> 00:58:35,360
What is wrong? Who really knows? You just kind of have to

859
00:58:35,360 --> 00:58:37,679
do it. People always we were
rolling out SLOs in a company. How

860
00:58:37,679 --> 00:58:39,960
do I know my slo's right?
I was like, well, you know,

861
00:58:40,000 --> 00:58:44,400
it's not. It's your first SLO. So you build it and you

862
00:58:44,519 --> 00:58:47,440
have post mortems and you adjust it
as you go. You know, were

863
00:58:47,480 --> 00:58:51,159
you at you know, did you
have an outage? Yes? Did your

864
00:58:51,280 --> 00:58:54,519
solo show it no cool? Make
it more aggressive? Did you not have

865
00:58:54,559 --> 00:58:58,239
an outage? And your SLO says
you had a outage? Make it less

866
00:58:58,239 --> 00:59:00,800
aggressive? Right? Like it sounds
simple, but that's just the feedback loop.

867
00:59:00,800 --> 00:59:05,079
And if you're doing it twelve months
later, you probably have a pretty

868
00:59:05,079 --> 00:59:09,599
decent setup and have a much better
idea of what your customers consider an outage

869
00:59:09,679 --> 00:59:14,000
or not. You hit on something
really interesting there. Actually, So if

870
00:59:14,159 --> 00:59:17,559
SLA is your you know, contracted
amount and the and the I is whatever,

871
00:59:17,559 --> 00:59:22,239
your indicator is always just an objective. So it does seem like it's

872
00:59:22,280 --> 00:59:25,039
it must be subjective in every way. How do you sort of pick that?

873
00:59:25,119 --> 00:59:28,599
How do you know what? First
off, if you have an A,

874
00:59:29,079 --> 00:59:31,320
the O must be more aggressive,
let's hope, right, right,

875
00:59:32,679 --> 00:59:37,960
But let's assume for a moment you
don't have a contractual you know, I

876
00:59:37,239 --> 00:59:40,079
say, you know slas are like
SLOs with lawyers, right, it's really

877
00:59:40,400 --> 00:59:45,800
kind of a yeah, you have
to guess, Like I mean, I

878
00:59:45,800 --> 00:59:49,320
think that's the unfortunate of it.
And that's where you know, hire somebody

879
00:59:49,320 --> 00:59:52,440
with experience. They'll be able to
guess more accurately maybe, but they'll at

880
00:59:52,519 --> 00:59:55,559
least also know when they're wrong.
So I don't know. It's a process

881
00:59:55,559 --> 01:00:00,119
where you just have to iterate and
sometimes you have a major miss You're like,

882
01:00:00,159 --> 01:00:02,360
oh man, we really thought we
were measuring this service and we had

883
01:00:02,360 --> 01:00:07,239
this massive outage in our dashboard.
Was like everything's green, nothing's wrong,

884
01:00:07,280 --> 01:00:10,159
and users are entre It's like,
okay, well we missed this super critical

885
01:00:10,199 --> 01:00:15,599
part of the picture, and then
you do the postmortem thing and you figure

886
01:00:15,599 --> 01:00:17,440
out, Okay, well I made
this common mistake. What other of my

887
01:00:17,599 --> 01:00:22,199
solos have this common You know that
I make this mistake more than once.

888
01:00:22,360 --> 01:00:27,679
Would it be fair to say that
it should be meaningful so that if you're

889
01:00:27,760 --> 01:00:30,639
violating it, then you're taking some
action as a result, and if you're

890
01:00:30,679 --> 01:00:34,960
not, then you don't do anything. So maybe it's about finding that sweet

891
01:00:34,960 --> 01:00:37,519
spot where it causes the right thing
to happen in your organization. Absolutely,

892
01:00:37,599 --> 01:00:42,000
I mean I think that. I
think to get there right, you need

893
01:00:42,039 --> 01:00:45,639
to have some kind of alerting and
reporting around it. I think alerting on

894
01:00:45,800 --> 01:00:50,119
solos is like a very hard problem. Like the Google book will talk about

895
01:00:50,119 --> 01:00:52,599
burden rate alerting. Have fun implementing
that, right, that's very hard.

896
01:00:52,840 --> 01:00:55,920
But if you can get there or
have some approximation of it and report on

897
01:00:55,960 --> 01:01:00,639
it, like I'm a big fan
of getting you know, it doesn't work

898
01:01:00,719 --> 01:01:02,840
right away, but eventually maturing to
a point where every alert has a ticket,

899
01:01:04,239 --> 01:01:08,440
right and the ticket is either fixed
the thing that caused the alert.

900
01:01:08,719 --> 01:01:13,199
You know, this needs to be
more resilient, This needs more replicas or

901
01:01:13,360 --> 01:01:15,599
this alert page mean nothing was actually
wrong. We should fix the alert and

902
01:01:15,639 --> 01:01:19,199
if you do that over time,
like that's how you can get solos and

903
01:01:19,280 --> 01:01:22,000
cause change, and when they really
are broken, the post mortem loop is

904
01:01:22,000 --> 01:01:27,480
the real fix. Like the trick
with SLOs is you kind of I don't

905
01:01:27,480 --> 01:01:30,239
know. There's that perpetual battle with
infrastructure and product right You're like, hey,

906
01:01:30,280 --> 01:01:32,719
we need you guys to write more
stable code and products, like we

907
01:01:32,760 --> 01:01:36,079
need features, we need this needs
to be a different color, right,

908
01:01:36,239 --> 01:01:38,480
which is fine, like that needs
there's a balance there. But if you

909
01:01:38,519 --> 01:01:42,800
get everyone to agree on SLOs and
you're really user focused, like, hey,

910
01:01:43,840 --> 01:01:46,719
users are happy when the latency you
know p ninety nine latency is this,

911
01:01:46,800 --> 01:01:50,519
and you just are sad when it's
over that, and everyone everyone,

912
01:01:50,599 --> 01:01:55,159
like business engineering management, agrees on
that. Your argument is a lot easier

913
01:01:55,199 --> 01:01:59,960
when you have an outage to say, our users weren't happy, Like objectively,

914
01:02:00,599 --> 01:02:05,079
our users weren't happy, So we
either fix the thing or we decide

915
01:02:05,119 --> 01:02:07,639
that our threshold was incorrect on what
is a happy user. But one of

916
01:02:07,639 --> 01:02:12,480
the two has to We can't do
nothing. We can't just write features and

917
01:02:12,559 --> 01:02:15,159
say, well, users might be
unhappy again, because we know this thing

918
01:02:15,199 --> 01:02:19,039
will blow up, and I find
that that is a good you know,

919
01:02:19,039 --> 01:02:21,599
you almost trick the trick product into
it and like, oh yeah, so

920
01:02:21,599 --> 01:02:22,280
those these are great, and then
later it's like, oh, yeah,

921
01:02:22,280 --> 01:02:25,119
I guess we have to fix this
now, so you know, we'll give

922
01:02:25,239 --> 01:02:30,880
we'll give you a sprint or you
know, whatever it is. You know,

923
01:02:30,039 --> 01:02:35,000
I did the same trick a long
time ago with okayrs. Very similar

924
01:02:35,079 --> 01:02:37,840
thing, where you know, once
they agree to the okay ares, you

925
01:02:37,880 --> 01:02:39,599
try to set the mindset up what
does this actually mean? You know,

926
01:02:39,599 --> 01:02:43,760
why are we setting it? And
you set the so we can know what

927
01:02:43,800 --> 01:02:45,599
to do when we know what to
do the right thing, and then later

928
01:02:45,840 --> 01:02:47,559
you can just point back to it
and be like, hey, you know,

929
01:02:47,599 --> 01:02:50,960
we decided what the right thing was
going to be in this situation.

930
01:02:51,320 --> 01:02:52,960
Now it's time to execute on it. Or you know, you have to

931
01:02:53,000 --> 01:02:58,639
make a trade off. As you
said, Yeah, it's a tough.

932
01:02:58,840 --> 01:03:04,400
You know, you can't sell people
problems with tech, but you try.

933
01:03:07,320 --> 01:03:10,519
That'll be the million dollars million dollars
you know thing. But I don't know,

934
01:03:10,599 --> 01:03:14,239
I feel like data is always helpful. You know, emotions are bad,

935
01:03:14,239 --> 01:03:17,840
you know, everyone has emotional arguments, but data is hard to People

936
01:03:17,880 --> 01:03:22,039
still argue with data, but if
I feel like if you're out the side

937
01:03:22,039 --> 01:03:27,400
of data, you at least have
a fighting chance of pushing for change,

938
01:03:27,519 --> 01:03:31,199
or how do you know, like
you ever feel like you're in a situation

939
01:03:31,280 --> 01:03:37,400
where you get scared of survivorship bias, where you even if you're doing root

940
01:03:37,440 --> 01:03:40,599
cause analysis and you're finding really what
the underlying problem is that even though you're

941
01:03:40,639 --> 01:03:45,280
going on and fixing it, that
you're missing some other side of the iceberg

942
01:03:45,440 --> 01:03:51,039
that is waiting out there to come
and crush you totally. But I mean

943
01:03:51,079 --> 01:03:54,519
I think that I don't know.
I just accepted it, right. It's

944
01:03:54,519 --> 01:04:00,800
a pestimistic, sary brain, Like
everything will break. Everything I fix will

945
01:04:00,800 --> 01:04:03,800
also break when I leave a company, everybody will blame me when it breaks.

946
01:04:03,840 --> 01:04:08,199
And that's whatever, you know,
Like I've just I've internalized it and

947
01:04:08,239 --> 01:04:11,000
accepted it. And it used to
stress me out a lot more and now

948
01:04:11,039 --> 01:04:15,719
it's just like, Yep, this
thing I'm rolling out might break, and

949
01:04:15,840 --> 01:04:17,760
you know, at least you know
that if it breaks, you're going to

950
01:04:17,800 --> 01:04:20,960
write a good postmartum and learn from
it. And that's kind of the Constellation

951
01:04:21,079 --> 01:04:25,039
prize of like, yeah, it
sucks to write a big Postmartum after a

952
01:04:25,039 --> 01:04:29,239
big adage. But also you know, that's that's the gig, and like

953
01:04:29,440 --> 01:04:32,320
we shouldn't have hired Pete, that's
that's the R. Well, you know,

954
01:04:32,400 --> 01:04:34,679
like the first place I ever with
FedEx, right, there was this

955
01:04:34,840 --> 01:04:40,280
meeting every week, the R squared
Meeting, the Redundancy and Reliability Meeting interesting

956
01:04:40,519 --> 01:04:44,440
and it was like how good of
a week is the VP having? Right?

957
01:04:45,760 --> 01:04:47,599
And there was always this rumor that
like, oh, somebody was fired

958
01:04:47,639 --> 01:04:50,239
once at this meeting because they had
an outage, and like no one can

959
01:04:50,320 --> 01:04:56,719
actually tell you that person's name,
what year it was. I'm pretty sure

960
01:04:56,800 --> 01:04:59,679
it was, you know, it
was trying to hype you up to prepare

961
01:04:59,760 --> 01:05:01,519
for it. I think it was. I don't know if they intentially did

962
01:05:01,519 --> 01:05:05,320
this or it was just a grow
but like, that's such a terrible you

963
01:05:05,360 --> 01:05:10,320
know. I think the blameless culture
is big people can't be a problem,

964
01:05:10,440 --> 01:05:15,000
right, But I think often it's
not the person that made the change that

965
01:05:15,119 --> 01:05:17,679
is actually the problem. It is
someone's reviewing code, right, Like code

966
01:05:17,679 --> 01:05:21,679
review should be first class, not
a checking a box. I feel I

967
01:05:21,719 --> 01:05:26,159
feel really bad when to change I
reviewed cause an adage that happened recently.

968
01:05:26,280 --> 01:05:30,679
I was like, who approoved has
changed? Oh, never mind, it

969
01:05:30,719 --> 01:05:33,760
should have done better due diligence,
And yeah, I mean it goes back

970
01:05:33,800 --> 01:05:36,280
even further than that though, because
if you do think that it is one

971
01:05:36,320 --> 01:05:40,800
person's responsibility that caused the problem,
you can look at, well, you

972
01:05:40,840 --> 01:05:43,559
know, what was the culture we
had that allowed them to make the mistake?

973
01:05:43,679 --> 01:05:45,519
Or you know why were they even
hired? Right? Were they a

974
01:05:45,559 --> 01:05:47,119
good fit for the role in the
first place? And you can definitely go

975
01:05:47,199 --> 01:05:51,320
back up somewhere else or a different
chain to really dive in there some of

976
01:05:51,320 --> 01:05:55,199
those things you might not write.
It's like a MetaPost mortem of the Yeah,

977
01:05:55,199 --> 01:05:57,599
for sure, how do we get
here? Right? How do we

978
01:05:57,679 --> 01:06:01,760
end up hiring people that understand networking
for a networking product or whatever? You

979
01:06:01,840 --> 01:06:05,159
know it? Right? Yeah,
I think that's suff's important. And then

980
01:06:05,199 --> 01:06:09,039
then change the process, right,
Okay, these are a new criteria for

981
01:06:09,119 --> 01:06:13,960
this for sure. Yeah. Yeah. When I was in the Navy,

982
01:06:14,039 --> 01:06:18,519
we had this process for getting your
certification for whatever job you were doing.

983
01:06:18,599 --> 01:06:25,960
That I've tried to bring into poor
request reviews that it takes a cultural shift

984
01:06:26,000 --> 01:06:28,960
to get fully implemented. But in
the Navy. It was set up so

985
01:06:29,000 --> 01:06:30,800
that, like I was a nuclear
engineer, so you had to learn all

986
01:06:30,800 --> 01:06:35,599
these different skills to operate the power
plant. And so you would go around

987
01:06:35,599 --> 01:06:43,559
the power plant and work with the
existing engineers and show them that you knew

988
01:06:43,679 --> 01:06:47,960
how to do something, and if
they felt like you understood it, they

989
01:06:47,960 --> 01:06:50,960
would sign it off in your book. And this was way back pre computer

990
01:06:51,079 --> 01:06:55,639
stuff, so you actually had a
physical book that you carried around, but

991
01:06:55,760 --> 01:06:58,400
you signed it off in that book. And then if at any point in

992
01:06:58,440 --> 01:07:02,079
your career you ever screwed that task
up to a point where your skills were

993
01:07:02,079 --> 01:07:06,000
called into question, they would open
up the book to see who signed it

994
01:07:06,199 --> 01:07:10,519
and go back to that person and
say, hey, why why did we'll

995
01:07:10,559 --> 01:07:15,320
screw this up? And and so
it was that like that. It did

996
01:07:15,320 --> 01:07:18,800
put that sense of pressure on you
so that before you would sign off on

997
01:07:18,840 --> 01:07:25,960
anyone's book on anything, you wanted
to make sure that you were reasonably confident

998
01:07:26,000 --> 01:07:29,840
that they actually knew what they were
doing. And so I pre poor requests

999
01:07:29,840 --> 01:07:33,079
the same way like if I approve
a poor request and it breaks something,

1000
01:07:33,679 --> 01:07:38,360
I don't consider that to be a
fault with the person who submitted the poor

1001
01:07:38,400 --> 01:07:41,800
request. I consider it to be
my fault for not catching it in the

1002
01:07:41,840 --> 01:07:47,840
review. This is like the Hurdos
number corollar ate the uh, there's the

1003
01:07:47,960 --> 01:07:50,239
you know, you chase it back
all the way up the chain, like,

1004
01:07:50,280 --> 01:07:51,920
well, you know, who did
that person? You know, what

1005
01:07:51,920 --> 01:08:01,920
does that person's book look like?
The reviewer? Right? No, I

1006
01:08:01,920 --> 01:08:04,440
think that's a good, good way
to look at it, and it's just

1007
01:08:04,480 --> 01:08:12,239
good for accountability and you know,
yeah the meta issues. Yeah, and

1008
01:08:12,599 --> 01:08:15,279
like a one off instance, you
know, is not that big a deal.

1009
01:08:15,319 --> 01:08:23,279
But over time, you know,
if like everyone that I signed off

1010
01:08:23,319 --> 01:08:27,880
on this particular skill is having problems, that's going to point back to the

1011
01:08:27,960 --> 01:08:33,800
root cause being me not actually understanding
either what the skill is or how to

1012
01:08:33,840 --> 01:08:41,119
evaluate that skill. Yeah. I've
been places where the root cause has just

1013
01:08:41,199 --> 01:08:45,640
been fatigue. It's like the call
rotations are too insane. This was at

1014
01:08:45,640 --> 01:08:48,199
the end of this person covered someone
else's on call. They were on two

1015
01:08:48,199 --> 01:08:51,000
weeks of twenty four by seven on
call. They had had a bunch of

1016
01:08:51,039 --> 01:08:55,079
major incidents overnight, they were running
on no sleep, and they just simply

1017
01:08:55,079 --> 01:08:58,600
did the wrong thing. You can't
fault the human for that. Why do

1018
01:08:58,720 --> 01:09:01,520
we put them in this grinder like
every other industry. Airline industry is a

1019
01:09:01,560 --> 01:09:04,199
great one, right, you can
only have limit limits on how much you

1020
01:09:04,239 --> 01:09:08,039
can fly and be responsible over people's
lives. I mean, obviously when lives

1021
01:09:08,039 --> 01:09:10,680
are instakes, like you said,
you have to be more rigorous. But

1022
01:09:10,880 --> 01:09:13,439
it doesn't have to be that rigorous
for you know, being on call.

1023
01:09:13,479 --> 01:09:15,840
But you know, we have an
informal policy on our team is if we

1024
01:09:15,880 --> 01:09:20,319
take overnight pages, someone will offer
to cover the next day, the next

1025
01:09:20,439 --> 01:09:26,319
night so that person can catch up. Yeah, there's nothing worse than having

1026
01:09:26,359 --> 01:09:29,960
a week of terror where you're losing
sleep every night like you're just you're dead

1027
01:09:30,119 --> 01:09:32,279
by there, and then a major
incident. You know, the worst timing

1028
01:09:32,319 --> 01:09:35,079
always happens in these things, right, Like the worst adages are never one

1029
01:09:35,119 --> 01:09:39,279
thing. It's a confluence of events. Right, So that terrible outage is

1030
01:09:39,279 --> 01:09:43,840
going to come Friday when you're running
on fumes and your brain isn't isn't there,

1031
01:09:44,399 --> 01:09:46,359
and you're going to make mistakes,
you're not going to see problems and

1032
01:09:46,399 --> 01:09:51,920
that's not your fault. But that's
tough to in a startup world like oh

1033
01:09:53,000 --> 01:09:56,760
yes, you're meant to grind,
but there has to be some reasonableness of

1034
01:09:57,600 --> 01:10:00,880
Okay, the people with responsibility are
keeping the sight up need to be aware

1035
01:10:00,960 --> 01:10:04,199
and awake because we're not just running
run books where we copy and paste off.

1036
01:10:04,319 --> 01:10:10,000
It's the run book is use your
brain. Yeah, I think we

1037
01:10:10,079 --> 01:10:14,439
really realized over the at least the
most recent decade that the grind is not

1038
01:10:14,479 --> 01:10:18,159
helpful. Even like doing more hours
of work, especially in knowledge work industry,

1039
01:10:18,760 --> 01:10:23,479
does not translate to an additional value. And so if you do have

1040
01:10:23,880 --> 01:10:27,680
outages every incidence, every day that
people are on, I like, it

1041
01:10:27,720 --> 01:10:31,520
seems like fundamental that you would intentionally
rotate them off so that someone else is

1042
01:10:31,520 --> 01:10:35,760
there because and you know, I
think there's like a pride issue here where

1043
01:10:35,800 --> 01:10:40,560
the engineer just wants to stay on
because you know, it's their rotation and

1044
01:10:40,920 --> 01:10:44,680
they don't realize that it's actually harming
the company. Like they should speak up

1045
01:10:44,680 --> 01:10:47,239
for the benefit of the company.
It's not about them necessarily. Right.

1046
01:10:47,439 --> 01:10:51,479
Heroics shouldn't be I mean, it's
great sometimes heroics just have to happen,

1047
01:10:51,520 --> 01:10:55,079
and they happen and it's good,
but that shouldn't be the goal of like,

1048
01:10:55,119 --> 01:10:57,399
oh man, I want to be
a hero of the Saturagye, please

1049
01:10:57,479 --> 01:11:00,439
know, like it was really fun
early in my career, and now it's

1050
01:11:00,479 --> 01:11:03,720
like, oh, there's heroics happening. This is awful. Yeah, it's

1051
01:11:03,760 --> 01:11:09,880
it's the cult of the hero because
then you you fulfill that hero role and

1052
01:11:09,920 --> 01:11:14,439
then you know, you get at
mentioned in slack and you know, everyone's

1053
01:11:14,479 --> 01:11:15,520
like, oh wow, that was
such a tough effort, you know,

1054
01:11:15,720 --> 01:11:20,840
and the worst yeah, the worst
word to use the NHR rockstar. Yeah.

1055
01:11:20,880 --> 01:11:24,319
Absolutely. I don't want to be
a rock star at work, man,

1056
01:11:24,479 --> 01:11:27,159
Like, please know, I want
to do I'm going to be a

1057
01:11:27,239 --> 01:11:30,439
rock star. I want it to
be on Motley Cruz, fueled by cocaine

1058
01:11:30,479 --> 01:11:32,680
and hookers. Let me be a
rock star, like being a rock star

1059
01:11:32,760 --> 01:11:38,279
and infrastructure is like, yeah,
where's my hotel room to tear up?

1060
01:11:38,439 --> 01:11:41,399
Yeah, I mean at the beginning
it was oh they described me as a

1061
01:11:41,479 --> 01:11:43,800
rockstar, that must mean I'm doing
great. And now it's like, oh,

1062
01:11:45,239 --> 01:11:47,640
now it's like what your mouth?
Yeah, don't call it that.

1063
01:11:50,079 --> 01:11:54,159
But I think I think the culture
is Yeah, I don't know. It's

1064
01:11:54,920 --> 01:11:58,039
it's a tough one to just you
don't want to actively discourage you. You

1065
01:11:58,039 --> 01:12:00,039
don't want to say long off of
the home, like if you want to

1066
01:12:00,079 --> 01:12:02,079
work. I don't know. My
thing is like I'm a workaholic, and

1067
01:12:02,119 --> 01:12:04,840
I always say I'm happy to work
long hours if it's what I want to

1068
01:12:04,880 --> 01:12:09,680
do. If I want to work
on Saturday, awesome. If other people

1069
01:12:09,680 --> 01:12:13,039
want me to work on Saturday,
that kind of falls apart from me.

1070
01:12:14,840 --> 01:12:16,680
And the social engineering trick is that, well, just to keep work that

1071
01:12:16,720 --> 01:12:19,720
he really likes and it's passionate about, and he'll work on Saturday. That's

1072
01:12:19,720 --> 01:12:23,680
a fine. Okay, that's a
good trick. It works on me,

1073
01:12:24,079 --> 01:12:29,279
but I don't think people have used
that. And some weekends I game all

1074
01:12:29,319 --> 01:12:31,960
weekend. Some weekends I work all
weekend. But usually it's it's my choice.

1075
01:12:33,079 --> 01:12:38,760
The point about that, I think
part of that is that we are

1076
01:12:39,319 --> 01:12:43,479
creative in the work that we do, you know, using our creativity to

1077
01:12:43,560 --> 01:12:49,600
solve problems. And creativity doesn't doesn't
show up at nine am when you hit

1078
01:12:49,680 --> 01:12:54,359
the time clock, you know,
and if it's something that you're excited about,

1079
01:12:54,479 --> 01:12:58,039
you you get this, you know, that creative bush like, oh,

1080
01:12:58,079 --> 01:13:00,119
I got to go do this,
which is what leads you to you

1081
01:13:00,960 --> 01:13:04,399
go in and work on Saturday.
Most of my best work is done yeah,

1082
01:13:04,439 --> 01:13:08,560
after midnight or on the weekends correct, Like it's exactly that is.

1083
01:13:08,600 --> 01:13:11,399
You get the idea and you're like, man, you're laying in bed and

1084
01:13:11,439 --> 01:13:13,039
you're like, I can build it
this way, that way. It's like

1085
01:13:13,079 --> 01:13:16,119
past draft to build it right,
Like I know. It always fueled me

1086
01:13:16,359 --> 01:13:20,520
was there was some really annoying problem
that I did, like some other problem

1087
01:13:20,560 --> 01:13:25,159
I didn't want to have to solve, Like someone was asking ridiculous things on

1088
01:13:25,199 --> 01:13:29,680
how to fix Jenkins and the solution
was something I didn't want them to do,

1089
01:13:30,199 --> 01:13:34,399
and so I felt the need motivated
to just have this problem completely go

1090
01:13:34,479 --> 01:13:39,479
away. And that's when I would
really work on things like non stop wast

1091
01:13:39,479 --> 01:13:45,680
nerd sniping. Right, someone can't
be done see you Monday. Right here

1092
01:13:45,720 --> 01:13:50,319
it is, but I don't think. I think the other problem with that

1093
01:13:50,520 --> 01:13:54,359
is it has to be clear on
the team that that's happening. Like some

1094
01:13:54,439 --> 01:13:59,199
people just don't ever work weekends.
I think that's great, Like you know,

1095
01:13:59,720 --> 01:14:02,199
that's that should shouldn't be a problem. So it shouldn't be a peer

1096
01:14:02,199 --> 01:14:04,560
pressure of like, oh Pete work
of the weekend, everyone else should.

1097
01:14:04,560 --> 01:14:08,760
Like I think that is a terrible
message to send, so you have to

1098
01:14:08,760 --> 01:14:13,119
be careful with it, like I've
learned not to send too many emails.

1099
01:14:13,199 --> 01:14:15,600
Well, I don't. Luckily the
company I work at it's not an email

1100
01:14:15,640 --> 01:14:19,039
company, but I like big companies
where email is life. I'm always very

1101
01:14:19,039 --> 01:14:24,680
careful about setting emails on the weekend
because you're implicitly setting expectations for other people

1102
01:14:24,760 --> 01:14:29,399
that are watching, and I feel
like that is trouble. Yeah, I

1103
01:14:29,439 --> 01:14:33,680
mean that's another avenue realistically, Like, I think there is a thing about

1104
01:14:34,000 --> 01:14:36,640
just like you want the load on
your systems to be constant. You don't

1105
01:14:36,640 --> 01:14:41,159
want to see spikes because they're incredibly
hard to deal with the same goes for

1106
01:14:41,680 --> 01:14:46,159
teams that are putting out work.
Right, If some engineers are incredibly spiky

1107
01:14:46,720 --> 01:14:50,399
on load, then you're unpredictable and
what you can deliver and how much,

1108
01:14:50,439 --> 01:14:55,239
and the reliability or quality of that. So it's you're not necessarily doing anyone

1109
01:14:55,279 --> 01:15:00,840
a favor by one day going out
and solving a problem. If that's your

1110
01:15:00,880 --> 01:15:02,560
pattern. Yeah, I think it's
a difficult lesson for a lot of people

1111
01:15:02,600 --> 01:15:08,359
to learn. I struggle with it
a lot. I often find myself like

1112
01:15:08,439 --> 01:15:11,199
I'll work a weekend because I want
to, and then Monday, I'm like,

1113
01:15:11,239 --> 01:15:15,079
oh, man, I don't want
to work today, Like, yeah,

1114
01:15:15,159 --> 01:15:16,520
well that's fine, right, because
then you're then you're still having the

1115
01:15:16,520 --> 01:15:20,159
same amount of work that you're sort
of putting out and productivity for the team,

1116
01:15:20,159 --> 01:15:24,159
but you're not overburning them because someone
will has to review that, right,

1117
01:15:24,199 --> 01:15:27,640
someone that's still creating followup work down
the road, and depending on the

1118
01:15:27,640 --> 01:15:30,920
statement, maybe that creates incidents as
well. Yeah, now, you're right,

1119
01:15:31,000 --> 01:15:35,359
it's a very tough balance to figure
out that. Yeah, I'm still

1120
01:15:35,399 --> 01:15:39,760
trying to figure out what that is. I would say my whole career is

1121
01:15:39,760 --> 01:15:44,600
pretty spiky over There are some places
I work at where I I'm always you

1122
01:15:44,600 --> 01:15:45,439
know, I can't help the work
aholic at me. But there are some

1123
01:15:45,439 --> 01:15:48,199
places where I chill a little more
in some places where I'm like eighty hour

1124
01:15:48,279 --> 01:15:51,479
weeks insanity, And I don't know. For me, I've learned it's a

1125
01:15:51,479 --> 01:15:56,479
healthy cycle. Like every couple jobs, I do the insane grind because that's

1126
01:15:56,520 --> 01:15:59,359
just where I'm at, and then
I have a couple of years of like

1127
01:15:59,680 --> 01:16:02,640
less insane grind, kind of relax
and find other things to do, and

1128
01:16:02,680 --> 01:16:05,000
then bright back to it because I
miss it, you know. I mean,

1129
01:16:05,279 --> 01:16:08,560
it doesn't matter what your pattern is. I mean, if you are

1130
01:16:08,600 --> 01:16:11,279
spikey, it's fine as long as
it's somehow consistent. Right. You know,

1131
01:16:11,399 --> 01:16:14,399
if every couple of weekend, you
know, every other weekend, you

1132
01:16:14,439 --> 01:16:17,000
do extra work, then you sort
of expect that into the realm of things

1133
01:16:17,000 --> 01:16:20,600
and how that's going to play out. But if it happens and it's unpredictable,

1134
01:16:20,680 --> 01:16:24,600
then you don't know what the impact
is on the team. You may

1135
01:16:24,680 --> 01:16:28,720
think that the team can have more
work done than is reasonable, and so

1136
01:16:28,760 --> 01:16:32,560
a new big project comes out and
now it's taking even longer or unexpected because

1137
01:16:32,640 --> 01:16:40,279
he's not pulling those weekends anymore and
doing the a real job that has a

1138
01:16:40,279 --> 01:16:45,279
lot of value, but messed up
some sort of prediction or timelines or deadlines.

1139
01:16:45,880 --> 01:16:48,399
I think engineering time prediction is like
the hard I just throw that out

1140
01:16:48,439 --> 01:16:55,640
the window, especially with infra.
Right, well, it's two weeks of

1141
01:16:55,640 --> 01:16:59,479
infra work. Oh so it'll be
done in two weeks, maybe like two

1142
01:16:59,520 --> 01:17:02,720
weeks, And for work sometimes takes
two months with outages and interrupts, and

1143
01:17:03,279 --> 01:17:08,279
it's hard to explain sometimes. Yeah, it's like Einstein's theory of relativity.

1144
01:17:08,399 --> 01:17:14,479
This is real the fast where we
go this is different where we go no.

1145
01:17:14,600 --> 01:17:16,479
But I think going back to your
you're talking about working in spikes.

1146
01:17:16,520 --> 01:17:24,640
I think that's like, that's how
we've worked as humans for you know,

1147
01:17:24,960 --> 01:17:28,640
forty thousand years. Like you go
out and you do the big grind to

1148
01:17:30,199 --> 01:17:31,880
you know, to to hunt the
animals, and then you go back and

1149
01:17:31,920 --> 01:17:35,399
you just your rest and you relax
for a while. Or you go out

1150
01:17:35,439 --> 01:17:40,399
and you work all summer to plant
the crops and harvest the crops and then

1151
01:17:40,439 --> 01:17:42,800
store them for the winter, and
then you ride the winter out. So

1152
01:17:42,840 --> 01:17:49,720
I think that behavior is actually something
that has been native to us for a

1153
01:17:49,800 --> 01:17:54,279
long long time, and to try
to break that in the course of a

1154
01:17:54,399 --> 01:17:59,920
three decade career is going to be
difficult. I mean you're on as a

1155
01:18:00,119 --> 01:18:02,840
there definitely what are you called it? Art? Right? So you look

1156
01:18:02,880 --> 01:18:08,039
at the Renaissance artists, famous ones
like you know, see what they did?

1157
01:18:08,079 --> 01:18:11,560
What are other artists doing before?
And even today? You know,

1158
01:18:11,560 --> 01:18:14,319
how are they working? Because that's
the same of expectation you can have for

1159
01:18:14,359 --> 01:18:18,079
any knowledge work, which is very
similar to a creative process. You have

1160
01:18:18,159 --> 01:18:21,640
to have the right motivation and sometimes
that's really hard to figure out. I

1161
01:18:21,640 --> 01:18:26,000
don't know what that is sometimes,
right, some weekends, I'm just I

1162
01:18:26,000 --> 01:18:29,640
want to be at and stare at
a screen. And some weeks I wake

1163
01:18:29,720 --> 01:18:30,960
up early on a Saturday and I'm
like, let's write some code, let's

1164
01:18:30,960 --> 01:18:35,159
do stuff, And yeah, I
have no idea what what drives it?

1165
01:18:36,000 --> 01:18:39,359
Sometimes I know, but often it's
just I don't know. It's hard to

1166
01:18:39,359 --> 01:18:43,920
say how I'm feeling or will feel
until I actually wake up. And that's

1167
01:18:44,079 --> 01:18:48,880
what to do for sure. I
think if we figure that out and can

1168
01:18:49,159 --> 01:18:54,279
define it and reproduce it, we've
got our next multi billion dollar startup.

1169
01:18:55,039 --> 01:19:01,640
Seriously, awesome. Is there anything
else we should talk about for internal platforms

1170
01:19:01,960 --> 01:19:06,159
of infrastructure? I think we covered
a lot. I think that Yeah,

1171
01:19:06,199 --> 01:19:11,199
all the things I want to talk
about awesome. Cool. Let's do some

1172
01:19:11,279 --> 01:19:15,640
picks. Warren have been picking on
you for picks the last couple of episodes,

1173
01:19:15,760 --> 01:19:20,039
but I gave Pete the heads up
before we started recording, so I

1174
01:19:20,119 --> 01:19:23,560
know he's preps. So I'm gonna
put Pete on the spot. What'd you

1175
01:19:23,560 --> 01:19:29,399
bring for us? Pete? I'm
a gamer, and I like, uh,

1176
01:19:29,880 --> 01:19:32,279
I don't know. I like games
that are really hard. I like

1177
01:19:32,319 --> 01:19:35,439
games that are kind of like a
second job, and grindy and I don't

1178
01:19:35,439 --> 01:19:39,560
know, I'm go utton for punishment. I guess this is the summary.

1179
01:19:40,279 --> 01:19:43,840
Some people play games just to have
casual. I have some casual you know,

1180
01:19:44,000 --> 01:19:45,840
hang out with the boys and play
some games kind of thing. But

1181
01:19:45,880 --> 01:19:48,880
I like spreadsheet on the second monitor
and gaming on the first monitor kind of

1182
01:19:48,880 --> 01:19:53,479
thing. So I really like aarpg's
you know, hack and slash anything where

1183
01:19:53,479 --> 01:19:58,119
you can mind max. It's just
interesting. Up my alley. A new

1184
01:19:58,159 --> 01:20:00,560
game came out. I don't know. I think I went full release ten

1185
01:20:00,640 --> 01:20:04,479
days ago. It's been around for
years, and like betas and Alpha,

1186
01:20:04,479 --> 01:20:10,359
it's called last Epoch. It's like
a you know, Diablo ish ARPG thing.

1187
01:20:11,399 --> 01:20:14,760
And there's Diablo. I played a
lot of Diablo. There's Poe,

1188
01:20:14,840 --> 01:20:17,720
which is like the ultimate. Like
I don't have enough hours to I'm also

1189
01:20:17,840 --> 01:20:20,520
like a very addicted personality, so
I have to choose my gamescre I don't

1190
01:20:20,520 --> 01:20:25,399
play a factorio because I know that
I would just stop working. Like Last

1191
01:20:25,399 --> 01:20:30,039
Epoch is this nice middle of It's
a lot more complex than Diablo. There's

1192
01:20:30,039 --> 01:20:32,680
a lot more things that can contribute
to your final build that you can nerd

1193
01:20:32,720 --> 01:20:36,600
out on trying this and trying that
there's a lot of RNG, so there's

1194
01:20:36,640 --> 01:20:43,039
the grind aspect of it. It
just checks all the boxes. And for

1195
01:20:43,159 --> 01:20:45,399
me, I know a game is
good when I lose track of time constantly.

1196
01:20:45,640 --> 01:20:49,079
Oh it's a very am I should
go to bed now. That's been

1197
01:20:49,199 --> 01:20:53,680
like the last week and a half
as the game came out, And so

1198
01:20:53,800 --> 01:20:57,960
that is my pick. If you
like ARPGs or min maxine games or things

1199
01:20:58,000 --> 01:21:01,399
like that, this game just checks
all the boxes for me. Written by

1200
01:21:01,520 --> 01:21:04,800
gamers. Sometimes you play a game
when it's pretty clear that the people that

1201
01:21:04,840 --> 01:21:09,279
wrote it don't actually play the game
they've written. It's actually way more common

1202
01:21:09,520 --> 01:21:13,239
than you think. And you realize
because there's no quality of life features and

1203
01:21:13,239 --> 01:21:15,640
it's like, well this is awkward
and I have to do it every two

1204
01:21:15,720 --> 01:21:18,079
raids and this sucks this game just
like oh I have to do this thing.

1205
01:21:18,119 --> 01:21:20,800
Oh wait, this is really easy
to do, Like okay, just

1206
01:21:20,960 --> 01:21:26,399
like someone plays the game wrote the
game. So yeah, I enjoy a

1207
01:21:26,439 --> 01:21:30,560
game like that. It's a smaller
the company. So is there an online

1208
01:21:30,600 --> 01:21:34,680
multiplayer mode for it? Yeah?
So it's like a whole way do you

1209
01:21:34,680 --> 01:21:38,319
go to the common areas there's a
bunch of other people there, and there's

1210
01:21:38,399 --> 01:21:42,159
trading and stuff, and there's you
know, public chat where half trolling,

1211
01:21:42,239 --> 01:21:46,399
half people asking questions. But you
can party up with your buddies and tackle

1212
01:21:46,479 --> 01:21:49,319
content together. And so I have
a small group I play with and we've

1213
01:21:49,359 --> 01:21:54,079
always played ARPGs, so this is
our current one that we're all just kind

1214
01:21:54,119 --> 01:21:57,640
of grinding up characters on and figuring
out what the best builds are and what

1215
01:21:57,720 --> 01:22:00,520
synergizes with each other and things like
that. All right, So the follow

1216
01:22:00,600 --> 01:22:03,760
up question is do you want to
share your gamer tag so that you're listening

1217
01:22:03,800 --> 01:22:09,760
for the show can jump on and
talk trash. Yeah, I'm pdf backwards

1218
01:22:09,760 --> 01:22:15,479
the TEP all right, someone out
there has p F places. So I'm

1219
01:22:15,479 --> 01:22:18,279
a yeah, you said difficulty.
I thought for sure you were going to

1220
01:22:18,319 --> 01:22:23,960
bring up Lost Souls or Dark Souls, and so I that kind of game

1221
01:22:24,600 --> 01:22:28,800
I if I was good at I
would play. I don't think anyone's good

1222
01:22:28,800 --> 01:22:31,159
at it the whole, like the
the you know those kind of games where

1223
01:22:31,159 --> 01:22:33,880
it's like jumping mechanics and stuff.
I don't know, I'm that's stuff.

1224
01:22:33,920 --> 01:22:38,079
He's not a platformer. That that's
it? Yeah, I mean really twitchy

1225
01:22:38,159 --> 01:22:41,520
games. Yeah, I'm totally with
you. I played Ninja Guide in a

1226
01:22:41,560 --> 01:22:44,600
lot in the past, and like
that was very there are some parts that

1227
01:22:44,600 --> 01:22:48,279
were incredibly challenging. Blessed Podcast.
Enough of that, Like some of the

1228
01:22:48,279 --> 01:22:51,760
boss fights are legitimately challenging. You
know, we just one dude we bought

1229
01:22:51,880 --> 01:22:55,239
probably forty times in a before we
beat him, but now were really good

1230
01:22:55,239 --> 01:23:00,840
at it because we yeah, we
post bared them each death. Okay,

1231
01:23:01,600 --> 01:23:04,960
you can't stand in the middle if
really where games are worth right, Like

1232
01:23:05,000 --> 01:23:09,600
it's I think that's a real lesson. You know, you really have to

1233
01:23:09,640 --> 01:23:14,239
take root cause analysis and post mortems
to your your private life, and you

1234
01:23:14,279 --> 01:23:17,039
know in your friend group when something
goes wrong, you really need to investigate.

1235
01:23:17,399 --> 01:23:20,000
I often joke that, like many
things about me are really good for

1236
01:23:20,039 --> 01:23:24,479
work and really terrible for personal relationships, but they work out really well in

1237
01:23:24,520 --> 01:23:30,600
gaming. So we're causing relationship problem
is usually a bad idea. We're causing

1238
01:23:30,600 --> 01:23:38,039
why you died to a boss.
Apply it where you can fair enough,

1239
01:23:39,680 --> 01:23:42,920
all right, Warren, would you
bring this week? Yeah, so a

1240
01:23:42,960 --> 01:23:45,479
couple weeks ago I mentioned this already, but I'm gonna plug it again.

1241
01:23:45,880 --> 01:23:51,359
On Friday, there's a decompiled conference
in dressed in Germany, UH which I'm

1242
01:23:51,359 --> 01:23:56,239
actually giving a talk on about our
journey at authors and adding security. But

1243
01:23:56,279 --> 01:23:58,760
there are some interesting talks there,
like there's one that I really want to

1244
01:23:58,800 --> 01:24:03,159
go to that's about migrating from kubernetties
to server lusts, and I think there

1245
01:24:03,239 --> 01:24:06,039
are a bunch of other ones that
are really interesting. I'm looking forward to

1246
01:24:06,560 --> 01:24:13,600
nice, excellent, cool. So
my pick is going to be no surprise

1247
01:24:13,680 --> 01:24:16,800
to anyone who's been listening to the
last few episodes. I am picking platform

1248
01:24:16,880 --> 01:24:23,439
Con coming out in June. It's
a five day virtual conference about platform engineering,

1249
01:24:24,199 --> 01:24:27,720
so check it out totally free,
and there's going to be tons of

1250
01:24:27,760 --> 01:24:32,279
great talks there and specific to me. At the end of the conference,

1251
01:24:32,319 --> 01:24:36,239
I will be doing a live Q
and A session with some of the speakers,

1252
01:24:36,760 --> 01:24:40,720
so they're finalizing who the speakers are
going to be, and then once

1253
01:24:40,800 --> 01:24:44,880
that's done, I am going to
try and turn this into a Q and

1254
01:24:44,920 --> 01:24:47,319
a session that you actually want to
listen to. So I'm going to go

1255
01:24:47,359 --> 01:24:53,800
out on X and start asking people
which speakers do you want me to interview

1256
01:24:54,199 --> 01:24:58,279
and what questions do you want me
to ask them, so that it becomes

1257
01:24:58,279 --> 01:25:00,800
the interview that you want to hear, and yes, I know that by

1258
01:25:00,840 --> 01:25:04,840
going out on X and asking you
what questions to ask, some of you

1259
01:25:04,880 --> 01:25:08,640
are going to ask some of the
wrong questions. So I'm just gonna be

1260
01:25:08,680 --> 01:25:12,760
honest here with you. I'm not
going to ask that, but thank you

1261
01:25:12,760 --> 01:25:17,640
for listening to the show, you
sick little pervert. And so whenever the

1262
01:25:17,760 --> 01:25:23,119
Q and A session comes up,
I will ask the questions, some of

1263
01:25:23,159 --> 01:25:26,600
them because we know what you're going
want to ask, but I will ask

1264
01:25:26,600 --> 01:25:29,800
some of them and then you will
be able to say, hey, I

1265
01:25:29,880 --> 01:25:32,760
heard about this on the X,
which takes it full circle because I did

1266
01:25:32,840 --> 01:25:36,439
go through that whole setup just to
walk all the way through and plug his

1267
01:25:36,560 --> 01:25:43,039
zz Top song in my pick for
today. So all of you listeners who

1268
01:25:43,039 --> 01:25:45,520
are Zezytop fans out there, that
was all done for you. Thanks for

1269
01:25:45,600 --> 01:25:56,319
listening. Cool, So I think
we got an episode here. Awesome.

1270
01:25:56,840 --> 01:26:00,159
Yeah, Pete, thanks for joining
us. This has been a great talks.

1271
01:26:00,159 --> 01:26:01,720
Fun to have you on the show. Yeah, thank you for having

1272
01:26:01,760 --> 01:26:05,399
me again. This was great.
Yeah, anytime. Warren, thanks again

1273
01:26:05,479 --> 01:26:09,239
for joining me as a his co
host here. Of course, I love

1274
01:26:09,279 --> 01:26:12,239
having you on the show. It's
been a lot of fun. Look forward

1275
01:26:12,239 --> 01:26:15,560
to seeing you next week. Yeah, and for all you listeners out there,

1276
01:26:15,640 --> 01:26:18,359
and we will see you all next
week too. Thanks everyone,