1
00:00:05,120 --> 00:00:08,960
Okay, what's going on? Everybody? Welcome to another episode of Adventures in

2
00:00:09,240 --> 00:00:14,240
DevOps. I'm your host, Will
Button and back after attending it was the

3
00:00:14,320 --> 00:00:18,920
DevOps conference in Zurich last week,
right Warren? Yeah for sure. And

4
00:00:19,039 --> 00:00:23,280
you know, I actually listened to
a little bit of last episode and I

5
00:00:23,320 --> 00:00:26,640
felt like you introduced me even though
I wasn't there. So I feel like

6
00:00:26,679 --> 00:00:30,280
I have a free you know,
I don't need to introduce myself this episode.

7
00:00:32,759 --> 00:00:37,679
I will say that I owe everyone
an interesting fact. And this time

8
00:00:37,719 --> 00:00:44,359
when we were at the conference,
we specifically went out and hosted an open

9
00:00:44,359 --> 00:00:50,000
space where I talked with a group
of experienced develops practitioners and we were really

10
00:00:50,039 --> 00:00:53,039
trying to define what a platform even
is. I think this word is thrown

11
00:00:53,079 --> 00:00:58,200
away around everywhere, and after an
hour we really came to a great conclusion.

12
00:00:58,719 --> 00:01:06,879
It's something horrioral and that's I think
there's an aspect here where it doesn't

13
00:01:06,920 --> 00:01:10,000
all doesn't offer value to the end
user. I think is an important thing.

14
00:01:10,079 --> 00:01:15,719
So one definition we had was everything
other than the product, which is

15
00:01:15,799 --> 00:01:19,959
interesting to think that the platforms we
build aren't aren't value add but I,

16
00:01:21,000 --> 00:01:23,200
you know, I there's some interesting
and deep dive there. But yeah,

17
00:01:23,319 --> 00:01:26,560
no, it was great. It
was great. Confidence. I'm back.

18
00:01:26,640 --> 00:01:29,560
I'm happy to be back though,
right, I'm cool. Happy to have

19
00:01:29,640 --> 00:01:34,560
you back. And then also joining
us in the studio the co founder and

20
00:01:34,719 --> 00:01:40,239
CEO of Kurtosis, Galen Marquetti.
Galen, welcome, thank you, thank

21
00:01:40,280 --> 00:01:42,280
you for having me, and hello
everybody. Yeah, I'm excited to have

22
00:01:42,359 --> 00:01:45,680
you here. You know, we've
talked a little bit in the past,

23
00:01:45,799 --> 00:01:51,519
and then we've also I've also been
using critosis a little bit and it's very

24
00:01:51,560 --> 00:01:57,239
basic level. But tell us just
give us like the high level overview of

25
00:01:57,439 --> 00:02:02,079
Kurtosis, and then we're going to
dig into what led us to this point

26
00:02:02,120 --> 00:02:07,479
in life. So the highest level
overview is that Krotosis makes it easy to

27
00:02:07,560 --> 00:02:15,520
get development environments and testing environments set
up for back end systems. So I

28
00:02:15,680 --> 00:02:21,800
think something like doctor composed or helm
charts with more parameterizability, more flexibility,

29
00:02:22,120 --> 00:02:27,879
and some things that give you greater
portability guarantees. Gotcha. So when we're

30
00:02:27,879 --> 00:02:32,039
talking about dev environments, as a
developer working on a product, I need

31
00:02:32,080 --> 00:02:37,520
to test the code that I'm writing, but that means I need to run

32
00:02:37,599 --> 00:02:40,919
the application. But then I also
need all the supporting micro services, and

33
00:02:40,960 --> 00:02:46,479
then those are completely worthless without the
data that gives me some kind of real

34
00:02:46,599 --> 00:02:52,800
world or pseudo real world scenario to
test against. Is that right? That's

35
00:02:52,800 --> 00:02:54,759
about right, And I think a
good way to think about the pain that

36
00:02:54,840 --> 00:03:00,639
kind of motivated the whole thing is
that part about running other people's code.

37
00:03:00,719 --> 00:03:04,520
We're running your own code. Usually
people have a handle on that one.

38
00:03:04,680 --> 00:03:09,120
It's when you need six or seven
other micro services plus the data. Now

39
00:03:09,159 --> 00:03:13,080
you have to dig into how do
I set these things up? And a

40
00:03:13,120 --> 00:03:16,879
lot of the time that I get
stuck at that phase for an in order

41
00:03:17,080 --> 00:03:20,879
amount of time, and that's kind
of the heart of what we're doing.

42
00:03:21,919 --> 00:03:24,960
Yeah, for sure. I can
think back of different scenarios where I've addressed

43
00:03:24,960 --> 00:03:30,400
this in the past, and some
of the approaches I've used are the you

44
00:03:30,400 --> 00:03:38,000
know, like the seven thousand line
Bash script. There's the golden machine or

45
00:03:38,039 --> 00:03:42,439
the golden VM that no one's allowed
to touch because it has everything on it.

46
00:03:42,560 --> 00:03:47,360
Right at the moment, there's trying
to bribe your security team to let

47
00:03:47,439 --> 00:03:53,240
you connect to the staging environment across
the firewall. So those are some of

48
00:03:53,240 --> 00:03:58,599
the approaches I've used in the past. Yeah, it's exactly right. It's

49
00:03:58,639 --> 00:04:02,039
exactly right, and it's what a
lot of the times people get a good

50
00:04:02,080 --> 00:04:05,639
system set up for kind of a
golden path they do all the time,

51
00:04:05,919 --> 00:04:11,360
but then there's this like long tail
of things that you always do a long

52
00:04:11,439 --> 00:04:14,319
tail thing maybe once a week,
but it's a different long tail thing,

53
00:04:14,479 --> 00:04:18,360
and it's that set of issues and
security and you have to touch the golden

54
00:04:18,360 --> 00:04:24,199
and VM for it, it break
it, and it just adds up for

55
00:04:24,240 --> 00:04:30,120
sure. Orren, what's what's your
approach in solving this problem? Oh,

56
00:04:30,160 --> 00:04:32,120
well, you know, I think
we only have like an hour on this.

57
00:04:34,959 --> 00:04:43,319
There's something here where if you have
really high separation, low coupling between

58
00:04:43,360 --> 00:04:46,160
your individual micro services, you can
just spin up the one that you've got

59
00:04:46,199 --> 00:04:51,279
on your machine. We use local
stack for our infrastructure, but I think

60
00:04:51,319 --> 00:04:57,959
that's a very extreme perspective and it
takes a while to really get there,

61
00:04:57,959 --> 00:05:00,160
and we focus on having that at
the beginning. And if you come from

62
00:05:00,199 --> 00:05:03,959
a monolith world or one where the
architecture wasn't really figured out, there's this

63
00:05:04,160 --> 00:05:10,000
sort of dead zone in the middle
where your services are still unfortunately coupled in

64
00:05:10,040 --> 00:05:14,439
some way and you really can't get
the value out of them without having some

65
00:05:14,560 --> 00:05:19,040
of the ancillary pieces also spun up
and working at the same time. Right

66
00:05:19,079 --> 00:05:27,759
on for sure. So Galen,
obviously there was a set of circumstances in

67
00:05:27,800 --> 00:05:30,800
your life that led you to say, this is a real problem and I'm

68
00:05:30,839 --> 00:05:34,319
going to I'm going to give it
my best shot at starting. And it

69
00:05:34,480 --> 00:05:39,120
led to a company. What what
kind of scenarios were you working in that

70
00:05:39,240 --> 00:05:43,959
made you realize this is what you
want to do? Yeah, I mean

71
00:05:43,959 --> 00:05:51,920
there's a couple. The earliest instance
of this innate pain it came at my

72
00:05:53,000 --> 00:05:56,240
senior year of college. My senior
year of college, I was doing a

73
00:05:56,279 --> 00:06:02,600
research project on distributed databases. And
this was right before Ethereum launch, which

74
00:06:02,680 --> 00:06:09,360
is a cryptocurrency, and I was
doing a research project to explore different consensus

75
00:06:09,439 --> 00:06:15,079
mechanisms, so like low level protocol
comms between notes. I wanted to spin

76
00:06:15,240 --> 00:06:23,319
up three nodes of this network just
to experiment with my different consensus mechanism.

77
00:06:24,240 --> 00:06:29,199
Nearly impossible to do that at the
time, and a lot of my project

78
00:06:29,279 --> 00:06:32,279
was oriented around simply getting the things
spun up in the first place. Of

79
00:06:32,319 --> 00:06:35,959
course, the code worked on the
developers' laptops that worked on the DevOps.

80
00:06:36,000 --> 00:06:41,800
Laptops didn't work on my laptop.
And it wasn't about it wasn't about building

81
00:06:41,920 --> 00:06:46,439
a particular note. It was about
getting the cluster of the system set up

82
00:06:46,480 --> 00:06:51,639
correctly where there's really no good abstraction
that gets you that very consistently. You

83
00:06:51,680 --> 00:06:58,160
can get it in certain architectures in
certain ways decently consistently. But that was

84
00:06:58,199 --> 00:07:00,639
the first instance. It was very
fresh trading, like why won't this thing

85
00:07:00,800 --> 00:07:05,240
just spin up? It's the simplest
application, just run it as it is.

86
00:07:08,759 --> 00:07:11,399
Yeah, that was the very first
one. I saw it later in

87
00:07:11,439 --> 00:07:17,360
my career as well as working at
a company that had had a internal tool

88
00:07:17,439 --> 00:07:21,360
that spun up a full micro service
deck you can think maybe two hundred micro

89
00:07:21,439 --> 00:07:28,399
services, and it was used for
a variety of purposes between dev demoing uat

90
00:07:28,680 --> 00:07:32,639
so like a high level just testing
user workflows to see how they would work.

91
00:07:33,519 --> 00:07:38,160
That took a lot of effort to
get that system spun up, and

92
00:07:38,199 --> 00:07:41,160
at the essence of it, it's
the same issue. It's I just want

93
00:07:41,160 --> 00:07:46,439
to run the system and a configuration
that makes sense for the work while I'm

94
00:07:46,480 --> 00:07:53,079
doing and it can be remarkably hard
to do that. So this these sets

95
00:07:51,839 --> 00:07:57,519
of issues. It's kind of the
feeling of like, this shouldn't be this

96
00:07:57,639 --> 00:08:01,519
way. It should be decently easy
to just run software. It should be

97
00:08:01,560 --> 00:08:07,399
hard to like build software that's that's
that's amazing, but it should be easy

98
00:08:07,480 --> 00:08:09,319
to run it. It's kind of
the feeling of the way the world should

99
00:08:09,319 --> 00:08:16,800
be that kind of motivated the entire
start into pretosis, right, And yeah,

100
00:08:16,800 --> 00:08:22,800
and I think there's I think maybe
just part of that is we build

101
00:08:22,839 --> 00:08:26,839
these complex systems, but for most
of us, once we build that,

102
00:08:26,920 --> 00:08:33,240
like once we build our production environment, the people who really have the skills

103
00:08:33,279 --> 00:08:37,240
to build those, they don't build
a second production. The part where this

104
00:08:37,279 --> 00:08:41,440
pain has really felt is in the
development teams where they're trying to you know,

105
00:08:41,519 --> 00:08:46,200
build and tear these down to test
different features and add new features to

106
00:08:46,240 --> 00:08:52,159
it. Yes, and that's one
of the main themes that I've learned while

107
00:08:52,320 --> 00:08:58,080
going through this journey is it's a
lot of the times these development environments they're

108
00:08:58,120 --> 00:09:03,279
about getting us close for production as
possible but safe and you can do the

109
00:09:03,279 --> 00:09:07,720
workfload you need to do. But
the main value access is how close can

110
00:09:07,759 --> 00:09:11,080
you get to production? So,
like Warren mentioned, local stack, you

111
00:09:11,120 --> 00:09:16,759
can run a tows services on your
laptop. It's one approach to it,

112
00:09:16,919 --> 00:09:24,039
bring production locally or bring production to
your development environment. The other approach is

113
00:09:24,080 --> 00:09:26,799
the opposite. You put your development
environment into the cloud, where you then

114
00:09:28,480 --> 00:09:33,840
basically fork everything that you have in
production and you have your cloud services are

115
00:09:33,919 --> 00:09:37,960
sitting in the exact same environment where
your development code is another approach to it.

116
00:09:37,519 --> 00:09:43,440
But basically, everyone who's putting a
lot of investment into getting these great

117
00:09:43,480 --> 00:09:46,919
development environments does one of those two
things, or some combination of both,

118
00:09:48,440 --> 00:09:54,399
bringing that bringing production locally or putting
your local into production in the sense of

119
00:09:54,480 --> 00:10:00,639
into your cloud environment. Right right, what's the barrier to intrig for using

120
00:10:00,679 --> 00:10:07,960
something like critosis, Yeah, right
now. The biggest barrier to entry is

121
00:10:09,399 --> 00:10:15,600
defining the setup logic for your environment. So we have a definition language.

122
00:10:15,600 --> 00:10:18,799
We use Starlarc, which is the
language. It's a Python dialect that Basil

123
00:10:18,919 --> 00:10:24,240
uses to define the build rules.
We use it for similar similar properties that

124
00:10:24,240 --> 00:10:28,200
it has for the build system.
It's you can determine that the program is

125
00:10:28,200 --> 00:10:33,120
going to halt because it's not a
full computing language. But you need to

126
00:10:33,159 --> 00:10:39,279
define your set up logic in this
language. And once you do it once,

127
00:10:39,679 --> 00:10:43,039
everyone else can reuse it. But
what happens a lot of the time

128
00:10:43,200 --> 00:10:48,240
is who knows how to do that? You have to It's especially in an

129
00:10:48,320 --> 00:10:52,799
environment where there's a lot of different
open source tooling being used to varying degrees

130
00:10:52,799 --> 00:10:58,679
of maturity. The person who has
the expertise to understand how to spin it

131
00:10:58,759 --> 00:11:03,639
up in a different right situations,
that person might not be maintained that project

132
00:11:03,639 --> 00:11:09,919
anymore, even so track that down
or relearn it somehow, for sure.

133
00:11:11,039 --> 00:11:18,240
Yeah. And then I think there's
a related piece to that is not only

134
00:11:18,360 --> 00:11:22,799
building it, but then making sure
like reverse engineering, I guess, for

135
00:11:22,919 --> 00:11:26,399
a lack of a better term,
to make sure that what you've built for

136
00:11:26,440 --> 00:11:31,840
your local dev tool actually mimics what's
really going on out in production, right,

137
00:11:31,879 --> 00:11:35,720
because there's a lot of different ways
you can make all of these different

138
00:11:35,759 --> 00:11:37,519
components talk to each other, and
you've got to try to replicate that.

139
00:11:39,600 --> 00:11:45,120
Yes, yes, and that's that's
a lot of work, and it's that's

140
00:11:45,200 --> 00:11:50,360
one of the reasons why a lot
of people do opt to forego that whole

141
00:11:50,360 --> 00:11:54,720
process and just put their development code
directly into a production like cluster in the

142
00:11:54,759 --> 00:12:00,720
cloud as as possible. Because they're
like a localist, never going to be

143
00:12:01,200 --> 00:12:05,399
close enough to production. It's going
to take a lot of never ending effort

144
00:12:05,480 --> 00:12:09,159
to get it close. So let's
just get to it a stage in cluster

145
00:12:09,240 --> 00:12:11,279
as fast as possible, let things
soak there, and then go to abroad.

146
00:12:11,799 --> 00:12:18,720
It's another approach to it. There's
benefits and cons to each one,

147
00:12:20,480 --> 00:12:24,799
for sure. Yeah, what are
some of the big Well, first of

148
00:12:24,799 --> 00:12:31,360
all, how long have you been
building critosis? Yeah? We've been doing

149
00:12:31,399 --> 00:12:35,360
this full time since December of twenty
twenty, So what is that? Okay,

150
00:12:35,600 --> 00:12:41,120
three years and change? Yeah,
yeah, yeah, cool. So

151
00:12:41,159 --> 00:12:43,960
what are some of the how is
how is your thought process changed in those

152
00:12:45,120 --> 00:12:48,799
three plus years for building development environments? I mean, I'm sure you've learned

153
00:12:48,879 --> 00:12:54,720
lots of lessons and tried a lot
of things that didn't quite work. Absolutely,

154
00:12:54,960 --> 00:13:03,559
absolutely. The couple of things.
First, we started out, we

155
00:13:03,600 --> 00:13:11,000
started out our company selling a basically
binaries. So one thing is, how

156
00:13:11,039 --> 00:13:16,679
do developers and development teams want to
consume developer tooling? So we started out

157
00:13:16,720 --> 00:13:20,519
saying we're going to sell a compiled
binary that you get to run as a

158
00:13:20,559 --> 00:13:26,159
license fee, and their first couple
of clients we operated that way and It

159
00:13:26,360 --> 00:13:31,799
was fine in the very beginning.
We were solving a very important problem for

160
00:13:31,840 --> 00:13:37,000
those clients. It was acceptable at
the time, but we very quickly learned

161
00:13:37,039 --> 00:13:41,240
that this is it's not the way
that developers want to and for really good

162
00:13:41,320 --> 00:13:46,639
reasons, want to be interacting with
software. You do have to have the

163
00:13:46,679 --> 00:13:50,120
source code available, You do have
to have a free license, you have

164
00:13:50,200 --> 00:13:54,159
to have a free period of trying
it where they know, because they're integrating

165
00:13:54,360 --> 00:13:58,399
at a deep level with their tooling, if they're going to try it out,

166
00:13:58,399 --> 00:14:00,759
if they're going to invest into it, I do want to know it's

167
00:14:00,759 --> 00:14:03,759
not going to get yanked out from
under them, and that's totally fair.

168
00:14:03,080 --> 00:14:09,039
So we change our licensing entirely,
made the reboot open, and that change

169
00:14:09,200 --> 00:14:13,799
changed everything about the rate at which
we were learning, the rate of the

170
00:14:13,799 --> 00:14:16,000
exposure to different developers, people able
to try out to too long gate different

171
00:14:16,039 --> 00:14:20,320
perspectives. In retrospect, it was
pretty obvious. Maybe a lot of these

172
00:14:20,399 --> 00:14:24,840
learnings are pretty obvious in retrospect,
but you know, that was that was

173
00:14:24,879 --> 00:14:26,720
maybe the biggest change that we had
in the business, was opening a tool

174
00:14:26,799 --> 00:14:31,240
like that. No, I think
you're in good company. The past few

175
00:14:31,360 --> 00:14:35,120
weeks months I feel like there's been
a huge shift and focus on the podcast.

176
00:14:35,200 --> 00:14:39,320
It keeps coming out. Wills bringing
up this point about selling your idea

177
00:14:39,399 --> 00:14:43,759
within the devlof space as a product
management challenge, So really understanding what your

178
00:14:45,440 --> 00:14:50,679
teams need that are your consumers.
It's difficult but also really interesting and can

179
00:14:50,720 --> 00:14:56,240
help you build a great a great
product for them. Yeah, and it's

180
00:14:56,519 --> 00:15:01,039
developers are they're amazing users to work
with, but it's a very different dynamic.

181
00:15:01,200 --> 00:15:07,759
I think then products that maybe are
selling the sales folks or business focused

182
00:15:07,799 --> 00:15:13,320
folks, you're the way that products
are integrated and interactive with is quite different

183
00:15:13,200 --> 00:15:18,080
and as different dynamics. But once
you're solving a problem that the developers really

184
00:15:18,120 --> 00:15:24,759
really care about, you really do
develop really deep relationships for users, especially

185
00:15:24,080 --> 00:15:28,519
with like the community based approach with
like an open product. So it's it's

186
00:15:28,519 --> 00:15:33,279
been game changing having that kind of
a relationship working for us. Yeah.

187
00:15:33,759 --> 00:15:39,240
I think that's a really cool point
there, because you're talking about who your

188
00:15:39,360 --> 00:15:45,480
target market is, and so whenever
you think about like production deployments, you

189
00:15:45,519 --> 00:15:50,360
know, deploying my application to what
my target customers, that's a very different

190
00:15:50,440 --> 00:15:56,639
process than deploying my stack to the
local development team so they can build on

191
00:15:56,679 --> 00:16:00,840
it. It's a different, different
customer and they have different goals and objectives

192
00:16:00,840 --> 00:16:06,320
like the customers, the consumers of
my product want high availability, up time

193
00:16:06,480 --> 00:16:12,000
and fast response, whereas the developers
want more granular access and the ability to

194
00:16:12,200 --> 00:16:18,279
tweak and turn the knobs across the
environment to see how they can solve their

195
00:16:18,320 --> 00:16:22,519
problems. Is that a fair statement. Absolutely? Absolutely. In fact,

196
00:16:22,519 --> 00:16:30,519
there's even three different profiles to consider. There's the developers that are the end

197
00:16:30,600 --> 00:16:36,639
users of the tool. There's the
we call it maybe the platform team DevOps

198
00:16:36,639 --> 00:16:40,399
team in for a team, they're
really the ones that are responsible for the

199
00:16:40,440 --> 00:16:44,320
implementation and configuration of the tool.
A lot of the times they're the ones

200
00:16:44,360 --> 00:16:48,279
getting access to the GitHub repos they're
doing the high level configuration administration of the

201
00:16:48,320 --> 00:16:55,200
installation, and they're concerned about the
security posture and all that. And then

202
00:16:55,519 --> 00:17:00,519
ultimately you also do have like the
director or head of engineering, whoever approves

203
00:17:00,559 --> 00:17:04,720
the budget for whatever you're charging for
an enterprise version. So there's three.

204
00:17:06,160 --> 00:17:11,680
There's three different concerns to worry about, and you have to think about each

205
00:17:11,720 --> 00:17:18,000
one differently. It's it's like what
when you think about like a minimal valuable

206
00:17:18,440 --> 00:17:22,720
product, Like how how batting of
the concerns at each of the three parties

207
00:17:22,039 --> 00:17:26,000
you really have to do to like
get something off the ground. Uh yeah,

208
00:17:26,160 --> 00:17:30,119
so it's a multi dimensional problem.
No one envies a startup pounder,

209
00:17:30,119 --> 00:17:33,519
I mean they do. They think
you get to build your your dream,

210
00:17:33,559 --> 00:17:37,559
but then you deal with these quite
challenging situations where your buyers are not your

211
00:17:37,759 --> 00:17:41,319
users that you are you know,
really focused on. I have a question

212
00:17:41,400 --> 00:17:45,839
on that. Actually, I feel
like and I know I'm gonna put myself

213
00:17:45,880 --> 00:17:49,799
in some bucket somewhere. I assume
like everyone's using open TOFO today. I

214
00:17:49,839 --> 00:17:52,440
mean, I know that's not the
case, but you know, maybe there's

215
00:17:52,480 --> 00:17:56,680
there's a huge audience there. Especially
if we talk about people that are in

216
00:17:56,960 --> 00:18:02,359
the quote unquote develop space. If
they are familiar with this terminology, they

217
00:18:02,400 --> 00:18:07,680
probably are using some sort of infrastructure
as code solution. We use extensively CloudFormation

218
00:18:07,759 --> 00:18:12,960
ourselves. What is the how does
that interact with krotosis here? Like where

219
00:18:12,960 --> 00:18:19,279
the touch points? Like if we've
already defined our infrastructure in unfortunately HCl or

220
00:18:19,359 --> 00:18:25,880
worth in a programming language that doesn't
really well match to declarative nature, what

221
00:18:25,920 --> 00:18:30,680
do we do to spin up a
dev environment absolutely, so right now,

222
00:18:30,759 --> 00:18:37,400
krtosis operates after you apply your open
tofu or your terraform scripts to spin up

223
00:18:37,680 --> 00:18:42,400
the cloud services that you would need
for your development environment or your testing environment.

224
00:18:42,640 --> 00:18:48,599
You can think of it really as
a abstraction layer over Kubernetes and Docker.

225
00:18:49,359 --> 00:18:55,519
So if you have can use your
terraform scripts open tofu to spin up

226
00:18:55,519 --> 00:19:00,440
your Kubernetes clusters and provision everything required
there and then deploy onto the clusters with

227
00:19:00,559 --> 00:19:15,759
grotosis, I see gotcha, And
so then that so like that, that's

228
00:19:15,799 --> 00:19:22,839
like a way to leverage some of
the tools that you're using to deploy into

229
00:19:22,960 --> 00:19:26,599
production for building that so you're not
duplicating a lot of effort, right so

230
00:19:26,599 --> 00:19:32,000
you can still leverage the learnings from
terra farm or open to tofu or any

231
00:19:32,039 --> 00:19:37,559
other production grade tools that you're deploying
with exactly exactly. And then the cloud

232
00:19:37,680 --> 00:19:41,480
provisioning side of things is an area
that we are now exploring, and it's

233
00:19:41,559 --> 00:19:45,880
not an area that's covered in the
current like open tool and like the direct

234
00:19:45,920 --> 00:19:53,359
aspaciates of the open tool where it's
hey, can you also handle maybe four

235
00:19:53,440 --> 00:19:56,640
K my production environment, and its
like as look as you could fork an

236
00:19:56,720 --> 00:20:02,759
environment, can you handle clone everything
necessary there? Right now? That's a

237
00:20:02,759 --> 00:20:07,559
lot of what's happened in people's information
and terraform scripts is something like that we

238
00:20:07,680 --> 00:20:11,119
haven't That's an area that we're exploring
right now learning about. You know,

239
00:20:11,759 --> 00:20:18,079
what's important when you're when you're looking
at designing those those scripts. Yeah,

240
00:20:18,079 --> 00:20:22,400
for sure when it comes to documenting, like because that's a I think that's

241
00:20:22,400 --> 00:20:30,240
a would be an important artifact of
this process is we've discovered how our production

242
00:20:30,319 --> 00:20:34,680
infrastructure is working, reverse engineered it. How do you approach documenting that so

243
00:20:34,720 --> 00:20:41,599
that when the next development team comes
along, maybe the learning curve is a

244
00:20:41,599 --> 00:20:48,400
little bit uh, not so sharp
for them. Yeah, right now,

245
00:20:48,440 --> 00:20:52,759
the document I guess there's two ways
to think about it. One is one

246
00:20:52,839 --> 00:20:57,799
is how much can you write?
How much can you have your users write

247
00:20:57,880 --> 00:21:03,880
code that is some self documenting,
so like, can the environment definition be

248
00:21:03,079 --> 00:21:10,200
the documentation and as much as possible. I like having code that is extraordinarily

249
00:21:10,200 --> 00:21:15,359
readable and it seems self documenting,
and the way that our tool is set

250
00:21:15,440 --> 00:21:18,599
up, you tend to get that
in the code. It tends to be

251
00:21:18,720 --> 00:21:21,640
very composable, it tends to be
very readable. It tends to be very

252
00:21:22,000 --> 00:21:27,359
intuitive, So that's one function of
it. Another function of it is the

253
00:21:27,400 --> 00:21:33,920
functionality that we offer, so we
have high level environment parameterizability. What that

254
00:21:33,960 --> 00:21:41,079
means is when you write your environment
definition, you can expose very easily how

255
00:21:41,480 --> 00:21:45,599
to modify it in a sensible way. So you start it and you say,

256
00:21:45,599 --> 00:21:48,720
well, maybe I want to modify
it this way, this way,

257
00:21:48,759 --> 00:21:52,359
this way. We don't need documentation
for it because when you want it with

258
00:21:52,480 --> 00:21:55,519
that configuration, you're going to get
it out of the box. So you're

259
00:21:55,519 --> 00:22:00,359
trying to reduce the need for documentation
as much as possible with the design of

260
00:22:00,400 --> 00:22:03,000
the definition language. But you can
only dose it so much so that after

261
00:22:03,039 --> 00:22:07,440
that it's it's traditional documentation. It's
you got to rely on your users to

262
00:22:07,440 --> 00:22:12,200
write the readings, to write the
docs for those specific systems. That it

263
00:22:12,240 --> 00:22:17,279
reminds me of a quote that I
heard a long long time ago, back

264
00:22:17,319 --> 00:22:21,279
when everyone used to have wikis for
documentation, and I worked with this.

265
00:22:21,319 --> 00:22:26,039
One guy said, yeah, the
wiki is where the documentation goes to die.

266
00:22:27,480 --> 00:22:32,599
But I think like one of the
cool things I think that's happened is

267
00:22:32,599 --> 00:22:36,279
if you've ever used Jupiter notebooks.
I love the way that they let you

268
00:22:37,160 --> 00:22:40,680
mix, you know, different different
syntax in there, so you can have

269
00:22:40,720 --> 00:22:42,680
like a markdown block and then a
code block, and so you can kind

270
00:22:42,680 --> 00:22:51,839
of integrate the documentation alongside the code. And for I think for iterative or

271
00:22:51,880 --> 00:22:55,960
interactive type environments like that, it
works really well because it then lets you

272
00:22:56,039 --> 00:23:00,079
keep everything in one place, because
when you try to update multiple places,

273
00:23:00,880 --> 00:23:06,920
it just never happens in reality.
Yeah, and I love Jupiter Notebooks the

274
00:23:07,079 --> 00:23:11,039
experience, Like, it's one of
my favorite That access in the world is

275
00:23:11,759 --> 00:23:15,720
like spinning up and hacking something together
in a Jupiter notebook. And the dream

276
00:23:15,839 --> 00:23:18,519
is to feel that way about all
of our development work. I'm not sure

277
00:23:18,359 --> 00:23:23,680
realistic, but I love that experience. I mean, that's an interesting thread.

278
00:23:23,799 --> 00:23:29,680
What would what would the world have
to look like that regular development could

279
00:23:29,880 --> 00:23:33,759
follow this sort of path here?
Is it a matter of what we're working

280
00:23:33,799 --> 00:23:40,920
on or can we already sort of
start doing that today? Yeah? I

281
00:23:40,960 --> 00:23:44,200
mean what's great about it? Like, one thing that's great is how fast

282
00:23:44,319 --> 00:23:48,400
you see your changes work. It's
like extraordinarily fast. You write it,

283
00:23:48,519 --> 00:23:52,519
run run and run, run,
run, and you get the feed back

284
00:23:52,599 --> 00:23:56,319
right away. That's the build deploy
like how the build deploy loop. It's

285
00:23:56,359 --> 00:24:00,160
like, how fast does it take
for you to see on real stuff off

286
00:24:00,160 --> 00:24:03,400
what your things look like? Uh, usually in Jupiter notebooks you're using.

287
00:24:03,839 --> 00:24:06,960
I mean, like a lot a
lot of the fun use cases for it

288
00:24:07,000 --> 00:24:11,599
are running on relatively small data sets
so it fits in memory. So you

289
00:24:11,799 --> 00:24:17,759
do that fast feed toploy build a
play loop, and that's just tricky to

290
00:24:17,960 --> 00:24:21,720
get. Like you can get it
with like hot reloading into containers that are

291
00:24:21,759 --> 00:24:23,839
running in your in your environment.
You can get decently close to it,

292
00:24:23,880 --> 00:24:27,319
but you're still stuck at the level
of like what is this What is the

293
00:24:27,359 --> 00:24:33,599
smallest build increment that you can that
you can like define for your background system.

294
00:24:33,880 --> 00:24:37,480
I know a lot of people spend
a lot of effort getting their their

295
00:24:37,519 --> 00:24:42,640
basal build rules so well defined that
they're going to like minimize the build a

296
00:24:42,640 --> 00:24:45,000
play loop as much as possible,
But that takes a lot of effort.

297
00:24:45,039 --> 00:24:49,559
You really have to look into it
to get that experience. I mean,

298
00:24:49,559 --> 00:24:53,519
I'm not sure that there's a really
good way to get that now, No

299
00:24:53,599 --> 00:24:57,160
for sure, for sure not but
I think you know, tools, tools

300
00:24:57,160 --> 00:25:00,599
like yours you know definitely help reduce
the loop here, which is like the

301
00:25:00,640 --> 00:25:08,119
most important aspect because no matter how
much time you spend, really every second

302
00:25:08,200 --> 00:25:14,079
you cut off your loop allows that
feedback to happen faster, which improves the

303
00:25:14,200 --> 00:25:21,279
time. I think it's a hackneyed
response now that realistically people waiting for software

304
00:25:21,319 --> 00:25:23,960
development, is there what makes them
unhappy? Right? If you have to

305
00:25:25,039 --> 00:25:27,039
wait for the response? Will said, You know, he's all about the

306
00:25:27,039 --> 00:25:32,160
instant gratifications. Nothing better than you
know, hitting F five and having your

307
00:25:32,200 --> 00:25:37,480
program instantly work or tell you what
you did wrong, which is you know,

308
00:25:37,519 --> 00:25:42,119
I don't think it's just me.
I'm convinced that we are a TikTok

309
00:25:42,240 --> 00:25:45,279
generation. You know, whether you're
a TikTok user or not, we've all

310
00:25:45,319 --> 00:25:51,039
gotten addicted to short form content.
You know, if you're going to do

311
00:25:51,079 --> 00:25:56,559
anything that requires an attention span of
over sixty seconds, you really have to

312
00:25:56,759 --> 00:26:04,039
engineer for that. And I we've
been digging a lot into this this little

313
00:26:04,920 --> 00:26:10,880
that little feedback loop there and one
thing that did the dream world. There's

314
00:26:10,960 --> 00:26:14,519
kind of two really important things.
One is getting that build deplay loop super

315
00:26:14,640 --> 00:26:17,480
super fast, so you're coding the
change, accoding to the change. The

316
00:26:17,519 --> 00:26:22,319
other one is when you see the
change, how close is the change that

317
00:26:22,359 --> 00:26:25,839
you're seeing to what would happen in
production? And I can you get both

318
00:26:25,880 --> 00:26:27,880
of those things maximized, so like
you're getting a really really fast build to

319
00:26:27,880 --> 00:26:34,160
play loop, but also the deployment
is safely basically a production when you're doing

320
00:26:34,160 --> 00:26:40,559
it. That's kind of the dream
the dream world. It's it would take

321
00:26:40,559 --> 00:26:42,279
a lot to get there, I
think all the way, but you kind

322
00:26:42,279 --> 00:26:47,599
of like, roughly, what's happening. I think with the more and more

323
00:26:47,599 --> 00:26:52,839
sophisticated like dev infra along the software
development bicycle, roughly what's happening is you're

324
00:26:52,839 --> 00:26:56,799
getting like more and more optimized on
both of those axis to the point that

325
00:26:56,880 --> 00:27:00,720
eventually it kind of feels like you're
doing that like the limit of what's possible.

326
00:27:02,880 --> 00:27:07,000
I think there's a sort of a
conundrum here where the providers that are

327
00:27:07,039 --> 00:27:11,359
offering production infrastructure or platforms, the
amount of money that you're paying into them

328
00:27:11,440 --> 00:27:17,240
is so large or so vast for
the environment you're running for production that it

329
00:27:17,319 --> 00:27:21,720
never makes sense. It's not something
that is ever justified for them to make

330
00:27:22,119 --> 00:27:26,839
it faster to run or get the
loop closed because they don't care. It's

331
00:27:27,000 --> 00:27:32,759
you know one like how much is
your test staging, fake feral environment cost

332
00:27:33,319 --> 00:27:37,920
in actual dollars, euros and pesos
whatever compared to production It's like what one

333
00:27:37,960 --> 00:27:42,920
percent less? Even so like Amazon, you know, Google, GCP platforms

334
00:27:42,920 --> 00:27:45,200
of the world, they're not going
to make anything there, and so I

335
00:27:45,640 --> 00:27:52,079
feel like there it would be on
them to really change that because that's sort

336
00:27:52,119 --> 00:27:55,799
of like the canonical interface that you
have to abstract or make the same as

337
00:27:55,880 --> 00:28:02,359
much as possible. Yes, absolutely, and basically everyone's working against that limitation

338
00:28:02,839 --> 00:28:06,319
as far as I can tell.
It is like that's the that's the blocker,

339
00:28:07,000 --> 00:28:11,839
the only way to like you're you're
basically just trying to design more and

340
00:28:11,880 --> 00:28:17,680
more like innovative solutions to get around
exactly that obstacle. Yeah, because let's

341
00:28:17,720 --> 00:28:22,880
be just like completely transparent here.
The gold standard is them on the production

342
00:28:23,000 --> 00:28:27,880
server. Yes, you said that. I feel like that's where I started

343
00:28:27,920 --> 00:28:33,359
my career actually right for sure,
Like I was working at a company where

344
00:28:33,519 --> 00:28:38,920
you did software development by sshing into
another service somewhere and there was an environment

345
00:28:40,039 --> 00:28:45,079
running there which then you logged into
and you literally changed production objects by opening

346
00:28:45,119 --> 00:28:48,559
a text editor there. We didn't
it wasn't VIM that would have been slightly

347
00:28:48,599 --> 00:28:52,400
better than what was being used,
uh and uh, and then changing them

348
00:28:52,440 --> 00:28:57,880
and then you had to record the
changes you made in an overgrown internal system

349
00:28:59,160 --> 00:29:03,079
that would somehow get those changes applied
to actual production when there was a release.

350
00:29:03,519 --> 00:29:06,559
So, you know, I really
do feel like what was old will

351
00:29:06,599 --> 00:29:10,480
be new again, and I honestly
don't know if I want that world.

352
00:29:12,880 --> 00:29:18,519
My introduction to programming was doing something
very similar to that, and the guy

353
00:29:18,519 --> 00:29:19,319
who showed me how to do it, he's like, okay, when you

354
00:29:19,359 --> 00:29:23,400
get here, you want to run
the LS command and you could see all

355
00:29:23,440 --> 00:29:26,759
of the files there. But then
it was like, you know, for

356
00:29:26,920 --> 00:29:33,000
any given file, you know,
like just say like Maine dot c,

357
00:29:33,279 --> 00:29:36,200
there was Maine dot c dot one, Maine dot C dot two, Maine

358
00:29:36,200 --> 00:29:37,960
dot c dot three. Is like, so you want to look and see

359
00:29:37,000 --> 00:29:42,200
what the biggest number here is and
copy Maine dot c to the next number

360
00:29:42,279 --> 00:29:45,480
so that if you screw anything up, you've got a backup for it.

361
00:29:45,839 --> 00:29:51,160
And that was our version control system. Oh no, I'm just like,

362
00:29:51,359 --> 00:29:53,519
I think people figured it out date
based, right, You don't. None

363
00:29:53,559 --> 00:29:56,319
of these are the fancy tools.
All you need to do is append the

364
00:29:56,359 --> 00:30:00,880
timestamp of the last time you edited
the file there and then you'd be good.

365
00:30:03,319 --> 00:30:10,440
Yeah, I mean it's the VIM
on the production server. At least

366
00:30:10,440 --> 00:30:15,960
for me, this is like clearly
the best debt X Like I love it.

367
00:30:15,400 --> 00:30:19,440
I love it. It's just dangerous, it's super dangerous, and like

368
00:30:19,519 --> 00:30:23,480
staying up all night to think there
of an outage is miserable. So that

369
00:30:23,599 --> 00:30:30,839
experience is not great. But the
feeling of the VIM directly to production is

370
00:30:30,839 --> 00:30:36,039
is is the ideal, So like
to what degree can you mitigate risk to

371
00:30:36,119 --> 00:30:38,200
get as close to that as possible? I mean, you said it right,

372
00:30:38,240 --> 00:30:41,799
you know, things like kurtosis,
you know, shortening the gap there

373
00:30:41,839 --> 00:30:44,720
on time to make a change so
that rather than VIM on the production server,

374
00:30:44,759 --> 00:30:47,559
it's VIM on your machine or you
know, using one of those better

375
00:30:47,599 --> 00:30:52,240
tools that's out there, and then
making that change and just immediately that gets

376
00:30:52,279 --> 00:30:56,240
to a clone of production that is
almost exactly this, I mean exactly the

377
00:30:56,279 --> 00:31:00,319
same, and you can see that
change running as if you had actually applied

378
00:31:00,319 --> 00:31:06,160
it and then get all the benefits
without having actually the danger of making a

379
00:31:06,160 --> 00:31:10,160
production change. Yes, yes,
absolutely, that's that's that's the that's theer's

380
00:31:10,200 --> 00:31:15,480
the dream Yes, So what is
the learning curve for someone who's just they've

381
00:31:15,519 --> 00:31:21,160
they've done the seven thousand line bash
script and they're looking for something different and

382
00:31:21,480 --> 00:31:25,200
wanting to check out critosis. What's
the learning curve or the amount of time

383
00:31:25,200 --> 00:31:30,519
it takes to be productive with this
so as framed very low. And the

384
00:31:30,599 --> 00:31:37,799
reason for that is because that person
did the seven thousand best script part.

385
00:31:37,920 --> 00:31:40,279
They figured out how it all works
together, they figured out who needs to

386
00:31:40,279 --> 00:31:44,480
be scripted, they figured out all
of this hard stuff. That learning curve

387
00:31:44,559 --> 00:31:48,559
is extraordinarily low. I mean,
in Krytosis, you're using a Python like

388
00:31:48,640 --> 00:31:53,759
dialect. Most people are familiar with
Python. The set of I guess functions

389
00:31:53,759 --> 00:31:59,519
and tools you have are very are
very basic. Half a day to just

390
00:31:59,599 --> 00:32:01,759
rewrite it and something that's uh,
that's cleaner, and then it's going to

391
00:32:01,799 --> 00:32:05,319
take you a lot further. It's
probably about what you're going to get.

392
00:32:06,279 --> 00:32:09,920
Now. The part that's you know, frustrating for us and we're trying to

393
00:32:09,920 --> 00:32:13,839
figure out how to make this better
is what if you haven't written the seven

394
00:32:13,839 --> 00:32:15,680
thousand mind mass script, then what
do you do? You say, Hey,

395
00:32:15,680 --> 00:32:19,240
someone else wrote the seven thousand mind
mass script. I don't understand it

396
00:32:19,319 --> 00:32:21,680
yet, but I want it to
be. I want to get all these

397
00:32:21,680 --> 00:32:24,119
benefits that I can get from protosis. Now we have a really big now

398
00:32:24,119 --> 00:32:27,839
we have a really big verded curve, but the learning curve is mostly inside

399
00:32:27,839 --> 00:32:37,440
of the side. Interesting for sure. And then can you give us like

400
00:32:37,559 --> 00:32:45,559
some some success stories of like different
problems that customers have solved. I always

401
00:32:45,599 --> 00:32:46,960
think that's kind of cool because then
it, you know, it gives us

402
00:32:47,000 --> 00:32:51,160
something that we can like resonate with
and be like, oh, yeah,

403
00:32:51,440 --> 00:32:52,920
I remember when I had to do
that. Like, what are some cool

404
00:32:53,160 --> 00:32:57,480
things that you've seen people do with
krotosis? Well, I feel like your

405
00:32:57,920 --> 00:33:01,400
whole work experience history here, you
know, just recount those things and it's

406
00:33:01,519 --> 00:33:12,200
like, yeah, yeah, my
career could be a like a Forrest Gump

407
00:33:12,319 --> 00:33:24,119
story in tech. Yeah. I
Mean the biggest and like the most proud

408
00:33:25,119 --> 00:33:30,920
use case that we have for cryptosis
is what the Ethereum Foundation is doing with

409
00:33:30,000 --> 00:33:35,920
cryptosis. That's probably the thing that
we're most it's the most incredible thing.

410
00:33:37,000 --> 00:33:42,240
So for folks who aren't like deep
into the blockchain ecosystem, the Ethereum main

411
00:33:42,400 --> 00:33:47,319
net, so it's like the production
deployment of Ethereum is now extraordinarily complicated.

412
00:33:47,519 --> 00:33:52,720
There is like twenty different open source
components that have to get splat up together.

413
00:33:52,400 --> 00:33:57,400
There's no centralized DEVLOPS team for this, so it's all different people running

414
00:33:57,440 --> 00:34:04,400
their own different service with moose communication
between them. What krotosis, what they're

415
00:34:04,480 --> 00:34:07,199
using Katosis do. The Ethereum foundation
is to package all of that together in

416
00:34:07,880 --> 00:34:14,639
a one line run with high level
configurability, so you can say, hey,

417
00:34:14,639 --> 00:34:19,079
I want to run it with this
topology like these clients and these clients,

418
00:34:19,480 --> 00:34:24,280
and it works remarkably well. It
works remarkably well as it's it's a

419
00:34:24,360 --> 00:34:29,159
large degree of complexity. The Beast. I don't know what it would look

420
00:34:29,239 --> 00:34:31,960
like if we like unfolded it into
the Beast script scenario that you described,

421
00:34:34,280 --> 00:34:37,840
it would be very remarkable. But
now this thing it runs, if it

422
00:34:37,920 --> 00:34:43,000
runs the UH, you know,
you can one click replicated Thethereum main that

423
00:34:43,199 --> 00:34:47,320
on your laptop or inside of one
of your cloud boxes or Recubernators cluster at

424
00:34:47,360 --> 00:34:51,280
scale. UH, so you can
kind of hit the dev all the way

425
00:34:51,360 --> 00:34:58,440
to late stage testing performance testing.
UH. There was a team that built

426
00:34:57,679 --> 00:35:04,519
h They built it's basically like a
sad path testing on top of this tool,

427
00:35:04,800 --> 00:35:08,079
so like a path network perturbin in
lots of different ways, as you

428
00:35:08,119 --> 00:35:13,079
call it, chaos testing on top
of this, and it takes you all

429
00:35:13,119 --> 00:35:16,960
the way almost to deploy the proud
if you're going to make like a big

430
00:35:17,079 --> 00:35:22,880
change in the deep layers of the
network. So that's that's the most That's

431
00:35:22,920 --> 00:35:25,960
the thing I'm the most proud of
when it comes to like what are tools

432
00:35:25,960 --> 00:35:32,840
been able to do. There's a
large degree of DevOps complexity encapsulated in that

433
00:35:32,920 --> 00:35:37,000
project, and then there's a bunch
of people using it to do some really

434
00:35:37,039 --> 00:35:38,960
cool things on top of it.
I feel like that makes a lot of

435
00:35:38,960 --> 00:35:43,719
sense there as well, Like when
you have a lot of different testing scenarios

436
00:35:43,800 --> 00:35:49,280
that need to be captured and the
configurability for them is also determined sort of

437
00:35:49,360 --> 00:35:53,079
by the constraints of the test,
then you have a huge dimensionality of the

438
00:35:53,199 --> 00:35:58,320
data and configuration that you need to
specify to actually execute it. There.

439
00:35:58,400 --> 00:36:00,440
Whereas I feel like, you know, some other environments that may not be

440
00:36:00,519 --> 00:36:02,719
total like always the case, but
in this one, like, yeah,

441
00:36:02,760 --> 00:36:07,039
for sure, I can totally imagine, especially given the fact that it's there

442
00:36:07,119 --> 00:36:13,800
is no you know, official production
environment where you know cloud provider where THEOEUM

443
00:36:13,840 --> 00:36:16,920
is running on exactly exactly, And
it happens in cases where you're doing a

444
00:36:16,920 --> 00:36:22,800
lot of different testing things. And
it also happens when you're I guess you

445
00:36:22,840 --> 00:36:27,480
could call it like a feature branch
environment. It's it's these situations where you're

446
00:36:27,880 --> 00:36:32,519
you're like pre merge and you want
to spin up a representation environment that's specific

447
00:36:32,559 --> 00:36:37,079
to you're not really deving on it
because it's not inside it. You're like

448
00:36:37,440 --> 00:36:39,719
build and run loop as you're in
your IDE but it's kind of close to

449
00:36:39,800 --> 00:36:44,119
like your death because they're not ready
to put it into CI yet. That

450
00:36:44,280 --> 00:36:50,400
situation as well as one where inside
of an organization or an open source community

451
00:36:50,639 --> 00:36:55,719
you might have hundreds of those a
week of slightly different configurations, so it's

452
00:36:55,719 --> 00:36:59,079
either easy to configure or it's not. But if it's not, like this

453
00:36:59,239 --> 00:37:02,000
is a a great way to do
it. This gives me the sort of

454
00:37:02,559 --> 00:37:08,199
PTSD flashbacks to the previous environment that
I was working in where there was like

455
00:37:08,719 --> 00:37:14,760
different production environments that we had because
they were for different customers or different internal

456
00:37:14,760 --> 00:37:19,719
business units, and then we had
copies of them that were test environments that

457
00:37:20,000 --> 00:37:24,079
we're supposed to match the production one
because realistically, given the ridiculous amount of

458
00:37:24,079 --> 00:37:28,760
configuration, you almost needed to run
your tests in every single sort of clone

459
00:37:28,800 --> 00:37:31,159
of production however it could be possibly
configured, and then of course it was

460
00:37:31,199 --> 00:37:36,480
a problem where like the actual switches
from production didn't match where you're in your

461
00:37:36,519 --> 00:37:38,800
testing environment, and so you had
not only did you have to run your

462
00:37:38,800 --> 00:37:43,519
test in all of these fake versions
of production, but also make some changes.

463
00:37:44,039 --> 00:37:49,079
That for sure was quite the nightmare. And there's a mode of thought

464
00:37:49,159 --> 00:37:52,840
that I'm honestly a fan of this
movement. There's a mode of thought where

465
00:37:52,880 --> 00:37:58,719
you just say, can we forego
all of that and just make shipping the

466
00:37:58,760 --> 00:38:01,079
production as safe as possible and do
all of it like test in production.

467
00:38:01,840 --> 00:38:06,199
That motive thought is like sometimes possible, I think. I think that's like

468
00:38:06,360 --> 00:38:10,880
Honeycomb's big big model. They're an
observability tool like test and production. We're

469
00:38:10,880 --> 00:38:15,840
just going to give you really good
monitoring tooling so the moment something goes wrong,

470
00:38:15,920 --> 00:38:20,079
you can catch it right away.
For an impacts your entire customer base

471
00:38:20,199 --> 00:38:25,119
rollback like some some some products I
guess can do this. Otherwise a tiny

472
00:38:25,159 --> 00:38:30,119
little bug is a disastrous like medical
tooling. I don't know, Like I

473
00:38:30,119 --> 00:38:32,960
mean, I want to do it
for sure, sign me up. I

474
00:38:34,039 --> 00:38:37,719
mean we're in the at authors with
the five nine SLA where for sure we

475
00:38:37,760 --> 00:38:43,119
can't let stuff go through. I
think the biggest challenge is depending on what

476
00:38:43,119 --> 00:38:45,039
your business is actually like, what
the business logic looks like. The fear

477
00:38:45,079 --> 00:38:50,239
I've always had with these sort of
automatic rollbacks in a lot of cases where

478
00:38:50,280 --> 00:38:53,079
what happens if the problem is compounded
by changing the database version? And I

479
00:38:53,079 --> 00:38:58,280
always keep getting back to the Night
Capital four hundred and forty million dollars loss

480
00:38:58,320 --> 00:39:01,880
in forty five minutes, kind of
like the ridiculous thing is they had the

481
00:39:01,920 --> 00:39:06,639
wrong version of the software only running
on one machine, and they made it

482
00:39:06,719 --> 00:39:10,679
worse by restarting the cluster. Like
if they hadn't restarted anything, it would

483
00:39:10,679 --> 00:39:14,719
have been fine. And they assumed
that the problem was with the rollout,

484
00:39:14,760 --> 00:39:19,079
so they rolled back everything and then
also compounded the issue. And it's those

485
00:39:19,119 --> 00:39:22,360
things that really scare me, and
I like, I would so totally before

486
00:39:22,800 --> 00:39:27,320
if the automatic rollback could calculate whether
or not the you know, the database

487
00:39:27,360 --> 00:39:30,280
is that fault and also include that, And I feel like I'm not going

488
00:39:30,360 --> 00:39:34,320
to see that in my lifetime.
I don't know. To me, it

489
00:39:34,360 --> 00:39:37,760
just sounds like we're making excuses to
keep from giving people VIM access to the

490
00:39:37,760 --> 00:39:42,960
production server. I mean, you
know you're I think in control of that

491
00:39:43,039 --> 00:39:45,880
actually had poligon right well, like
you could just go and turn that on

492
00:39:45,000 --> 00:39:49,280
and let that happen and it could
be like it could be it could be

493
00:39:49,320 --> 00:39:53,239
the real case study. You could
do that right shop, whether it was

494
00:39:53,280 --> 00:39:55,639
a success. You know, published
that blog post. You know that you're

495
00:39:55,679 --> 00:40:00,280
doing this, why it's so great, and you know you could get on

496
00:40:00,320 --> 00:40:04,800
board with it, and then next
week's episode will be reviewing my resume because

497
00:40:04,840 --> 00:40:12,440
I'm looking for a job. I'll
never forget that article that was published by

498
00:40:12,559 --> 00:40:15,719
get lab about how they you know, accidentally deleted their production database. Uh

499
00:40:15,800 --> 00:40:20,039
it was like a decade ago.
Now. I was like, oops,

500
00:40:20,039 --> 00:40:23,079
Like I ran this command and I
thought I was on the replication uh server,

501
00:40:23,119 --> 00:40:32,000
and I was actually in production.
I've done that to do. Yeah,

502
00:40:32,280 --> 00:40:37,840
it was in AWS, and I
thought that I was on the on

503
00:40:37,880 --> 00:40:45,360
the staging environment and uh just dropped
an entire database cluster. Oh that that's

504
00:40:45,360 --> 00:40:47,840
not that bad, right, I
mean, you just restore that from from

505
00:40:47,880 --> 00:40:52,159
the one that you you know,
got snapshotted from an hour ago. Like

506
00:40:52,159 --> 00:40:54,559
it's much worse when you, like, you know, write that update statement

507
00:40:54,639 --> 00:40:59,280
in sequel and forget the select that
comes after it, right, you know,

508
00:40:59,280 --> 00:41:01,760
people say you should write that first, like write the from statement first

509
00:41:01,800 --> 00:41:06,480
and then change your select to an
update. Yeah, it's just it's ridiculous

510
00:41:06,519 --> 00:41:09,000
that those mistakes can be made.
But I think at every company I've been

511
00:41:09,039 --> 00:41:12,440
at, there's been a special name
for that, like, oh, don't

512
00:41:12,480 --> 00:41:15,280
eighty four the database, because like
that's just happened to be what the update

513
00:41:15,480 --> 00:41:21,320
was, where someone didn't configure the
restriction correctly and now there's some column with

514
00:41:21,400 --> 00:41:24,000
this wrong value in it and it's
so noticeable too, right, Like it's

515
00:41:24,000 --> 00:41:27,920
not like, oh, database is
gone, let's fix that. It's oh,

516
00:41:27,960 --> 00:41:31,039
when did someone change the name of
all of our customers to eighty four?

517
00:41:34,039 --> 00:41:36,760
I mean that's kind of like a
badge of honor though, like if

518
00:41:36,760 --> 00:41:39,960
you were if you were the person
who implemented eighty four, like, would

519
00:41:39,960 --> 00:41:46,360
you put that on your LinkedIn?
So yeah, I'm that guy. I

520
00:41:46,400 --> 00:41:52,280
think that was my biggest fear when
I started my engineering career was doing one

521
00:41:52,320 --> 00:41:59,119
of these things like the terrified of
touching anything that could look and feel something

522
00:41:59,199 --> 00:42:02,719
like deleting a duction database. And
I think it is about your honor.

523
00:42:02,719 --> 00:42:07,000
It's the moment when you realize,
like, once you do something like this,

524
00:42:07,440 --> 00:42:12,199
hopefully to a smaller scale, you
kind of learn like this is a

525
00:42:12,239 --> 00:42:16,280
process issue, like this should have
this should have never I never should have

526
00:42:15,480 --> 00:42:19,400
been able to do it. Shouldn't
have been a risk that I would do

527
00:42:19,519 --> 00:42:22,760
this. I don't. But when
you start out, you feel you feel

528
00:42:22,880 --> 00:42:28,760
I'm gonna get fired if I need
one big wrong. Yeah, and it's

529
00:42:28,760 --> 00:42:30,519
one of those things. I don't
think you can communicate that. I think

530
00:42:30,559 --> 00:42:37,239
the only way to learn that is
to to earn that badge. This is

531
00:42:37,280 --> 00:42:39,480
a failure everyone has to go through
in your life. You know, suffer

532
00:42:39,519 --> 00:42:44,960
that trauma. You'll live with the
PTSD from that moment on about how the

533
00:42:45,000 --> 00:42:47,079
more the larger system, because you
know you've gained more experience at this time,

534
00:42:47,119 --> 00:42:51,760
you're in a more important position.
Your impact to the company is obviously

535
00:42:51,840 --> 00:42:54,320
going to be higher. You'll never
make that same mistake again. Yeah,

536
00:42:54,400 --> 00:42:59,920
it's like riding a bike, Like
you don't really understand the pros and con

537
00:43:00,119 --> 00:43:02,639
of riding a bike until you face
plant into the side of your house.

538
00:43:02,840 --> 00:43:07,719
You're like, oh, gotcha,
breaks, it was a mailbox for me.

539
00:43:14,360 --> 00:43:21,800
Cool. So let's talk about like
we talk about mimicking the production environment,

540
00:43:21,800 --> 00:43:27,920
and production environments are always changing with
you know, updated code and updates

541
00:43:27,960 --> 00:43:32,440
to the database and sometimes breaking changes, especially like since we've been talking about

542
00:43:32,480 --> 00:43:37,840
databases, like breaking schema changes with
cartosis. How do you make sure that

543
00:43:37,880 --> 00:43:45,760
those production changes get integrated back into
the local dev tool environments. Yeah,

544
00:43:45,280 --> 00:43:50,800
right now, that's something that's something
the end user has to configure. We're

545
00:43:50,840 --> 00:43:54,519
looking into ways to get this more
to provide like a product doze, way

546
00:43:54,679 --> 00:44:00,880
to automatically get this. But right
now, when it comes to reduction change,

547
00:44:00,920 --> 00:44:04,360
my dev environment has to update to
it. You got to. You

548
00:44:04,440 --> 00:44:08,920
gotta have a process in place to
check out updating the configurations. Ideally,

549
00:44:08,960 --> 00:44:14,320
what you do is you any time
that you make a production change, it

550
00:44:14,360 --> 00:44:19,960
goes through your krotosis set up first. That's the ideal scenario, but it's

551
00:44:20,000 --> 00:44:23,159
not always going to happen that way, of course. Yeah. No,

552
00:44:23,280 --> 00:44:29,039
I mean I imagine that you're making
the database changes in your source code anyway

553
00:44:29,079 --> 00:44:32,960
to run a migration, if it's
schemed or schema less you're making the code

554
00:44:34,039 --> 00:44:38,199
changes that are associated, and you
would just you can just refresh your dev

555
00:44:38,280 --> 00:44:42,360
environment, like you know, blow
it away and start fresh, because the

556
00:44:42,679 --> 00:44:45,119
migration should just not even need to
be run potentially and it would just be

557
00:44:45,159 --> 00:44:50,360
the same as production. Well at
least that would be my hope. Yes,

558
00:44:50,760 --> 00:44:58,880
yes, it's it gets goofy when
people do things like like hot fixes,

559
00:44:58,880 --> 00:45:05,480
where like other people hot fixed stuff
and you never looked at it.

560
00:45:05,480 --> 00:45:09,840
It can get goofy. But happy
past scenario, it's not that hard to

561
00:45:09,840 --> 00:45:13,639
get it back up to. You
know, these were these experiences are important.

562
00:45:13,639 --> 00:45:16,760
I remember having to do a production
fix and at the time that meant

563
00:45:16,760 --> 00:45:22,920
that a different team created a SPN
feature branch for us. It was a

564
00:45:22,960 --> 00:45:28,039
hot patch of production, and I
ended up with this problem where there was

565
00:45:28,159 --> 00:45:31,639
code that should have worked in production, but it didn't play nicely with what

566
00:45:31,760 --> 00:45:34,639
was there, and you know,
I compared it to what was actually in

567
00:45:34,920 --> 00:45:37,639
and Devin, it didn't match.
And what I had found out is that

568
00:45:37,679 --> 00:45:39,719
there could be multiple production fixes that
had to be done at the same time,

569
00:45:40,039 --> 00:45:44,519
and merging in SVN didn't really work
that well. And what happened is

570
00:45:44,519 --> 00:45:47,559
is you would get merge conflicts there, and this team that was not the

571
00:45:47,599 --> 00:45:52,480
development team or knew how the feature
should actually work, was going and actually

572
00:45:52,880 --> 00:45:58,480
doing the merge conflict resolution themselves and
pushing that out. And that was a

573
00:45:58,559 --> 00:46:05,400
very horrifying learning experience for me.
I think I'd rather drop the production database.

574
00:46:07,119 --> 00:46:09,320
I mean, you don't even have
the code to look at to know

575
00:46:09,360 --> 00:46:12,960
what the issue was, because like
I had to go find these other branches

576
00:46:12,960 --> 00:46:15,519
that weren't merged to production and like
merge them into my code and see like

577
00:46:15,639 --> 00:46:20,000
it still wasn't right, Like where
is this? Where is this? Wrong?

578
00:46:20,079 --> 00:46:22,599
Code just didn't exist anywhere. And
then of course you have like the

579
00:46:22,639 --> 00:46:27,760
incident response team coming to your desk
because in these days, of course you

580
00:46:27,800 --> 00:46:30,519
had to be in the office and
like asking you, why are we losing

581
00:46:30,559 --> 00:46:35,840
one hundred thousand dollars every minute because
of this problem? I'm like, I

582
00:46:35,880 --> 00:46:45,280
have literally no idea. Yeah,
that's always cool too, That's just another

583
00:46:45,320 --> 00:46:51,840
fun experience of being able to quantify
the cost of downtime, like, oh

584
00:46:51,880 --> 00:47:00,480
wow, this is serious. I
feel bad enough to work like twenty four

585
00:47:00,480 --> 00:47:02,199
hours a day, seven days a
week, and you'll never be able to

586
00:47:02,199 --> 00:47:07,719
wait from your computer in case something
happens there. Right, that's the solution

587
00:47:08,239 --> 00:47:12,840
four seven with VIM on the production
server. Yeah, it's just going to

588
00:47:12,960 --> 00:47:15,599
keep you know, if you keep
bringing that up like I'm gonna, I'm

589
00:47:15,599 --> 00:47:17,239
going to write down this in my
in my notebook, you know, see

590
00:47:17,239 --> 00:47:22,159
when Will got this product ready to
go, where we'll let you you know

591
00:47:22,400 --> 00:47:24,559
it uses maybe s s H under
the hood or your ID you know,

592
00:47:24,760 --> 00:47:29,840
just you called VIM and production you
can even you know it even it lends

593
00:47:29,880 --> 00:47:36,280
itself a name VIP, right,
and we'll see where that goes. Right,

594
00:47:36,599 --> 00:47:38,320
I'm just going to keep campaigning until
they let me give a talk at

595
00:47:38,360 --> 00:47:43,480
like platform con about how to do
this. Well, you know, how

596
00:47:43,519 --> 00:47:46,840
to get accepted to those conferences.
What you do is you first just say

597
00:47:46,880 --> 00:47:51,679
you've done it, and and then
if they get accepted, then you'll have

598
00:47:51,679 --> 00:47:54,599
to come with the whole product and
actually try it out and I then present

599
00:47:54,679 --> 00:48:05,000
it were full of useful tips today. I mean I really I really like

600
00:48:05,039 --> 00:48:09,719
this idea of getting a tool in
the mix here for helping build the the

601
00:48:09,719 --> 00:48:16,480
the environment for real. Yeah,
and if you like to help illustrate the

602
00:48:17,119 --> 00:48:21,039
scale of this, you know,
with your success with the Ethereum foundation,

603
00:48:21,239 --> 00:48:28,440
that's a monster project with completely open
source, you know, and so I'm

604
00:48:28,440 --> 00:48:30,440
assuming that like if you want to
if like, if I wanted to contribute

605
00:48:30,480 --> 00:48:39,639
to the Ethereum project, I could
laugh and Warren, I think I'm going

606
00:48:39,679 --> 00:48:43,440
to ask for access to, you
know, update this tool, and I'd

607
00:48:43,440 --> 00:48:45,039
be like, well, you know, will you know, making changes to

608
00:48:45,119 --> 00:48:52,599
it? I'm not. I'm not
sure I know. But if I wanted

609
00:48:52,599 --> 00:48:57,920
to to contribute to Ethereum, like
I could use crytosis to build my local

610
00:48:58,000 --> 00:49:01,400
div environment which helps me write the
code, test it, and then create

611
00:49:02,320 --> 00:49:07,840
a poll or open a poll request
that's going to be more likely to meet

612
00:49:07,960 --> 00:49:13,880
their contribution guidelines. Is that a
fair statement? It's I mean, it

613
00:49:13,920 --> 00:49:17,000
probably is the easiest way to do
that, Yeah, yeah, versus the

614
00:49:17,079 --> 00:49:23,719
alternative prior to because I worked with
Ethereum before they started at adopting cretosis,

615
00:49:23,800 --> 00:49:29,719
like prior to that, you know, it was like twenty or thirty steps

616
00:49:29,760 --> 00:49:32,239
to get this thing running, you
know, where you were downloading from all

617
00:49:32,280 --> 00:49:36,440
these different repositories, and then you
were compiling it, and then you had

618
00:49:36,440 --> 00:49:40,599
to figure out in the config files
which parts you had to change to make

619
00:49:40,679 --> 00:49:44,519
these different things that you just compiled, but you weren't really sure what they

620
00:49:44,559 --> 00:49:49,679
were talk to each other and so
yeah, this like having this type of

621
00:49:49,719 --> 00:49:52,800
solution makes it easier, and that
could be that could be another benefit for

622
00:49:52,840 --> 00:49:58,079
you, especially if you're an open
source project, making it easier for people

623
00:49:58,119 --> 00:50:05,239
to contribute to your product. There's
actually a good rule of thumb for when

624
00:50:05,360 --> 00:50:10,079
an open source project or community would
would benefit from from using for dosis,

625
00:50:10,079 --> 00:50:14,400
and it's when you go to the
docks and it says how to run locally

626
00:50:14,719 --> 00:50:17,719
if it's like the longer that thing
is, the better you're going to get.

627
00:50:20,400 --> 00:50:22,480
It's kind of the general rule of
thumb is how long is that thing?

628
00:50:23,400 --> 00:50:25,880
So yeah, for the ethereum,
it was, it was huge.

629
00:50:27,239 --> 00:50:30,039
There's other general rule of thumb is
that that that that's pretty much how it

630
00:50:30,039 --> 00:50:34,639
works, uh and and it is
help work for contributing. It's most of

631
00:50:34,679 --> 00:50:40,199
the people, most of our implementations
are open source entirely, so it's you

632
00:50:40,199 --> 00:50:45,559
know, it's it's people contributing back
to those uh, people that use them,

633
00:50:45,719 --> 00:50:49,199
putting things back, getting more options
available. And that really really helps

634
00:50:49,239 --> 00:50:53,599
because our team doesn't have time to
handle all of the different scenarios. UH.

635
00:50:53,960 --> 00:50:59,239
When we start building examples and everything. So having people contribute back is

636
00:50:59,519 --> 00:51:04,199
I think the the Ethereum package,
it's called a package, the definition of

637
00:51:04,239 --> 00:51:07,000
how to spin up an environment,
the Ethereum package. It really is a

638
00:51:07,039 --> 00:51:10,840
remarkable work of open source contributions coming
back. They're very, very engaged users.

639
00:51:12,679 --> 00:51:15,559
I think you touched on this a
little bit earlier on about first run

640
00:51:15,599 --> 00:51:17,000
the licensing model and then and then
you change off to that. Like if

641
00:51:17,079 --> 00:51:22,000
someone's going to get started today with
this, is it I install a CLI

642
00:51:22,159 --> 00:51:24,800
and write a configuration file. Is
there a hosted version? Like on these

643
00:51:24,880 --> 00:51:30,400
packages available and you'll deploy them automatically
to my cloud environment. What does this

644
00:51:30,400 --> 00:51:37,119
look like? There's actually two there's
two versions right now. The standard way

645
00:51:37,199 --> 00:51:40,400
is download the Clyde the a homebrew
or app get or whatever is the preferred

646
00:51:40,719 --> 00:51:45,840
installation thing. Most people who start, they're starting because they see a package

647
00:51:45,880 --> 00:51:50,079
they want to run. So so
most people they're just running a package when

648
00:51:50,079 --> 00:51:53,320
they start to see how it spins
up, and then they start modifying that

649
00:51:53,360 --> 00:51:55,639
package if they want to modify it, or they're like, hey, this

650
00:51:55,719 --> 00:51:59,760
is cool, I'm going to brite
my own my own system, and it's

651
00:51:59,800 --> 00:52:01,840
all all happens on your own laptop
and you're own hardware, and you just

652
00:52:01,920 --> 00:52:07,199
you just run with it and go. And then we do have a cloud

653
00:52:07,679 --> 00:52:10,280
right now that we host where Hey, let's say you don't want to down

654
00:52:10,679 --> 00:52:14,440
the CLI and you don't want to
be using your own compute to be running

655
00:52:14,760 --> 00:52:17,400
the environment, you can just button
click. It's like a no code interface

656
00:52:17,400 --> 00:52:24,239
to run any existing package. There
is also no code interface for modifying the

657
00:52:24,280 --> 00:52:28,679
package to a certain degree. That's
evolving, so you can get more and

658
00:52:28,719 --> 00:52:32,679
more power over modifying it. And
then once you're excited about it, you

659
00:52:32,679 --> 00:52:36,079
can write your own package and then
you can deplay that there as well.

660
00:52:37,880 --> 00:52:43,519
And then the packages are those the
ones that end up showing up in the

661
00:52:43,639 --> 00:52:47,800
Kurtosis catalog. Yes, yeah,
because the Curtosis catalog is actually really cool.

662
00:52:47,840 --> 00:52:52,480
Specifically, you know, if you
going back to the Ethereum example,

663
00:52:52,639 --> 00:52:57,639
you know, you can just grab
that from the catalog and it runs for

664
00:52:57,679 --> 00:53:00,480
you, right, and then there's
there's actually quite a few out there in

665
00:53:00,519 --> 00:53:04,440
your catalog. One that I happened
to know of is the Polygon CDK as

666
00:53:04,480 --> 00:53:07,280
of last week, just went up
in the catalog. So if you want

667
00:53:07,280 --> 00:53:10,159
to run that, you know,
just grab it from the catalog and there

668
00:53:10,199 --> 00:53:14,199
you go. Interesting. Yeah,
they're right there. And and what the

669
00:53:14,400 --> 00:53:16,880
I mean the way the catalog works
right now is it scans public GitHub.

670
00:53:16,960 --> 00:53:22,679
So if it's an open source package, it's going to get detected and populated

671
00:53:22,719 --> 00:53:25,480
into the catalog and then you can
just run it. Because it's an open

672
00:53:25,559 --> 00:53:30,880
source, it's an open source tool. So the Polygon CDK, for example,

673
00:53:31,320 --> 00:53:35,480
the Polygon team put a lot of
work into to get into the package

674
00:53:35,559 --> 00:53:38,559
very in a very nice format,
and now it's there and you can just

675
00:53:38,599 --> 00:53:45,159
button click run it. And one
of the most exciting things about having made

676
00:53:45,159 --> 00:53:49,760
this step of opening everything is that
we're seeing stuff happen without us knowing it

677
00:53:49,880 --> 00:53:52,079
or tracking it too closely, so
like, oh, look like look at

678
00:53:52,119 --> 00:53:54,920
what the Polygon team has done.
This is amazing And that's very exciting.

679
00:53:57,280 --> 00:54:00,920
Yeah, for sure. And I
think that's because y'all are completely open source

680
00:54:01,039 --> 00:54:07,599
as well. Right, So the
tool is actually the BSL license, okay,

681
00:54:07,000 --> 00:54:15,280
source available. You can do everything
except for recreate our right on it,

682
00:54:15,639 --> 00:54:22,280
which seems like a fair constrained exactly. Yeah, no, but I

683
00:54:22,320 --> 00:54:29,719
think that's really cool having the catalog
there, especially with the open source communities,

684
00:54:30,760 --> 00:54:35,039
and it seems that Kurtosis is really
resonating there to make it easy for

685
00:54:35,119 --> 00:54:40,280
people to just run and then contribute
and then improve your product and in a

686
00:54:40,320 --> 00:54:44,800
consistent format, you know, because
I think that's one of the biggest challenges

687
00:54:44,800 --> 00:54:52,119
of running an open source project is
verifying or vetting out the contributions that you

688
00:54:52,199 --> 00:54:58,239
get to make sure that they have
done in some cases just tribal knowledge.

689
00:54:59,079 --> 00:55:02,360
Yes, yes, yes. And
it's one thing we were noticing, especially

690
00:55:02,360 --> 00:55:07,400
as we work with more and more
people who have cloud dependencies, is the

691
00:55:07,440 --> 00:55:14,280
more that your system is is composed
of open source components, the more this

692
00:55:14,360 --> 00:55:19,880
problem is important to you because you're
you're dealing with a launch of different software

693
00:55:19,880 --> 00:55:23,400
components built by different organizations, where
where you can run that entire software component

694
00:55:23,440 --> 00:55:27,840
yourself. That's the whole point of
the open source So you don't have the

695
00:55:28,000 --> 00:55:32,039
issue of you know, cloud provider
run a production environment has zero incentive to

696
00:55:32,039 --> 00:55:36,079
like make you able to like modify
it or move it. In the open

697
00:55:36,079 --> 00:55:38,239
source communities, not the case,
you actually can do it entirely. So

698
00:55:38,639 --> 00:55:45,320
the more open source your tool is
in terms of like the percentage of services

699
00:55:45,320 --> 00:55:49,199
that are involved in your environment which
are open source. The more the more

700
00:55:49,239 --> 00:55:54,199
you get this benefit of like how
do we verify that the contribution is going

701
00:55:54,280 --> 00:55:57,599
to work and makes sense? And
you do kind of have to run it

702
00:55:57,639 --> 00:56:00,519
all to see it. Yeah,
I would even take it further, honestly

703
00:56:00,679 --> 00:56:07,360
that most open source solutions out there
that aren't using a strategy or have a

704
00:56:07,400 --> 00:56:12,480
strategy for easy for contributors to come
in and run the software would benefit a

705
00:56:12,480 --> 00:56:15,199
lot from a strategy like this,
because realistically, if you go out there

706
00:56:15,239 --> 00:56:20,199
and find an open solution, no
matter what it is, even one used

707
00:56:20,199 --> 00:56:23,599
by thousands of organizations, it is
quite the challenge to get that up and

708
00:56:23,679 --> 00:56:28,280
running. Even if it's a darker
container, even if it's you know,

709
00:56:28,320 --> 00:56:30,360
in a registry like it's still there's
a lot that goes on with it.

710
00:56:30,400 --> 00:56:37,760
I remember it was it was even
recently that you didn't need a zookeeper anymore

711
00:56:37,800 --> 00:56:42,199
to run Kafka, Like it was
always this sort of mess that if you

712
00:56:42,239 --> 00:56:45,719
ever wanted to run Kafka yourself,
you needed three different services running on your

713
00:56:45,960 --> 00:56:51,840
machine with multiple nodes and clusters at
the same time in order to even test

714
00:56:51,880 --> 00:56:55,000
this out. And I'm like,
what, yes, yes, it's a

715
00:56:55,039 --> 00:56:59,440
big issue, and it's and it's
for the most part, in my opinion,

716
00:56:59,480 --> 00:57:04,039
it's a bit of a silent issue
because if you're contributing back to an

717
00:57:04,039 --> 00:57:07,159
open source project, and a lot
of the time you're doing it out of

718
00:57:07,880 --> 00:57:10,559
almost like a hobby mentality. It's
like, I want to help this project,

719
00:57:10,599 --> 00:57:14,920
I'm interested in it. If there's
a really big friction bar for me

720
00:57:15,000 --> 00:57:17,039
to do that, I'm not getting
paid to contribute back to it. That's

721
00:57:17,079 --> 00:57:22,360
the whole idea. So I'm not
going to put an inorder the amount of

722
00:57:22,400 --> 00:57:25,639
effort into knocking down the door to
say, like make my testing environment better,

723
00:57:25,760 --> 00:57:29,639
like make my environment better, I'm
just going to disappear. So like

724
00:57:29,679 --> 00:57:35,159
the amount of open source contributions that
you're missing because because people just got frustrated

725
00:57:35,159 --> 00:57:37,519
in the first five minutes and say, you know what, like someone else

726
00:57:37,599 --> 00:57:42,559
is going to handle this contribution at
some point hopefully is I can just imagine

727
00:57:42,559 --> 00:57:45,440
the velocity that you're losing there.
I mean, I hope that's really the

728
00:57:45,760 --> 00:57:50,920
case, that that's even where they're
falling off. I find that in Unfortunately,

729
00:57:50,960 --> 00:57:52,400
the state of the open source today
is like you find an issue out

730
00:57:52,440 --> 00:57:55,719
there, or you want to even
make a change and you may open an

731
00:57:55,719 --> 00:58:00,000
issue or open that poor request and
just never hear back. And with stuff

732
00:58:00,000 --> 00:58:05,119
like this, you know that the
team has invested in what that experience is

733
00:58:05,239 --> 00:58:07,320
on building stuff up. So it
could be a badge of honor really to

734
00:58:07,360 --> 00:58:13,199
say this open source project is using
kurtosis or another tool that makes this experience

735
00:58:13,239 --> 00:58:15,719
better because you know that you thought
about it, that they care about getting

736
00:58:15,760 --> 00:58:24,679
those contributions. Then, so,
is there is there a thought pattern behind

737
00:58:24,679 --> 00:58:29,519
you, Like when we talk about
production servers with the whole DevOps movement,

738
00:58:30,519 --> 00:58:36,719
we kind of came up with this
philosophy of servers are cattle not petch,

739
00:58:36,800 --> 00:58:39,599
you know, referring to the fact
that they should be short lived and you

740
00:58:39,639 --> 00:58:45,159
shouldn't have emotional attachment tool them.
You shouldn't be like hand curating them.

741
00:58:45,440 --> 00:58:52,559
But I feel like dev environments maybe
didn't fall under that umbrella. Is there

742
00:58:52,800 --> 00:58:59,559
like a thought process or something that
you've noticed through critosis where you kind of

743
00:58:59,760 --> 00:59:05,400
see people shifting away from the long
lived development environments because it's a little easier

744
00:59:05,599 --> 00:59:10,000
and standardized to bring them up.
Now, it's I don't know if I

745
00:59:10,039 --> 00:59:14,480
would say that it's a shift away
from it. It's in some ways it

746
00:59:14,559 --> 00:59:19,880
is for certain use cases, like
so how could I describe this? What

747
00:59:19,920 --> 00:59:24,320
one thing crtosis does is if you
run your integration tests in CI and cider

748
00:59:24,360 --> 00:59:29,440
crtosis, it's quite easy to rerun
that locally in your laptop. So if

749
00:59:29,480 --> 00:59:35,320
you want to debug or failing integration
and the end test, rather than recreating

750
00:59:35,320 --> 00:59:37,760
it in a remote cluster that's like
a shared cluster, you would just rerun

751
00:59:37,800 --> 00:59:45,079
it on your laptop. And in
that case there's a shift towards ephemeral environments,

752
00:59:45,320 --> 00:59:49,280
like you're just for depth, like
you're just running vision. But there's

753
00:59:49,320 --> 00:59:52,840
also people who are like, I'm
going to use kryptosis to set up a

754
00:59:52,880 --> 00:59:59,119
persistent development environment because the other deaths
of my team it's just significantly easier for

755
00:59:59,159 --> 01:00:01,960
them to ping an endpoint that's sitting
there stable and to get into that.

756
01:00:02,119 --> 01:00:07,239
Then even to like download a tool
and then re run some commands every single

757
01:00:07,280 --> 01:00:10,800
time, and and then you get
into the problem of like data management.

758
01:00:10,920 --> 01:00:17,199
So if you've invested in your dev
environment, having a particular data set behind

759
01:00:17,239 --> 01:00:22,199
it that's useful for that dev project, and then investing the extra effort into

760
01:00:22,320 --> 01:00:25,960
cloning and moving that data for an
ephemeral use, it often doesn't make sense

761
01:00:27,000 --> 01:00:31,400
you just use the same thing.
So I think it really is It really

762
01:00:31,440 --> 01:00:40,119
is dependent on the use case.
But there definitely is a meaningful difference between

763
01:00:40,280 --> 01:00:45,039
how you treat your dev servers and
your broad servers. So like the cattle,

764
01:00:45,119 --> 01:00:47,800
the cattle not pets thing. I
think it's totally fine for your dev

765
01:00:47,840 --> 01:00:52,519
server to be a pet. Like
it's there's there's there's times when it makes

766
01:00:52,519 --> 01:00:54,519
a lot of sense. The moment
you want to see that it looks like

767
01:00:54,559 --> 01:00:58,159
braud, you do kind of want
to turn it into a cattle just to

768
01:00:58,199 --> 01:01:00,079
see how it works. But there's
a lot of telling reasons to treat it

769
01:01:00,119 --> 01:01:07,760
like a pet, especially if you
are debugging an issue or trying to figure

770
01:01:07,800 --> 01:01:10,920
out the way things kind of work. You want to have that state sitting

771
01:01:10,960 --> 01:01:15,719
around, and you want to really
preserve that because it's going to take you

772
01:01:15,760 --> 01:01:19,639
some time thinking and creative time to
work with it. And if it's a

773
01:01:19,719 --> 01:01:23,239
cattle and you lose it halfway through
your work, it's miserable. You got

774
01:01:23,239 --> 01:01:29,000
to restart, right And I think
I heard you say that there is a

775
01:01:29,679 --> 01:01:34,599
there's a use case here for using
something like critosis to create a persistent dev

776
01:01:34,719 --> 01:01:39,119
environment that provides a common set of
services for your dev team to access.

777
01:01:39,159 --> 01:01:44,840
So not everyone has to run that
locally, but there's one that's centrally available

778
01:01:44,880 --> 01:01:50,920
for them that maybe has a specific
configuration or has a something unique to it

779
01:01:50,960 --> 01:01:55,559
that everyone needs to be thinking about
or addressing in their local development efforts.

780
01:01:55,599 --> 01:02:00,079
So you just provide that as a
shared service for everyone. Yes, and

781
01:02:00,400 --> 01:02:06,679
that's useful for a lot of development
workflows. But then there's another set of

782
01:02:06,679 --> 01:02:10,519
development workflows where you're going to say, I actually do want to isolate for

783
01:02:10,960 --> 01:02:16,719
an ephemeral purpose a particular set of
micro services or a data set, and

784
01:02:16,760 --> 01:02:22,400
I want that to be my sandbox
that nobody else is touching. And that

785
01:02:22,480 --> 01:02:23,480
happens a lot as well, because
it's like, hey, what if three

786
01:02:23,519 --> 01:02:27,760
different teams want to use the dev
cluster and they're all hitting each other,

787
01:02:27,920 --> 01:02:30,639
and then you have to go on
Slack and say, hey, wait a

788
01:02:30,639 --> 01:02:31,440
minute, like, let me use
it for the next two hours, you

789
01:02:31,519 --> 01:02:36,679
guys, when you blow away their
work, and it's miserable, But that's

790
01:02:36,760 --> 01:02:40,199
only sometimes. So the best world
is you have the choice. When you

791
01:02:40,280 --> 01:02:43,920
need to get the isolation, you
can get it quite easily. When you

792
01:02:44,679 --> 01:02:47,400
don't need it, you can use
other people's work. And that's the direction

793
01:02:47,440 --> 01:02:54,599
that we're trying to go towards,
is how do we provide we like high

794
01:02:54,679 --> 01:02:58,519
high level. You want it to
feel like you're in Harry Potter and you

795
01:02:58,519 --> 01:03:00,519
have a wand and you're just like
that, you have the figure need Like

796
01:03:00,559 --> 01:03:05,239
that's the highest level user experience that
you want to get to. It's like

797
01:03:05,239 --> 01:03:07,760
cool, Like, how do you
design the want such that you can do

798
01:03:07,880 --> 01:03:10,960
that? And I think you just
want to have the configurability to be able

799
01:03:12,000 --> 01:03:15,000
to pick which in this scenario,
I want the shared duftbuster or I want

800
01:03:15,039 --> 01:03:21,960
the isolated if I'm one right on, So, continuing on that thought,

801
01:03:22,039 --> 01:03:24,800
what does the future of krotosis look
like? What are the in addition to

802
01:03:24,840 --> 01:03:29,920
that, what are some of the
things that you would like to see getting

803
01:03:30,000 --> 01:03:36,920
implemented? Absolutely, so I said
before, it's right now. Crotosis operates

804
01:03:37,000 --> 01:03:42,679
to you give your terraform scripts or
your open TOFU and it deploys your provisions,

805
01:03:42,719 --> 01:03:46,239
your hardware so to speak, and
maybe your virtualization later like the communities

806
01:03:46,320 --> 01:03:52,679
Buster or Ductor Engine, and then
Crotosis deploys on top of that. We

807
01:03:52,719 --> 01:03:55,079
want crtosis to move in the direction
where you no longer have to do that

808
01:03:55,119 --> 01:03:59,440
first part. You no longer have
to think about and dig into like how

809
01:03:59,440 --> 01:04:02,480
do I how do I split my
my cloud assets? Over how do I

810
01:04:02,519 --> 01:04:06,280
provision my hardware? We want to
get better and better and better at providing

811
01:04:08,679 --> 01:04:13,039
tools to not have to worry about
that, so that that's where we're going.

812
01:04:13,920 --> 01:04:18,519
You know, the more that the
folks in the blockchain space they use

813
01:04:18,599 --> 01:04:23,159
a lot of open source tools where
it's just all can be containerized and you

814
01:04:23,199 --> 01:04:26,280
get everything in the container. The
more that you're working with folks who have

815
01:04:26,360 --> 01:04:30,800
alliances on like a to bus services
or manage authentication services and this kind of

816
01:04:30,800 --> 01:04:34,440
thing, the more that that abstraction
is not so useful anymore because those are

817
01:04:34,440 --> 01:04:38,480
the issues, the issues of the
cloud stuff. So the more that we

818
01:04:38,519 --> 01:04:41,599
can build an instruction there that captures
that stuff in a way that's useful,

819
01:04:42,000 --> 01:04:44,519
that's where we want to go,
and that's where we're doing all this research,

820
01:04:44,559 --> 01:04:47,440
and that's where that's where this thought
experiment of like, well, to

821
01:04:47,559 --> 01:04:54,880
some degree you're limited by the cloud
provider allowing you to easily get like everything

822
01:04:55,039 --> 01:04:59,440
worked over to some degree, have
those limitations. So then, given that's

823
01:04:59,440 --> 01:05:01,039
the world we live in, how
do we get as close as possible?

824
01:05:01,079 --> 01:05:03,679
And that's kind of the design work
that we're looking at right now. Well,

825
01:05:03,719 --> 01:05:08,840
I think there's totally a gap here
in the IAIC world where it doesn't

826
01:05:08,840 --> 01:05:13,679
feel like any of the offerings are
super compelling. There's I mean, I

827
01:05:13,719 --> 01:05:20,440
think anyone that assigns themselves a DevOps
title has a unique perspective on complaining about

828
01:05:20,480 --> 01:05:25,320
literally everything that exists. So you
know, of course the tools that we

829
01:05:25,519 --> 01:05:29,000
have, you know, are not
as good as they should be. And

830
01:05:29,119 --> 01:05:31,639
I think there's always a space for
someone to say I wrote a better terror

831
01:05:31,639 --> 01:05:34,559
form or whatever, because I feel
like that's sort of the abstraction layer that

832
01:05:35,159 --> 01:05:41,440
exists over the cloud providers. And
I'm not saying that anyone should go and

833
01:05:41,480 --> 01:05:44,719
create a new is C tool.
I'm sure someone just got that idea right

834
01:05:44,840 --> 01:05:47,239
when you can do that, you
absolutely can do that and wrap the SDKs

835
01:05:47,280 --> 01:05:51,000
out there. But I do I
do feel like that there's always there is

836
01:05:51,000 --> 01:05:56,800
definitely a room here to grow in
that direction. Yeah, and then that's

837
01:05:58,159 --> 01:06:00,760
it's kind of a maybe a fork
in the road. But the two flip

838
01:06:00,840 --> 01:06:04,840
philosophies is like create that layer so
that it's very easy to do those forks.

839
01:06:05,079 --> 01:06:10,519
The other one is, I guess
it's you call it like the honeycomb

840
01:06:10,679 --> 01:06:14,440
or the launch Darkly philosophy, where
it's like, can you build tools that

841
01:06:14,559 --> 01:06:18,239
make production just safer, like feature
flags, so now just don't worry about

842
01:06:18,239 --> 01:06:23,440
testing or like like so much.
Just get into production under feature flag and

843
01:06:23,480 --> 01:06:26,480
then and then just turn it on
for a certain set of customers and then

844
01:06:26,480 --> 01:06:30,599
you'll figure out how things work or
like deserveability thing. Instead of focusing so

845
01:06:30,679 --> 01:06:33,960
much on like fourteen year staging and
testing environment, you just go into production.

846
01:06:34,039 --> 01:06:38,000
We're going to monitor everything so closely
that we're going to minimize any kind

847
01:06:38,000 --> 01:06:43,519
of damage. And there's I think
there's gaps on both sides, and there

848
01:06:43,840 --> 01:06:47,599
the different engineering leaders and platform leaders
have different opinions about about which way they

849
01:06:47,639 --> 01:06:54,360
want to go, so it's kind
of a lots of different picks on.

850
01:06:54,800 --> 01:06:58,679
I mean, I definitely have an
opinion on that. Realistically, it would

851
01:06:58,719 --> 01:07:00,400
be nice if we could get to
that world. I don't think the cloud

852
01:07:00,400 --> 01:07:04,880
providers offer us the sort of control
over the environment that would allow us to

853
01:07:04,960 --> 01:07:10,280
perform like one out of every million
requests. Yeah, go and changes.

854
01:07:10,440 --> 01:07:15,639
I just feel like we're much better
off pulling production down to another environment to

855
01:07:15,960 --> 01:07:20,400
validate that the code that we have
is exactly executing as we want, and

856
01:07:20,440 --> 01:07:25,400
we can do it, you know, creating synthetic requests from our production environment

857
01:07:25,400 --> 01:07:30,599
and replicating that against the ephemeral,
dynamically created dev one. This is a

858
01:07:30,679 --> 01:07:36,559
perfect example of what we're seeing is
Warren who is like, let's get things

859
01:07:36,760 --> 01:07:41,519
synthetically mocked, Let's get everything running
locally, Let's get it as close as

860
01:07:41,519 --> 01:07:44,400
possible, make sure things work,
and then we have Will who wants to

861
01:07:44,440 --> 01:07:54,440
film Upbraud team VIM right here.
I think valid approaches about valid approaches.

862
01:07:54,480 --> 01:07:58,159
People have different appetites what they want
to do. No, I mean,

863
01:07:58,199 --> 01:08:01,519
there's still definitely things where I'm a
huge proponent of pushing stuff to testing in

864
01:08:01,639 --> 01:08:05,760
production, and we do that under
certain circumstances where we make sure that we

865
01:08:05,920 --> 01:08:10,119
wrapped. Like parallel execution is a
huge thing for us. When we're changing

866
01:08:10,360 --> 01:08:14,079
some of the business logic of how
something works, we'll usually execute two different

867
01:08:14,079 --> 01:08:18,840
paths actually simultaneously in production and validate
whether or not the outcome is exactly as

868
01:08:18,840 --> 01:08:24,760
it should be, and before turning
off the legacy one and when you do

869
01:08:24,840 --> 01:08:30,000
that, When you do that,
does that output still go back to like

870
01:08:30,079 --> 01:08:35,159
your fork traffic or do you know
it's in process? Like it'll basically the

871
01:08:35,159 --> 01:08:40,359
equivalent of having two threads. You
know, you call this method and it

872
01:08:40,399 --> 01:08:43,159
does two things at the same time, and then we throw away the new

873
01:08:43,239 --> 01:08:45,880
result until we like we validate it, and then we throw it away and

874
01:08:45,960 --> 01:08:49,239
if we get any invalidations, then
we you know, that's an alert that

875
01:08:49,279 --> 01:08:53,279
someone goes and looks at and comes
back and says, Okay, you know

876
01:08:53,279 --> 01:08:56,680
what happened here. And if we
don't get any alerts after a period of

877
01:08:56,720 --> 01:09:00,279
time, then that sort of level
of testing has been completed and we can

878
01:09:00,359 --> 01:09:03,119
just delete the old code and own
and actually return the new stuff. I

879
01:09:03,199 --> 01:09:06,199
love that setup. I think that's
I think that's great. I think I

880
01:09:06,199 --> 01:09:12,199
think that's like decently. I mean, it's it's a far cry from like

881
01:09:12,279 --> 01:09:15,279
VIM and prade, but like it's
almost a step towards that. Right.

882
01:09:15,359 --> 01:09:18,680
It's like like you're at least able
to get something into production more safely,

883
01:09:18,920 --> 01:09:25,000
even though it's I imagine there's constraints
on like you wouldn't be able to do

884
01:09:25,039 --> 01:09:27,920
that if there was like very secure
data that was going into the response that

885
01:09:27,960 --> 01:09:29,880
you're like, you shouldn't see.
You wouldn't be able to nidate it that

886
01:09:29,960 --> 01:09:31,239
well, well, I mean we
we sort of had that because we're an

887
01:09:31,279 --> 01:09:36,680
identity provider federated and aggregation and no
one sees the data. It's just you

888
01:09:36,680 --> 01:09:41,399
know, hey, there's something here
that doesn't match, and you know,

889
01:09:41,439 --> 01:09:45,119
you can sort of revalidate that and
realistically the actual who the user is doesn't

890
01:09:45,119 --> 01:09:47,760
really matter in those circumstances. You
can just sort of see what it looks

891
01:09:47,840 --> 01:09:53,920
like. If for some reason there
is some concern. We encrypt the logs

892
01:09:53,960 --> 01:09:57,920
that get written even before they leave
the production environment. We hash the data

893
01:09:57,920 --> 01:10:01,439
that we'll get seen automatically by any
of our engineers before it leaves. So

894
01:10:01,960 --> 01:10:06,520
I mean, very rarely it's the
case that that actually, like what someone's

895
01:10:06,560 --> 01:10:12,039
last name was was relevant for us. That's a very cool setup. I

896
01:10:12,159 --> 01:10:16,880
like that. So that's some discipline
there, though, Like that's very that

897
01:10:16,920 --> 01:10:21,159
sounds to me like that sounds like
a very disciplined approach because you're being very

898
01:10:23,000 --> 01:10:30,000
deliberate about what you're testing, writing
that second path for it, and then

899
01:10:30,039 --> 01:10:36,439
you're having to specifically define and measure
what the success criteria for that is.

900
01:10:36,840 --> 01:10:42,399
So that's a charitable way of looking
at it. You know, maybe that's

901
01:10:42,399 --> 01:10:46,239
my na You don't do that at
all. I mean there was nobly a

902
01:10:46,279 --> 01:10:48,319
time in my life where I'm like, you know, you're releasing that code

903
01:10:48,359 --> 01:10:53,479
to production. Why is there an
arbitrary try catch block you know around that?

904
01:10:54,279 --> 01:10:58,520
But there is another aspect here when
if you're if you care about reliability

905
01:10:58,720 --> 01:11:01,439
and that your stuff don't really go
down and production and the code you're changing,

906
01:11:01,600 --> 01:11:05,600
like there are too many unknowns there. Realistically, you didn't test literally

907
01:11:05,720 --> 01:11:11,720
every case, and sometimes it's not
so critical that you have tested everything,

908
01:11:11,800 --> 01:11:15,560
but you know, validating something happens
in production, like oh, we don't

909
01:11:15,560 --> 01:11:16,920
know if a customer is using this. Well you can at least create a

910
01:11:16,960 --> 01:11:20,560
log message that says, uh,
hey, customer is using this thing,

911
01:11:20,640 --> 01:11:25,439
don't turn it off yet and run
that for a bunch of weeks so that

912
01:11:25,479 --> 01:11:30,800
you have the data to validate.
Uh yeah, I mean you laugh,

913
01:11:30,960 --> 01:11:34,399
but you're not if you don't take
that first step, Like this is absolutely

914
01:11:34,439 --> 01:11:40,600
the advice that someone could be utilizing
to improve their decision making apility, not

915
01:11:40,680 --> 01:11:44,600
like some spreadsheet with a list of
features that are on or off per customer,

916
01:11:44,840 --> 01:11:46,840
but really, like what is happening
in your production environment, and this

917
01:11:46,920 --> 01:11:49,560
is just the next level after that, which is release the code, have

918
01:11:49,640 --> 01:11:54,800
multiple versions running at the same time, measure performance, reliability, you know,

919
01:11:55,000 --> 01:11:59,760
consistency, whatever that is. This
is just how we've happened to be

920
01:11:59,800 --> 01:12:02,680
doing and I feel like it works
really well for us. Unfortunately, so

921
01:12:03,039 --> 01:12:05,520
you know, we're not going to
migrate to something else. I don't know.

922
01:12:05,600 --> 01:12:10,000
I haven't heard anyone else doing something
that you know really changes this.

923
01:12:10,199 --> 01:12:14,920
I mean there's blue green deployments right
where you're siphoning traffic off from your new

924
01:12:15,399 --> 01:12:18,640
version. It sometimes it doesn't really
make sense a lot when you have a

925
01:12:18,640 --> 01:12:24,439
lot of features going to production over
time, where the features especially aren't interacting

926
01:12:24,439 --> 01:12:27,760
with each other. Having them weigh
in a queue to get through this and

927
01:12:27,760 --> 01:12:30,479
then deciding to roll back or not
is a huge expensive process. As you

928
01:12:30,560 --> 01:12:32,920
said, you know, feature flags
can work here, although that's again a

929
01:12:32,960 --> 01:12:38,000
switch. So really parallel execution has
been the only thing I can think of

930
01:12:38,079 --> 01:12:42,199
that really made sense for us early
on. And how do you think about

931
01:12:42,279 --> 01:12:46,800
how that compares to like a canary
environment. I mean, this is sort

932
01:12:46,840 --> 01:12:49,720
of the same thing, like when
you if you have millions of requests that

933
01:12:49,760 --> 01:12:55,039
are that are coming into your service
all the time, then some number of

934
01:12:55,039 --> 01:12:59,399
people would be affected by any change, no matter how small. And if

935
01:12:59,399 --> 01:13:01,039
you have the budget, which is
you know, one of the key terms

936
01:13:01,039 --> 01:13:04,560
that we talk a lot about in
observability, then you know, go for

937
01:13:04,640 --> 01:13:06,800
it, right, you know you
don't need to take some extra steps.

938
01:13:08,079 --> 01:13:12,399
But if it's just a matter of
you know, changing a bunch of parameters

939
01:13:12,439 --> 01:13:15,359
somewhere and you want to see how
it validates it, especially when you have

940
01:13:15,399 --> 01:13:16,920
all the business logic on your side
already, right, you know you're not

941
01:13:17,000 --> 01:13:20,960
calling a third party service, you're
not doing some other things like if you

942
01:13:20,960 --> 01:13:25,800
can really test it all in your
service in production at runtime, why not

943
01:13:26,399 --> 01:13:31,960
do this to validate? Oh was
that hard to set up? I mean

944
01:13:32,000 --> 01:13:34,319
you really, like, you know, think of you're changing a method,

945
01:13:34,359 --> 01:13:38,159
you know, any any engineer out
there. You're changing the internals of some

946
01:13:38,239 --> 01:13:42,800
method. All you need to do
is duplicate that method down to a second

947
01:13:42,800 --> 01:13:45,920
block, call it V two and
have the caller execute both of those things,

948
01:13:45,079 --> 01:13:48,079
wrap the other one in a trycatch
block, and then validate the outputs

949
01:13:48,079 --> 01:13:51,239
in your code and if it doesn't
work, you know, log a message

950
01:13:51,399 --> 01:13:55,880
and that's it's not It's honestly not
a lot there. You can create some

951
01:13:55,920 --> 01:14:00,520
structure and around that with a framework
where you can wrap you know, parallel

952
01:14:00,520 --> 01:14:04,399
execution and then pass it into methods
and have it automatically do it. It's

953
01:14:04,439 --> 01:14:08,880
not a lot of extra overhead if
you feel like you're doing this very very

954
01:14:08,920 --> 01:14:12,199
often. I think this happens,
like one out of ten features that gets

955
01:14:12,199 --> 01:14:15,840
implemented goes in like this. It
also gives us some security around it,

956
01:14:15,960 --> 01:14:20,079
so I wouldn't say it's complicated.
I think a lot of engineers know that

957
01:14:20,760 --> 01:14:25,199
this is a great a way that
you can test and this helps I think

958
01:14:25,279 --> 01:14:28,680
junior engineers, you know, they
navigated this right, you know, throw

959
01:14:28,720 --> 01:14:31,479
a try catch around there. I'm
scared about this affecting production. I think

960
01:14:31,640 --> 01:14:35,720
bringing that fear back in it,
you know, helps us. You know,

961
01:14:35,760 --> 01:14:40,600
thinking that our code that we write
is invincible is the problem here?

962
01:14:41,039 --> 01:14:45,399
Yes? Yes, So can you
just share your screen for us? Warren,

963
01:14:47,199 --> 01:14:51,920
I'm kidding it's really boring right now. I mean, when I'm doing

964
01:14:51,920 --> 01:15:00,000
the podcast, I literally have a
notes page and my audio equipment uli up

965
01:15:00,079 --> 01:15:02,199
in running and that's it. There's
nothing else on my desktop right now.

966
01:15:03,760 --> 01:15:10,039
We are. We're completely different.
I never I never closed windows, and

967
01:15:10,159 --> 01:15:16,279
never reboot. That's exactly what somebody
who wants to be proud would say.

968
01:15:16,279 --> 01:15:20,359
Well, I saw a meme yesterday
and it cracked me up, and it

969
01:15:20,439 --> 01:15:26,119
said, damn, I just accidentally
closed that browser tab that I've been meaning

970
01:15:26,119 --> 01:15:30,199
to read for two years. It
was very careful about my usage there that

971
01:15:30,239 --> 01:15:33,720
control shift T brings that up or
the previous window that I've closed, you

972
01:15:33,760 --> 01:15:36,359
start all my tab Yeah, for
sure. That. Like I was using

973
01:15:36,600 --> 01:15:42,399
x marks on Firefox back in the
day before they bought into it, because

974
01:15:42,439 --> 01:15:45,880
it's just so critical for me that
I can actually get that. The worst

975
01:15:45,880 --> 01:15:49,880
one I have is emails in my
inbox because when you click archive accidentally or

976
01:15:49,960 --> 01:15:55,159
delete like at that email was months
old. Good luck finding what that was.

977
01:15:58,359 --> 01:16:03,279
We have a security polow see that
automatically deletes our emails after a specified

978
01:16:03,279 --> 01:16:15,680
period of time. So email email
management is automated for me. Cool.

979
01:16:15,720 --> 01:16:25,399
So what well on critosis, have
you found an area where it's not a

980
01:16:25,399 --> 01:16:32,239
good fit. Yes, definitely crotosis
with where we are right now, if

981
01:16:32,439 --> 01:16:40,720
you are able to very adequately express
your system with a doctor composed file or

982
01:16:40,760 --> 01:16:45,760
a HELM chart with a relatively small
helm chart in the sense of like your

983
01:16:45,800 --> 01:16:47,319
helptot rebo doesn't have some dozen lines
in it, it's got like, you

984
01:16:47,319 --> 01:16:53,560
know, the same number of configuration
settings. If you have those two things,

985
01:16:53,640 --> 01:16:56,119
you're you're doing great. Krotosis is
not going to be great for you.

986
01:16:56,439 --> 01:17:00,600
It's the situation where those things have
to be augmented by the seven thousand

987
01:17:00,600 --> 01:17:04,000
of my mast script, the bunch
of different documentations on how to configure these

988
01:17:04,039 --> 01:17:09,039
things. If your doctor post file
has or your home charts have a bunch

989
01:17:09,079 --> 01:17:13,640
of lines commented out that say if
you want this behavior, please uncomment these

990
01:17:13,680 --> 01:17:16,399
lines, and it's all over the
place. That's where that's where you should

991
01:17:16,399 --> 01:17:19,359
think a look at prtosis uh,
and that's where we're going to provide a

992
01:17:19,359 --> 01:17:30,520
lot of value at the moment.
Right on excellent. So you mentioned doctor

993
01:17:30,560 --> 01:17:36,560
compose and HELM So I think I
can safely infer there that you're a big

994
01:17:36,600 --> 01:17:44,359
favor of containerization. Yes. Is
that specific or is that a result of

995
01:17:45,239 --> 01:17:47,760
like this is what you're you learned
from your customers or from the beginning,

996
01:17:47,760 --> 01:17:54,359
where you just like containers all the
way I mean, I would love to

997
01:17:54,399 --> 01:17:58,399
say it was entirely deliberate. Uh, and from like the very beginning.

998
01:17:59,079 --> 01:18:01,920
Definitely in the beginning it was like, hey, this is the standard that

999
01:18:01,920 --> 01:18:05,760
people are using to ship these things. I think it's a great abstraction.

1000
01:18:05,880 --> 01:18:11,840
I think containers. I think it's
wonderful to embrace the world of containers.

1001
01:18:11,880 --> 01:18:16,039
I think right now, container right
now, When people think about containers,

1002
01:18:16,039 --> 01:18:21,159
they often also think about the Docker
run time, the Docker build process,

1003
01:18:21,600 --> 01:18:26,000
and so like the of a container, the thing about not only like the

1004
01:18:26,039 --> 01:18:29,199
idea of the container, but also
like what does it look and feel like

1005
01:18:29,239 --> 01:18:32,279
when you build everything inside of the
Docker file, which is like separate from

1006
01:18:32,359 --> 01:18:36,399
having a container. So like what
I love is the container abstraction. I

1007
01:18:36,439 --> 01:18:41,560
don't so much love. It's very
easy to bloat the Docker file. It's

1008
01:18:41,600 --> 01:18:45,000
very easy to blow the container so
that it's like significantly bigger than it really

1009
01:18:45,079 --> 01:18:47,600
needs to be. There's a lot
of stuff to improve there. But when

1010
01:18:47,600 --> 01:18:51,000
it comes to like how you wrap
things up, the ideal world is like

1011
01:18:51,159 --> 01:18:59,239
containers as lightweight as possible, wonderful. The only issue is it is pretty

1012
01:19:00,039 --> 01:19:01,800
It is pretty annoying to get in
and out of containers and deal with the

1013
01:19:02,600 --> 01:19:06,159
like is this a port that is
exposed inside of the container, or is

1014
01:19:06,199 --> 01:19:09,800
it on my laptop or is it
like like which port? Is it?

1015
01:19:09,840 --> 01:19:13,840
At what level of networking? It's
pretty annoying, and it can be.

1016
01:19:13,960 --> 01:19:18,560
It can be also difficult to map
between filesystems and different containers to your own

1017
01:19:18,600 --> 01:19:21,920
file system. When you have mounts
mapped or not, that stuff can be

1018
01:19:21,960 --> 01:19:26,359
tricky. A lot of the abstraction
that we provide in our like run time,

1019
01:19:27,439 --> 01:19:30,359
tries to make that stuff better when
it comes to interacting with the def

1020
01:19:30,479 --> 01:19:34,800
environment. But I think really the
the reaction folks have against containers when it

1021
01:19:34,840 --> 01:19:39,279
comes to environments, a lot of
the time it comes from them being perceived

1022
01:19:39,319 --> 01:19:43,560
as heavyweight, which is often true. If you're running a doctor it often

1023
01:19:43,640 --> 01:19:45,880
is true. It's like there's a
lot of there's a lot of stuff going

1024
01:19:45,920 --> 01:19:48,119
on there. But a lot of
the time it doesn't have to be heavyweight.

1025
01:19:48,159 --> 01:19:54,119
It's just that the way we build
containers now tends to lean towards a

1026
01:19:54,159 --> 01:19:58,399
more heavyweight approach. No, I
think you nailed it there. Realistically,

1027
01:19:58,439 --> 01:20:01,720
it's like this, there's no hobb
load automation. When you make a code

1028
01:20:01,800 --> 01:20:06,239
change, how do you get the
necessary like auto compiling and run like you

1029
01:20:06,359 --> 01:20:11,399
just want to sort of debug it
as soon as possible. I think it's

1030
01:20:11,439 --> 01:20:14,840
a small like that's suset of things. And then there's this other set which

1031
01:20:14,880 --> 01:20:17,399
you for sure mentioned like what if
you're changing something at the container level right

1032
01:20:17,399 --> 01:20:21,520
like to You're definitely going to make
some changes and you're going to need to

1033
01:20:21,560 --> 01:20:25,279
test that in some way. I
don't think I know anyone that wrote a

1034
01:20:25,279 --> 01:20:29,119
Docker file and it was right the
first time or the second time or the

1035
01:20:29,159 --> 01:20:32,399
third time. I did learn about
this thing that you can like now make

1036
01:20:32,399 --> 01:20:35,960
a builder out of it, and
that can really help reduce the size of

1037
01:20:36,000 --> 01:20:41,720
your containers by not storing ninety percent
of what's necessary there. Like, I

1038
01:20:41,760 --> 01:20:45,560
still feel like that's really esoteric,
and most people are not aware that the

1039
01:20:45,560 --> 01:20:49,600
overcon container initiative supports that sort of
reduction. Are you talking about like using

1040
01:20:50,560 --> 01:20:55,439
like a multi stage Docker build where
you run your build process and then store

1041
01:20:55,479 --> 01:20:58,680
that as an artifact and then launch
a new Docker in it. It was

1042
01:20:58,760 --> 01:21:03,119
actually in the door one thing,
right for sure? In multi stage No,

1043
01:21:03,159 --> 01:21:08,840
there's actually a in the darker file
you can specify multiple images and one

1044
01:21:08,840 --> 01:21:13,079
of the images you can say is
this is my builder, and then have

1045
01:21:13,199 --> 01:21:15,960
the tools in there, and then
another image lower down that says, this

1046
01:21:15,039 --> 01:21:19,439
is the actual source image I want, because there's just so many run time

1047
01:21:19,560 --> 01:21:25,520
versus execution time problems that come up, where in the previous world we were

1048
01:21:25,520 --> 01:21:29,279
in a state where they're the same
thing and it's really expensive, but now

1049
01:21:29,359 --> 01:21:32,640
we're much better off. Like you
can get rust containers down from gigabytes to

1050
01:21:33,279 --> 01:21:39,119
really megabytes because the builder is big
and expensive, but the actual built image

1051
01:21:39,199 --> 01:21:43,399
is just pretty much a wrap binary. Yeah, that's the crazy thing,

1052
01:21:44,800 --> 01:21:48,880
especially in the area of like build
optimization. There's so much low hanging fruit

1053
01:21:49,399 --> 01:21:55,600
everywhere, but it it's all different
low hanging fruit, and you have to,

1054
01:21:55,680 --> 01:21:59,039
like you have to do some degree
of looking into it to figure out

1055
01:21:59,079 --> 01:22:00,800
that, oh, you can grab
the thing and put it in here,

1056
01:22:00,840 --> 01:22:02,680
and all of a sudden, I
forgive your buys the megabytes. But it's

1057
01:22:02,800 --> 01:22:06,319
so common to hear that, oh, I brought my build time down from

1058
01:22:06,399 --> 01:22:10,279
ninety seconds to nine seconds, or
I brought it down from five minutes to

1059
01:22:10,319 --> 01:22:13,960
one minute, and it's these are
massive improvements and you build all the time,

1060
01:22:14,319 --> 01:22:18,239
but they're right there in the city
there, but there's no it's not

1061
01:22:18,359 --> 01:22:23,319
often connected, right, Yeah.
I think a lot of For a lot

1062
01:22:23,399 --> 01:22:29,880
of people, there's just they've got
they've done the build process and it took

1063
01:22:29,920 --> 01:22:32,319
a long time, and they just
like, oh, I guess that's the

1064
01:22:32,359 --> 01:22:36,640
way it is, not realizing that
there are a lot of knobs you can

1065
01:22:36,680 --> 01:22:43,239
tweak to change that. Yeah,
And I think I mean, depending on

1066
01:22:43,279 --> 01:22:45,560
where you are on the build process. If you're if you're on your laptop

1067
01:22:45,600 --> 01:22:46,840
in your building and you want it
to be like super fast, and if

1068
01:22:46,880 --> 01:22:49,880
it's two minutes, it's going to
be a bit annoying. But if you're

1069
01:22:50,359 --> 01:22:53,760
if you're merging your pr and you're
waiting for it to go to a staging

1070
01:22:53,840 --> 01:22:57,800
environment, like if it's if it's
five minutes, you're probably okay with it.

1071
01:22:57,840 --> 01:23:00,319
You're probably like, Okay, this
is fine, and try to get

1072
01:23:00,319 --> 01:23:02,399
it from five to two minutes.
It's probably like not that big of a

1073
01:23:02,439 --> 01:23:06,479
deal for you because you don't do
it that often. So then like when

1074
01:23:06,479 --> 01:23:11,880
would you ever want to optimize that? But across all the debts and all

1075
01:23:11,920 --> 01:23:14,479
the mergers, that's it adds up. It's a lot of time, but

1076
01:23:14,640 --> 01:23:21,600
you could have gone faster for sure, well, sweet, is there anything

1077
01:23:21,600 --> 01:23:27,279
else we should talk about? I
feel like we kind of just ran the

1078
01:23:27,319 --> 01:23:34,720
gauntlet here we did. So if
folks are interested in learning more about Kurtosis,

1079
01:23:36,720 --> 01:23:44,039
what where do y'all hang out at
online discord? Well, first of

1080
01:23:44,079 --> 01:23:45,640
all, I recommend going to our
GitHub. The GitHub is the place for

1081
01:23:45,680 --> 01:23:48,920
you enter. So like GitHub,
it's got our code, it's got our

1082
01:23:48,960 --> 01:23:51,600
docks there, it's got the reading
me. So if you want to learn

1083
01:23:51,640 --> 01:23:56,119
my business, go right to the
getthub. And then if you want to

1084
01:23:56,199 --> 01:24:00,199
chat with us, we're always chatting
with the community, So join our discord

1085
01:24:00,560 --> 01:24:03,279
and we'll be there. You can
DM us, you can chat with us

1086
01:24:03,279 --> 01:24:08,359
in the general channel. It's fun
right on cool. Yeah, and I've

1087
01:24:08,359 --> 01:24:12,199
been chatting with y'all a little bit. Y'all are you really are super active

1088
01:24:12,239 --> 01:24:18,039
and super responsive online, which is
nice and it's refreshing to see. It's

1089
01:24:18,079 --> 01:24:23,319
the fun part. The fun part
is chatting with the users. See,

1090
01:24:23,359 --> 01:24:30,359
that's just so anti tech right there, Like, like I would love tech

1091
01:24:30,399 --> 01:24:33,199
if it weren't for the damn users. I feel like there's the difference of

1092
01:24:33,279 --> 01:24:39,319
you know, the outward appearance versus
what you're thinking when they're actually saying things.

1093
01:24:40,640 --> 01:24:43,319
And it's also the users that you're
talking with too. You know,

1094
01:24:43,319 --> 01:24:48,720
when you're talking with with people who
are doing the same thing, you know,

1095
01:24:49,159 --> 01:24:53,600
whether they're software engineers or whatever.
I think that's a different experience than

1096
01:24:54,479 --> 01:25:00,399
trying to talk mod through getting her
dial up modem connected and clearing out her

1097
01:25:00,399 --> 01:25:02,279
Internet Explorer cash or whatever. Yeah, I mean, I think that's definitely

1098
01:25:02,279 --> 01:25:08,800
difference between support right for something and
real engagement with your customers. Yeah,

1099
01:25:08,800 --> 01:25:12,039
for sure. You know, I
usually come away from them like, oh,

1100
01:25:12,199 --> 01:25:15,159
no, we should have that feature. I can't believe we didn't think

1101
01:25:15,159 --> 01:25:19,159
of it. I mean, one
of the fun things about having a developer

1102
01:25:19,239 --> 01:25:26,239
tool is it's buy engineers for engineers. There's a lot of empathetic connection across

1103
01:25:26,279 --> 01:25:31,319
the the product user barrier, where
maybe if we weren't the exact same profile,

1104
01:25:31,600 --> 01:25:36,600
you'd have to you have to bridge
the gap somehow. Well. You

1105
01:25:36,640 --> 01:25:41,279
know, I've heard many times that
that's what makes us successful startup is whenever

1106
01:25:41,319 --> 01:25:46,439
you're solving a problem that you personally
have. I totally agree with that.

1107
01:25:47,119 --> 01:25:50,640
Yeah. Cool, Well, Gayalen, thanks for being on the show man.

1108
01:25:50,680 --> 01:25:57,279
This has been a really cool talk. Thanks for having me. Cool

1109
01:25:57,319 --> 01:26:00,000
Warren thanks for joining me, welcome
back, Glad to have you back,

1110
01:26:00,079 --> 01:26:03,039
kind of show, glad to be
here. And for all of the listeners,

1111
01:26:03,279 --> 01:26:06,560
thank you for listening. Really appreciate
it. Hope you found it useful.

1112
01:26:06,840 --> 01:26:11,840
Oh right, yeah, yeah,
picks, all right, you brought

1113
01:26:11,880 --> 01:26:14,840
it up. Kick us off oring. I just you know, I know

1114
01:26:14,920 --> 01:26:17,880
some listeners were going to be mad
if you know they didn't hear on the

1115
01:26:17,920 --> 01:26:28,319
report. Yeah, So mine this
week is Turn the Ship Around by David

1116
01:26:28,399 --> 01:26:32,399
Marque, which is a book that, irrelevant of your experience and technology or

1117
01:26:32,760 --> 01:26:40,159
why you're focused on either core engineering
or somewhere else, can really help take

1118
01:26:40,279 --> 01:26:44,479
your understanding of how to work effectively
to the next level. It's really about

1119
01:26:45,159 --> 01:26:48,399
leadership in every way, starting from
the bottom all the way up, and

1120
01:26:48,439 --> 01:26:51,439
how to think about it. And
I think it's really it's really interesting to

1121
01:26:51,479 --> 01:26:55,520
read, and if you feel like
you can't commit to doing that, you

1122
01:26:55,560 --> 01:27:00,840
can find ten minutes to watch the
shortened video online. It's something of greatness.

1123
01:27:00,920 --> 01:27:03,439
I can never win the name of
the title. It's a YouTube video.

1124
01:27:03,479 --> 01:27:08,640
You can find it, which is
it's a hilarious and it's really good

1125
01:27:08,640 --> 01:27:15,239
content right on. Awesome Galen putting
on the spot. What'd you bring for

1126
01:27:15,279 --> 01:27:18,720
a pick my pig? It can
be it can be a music Oh for

1127
01:27:18,760 --> 01:27:25,239
sure. Yeah, so I have
been very into jelly Roll lately. I

1128
01:27:25,239 --> 01:27:29,000
don't know if you guys know jelly
roll country singer. Oh yeah, yeah,

1129
01:27:29,039 --> 01:27:33,760
yeah, he's he's uh the country
post Malone. Yes, yes,

1130
01:27:34,680 --> 01:27:38,079
that's hilarious. I've never heard it
before, but as soon as you said

1131
01:27:38,119 --> 01:27:42,119
it, I knew that you were
thinking of the right person. Yes,

1132
01:27:42,239 --> 01:27:46,000
Country post Malone. Jelly Roll is
amazing. I've just been It's rare that

1133
01:27:46,039 --> 01:27:49,720
I find an artist. And then
I just listened to a dozen of his

1134
01:27:49,880 --> 01:27:54,119
songs on a regular basis. He
hit me with, hit me with all

1135
01:27:54,199 --> 01:27:57,000
of them. So if you haven't
checked out jelly Roll, if you're into

1136
01:27:57,039 --> 01:28:00,079
country, if you're into hip hop, it's a really nice combination of both.

1137
01:28:00,359 --> 01:28:08,920
So for sure, Yeah, I
last year I stumbled across hardy and

1138
01:28:08,960 --> 01:28:15,840
which hardy is like country meets heavy
metal or or hard rock. And then

1139
01:28:15,840 --> 01:28:17,880
at some point he did a collaboration
with jelly Roll, which took me down

1140
01:28:17,880 --> 01:28:21,399
that path. Yeah, so I'll
agree with you on that one. That's

1141
01:28:21,399 --> 01:28:28,000
pretty cool and it's you know,
it's definitely not what you would expect from

1142
01:28:28,199 --> 01:28:33,680
country music. It's it's quite a
bit different. I'm not sure Hank william

1143
01:28:33,720 --> 01:28:41,920
Senior Senior would approve of it,
but here we are cool. So I

1144
01:28:42,000 --> 01:28:48,039
brought two picks this week. The
first one is going to totally destroy my

1145
01:28:48,920 --> 01:28:56,199
nerd credibility because I have never read
the book Dune or seen the movie Dune.

1146
01:28:56,720 --> 01:29:02,119
So I am starting to read the
book Dune, and dude, maybe

1147
01:29:02,239 --> 01:29:06,319
one of the best books I've ever
read. I mean, the things that

1148
01:29:06,319 --> 01:29:12,319
that Herbert does in the book is
just unmatched by other stories realistically, like

1149
01:29:12,359 --> 01:29:15,520
that you can tell us by the
number of new concepts and things that were

1150
01:29:15,560 --> 01:29:19,640
just made up. It's not like
a world that just easily imaginable, right,

1151
01:29:19,680 --> 01:29:23,399
And I'm looking forward to it and
I've I've heard from multiple people.

1152
01:29:23,840 --> 01:29:26,279
You know, it's a common saying, oh, the book's better than the

1153
01:29:26,279 --> 01:29:28,880
movie, but I've heard from from
multiple people like, no, No,

1154
01:29:29,520 --> 01:29:32,960
the book is better than the movie
on many, many levels. So I'm

1155
01:29:32,960 --> 01:29:36,760
looking forward, looking forward to getting
into that. The other pick I'm going

1156
01:29:36,840 --> 01:29:44,279
to bring up is I kind of
just lost I kind of fell out of

1157
01:29:44,319 --> 01:29:47,159
love with Chrome as my browser,
and I wanted to see what else was

1158
01:29:47,199 --> 01:29:51,439
out there, Chrome, It's not
you, it's me. I just felt

1159
01:29:51,479 --> 01:30:00,560
like to see other browsers across the
ARC Browser, and I think I completely

1160
01:30:00,640 --> 01:30:14,239
locked up here there you I did
here may be back. I got completely

1161
01:30:14,359 --> 01:30:20,680
kicked out for talking trash about Chrome, didn't I. I'm going to go

1162
01:30:20,720 --> 01:30:25,079
through with it anyway. The ARC
Browser has been really cool. I've been

1163
01:30:25,159 --> 01:30:29,039
using it for about a week now, and there's so many cool, subtle

1164
01:30:29,039 --> 01:30:33,760
little features that I didn't realize belonged
in a browser. Like my favorite one

1165
01:30:34,520 --> 01:30:41,039
is when you're in a Google meet, you know, and you navigate off

1166
01:30:41,079 --> 01:30:45,800
to another tab, and then somebody
asked you a question. You can never

1167
01:30:45,920 --> 01:30:48,680
find which tab is the one for
meat, right, So then you got

1168
01:30:48,680 --> 01:30:51,479
this awkward silence and you gotta admit
that you were screwing off and not paying

1169
01:30:51,479 --> 01:30:56,199
attention to the meeting. Well,
with the ARC Browser, when you leave

1170
01:30:56,279 --> 01:31:00,000
the tab, it just puts a
little thumbnail in the corner of your screw

1171
01:31:00,560 --> 01:31:04,239
with that video meet still going.
So when you never have to go back

1172
01:31:04,239 --> 01:31:06,439
to it, it's just down there
in the corner, you click into it

1173
01:31:06,479 --> 01:31:09,640
and it pops you right back up. And I just thought that was super

1174
01:31:09,640 --> 01:31:15,560
cool. So that's my second pick
for the week, and with that,

1175
01:31:16,399 --> 01:31:19,640
I think we're done here. Galen. Thanks again, Ben, this has

1176
01:31:19,680 --> 01:31:23,760
been pretty cool. I've enjoyed it. I hope the listeners enjoyed it.

1177
01:31:23,800 --> 01:31:29,359
Thank you for listening. We appreciate
your support, and yeah, I hope

1178
01:31:29,439 --> 01:31:30,239
see you all next week.