1
00:00:08,080 --> 00:00:11,560
Speaker 1: And we're live. Welcome back, everyone to another episode of

2
00:00:11,720 --> 00:00:16,079
Adventures in DevOps and I have today no co host,

3
00:00:16,239 --> 00:00:20,399
so I'm flying solo for the what looks like foreseeable future.

4
00:00:21,079 --> 00:00:24,280
So as consolation, I have a fact. There's a great

5
00:00:24,320 --> 00:00:28,640
white paper by Portainer talking about the operational overhead of Kubernetes,

6
00:00:28,719 --> 00:00:31,600
and so I'll link that in the talk description below

7
00:00:31,640 --> 00:00:34,000
after the episode, so everyone get at it. Basically, the

8
00:00:34,359 --> 00:00:37,159
TLDR is you're basically going to be spending an extra

9
00:00:37,280 --> 00:00:40,200
million dollars a year just because you decided to pick

10
00:00:40,399 --> 00:00:44,520
one orchestration platform over another one. More on that maybe later,

11
00:00:44,640 --> 00:00:47,200
but let's get to the point of the episode. I've

12
00:00:47,240 --> 00:00:50,200
got and I'm really excited today to have Andrew Morland,

13
00:00:50,280 --> 00:00:53,039
the co founder of Chalk, with us. So welcome to

14
00:00:53,079 --> 00:00:53,439
the show.

15
00:00:53,560 --> 00:00:55,280
Speaker 2: Thanks Wan, it's great to be here. And I have

16
00:00:55,359 --> 00:00:58,079
to say I do agree A million dollars sounds about right.

17
00:00:58,159 --> 00:00:59,759
Speaker 1: It sounds like a lot of money, I think, But

18
00:00:59,799 --> 00:01:02,840
if we realize that's pretty much just five engineers, Oh well,

19
00:01:02,960 --> 00:01:05,400
I actually totally get that, but I think a lot

20
00:01:05,439 --> 00:01:07,920
of people throw that under And it's interesting because if

21
00:01:07,959 --> 00:01:10,799
I understand correctly, Chuck is a fairly recent company. Therefore,

22
00:01:10,840 --> 00:01:12,439
maybe it's okay to call it a startup, and I

23
00:01:12,480 --> 00:01:14,799
can wonder whether or not you've had to actually make

24
00:01:14,799 --> 00:01:17,040
a decision on whether or not Kubernetes will be part

25
00:01:17,079 --> 00:01:19,439
of the tech stackt that you've decided to run with.

26
00:01:19,599 --> 00:01:22,599
Speaker 2: Yeah, so we do deploy on Kubernetes, and we can

27
00:01:22,640 --> 00:01:24,840
maybe talk about some of the details later. We run

28
00:01:24,879 --> 00:01:27,840
on Kubernet's clusters in our customer clouds. We did decide

29
00:01:27,920 --> 00:01:29,879
that it made sense to use. I think one of

30
00:01:29,879 --> 00:01:31,719
the things which pushed us in that direction is we

31
00:01:32,000 --> 00:01:33,840
since we deploy into customer clou accounts, we have to

32
00:01:33,840 --> 00:01:36,760
deal with multiple different kinds of clouds. So for us,

33
00:01:36,799 --> 00:01:40,120
Kubernetes is kind of a normalizing layer across the specific

34
00:01:40,120 --> 00:01:41,640
details of easy two or GCE.

35
00:01:42,239 --> 00:01:44,359
Speaker 1: Seeah. I think that's a really intelligent way of putting

36
00:01:44,400 --> 00:01:45,799
out because a lot of times I hear we need

37
00:01:45,840 --> 00:01:48,840
to use Kubernetes because either we need to pad our resumes,

38
00:01:49,239 --> 00:01:51,959
or more likely we want to be cloud agnostic, even

39
00:01:52,000 --> 00:01:54,439
though there's no business reason to do so. But what

40
00:01:54,480 --> 00:01:57,159
I heard is we actually have a business justification for

41
00:01:57,200 --> 00:01:59,359
being cloud agnostic, and you're doing the ridiculous thing of

42
00:01:59,439 --> 00:02:02,519
if I've got the right deploying to your customer's cloud

43
00:02:02,840 --> 00:02:06,519
environments directly. That doesn't sound like something I ever want

44
00:02:06,560 --> 00:02:06,840
to do.

45
00:02:07,159 --> 00:02:09,439
Speaker 2: Yeah, it wasn't something that we wanted to do initially

46
00:02:09,479 --> 00:02:11,960
either or that we intended to do. We started out

47
00:02:12,000 --> 00:02:15,960
as like effectively trying to present a hosted SaaS, So

48
00:02:16,000 --> 00:02:18,000
we wanted people to just come onto our website kind

49
00:02:18,000 --> 00:02:20,680
of like for Seal or like you know, any of

50
00:02:20,719 --> 00:02:22,759
the other platform the service companies click sign up and

51
00:02:22,759 --> 00:02:24,759
then like deploy their software and move on with their lives.

52
00:02:24,800 --> 00:02:26,599
But it turns out that when you work with more

53
00:02:26,639 --> 00:02:28,639
mature companies and you ask them for all their most

54
00:02:28,680 --> 00:02:32,719
sensitive data like their PII, their medical records, their financial records,

55
00:02:32,919 --> 00:02:35,800
they don't really want to hand that to a random

56
00:02:35,840 --> 00:02:38,479
seed stage staff startup. So it's kind of mandatory for

57
00:02:38,560 --> 00:02:41,400
us to start processing data in our customer cloud accounts a.

58
00:02:41,319 --> 00:02:43,800
Speaker 1: Pretty great justification, And we see the cost of paying

59
00:02:43,840 --> 00:02:47,000
cloud provider storage and even transferring between one account to

60
00:02:47,039 --> 00:02:50,639
another one increase over time as well, and so the

61
00:02:50,639 --> 00:02:53,800
ability to go into any sort of account and utilize

62
00:02:53,800 --> 00:02:55,240
the data where it is makes a lot of sense.

63
00:02:55,280 --> 00:02:57,719
Can I ask, like, what is TALK doing where it

64
00:02:57,800 --> 00:03:02,639
becomes the competitive advantage to be deploying into your customer's environment.

65
00:03:02,719 --> 00:03:04,319
Speaker 2: Yeah. So, like, the key thing that we do for

66
00:03:04,360 --> 00:03:07,240
our customers is help them process, transform serve data to

67
00:03:07,280 --> 00:03:10,120
machine learning models. So we don't have specific opinions about

68
00:03:10,159 --> 00:03:12,439
the exact transformations they want to do or what their

69
00:03:12,560 --> 00:03:15,840
models are for. Oftentimes it's things like fraud or logistic

70
00:03:15,960 --> 00:03:18,800
or logistics or recommendation systems or search or something like that.

71
00:03:18,919 --> 00:03:21,080
But because these models tend to be really core to

72
00:03:21,159 --> 00:03:23,879
their business or to their risk and fraud functions, they

73
00:03:23,919 --> 00:03:26,759
process all the most sensitive things that they have. There's

74
00:03:26,759 --> 00:03:28,639
a few different dimensions in which being in the customer

75
00:03:28,680 --> 00:03:31,719
account matters. One is maybe the most obvious, which is

76
00:03:31,719 --> 00:03:33,599
that their data doesn't leave their account. When we talk

77
00:03:33,680 --> 00:03:36,120
to companies, we always talk to their infrastructure and compliance

78
00:03:36,159 --> 00:03:38,360
and security teams, and it gives those teams a lot

79
00:03:38,400 --> 00:03:41,400
of comfort to know that the data stay is resident

80
00:03:41,560 --> 00:03:44,159
and that we don't actually have like we can't access

81
00:03:44,159 --> 00:03:46,280
it if they don't want us to say. Another maybe

82
00:03:46,319 --> 00:03:48,639
less obvious point is kind of what you touched on

83
00:03:48,680 --> 00:03:51,560
with the network transit. Like a there's obviously no network transit,

84
00:03:51,599 --> 00:03:53,719
but b it lowers the latency. So a lot of

85
00:03:53,719 --> 00:03:56,919
our customers care a lot about super low latency for

86
00:03:56,960 --> 00:03:58,120
these types of applications.

87
00:03:58,280 --> 00:04:01,280
Speaker 1: My understanding was, and this is how zero experience and

88
00:04:01,360 --> 00:04:04,960
either any sort of LM data modeling or any sort

89
00:04:04,960 --> 00:04:08,000
of machine learning as far as it goes for the

90
00:04:08,000 --> 00:04:11,280
actual data, say something like fraud, I wouldn't imagine necessarily

91
00:04:11,280 --> 00:04:14,080
the PII would be relevant in the fraud detection. But again,

92
00:04:14,360 --> 00:04:16,839
like I said, not the expert here, so I can

93
00:04:16,920 --> 00:04:19,560
imagine the flip side, it may not be directly relevant,

94
00:04:19,600 --> 00:04:22,839
but super difficult to separate it out to be able

95
00:04:22,839 --> 00:04:24,759
to hand over only the relevant pieces.

96
00:04:24,959 --> 00:04:27,000
Speaker 2: Yeah, it's kind of i'd saying both. So some of

97
00:04:27,000 --> 00:04:29,920
the companies who work with are literally about the PII.

98
00:04:30,120 --> 00:04:33,040
So like one customer, so cure, they're kind of their

99
00:04:33,079 --> 00:04:36,120
whole business is is this particular set of PII Actually

100
00:04:36,120 --> 00:04:38,120
the PII for a real person if you look at

101
00:04:38,319 --> 00:04:40,480
their website, like their API is literally I mean not

102
00:04:40,800 --> 00:04:43,959
literally but close enough is basically post US and social

103
00:04:44,000 --> 00:04:46,920
security number, phone number, email address, and address and we'll

104
00:04:46,920 --> 00:04:49,000
give you back a yep, that's a person or not score.

105
00:04:49,360 --> 00:04:51,279
So as you can imagine, that's pretty sensitive.

106
00:04:51,600 --> 00:04:54,680
Speaker 1: Yeah, it's always interesting being in that position where it's

107
00:04:54,720 --> 00:04:56,759
almost like you don't know what the data is and

108
00:04:56,800 --> 00:04:59,560
you're in a way praying that it's not sensitive. But

109
00:04:59,680 --> 00:05:01,920
then you spin up an API where the answer is

110
00:05:01,920 --> 00:05:04,079
is it sensitive data? And the answer like, you're getting

111
00:05:04,120 --> 00:05:05,160
sensitive data and there for.

112
00:05:05,079 --> 00:05:08,439
Speaker 2: Sure, yes, yeah, it's it's it's almost like nuclear waste.

113
00:05:08,600 --> 00:05:10,360
You really don't want to have to process it. But

114
00:05:10,480 --> 00:05:12,920
in this in particularly in fintech, you often can avoid it.

115
00:05:12,959 --> 00:05:15,120
Speaker 1: Share with me here, if you've working with these fraud

116
00:05:15,160 --> 00:05:17,199
detection systems, you must know all the secrets on how

117
00:05:17,240 --> 00:05:18,399
to evade them.

118
00:05:18,480 --> 00:05:22,360
Speaker 2: That well, I think part of the part of the

119
00:05:22,360 --> 00:05:24,199
reason why we deploy into the cloud accounts is that

120
00:05:24,199 --> 00:05:25,759
I also don't get to see the code the customers

121
00:05:25,800 --> 00:05:28,879
are running oftentimes. I mean we talk about it. We

122
00:05:29,199 --> 00:05:31,160
obviously very hands on with the people we work with.

123
00:05:31,399 --> 00:05:33,399
One i'd say ideally there is no way to evade

124
00:05:33,399 --> 00:05:36,000
these systems, and two I don't get to know all

125
00:05:36,000 --> 00:05:37,680
the details of how they work. But I mean there's

126
00:05:37,680 --> 00:05:40,040
some obvious things like probably anyone couldn't do it. Like

127
00:05:40,439 --> 00:05:42,480
it turns out, identity theft does work if you're using

128
00:05:42,519 --> 00:05:44,600
a real person's identity. It's a lot harder to you know,

129
00:05:44,639 --> 00:05:47,000
detect that. You know, it's kind of all the obvious

130
00:05:47,040 --> 00:05:50,240
things too. Like you if you look at some of

131
00:05:50,279 --> 00:05:52,800
the other customers we work with, like Varasol, et cetera.

132
00:05:53,000 --> 00:05:54,600
Just on the on the homepage of the website talk

133
00:05:54,600 --> 00:05:57,920
about device fingerprinting, So you know, people who are trying

134
00:05:57,959 --> 00:05:59,959
to commit fraud go to great lengths to try to

135
00:06:00,040 --> 00:06:02,439
avoid these systems, and that mostly involves pretending to be

136
00:06:02,480 --> 00:06:04,439
real people for very long periods of time so that

137
00:06:04,480 --> 00:06:06,560
they can kind of later on flip and become malicious.

138
00:06:06,639 --> 00:06:09,920
Speaker 1: There's a parallel to what we see in advanced persistent

139
00:06:09,959 --> 00:06:14,800
threats on digital systems, where that attacker lives off the

140
00:06:14,839 --> 00:06:17,839
resources for I think it's up to nine months is

141
00:06:17,920 --> 00:06:21,600
usually the average time before actually deciding to actual trade resources.

142
00:06:21,720 --> 00:06:24,959
And I think the other metric numbers like thirteen months,

143
00:06:25,519 --> 00:06:29,279
where people start throwing away the log data at that

144
00:06:29,360 --> 00:06:31,639
point and so you don't even know what systems they

145
00:06:31,680 --> 00:06:33,600
had access to at that moment. I want to get

146
00:06:33,600 --> 00:06:36,000
back to the deploying into the customer accounts though, because

147
00:06:36,040 --> 00:06:38,399
it's something that I said that I've always tried to

148
00:06:38,439 --> 00:06:40,560
avoid and never wanted to do, and you're doing it,

149
00:06:40,600 --> 00:06:43,879
and so this must have come with either some risks

150
00:06:43,879 --> 00:06:46,560
involved or some challenges as far as actually getting the

151
00:06:46,560 --> 00:06:49,199
technology to work. Like, it's not just so simple even

152
00:06:49,240 --> 00:06:53,360
in AWS, getting the access to the individual accounts hand

153
00:06:53,360 --> 00:06:58,199
over with external account IDs or identifiers, i AM rules,

154
00:06:58,199 --> 00:06:58,600
et cetera.

155
00:06:58,759 --> 00:07:01,279
Speaker 2: I would say there's like a lot of things that

156
00:07:01,319 --> 00:07:03,639
are hard about it. Some are obvious, some are not obvious.

157
00:07:04,079 --> 00:07:05,879
The obvious ones are like, Okay, how do you actually

158
00:07:05,920 --> 00:07:08,399
write down all the im permissions that your application needs?

159
00:07:08,480 --> 00:07:11,920
Like who knows? Right? So the answer was kind of like, well,

160
00:07:11,920 --> 00:07:14,600
what scope it to zero, build a tool to automatically

161
00:07:14,639 --> 00:07:18,519
deploy it, and then just loop for days until we

162
00:07:18,560 --> 00:07:21,560
finally find the minimal set. But there's more subtle things too,

163
00:07:21,720 --> 00:07:24,079
like it turns out that in all these cloud platforms

164
00:07:24,120 --> 00:07:26,519
there are global policies you can apply across all the accounts.

165
00:07:26,560 --> 00:07:28,920
And of course, like really savvy infrastructure people are out

166
00:07:28,920 --> 00:07:31,639
there secretly saying things like oh, you can't transitively assume

167
00:07:31,639 --> 00:07:34,279
i AM rules, But we have no way of knowing

168
00:07:34,279 --> 00:07:36,079
whether these policies are applied, and now do the people

169
00:07:36,079 --> 00:07:38,800
we're working with these companies. So even if you control

170
00:07:38,839 --> 00:07:43,319
all the configuration, right, you're still host. But there's all

171
00:07:43,319 --> 00:07:45,879
sorts of other things that go wrong too. You mentioned

172
00:07:45,879 --> 00:07:48,319
again earlier the network transit. It turns out that there's

173
00:07:48,360 --> 00:07:51,399
like a known problem between GCP US East four or

174
00:07:51,439 --> 00:07:53,879
Native US US East one, for example, like the network

175
00:07:53,920 --> 00:07:56,399
pipe there is just full, so it is understood and

176
00:07:56,439 --> 00:07:59,319
expected that you'll suffer packet loss over that pipe fairly often.

177
00:08:00,000 --> 00:08:01,800
They actually have a whole product page which I didn't

178
00:08:01,839 --> 00:08:04,519
expect to find, where you can buy a specific dedicated

179
00:08:04,560 --> 00:08:08,439
link on that pipe to guarantee yourself bandwidth. But basically

180
00:08:08,439 --> 00:08:11,040
the cross clouds defends up being super wacking in new scenarios.

181
00:08:11,079 --> 00:08:13,120
Speaker 1: Also, that's actually really interesting. You just brought it up

182
00:08:13,120 --> 00:08:15,839
because cloud Flair just had a downtime that affected a

183
00:08:15,839 --> 00:08:18,199
whole bunch of customers, and I believe the same thing

184
00:08:18,319 --> 00:08:21,199
actually happened with It was about network congestion a lot

185
00:08:21,279 --> 00:08:24,360
from cloud Flare to their US East one in Ashburn,

186
00:08:24,439 --> 00:08:26,279
And I'm like, I have no idea what's going on there,

187
00:08:26,319 --> 00:08:28,199
And you've said that it's like the second time in

188
00:08:28,600 --> 00:08:30,759
a month that I've I've heard about this, and now

189
00:08:30,800 --> 00:08:32,159
I start getting really concerned.

190
00:08:32,200 --> 00:08:33,759
Speaker 2: I think they're running out of internet in Virginia. It

191
00:08:33,759 --> 00:08:34,919
seems like the problem.

192
00:08:34,639 --> 00:08:37,080
Speaker 1: At this point, It's like does that data center still exist?

193
00:08:37,080 --> 00:08:39,399
And I think the meme is for sure, like why

194
00:08:39,399 --> 00:08:41,759
are you doing anything in US East one? And the

195
00:08:41,799 --> 00:08:45,080
answer is because our customers are exactly how any choice?

196
00:08:45,399 --> 00:08:47,000
Speaker 2: Yeah, I have no choice? Unfortunately.

197
00:08:48,080 --> 00:08:52,120
Speaker 1: Wow, Okay, you know, the network going down was not

198
00:08:52,360 --> 00:08:55,240
a top three of what I would have expected to

199
00:08:55,279 --> 00:08:57,279
have to deal with, you know, as a startup. I

200
00:08:57,279 --> 00:09:00,639
mean we're almost ten years old now, and I still

201
00:09:00,679 --> 00:09:04,840
remember every single challenge we went through. Technically deploying the

202
00:09:04,840 --> 00:09:07,360
customer accounts though, is definitely not one of the things

203
00:09:07,519 --> 00:09:10,480
on the list. But yeah, crosscount im roles and service

204
00:09:10,519 --> 00:09:12,399
control policies. It used to be much worse, right, used

205
00:09:12,399 --> 00:09:14,799
to get to deny and you had no idea in AWS.

206
00:09:14,919 --> 00:09:17,600
Now at least it says, yeah, denied due to service

207
00:09:17,600 --> 00:09:20,759
control policy and your good good luck you know getting

208
00:09:20,759 --> 00:09:22,639
that changed. But at least you know you know what

209
00:09:22,679 --> 00:09:24,679
you're trying to avoid there. How do you deal with

210
00:09:24,960 --> 00:09:27,720
expanding the footprint that you need in your customer environment?

211
00:09:27,840 --> 00:09:31,600
So if additional permissions are required, is it a manual

212
00:09:31,639 --> 00:09:34,799
process to get them to basically update something, or is

213
00:09:34,879 --> 00:09:36,519
I can't imagine what the workflow would be.

214
00:09:36,720 --> 00:09:39,759
Speaker 2: Yeah, it's not my favorite thing. Yeah, it ends up

215
00:09:39,759 --> 00:09:42,159
being kind of a negotiation with every single customer. So

216
00:09:42,879 --> 00:09:45,720
I think fortunately these days, most of the im permissions

217
00:09:45,759 --> 00:09:48,879
are introspectable, like you can say do I have this

218
00:09:48,919 --> 00:09:51,320
permission and before you go and use it. So we

219
00:09:51,360 --> 00:09:53,039
try to write our software to do that sort of

220
00:09:53,039 --> 00:09:55,919
thing and then present like warnings on the dashboard about oh,

221
00:09:55,919 --> 00:09:58,840
we're missing some permissions. In general, it kind of means

222
00:09:58,879 --> 00:10:01,399
we can't use a lot of stand tooling, Like terraform

223
00:10:01,440 --> 00:10:03,639
doesn't work really well in this workflow because you know,

224
00:10:03,679 --> 00:10:05,799
a terraform's kind of awkward if you want to automate

225
00:10:05,840 --> 00:10:07,480
it as part of the sign up button flow. B

226
00:10:07,759 --> 00:10:11,240
terraform really does not like not having permissions for things.

227
00:10:11,639 --> 00:10:14,240
You'll end up in situations where the DAG can't totally resolve.

228
00:10:14,279 --> 00:10:15,960
I can't even plan, So we end up having to

229
00:10:16,000 --> 00:10:17,879
build a lot of the functionality that you would normally

230
00:10:17,919 --> 00:10:19,559
get out of the box and terraform for free kind

231
00:10:19,600 --> 00:10:22,039
of in our own go application which does the orchestration

232
00:10:22,120 --> 00:10:25,720
all these resources, so that we can recover from missing permissions.

233
00:10:25,759 --> 00:10:28,159
Speaker 1: I don't want to understate that It's just a simple

234
00:10:28,159 --> 00:10:30,600
thing like you write in some terraform which are open

235
00:10:30,639 --> 00:10:33,679
TOFU now to update a resource, and the first thing

236
00:10:33,720 --> 00:10:35,399
you do is you try to go get that resource

237
00:10:35,440 --> 00:10:37,000
and you get to deny there. So even if you

238
00:10:37,000 --> 00:10:39,759
have an update like the ability to create that resource,

239
00:10:39,919 --> 00:10:42,120
there is a hidden get there that you may not have.

240
00:10:42,200 --> 00:10:44,919
And AWS or the other cloud providers do change over time,

241
00:10:44,919 --> 00:10:47,919
and so that one resource can become two resources, sure can,

242
00:10:48,360 --> 00:10:52,960
and the open TOFU terraform whatever bloomy provider will change

243
00:10:53,039 --> 00:10:55,639
to make the two gets, and all of a sudden

244
00:10:55,679 --> 00:10:58,879
you would start running into two issues there. So yeah,

245
00:10:58,879 --> 00:11:01,480
I can. I can totally image. It's more of a

246
00:11:01,559 --> 00:11:04,000
feature flag flow where it's like, well, in order to

247
00:11:04,039 --> 00:11:06,240
get this feature, you have to first go through the

248
00:11:06,279 --> 00:11:08,799
security steps and then we'll validate it on our side,

249
00:11:08,840 --> 00:11:11,000
and once it's been validated, then we'll actually enable it

250
00:11:11,000 --> 00:11:11,399
for you.

251
00:11:11,480 --> 00:11:13,759
Speaker 2: But I'd say, like maybe the other answer too is

252
00:11:13,799 --> 00:11:16,200
we've been trying to pull more things in house over time,

253
00:11:16,279 --> 00:11:18,159
and in the beginning we're startup, we don't want to

254
00:11:18,159 --> 00:11:20,919
build everything on top to bottom. We want to delegate

255
00:11:20,960 --> 00:11:23,000
to the best in class solutions where we can as

256
00:11:23,000 --> 00:11:25,600
we have more resources and more experience and more opinions.

257
00:11:25,639 --> 00:11:27,879
We found that it's better to bring things like you know,

258
00:11:27,960 --> 00:11:30,480
log aggregation in house rather than relying on like cloud

259
00:11:30,480 --> 00:11:33,279
watch or cloud Trail whatever the Amazon products called, or

260
00:11:33,440 --> 00:11:36,679
stack driver logging. We'd rather use like ClickHouse logs or

261
00:11:36,679 --> 00:11:39,240
something like that that we control so that we can

262
00:11:39,279 --> 00:11:41,600
make sure that it's a consistent experience and also kind

263
00:11:41,639 --> 00:11:43,000
of lower our permission footprint.

264
00:11:43,039 --> 00:11:44,799
Speaker 1: I can really understand that, especially if you're getting the

265
00:11:44,840 --> 00:11:47,240
logs directly out from the customer account. It's one thing

266
00:11:47,360 --> 00:11:50,960
if your abuse account or GCP account is creating logs

267
00:11:50,960 --> 00:11:53,960
in the log solution itself, but if they're being generated

268
00:11:53,960 --> 00:11:56,600
in somewhere that you don't control primarily, and then you

269
00:11:56,679 --> 00:12:00,440
have to then write some sort of funnel to get

270
00:12:00,440 --> 00:12:02,279
from one place to another one, you're going to immediately

271
00:12:02,279 --> 00:12:04,240
start questioning where do we want this data to end

272
00:12:04,320 --> 00:12:06,480
up and then pay the least amount for it and

273
00:12:06,639 --> 00:12:09,320
have the most reasonable retention in that regard, Like what's

274
00:12:09,360 --> 00:12:12,080
the onboarding process? Is it manual? Is it? You know,

275
00:12:12,159 --> 00:12:13,840
do you take like a white glove approach each of

276
00:12:13,840 --> 00:12:15,840
these customers you said, you know much larger they do

277
00:12:15,919 --> 00:12:18,360
care about their data in a lot of ways, it's

278
00:12:18,360 --> 00:12:20,120
not going to be probably self sign up in a

279
00:12:20,120 --> 00:12:20,960
lot of scenarios.

280
00:12:21,080 --> 00:12:23,759
Speaker 2: Yeah, we are marching towards self sign up honestly, just

281
00:12:23,799 --> 00:12:26,519
to lower the operational burden of the White Glove version.

282
00:12:26,639 --> 00:12:28,480
You know, I don't think anybody really likes having a

283
00:12:28,519 --> 00:12:30,960
conversation about how to deplay software, so we would rather

284
00:12:31,000 --> 00:12:33,679
have a sign up form people can go through. Whether

285
00:12:33,759 --> 00:12:35,639
or not we actually make a publicly available as a

286
00:12:35,679 --> 00:12:38,320
separate question, because in some ways we're a remote code

287
00:12:38,320 --> 00:12:41,840
execution platform, so it is really challenging to make sure

288
00:12:41,919 --> 00:12:45,159
that our software is secure in an untrusted environment. But

289
00:12:45,720 --> 00:12:47,720
that aside. The way that we do it these days

290
00:12:47,799 --> 00:12:50,559
is we do a little handshake where we exchange i

291
00:12:50,639 --> 00:12:53,679
AM rolls or things like that, maybe some private keys

292
00:12:53,840 --> 00:12:55,960
and public keys, and then we do work with people.

293
00:12:56,000 --> 00:12:58,960
Probably for kind of cookie cutter deployments, it's usually a

294
00:12:58,960 --> 00:13:03,519
couple hours. For more involved deployments, maybe across multiple cloud

295
00:13:03,559 --> 00:13:07,000
regions or you know, or cloud accounts or cloud providers,

296
00:13:07,039 --> 00:13:09,519
even that can be you know, two three weeks to

297
00:13:09,639 --> 00:13:11,480
get stuff all the way world out and kind of

298
00:13:11,519 --> 00:13:12,559
certified for production.

299
00:13:12,799 --> 00:13:16,039
Speaker 1: When you ineavily get to the point where you've decided,

300
00:13:16,120 --> 00:13:18,399
after many years of running your company that it's time

301
00:13:18,399 --> 00:13:21,600
to deploy a second version of whatever you had. How

302
00:13:21,600 --> 00:13:23,240
does that get to the customer environment.

303
00:13:23,279 --> 00:13:24,720
Speaker 2: In the beginning, when we thought we were going to

304
00:13:24,720 --> 00:13:27,399
be like the versell for data and to today we're

305
00:13:27,440 --> 00:13:30,480
more enterprising. We used to just publish releases and people

306
00:13:30,519 --> 00:13:32,720
will pick them up kind of automatically. But it turns

307
00:13:32,759 --> 00:13:34,919
out again, you know, like large enterprises don't really like

308
00:13:34,960 --> 00:13:38,080
surprise software updates for lots of reasons, you know, both.

309
00:13:38,360 --> 00:13:40,639
You know, maybe we make a mistake, maybe we make

310
00:13:40,679 --> 00:13:42,559
a change that they don't like, maybe we make an

311
00:13:42,559 --> 00:13:45,879
improvement that they don't want. So we publish version releases

312
00:13:45,879 --> 00:13:48,279
of our software now, and we do ask that customers,

313
00:13:48,399 --> 00:13:50,679
you know, basically choose which version they'd like to execute.

314
00:13:50,759 --> 00:13:52,879
You know, obviously that has all the downsides of version

315
00:13:52,919 --> 00:13:55,000
software releases, i e. My mistakes live forever.

316
00:13:55,200 --> 00:13:57,639
Speaker 1: Yeah, as you have like some HELM charts or something

317
00:13:57,639 --> 00:14:00,799
that their customer is pulling in and running in their environment.

318
00:14:00,600 --> 00:14:03,240
Speaker 2: Yeah, we version the HELM charts for like the full

319
00:14:03,240 --> 00:14:06,519
metadata plane deployments when people host that themselves, we version

320
00:14:06,600 --> 00:14:09,120
like the underlying software version or SDKs. All those things

321
00:14:09,399 --> 00:14:09,840
are there.

322
00:14:09,919 --> 00:14:13,879
Speaker 1: Unexpected downsides here of going down this approach rather than

323
00:14:14,000 --> 00:14:16,519
you know, forced rollout or upgrades. Do you find that

324
00:14:16,559 --> 00:14:19,840
you're getting support scenarios for the older versions that are

325
00:14:19,919 --> 00:14:20,879
missing critical stuff.

326
00:14:21,080 --> 00:14:24,200
Speaker 2: Yeah, it's definitely a problem. I mean I always what

327
00:14:24,200 --> 00:14:25,960
I always tell people is, you know, I think my

328
00:14:26,080 --> 00:14:28,879
latest version is my best version, and it should have

329
00:14:28,960 --> 00:14:31,000
all of my my newest and best things in it.

330
00:14:31,879 --> 00:14:33,559
So if people are not on it, then you know,

331
00:14:33,799 --> 00:14:35,360
I think that they're missing some stuff I think they

332
00:14:35,360 --> 00:14:37,320
should have. So we do run into issues where we

333
00:14:37,639 --> 00:14:40,799
fix bugs, we make improvements, and they're important, like we

334
00:14:40,840 --> 00:14:43,679
add better load shedding for instance. And if someone's runn

335
00:14:43,679 --> 00:14:45,399
an old version that doesn't have the ability to like

336
00:14:45,440 --> 00:14:47,840
properly do dynamic rate limiting, they may run into a

337
00:14:47,840 --> 00:14:50,480
scenario where they get overwhelmed and end up having a

338
00:14:50,559 --> 00:14:52,759
latency you know, espalation. We do keep track to what

339
00:14:52,799 --> 00:14:55,360
versions people are running, and we do, like our kind

340
00:14:55,399 --> 00:14:57,120
of solutions team works with them to make sure they're

341
00:14:57,159 --> 00:14:59,879
updating regularly. We don't really want people to be stuck

342
00:15:00,080 --> 00:15:01,159
a year old version forever.

343
00:15:01,320 --> 00:15:01,960
Speaker 1: Are there though?

344
00:15:03,200 --> 00:15:05,919
Speaker 2: I think we had one person who was running January

345
00:15:06,000 --> 00:15:09,279
software until about four weeks ago but in general we

346
00:15:09,480 --> 00:15:11,200
most people are within two or three months.

347
00:15:11,639 --> 00:15:14,240
Speaker 1: I mean that's pretty recent, like even January, you've got

348
00:15:14,320 --> 00:15:18,120
same year there, twenty twenty five, without any previous hiccups.

349
00:15:18,440 --> 00:15:21,440
That's I would say. I'm a little surprised. I see

350
00:15:21,440 --> 00:15:24,879
a lot of companies running older versions of stuff that like,

351
00:15:25,000 --> 00:15:28,039
we have open source SDKs for every language for our product,

352
00:15:28,120 --> 00:15:31,759
and I can guarantee you there are some that have

353
00:15:31,840 --> 00:15:35,200
never upgraded since the moment they deployed their original version.

354
00:15:35,240 --> 00:15:37,759
And those early customers of ours, you know, that's almost

355
00:15:37,799 --> 00:15:40,399
ten years ago. So that's kind of scary in some regard.

356
00:15:40,679 --> 00:15:43,720
Speaker 2: Yeah, No, I think that like the main way we

357
00:15:44,240 --> 00:15:46,320
try to like engineer for that as we make sure

358
00:15:46,360 --> 00:15:49,840
that we have never intentionally made a backwards incompatible change.

359
00:15:50,679 --> 00:15:52,440
One day we may make a breaking change. We have

360
00:15:52,480 --> 00:15:54,559
made some mistakes in API that we love to correct,

361
00:15:54,799 --> 00:15:57,799
but we really want to prioritize that upgrade experience, so

362
00:15:57,840 --> 00:16:01,120
we'd never ever intentionally break any time the customer can't

363
00:16:01,240 --> 00:16:03,279
upgrade because of a regression, we make sure that's a

364
00:16:03,279 --> 00:16:05,080
p zero issue. So that's top of the heap for

365
00:16:05,159 --> 00:16:06,159
us in terms of triage.

366
00:16:06,360 --> 00:16:08,679
Speaker 1: I think I really like this perspective I went into

367
00:16:08,720 --> 00:16:11,039
it a little bit earlier in one of our episodes

368
00:16:11,759 --> 00:16:14,039
in the Off Showdown where we were discussing multi tenant

369
00:16:14,080 --> 00:16:16,639
versus single tenant solutions, and one of the things that

370
00:16:16,679 --> 00:16:21,279
came up was like, can you manage backwards compatible like forever?

371
00:16:21,399 --> 00:16:23,960
And because we have a SaaS, that's actually what we

372
00:16:24,000 --> 00:16:26,679
really try to focus on doing. Because I'm totally with you,

373
00:16:27,480 --> 00:16:30,080
it's so critical to be able to migrate from one

374
00:16:30,159 --> 00:16:32,120
version to another one. And I feel like one of

375
00:16:32,120 --> 00:16:35,240
the problems with like open source software or you know,

376
00:16:35,440 --> 00:16:38,120
you choose the version as a customer is that there's

377
00:16:38,159 --> 00:16:40,799
less discipline involved from the company. Right if you can

378
00:16:40,879 --> 00:16:43,879
break things and because you say, well, people don't have

379
00:16:43,919 --> 00:16:46,720
to upgrade, then you don't care about it. But you've

380
00:16:46,759 --> 00:16:50,759
taken a like really smart, mature approach there, which is, yes,

381
00:16:50,759 --> 00:16:52,960
it's not critical for any of our customers to do this,

382
00:16:53,080 --> 00:16:55,720
but it's so critical from us, and we know the

383
00:16:55,759 --> 00:16:57,879
pain that they're going to find at some point that

384
00:16:57,919 --> 00:16:59,759
if they can't do the upgrade, it's something that we

385
00:16:59,799 --> 00:17:01,639
have to have to fix. For sure.

386
00:17:01,879 --> 00:17:04,400
Speaker 2: Early moments in the company's history that kind of shape

387
00:17:04,440 --> 00:17:07,960
this philosophy. In addition just our opinions was we had

388
00:17:07,960 --> 00:17:10,759
a company churned to us from a competitor because they said, well,

389
00:17:10,799 --> 00:17:13,759
doing the upgrade for our competitors software is approximately as

390
00:17:13,759 --> 00:17:15,440
hard as adopting a new solution, so we might as

391
00:17:15,440 --> 00:17:17,759
well do a competitive email. I was like, well, okay,

392
00:17:17,799 --> 00:17:19,720
we're never going to give anyone that opportunity.

393
00:17:19,799 --> 00:17:22,079
Speaker 1: So many different things that we think about in our

394
00:17:22,119 --> 00:17:24,400
product when it comes to our customers and trying to

395
00:17:24,400 --> 00:17:27,000
avoid mental challenge of the churn, right, I mean, even

396
00:17:27,039 --> 00:17:28,640
if they're not actually going to do it, you know,

397
00:17:28,680 --> 00:17:31,680
having customers tell you the exact reason that they're churning

398
00:17:31,680 --> 00:17:34,200
and be like, well, let's bake that into the future.

399
00:17:34,240 --> 00:17:36,640
DNA is really is really something there. So before we

400
00:17:36,680 --> 00:17:39,559
got started in today's episode, there was a lot of

401
00:17:39,599 --> 00:17:46,079
technical problems, but many minutes of dealing with the current

402
00:17:46,079 --> 00:17:48,759
state of microphones and software. But one of the things

403
00:17:48,839 --> 00:17:51,960
you had mentioned that I completely glossed over, and I

404
00:17:52,000 --> 00:17:53,759
have to be honest, I'd never heard anyone say this

405
00:17:53,799 --> 00:17:58,680
before was we deal with time travel in data and

406
00:17:58,799 --> 00:18:00,839
I honestly have no idea, like I know, the idea

407
00:18:00,839 --> 00:18:03,759
of time travel like replaying events from some sort of

408
00:18:03,759 --> 00:18:06,640
event stream that's not exactly the model that you have

409
00:18:06,680 --> 00:18:07,160
to deal with.

410
00:18:07,359 --> 00:18:10,480
Speaker 2: Yeah, it's not totally dissimilar. So one of the things

411
00:18:10,519 --> 00:18:12,640
we think about a lot is bi temporal modeling, which

412
00:18:12,680 --> 00:18:15,279
is a really fancy way of saying that there's you know,

413
00:18:15,599 --> 00:18:18,079
at least two time stamps you actually care about in data.

414
00:18:18,400 --> 00:18:21,000
One is you know, when did a thing happen, But

415
00:18:21,079 --> 00:18:23,000
the other thing is when did you know that it happened.

416
00:18:23,039 --> 00:18:25,200
And that's a really important distinction to make when you're

417
00:18:25,200 --> 00:18:27,759
trying to train machine learning models. Thinking about a fraud model,

418
00:18:28,279 --> 00:18:31,319
someone commits fraud on your platform, maybe on like January first,

419
00:18:31,319 --> 00:18:33,079
but you don't actually find out that it was fraud

420
00:18:33,160 --> 00:18:36,920
until maybe two months later when they issue an ach

421
00:18:36,960 --> 00:18:38,839
return at the end of the sixty d expiration window.

422
00:18:39,480 --> 00:18:41,319
So you did not know that it was fraud until

423
00:18:41,440 --> 00:18:45,319
say March. So when you're building your training sets to

424
00:18:45,359 --> 00:18:47,119
train your machine learning models, you need to be very

425
00:18:47,119 --> 00:18:49,519
careful to make sure that no information which you learned

426
00:18:49,559 --> 00:18:53,359
after the time of the decision leaks backwards into the

427
00:18:53,519 --> 00:18:56,519
effectively row of data which represents that event, even though

428
00:18:56,559 --> 00:18:59,039
you have been collecting data during that whole window and

429
00:18:59,119 --> 00:19:01,759
learning new things. To make sure you filtered out and

430
00:19:01,759 --> 00:19:05,559
that's not so hard if you have immutable data sources

431
00:19:05,559 --> 00:19:07,960
in simple relationships, but it gets really nasty and you're

432
00:19:07,960 --> 00:19:13,759
doing things like computing aggregations over transactions or applying complicated

433
00:19:13,799 --> 00:19:18,480
filters on statuses which are evolving over time. So at

434
00:19:18,480 --> 00:19:21,359
the core of our software is effectively a SQL engine,

435
00:19:21,359 --> 00:19:22,920
but one of the ways in which it's different from

436
00:19:22,920 --> 00:19:24,880
most equal engines is that it has this notion of

437
00:19:24,880 --> 00:19:29,359
the effective timestamp of sell data in the system, so

438
00:19:29,400 --> 00:19:31,240
we can kind of manage that filtering for you.

439
00:19:31,440 --> 00:19:34,160
Speaker 1: If I'm doing some sort of machine learning, and in

440
00:19:34,240 --> 00:19:36,400
the data set that I'm using to train the model,

441
00:19:37,000 --> 00:19:39,799
I actually need to make sure I'm not including the

442
00:19:39,839 --> 00:19:42,000
sort of output of the model, which in a lot

443
00:19:42,000 --> 00:19:45,799
of ways is something you today with lllms. We actually

444
00:19:45,880 --> 00:19:48,039
want in there because we want it to actually affect

445
00:19:48,039 --> 00:19:51,759
the driver of the token selection on the output tokens.

446
00:19:52,000 --> 00:19:53,960
But if it's included in there, then the model sort

447
00:19:54,000 --> 00:19:57,599
of knows about that data, which will in effect defeat

448
00:19:57,599 --> 00:19:59,599
the whole purpose of the model, which is to figure

449
00:19:59,599 --> 00:20:03,160
out not knowing that. I mean, I can understand associations

450
00:20:03,640 --> 00:20:05,480
or matching that where you would want it, but when

451
00:20:05,480 --> 00:20:08,039
it comes to date based things. I totally get that.

452
00:20:07,960 --> 00:20:09,759
Speaker 2: You can kind of apply the same logic to the

453
00:20:09,880 --> 00:20:12,599
LM applications too. Imagine like you present a transaction to

454
00:20:13,359 --> 00:20:15,960
GPT five and you say, like, is this fraud or not?

455
00:20:16,319 --> 00:20:18,599
It'd be really unfortunate if part of the information you

456
00:20:18,640 --> 00:20:21,680
gave it included the label is fraud or not. It

457
00:20:21,680 --> 00:20:24,119
would hopefully pick up on that label and say, ah, well,

458
00:20:24,160 --> 00:20:27,119
it's in fact not fraud, So we know that a priori.

459
00:20:27,920 --> 00:20:30,319
That wouldn't be In general, when people go and swipe

460
00:20:30,319 --> 00:20:32,359
credit cards at merchants, they don't actually tell the merchant

461
00:20:32,359 --> 00:20:34,359
whether they're committing fraud, so it's not super useful to

462
00:20:34,359 --> 00:20:35,079
train on that signal.

463
00:20:35,200 --> 00:20:39,039
Speaker 1: This remind me of two scenarios in the recent history

464
00:20:39,119 --> 00:20:42,000
for the creation of LMS and machine learning in general.

465
00:20:42,039 --> 00:20:45,440
The first one is the presence of the medical ruler

466
00:20:45,839 --> 00:20:50,200
in the detection of cancerous moles and termatology. Like, if

467
00:20:50,200 --> 00:20:53,720
the ruler is there, there's a problem, right yeah, And

468
00:20:54,079 --> 00:20:56,400
I think there's a generic base for this known as

469
00:20:56,440 --> 00:20:58,920
like the giraffe problem early on, as if you gave

470
00:20:58,960 --> 00:21:01,839
an image model a picture of a safari and said,

471
00:21:01,880 --> 00:21:04,240
is there a giraffe? The answer is always yes. Because

472
00:21:04,279 --> 00:21:07,119
there was, because it has not because there was ever

473
00:21:07,160 --> 00:21:10,319
a giraft there. It was because if you ever asked

474
00:21:10,359 --> 00:21:12,920
if there was a giraffe, it was highly correlated with

475
00:21:12,920 --> 00:21:15,240
there actually being a giraffe. So therefore the presence of

476
00:21:15,279 --> 00:21:18,559
the question eliminated the possibility of it being no. So

477
00:21:18,799 --> 00:21:20,279
as soon as the question was asked, there was always

478
00:21:20,319 --> 00:21:22,079
a draft, even if there wasn't one in the picture.

479
00:21:22,279 --> 00:21:24,079
Speaker 2: Yeah, it's the same sort of problem that we're trying

480
00:21:24,119 --> 00:21:28,400
to stop. It ends up being extremely painful and subtle

481
00:21:28,440 --> 00:21:31,880
because you know, you have database records that have timestamps

482
00:21:31,880 --> 00:21:34,519
on them, but it turns out in production systems the

483
00:21:34,559 --> 00:21:38,599
timestamps are approximate, or maybe someone else came along and

484
00:21:38,640 --> 00:21:41,000
updated the row and the original domain modelor didn't know

485
00:21:41,000 --> 00:21:43,000
about it, or things like that, So I don't know.

486
00:21:43,039 --> 00:21:45,759
Twenty percent of the time in the entire process of

487
00:21:45,799 --> 00:21:48,960
doing machine learning is about actually figuring out when things

488
00:21:49,039 --> 00:21:51,480
did happen and when you did really learn about them,

489
00:21:51,480 --> 00:21:53,559
and would that information have been available.

490
00:21:53,200 --> 00:21:55,680
Speaker 1: If someone had the ridiculous idea to try to, you know,

491
00:21:56,000 --> 00:21:59,200
go build time travel themselves and not rely on chalk

492
00:21:59,319 --> 00:22:02,039
to solve it for them, Like, is there some secret here.

493
00:22:02,200 --> 00:22:04,440
I mean, don't don't tell us everything, but you know

494
00:22:04,480 --> 00:22:06,680
something specific about how you're thinking about the problems in

495
00:22:06,799 --> 00:22:10,160
order to evade this potential issue in the output.

496
00:22:10,200 --> 00:22:12,000
Speaker 2: I think there's a couple of different dimensions we think

497
00:22:12,039 --> 00:22:14,079
about it. One is like at the SDK layer, when

498
00:22:14,079 --> 00:22:16,319
people are defining, you know, how data is integrated into

499
00:22:16,319 --> 00:22:18,799
the system. We can kind of have things which encourage

500
00:22:18,799 --> 00:22:21,759
them to think about the distinctions between these types of timestamps.

501
00:22:21,960 --> 00:22:25,880
Then there's like the actual query execution layer, so inside

502
00:22:25,920 --> 00:22:28,279
of the system, which is doing things that are like

503
00:22:28,319 --> 00:22:30,759
seqal semantics. You know, in a normalst equal database you

504
00:22:30,799 --> 00:22:34,720
track tuples. We're tracking tooples augmented with a lot of metadata,

505
00:22:34,759 --> 00:22:38,160
so we know extra information, extra bits basically about every

506
00:22:38,480 --> 00:22:42,279
value we're moving around, which tells us when did it

507
00:22:42,319 --> 00:22:44,759
become effective, when did it become replaced? You know, is

508
00:22:44,799 --> 00:22:48,119
this an actually valid value? Is it noll because it's missing,

509
00:22:49,039 --> 00:22:51,279
and we're making sure that all the operations we do

510
00:22:51,400 --> 00:22:54,119
kind of properly transform that metadata. In addition to the

511
00:22:54,160 --> 00:22:56,000
source data. You would have to do something like that,

512
00:22:56,039 --> 00:22:58,079
I think in order to build a replacement. And then

513
00:22:58,119 --> 00:23:00,440
the third thing is down at the actual physical execute layer.

514
00:23:00,559 --> 00:23:02,240
So we care a lot about making sure things want

515
00:23:02,319 --> 00:23:05,559
run really fast. So we have to implement like custom

516
00:23:05,720 --> 00:23:08,720
join algorithms and custom you know, actually ways of adding

517
00:23:08,759 --> 00:23:11,079
numbers together so that we can respect all this metadata

518
00:23:11,759 --> 00:23:14,519
and do these operations efficiently. It turns out that just

519
00:23:14,559 --> 00:23:17,519
like if you just do straight standard SQL aggregations, you

520
00:23:17,599 --> 00:23:20,640
run into enormous cardinality explosions and in the size of

521
00:23:20,640 --> 00:23:22,759
the data you're processing when you have to think about

522
00:23:22,759 --> 00:23:23,559
all the different time points.

523
00:23:23,599 --> 00:23:26,680
Speaker 1: It occurs at just for clarity, the models, they're being

524
00:23:26,720 --> 00:23:29,960
built on your customer side based off of the data

525
00:23:29,960 --> 00:23:32,319
that you're providing, or are you helping to build the model.

526
00:23:32,599 --> 00:23:34,640
Speaker 2: Yeah, So we don't usually get involved in the actual

527
00:23:34,640 --> 00:23:38,480
model construction. We do oftentimes provide advice about how to

528
00:23:38,640 --> 00:23:41,359
model the data that's flowing through the system, like what

529
00:23:41,440 --> 00:23:44,160
schemas make sense? How should you think about time right now?

530
00:23:44,200 --> 00:23:48,000
We don't do any model training, Like, we don't do

531
00:23:48,000 --> 00:23:50,400
any like solutions stuff for model training for people.

532
00:23:50,240 --> 00:23:53,400
Speaker 1: Think it's like immutable databases for machine learning.

533
00:23:53,599 --> 00:23:55,599
Speaker 2: Yeah, and we call ourselves the data platform Free AI

534
00:23:55,680 --> 00:23:59,599
machine learning. We don't think it's specifically fraud related like

535
00:23:59,640 --> 00:24:03,400
we do recommendation systems search logistics models a lot of

536
00:24:03,400 --> 00:24:06,960
content moderation, which is kind of fun. We basically think about, Okay,

537
00:24:07,000 --> 00:24:08,640
what are all the workflows that are involved in our

538
00:24:08,880 --> 00:24:11,799
productionizing AI machine learning? And can we build like an

539
00:24:11,920 --> 00:24:16,000
end to end answer for that versus building horizontal infrastructure.

540
00:24:16,160 --> 00:24:20,000
Speaker 1: You found one of the magical shovels in this AI bubble.

541
00:24:20,160 --> 00:24:23,200
Speaker 2: Elliott and I both did effectively the job our customers

542
00:24:23,240 --> 00:24:25,200
do at a bunch of different companies before. So we

543
00:24:25,279 --> 00:24:27,759
saw a system like chalk get built a bunch of times,

544
00:24:27,799 --> 00:24:29,759
and we've tried to, you know, make an actually good

545
00:24:29,839 --> 00:24:31,440
version of it instead of the like, you know, four

546
00:24:31,440 --> 00:24:32,960
engineers two quarters version of it.

547
00:24:33,000 --> 00:24:35,480
Speaker 1: Having been in a similar circumstance, I find the challenge

548
00:24:35,519 --> 00:24:37,960
there is you shift from being a technical person, which

549
00:24:38,000 --> 00:24:41,480
it very much seems you are, into what I absolutely

550
00:24:41,519 --> 00:24:44,319
hate doing, which is marketing and sales to try to

551
00:24:44,319 --> 00:24:46,400
convince people to buy the product that you already know

552
00:24:46,519 --> 00:24:48,759
they need. And you're nodding your head, which means you

553
00:24:49,160 --> 00:24:50,680
know exactly what I'm talking about.

554
00:24:50,839 --> 00:24:52,720
Speaker 2: I don't know if this is like something we should say,

555
00:24:52,720 --> 00:24:55,720
but like, effectively we have one hundred percent pilot closed rate.

556
00:24:55,799 --> 00:24:56,039
Speaker 1: I e.

557
00:24:56,519 --> 00:24:58,759
Speaker 2: If you come and try our software. It's almost inevitable

558
00:24:58,799 --> 00:25:01,240
you will buy it. Of course, Then the problem becomes,

559
00:25:01,240 --> 00:25:02,319
how do you get the word out? How do you

560
00:25:02,319 --> 00:25:05,000
get people to actually try it? So you know here

561
00:25:05,000 --> 00:25:05,279
we are?

562
00:25:05,599 --> 00:25:06,200
Speaker 1: It is.

563
00:25:06,319 --> 00:25:08,559
Speaker 2: It is interesting because I think it's really difficult to

564
00:25:08,640 --> 00:25:11,400
convey what we do and what problems we're solving. And

565
00:25:11,440 --> 00:25:13,599
I don't think we've really found the two sentence elevator

566
00:25:13,640 --> 00:25:15,400
pitch it despite thinking about it for three years.

567
00:25:15,640 --> 00:25:18,519
Speaker 1: I'd maybe give you some more confidence in your ability here.

568
00:25:18,680 --> 00:25:20,920
I do think you're you know, you're clear about talking

569
00:25:21,039 --> 00:25:24,440
about what is going on there, and even at our point,

570
00:25:24,480 --> 00:25:27,400
which is I don't know, I forget the exact year,

571
00:25:27,480 --> 00:25:29,640
how long we've been around it. It's not even competitors.

572
00:25:29,680 --> 00:25:33,480
It's other companies that have used similar language and basically

573
00:25:33,519 --> 00:25:36,559
stolen it to mean something completely different. And now that

574
00:25:36,640 --> 00:25:39,920
all that's left is either you compete with them and

575
00:25:39,960 --> 00:25:42,480
say the same things they're saying, but mean something totally

576
00:25:42,480 --> 00:25:45,440
different the right thing, I'll say, not the wrong thing

577
00:25:45,440 --> 00:25:48,000
that they're they're peddling, of course, so you say something

578
00:25:48,039 --> 00:25:50,359
completely different and your customers are like, we have no

579
00:25:50,440 --> 00:25:52,880
idea what you're talking about, and they you know, churn

580
00:25:52,920 --> 00:25:54,839
from your marketing page, or they don't even get there

581
00:25:54,880 --> 00:25:57,880
because they don't understand what you're actually going at. I'll

582
00:25:57,880 --> 00:26:01,240
say from experience, one of the secrets here, especially going

583
00:26:01,240 --> 00:26:04,319
out to the larger enterprise customers, is it doesn't matter

584
00:26:04,319 --> 00:26:06,319
because you don't care about your marketing page. You care

585
00:26:06,359 --> 00:26:10,680
about you know, connecting directly with whoever is making the decision.

586
00:26:10,440 --> 00:26:12,680
Speaker 2: Right, Yeah, FUAL high touch sales.

587
00:26:12,880 --> 00:26:14,720
Speaker 1: Yeah, high touch sales, So you must love that.

588
00:26:14,880 --> 00:26:17,839
Speaker 2: Fortunately, both of my co funders are much more charismatic

589
00:26:17,920 --> 00:26:19,920
than I am, so they focus mostly on the pre

590
00:26:20,000 --> 00:26:22,519
sales stuff before people actually sign to buy the software,

591
00:26:22,519 --> 00:26:24,400
and I get to think about the hard, interesting problems

592
00:26:24,440 --> 00:26:27,599
post sales. But it's definitely a big challenge, and I

593
00:26:27,640 --> 00:26:30,359
think like the kind of reason we constructed this company

594
00:26:30,400 --> 00:26:32,559
is because Elliot and I did a consumer fintech before,

595
00:26:32,599 --> 00:26:34,720
and that was one hundred percent about marketing. If you

596
00:26:34,759 --> 00:26:38,039
don't love changing the background color of ad creative on

597
00:26:38,079 --> 00:26:41,400
Facebook Ads Console for your entire life, you shouldn't do

598
00:26:41,440 --> 00:26:44,759
consumer fintech. And we basically spent the time during our

599
00:26:44,799 --> 00:26:47,240
earn out at Credit Crema thinking about, Okay, how can

600
00:26:47,279 --> 00:26:49,720
we make a company where we get paid money to

601
00:26:49,799 --> 00:26:53,160
solve interesting systems problems for you know, big companies, and

602
00:26:53,240 --> 00:26:54,599
this is what came up with you went.

603
00:26:54,480 --> 00:26:57,839
Speaker 1: Down a third path. There, if I understood your LinkedIn

604
00:26:58,200 --> 00:27:01,279
experience correctly, you made a great company and it got acquired.

605
00:27:01,480 --> 00:27:04,559
Speaker 2: Yeah, we built a consumer fintech company. We basically stapled

606
00:27:04,680 --> 00:27:06,920
a bank to an investment brokerage and then added like,

607
00:27:07,000 --> 00:27:10,359
we call it self driving finance. But that's like you say,

608
00:27:10,680 --> 00:27:13,279
Everyone says that now machine learning based wealth management, which

609
00:27:13,279 --> 00:27:16,279
would do constraint solving to move money around automatically for you.

610
00:27:16,359 --> 00:27:18,400
And it turns out that the set of people who

611
00:27:19,000 --> 00:27:22,799
are really interested in personal finance such that that sounds

612
00:27:22,839 --> 00:27:24,559
like a cool thing to them, and also who don't

613
00:27:24,559 --> 00:27:26,359
want to be involved in the management of the personal

614
00:27:26,359 --> 00:27:29,200
finance is approximately the nul SAT. Basically, everyone who cares

615
00:27:29,200 --> 00:27:31,440
about personal finance wants to manage it, and everyone who

616
00:27:31,440 --> 00:27:33,160
doesn't care about personal finance is not going to buy

617
00:27:33,160 --> 00:27:36,480
a fancy financial planning product. So the tech was cool.

618
00:27:36,720 --> 00:27:39,039
We grew the business. Eventually, Credit come I bought the

619
00:27:39,039 --> 00:27:42,079
company because they wanted to offer that type of financial

620
00:27:42,119 --> 00:27:45,519
management to the one hundred million some people that they serve.

621
00:27:45,839 --> 00:27:48,200
I learned that I really don't enjoy doing consumer.

622
00:27:48,240 --> 00:27:51,759
Speaker 1: Yeah, I have a slightly different takeaway there. I make

623
00:27:51,799 --> 00:27:53,640
a lot of mistakes with what to do with my

624
00:27:53,880 --> 00:27:58,200
personal finances, and that made me one time, say, you know,

625
00:27:59,279 --> 00:28:02,839
doing some hypothesis and guessing it's going to be more

626
00:28:03,640 --> 00:28:05,799
of a better strategy, And so I threw some money

627
00:28:05,839 --> 00:28:11,640
out a product that automatically promises he returns by buying

628
00:28:11,720 --> 00:28:15,880
and selling shares of different index funds. And it wasn't

629
00:28:15,920 --> 00:28:18,000
the worst decision I ever made. I mean, I didn't

630
00:28:18,000 --> 00:28:20,559
put all my money there, but I definitely put some there.

631
00:28:20,559 --> 00:28:22,599
And so, like I do, if you think you're good

632
00:28:22,640 --> 00:28:27,279
with money, like just dispel that notion. You're definitely worse

633
00:28:27,400 --> 00:28:30,920
than just buying the index fund in in most scenarios.

634
00:28:30,960 --> 00:28:32,799
But you can do a little bit better than that

635
00:28:33,000 --> 00:28:35,359
if you happen to find an engine that is willing

636
00:28:35,400 --> 00:28:39,279
to manage it. So being in that nulset, that magic

637
00:28:39,279 --> 00:28:42,559
space where you know what you're doing but also want

638
00:28:42,599 --> 00:28:44,519
to do the management, and then just stop doing the management.

639
00:28:44,519 --> 00:28:47,039
It's like a form of delegation. Yeah, I like the

640
00:28:47,039 --> 00:28:49,359
pivot because you're doing something that seems like way more

641
00:28:49,400 --> 00:28:51,960
interesting than moving money around. I mean, I don't know.

642
00:28:52,000 --> 00:28:53,920
I'm saying that I don't know anything about the technology

643
00:28:53,920 --> 00:28:55,799
and either of these pieces, so maybe that's a bit unfair,

644
00:28:55,839 --> 00:28:59,480
But I'm what you've shared so far is like, in

645
00:28:59,519 --> 00:29:02,279
a way, really exciting because a you're not talking about

646
00:29:02,279 --> 00:29:05,960
building AI. You're talking about your customers doing machine learning

647
00:29:06,039 --> 00:29:09,680
on a complexity set of data that they just wouldn't

648
00:29:09,680 --> 00:29:12,960
be able to achieve otherwise. And you're deploying your software

649
00:29:13,000 --> 00:29:15,799
into their accounts so that you can get access or

650
00:29:15,839 --> 00:29:18,519
integrate with their cloud provider where the data actually is,

651
00:29:18,640 --> 00:29:20,559
not to do it yourself. I mean, you're avoiding a

652
00:29:20,559 --> 00:29:23,440
lot of complexity that didn't bring value, and you figure

653
00:29:23,480 --> 00:29:26,880
out a way to still, you know, solve the real

654
00:29:27,119 --> 00:29:29,799
challenging problem, which you already experienced. I mean, I think

655
00:29:30,039 --> 00:29:32,559
you're onto something. And you said the company's only three

656
00:29:32,640 --> 00:29:33,079
years old.

657
00:29:33,119 --> 00:29:35,759
Speaker 2: Now I would say that so far it has been.

658
00:29:36,319 --> 00:29:38,839
It's been interesting because you see the challenges coming, but

659
00:29:38,880 --> 00:29:42,519
maybe you choose to run into them anyways. So I

660
00:29:42,519 --> 00:29:44,599
think we've dodged a lot of the dumb things like

661
00:29:44,680 --> 00:29:47,640
messing up with fundraising, like how do you incorporate a company?

662
00:29:47,799 --> 00:29:50,440
How do you business ops? Like accounting all those things like,

663
00:29:50,960 --> 00:29:53,039
we're good at that because we've all been through this before.

664
00:29:54,200 --> 00:29:56,440
But there are other challenges like that we kind of

665
00:29:56,440 --> 00:29:58,680
signed up for. Like we started out roll the software

666
00:29:58,680 --> 00:30:00,440
in Python because that was the fact thing to do

667
00:30:00,559 --> 00:30:02,519
to get started, and we needed to run our customers

668
00:30:02,519 --> 00:30:05,640
Python code. But we knew inevitably that that was not

669
00:30:05,759 --> 00:30:08,079
going to scale, that it was not going to be

670
00:30:08,119 --> 00:30:11,559
fast enough, and it has been a herculean project to

671
00:30:11,680 --> 00:30:13,480
rebuild a lot of that software and C plus plus

672
00:30:13,480 --> 00:30:15,920
and Rust, and it would have been simpler to start there,

673
00:30:15,960 --> 00:30:18,480
but then we wouldn't have known whether anyone cared. Like basically,

674
00:30:18,519 --> 00:30:21,839
if you plus for no reason, it doesn't help anyone.

675
00:30:22,759 --> 00:30:25,519
Speaker 1: I had There was a great quote that one of

676
00:30:25,519 --> 00:30:27,319
my early mentors had shared with me a couple of

677
00:30:27,319 --> 00:30:31,000
times where it was it was all about it doesn't

678
00:30:31,039 --> 00:30:32,880
matter if you do that in the in the perfect way,

679
00:30:33,039 --> 00:30:35,200
because you have a need to have it actually deliver

680
00:30:35,319 --> 00:30:38,799
value to your customers, which if you don't have is

681
00:30:38,839 --> 00:30:42,359
normally a challenge. Yeah, I would have personally avoided Python,

682
00:30:42,440 --> 00:30:44,559
but you know, if you're in the machine learning realm,

683
00:30:44,599 --> 00:30:47,720
and you are, it's twenty twenty two, that's pretty much

684
00:30:47,720 --> 00:30:50,200
on the interface around and what people are trying to

685
00:30:50,279 --> 00:30:52,599
use to get running, And if you had started with

686
00:30:52,680 --> 00:30:54,839
Rust at that point, it would have been way too

687
00:30:54,920 --> 00:30:57,559
early on, I think, to actually have the infrastructure available,

688
00:30:57,799 --> 00:31:00,279
but in the space going for performance, and we see

689
00:31:00,319 --> 00:31:03,200
a lot of the database engines being written in rewritten

690
00:31:03,200 --> 00:31:06,079
in RUSS now, which I can definitely see if you're

691
00:31:06,279 --> 00:31:08,359
closer on the database side. That makes a lot of sense.

692
00:31:08,640 --> 00:31:13,240
Speaker 2: Yeah, And so also it turns out that, to my surprise, honestly,

693
00:31:13,279 --> 00:31:16,160
the interrupt between both RUSS, Dancy plus plus and Python

694
00:31:16,359 --> 00:31:19,920
is really fantastic. So it's been fairly achievable to kind

695
00:31:19,920 --> 00:31:23,480
of rip pieces out and rebuild them in the faster language.

696
00:31:24,200 --> 00:31:26,319
Definitely would have been a lot simple to start there.

697
00:31:26,480 --> 00:31:29,839
Speaker 1: Did you have Rust experience before going in here or

698
00:31:29,880 --> 00:31:31,519
was that like we'd have to learn that's.

699
00:31:31,359 --> 00:31:34,599
Speaker 2: Because right, yeah, very little. And I think that's actually

700
00:31:34,640 --> 00:31:36,079
one of the things that's pushed us more towards C

701
00:31:36,160 --> 00:31:38,720
plus plus. Although originally we thought we'd be a Rust

702
00:31:38,720 --> 00:31:42,079
company because there's just so many things that are going

703
00:31:42,119 --> 00:31:44,960
wrong every day that you don't want to add on

704
00:31:45,000 --> 00:31:47,519
the extra layer if I don't even know my programming language.

705
00:31:47,799 --> 00:31:50,640
So again, it's like if I had known Rust at

706
00:31:50,640 --> 00:31:52,759
the very start of the company. Really well, I'm an expert.

707
00:31:52,799 --> 00:31:55,160
I would have just started there, but for lots of reasons,

708
00:31:55,200 --> 00:31:59,519
C plus plus was a better intermediate for us from Python, say,

709
00:31:59,559 --> 00:32:02,319
aside from the complexity the language or the familiarity A lot.

710
00:32:02,359 --> 00:32:04,759
Turns out a lot of database drivers are actually like native,

711
00:32:04,799 --> 00:32:07,079
implent and C plus plus. So if you want like

712
00:32:07,119 --> 00:32:10,920
the full suite of interfaces to things like, you know,

713
00:32:11,000 --> 00:32:14,680
Dynamo dB or whatever, you don't get that through the

714
00:32:14,680 --> 00:32:17,160
Python interface, and maybe the Rust driver doesn't even exist.

715
00:32:17,240 --> 00:32:21,720
Speaker 1: You're integrating with the databases that exist through their APIs directly,

716
00:32:22,000 --> 00:32:26,079
rather than utilizing say the AWSSDK for doing that.

717
00:32:26,079 --> 00:32:28,079
Speaker 2: That's right, yeah, I mean it turns out like DATABASESDK

718
00:32:28,200 --> 00:32:30,279
uses an old version of lib curl and it's really slow.

719
00:32:31,039 --> 00:32:33,640
So if you want like really fast access to most

720
00:32:33,640 --> 00:32:35,880
of the bos data stores, it makes sense to build

721
00:32:35,880 --> 00:32:37,920
your own just a shapey connection to them.

722
00:32:38,000 --> 00:32:40,200
Speaker 1: I think a lot of people either just cringed or

723
00:32:40,240 --> 00:32:42,960
got really excited that you said that, because now they

724
00:32:42,960 --> 00:32:44,960
have a new thing that they can spend all their

725
00:32:45,000 --> 00:32:48,160
time doing. I can make my application faster by getting

726
00:32:48,240 --> 00:32:50,920
rid of the a DOSSDK and replacing it with one

727
00:32:50,920 --> 00:32:51,839
that I wrote myself.

728
00:32:52,119 --> 00:32:54,279
Speaker 2: A lot of this company has been about that exact

729
00:32:54,400 --> 00:32:58,200
kind of intuition breaking. Like I'm a library maximalist, like

730
00:32:58,200 --> 00:33:00,400
I always start with the library, and I'm like, surely

731
00:33:00,440 --> 00:33:02,640
this must work. But it turns out that if you

732
00:33:02,720 --> 00:33:05,359
are trying to count microseconds, a lot of times the

733
00:33:05,359 --> 00:33:07,680
library's written on different constraints, like maybe the author wanted

734
00:33:07,720 --> 00:33:10,359
to autogenerate it, or maybe they wanted to have, you know,

735
00:33:10,440 --> 00:33:13,720
really nice error messages or something. They have some other concerns.

736
00:33:13,759 --> 00:33:16,119
But if literally all you care about is okay, I

737
00:33:16,200 --> 00:33:18,400
have to make this happen in less than two milliseconds

738
00:33:18,960 --> 00:33:21,599
and at a high, like a high through put, weird

739
00:33:21,599 --> 00:33:23,440
stuff starts to creep in and you have to develop

740
00:33:23,440 --> 00:33:25,000
a pains about a lot of layers in the stack.

741
00:33:25,160 --> 00:33:26,640
Speaker 1: And I'm going to get this wrong because it's been

742
00:33:26,720 --> 00:33:31,319
way too long. I'm starting to recall a video about

743
00:33:31,519 --> 00:33:35,519
getting rid of all of the solid principles on software development.

744
00:33:35,599 --> 00:33:39,720
So the set of solid which are like single responsibility principle,

745
00:33:39,880 --> 00:33:44,440
open open closed, go substitution principle. I don't whatever YouTube

746
00:33:44,680 --> 00:33:46,720
video creator basically goes thro each one of them will

747
00:33:46,720 --> 00:33:49,640
be like why is it a mistake from a performance standpoint,

748
00:33:49,680 --> 00:33:52,000
like we did we write better software by following these

749
00:33:52,039 --> 00:33:54,359
and like it's like, well, no, actually we wrote worse software.

750
00:33:54,599 --> 00:33:56,839
It's more confusing, but even worse if you look at

751
00:33:56,839 --> 00:33:59,680
the performance, it's it's not a factor of two or

752
00:33:59,680 --> 00:34:01,640
a fact or three or even ten where you're like, oh,

753
00:34:01,720 --> 00:34:03,640
we'll just ignore it for the most part because we

754
00:34:03,680 --> 00:34:06,160
have a set of ten items or even one hundred items,

755
00:34:06,240 --> 00:34:07,880
but you have a set of one thousand or a million,

756
00:34:07,960 --> 00:34:10,360
and it really makes an impact. There means a good

757
00:34:10,360 --> 00:34:13,800
point because we saw this early on with especially with

758
00:34:13,880 --> 00:34:18,840
JavaScript in AWS the library was managed manually, and then

759
00:34:19,199 --> 00:34:20,840
I think it was like five years ago they switched

760
00:34:20,880 --> 00:34:24,719
to AWS SDK version three where they were using Smithy

761
00:34:24,840 --> 00:34:27,039
to generate all their SDKs. Now and I think it's

762
00:34:27,039 --> 00:34:30,519
gotten significantly worse, not initially from a performance standpoint, but

763
00:34:30,800 --> 00:34:33,960
definitely from a usability one. And it's okay if you're

764
00:34:33,960 --> 00:34:36,199
a massive platform where people are like, we'll do whatever

765
00:34:36,239 --> 00:34:38,159
we have to to make it work, and there is

766
00:34:38,199 --> 00:34:41,000
consistency across languages, it's just not that great. Like if

767
00:34:41,039 --> 00:34:43,679
you know your language really well, you know Python really well,

768
00:34:43,719 --> 00:34:45,800
you know JavaScript, and you go to u ZSK it's

769
00:34:45,800 --> 00:34:48,960
like nothing you've ever used before. But now you're saying also,

770
00:34:49,400 --> 00:34:51,440
you know, security aside, because as is great at that

771
00:34:51,679 --> 00:34:54,199
performance wise, it could be a concern like that didn't

772
00:34:54,199 --> 00:34:56,559
really ever occur to me, Like unless you're interacting with

773
00:34:56,599 --> 00:34:59,719
like val Key or rds, you know my sequel Aurora,

774
00:34:59,760 --> 00:35:02,679
they're customer SDKs. Like you, we're not using ADUs one,

775
00:35:02,679 --> 00:35:05,000
but if you're using one of the Servilus options, you

776
00:35:05,119 --> 00:35:08,000
are probably using the SDK, So that's really surprising.

777
00:35:08,039 --> 00:35:10,280
Speaker 2: Actually yeah, no, I mean actually valky is a great

778
00:35:10,320 --> 00:35:11,960
example of where we had to spend a lot of

779
00:35:11,960 --> 00:35:14,360
time thinking about stuff because like for one of our customers,

780
00:35:14,360 --> 00:35:17,199
we're pushing back and forth gigabytes a second of floats basically,

781
00:35:17,320 --> 00:35:19,840
so we had to think about like the encoding, the

782
00:35:20,199 --> 00:35:24,519
connection pooling policies, the retry policies for intermittent failures, all

783
00:35:24,559 --> 00:35:27,440
of those details. And when there's like two or three

784
00:35:27,519 --> 00:35:31,079
layers of SDK and astraction in the way, it really

785
00:35:31,079 --> 00:35:33,079
messes with your ability to reason about the system in

786
00:35:33,360 --> 00:35:36,440
that in like the intense scenarios, it like works fine

787
00:35:36,440 --> 00:35:38,599
on day one, but it's just you know, day ninety

788
00:35:38,760 --> 00:35:40,679
regretting not understanding how this stuff works.

789
00:35:40,840 --> 00:35:43,880
Speaker 1: You've mentioned that there's quite a few of those lessons

790
00:35:43,880 --> 00:35:45,880
that you've learned, Like does this conversation just go on

791
00:35:45,960 --> 00:35:48,360
for hours with all those lesson learners? Like you know

792
00:35:48,400 --> 00:35:50,519
another one that just like immediately pops out at you.

793
00:35:50,719 --> 00:35:53,440
Speaker 2: Well, I mean another one is probably like I originally

794
00:35:53,519 --> 00:35:56,000
assumed that Python was not that slow, so you know,

795
00:35:56,039 --> 00:35:57,719
I mean it has like a sin kio. You're like, cool,

796
00:35:57,760 --> 00:36:00,639
I can use it to do parallelism. How slow could

797
00:36:00,639 --> 00:36:02,400
adding numbers together be? Like, I have a lot of

798
00:36:02,400 --> 00:36:05,239
experience writing JavaScript, and like it's an interpreted language sort

799
00:36:05,239 --> 00:36:08,199
of like obviously the git's incredible, but if you do

800
00:36:08,280 --> 00:36:10,039
a for loop in some integers, it will not be

801
00:36:10,079 --> 00:36:12,280
that much slower than writing the equivalent program. And see right,

802
00:36:12,559 --> 00:36:17,039
But those feelings don't really hold true for Python. It's

803
00:36:17,079 --> 00:36:19,320
really easy to end up with function calls or the

804
00:36:19,400 --> 00:36:22,639
trivial operations taking single digit or tens of microseconds because

805
00:36:22,679 --> 00:36:25,280
of all the interaction and interpret overhead and the absence

806
00:36:25,280 --> 00:36:27,760
of legit. And if you're trying to do something you

807
00:36:27,800 --> 00:36:30,320
know billions or trillions of times and it takes tens

808
00:36:30,360 --> 00:36:31,960
of micros, all of a sudden you're looking at like

809
00:36:32,000 --> 00:36:36,199
real seconds adding up, which makes entire categories of operations impossible.

810
00:36:36,960 --> 00:36:39,480
So obviously we wrote a lot of software to peoples

811
00:36:39,480 --> 00:36:42,639
plus and rust, but we also try to transpile effectively

812
00:36:42,679 --> 00:36:48,119
our customer software out of Python into an abstract expression representation.

813
00:36:48,400 --> 00:36:48,960
Speaker 1: Wow.

814
00:36:49,360 --> 00:36:51,679
Speaker 2: So that's the whole thing too. So we let people

815
00:36:51,760 --> 00:36:55,079
write transformations over data and the maybe one of the

816
00:36:55,079 --> 00:36:57,119
weird twists is that they want to write those transformations

817
00:36:57,119 --> 00:36:59,239
in Python when they ran on our platform as opposed

818
00:36:59,239 --> 00:37:02,239
to standard sequel. So what we do is we run

819
00:37:02,280 --> 00:37:04,920
this thing which is effectively a symbolic interpreter. It has

820
00:37:05,039 --> 00:37:08,480
roots in Google's type checker, which they just sunsets had,

821
00:37:09,360 --> 00:37:12,679
but we basically evaluate all of their functions symbolically, so

822
00:37:12,719 --> 00:37:15,440
instead actually executing them, we model the control flow and

823
00:37:15,480 --> 00:37:18,480
try to do little proofs about nulity and non zeroness

824
00:37:18,519 --> 00:37:20,679
and positive and negative veneagers and things like that to

825
00:37:20,840 --> 00:37:23,280
prove we understand the semantics of it. And if we

826
00:37:23,320 --> 00:37:25,400
can prove that we fully understand a function all the

827
00:37:25,400 --> 00:37:28,000
associated functions of calls, then we don't need Python to

828
00:37:28,079 --> 00:37:31,760
run it. We can basically interpret it into a different

829
00:37:31,960 --> 00:37:35,159
DSL for executing expressions and then we can run A

830
00:37:35,280 --> 00:37:37,880
plus B from Python or even for loops or conditionals

831
00:37:37,960 --> 00:37:40,440
or like some library code entirely Python free.

832
00:37:40,480 --> 00:37:44,559
Speaker 1: This is normally used for semantic validation of code, for

833
00:37:44,960 --> 00:37:48,400
ensuring that the outputs, especially in the financial domain, are

834
00:37:48,599 --> 00:37:51,599
like reasonable and I think good corollary here is if

835
00:37:51,599 --> 00:37:54,119
you have an ad function, you're not literally going to

836
00:37:54,119 --> 00:37:58,199
pass in every possible integer value or float value into

837
00:37:58,199 --> 00:38:01,199
both parameters and then validate the output. Just not possible.

838
00:38:01,239 --> 00:38:05,239
So instead you write your programming language or your tree

839
00:38:05,280 --> 00:38:07,719
in a way that is very verifiable, and then you

840
00:38:07,800 --> 00:38:10,159
verify it. But you're using it to actually then board

841
00:38:10,199 --> 00:38:12,960
it over and do it not only interoperability, but actually

842
00:38:13,239 --> 00:38:16,599
runtime execution and whatever engine you want. Are you then

843
00:38:17,159 --> 00:38:19,599
running it in roster C plus plus at that point.

844
00:38:19,840 --> 00:38:22,880
Speaker 2: Yeah. So the underlying execution engine for us is vlox,

845
00:38:22,880 --> 00:38:25,599
which is what Facebook uses to accelerate Presto, which is

846
00:38:25,639 --> 00:38:29,119
their internal analytical database. People are also using it to

847
00:38:29,159 --> 00:38:31,599
accelerate Spark. It's called gluten that is implemented in C

848
00:38:31,679 --> 00:38:34,239
plus plus. A lot of the operations are either GPU

849
00:38:34,280 --> 00:38:36,480
accelerated or sin be accelerated. So like DOWNE in the

850
00:38:36,480 --> 00:38:38,639
assembly code. But yeah, it's exactly what you said that.

851
00:38:38,679 --> 00:38:40,400
The point is if you if you understand what a

852
00:38:40,440 --> 00:38:42,519
function does, you don't need to literally run it with

853
00:38:42,599 --> 00:38:45,199
the original programm language was written in. And then for

854
00:38:45,280 --> 00:38:47,960
us the game becomes, you know, making sure we're perfectly sound,

855
00:38:48,000 --> 00:38:49,679
because we're going to take your Python and like not

856
00:38:49,760 --> 00:38:53,079
run Python. We better really know, we better be really

857
00:38:53,119 --> 00:38:56,199
sure we understand exactly what that's doing. And that is

858
00:38:56,239 --> 00:38:58,960
tricky in the presence of a lot of Python constructs

859
00:38:59,000 --> 00:39:02,360
like if people are using you know, like if conditionals

860
00:39:02,360 --> 00:39:06,039
to implicitly check nullity, but also like non falsiness. We

861
00:39:06,119 --> 00:39:08,320
have to represent all the semantics. So it ends up

862
00:39:08,360 --> 00:39:10,599
being kind of like a type checker, and we do

863
00:39:10,639 --> 00:39:12,480
a lot of the stuff that like liquid hasple does

864
00:39:13,400 --> 00:39:16,119
to trap not just like course categories of types, but

865
00:39:16,159 --> 00:39:18,760
also the values within the types. So we can prove that,

866
00:39:18,880 --> 00:39:23,079
like array indexes are inbounds for a list, so we

867
00:39:23,199 --> 00:39:25,599
know that like indexing a list by some in some

868
00:39:25,679 --> 00:39:29,079
integer value, I never throws the index error or something

869
00:39:29,159 --> 00:39:29,880
like that because.

870
00:39:29,719 --> 00:39:32,159
Speaker 1: You already know the length of the array fundamentally at

871
00:39:32,440 --> 00:39:35,079
equivalently compile time of the code.

872
00:39:35,159 --> 00:39:37,519
Speaker 2: Yeah, yeah, we track like we know it's a constant list,

873
00:39:37,519 --> 00:39:39,360
and we know the indexes less than the constant length

874
00:39:39,400 --> 00:39:40,840
the list, et cetera. That sort of stuff.

875
00:39:40,920 --> 00:39:43,880
Speaker 1: You've got a future in so many technology directions here

876
00:39:44,039 --> 00:39:47,320
because not just interacting with the database, but like unit testing,

877
00:39:47,320 --> 00:39:50,840
our code validation, semantic validation. I think there's another one

878
00:39:50,840 --> 00:39:53,159
here that I'm not thinking about the moment, but like

879
00:39:53,280 --> 00:39:55,880
just you can just easily go down any of these

880
00:39:56,119 --> 00:39:59,239
pass as added value for your customers in a way.

881
00:39:59,400 --> 00:40:01,360
Speaker 2: Yeah, that's right. I mean I think like we have

882
00:40:01,400 --> 00:40:04,119
an LSP plug in. I mean it's incredibly slow right now,

883
00:40:04,159 --> 00:40:06,880
so no one uses it, but we're working on making

884
00:40:06,920 --> 00:40:07,280
it faster.

885
00:40:07,920 --> 00:40:10,559
Speaker 1: That's that's the reason no one's using it because it's slow.

886
00:40:10,800 --> 00:40:13,079
Speaker 2: It's like takes forty five seconds to type check right now.

887
00:40:13,079 --> 00:40:15,920
But basically, like the intent and I'm told that we'll

888
00:40:15,920 --> 00:40:18,000
have this fixed in is that like as you're typing in,

889
00:40:18,039 --> 00:40:19,559
your editor will be able to tell you whether or

890
00:40:19,599 --> 00:40:22,000
not we can transpile the program or not. But also

891
00:40:22,039 --> 00:40:23,800
we can we can tell you, oops, you've made a mistake,

892
00:40:23,880 --> 00:40:26,480
because we know that you've like you know, done a

893
00:40:26,519 --> 00:40:28,719
null point or error, so we can let you know

894
00:40:28,760 --> 00:40:30,320
that right there in the editor, and then the kind

895
00:40:30,320 --> 00:40:31,960
of the next thing we're going to start doing is

896
00:40:32,079 --> 00:40:34,760
actually taking the historical values that this function would have

897
00:40:34,760 --> 00:40:36,920
been computed on, Like we know all the values of

898
00:40:36,960 --> 00:40:39,760
all the inputs to the functions ecstatically, so we can

899
00:40:39,800 --> 00:40:42,800
tell you, like as you're typing, what the output distribution

900
00:40:42,880 --> 00:40:44,599
of your function would be, which I think is an

901
00:40:44,599 --> 00:40:46,719
interesting thing for data engineers, because you know, if your

902
00:40:46,719 --> 00:40:49,320
function only produces zero, it's probably not a very useful function.

903
00:40:49,599 --> 00:40:53,480
Speaker 1: Is this something that you've developed for your company that

904
00:40:53,599 --> 00:40:56,199
is prorietary internal or something you've open sourced.

905
00:40:56,320 --> 00:40:58,800
Speaker 2: Yeah, it's proprietary right now. We did a presentation on

906
00:40:58,840 --> 00:41:01,880
how it works at vlock. There's an open question about

907
00:41:01,880 --> 00:41:04,440
exactly how much of this will upstream, but probably not

908
00:41:04,519 --> 00:41:04,880
none of it.

909
00:41:05,199 --> 00:41:06,800
Speaker 1: The question came to me because I do remember someone

910
00:41:06,800 --> 00:41:08,880
asking a number of years ago, like hey, I've got

911
00:41:08,960 --> 00:41:12,880
I mean, it wasn't linear algebra like calculating the solution set,

912
00:41:12,880 --> 00:41:14,800
but it was similar to like, hey, I'm just sort

913
00:41:14,840 --> 00:41:18,280
of curious for these variables that are parameters in my function?

914
00:41:18,719 --> 00:41:22,320
What values input and output? Makes sense? Yeah? And it

915
00:41:22,639 --> 00:41:24,360
sounds a lot like that, like oh, yeah, you know

916
00:41:24,519 --> 00:41:27,400
the output bounds are here's the distribution. You know, we

917
00:41:27,480 --> 00:41:29,480
know it is always been between zero and one or

918
00:41:29,800 --> 00:41:32,920
zero and you know maxin or whatever guaranteed never negative.

919
00:41:33,079 --> 00:41:35,320
And I think there is something to be said about

920
00:41:35,360 --> 00:41:38,159
the value there. I do worry that in today's world

921
00:41:38,599 --> 00:41:40,360
there is more and more companies that are trying to

922
00:41:40,360 --> 00:41:43,440
go faster and sacrificing quality and caring about that. Is

923
00:41:43,480 --> 00:41:46,599
that just my network or you know, do you see

924
00:41:46,639 --> 00:41:48,000
something similar? Yeah?

925
00:41:48,039 --> 00:41:50,239
Speaker 2: Well, I mean it's actually a big support challenge for US.

926
00:41:50,239 --> 00:41:52,719
I mean, every customer we have wants to use Cursor

927
00:41:52,880 --> 00:41:55,519
cloud code to generate all the integration code of US.

928
00:41:55,599 --> 00:41:58,199
And it turns out Curser and cloud Code have a

929
00:41:58,239 --> 00:42:00,440
lot of great ideas for functionality we should build but

930
00:42:00,519 --> 00:42:03,960
have not yet. And suddenly it becomes like a customer

931
00:42:04,000 --> 00:42:08,639
relationship issue for US that you know, they're they're generating

932
00:42:08,719 --> 00:42:11,679
code without verifying that it even matches anything they've everclaimed

933
00:42:11,679 --> 00:42:14,039
to support. I feel that really acutely right now, people

934
00:42:14,079 --> 00:42:14,960
are moving very fast.

935
00:42:15,239 --> 00:42:18,880
Speaker 1: A GMT Polumi two episodes ago told me that they

936
00:42:19,000 --> 00:42:23,519
use that to determine originally some features they should build. Yeah,

937
00:42:23,760 --> 00:42:26,880
because at which which like I already don't like that

938
00:42:26,920 --> 00:42:29,239
we're changing humanity to figure out how to talk to

939
00:42:29,239 --> 00:42:32,519
the LMS rather than improving the LMS to understand you know,

940
00:42:32,639 --> 00:42:35,639
us individually. But now we're also letting the LMS aside

941
00:42:35,920 --> 00:42:39,599
which features we should have, just because they're continuing to

942
00:42:39,679 --> 00:42:41,880
lie to our customers about having that feature, even if

943
00:42:41,880 --> 00:42:43,079
it doesn't necessarily make sense.

944
00:42:43,280 --> 00:42:47,320
Speaker 2: Yes, I think that we are starting to evaluate all

945
00:42:47,360 --> 00:42:50,000
of the dsls and things like that that we expose

946
00:42:50,119 --> 00:42:53,559
by like does this work with cursor or not? So, yeah,

947
00:42:53,639 --> 00:42:56,199
it's it's definitely a new kind of dynamic in that

948
00:42:56,599 --> 00:42:57,679
product management.

949
00:42:57,760 --> 00:43:00,679
Speaker 1: Would it make sense to take your I mean, obviously

950
00:43:00,760 --> 00:43:02,960
you're a small, small team in a small company. You know,

951
00:43:03,000 --> 00:43:05,440
still startup three years old. You know, you have the

952
00:43:05,440 --> 00:43:07,639
whole runway in front of you basically of whether or

953
00:43:07,639 --> 00:43:09,960
not it's going to be successful. So this is probably

954
00:43:09,960 --> 00:43:13,360
incredibly irresponsible of me to ask, But it does seem

955
00:43:13,360 --> 00:43:16,679
like there's a there's another product here where you can

956
00:43:16,800 --> 00:43:22,000
absolutely use your engine building the ASTs to validate the

957
00:43:22,000 --> 00:43:25,280
semantic correctness of the programs that the alums are generating,

958
00:43:25,360 --> 00:43:28,199
and I don't think anyone is doing this effectively right now.

959
00:43:28,320 --> 00:43:30,760
Speaker 2: Yeah, that's definitely something we I mean, even just for

960
00:43:30,840 --> 00:43:35,639
my own stanity, we need to start doing because I

961
00:43:35,760 --> 00:43:38,679
can't keep answering questions about the hallucinated code for the

962
00:43:38,679 --> 00:43:40,679
rest of my life. But I think that kind of

963
00:43:40,719 --> 00:43:43,360
the original inspiration, part of the inspiration for the company

964
00:43:43,440 --> 00:43:45,719
dates back to I can't remember the name of the editor,

965
00:43:45,760 --> 00:43:48,079
but there was that kickstartered editor about ten years ago

966
00:43:48,119 --> 00:43:51,599
where someone wanted to called light Table, where basically people

967
00:43:51,639 --> 00:43:55,280
were like, let's reimagine what an ide could be. I

968
00:43:55,320 --> 00:43:58,239
think a lot of the ideas in there were fascinating, like, Okay,

969
00:43:58,320 --> 00:44:00,480
what if functions were entries in a database instead of

970
00:44:00,519 --> 00:44:03,480
like lines in a file. What if we did track

971
00:44:03,519 --> 00:44:05,480
all the inputs and outputs to functions and productions so

972
00:44:05,519 --> 00:44:07,599
we could show you, like what the distributions were. But also,

973
00:44:07,639 --> 00:44:09,360
if you make a bug, we can tell you immediately

974
00:44:09,360 --> 00:44:11,400
that you're about to ship abug to production. I do

975
00:44:11,480 --> 00:44:13,559
want to get back into the like the core thesis

976
00:44:13,559 --> 00:44:15,519
of Chocolate that we could provide really good data developer

977
00:44:15,519 --> 00:44:17,440
experience for data engineers. And I do want to get

978
00:44:17,440 --> 00:44:19,119
back into the nuts and bolts of like working on

979
00:44:19,159 --> 00:44:21,480
those sorts of problems rather than just you know, fixing

980
00:44:21,519 --> 00:44:24,440
TCP congestion issues admirable.

981
00:44:24,599 --> 00:44:28,119
Speaker 1: Definitely, I approve on solving the technical problems. I mean,

982
00:44:28,119 --> 00:44:30,400
because I feel like in my career it definitely started

983
00:44:30,440 --> 00:44:33,079
out as this lie. We tell ourselves like, we do

984
00:44:33,159 --> 00:44:36,199
software development to solve challenging problems. And I think we

985
00:44:36,280 --> 00:44:39,239
just say that because that was our education experience where

986
00:44:39,280 --> 00:44:41,119
we were forced to solve challenging problems and then we

987
00:44:41,199 --> 00:44:43,559
got credit for it and created this positive feedback loop,

988
00:44:43,599 --> 00:44:46,159
and we thought, that's our personal identity. And then we

989
00:44:46,199 --> 00:44:49,119
go out into the workforce and we keep on repeating that,

990
00:44:49,159 --> 00:44:50,920
and then we learn after some time that we don't

991
00:44:50,960 --> 00:44:55,400
get rewarded or promoted for solving technically challenging problems, and

992
00:44:55,440 --> 00:44:57,119
so we get away from that, and then we have

993
00:44:57,239 --> 00:45:01,760
this lie that transformed into it's all about talking about

994
00:45:01,760 --> 00:45:04,599
our successes, et cetera. Although I think at this point

995
00:45:05,199 --> 00:45:07,960
I'm trying to realize that the original thing isn't such

996
00:45:07,960 --> 00:45:09,960
a lot. Like at a company level, if you build

997
00:45:10,199 --> 00:45:14,719
a great technical solution that solves a real technical problem,

998
00:45:14,800 --> 00:45:16,559
that it will get picked up and start being used,

999
00:45:16,559 --> 00:45:19,119
and the better job you do at it, the more

1000
00:45:19,159 --> 00:45:20,960
people that will flock to it. Because, as you said,

1001
00:45:20,960 --> 00:45:23,079
you like you won't have any churn. You literally are

1002
00:45:23,119 --> 00:45:25,360
the only ones that build this thing and are the

1003
00:45:25,400 --> 00:45:28,199
best at building it. Everyone will come and start using it.

1004
00:45:28,280 --> 00:45:31,440
And that makes me a little bit happier than believing that, yeah,

1005
00:45:32,599 --> 00:45:34,679
some company out there is going to replicate every single

1006
00:45:34,760 --> 00:45:37,719
SaaS product that will ever exists. As if it's like

1007
00:45:37,880 --> 00:45:40,039
it's going to work. Yeah, sure it may look like

1008
00:45:40,079 --> 00:45:41,960
it works, but I don't think it's going to actually

1009
00:45:42,000 --> 00:45:46,239
work until they have chalks newly open sourced. I shouldn't

1010
00:45:46,239 --> 00:45:48,880
start it because it's not open source yet. Would be

1011
00:45:48,920 --> 00:45:52,400
open sourced some part of it ast validator.

1012
00:45:52,519 --> 00:45:54,199
Speaker 2: Yeah, one day we'll Bible to approve all software is

1013
00:45:54,239 --> 00:45:56,039
correct and solve the whole thing problem, but not today.

1014
00:45:56,079 --> 00:45:58,480
Speaker 1: Unfortunately that's not a solvable problem. Actually, and there's a

1015
00:45:58,519 --> 00:46:03,079
great Vertalitia video about the whole at the bottom of math,

1016
00:46:03,159 --> 00:46:06,719
but that's not going to be my pick for this episode.

1017
00:46:06,760 --> 00:46:09,400
So before we get to picks, I'll ask Andrew, is

1018
00:46:09,400 --> 00:46:12,599
there anything else that about chalk or about LMS, or

1019
00:46:12,760 --> 00:46:17,320
about immutable data structures Python interpretability that you want to

1020
00:46:17,360 --> 00:46:18,760
share with our listeners.

1021
00:46:18,800 --> 00:46:21,880
Speaker 2: Oh, mutable data structures, I mean I feel like they're

1022
00:46:21,960 --> 00:46:24,639
a constant siren you always want to use a mutable

1023
00:46:24,679 --> 00:46:28,440
data structure when you are, you know, working in your software,

1024
00:46:28,559 --> 00:46:30,440
because it's got all the properties you want right like,

1025
00:46:30,480 --> 00:46:33,559
you can't mutate it so you can avoid the thread

1026
00:46:33,599 --> 00:46:36,679
safety issues, etcetera, etcetera. But they're just always so slow.

1027
00:46:37,000 --> 00:46:39,519
They're just always so slow, so we've always had to

1028
00:46:39,599 --> 00:46:40,119
rip them out.

1029
00:46:40,320 --> 00:46:43,000
Speaker 1: I would never have guessed that. I mean, I could

1030
00:46:43,000 --> 00:46:45,159
imagine that you're a huge fan of like Haskell.

1031
00:46:45,440 --> 00:46:47,320
Speaker 2: Then I don't have my high schoolook on my desk

1032
00:46:47,400 --> 00:46:49,679
right now. I think way back in high school, like

1033
00:46:49,920 --> 00:46:51,960
senior or high school. I found learning you a high

1034
00:46:51,960 --> 00:46:53,880
school on the Internet or something like that and absolutely

1035
00:46:53,880 --> 00:46:57,400
blew my mind. But I mean, it's the same lie

1036
00:46:57,480 --> 00:46:59,360
that Haigh School has to tell, Like I'm presenting a

1037
00:46:59,440 --> 00:47:03,840
mutable Internferce Spender the hood as beautable data structures. And unfortunately, my.

1038
00:47:03,960 --> 00:47:07,239
Speaker 1: Problem with functional programming was that everyone who tried to

1039
00:47:07,239 --> 00:47:09,559
convince me that functional programming was the best thing ever

1040
00:47:09,920 --> 00:47:12,599
always had the same message, which is, functional programming is

1041
00:47:12,599 --> 00:47:17,440
the best thing ever. There's like monads and immutability, and

1042
00:47:17,920 --> 00:47:20,360
that never sold me, honestly, and I never got there.

1043
00:47:20,440 --> 00:47:23,480
And after years and years of software development, I came

1044
00:47:23,519 --> 00:47:27,239
to the realization that there's a much better marketing sales

1045
00:47:27,280 --> 00:47:32,599
pitch for these languages, especially things like Rust or Haskell,

1046
00:47:32,639 --> 00:47:36,360
where you can actually look at a function and know

1047
00:47:36,920 --> 00:47:40,559
that you've handled every edge case and you don't and

1048
00:47:40,719 --> 00:47:42,760
like that. For me, like this was always something I

1049
00:47:42,760 --> 00:47:46,039
was like, I hate Java, but I appreciate Java because

1050
00:47:46,079 --> 00:47:50,039
it captures the exceptions as part of the function signature.

1051
00:47:50,559 --> 00:47:54,599
And until I realized that functional programming is actually baked

1052
00:47:54,639 --> 00:47:57,639
this notion in by default into the language, it never

1053
00:47:57,679 --> 00:47:59,760
really connected with me. But if anyone had ever said,

1054
00:48:00,000 --> 00:48:02,119
aren't you worried that you're missing some edge cases, and

1055
00:48:02,159 --> 00:48:04,440
I'd be like, yeah, I am totally worried, then I

1056
00:48:04,440 --> 00:48:07,039
would have started with Rust much sooner. I would have

1057
00:48:07,079 --> 00:48:10,400
switched to f sharp, probably earlier in my career, and

1058
00:48:11,199 --> 00:48:14,199
I think I would be happier for it. So people

1059
00:48:14,199 --> 00:48:16,440
who say, yeah, oh Haskell is great for like rule

1060
00:48:16,480 --> 00:48:19,480
engines and stuff like that, you know whatever, But yeah,

1061
00:48:19,559 --> 00:48:21,119
this this edge case for and you know what we

1062
00:48:21,119 --> 00:48:22,960
talked to hear about the ASTs, I think is a

1063
00:48:23,000 --> 00:48:26,719
really good justific justification to spend time to switch out

1064
00:48:26,760 --> 00:48:28,639
and try some of these alternative solutions.

1065
00:48:28,719 --> 00:48:32,119
Speaker 2: Yeah, definitely, I mean checked errors and rust have made

1066
00:48:32,559 --> 00:48:34,599
so many categories of bugs and not happen. I sleep

1067
00:48:34,599 --> 00:48:35,320
a love better at night.

1068
00:48:35,480 --> 00:48:40,199
Speaker 1: I early on in our company started categorizing every single

1069
00:48:40,239 --> 00:48:43,239
possible bug that we ever saw in production or caught

1070
00:48:43,320 --> 00:48:45,599
and didn't expose in production, and like what caused that?

1071
00:48:45,800 --> 00:48:48,480
And there's a number of bugs that are type related

1072
00:48:48,519 --> 00:48:51,000
things which are just like, yes, we had the wrong

1073
00:48:51,039 --> 00:48:53,079
type or it was too loose or whatever, and we

1074
00:48:53,079 --> 00:48:55,400
should fix that by switching to something else. But then

1075
00:48:55,400 --> 00:48:57,199
there's a whole class of things of like, no, it's

1076
00:48:57,239 --> 00:48:59,679
actually not possible to write the validation in this way,

1077
00:48:59,800 --> 00:49:03,039
and switching to a different language would have just prevented

1078
00:49:03,079 --> 00:49:04,280
this from ever happening.

1079
00:49:04,440 --> 00:49:07,480
Speaker 2: Yeah, absolutely, I mean the then DNS is down and

1080
00:49:07,519 --> 00:49:10,719
it didn't matter anyways. It turns out that like DNS

1081
00:49:10,840 --> 00:49:14,000
in Kubernetes is not really a very reliable thing unless

1082
00:49:14,000 --> 00:49:16,039
you spend a lot of time thinking about it. So

1083
00:49:16,519 --> 00:49:18,320
that's been a recurring theme in my life this year.

1084
00:49:18,679 --> 00:49:21,199
Speaker 1: I'm horrified that you could even say those words, and

1085
00:49:21,400 --> 00:49:25,639
I'm even more concerned to ask for more details. We're

1086
00:49:25,679 --> 00:49:27,400
on the cusp of the end of the episode. So

1087
00:49:27,559 --> 00:49:29,320
if it's if it's if it's quick and concise.

1088
00:49:29,480 --> 00:49:31,599
Speaker 2: Uh oh, I don't. I don't know if networking debuggings

1089
00:49:31,599 --> 00:49:34,639
are that quick or concise, but I would everyone's using

1090
00:49:34,639 --> 00:49:38,679
cube DNS still, maybe stop, but also check your scaling policy.

1091
00:49:38,719 --> 00:49:40,639
It's probably underscaled and gk by.

1092
00:49:40,480 --> 00:49:42,000
Speaker 1: De fault and this affects DNS.

1093
00:49:42,039 --> 00:49:44,519
Speaker 2: How if you don't have enough replicas, dns will start

1094
00:49:44,559 --> 00:49:46,559
to time out when you try to resolve names. So

1095
00:49:46,719 --> 00:49:49,119
I mean the classic issues your auto scaling group says, cool,

1096
00:49:49,119 --> 00:49:51,159
tend to double the podcast. Everyone boots up and tries

1097
00:49:51,199 --> 00:49:54,800
to resolve you know, your database IP address, and then

1098
00:49:54,880 --> 00:49:56,880
they fail to resolve your datase hypea address and then

1099
00:49:56,960 --> 00:49:58,639
unfortunate things happened as a consequence.

1100
00:49:58,719 --> 00:50:02,320
Speaker 1: Absolutely amazing. Okay, So with that, let's switch over to

1101
00:50:02,960 --> 00:50:05,280
picks for the episode. Andrew, what did you bring for

1102
00:50:05,360 --> 00:50:05,760
us today?

1103
00:50:06,559 --> 00:50:09,639
Speaker 2: Yeah? So I had a serious answer, but I think

1104
00:50:09,679 --> 00:50:12,119
my silly answer is everyone should get an e bike.

1105
00:50:12,840 --> 00:50:14,679
It's a lot of fun. I would recommend getting one

1106
00:50:14,679 --> 00:50:16,559
of the e bikes, which looks exactly like a regular

1107
00:50:16,639 --> 00:50:18,920
road bike, so that when you're going up a hill,

1108
00:50:19,320 --> 00:50:23,840
all the you know, really extremely serious cyclists get jealous

1109
00:50:23,880 --> 00:50:26,239
and you can hear their gears change as you go by.

1110
00:50:26,280 --> 00:50:27,880
It just goes chunk ka chunk ka, chunk goes. They

1111
00:50:27,880 --> 00:50:28,639
all cry to catch up.

1112
00:50:28,840 --> 00:50:30,639
Speaker 1: You've had this experience multiple times?

1113
00:50:30,840 --> 00:50:33,440
Speaker 2: Yeah, of course, it's my favorite weekend activity.

1114
00:50:33,559 --> 00:50:36,199
Speaker 1: Is this a newfound hobby of yours or one that

1115
00:50:36,440 --> 00:50:38,719
you've been living with since the invention of e bikes?

1116
00:50:38,840 --> 00:50:41,679
Speaker 2: Yeah. So I've actually been really into road cycling for

1117
00:50:41,719 --> 00:50:44,000
a long time, but my wife bought an e bike

1118
00:50:44,039 --> 00:50:46,159
recently so that we could kind of keep pace better

1119
00:50:46,199 --> 00:50:49,239
on rides together. She doesn't want to get super beat

1120
00:50:49,320 --> 00:50:51,719
up and I do, but we've been doing that basically

1121
00:50:51,760 --> 00:50:53,320
every weekend for maybe the past six months.

1122
00:50:53,599 --> 00:50:54,840
Speaker 1: What sort of e bike do you have?

1123
00:50:55,039 --> 00:50:57,159
Speaker 2: She has a truck e bike. It's like a truck

1124
00:50:57,159 --> 00:50:59,119
to money or something like that. But it looks exactly

1125
00:50:59,159 --> 00:51:02,199
like a regular road bike, so it's it passes the

1126
00:51:02,280 --> 00:51:02,880
stealth check.

1127
00:51:03,000 --> 00:51:05,920
Speaker 1: Okay. I was racking my brain what to pick today,

1128
00:51:05,960 --> 00:51:09,000
but since I'm traveling to the US, I decided to

1129
00:51:09,079 --> 00:51:11,599
bring some Swiss chocolate. And I have to say that

1130
00:51:11,639 --> 00:51:14,159
something about Swiss chocolate. Since I'm originally for the United States,

1131
00:51:14,199 --> 00:51:16,119
I thought I just always hated chocolate. I have to

1132
00:51:16,159 --> 00:51:19,440
say that it's because the chocolate is bad, honestly. And

1133
00:51:19,719 --> 00:51:22,360
if you're ever in Switzerland and you come here and

1134
00:51:22,360 --> 00:51:24,039
you're like, I want to get some really great chocolate,

1135
00:51:24,320 --> 00:51:27,039
the number one thing you should not do is go

1136
00:51:27,079 --> 00:51:30,440
to a famous like chocolate manufacturer, or go to a

1137
00:51:30,480 --> 00:51:33,800
special store that's shrouded in controversy. Instead, just like go

1138
00:51:33,880 --> 00:51:36,840
to your local grocery store and just grab a random one.

1139
00:51:36,880 --> 00:51:38,960
It will be absolutely fantastic. So my pick today is

1140
00:51:39,000 --> 00:51:42,000
actually going to be the grocery store Migro, which is

1141
00:51:42,000 --> 00:51:44,480
like one of two major grocery stores in Switzerland, and

1142
00:51:44,480 --> 00:51:46,119
they have a special brand called Frey and this is

1143
00:51:46,159 --> 00:51:49,119
like the pistachio version, and it's like, honestly the best

1144
00:51:49,199 --> 00:51:51,000
chocolate I've ever had. And it's like you just go

1145
00:51:51,000 --> 00:51:52,559
to the store and you just buy it. It's there's

1146
00:51:52,599 --> 00:51:55,800
no like special dance you have to do. It's not

1147
00:51:55,840 --> 00:51:57,559
like a special company or anything. And the interesting thing

1148
00:51:57,599 --> 00:52:01,199
is like their brand that actually produces this is like

1149
00:52:01,280 --> 00:52:05,320
totally social conscious and concerned about environmental protection laws and

1150
00:52:06,199 --> 00:52:09,039
where the cocoa has grown and manufactured and assembled and

1151
00:52:09,079 --> 00:52:11,440
everything like that, so you can be sure that no

1152
00:52:11,599 --> 00:52:12,719
corners are being cut there.

1153
00:52:12,800 --> 00:52:14,719
Speaker 2: Well, I got a mood to Special Land. It's the dream.

1154
00:52:14,880 --> 00:52:18,199
Speaker 1: The second best time is now. All right, So that's

1155
00:52:18,239 --> 00:52:21,559
it for today's episode. Thank you Andrew for coming and

1156
00:52:21,639 --> 00:52:24,880
joining us and talking through all those really interesting technical

1157
00:52:24,960 --> 00:52:25,800
challenges that you had.

1158
00:52:25,920 --> 00:52:27,760
Speaker 2: Of course, thanks Warren, I really appreciate you having me.

1159
00:52:27,880 --> 00:52:30,000
Speaker 1: Yeah, I mean, this has been a great conversation and

1160
00:52:30,159 --> 00:52:35,000
we'll be back again next week.