1
00:00:07,719 --> 00:00:10,359
Speaker 1: Hello everyone, and welcome back to another episode of Adventures

2
00:00:10,400 --> 00:00:13,800
and DevOps. You can see today I'm flying solo slight

3
00:00:13,919 --> 00:00:17,719
replacement instead of I have a fact Plorox sues Cognizant

4
00:00:18,160 --> 00:00:21,160
for a ridiculous amount of money because they outsource their

5
00:00:21,199 --> 00:00:25,800
customer support and cognitives like, hey, hackers, you want access

6
00:00:25,839 --> 00:00:28,839
to Chlorox, no problem. Here are their passwords. I don't know,

7
00:00:28,920 --> 00:00:31,000
really interesting read. It will be in the podcast description

8
00:00:31,039 --> 00:00:34,159
after the episode. But today I'm really looking forward to

9
00:00:34,200 --> 00:00:37,640
our episode. I have general manager of Plumy with me here,

10
00:00:37,799 --> 00:00:38,640
Megan Kojakar.

11
00:00:38,759 --> 00:00:40,560
Speaker 2: Welcome, Hi, Thanks Warren.

12
00:00:40,600 --> 00:00:42,600
Speaker 1: It's nice to be I saw you have a really

13
00:00:42,600 --> 00:00:45,679
strong product background. Before moving into the general manager role

14
00:00:45,679 --> 00:00:48,359
at Plumy, you were a senior product manager. What does

15
00:00:48,399 --> 00:00:49,359
a general manager do?

16
00:00:49,640 --> 00:00:52,840
Speaker 3: It's funny because LinkedIn gives me ads for working at

17
00:00:52,880 --> 00:00:56,560
McDonald's and car dealerships, So I feel like I have

18
00:00:56,640 --> 00:01:01,159
good career progression in that sense. So Plimian, I think

19
00:01:01,200 --> 00:01:02,479
when I was at a Tobs is kind of a

20
00:01:02,520 --> 00:01:05,040
similar model. A general manager is kind of like the

21
00:01:05,079 --> 00:01:08,120
owner of an entire product area, so that means everything

22
00:01:08,159 --> 00:01:12,879
from product to engineering to you know, I have documentation

23
00:01:12,959 --> 00:01:16,359
folks have data engineers, and it's kind of the whole

24
00:01:16,400 --> 00:01:20,200
product is under one organizational structure, and you're the point

25
00:01:20,239 --> 00:01:23,760
person for it, everything from pricing to interacting with sales,

26
00:01:23,799 --> 00:01:26,680
to working with marketing, and so it's kind of I

27
00:01:26,680 --> 00:01:29,439
think a model where you have really high ownership and

28
00:01:29,480 --> 00:01:34,400
you want product teams to operate pretty sufficiently within their

29
00:01:34,519 --> 00:01:38,239
organizational structure. And it's pretty common at a TOBS. Folks

30
00:01:38,280 --> 00:01:40,280
always joke that a TOBS is just a collection of

31
00:01:40,280 --> 00:01:43,599
startups and that's part of them building their org structure

32
00:01:43,599 --> 00:01:46,000
of just being able to let people run within their

33
00:01:46,079 --> 00:01:46,680
little Yeah.

34
00:01:46,680 --> 00:01:50,439
Speaker 1: I mean, I've heard the title often like managing director

35
00:01:50,640 --> 00:01:53,680
for a line of business, for a particular whole business unit.

36
00:01:54,239 --> 00:01:56,640
So I mean, I think it's much smarter than a

37
00:01:56,719 --> 00:01:58,959
lot of these companies that have a bunch of senior

38
00:01:59,599 --> 00:02:02,359
technical folks and they just have them all report directly

39
00:02:02,439 --> 00:02:07,000
the CEO or some leadership team. But really identifying that

40
00:02:07,400 --> 00:02:10,840
you're fundamentally responsible for this whole product area you're making

41
00:02:10,840 --> 00:02:14,919
all the critical decisions, really shows that there's an alignment

42
00:02:15,080 --> 00:02:18,680
of how critical it is and who's accountable for the

43
00:02:18,800 --> 00:02:19,840
success of the organization.

44
00:02:19,919 --> 00:02:21,719
Speaker 3: Yeah, I think in Poluma's case, it was really about,

45
00:02:21,800 --> 00:02:24,879
like we want it to feel like a small startup

46
00:02:24,879 --> 00:02:27,199
inside a bigger startup, if that makes sense, and have

47
00:02:27,560 --> 00:02:29,400
folks be able to all be aligned on the same

48
00:02:29,479 --> 00:02:33,719
goals across many different code bases. And there's president cons

49
00:02:33,719 --> 00:02:35,800
to any model, but so far, I think there's a

50
00:02:35,840 --> 00:02:36,439
lot of benefits.

51
00:02:36,520 --> 00:02:37,800
Speaker 1: Yeah, and I don't think you have to be worried

52
00:02:37,840 --> 00:02:40,639
about your career progression because if you leave Polumi, I've

53
00:02:40,680 --> 00:02:43,080
heard there's a general manager role open at like a

54
00:02:43,080 --> 00:02:46,439
lot of large sports organization. The GM is often like

55
00:02:46,599 --> 00:02:49,800
you know, running the football team for instance, for you know,

56
00:02:49,800 --> 00:02:53,199
a multi billion dollar organization. So you know, there's definitely

57
00:02:53,240 --> 00:02:55,759
up I guess or I guess side grade depending you know,

58
00:02:55,840 --> 00:02:59,240
moving from SaaS to sports. But yeah, you don't have

59
00:02:59,280 --> 00:03:02,759
to become as.

60
00:03:01,800 --> 00:03:04,599
Speaker 3: I'm Canadian and I play ice hockey, so I love

61
00:03:04,639 --> 00:03:05,960
hearing that. That's great news.

62
00:03:07,360 --> 00:03:09,360
Speaker 1: We're going a fight about the best sports teams. Then

63
00:03:09,680 --> 00:03:11,240
I have to ask, like, what made you want to

64
00:03:11,280 --> 00:03:13,879
shift from being a senior product manager at a AWS,

65
00:03:13,960 --> 00:03:15,879
which I think you were like in the database space,

66
00:03:16,159 --> 00:03:19,080
to eventually become the GM at Bloom was there like

67
00:03:19,199 --> 00:03:21,919
already writing at the wall like four years ago or

68
00:03:21,960 --> 00:03:24,280
so that you just like I have to leave or

69
00:03:24,479 --> 00:03:25,639
was it something about your career.

70
00:03:25,759 --> 00:03:27,960
Speaker 3: I loved my time at ABS in a lot of ways.

71
00:03:28,000 --> 00:03:30,439
I feel like, especially for a product manager, going to

72
00:03:30,560 --> 00:03:33,639
Amazon is like going to college, like you're going to

73
00:03:33,680 --> 00:03:36,240
like PM school, because it's like very regimented and how

74
00:03:36,240 --> 00:03:38,919
they build product there, and you learn so much, you

75
00:03:38,960 --> 00:03:42,400
know the infamous, like every doc you right getting reviewed

76
00:03:42,599 --> 00:03:44,879
in a a like in a synchronous meeting where someone's

77
00:03:44,919 --> 00:03:47,080
like redlining it in front of you. Truly like going

78
00:03:47,120 --> 00:03:49,520
to school. But I started my career at a startup.

79
00:03:49,560 --> 00:03:52,280
I kind of fell into founding a startup. I was

80
00:03:52,319 --> 00:03:55,960
the first employee, and that was like an amazing roller coaster.

81
00:03:56,039 --> 00:03:58,000
I think we had like forty employees when I left,

82
00:03:58,479 --> 00:04:00,240
and I was there for four years, and I ended

83
00:04:00,319 --> 00:04:01,800
up in a similar role to what I'm in now,

84
00:04:01,960 --> 00:04:04,759
leading products and engineering there. And so I knew I

85
00:04:04,800 --> 00:04:06,680
wanted to try the big company thing. And you hear

86
00:04:06,719 --> 00:04:09,240
a lot about fang companies and what it's like to

87
00:04:09,240 --> 00:04:11,520
work at them, and so moved to Seattle, did the

88
00:04:11,520 --> 00:04:14,000
A OFS thing, and like I said, I learned a lot,

89
00:04:14,039 --> 00:04:17,759
but ultimately I missed having a really high amount of ownership,

90
00:04:18,040 --> 00:04:20,360
and at an organization like Aight of Us, you have

91
00:04:20,480 --> 00:04:22,959
so many talented leaders around you, and I learned a

92
00:04:22,959 --> 00:04:24,800
lot from them. But it also means you have just

93
00:04:24,839 --> 00:04:27,560
like small scope in nature, but we had so many

94
00:04:27,560 --> 00:04:29,800
customers and so much data. And one of the things

95
00:04:29,800 --> 00:04:32,399
that made you really excited about Pollumi is being open source.

96
00:04:32,439 --> 00:04:34,759
It has a ton of users, and so you're able

97
00:04:34,800 --> 00:04:36,879
to get really quick signal on like what your users

98
00:04:36,920 --> 00:04:38,720
are interested in and like, hey, did we build the

99
00:04:38,800 --> 00:04:41,759
right thing? And I knew I would miss that at

100
00:04:41,800 --> 00:04:45,040
as is so nice. You know, you have unlimited amounts

101
00:04:45,079 --> 00:04:47,000
of data on like what customers are doing with it

102
00:04:47,040 --> 00:04:49,319
and like how users are using your product, and so

103
00:04:49,680 --> 00:04:51,160
moving to Plumi, I knew I wanted to go to

104
00:04:51,199 --> 00:04:53,120
a space that was smaller and I had a lot

105
00:04:53,160 --> 00:04:56,480
more autonomy and ownership of the area. But I didn't

106
00:04:56,480 --> 00:04:58,759
want to give up on having that signal. And so

107
00:04:59,319 --> 00:05:01,920
it's it's been amazing like working an open source company

108
00:05:01,959 --> 00:05:04,759
where anyone could open a GitHub issue and that is

109
00:05:04,800 --> 00:05:07,959
your roadmap, and I really have liked those challenges.

110
00:05:08,079 --> 00:05:10,319
Speaker 1: Well, there's something definitely to be said being careful about

111
00:05:10,439 --> 00:05:12,839
responding to every issue as if it's the most critical

112
00:05:12,879 --> 00:05:14,839
thing for all your customers. But I totally get that

113
00:05:14,839 --> 00:05:17,759
with AWS, you're limited on the number of open source

114
00:05:18,079 --> 00:05:20,560
technologies and they're not the core business and unless you're

115
00:05:20,600 --> 00:05:24,639
in a technical account manager role or a solution architects side, like,

116
00:05:24,639 --> 00:05:27,000
you're not as close to the actual challenges that customers

117
00:05:27,040 --> 00:05:28,720
are having at that moment, even though you're in a

118
00:05:28,720 --> 00:05:30,720
product focused responsibility.

119
00:05:30,800 --> 00:05:33,040
Speaker 3: Yeah, they definitely do a good job of like bringing

120
00:05:33,079 --> 00:05:35,360
that into their DNA, both by having all the data

121
00:05:35,480 --> 00:05:38,639
that the folks interacting with customers bring in, but also

122
00:05:39,240 --> 00:05:41,240
product managers of a to BS talk to customers a ton,

123
00:05:42,040 --> 00:05:44,240
but pluently being a smaller company and like having this

124
00:05:44,319 --> 00:05:47,079
huge community, it's amazing, Like we meet with I meet

125
00:05:47,079 --> 00:05:49,319
with multiple customers a week, and I feel like that's

126
00:05:49,439 --> 00:05:51,879
what makes our jobs great, right, is like seeing the

127
00:05:51,920 --> 00:05:54,399
person using your product and like how they use it.

128
00:05:54,399 --> 00:05:56,079
It makes it so much more real. You feel the

129
00:05:56,120 --> 00:05:57,279
impact of what you're working on.

130
00:05:57,680 --> 00:06:00,000
Speaker 1: Yeah, having worked in some bigger companies, I always felt

131
00:06:00,120 --> 00:06:03,040
slighted a bit when I had intermediaries between me and

132
00:06:03,079 --> 00:06:06,399
the actual customers, Like, I trust you are conveying the

133
00:06:06,439 --> 00:06:08,360
right information to me, but I really want to hear

134
00:06:08,399 --> 00:06:10,959
it from their mouth exactly what they're saying, because there's

135
00:06:10,959 --> 00:06:13,720
definitely things lost in the telephone game and just some

136
00:06:13,879 --> 00:06:17,160
nuances or priorities, et cetera that they're not really sharing.

137
00:06:17,279 --> 00:06:19,560
And unless you really have someone that's a really great

138
00:06:19,600 --> 00:06:23,199
communicator and understands both the product side and also the

139
00:06:23,199 --> 00:06:26,360
customer side in that responsibility, you're definitely going to lose

140
00:06:26,399 --> 00:06:28,680
things there, and then you're gonna get You're gonna have

141
00:06:28,720 --> 00:06:30,160
an issue in the long run when you end up

142
00:06:30,160 --> 00:06:32,680
building not exactly quite the right thing and then have

143
00:06:32,720 --> 00:06:34,480
to go back to the drawing board to actually deliver

144
00:06:34,560 --> 00:06:35,240
the real value.

145
00:06:35,319 --> 00:06:37,560
Speaker 3: Yeah, what's interesting, It's like open source helps with that too, right,

146
00:06:37,639 --> 00:06:39,759
that game of telephone, because it's like you're going from

147
00:06:39,800 --> 00:06:42,399
the community member straight to the engineering team. You know,

148
00:06:42,480 --> 00:06:45,839
we have zero in intermediary between whoever's opening the issue

149
00:06:45,839 --> 00:06:48,279
and the engineer working on it, so that's really nice.

150
00:06:48,319 --> 00:06:51,040
They just interact directly. They're like, Hey, I can't repro this,

151
00:06:51,160 --> 00:06:53,000
can you please give me more info? I?

152
00:06:53,079 --> 00:06:55,680
Speaker 1: Oh wow, I'm like on both sides of this because

153
00:06:55,720 --> 00:06:58,279
we have our products is totally proprietary, but we have

154
00:06:58,279 --> 00:07:00,519
lots of open source SDKs and whatnot, and we always

155
00:07:00,560 --> 00:07:02,560
get questions about open sourcing it, and I think we're

156
00:07:02,560 --> 00:07:05,040
going to have an open source something in the near future,

157
00:07:05,439 --> 00:07:08,360
but it like I have this fear of there's just

158
00:07:08,399 --> 00:07:11,199
like there's a lot of issues that pop up that

159
00:07:11,319 --> 00:07:13,879
are just you know, basically support tickets.

160
00:07:13,920 --> 00:07:14,879
Speaker 4: Can we have help with this?

161
00:07:14,959 --> 00:07:17,680
Speaker 1: Can we do this thing and don't necessarily align with

162
00:07:18,639 --> 00:07:20,920
like a long term direction that we want to go

163
00:07:21,079 --> 00:07:22,480
or would be even benefit.

164
00:07:22,120 --> 00:07:23,199
Speaker 4: Official for your users?

165
00:07:23,240 --> 00:07:25,959
Speaker 1: How do you filter out like the number of support

166
00:07:25,959 --> 00:07:28,439
requests you're getting to actually make sure that the issues

167
00:07:28,439 --> 00:07:31,600
on your GitHub are useful both for them and for

168
00:07:31,879 --> 00:07:32,600
Blooming as well.

169
00:07:32,720 --> 00:07:34,519
Speaker 3: That's interesting because we definitely get some of that, but

170
00:07:34,560 --> 00:07:36,959
it's not a huge problem for us. And I think

171
00:07:37,000 --> 00:07:38,680
part of it is we have a very active Slack

172
00:07:38,759 --> 00:07:41,879
community and so a lot of our community support happens there.

173
00:07:42,240 --> 00:07:47,000
And it's so cool, like, like users helping users is

174
00:07:47,079 --> 00:07:49,959
like the funnest interaction model because you know, they have

175
00:07:50,000 --> 00:07:52,000
similar challenges at the same time, and so they're meeting

176
00:07:52,040 --> 00:07:54,399
each other at the same phase and they they're much

177
00:07:54,439 --> 00:07:56,560
more empathetic, right, They're like, hey, I just went through this,

178
00:07:56,639 --> 00:07:59,040
Like let me help you versus an engineer who works

179
00:07:59,040 --> 00:08:00,639
on it who's like, oh, I don't know how you

180
00:08:00,680 --> 00:08:03,040
run into this. I mean, what's interesting about what you

181
00:08:03,079 --> 00:08:04,959
said there is like part of it is how you

182
00:08:05,000 --> 00:08:07,639
develop product in that if you over an index on

183
00:08:07,920 --> 00:08:10,399
like the really noisy customer or a customer that needs

184
00:08:10,439 --> 00:08:13,879
a lot of support, and build something very useful to them,

185
00:08:13,879 --> 00:08:16,240
but it's kind of custom and therefore not generic for

186
00:08:16,279 --> 00:08:18,839
other customers. I mean that is like a huge part

187
00:08:18,839 --> 00:08:20,879
of building product is thinking about how do I make

188
00:08:20,879 --> 00:08:23,839
sure that this is an investment that other customers have

189
00:08:23,920 --> 00:08:26,439
that pain point instead of just like building some super

190
00:08:26,560 --> 00:08:29,680
custom thing for one customer. And like you know, Plummy

191
00:08:29,720 --> 00:08:32,279
has some massive, massive customers and it's hard at times

192
00:08:32,279 --> 00:08:35,080
not to get pulled into that direction. But I feel

193
00:08:35,080 --> 00:08:37,559
like it's often just understanding the root of things instead

194
00:08:37,559 --> 00:08:40,000
of the solution itself. And so a lot of times,

195
00:08:40,600 --> 00:08:42,519
either in a gihub issue or talking to your customer,

196
00:08:42,600 --> 00:08:44,639
they'll say, hey, we need this, but it's actually a solution,

197
00:08:44,720 --> 00:08:46,440
and you have to figure out, all right, how to

198
00:08:46,519 --> 00:08:48,440
do you what is the actual pain point here? And

199
00:08:48,480 --> 00:08:50,279
then once you get to that, often that is the

200
00:08:50,320 --> 00:08:52,600
same thing with other customers. It's just the solution might

201
00:08:52,639 --> 00:08:53,080
be different.

202
00:08:53,200 --> 00:08:55,600
Speaker 1: Just have to worry about just as much about other

203
00:08:55,639 --> 00:08:58,559
customers when they are throwing money at trying to get

204
00:08:58,559 --> 00:09:02,200
you to solve the problems as if you were have.

205
00:09:02,200 --> 00:09:03,440
Speaker 4: Only proprietary code based.

206
00:09:03,480 --> 00:09:06,000
Speaker 1: So like that problem obviously doesn't go away, but it's

207
00:09:06,080 --> 00:09:07,759
I feel I do agree with you. There is something

208
00:09:07,799 --> 00:09:10,279
about having an open source product where you're able to

209
00:09:10,279 --> 00:09:13,279
build a community around it to have those conversations happen,

210
00:09:13,320 --> 00:09:15,320
whereas with a proprietary product.

211
00:09:15,320 --> 00:09:16,840
Speaker 4: It does especially.

212
00:09:16,440 --> 00:09:19,519
Speaker 1: Feel like customers that you have aren't interested in really

213
00:09:19,519 --> 00:09:21,639
talking with each other. And I don't know if it's

214
00:09:22,279 --> 00:09:25,919
just the result of the culture around it or whether

215
00:09:26,039 --> 00:09:28,720
or not there is is just necessary Like when you

216
00:09:28,720 --> 00:09:31,639
have something open source, people expect that they can communicate

217
00:09:31,679 --> 00:09:33,840
and they want to chat and the types of in

218
00:09:33,879 --> 00:09:37,200
your case, engineers users that come on board are thinking

219
00:09:37,200 --> 00:09:40,000
about the community. I assume, of course, having open source

220
00:09:40,519 --> 00:09:43,200
fundamental as your you know, the core of your product,

221
00:09:43,279 --> 00:09:45,919
it's intentional. Any thought about how that would have driven

222
00:09:45,960 --> 00:09:48,679
the community like this like a nice win where you

223
00:09:48,799 --> 00:09:51,320
saw that happening, or was this like a huge expectation

224
00:09:51,440 --> 00:09:54,039
that went into how you were designing how the business

225
00:09:54,039 --> 00:09:54,559
would work well.

226
00:09:54,600 --> 00:09:56,720
Speaker 3: Ploye's been around for eight years and it's been open

227
00:09:56,759 --> 00:09:58,200
source a whole time, and I've only been at the

228
00:09:58,200 --> 00:09:59,840
company like four years, so I feel like I missed

229
00:09:59,840 --> 00:10:03,120
some these like core decisions on like let's go open source,

230
00:10:03,159 --> 00:10:04,720
and like I don't know, did they have in their

231
00:10:04,759 --> 00:10:06,399
head that this would all happen, that we would have

232
00:10:06,559 --> 00:10:09,240
like hundreds of thousands of users who are helping each other,

233
00:10:09,320 --> 00:10:13,200
Like probably not right, I think that's that's likely, Like

234
00:10:13,279 --> 00:10:16,080
wasn't part of the strategy, but it's very much part

235
00:10:16,080 --> 00:10:19,320
of pluming strategy to be open source. And it's interesting,

236
00:10:19,399 --> 00:10:22,080
like as the side we could talk about like the

237
00:10:22,159 --> 00:10:24,600
industry and open source right now, because there was kind

238
00:10:24,639 --> 00:10:26,360
of a huge boom during the time I was at

239
00:10:26,360 --> 00:10:29,080
a tows where we had like you know, Elastic Search

240
00:10:29,120 --> 00:10:31,679
and like all of these products who were open source communities,

241
00:10:32,000 --> 00:10:34,480
you know, move away from a patch of two licensing

242
00:10:34,799 --> 00:10:38,600
and in like our space, you know, Hashi court ter

243
00:10:38,759 --> 00:10:42,279
form moving away from being open source to a BUSL license.

244
00:10:42,639 --> 00:10:45,360
It's been a huge shift, Like I can I feel

245
00:10:45,360 --> 00:10:47,080
like I can count on like one hand how many

246
00:10:47,120 --> 00:10:50,200
companies are still fully open source. And it's been it

247
00:10:50,279 --> 00:10:52,919
used to be like completely normal and now it's it's

248
00:10:52,960 --> 00:10:56,279
really changed, and it's been interesting seeing the market adapt

249
00:10:56,279 --> 00:10:58,159
to that. And we have a lot of customer distrust

250
00:10:58,240 --> 00:11:00,639
because of all this, where they're like, what if you

251
00:11:00,679 --> 00:11:02,399
what if you go close doors tomorrow? And what if

252
00:11:02,480 --> 00:11:04,840
you know all future releases I need a license for

253
00:11:04,879 --> 00:11:07,440
and I have to pay you for And how do

254
00:11:07,480 --> 00:11:09,720
you You can't promise them the future, right, how do

255
00:11:09,759 --> 00:11:11,879
you say, like no, no, no, will be different, Like

256
00:11:11,919 --> 00:11:13,679
I know everyone else said that, but will be different.

257
00:11:13,720 --> 00:11:16,240
And so we've been really intentional about just talking about

258
00:11:16,240 --> 00:11:19,039
our why of why open source means so much to

259
00:11:19,080 --> 00:11:22,159
our business and like our founders are like huge lifetime

260
00:11:22,200 --> 00:11:25,639
believers in it. But it's it's a tough one, like

261
00:11:26,440 --> 00:11:27,519
there's no assurances.

262
00:11:27,879 --> 00:11:29,919
Speaker 1: I feel like there's actually an easy answer there is,

263
00:11:29,960 --> 00:11:32,320
Like we like, we're not stupid. We can see what

264
00:11:32,399 --> 00:11:36,360
happened with reddis and Mango dB and Elastic Search and

265
00:11:37,840 --> 00:11:40,360
Hashi kar By doing this, and like what they're happening

266
00:11:40,360 --> 00:11:43,480
to their user base. Well, they're all like, not only

267
00:11:43,519 --> 00:11:45,799
are their competitors that spawned up in every single one

268
00:11:45,799 --> 00:11:49,639
of those examples, the communities for those all boomed, right

269
00:11:49,960 --> 00:11:53,320
like Valki now is now seen as like not not

270
00:11:53,360 --> 00:11:56,559
just a replacement, but really total successor of reddis in

271
00:11:56,600 --> 00:11:58,960
every way, even though they've walked back on the license.

272
00:11:59,080 --> 00:12:02,039
Speaker 4: I think walked back on the license. I don't know,

273
00:12:02,080 --> 00:12:03,759
we're open sourced really went there.

274
00:12:03,799 --> 00:12:06,480
Speaker 1: So maybe that's not the best example, but like the

275
00:12:06,519 --> 00:12:09,600
CNCF has, you know, totally bought into open TOFU. And

276
00:12:09,639 --> 00:12:12,759
we see a lot of the guests that we have

277
00:12:12,799 --> 00:12:14,399
on the show, you know, they talk about open TOFU.

278
00:12:14,480 --> 00:12:16,639
Now we see in the communities it's open TOFU much

279
00:12:16,720 --> 00:12:17,480
less terror form.

280
00:12:17,600 --> 00:12:18,000
Speaker 4: So so.

281
00:12:19,480 --> 00:12:21,720
Speaker 1: Yeah, I mean you say you bring these up as examples, like, yeah,

282
00:12:21,799 --> 00:12:24,000
we see this, like we know what will happen.

283
00:12:24,120 --> 00:12:26,039
Speaker 4: If this happens, someone will just be like, yep, we're

284
00:12:26,039 --> 00:12:27,919
forking it and that's the end of the story.

285
00:12:28,039 --> 00:12:31,600
Speaker 1: So you realize that it's you're not providing the benefit

286
00:12:31,639 --> 00:12:34,960
to your users, but because it's open source, they're making

287
00:12:35,360 --> 00:12:38,320
it possible for you to run a business. And I

288
00:12:38,320 --> 00:12:40,480
think that's a that's an answer that everyone should be

289
00:12:40,480 --> 00:12:41,200
comfortable hearing.

290
00:12:41,279 --> 00:12:44,440
Speaker 3: You know, it's interesting. It's so I really like hearing

291
00:12:44,440 --> 00:12:47,159
that from your perspective because you're like seeing it from

292
00:12:47,279 --> 00:12:49,240
the outside end. And I feel like sometimes working in

293
00:12:49,279 --> 00:12:52,080
the industry, you see such a different lens and so

294
00:12:52,159 --> 00:12:55,679
that's really fascinating. To give you an interesting example of

295
00:12:55,759 --> 00:12:58,360
like how it could be hard to see it in

296
00:12:58,399 --> 00:13:02,080
that way. Specifically around the open tofu stuff is recently,

297
00:13:02,159 --> 00:13:04,080
like a month ago, we launched so that you can

298
00:13:04,159 --> 00:13:06,919
run a terraform or open tofu module within Plumi, so

299
00:13:06,919 --> 00:13:09,679
you don't have to convert everything right away, which is

300
00:13:09,720 --> 00:13:12,080
really exciting. And so I was demoing it to customers

301
00:13:12,200 --> 00:13:16,639
and I had a handful of like large companies not

302
00:13:16,759 --> 00:13:19,080
know what open tofu was, and we were like, oh,

303
00:13:19,200 --> 00:13:21,559
are we like way too close to it where we

304
00:13:21,559 --> 00:13:24,440
think that this is like the norm now. And there's

305
00:13:24,440 --> 00:13:27,120
so many people who, like, you know, their job, they've

306
00:13:27,120 --> 00:13:29,679
been doing terraform for like ten years and they have

307
00:13:29,840 --> 00:13:32,000
never heard about open tofu. Were like knew about all

308
00:13:32,039 --> 00:13:35,240
of this, and it's it's just fascinating because you think

309
00:13:35,679 --> 00:13:37,639
there's a lot of people who are like always, you know,

310
00:13:37,759 --> 00:13:39,519
reading the news and like staying up with tech news.

311
00:13:39,559 --> 00:13:41,559
But that's not everyone, right, Like sometimes it's your job,

312
00:13:41,559 --> 00:13:43,039
and sometimes you just go to your job and use

313
00:13:43,080 --> 00:13:45,600
the tools that they use, you know, and the stuffitly

314
00:13:45,600 --> 00:13:47,480
not a bad thing. That's just a different mode of operation.

315
00:13:47,720 --> 00:13:50,919
And it was a really good learning for us because

316
00:13:50,919 --> 00:13:52,720
we had kind of planned to launch it being like, oh,

317
00:13:52,759 --> 00:13:55,600
open tofu, you can use it within Plumi, and then

318
00:13:55,720 --> 00:13:58,200
actually talking to folks inside large companies they were like,

319
00:13:58,240 --> 00:14:00,759
what's open tofu And so we pivoted feel like terrorforma

320
00:14:00,799 --> 00:14:03,639
and open Tofu in some of our messaging. But it

321
00:14:03,679 --> 00:14:05,840
is interesting to hear you say like, oh, that's you know,

322
00:14:06,000 --> 00:14:10,960
that's the default now, because it's not always so black

323
00:14:10,960 --> 00:14:11,360
and white.

324
00:14:11,639 --> 00:14:12,519
Speaker 4: Yeah, no, for sure.

325
00:14:12,759 --> 00:14:16,360
Speaker 1: I mean I like being like as much on one

326
00:14:16,440 --> 00:14:18,120
end of the spectrum as possible because I feel like

327
00:14:18,159 --> 00:14:19,720
if I go down that way, there's a lot of

328
00:14:19,720 --> 00:14:21,440
people on the other side of the spectrum, more somewhere

329
00:14:21,440 --> 00:14:23,720
in the middle, and so it helps shift people in

330
00:14:23,759 --> 00:14:26,360
that direction. I think open Tofu is great for the

331
00:14:26,879 --> 00:14:31,080
whole ecosystem compared to having used terraforma. I feel like,

332
00:14:31,360 --> 00:14:33,799
and you brought this up earlier in the episode, basically

333
00:14:34,639 --> 00:14:37,399
how you manage the get up issues Like I remember

334
00:14:37,559 --> 00:14:40,559
years and years ago I had filed problems there, like

335
00:14:40,679 --> 00:14:43,360
real issues sometimes with poor request and this is how

336
00:14:43,360 --> 00:14:46,200
I completely change my whole approach to doing software development

337
00:14:46,240 --> 00:14:48,440
when I was still doing it for open source stuff.

338
00:14:48,519 --> 00:14:49,519
Speaker 4: You open the issue first.

339
00:14:49,559 --> 00:14:51,639
Speaker 1: You don't do any work, don't don't create a poor

340
00:14:51,720 --> 00:14:53,559
QUTS first, just open the ticket and see if a

341
00:14:53,639 --> 00:14:57,480
human response, because if it's a hashy corporate repository, after

342
00:14:57,600 --> 00:15:00,200
thirty days, there'll be a blot that comes and says, hey, hey,

343
00:15:00,320 --> 00:15:02,440
no one's messaged on this, We're going to auto close

344
00:15:02,480 --> 00:15:06,200
this ticket for you. And I'm like, yeah, like fix

345
00:15:06,240 --> 00:15:09,559
your process. Honestly, this is clearly a bug. No one

346
00:15:09,559 --> 00:15:11,360
cares about it, and so it auto gets closed. Like

347
00:15:11,399 --> 00:15:13,679
that's a huge problem, and so what you You make

348
00:15:13,720 --> 00:15:15,639
a message there to keep it open, and again and again.

349
00:15:15,759 --> 00:15:18,559
After a while you're just like I'm done. So, you know,

350
00:15:18,679 --> 00:15:22,360
having a mature response to how to handle your issues

351
00:15:22,399 --> 00:15:23,720
that show up and get up, you know, like that's

352
00:15:23,759 --> 00:15:26,240
really critical and so like it's really great to hear that.

353
00:15:26,360 --> 00:15:26,519
Speaker 4: You know.

354
00:15:26,559 --> 00:15:30,240
Speaker 1: It's really interesting though about the bias of like how

355
00:15:30,279 --> 00:15:33,240
close we are to things we're in the quote unquote

356
00:15:33,279 --> 00:15:33,840
security demain.

357
00:15:33,840 --> 00:15:36,120
Speaker 4: I mean, we do log in and access control, we.

358
00:15:36,039 --> 00:15:39,279
Speaker 1: Generate JWTs for our users, and we have to remember

359
00:15:39,559 --> 00:15:42,559
just don't bring up competitors ever. Actually, because while we

360
00:15:42,639 --> 00:15:46,200
know every single competitor that's in the market, and there's

361
00:15:46,279 --> 00:15:49,360
like another one like every single day. Our users never

362
00:15:49,440 --> 00:15:52,039
do and they don't care because they don't find these

363
00:15:52,279 --> 00:15:53,960
and so it doesn't make sense to even really like

364
00:15:53,960 --> 00:15:54,960
talk about it most of the time.

365
00:15:55,039 --> 00:15:57,759
Speaker 2: That's interesting, So you.

366
00:15:57,720 --> 00:16:00,279
Speaker 1: Know, that's one perspective. I mean, if you know, you

367
00:16:00,320 --> 00:16:02,200
obviously have customers coming to you and be like, yeah,

368
00:16:02,240 --> 00:16:05,200
you know, there's a huge challenge getting and I think

369
00:16:05,200 --> 00:16:06,840
this is one of the challenges that actually we went

370
00:16:06,840 --> 00:16:07,360
through with Plume.

371
00:16:07,440 --> 00:16:08,960
Speaker 4: It's like it is a challenge to manage.

372
00:16:09,000 --> 00:16:12,279
Speaker 1: You know, we have like nine or ten language SDKs,

373
00:16:12,320 --> 00:16:15,960
Like getting a provider SDK in every single language is

374
00:16:16,000 --> 00:16:19,279
just something that's just a huge amount of overhead and

375
00:16:19,399 --> 00:16:24,360
being able to pull already existing go and like the

376
00:16:24,399 --> 00:16:27,879
propository that's running your terraform or open tofu in it's

377
00:16:27,919 --> 00:16:31,240
just a huge win for the users and like the

378
00:16:31,320 --> 00:16:33,559
end users, it's a challenge with the open sided platform

379
00:16:33,879 --> 00:16:36,759
or multi sided. So yeah, I mean you're in a

380
00:16:36,759 --> 00:16:39,120
weird space there, right. Obviously you need to admit that

381
00:16:39,159 --> 00:16:41,320
there's multiple pieces here that are at play and some

382
00:16:41,440 --> 00:16:43,399
users care about but you know, it like depends on

383
00:16:43,399 --> 00:16:47,639
that type of conversation, and so yeah, I mean it's

384
00:16:47,720 --> 00:16:49,679
like the curse of being a great product manager. You

385
00:16:49,720 --> 00:16:51,399
know everything's going on in this space, you know what's

386
00:16:51,399 --> 00:16:53,399
going internally, you have your own ideas for innovation as

387
00:16:53,399 --> 00:16:55,480
well as what your competitors are doing. But then you

388
00:16:55,519 --> 00:16:58,080
have to remember, yeah, actually our users don't even really

389
00:16:58,120 --> 00:16:59,080
know all those things.

390
00:16:59,200 --> 00:17:01,120
Speaker 3: Yeah, totally. It's I feel like you summed it up.

391
00:17:01,159 --> 00:17:03,360
It's like you have to be able to see through

392
00:17:03,360 --> 00:17:06,519
your users' eyes, and you're often way too biased to

393
00:17:06,599 --> 00:17:08,680
actually see that. And there's a lot to be said there,

394
00:17:08,759 --> 00:17:11,880
like when using a product like you're you know, let's

395
00:17:11,880 --> 00:17:14,319
say I'm like building Honeycomb, Like I understand fully how

396
00:17:14,319 --> 00:17:17,039
to write that query, right, but it's very hard to

397
00:17:17,039 --> 00:17:18,680
see how a user would see it for the first

398
00:17:18,680 --> 00:17:20,559
time and know what that learning curve would be like.

399
00:17:20,680 --> 00:17:23,640
And so it's it's good acknowledging your bias and like

400
00:17:23,640 --> 00:17:24,640
how close to it you are.

401
00:17:24,960 --> 00:17:26,480
Speaker 1: I mean it's like there's another one like you brought

402
00:17:26,559 --> 00:17:28,680
up Honeycomb, Like everyone knows what this is. It's like,

403
00:17:28,960 --> 00:17:32,119
you know, they may know what what elastic search is,

404
00:17:32,240 --> 00:17:35,960
or they may know what you know, the kabana who

405
00:17:35,960 --> 00:17:38,799
they are or whatever, and any of the n other

406
00:17:38,920 --> 00:17:41,160
options are out there, it's like it's another one and

407
00:17:41,240 --> 00:17:43,599
you talk to people who just have never been in observability.

408
00:17:44,079 --> 00:17:46,720
I have no idea what hotel is or collectors and

409
00:17:46,759 --> 00:17:48,359
you bring up a product or whether, it's like I

410
00:17:48,400 --> 00:17:49,599
have no idea what that is.

411
00:17:50,000 --> 00:17:50,759
Speaker 2: Yeah, it's a good.

412
00:17:50,640 --> 00:17:54,319
Speaker 3: Call, it like, especially just like in any form like this,

413
00:17:54,599 --> 00:17:57,400
ensuring there's an intro. I feel like often we do

414
00:17:57,400 --> 00:18:00,920
this a lot with acronyms too, like which I guess

415
00:18:00,960 --> 00:18:03,880
is a lot more standardized. But it's like I'll say

416
00:18:03,880 --> 00:18:06,319
all the time, like ICP and my engineers are like,

417
00:18:06,359 --> 00:18:08,519
what is that. It's like it's our ideal customer profile,

418
00:18:08,559 --> 00:18:10,799
Like this is the customer we want. But it's it's

419
00:18:10,799 --> 00:18:14,200
a good point. It's easy to just like stick to

420
00:18:14,200 --> 00:18:15,279
the things you already know and not.

421
00:18:15,279 --> 00:18:15,920
Speaker 2: Know what you don't know.

422
00:18:16,039 --> 00:18:19,119
Speaker 1: Actually, on this podcast, I'm a huge stickler for acronyms

423
00:18:19,160 --> 00:18:21,480
because you never know, you know what the experiences that

424
00:18:21,519 --> 00:18:24,000
people have had, and just saying what those things are

425
00:18:24,039 --> 00:18:26,359
and calling the manage is like so much easier for

426
00:18:26,359 --> 00:18:28,079
people to see. So I'm going to use another acronyum

427
00:18:28,160 --> 00:18:32,759
right now, is V or independent software vendor. And you

428
00:18:32,759 --> 00:18:34,920
know it's really interesting, is like we end up having

429
00:18:34,960 --> 00:18:36,799
to go through a lot of tools out there to

430
00:18:36,880 --> 00:18:39,240
add in integrations to work with our product or one

431
00:18:39,240 --> 00:18:41,839
of our products, and I find this is a miss

432
00:18:41,839 --> 00:18:44,400
and a lot of platforms. So we had integrations into

433
00:18:44,440 --> 00:18:47,440
Bubble and WordPress and they're just so different compared to

434
00:18:47,480 --> 00:18:50,839
open tofu. The experience that we see so much so

435
00:18:50,880 --> 00:18:53,319
that like we don't support Bubble and WordPress really anymore

436
00:18:53,559 --> 00:18:55,960
because it's just such a huge challenge they don't care about.

437
00:18:56,000 --> 00:18:59,079
There's a platform is vs are on one side and

438
00:18:59,200 --> 00:19:03,359
customers users on the other one, and we've decided to

439
00:19:03,440 --> 00:19:05,880
make Bubble and word Press, it's say, worse by not

440
00:19:05,960 --> 00:19:08,039
offering a first class integration there. I feel like this

441
00:19:08,079 --> 00:19:11,720
is an important thing to really consider because your partnerships

442
00:19:11,799 --> 00:19:14,319
within your platform can have a huge impact for your customers.

443
00:19:14,319 --> 00:19:16,400
You've really recognized this, And I was actually going to

444
00:19:16,440 --> 00:19:18,200
bring up, like how much do you think about this

445
00:19:18,240 --> 00:19:19,000
problem at blow me?

446
00:19:19,119 --> 00:19:21,920
Speaker 3: So when you draw that analogy there of like integrating

447
00:19:22,000 --> 00:19:25,160
not being easy and versus like open tofu being easy,

448
00:19:25,319 --> 00:19:28,240
what you mean there is more like the fact that

449
00:19:28,279 --> 00:19:31,119
Bubble and word plus haven't built an integration at all,

450
00:19:31,279 --> 00:19:34,039
or that they're not being responsive to like issues because

451
00:19:34,039 --> 00:19:35,519
they don't have a method for that.

452
00:19:36,119 --> 00:19:38,319
Speaker 1: I mean, you look at your core customer, your ICP,

453
00:19:38,480 --> 00:19:42,079
and you're like, well, Bubble and WordPress, the customers are

454
00:19:42,519 --> 00:19:45,599
specifically the ones that are using the site or building

455
00:19:45,599 --> 00:19:48,480
the site, but they're not. They're like, we're not a customer,

456
00:19:48,480 --> 00:19:51,240
We're an ISV. We're just providing plugins that customers use.

457
00:19:51,279 --> 00:19:54,400
But our experience building those plugins is so bad that

458
00:19:54,400 --> 00:19:56,559
we're choosing to not build those plugins, which means we're

459
00:19:56,559 --> 00:20:00,319
actually providing less value to their customers because they less

460
00:20:00,319 --> 00:20:03,559
to pick from. It's like Microsoft Teams versus Slack. Slack

461
00:20:03,640 --> 00:20:05,720
is much easier to write a Slack bot for than

462
00:20:05,720 --> 00:20:07,920
Microsoft Teams is to write a pot for, so more

463
00:20:07,920 --> 00:20:10,440
people will write bots for Slack. Therefore, Slack offers a

464
00:20:10,480 --> 00:20:13,480
platform which can bribe more value to the users that join,

465
00:20:13,599 --> 00:20:15,960
and why people like Slack more is just one of

466
00:20:15,960 --> 00:20:17,599
the reasons compared to Microsoft Team.

467
00:20:17,640 --> 00:20:19,039
Speaker 3: Got I got it, Yeah, I see what you're saying.

468
00:20:19,160 --> 00:20:21,359
It's interesting because like I wonder if I'm at word

469
00:20:21,359 --> 00:20:23,039
Press as a PM, like, what is the metric I'm

470
00:20:23,039 --> 00:20:25,359
trying to drive? And that would be interesting because it

471
00:20:25,400 --> 00:20:28,640
probably all trickles down from there. So like Polumi's equivalent

472
00:20:28,880 --> 00:20:32,519
is like we there's a lot of equivalents within Plumby,

473
00:20:32,559 --> 00:20:35,039
but like there's a lot of ways in which Plumy

474
00:20:35,039 --> 00:20:38,000
could not integrate and feel like you want people to

475
00:20:38,000 --> 00:20:40,359
be within your closed garden. So a direct example, as

476
00:20:40,400 --> 00:20:45,079
we've been building an IDP product or sorry, yeah, internal

477
00:20:45,119 --> 00:20:50,440
developer platform and we non internal developer portal. You say IDP,

478
00:20:50,559 --> 00:20:55,680
and I think you think identity. We actually struggle with

479
00:20:55,680 --> 00:20:58,680
this internally, even because we have integrations with all the

480
00:20:58,720 --> 00:21:02,920
identity providers as well. But basically a home where you

481
00:21:02,920 --> 00:21:05,319
could have your documentation and all of your best practices

482
00:21:05,400 --> 00:21:07,519
and your services that like your platform team can set

483
00:21:07,559 --> 00:21:10,599
up and then developers can self serve. And there is

484
00:21:11,279 --> 00:21:15,519
Spotify's open source backstage product in this space, and so

485
00:21:15,559 --> 00:21:16,799
we have customers who are like.

486
00:21:16,799 --> 00:21:19,559
Speaker 2: Oh, you know, while you're building that, it doesn't.

487
00:21:19,279 --> 00:21:22,680
Speaker 3: Have you know, the full functionality of a full IDP,

488
00:21:23,000 --> 00:21:25,960
like all the docks integrations and whatnot, like can we

489
00:21:25,960 --> 00:21:28,480
have an integration with backstage? And we could have easily

490
00:21:28,559 --> 00:21:30,480
been like, nah, we have this vision to have this thing.

491
00:21:30,599 --> 00:21:33,400
But Pulumia is very much like our ultimate goal is

492
00:21:33,480 --> 00:21:36,480
like resources under management. Like our ultimate goal is like

493
00:21:36,519 --> 00:21:39,559
you're getting value from having your instructure managed by Pulumi.

494
00:21:39,759 --> 00:21:41,599
Our goal is not like that we get a lot

495
00:21:41,640 --> 00:21:44,519
of clicks, or that we get like daily active like

496
00:21:44,920 --> 00:21:47,000
number of active users and things like that, Like that's

497
00:21:47,039 --> 00:21:50,799
all great, but we care about growing our overall infrastructure

498
00:21:50,839 --> 00:21:52,880
manage and so that's a no brainer. It's like, let's

499
00:21:52,880 --> 00:21:54,759
go build a backstage integration and so we build a

500
00:21:54,759 --> 00:21:57,039
plug in for backstage that like scaffolds out for you

501
00:21:57,440 --> 00:22:00,440
the ability to have your Pumy sacks and deployments all

502
00:22:00,440 --> 00:22:04,000
within your backstage environment. And that's very much like core

503
00:22:04,119 --> 00:22:07,200
to our beliefs is like APIs on everything. Like if

504
00:22:07,200 --> 00:22:08,839
you don't want to use our UI and you want

505
00:22:08,880 --> 00:22:10,920
to build something else, great, Like if you want to

506
00:22:11,000 --> 00:22:14,279
use our CLI directly and you know, not interact with

507
00:22:14,319 --> 00:22:16,640
our UI, we try and have future parity across everything.

508
00:22:17,119 --> 00:22:20,119
And similarly, like web hoooks, we have you can build

509
00:22:20,279 --> 00:22:26,440
integrations with black or anything custom and service your needs directly.

510
00:22:26,559 --> 00:22:28,480
It doesn't need to be like some Plumy feature that

511
00:22:28,559 --> 00:22:30,079
has like a bow on it. We're happy to just

512
00:22:30,119 --> 00:22:33,119
like meet you where you are, and so that's interesting.

513
00:22:33,119 --> 00:22:34,599
That's how we think about some of those trade offs.

514
00:22:34,599 --> 00:22:36,519
And the backstage one is like a direct example of

515
00:22:36,599 --> 00:22:38,400
like where we could have just been like, no, we're

516
00:22:38,440 --> 00:22:38,759
not going to.

517
00:22:38,759 --> 00:22:39,160
Speaker 2: Go do this.

518
00:22:40,079 --> 00:22:43,519
Speaker 1: That must contribute though, to like extra work, extra time

519
00:22:43,599 --> 00:22:45,519
before being able to roll out something I could imagine.

520
00:22:45,519 --> 00:22:49,079
Now you have the maintenance of managing the backstage plug in,

521
00:22:49,119 --> 00:22:51,440
how do you, like, how do you organize around that

522
00:22:51,559 --> 00:22:53,960
challenge to make sure that you are staying up to

523
00:22:54,039 --> 00:22:56,920
date with whatever breaking changes backstage has to make sure

524
00:22:56,960 --> 00:22:59,519
the plug is still valid. Are You're now supporting a

525
00:22:59,559 --> 00:23:02,119
whole bunch of new users who are probably coming to

526
00:23:02,200 --> 00:23:04,599
your community and be like, hey, this thing with backstage

527
00:23:04,640 --> 00:23:07,000
is broken, and you're like, well, it's not really us.

528
00:23:07,039 --> 00:23:08,799
We can't really help you, hear, this is the backstaging,

529
00:23:08,920 --> 00:23:11,279
and now you're teaching them about how to use backstage.

530
00:23:11,319 --> 00:23:13,000
Speaker 3: I mean, the backstage chef has been kind of simple.

531
00:23:13,000 --> 00:23:14,880
But what you're talking about is true of like our

532
00:23:15,039 --> 00:23:18,599
entire provider landscape, So like we build providers for like

533
00:23:18,880 --> 00:23:24,079
two hundred plus like cloud or SaaS Offerings and so

534
00:23:24,160 --> 00:23:26,319
like AWS GCP all the way to like Snowflake and

535
00:23:26,880 --> 00:23:29,880
Launch Darkly, so like everything has a Plumby provider, and

536
00:23:29,920 --> 00:23:32,000
that's exactly what you're describing, Like will the customer come

537
00:23:32,079 --> 00:23:33,400
to us to be like, hey, I'm trying to use

538
00:23:33,440 --> 00:23:37,720
this like Opta provider you have and something's broken and

539
00:23:37,759 --> 00:23:39,720
it's like, okay, it broke upstream, and like we don't

540
00:23:39,720 --> 00:23:41,400
have access to their code, so like we can't help,

541
00:23:41,440 --> 00:23:43,279
and so it's there's.

542
00:23:43,200 --> 00:23:44,240
Speaker 2: A lot of things to unpack.

543
00:23:44,279 --> 00:23:46,759
Speaker 3: Their one is like how plumy spends its time, and

544
00:23:46,799 --> 00:23:50,240
that's honestly an ongoing struggle because kind of what you

545
00:23:50,279 --> 00:23:53,440
said earlier is like every GitHub issue has the same equivalents,

546
00:23:53,480 --> 00:23:56,400
Like you never know what the impact that's going to have,

547
00:23:56,440 --> 00:23:58,160
and you don't want to pick and choose like, oh

548
00:23:58,160 --> 00:24:00,680
I think this one's important versus that, and so you

549
00:24:00,759 --> 00:24:03,160
want to help every customer button saying that we're like

550
00:24:03,200 --> 00:24:06,519
a tiny amount of engineers supporting this whole thing. So

551
00:24:06,759 --> 00:24:09,880
we try and do like prioritization based on volume as

552
00:24:09,960 --> 00:24:12,039
much as possible, So like if we have a huge

553
00:24:12,119 --> 00:24:15,000
volume of customers using one provider, we make sure that

554
00:24:15,000 --> 00:24:18,319
that's like first class. We basically have like a teering

555
00:24:18,359 --> 00:24:21,160
system and so you know what support levels and slas

556
00:24:21,160 --> 00:24:24,720
we're going to have for each of our providers, And yeah,

557
00:24:24,759 --> 00:24:27,880
there's definitely times where like oh, you're like, you know,

558
00:24:28,000 --> 00:24:30,880
one of a small like a few hundred using a

559
00:24:30,880 --> 00:24:32,960
certain provider that's like very niche, and so you might

560
00:24:33,000 --> 00:24:35,359
get like a longer wait time on getting a fix.

561
00:24:35,440 --> 00:24:38,039
But like ideally you understand that, Like that's what's most

562
00:24:38,039 --> 00:24:40,160
important to us is like you know what the expectations are,

563
00:24:40,160 --> 00:24:43,759
you have a response. You aren't just like flying blind,

564
00:24:43,799 --> 00:24:45,119
which would be like the worst scenario.

565
00:24:45,240 --> 00:24:47,599
Speaker 1: So can I pad the metrics for our provider by

566
00:24:47,880 --> 00:24:51,519
going and just downloading it a lot of times, be

567
00:24:51,599 --> 00:24:53,680
a different IP addresses or something like that, so it

568
00:24:53,720 --> 00:24:55,720
looks like there's a more usage there.

569
00:24:55,759 --> 00:24:56,400
Speaker 2: Yeah, go for it.

570
00:24:58,079 --> 00:24:59,640
Speaker 3: We would one thing that we've been talking about, We

571
00:24:59,640 --> 00:25:01,759
would love like to add these metrics to also be

572
00:25:01,880 --> 00:25:05,559
user facing because like we can say like what percentage

573
00:25:05,599 --> 00:25:09,079
of at least like polumy cloud state has the that

574
00:25:09,119 --> 00:25:11,119
provider like how many resources, right, and so we could

575
00:25:11,160 --> 00:25:13,160
say like, oh, you're in like the tenth percentile of

576
00:25:13,200 --> 00:25:16,640
like providers for PULYMI or something like that, because yeah,

577
00:25:16,640 --> 00:25:20,160
that transparency could be cool having just general insights into

578
00:25:20,200 --> 00:25:22,240
like how your provider is being used.

579
00:25:22,960 --> 00:25:25,680
Speaker 1: Yeah, No, I mean I always find the metrics interesting there,

580
00:25:25,720 --> 00:25:29,599
like something that certain platforms are always in an opportunity

581
00:25:29,599 --> 00:25:31,000
to do, and then that it seems like they don't

582
00:25:31,039 --> 00:25:34,000
actually go forward. Like if you know that someone is

583
00:25:34,079 --> 00:25:37,960
using I don't know, grafhauna in someplace and you like

584
00:25:37,960 --> 00:25:39,400
you can actually go in and be like, oh, yeah,

585
00:25:39,400 --> 00:25:41,160
you know, like did you know everyone else is using

586
00:25:41,200 --> 00:25:44,440
polluming Graffhanna has this configuration, but you don't like that

587
00:25:44,519 --> 00:25:46,039
may be something to actually investigate.

588
00:25:46,279 --> 00:25:47,160
Speaker 2: Pullumy cloud like.

589
00:25:47,200 --> 00:25:49,559
Speaker 3: We think about that a lot, especially as you're getting

590
00:25:49,559 --> 00:25:52,440
into these more common that people are using AI tools

591
00:25:52,519 --> 00:25:54,720
to write poluming. It's like we can pull in that

592
00:25:54,759 --> 00:25:58,920
context for your AI developer tool to be like, hey, hey,

593
00:25:58,920 --> 00:26:01,160
this is an architecture that most customers have with this

594
00:26:01,279 --> 00:26:01,920
resource or.

595
00:26:01,839 --> 00:26:02,480
Speaker 2: Something like that.

596
00:26:02,799 --> 00:26:04,680
Speaker 1: I'm smiling because you said the magic word that I

597
00:26:04,720 --> 00:26:07,960
think I need a Clackson here for. So now now

598
00:26:07,960 --> 00:26:10,000
that you've mentioned it, I'm going to have to ask

599
00:26:10,039 --> 00:26:13,160
you some LLM based questions here. So now that you

600
00:26:13,240 --> 00:26:15,240
know that, do you know how much of the code

601
00:26:15,279 --> 00:26:18,079
for Poluomi that you know written by your customers is

602
00:26:18,119 --> 00:26:22,279
written by an LM versus by an engineer.

603
00:26:22,319 --> 00:26:23,400
Speaker 4: Is this even something you can track?

604
00:26:23,640 --> 00:26:26,319
Speaker 3: It's not something we can track general open source Pulmovi

605
00:26:26,319 --> 00:26:29,880
we have zero telemetry. But in terms of like our

606
00:26:29,960 --> 00:26:32,960
existing customer base we are we kind of get things

607
00:26:32,960 --> 00:26:36,599
like potentially ID and like your VCS provider, but not

608
00:26:36,880 --> 00:26:39,359
like if you use an LM or not. It's interesting

609
00:26:39,359 --> 00:26:41,599
because you could maybe think about like poll requests that's

610
00:26:41,599 --> 00:26:43,599
say like open with cloud code or like codex or

611
00:26:43,640 --> 00:26:45,799
something like that, but for the most part we're like

612
00:26:45,799 --> 00:26:50,160
blind to it. However, we have a AI product within

613
00:26:50,279 --> 00:26:52,400
plumy Cloud that you can use for code generations, so

614
00:26:52,400 --> 00:26:55,319
we have metrics on that, but just the general ecosystem

615
00:26:55,599 --> 00:26:58,680
unsure sample size of it probably still less than you

616
00:26:58,720 --> 00:27:01,960
would think at this point, growing extremely fast, especially talking

617
00:27:02,000 --> 00:27:04,839
to customers. But infrastructure is a space where like people

618
00:27:04,839 --> 00:27:08,799
are inherently more cautious and are it's not moving at

619
00:27:08,799 --> 00:27:10,640
the pace of app development. There needs to be like

620
00:27:10,640 --> 00:27:12,920
guardrails and ways to make sure that that isn't gonna

621
00:27:13,359 --> 00:27:15,200
you know, you're not going to accidentally vibe code your

622
00:27:15,240 --> 00:27:15,920
production cluss.

623
00:27:16,720 --> 00:27:19,759
Speaker 1: I mean you say that, but Replet just had a

624
00:27:19,880 --> 00:27:24,519
huge controversy over deleting all the infrastructure and databasechema related

625
00:27:24,559 --> 00:27:28,359
stuff for one of their customers because the LM they

626
00:27:28,440 --> 00:27:32,200
had decided to during what they had called code freeze,

627
00:27:32,319 --> 00:27:35,480
still make changes and push database changes and because querries

628
00:27:35,480 --> 00:27:38,920
weren't responding, then wipe the whole thing. So you know,

629
00:27:38,960 --> 00:27:41,440
it's getting there. I think maybe the question I want

630
00:27:41,440 --> 00:27:44,039
to ask is were you ahead of the curve here?

631
00:27:44,079 --> 00:27:47,759
I mean, I know Pulumius had LM based answer generation

632
00:27:47,880 --> 00:27:50,359
inside the docks for a while now, way before, way

633
00:27:50,400 --> 00:27:52,640
ahead of the curve. But as far as the product

634
00:27:52,640 --> 00:27:56,000
goes or the features that you're building, have you pivoted

635
00:27:56,119 --> 00:28:00,519
in any way to potentially deal with the packed of

636
00:28:00,640 --> 00:28:03,079
customers now utilizing more LM tools.

637
00:28:03,319 --> 00:28:06,119
Speaker 3: Yeah, definitely, I mean it's it's definitely a huge focus.

638
00:28:06,680 --> 00:28:07,799
Speaker 2: It's interesting. So yeah, pullin.

639
00:28:07,880 --> 00:28:12,359
Speaker 3: We had I think we had an AI product before

640
00:28:12,559 --> 00:28:15,240
Chattobt had a UI, like when it was just an API,

641
00:28:16,519 --> 00:28:18,319
but I would have to check my exact timing on that,

642
00:28:18,359 --> 00:28:20,279
But around that time we were like very quick to

643
00:28:20,359 --> 00:28:23,319
adding an LM to our product and the main use

644
00:28:23,359 --> 00:28:25,319
case at the time, which it is funny looking back

645
00:28:25,359 --> 00:28:27,200
because at the time everyone thought this would be a

646
00:28:27,200 --> 00:28:28,640
good idea, But like if had said it was like

647
00:28:28,640 --> 00:28:30,319
not a good idea. We were like, oh yeah, you

648
00:28:30,319 --> 00:28:33,079
can like ask questions and we'll have like an LM

649
00:28:33,160 --> 00:28:35,799
respond to it. And at the time they just like

650
00:28:35,839 --> 00:28:38,880
hallucinated a bunch. They would like give incorrect links and

651
00:28:38,960 --> 00:28:41,039
like in some ways it was like a funny way

652
00:28:41,039 --> 00:28:44,200
to get product feedback because it would like hallucinate API

653
00:28:44,799 --> 00:28:46,720
endpoints and we'd be like, actually, it probably should be

654
00:28:46,759 --> 00:28:50,000
damned bad, like if the LM probably thinks that, because

655
00:28:50,000 --> 00:28:52,920
that is how most products would name this, And so

656
00:28:53,079 --> 00:28:56,400
it was an interesting feedback loop. But originally definitely like

657
00:28:56,440 --> 00:28:59,400
took some time to figure out, like we didn't limit

658
00:28:59,680 --> 00:29:02,279
what assets, so like people were just like using our

659
00:29:02,799 --> 00:29:06,200
like in product chatbot as their GPT basically like just

660
00:29:06,240 --> 00:29:10,200
like completely unrelated to POLU meant questions like help with things.

661
00:29:10,680 --> 00:29:12,640
But we rEFInd it a lot over time. And I

662
00:29:12,680 --> 00:29:17,079
think to your question around like knowing that you know

663
00:29:17,119 --> 00:29:20,079
your user base is changing how they're doing things pretty drastically,

664
00:29:20,119 --> 00:29:24,359
which is using lms to write code, like how how

665
00:29:24,359 --> 00:29:26,880
does like pollum me inform like how does that change

666
00:29:26,920 --> 00:29:29,279
what we think about? And I think there's a couple

667
00:29:29,319 --> 00:29:32,400
of things. One is one thing we're hearing is there's

668
00:29:32,440 --> 00:29:35,200
a ton of app code that's now coming to like

669
00:29:35,279 --> 00:29:38,640
the speed of application code is increasing due to these tools,

670
00:29:38,680 --> 00:29:40,559
which means the infrastructure team is becoming a bit of

671
00:29:40,599 --> 00:29:42,880
a bottleneck. And so we have teams who are like,

672
00:29:42,920 --> 00:29:44,920
we have so much work right now like this, like

673
00:29:44,960 --> 00:29:48,240
we're feeling the increasing speed and so like for us,

674
00:29:48,240 --> 00:29:50,000
it's like figuring out how do we automate things that

675
00:29:50,079 --> 00:29:51,839
like make it easier to stay on top, so like

676
00:29:51,880 --> 00:29:54,960
how do we help platform teams with this new dynamic change.

677
00:29:55,039 --> 00:29:57,640
But then there's also the people actually writing plumy right,

678
00:29:58,000 --> 00:30:00,599
writing plumy code. And the first thing we ever did

679
00:30:00,640 --> 00:30:03,680
in this space was one of the early problems with

680
00:30:03,799 --> 00:30:07,279
using LLLS or PULLUMEI code was that it hallucinated resource

681
00:30:07,359 --> 00:30:10,079
names a lot, because there's a lot of different versions

682
00:30:10,240 --> 00:30:13,359
of a provider, and it's actually very important that it

683
00:30:13,400 --> 00:30:15,680
gets that right. Otherwise you're just going to be constantly

684
00:30:15,680 --> 00:30:17,039
having to go fix a bunch of things to get

685
00:30:17,039 --> 00:30:18,920
it to run. So the first thing we did was

686
00:30:18,960 --> 00:30:21,559
provide it context for all of the latest versions of

687
00:30:21,559 --> 00:30:24,559
every provider so that it was just much more accurate, and.

688
00:30:24,519 --> 00:30:25,200
Speaker 2: That was pretty good.

689
00:30:25,200 --> 00:30:27,640
Speaker 3: We got a good amount of usage, like more than

690
00:30:27,640 --> 00:30:30,000
we would have expected of customers writing code using that.

691
00:30:30,519 --> 00:30:32,680
And it's very simple, right, but it's really just like

692
00:30:32,720 --> 00:30:35,359
what mcps are today, like giving the context that the

693
00:30:35,400 --> 00:30:37,680
model needs at the time it needs it. And now

694
00:30:37,680 --> 00:30:39,880
we've grown a lot, so like you can get any

695
00:30:40,079 --> 00:30:44,960
information about your PULLUMI environment from our LM and that

696
00:30:45,079 --> 00:30:47,440
is like in our MCP, but also in our in

697
00:30:47,480 --> 00:30:51,279
big product which there's some interesting product dynamics there now

698
00:30:51,319 --> 00:30:53,440
with like building mcps, and so we're going to make

699
00:30:53,519 --> 00:30:56,759
everything that's available in any of our features available in

700
00:30:56,799 --> 00:30:57,920
our MCP.

701
00:30:57,680 --> 00:31:01,279
Speaker 2: Which using acronym Model Context Protocol.

702
00:31:01,319 --> 00:31:02,920
Speaker 3: I know you guys had a session on this already,

703
00:31:02,960 --> 00:31:05,720
so your listeners will kind of be familiar.

704
00:31:06,079 --> 00:31:07,799
Speaker 4: Definitely. I'll just plug that.

705
00:31:07,880 --> 00:31:10,640
Speaker 1: You know, if you don't know what MCP is, go

706
00:31:10,680 --> 00:31:13,519
watch the previous episode where we talked a lot about this.

707
00:31:13,839 --> 00:31:14,359
Speaker 4: Pretty good.

708
00:31:14,640 --> 00:31:17,480
Speaker 1: So yeah, I mean one thing that comes up here actually,

709
00:31:17,519 --> 00:31:19,759
and maybe i'll say it, I'll preface it with I

710
00:31:19,759 --> 00:31:21,000
think this is going to be controversial.

711
00:31:22,079 --> 00:31:22,799
Speaker 4: Your trading.

712
00:31:22,839 --> 00:31:24,519
Speaker 1: I mean, I think we know this is the case

713
00:31:24,599 --> 00:31:26,480
with LMS, your trading quality.

714
00:31:26,759 --> 00:31:28,880
Speaker 4: For speed, you need to be.

715
00:31:28,880 --> 00:31:31,640
Speaker 1: Most concerned about the quality, even more so in your

716
00:31:31,680 --> 00:31:36,119
infrastructure given that small changes, especially during refactoring, can have

717
00:31:36,279 --> 00:31:39,000
huge production impact. Whereas a random bug in one of

718
00:31:39,000 --> 00:31:41,359
your websites or one of your endpoints isn't so bad.

719
00:31:41,759 --> 00:31:43,960
Speaker 4: Having a small change in.

720
00:31:43,759 --> 00:31:47,160
Speaker 1: Your say database provider, or if you're using rds and

721
00:31:47,160 --> 00:31:49,079
you change the schema or roll out in version, you

722
00:31:49,079 --> 00:31:51,680
could have downtime or worse, you know, your whole database

723
00:31:51,720 --> 00:31:56,319
guts crash. I'm going to argue making sure the users

724
00:31:56,359 --> 00:32:00,400
are still going slow is providing more value than allowing

725
00:32:00,400 --> 00:32:01,559
the ability to go fast.

726
00:32:02,359 --> 00:32:04,880
Speaker 4: I know your users are probably going to disagree with

727
00:32:04,880 --> 00:32:08,640
that statement. Any thoughts about that though?

728
00:32:08,880 --> 00:32:13,079
Speaker 3: That's really interesting. What are our CEO Joe Duffy. He

729
00:32:13,319 --> 00:32:16,839
has this really good analogy for what we're seeing right now,

730
00:32:16,839 --> 00:32:20,680
which is you wouldn't vibe code without get right, Like

731
00:32:20,680 --> 00:32:22,720
you're not going to directly change all your code and

732
00:32:22,720 --> 00:32:25,240
like push it to your production server, and you kind

733
00:32:25,240 --> 00:32:27,720
of want that GET layer. Maybe you would, Sorry, but

734
00:32:27,839 --> 00:32:31,240
let's say it's a helpful less to have, Like here's

735
00:32:31,240 --> 00:32:33,000
the GIT changes and I can review that and see

736
00:32:33,000 --> 00:32:34,960
what's going to change, right, And that's a lot of

737
00:32:35,160 --> 00:32:39,079
application coding is going in the space of like reviewing code.

738
00:32:39,160 --> 00:32:41,440
Now you can like tag the GitHub copilot agent on

739
00:32:41,480 --> 00:32:43,359
a PR and they'll write it and like you can

740
00:32:43,400 --> 00:32:46,640
review it. And for small stuff like that, that largely works,

741
00:32:46,640 --> 00:32:48,279
like we do that in plume me for a handful

742
00:32:48,279 --> 00:32:52,359
of things, but you need that for infrastructure too, right,

743
00:32:52,519 --> 00:32:55,119
Like you want a way to understand like what is

744
00:32:55,119 --> 00:32:58,319
desired satan, what's going to change, and like Plumy is

745
00:32:58,359 --> 00:33:00,920
great for that in the sense, and you know open

746
00:33:00,960 --> 00:33:04,039
tofu has similar things. But the one difference with Polumi

747
00:33:04,119 --> 00:33:05,960
is that you're using a programming language. So like these

748
00:33:06,000 --> 00:33:08,440
models have a ton of sample data and context on

749
00:33:08,680 --> 00:33:12,279
using a programming language, but Pulumi is a desired state

750
00:33:12,400 --> 00:33:15,000
versus actual engine, and so you can see exactly what's

751
00:33:15,000 --> 00:33:16,920
going to change and run a preview on it. And

752
00:33:17,000 --> 00:33:19,880
so to your point about like helping users by like

753
00:33:20,079 --> 00:33:22,559
moving slowly, Like the best thing plymy can do is

754
00:33:22,559 --> 00:33:24,799
like give you previews of what's actually going to happen,

755
00:33:24,960 --> 00:33:27,319
And that layer becomes so much more important. And so

756
00:33:27,440 --> 00:33:29,839
as we're building stuff with AI, you know, we're starting

757
00:33:29,880 --> 00:33:32,640
to get into the space of automating things within Plumy

758
00:33:33,119 --> 00:33:35,359
with AI, the most important thing is like what are

759
00:33:35,359 --> 00:33:36,279
your checks and balances?

760
00:33:36,680 --> 00:33:40,519
Speaker 1: Yeah, I mean, I know you said you wouldn't vibe

761
00:33:40,559 --> 00:33:43,240
code without GET, and I'd agree with that. I mean,

762
00:33:43,279 --> 00:33:45,960
I don't vibe code to begin with, but I definitely

763
00:33:45,960 --> 00:33:47,920
wouldn't vibe code without GET. But there are for sure

764
00:33:48,000 --> 00:33:50,319
lots of people who would vibe code without GET, and

765
00:33:50,359 --> 00:33:52,759
the idea of using some sort of version control there

766
00:33:53,160 --> 00:33:55,759
is a whole complexity that they probably have never even

767
00:33:55,799 --> 00:34:00,000
thought of, especially because we see LMS as raising the floor,

768
00:34:00,319 --> 00:34:04,759
so those with less software experience being able to start

769
00:34:04,759 --> 00:34:07,400
doing things they haven't done before, which means they're not

770
00:34:07,480 --> 00:34:09,920
going to be using all the tools and best practices

771
00:34:09,960 --> 00:34:13,559
that the industry has created to deal with quality issues

772
00:34:13,679 --> 00:34:15,559
or production impact.

773
00:34:15,599 --> 00:34:18,239
Speaker 4: As you know that has happened in the past, so

774
00:34:19,119 --> 00:34:20,480
you know there's something interesting there.

775
00:34:20,519 --> 00:34:22,960
Speaker 1: I think the other thing we see is that because

776
00:34:23,000 --> 00:34:25,559
of the quality speed trade off, there's a lot more

777
00:34:25,559 --> 00:34:29,480
code being generated, which really reduces the value of what's

778
00:34:29,519 --> 00:34:30,199
being checked in.

779
00:34:30,760 --> 00:34:32,280
Speaker 4: And there's actually a corrollary to this.

780
00:34:32,400 --> 00:34:35,360
Speaker 1: Whereas I don't want to read an LLM generated post,

781
00:34:36,519 --> 00:34:39,519
that means that the value in that post is probably

782
00:34:39,559 --> 00:34:42,719
whatever the prompt was, which is way more valuable than

783
00:34:42,800 --> 00:34:43,440
the output.

784
00:34:43,519 --> 00:34:44,920
Speaker 4: So if you are vibe coding.

785
00:34:45,400 --> 00:34:47,559
Speaker 1: The thing that makes sense more to be committing is

786
00:34:47,599 --> 00:34:51,440
the intent rather than the output. There, so I can

787
00:34:51,480 --> 00:34:53,280
see a world that you know, get in. The whole

788
00:34:53,280 --> 00:34:55,760
development workflow does change in a way, there is still

789
00:34:55,760 --> 00:34:57,960
this intermediary step, and I really like to call out

790
00:34:58,000 --> 00:35:01,559
that it's the review and when we may be automating

791
00:35:01,559 --> 00:35:03,280
some parts of the review for you, because I do

792
00:35:03,320 --> 00:35:06,599
see a lot of value in Hey, you know, does

793
00:35:06,599 --> 00:35:10,199
this match with certain policies or other expectations that we

794
00:35:10,280 --> 00:35:12,039
have as an organization or even just as a team

795
00:35:12,119 --> 00:35:14,519
or a service, or best practices or what everyone else

796
00:35:14,599 --> 00:35:16,440
is doing all what comes in through there.

797
00:35:16,559 --> 00:35:18,960
Speaker 3: Yeah, I think that's super important, like plumbing, knowing all

798
00:35:19,000 --> 00:35:21,159
of your best practices and ensuring that that's what it's

799
00:35:21,159 --> 00:35:23,480
putting out. And I was going to say to put

800
00:35:23,679 --> 00:35:26,159
to challenge a little bit. You're like, we're trading speed

801
00:35:26,239 --> 00:35:29,239
versus quality. The one kind of difference there is like that,

802
00:35:29,280 --> 00:35:31,480
I think that's largely true if like you've done this before.

803
00:35:31,639 --> 00:35:33,360
But we have a lot of users who are like, oh,

804
00:35:33,400 --> 00:35:35,599
I'm like new to using pluming, Like my platform team

805
00:35:35,679 --> 00:35:37,679
uses it, but I've never written it. And this is

806
00:35:37,760 --> 00:35:39,760
like a lot of helping with the zero to one

807
00:35:39,760 --> 00:35:42,440
where like it can actually if you've never done something before,

808
00:35:42,559 --> 00:35:44,440
it helps you learn it a lot better in the

809
00:35:44,480 --> 00:35:46,760
sense of like let's say you come to pluming, you're like, hey,

810
00:35:46,800 --> 00:35:49,280
I need a program for like a cloud flow worker

811
00:35:49,400 --> 00:35:52,119
or something, and it generates it for you and it

812
00:35:52,239 --> 00:35:55,320
uses Hey, this is how your organization best practices are

813
00:35:55,360 --> 00:35:57,360
for this resource, and it pulls in all of your

814
00:35:57,400 --> 00:36:00,159
like template of how to do things. Then like you're

815
00:36:00,199 --> 00:36:02,360
like up and running a lot faster. And so yeah,

816
00:36:02,440 --> 00:36:04,119
maybe the quality is not as good as like someone

817
00:36:04,119 --> 00:36:05,840
who does this for a full time job, but someone

818
00:36:05,880 --> 00:36:08,119
new to it gets a lot of benefits from having

819
00:36:08,159 --> 00:36:11,320
all this context of this is how we do it

820
00:36:11,360 --> 00:36:13,360
in our organization. This is our best practice. These are

821
00:36:13,400 --> 00:36:15,920
security best practices of doing things by the way we've

822
00:36:16,000 --> 00:36:18,800
enabled like policies and ensured that like you have short

823
00:36:18,800 --> 00:36:21,039
lived access tokens on this deployment and like all of

824
00:36:21,079 --> 00:36:23,360
these things come with it, and so in a bunch

825
00:36:23,400 --> 00:36:25,119
of cases we have like new users that are like

826
00:36:25,119 --> 00:36:25,639
this is great.

827
00:36:25,639 --> 00:36:26,679
Speaker 2: This has helped me so much.

828
00:36:26,920 --> 00:36:31,079
Speaker 1: So I will agree definitely on the if you don't

829
00:36:31,119 --> 00:36:34,639
have experience in something that the quality of the LM

830
00:36:34,679 --> 00:36:37,920
generated output, especially in spaces where there are examples or

831
00:36:38,000 --> 00:36:40,719
can be moderated or validated or reviewed, is going to

832
00:36:40,719 --> 00:36:42,400
be much higher than what they would have put out

833
00:36:43,119 --> 00:36:44,199
previously without that that.

834
00:36:44,400 --> 00:36:45,840
Speaker 4: That for sure is true.

835
00:36:45,960 --> 00:36:49,239
Speaker 1: However, I will debate whoever, Like, if you're inexperience in

836
00:36:49,280 --> 00:36:51,320
a particular area, So a new user comes in and

837
00:36:51,360 --> 00:36:53,960
hasn't used polluming before, I don't think they would you

838
00:36:54,199 --> 00:36:56,480
be actually learning anything. They would not be learning about

839
00:36:56,519 --> 00:36:58,840
cloud flare or polluming if they're using an LM to

840
00:36:58,880 --> 00:37:01,000
generate that code. There is there is a study that

841
00:37:01,400 --> 00:37:04,320
was sponsored by Microsoft about the loss of critical thinking

842
00:37:04,360 --> 00:37:07,079
as a result of utilizing LM models.

843
00:37:07,199 --> 00:37:07,920
Speaker 4: So it does.

844
00:37:08,039 --> 00:37:11,760
Speaker 1: It does, for sure helps those users get to valuable

845
00:37:12,119 --> 00:37:15,159
output of a higher quality then they would have had

846
00:37:15,440 --> 00:37:19,800
without it, But it doesn't help them become experienced engineers

847
00:37:20,159 --> 00:37:22,199
bloomy experts, or even be able to use it to

848
00:37:22,199 --> 00:37:24,400
build new things without also relying on the LM in

849
00:37:24,440 --> 00:37:27,039
the future. So I do want to ask about that,

850
00:37:27,079 --> 00:37:29,599
maybe something like I think we all have to contend

851
00:37:29,599 --> 00:37:33,719
with our engineers utilizing LMS within our own company. Have

852
00:37:33,880 --> 00:37:38,400
you seen this in any way, like, are your inexperienced engineers,

853
00:37:38,440 --> 00:37:40,920
you know, ones that you hire from university or from

854
00:37:40,960 --> 00:37:44,559
other companies that don't have infrastructure as code experience, helping

855
00:37:44,639 --> 00:37:47,440
them in some way to combat the loss of experience

856
00:37:47,480 --> 00:37:50,239
that they would have gained now that they're using LMS.

857
00:37:50,639 --> 00:37:55,000
Speaker 3: Yeah, that's very interesting in terms of internally, we actually,

858
00:37:55,079 --> 00:37:57,559
to be super transparent, don't have a ton of more

859
00:37:57,639 --> 00:38:00,800
junior engineers. And this isn't like a new LM thing.

860
00:38:00,880 --> 00:38:03,239
This is just like, as long as Plumi has existed,

861
00:38:03,320 --> 00:38:05,400
we basically hired people who have a lot of experience

862
00:38:05,440 --> 00:38:08,480
like building languages or seks and there you.

863
00:38:08,400 --> 00:38:10,280
Speaker 2: Know, obviously aren't a ton of new roads that have that.

864
00:38:10,519 --> 00:38:13,000
Speaker 3: As we grow as a company, we'll have like, you know,

865
00:38:13,199 --> 00:38:14,960
a need for a lot more.

866
00:38:16,239 --> 00:38:17,159
Speaker 2: Like people who are.

867
00:38:17,039 --> 00:38:19,440
Speaker 3: Just coming out of school and like the But today

868
00:38:19,480 --> 00:38:21,519
we don't have like a ton of great mechanisms to

869
00:38:21,599 --> 00:38:24,960
onboard people and like train them, and so it might

870
00:38:25,000 --> 00:38:26,960
be like not the best experience, but we have some

871
00:38:27,320 --> 00:38:30,440
and it's just like I'm giving the disclaimer of like

872
00:38:30,480 --> 00:38:32,079
this is not the thing that we do best. You know,

873
00:38:32,119 --> 00:38:35,679
I'm gonna call that a day one. But those we

874
00:38:35,840 --> 00:38:37,719
do have, it's interesting. I mean, in Plumy, I think

875
00:38:37,760 --> 00:38:39,920
a lot of organizations are probably having this similar thing

876
00:38:39,920 --> 00:38:40,440
where you're.

877
00:38:40,320 --> 00:38:42,760
Speaker 2: Thinking about how far do we go with this AI?

878
00:38:42,840 --> 00:38:46,320
Speaker 3: Think like there's companies, like large organizations that send emails

879
00:38:46,320 --> 00:38:48,719
to every manager with the number of AI queries that

880
00:38:48,760 --> 00:38:51,119
each developer is doing. So like that's one end of

881
00:38:51,119 --> 00:38:53,760
the spectrum where you're like forcing it down, and then

882
00:38:53,760 --> 00:38:55,760
there's the other end where you're just leaving it wide open.

883
00:38:55,880 --> 00:38:57,760
Speaker 4: Yeah, I mean, can we collectively agree that, like that's

884
00:38:57,800 --> 00:38:58,519
all that's wrong?

885
00:39:00,559 --> 00:39:04,199
Speaker 3: Yeah, it's it's it's an approach, right, I don't know

886
00:39:04,239 --> 00:39:06,960
what it depends. All depends on your desired outcome is

887
00:39:07,039 --> 00:39:08,480
in this case you know, well.

888
00:39:08,320 --> 00:39:11,639
Speaker 1: I mean, like Eddie, I'm gonna repeat the age old

889
00:39:11,719 --> 00:39:14,239
quote which I'm sure some people still haven't heard before.

890
00:39:14,280 --> 00:39:16,760
Any metric that becomes the target ceases to be a

891
00:39:16,800 --> 00:39:18,679
good metric, right, And I think this is this is

892
00:39:18,719 --> 00:39:19,800
an indication.

893
00:39:19,400 --> 00:39:22,079
Speaker 4: Of knowing like using an LM.

894
00:39:22,119 --> 00:39:23,960
Speaker 1: And I'm gonna keep saying LM for as long as

895
00:39:24,000 --> 00:39:26,239
I'm a host of this podcast, because it's not AI

896
00:39:26,360 --> 00:39:30,960
for me to solve a problem where it can help you. Right,

897
00:39:31,159 --> 00:39:34,960
you lack experience in a particular area and you need something,

898
00:39:35,000 --> 00:39:37,280
you need a second review on it, or you know,

899
00:39:37,360 --> 00:39:40,239
to generate that blooming code the first time it gets

900
00:39:40,239 --> 00:39:42,679
you there. It for sure does, and that's a good usage.

901
00:39:42,760 --> 00:39:44,679
It's a bad usage if you take that output and

902
00:39:44,679 --> 00:39:47,239
you send it to someone else and say this is

903
00:39:47,280 --> 00:39:49,679
the right answer, Like you know, I just use an

904
00:39:49,760 --> 00:39:52,679
LM to generate it, and you can't distinguish between those

905
00:39:52,679 --> 00:39:55,280
two things in a metrics support So I think, you know,

906
00:39:55,280 --> 00:39:57,440
that's a huge problem. Or I use this LM to

907
00:39:57,480 --> 00:40:00,800
make critical business decisions, like Megan, I can ask you

908
00:40:00,840 --> 00:40:02,239
how many times have you used an LM in the

909
00:40:02,280 --> 00:40:05,360
last week to make critical business decisions for your organization?

910
00:40:05,760 --> 00:40:06,760
Speaker 2: I mean it's not zero.

911
00:40:08,000 --> 00:40:10,480
Speaker 4: I put the question in and I just do whatever

912
00:40:10,519 --> 00:40:11,000
it says.

913
00:40:11,239 --> 00:40:12,239
Speaker 2: Oh, definitely not that.

914
00:40:12,320 --> 00:40:16,840
Speaker 3: I mean I strongly believe in like writing down large

915
00:40:16,840 --> 00:40:19,800
product decisions and making sure that you have the options

916
00:40:19,800 --> 00:40:22,480
you considered and why you didn't consider each of them,

917
00:40:22,480 --> 00:40:25,920
like why you recommended what you did, and so if

918
00:40:26,039 --> 00:40:28,559
the LM's like, oh, yeah, you should use this pricing metric,

919
00:40:28,639 --> 00:40:31,079
but ultimately, you know, it's your logic that has to

920
00:40:31,079 --> 00:40:32,719
stand up. And I'm a huge fan of still doing

921
00:40:32,800 --> 00:40:34,239
doctor reviews, so we do that for like all of

922
00:40:34,280 --> 00:40:36,480
our major product decisions, and so it's like a room

923
00:40:36,519 --> 00:40:40,599
full of people are to like criticizing my logic on

924
00:40:40,639 --> 00:40:42,960
a product decision, which I love. That's the best way

925
00:40:43,000 --> 00:40:45,039
to figure out if you're doing the right thing. So

926
00:40:45,079 --> 00:40:47,800
probably zero in the sense of if the barometer is

927
00:40:48,119 --> 00:40:50,559
put it in and then do exactly what it says.

928
00:40:50,599 --> 00:40:51,800
But I think it's helpful.

929
00:40:52,079 --> 00:40:54,079
Speaker 1: Yeah, I mean that's the thing though, right, Like you're

930
00:40:54,199 --> 00:40:57,719
utilizing in an intelligent way to critique what you've got

931
00:40:57,760 --> 00:40:59,079
and be willing to throw away the.

932
00:40:59,000 --> 00:41:01,480
Speaker 4: Output rather than seeing it as the expert.

933
00:41:01,519 --> 00:41:03,119
Speaker 1: And I think this is the metric that we've come

934
00:41:03,199 --> 00:41:05,960
up with internally actually at my company, which is realistically

935
00:41:06,360 --> 00:41:08,960
how like who's ever using l M Are they using

936
00:41:08,960 --> 00:41:12,119
it in a critical first manner where they're challenging the

937
00:41:12,199 --> 00:41:14,800
output as whether or not not just you know, what

938
00:41:14,800 --> 00:41:17,519
it's saying, but whether or not it is accurate, rather

939
00:41:17,559 --> 00:41:20,119
than taking it for granted and believing that the lms

940
00:41:20,199 --> 00:41:23,639
always give you better than accurate answers. So, you know,

941
00:41:23,719 --> 00:41:26,199
as an expert, I think the ad in the area

942
00:41:26,519 --> 00:41:29,920
and also being critical of the you know results, I mean,

943
00:41:29,920 --> 00:41:31,800
you're you're a you're in a special mode where you're

944
00:41:31,800 --> 00:41:34,480
actually looking for holes in what you're saying, and so

945
00:41:34,679 --> 00:41:36,559
taking each one of those as valid arguments is a

946
00:41:36,559 --> 00:41:38,840
great way of utilizing it.

947
00:41:38,880 --> 00:41:40,119
Speaker 4: So I applaud you for that.

948
00:41:40,159 --> 00:41:42,039
Speaker 1: You're definitely not one of those leaders who is just

949
00:41:43,360 --> 00:41:45,280
tracking people on their LLM usage.

950
00:41:46,960 --> 00:41:49,119
Speaker 3: I'm curious what your thoughts are on Like you mentioned,

951
00:41:49,119 --> 00:41:50,960
you're always going to say LM as long as this

952
00:41:51,199 --> 00:41:55,199
podcast is happening, what do you think about like the

953
00:41:55,239 --> 00:41:57,599
models progressing in the sense of, like everything we're talking

954
00:41:57,639 --> 00:42:00,239
about has a lot of bigg assumptions that the the

955
00:42:00,320 --> 00:42:02,119
quality of the models is going to stay around the

956
00:42:02,159 --> 00:42:05,000
same amount. But what if they do get so good

957
00:42:05,000 --> 00:42:07,480
that they are the same quality of as like a

958
00:42:07,519 --> 00:42:10,519
certain tasks that you would do already. Okay, so what happens?

959
00:42:11,280 --> 00:42:13,599
Speaker 1: I guess I haven't shared this out loud too many

960
00:42:13,639 --> 00:42:17,199
times before, so this will be a special thing.

961
00:42:17,119 --> 00:42:18,039
Speaker 4: For our viewers here.

962
00:42:18,440 --> 00:42:21,599
Speaker 1: So I think so far we've used lms to solve

963
00:42:21,719 --> 00:42:24,559
easy problems, and the idea that they'll just keep on

964
00:42:24,599 --> 00:42:27,840
getting better is a little bit misguided because at some

965
00:42:27,920 --> 00:42:30,239
point we're going to have to solve a hard problem,

966
00:42:30,280 --> 00:42:33,360
and no hard problems have ever been solved as far

967
00:42:33,400 --> 00:42:36,119
as the creation of lms go, So it's actually a

968
00:42:36,159 --> 00:42:40,039
technical difficulty and maybe we're at a fundamental limit to

969
00:42:40,079 --> 00:42:45,760
actually getting there. A common argument against this is that

970
00:42:46,320 --> 00:42:50,800
humans are biological computers, and if we're just a machine,

971
00:42:50,840 --> 00:42:52,719
then of course we can build a machine or a

972
00:42:52,719 --> 00:42:55,840
computer to compete with that. But that statement is an

973
00:42:55,840 --> 00:42:59,239
analogy which doesn't actually mean it's true. What if we're

974
00:42:59,280 --> 00:43:02,960
not biological computers, which means that we can't just make

975
00:43:03,000 --> 00:43:05,840
a computer better to solve problems that humans can solve.

976
00:43:06,079 --> 00:43:09,800
So it begs the question are humans biological computers? And

977
00:43:09,840 --> 00:43:12,039
you'd have to prove the answer is yes first before

978
00:43:12,079 --> 00:43:14,760
you can prove that a turn complete language or a

979
00:43:14,800 --> 00:43:17,039
turning machine, or something that can be distilled down to

980
00:43:17,039 --> 00:43:20,440
basically just a turning machine can solve more difficult problems.

981
00:43:20,639 --> 00:43:22,239
Speaker 4: So no one's proven that yet.

982
00:43:22,239 --> 00:43:24,039
Speaker 1: So we're still at the point where for sure, we

983
00:43:24,119 --> 00:43:26,679
don't have what I'll call AI, which is a replication

984
00:43:26,880 --> 00:43:30,159
of the intelligence that let's say humans have, I mean

985
00:43:30,480 --> 00:43:32,920
not even getting into the story of like other species

986
00:43:33,039 --> 00:43:36,280
or sentience or anything like that. So that's the first

987
00:43:36,280 --> 00:43:39,559
step really there for me and all the things that

988
00:43:39,559 --> 00:43:42,199
we've seen innovated in the last five years come out

989
00:43:42,239 --> 00:43:45,079
of the transformer architecture paper that was written at Google,

990
00:43:46,360 --> 00:43:48,519
and we haven't really gotten any better than that all

991
00:43:48,519 --> 00:43:51,079
the improvements we've made. A good example is like, well

992
00:43:51,199 --> 00:43:53,360
it couldn't do math before, and now it can start

993
00:43:53,360 --> 00:43:55,920
trying to do math. Well, that's easy. You just reject

994
00:43:56,079 --> 00:43:58,800
the result. You say, hey, LM, where's the math here?

995
00:43:59,079 --> 00:44:01,679
You'd get you extra the math. You send it to

996
00:44:01,760 --> 00:44:04,440
some mathematical solver, get back the result, and plug it

997
00:44:04,440 --> 00:44:05,480
in as part of your answer.

998
00:44:06,159 --> 00:44:07,039
Speaker 4: Stuff like RAG.

999
00:44:07,239 --> 00:44:10,599
Speaker 1: Yeah, it's sort of interesting resource augmented generation where you

1000
00:44:10,679 --> 00:44:13,039
do part of the LM response. You send it to

1001
00:44:13,039 --> 00:44:15,760
your database, get the query, get the results back included

1002
00:44:15,800 --> 00:44:18,840
in the last level of your transform architecture, and generate

1003
00:44:18,880 --> 00:44:22,400
the real result. Yes, that's still solving a simple problem

1004
00:44:22,480 --> 00:44:25,239
making it more accurate, but it's not getting over the

1005
00:44:25,280 --> 00:44:26,000
real hard problem.

1006
00:44:26,039 --> 00:44:27,360
Speaker 4: And that's why I get stuck with this.

1007
00:44:27,800 --> 00:44:28,440
Speaker 2: That makes sense.

1008
00:44:28,559 --> 00:44:31,119
Speaker 3: I mean, I will tell you I'm a huge LLLM

1009
00:44:31,159 --> 00:44:33,679
believer in a lot of ways, so this is a

1010
00:44:33,679 --> 00:44:36,199
fun discussion. I feel like, though the value I see

1011
00:44:36,199 --> 00:44:40,039
in using LMS isn't solving hard problems. It's like speeding

1012
00:44:40,119 --> 00:44:41,760
up on all the things that don't need to be

1013
00:44:41,840 --> 00:44:45,679
high quality basically, and so like the value of speed

1014
00:44:46,000 --> 00:44:48,639
is like a lot of times, if we think about

1015
00:44:48,639 --> 00:44:51,159
things in engineering, if there is a speed to quality

1016
00:44:51,199 --> 00:44:53,000
trade off, it's like, Okay, you probably don't want that.

1017
00:44:53,079 --> 00:44:55,760
Like the a lot of times the opportunity cost of

1018
00:44:55,800 --> 00:44:57,800
like shipping a bug or something like that is so high.

1019
00:44:58,000 --> 00:45:01,760
But in a lot of like every operations, speed is

1020
00:45:01,800 --> 00:45:04,599
like very valuable, right, And like if you know, coming

1021
00:45:04,599 --> 00:45:07,800
from a startup perspective, being able to automate a lot

1022
00:45:07,840 --> 00:45:10,119
of things and build faster and like win more of

1023
00:45:10,159 --> 00:45:12,519
the market quicker is very high value. And so I

1024
00:45:12,559 --> 00:45:14,880
totally hear you. And there's like, you know, the big

1025
00:45:14,920 --> 00:45:18,960
discussion about like our human humans ultimately a machine, but

1026
00:45:19,079 --> 00:45:23,199
like there's really that on the side. But I do

1027
00:45:23,239 --> 00:45:25,039
see a ton of value in it, even if it

1028
00:45:25,119 --> 00:45:26,559
never can solve hard problems.

1029
00:45:27,039 --> 00:45:29,119
Speaker 4: I want to I want to be clear here that our.

1030
00:45:30,519 --> 00:45:33,360
Speaker 1: Categorization of whether a problem is hard or not isn't

1031
00:45:33,440 --> 00:45:37,199
actually my challenge to the l Alams. It's in the

1032
00:45:37,599 --> 00:45:40,639
manufacturing of l alams. What problems have we had to

1033
00:45:40,679 --> 00:45:44,440
overcome and order to manufacture them? The interesting the next

1034
00:45:44,519 --> 00:45:48,159
levels like basically what we have today is a statistical

1035
00:45:48,199 --> 00:45:52,280
next word predictors and that has been a thing since

1036
00:45:52,360 --> 00:45:53,840
like nineteen fifty eight or something.

1037
00:45:54,519 --> 00:45:57,039
Speaker 4: Uh, And I I don't know if that's the year.

1038
00:45:57,119 --> 00:45:59,079
Speaker 1: I'm terrible with years, but I swear there was a

1039
00:45:59,119 --> 00:46:03,400
Veritassic video where we're actually talking about this. For any

1040
00:46:03,440 --> 00:46:05,280
of the viewers who go to a Veritasium is like

1041
00:46:05,719 --> 00:46:09,480
highly rated, fantastic YouTube channel, you should definitely go out

1042
00:46:09,480 --> 00:46:11,119
and subscribe that. It's not going to be my pick

1043
00:46:11,119 --> 00:46:12,480
for this episode, because it was a pick for a

1044
00:46:12,519 --> 00:46:16,440
previous episode actually talks about this, And yes, we got

1045
00:46:16,480 --> 00:46:18,199
better at next word predicting, and that's.

1046
00:46:18,119 --> 00:46:18,599
Speaker 4: All we're doing.

1047
00:46:18,599 --> 00:46:21,519
Speaker 1: We're just keep improving our ability to predict next words

1048
00:46:21,559 --> 00:46:24,320
better by not only using the previous token or the

1049
00:46:24,320 --> 00:46:27,159
previous word or the previous paragraph, but also pulling in

1050
00:46:27,239 --> 00:46:29,400
all the context everywhere. We're getting better at that and

1051
00:46:29,440 --> 00:46:32,480
building better technology to solve that problem. But all we're

1052
00:46:32,559 --> 00:46:35,159
doing is improving the statistical analysis, and we have to

1053
00:46:35,960 --> 00:46:39,159
change fundamentally the technology to get much further away from

1054
00:46:39,199 --> 00:46:42,159
that or else will never eliminate hallucinations. And I think

1055
00:46:42,360 --> 00:46:44,679
that's one of the biggest challenges that we have. So

1056
00:46:44,719 --> 00:46:47,599
when I say solve hard problems. I mean until someone

1057
00:46:47,679 --> 00:46:51,800
has a new technology that fundamentally eliminates hallucinations, will never

1058
00:46:51,880 --> 00:46:54,079
have loms that I'm comfortable calling AI.

1059
00:46:54,480 --> 00:46:58,679
Speaker 2: I'm curious. Have you tried ide LM usage like cursor?

1060
00:46:59,519 --> 00:47:01,320
Speaker 4: I have. I have tried these things.

1061
00:47:01,920 --> 00:47:05,840
Speaker 1: I don't get very far because I find a lot

1062
00:47:05,920 --> 00:47:09,800
of the work goes into articulating with words what the

1063
00:47:09,840 --> 00:47:13,079
problem is. And once I've done that, I've done ninety

1064
00:47:13,079 --> 00:47:15,039
percent of the work, and doing the last part of

1065
00:47:15,079 --> 00:47:17,320
it is now a fight with an LM to even

1066
00:47:17,360 --> 00:47:21,719
produce the appropriate results. So an example of something like

1067
00:47:21,920 --> 00:47:25,320
I never use it to generate code whatsoever for two reasons.

1068
00:47:25,320 --> 00:47:25,719
Speaker 4: Actually.

1069
00:47:25,880 --> 00:47:28,639
Speaker 1: The first one is it's usually in a domain that

1070
00:47:28,639 --> 00:47:31,960
there aren't good correct examples, and it's like security related,

1071
00:47:32,400 --> 00:47:34,880
and usually the outcomes that I get.

1072
00:47:34,719 --> 00:47:36,400
Speaker 4: Are have to have high quality.

1073
00:47:36,519 --> 00:47:38,480
Speaker 1: An example where I did use it is I wanted

1074
00:47:39,039 --> 00:47:42,280
to use a government website that requires you to click

1075
00:47:42,320 --> 00:47:44,599
a link and schedule a meeting, and there are no

1076
00:47:44,639 --> 00:47:47,159
meetings available in any close location that I can possibly

1077
00:47:47,159 --> 00:47:49,880
get to, So I wanted a bunch of scripts that

1078
00:47:50,079 --> 00:47:53,880
goes and uses the CURL command to download the open

1079
00:47:54,159 --> 00:47:56,400
schedule appointments and then filter them and do something else.

1080
00:47:56,400 --> 00:47:58,440
And I'm like, I don't care the quality of this.

1081
00:47:58,440 --> 00:48:00,360
I don't care if it crashes whatever, I can iterate

1082
00:48:00,400 --> 00:48:02,519
on it, just go in throw it at that and

1083
00:48:02,559 --> 00:48:04,679
so like we'll definitely use that, and so like it

1084
00:48:04,760 --> 00:48:07,239
helps me get to that answer faster, and I don't

1085
00:48:07,280 --> 00:48:09,320
care about the accuracy or quality or whatever.

1086
00:48:10,320 --> 00:48:11,159
Speaker 4: That's a good example.

1087
00:48:11,199 --> 00:48:14,000
Speaker 1: But in anything that I absolutely do care about, it

1088
00:48:14,519 --> 00:48:15,800
only gets in my way for sure.

1089
00:48:16,039 --> 00:48:19,119
Speaker 3: Yeah, that makes a lot of sense. The reason I

1090
00:48:19,159 --> 00:48:21,480
ask is because like I think that the way that

1091
00:48:21,519 --> 00:48:24,960
Cursor has handled the user experience of hallucination is really

1092
00:48:25,000 --> 00:48:26,800
good in the sense Cursor and similar things like the

1093
00:48:26,880 --> 00:48:30,760
inscode also has but in that it returns code examples

1094
00:48:30,760 --> 00:48:32,719
for everything it says, and so like if you click

1095
00:48:32,760 --> 00:48:34,400
on the thing and it does not exist, then like

1096
00:48:34,880 --> 00:48:36,960
you know right away this was hallucinated. So there's like

1097
00:48:37,000 --> 00:48:40,119
a very good on the rails of like everything across

1098
00:48:40,159 --> 00:48:42,320
my code base needs to have a reference, and those

1099
00:48:42,320 --> 00:48:45,519
models work fairly well. But that was mainly just like

1100
00:48:45,559 --> 00:48:48,360
a thought experiment. It is interesting though, that space where

1101
00:48:48,400 --> 00:48:51,079
you have very little documentation, that's tough, and like we

1102
00:48:51,119 --> 00:48:53,159
have edge cases like that at plumy as well. But

1103
00:48:53,199 --> 00:48:54,920
there's use cases and I wonder if you guys run

1104
00:48:54,920 --> 00:48:57,760
into this given what you build, where if let's say,

1105
00:48:57,760 --> 00:49:00,559
for example, we implement something in Java, we have a

1106
00:49:00,679 --> 00:49:03,599
very good context reference for the model to go do

1107
00:49:03,679 --> 00:49:05,840
it in multiple languages. And that's a model that works

1108
00:49:05,840 --> 00:49:08,440
pretty well because the ele ones aren't good at novel things, right,

1109
00:49:08,480 --> 00:49:10,199
but if you give them an example, they're pretty good

1110
00:49:10,199 --> 00:49:12,760
at like using their knowledge base to figure out how

1111
00:49:12,800 --> 00:49:13,960
to do it in another language.

1112
00:49:14,079 --> 00:49:17,039
Speaker 1: So hypothetically translation from one language to another one, especially

1113
00:49:17,079 --> 00:49:20,159
like natural human languages, is the exact way in which

1114
00:49:20,199 --> 00:49:23,079
the models were built. And obviously large language models are

1115
00:49:23,079 --> 00:49:27,440
still slightly different, depending if they're or human readable or

1116
00:49:27,480 --> 00:49:33,079
understandable language or some other custom lexicon that is map

1117
00:49:33,119 --> 00:49:37,480
to your domain or even like software development. However, this

1118
00:49:37,639 --> 00:49:40,079
was actually one of our primary examples where we were struggling.

1119
00:49:40,519 --> 00:49:44,519
We need to write JBTS or jaw Jason webtokens de

1120
00:49:44,559 --> 00:49:48,960
serialization and token validation for security purposes, and we need

1121
00:49:49,000 --> 00:49:52,599
an example for every single language, and some languages are

1122
00:49:52,760 --> 00:49:55,079
very easy and come up with a correct answer because

1123
00:49:55,079 --> 00:49:56,960
there are a library is dedicated to solving this problem.

1124
00:49:57,360 --> 00:50:01,079
In other languages, the primitives don't really exist that well,

1125
00:50:01,559 --> 00:50:05,719
and you have to stack them all up together in

1126
00:50:05,760 --> 00:50:08,320
a complex way that no one's ever really done or

1127
00:50:08,360 --> 00:50:10,400
the people have done it. It don't really work anymore

1128
00:50:10,440 --> 00:50:13,079
with the particular versions that are available, et cetera, et cetera,

1129
00:50:13,159 --> 00:50:16,960
and the models are trocious at that. So even knowing

1130
00:50:17,039 --> 00:50:19,400
how this should work in ninety nine percent of the

1131
00:50:19,519 --> 00:50:22,400
implementations across all the languages still does not help you

1132
00:50:22,800 --> 00:50:25,320
get the last one out. And this is actually I

1133
00:50:25,400 --> 00:50:28,400
used to think something like, oh, all languages are pretty

1134
00:50:28,440 --> 00:50:30,960
much the same for the most part. There's some built

1135
00:50:31,000 --> 00:50:33,559
in stuff that causes you to write code one way

1136
00:50:33,639 --> 00:50:34,880
or another one. But now I can come out here

1137
00:50:34,920 --> 00:50:37,400
and actually say some languages cause you to write more

1138
00:50:37,440 --> 00:50:40,239
insecure code than other languages. For instance, I can tell

1139
00:50:40,239 --> 00:50:44,119
you very specifically Python and Ruby are more insecure languages

1140
00:50:44,199 --> 00:50:47,639
than even Php and JavaScript because the code that comes

1141
00:50:47,679 --> 00:50:50,559
out of those is less they are less examples of

1142
00:50:50,599 --> 00:50:54,400
having secure code be generated, and so models will more

1143
00:50:54,519 --> 00:50:57,079
likely write insecure code for those languages. So if security

1144
00:50:57,119 --> 00:50:59,000
is a concern for you, stop using those languages.

1145
00:50:59,199 --> 00:51:00,880
Speaker 3: And that's just like, sorry, why do you have that

1146
00:51:01,320 --> 00:51:04,880
observation for Python RVIE specifically because of like there's more

1147
00:51:04,920 --> 00:51:06,880
hobbyists like building with them, and so you.

1148
00:51:06,840 --> 00:51:10,960
Speaker 1: Get well, yes, exactly, there's there's almost no examples of

1149
00:51:11,639 --> 00:51:14,079
getting this right or like trying to get it right,

1150
00:51:14,920 --> 00:51:16,719
of what we actually need. And I think there was

1151
00:51:16,719 --> 00:51:20,320
an example from even a cloud Flairer did an experiment

1152
00:51:20,320 --> 00:51:23,679
where they were generating an o ath to compatible client,

1153
00:51:24,199 --> 00:51:29,079
and an expert in identity providers and oth to, you know,

1154
00:51:29,119 --> 00:51:31,280
went and tried to get it done, and there's just

1155
00:51:31,559 --> 00:51:34,159
riddle lots of problems and you just won't even see these.

1156
00:51:34,199 --> 00:51:37,119
And in some languages there are working examples of this,

1157
00:51:37,199 --> 00:51:39,800
and other languages there are not very good working examples

1158
00:51:39,800 --> 00:51:40,000
of this.

1159
00:51:40,360 --> 00:51:42,519
Speaker 4: So you know, if you just have no.

1160
00:51:42,480 --> 00:51:43,880
Speaker 1: Idea what you're doing here, or even if you do,

1161
00:51:44,159 --> 00:51:47,400
trying to get it to pop out just will always

1162
00:51:47,400 --> 00:51:50,480
be a problem, unfortunately. And I think I'm going to

1163
00:51:50,559 --> 00:51:53,119
keep repeating this because I like this idea that the

1164
00:51:53,159 --> 00:51:57,840
next successful programming language that humans utilize will be one

1165
00:51:57,880 --> 00:52:02,719
that is optimized. For examples, for LLM, so the generation

1166
00:52:02,800 --> 00:52:05,039
of code and also consumption as far as context goes,

1167
00:52:05,280 --> 00:52:07,239
rather than what we're doing today, which is like automating

1168
00:52:07,239 --> 00:52:10,039
the hands on the keyboard where you see we try

1169
00:52:10,039 --> 00:52:12,480
to merge all the code together and optimize the context

1170
00:52:12,480 --> 00:52:14,840
that we're passing to cursor or windsurf or whatever so

1171
00:52:14,880 --> 00:52:17,079
that it can actually fit all that all the tokens

1172
00:52:17,079 --> 00:52:19,079
in its context window. I think we're going to start

1173
00:52:19,079 --> 00:52:22,079
seeing new languages that are that are terrible to program

1174
00:52:22,119 --> 00:52:25,400
with but are great for llms to generate, because at

1175
00:52:25,440 --> 00:52:27,119
the end of the day, we want the working program

1176
00:52:27,159 --> 00:52:29,199
more than we care about the code that's actually being used.

1177
00:52:29,480 --> 00:52:30,280
Speaker 2: That's so interesting.

1178
00:52:30,360 --> 00:52:32,440
Speaker 3: So do you feel like languages like Java are probably

1179
00:52:32,719 --> 00:52:35,760
great by that the same framing where it's mainly enterprised

1180
00:52:35,800 --> 00:52:38,559
people who are using it and have examples online.

1181
00:52:38,760 --> 00:52:41,159
Speaker 1: Yeah, I mean, I think the examples and of the

1182
00:52:41,199 --> 00:52:43,920
thing that you're trying to do is paramount or using

1183
00:52:43,960 --> 00:52:45,719
the LLM. So if you're trying to do something that

1184
00:52:45,719 --> 00:52:48,159
no one has written before or isn't frequently done in

1185
00:52:48,199 --> 00:52:50,719
that language, yeah, for sure, stop doing that and you

1186
00:52:50,760 --> 00:52:52,199
can you can actually perform this test.

1187
00:52:52,559 --> 00:52:53,320
Speaker 4: It's pretty interesting.

1188
00:52:53,360 --> 00:52:55,239
Speaker 1: Go to an LLM, don't tell what language to use,

1189
00:52:55,400 --> 00:52:57,079
and give it a problem to solve, and see what

1190
00:52:57,199 --> 00:52:59,760
language it picks to write the solution. In and it

1191
00:52:59,760 --> 00:53:02,119
will pick different languages based off of the problem you're solving,

1192
00:53:02,239 --> 00:53:04,880
and that should actually tell you stop picking the language.

1193
00:53:05,039 --> 00:53:06,639
The LM will pick the language for you, and you

1194
00:53:06,639 --> 00:53:08,719
should use that one because it may not even be

1195
00:53:08,800 --> 00:53:11,000
correct in other language, or it may not even be possible.

1196
00:53:11,199 --> 00:53:12,480
Speaker 2: That's funny, that's very interesting.

1197
00:53:13,320 --> 00:53:16,199
Speaker 3: Well, yeah, I'm very applicable to Plumy's world of supporting

1198
00:53:16,639 --> 00:53:17,480
programming languages.

1199
00:53:18,599 --> 00:53:21,760
Speaker 1: I totally agree with that. So, I mean, you do,

1200
00:53:21,920 --> 00:53:23,360
you are hitting for sure a lot of points. And

1201
00:53:23,400 --> 00:53:25,239
as much as I would love to have a whole

1202
00:53:27,599 --> 00:53:30,679
debate on to LM or or not to LM, I

1203
00:53:30,920 --> 00:53:34,039
think I think the agreement is likely. There are use

1204
00:53:34,079 --> 00:53:37,039
cases where it makes sense and ones that it should not,

1205
00:53:37,159 --> 00:53:39,159
And it's anyone's best bet what is going to happen,

1206
00:53:39,239 --> 00:53:42,280
even a couple of years from now. So I'd rather not,

1207
00:53:43,239 --> 00:53:44,679
you know, spend too much time speculating.

1208
00:53:44,840 --> 00:53:47,079
Speaker 3: Yeah, makes sense. I think just to like wrap a

1209
00:53:47,079 --> 00:53:49,079
bow on a lot of what we talked about. There's

1210
00:53:49,159 --> 00:53:51,400
a lot of change happening in the developer space right now,

1211
00:53:51,440 --> 00:53:54,000
and there's a lot of change happening in infrastructure, and

1212
00:53:54,559 --> 00:53:56,639
I think we've talked about a lot of it, which

1213
00:53:56,679 --> 00:53:59,639
is like speeding up and the importance of slowing down

1214
00:53:59,719 --> 00:54:02,360
and how you can have like checks and balances along

1215
00:54:02,400 --> 00:54:05,440
that process. And it's something that I'm really passionate about,

1216
00:54:06,559 --> 00:54:09,239
like helping build tools for which is how do you,

1217
00:54:09,239 --> 00:54:12,679
you know, feel confident about the changes you're making in

1218
00:54:12,719 --> 00:54:13,480
this new age.

1219
00:54:14,280 --> 00:54:16,840
Speaker 2: And so it's been a good conversation.

1220
00:54:17,079 --> 00:54:17,800
Speaker 4: Oh yeah, of course.

1221
00:54:17,880 --> 00:54:19,880
Speaker 1: So with that, I guess we can move over to

1222
00:54:19,960 --> 00:54:21,559
our last thing, which is a picks.

1223
00:54:22,159 --> 00:54:23,400
Speaker 4: So I'll go first.

1224
00:54:24,119 --> 00:54:27,440
Speaker 1: My pick is going to be a specific community dedicated

1225
00:54:27,440 --> 00:54:30,360
to leadership that anyone can join. It's called the Rand's

1226
00:54:30,400 --> 00:54:34,840
Leadership Community, and it's just I think it's over thirty

1227
00:54:34,920 --> 00:54:39,159
thousand people now from tech backgrounds, non tech backgrounds, but

1228
00:54:39,199 --> 00:54:43,159
work in tech adjacent stuff where I think even we

1229
00:54:43,440 --> 00:54:44,920
had a little bit of a not a great time

1230
00:54:44,960 --> 00:54:47,519
for leaders in the last maybe couple of years, where

1231
00:54:47,519 --> 00:54:50,039
companies were like, we don't need leaders, we have lms

1232
00:54:50,079 --> 00:54:52,719
to replace everything. But I think some of them are

1233
00:54:52,719 --> 00:54:54,599
starting to come around. I think it's going to be

1234
00:54:54,599 --> 00:54:57,840
another year or so and we're going to see engineers

1235
00:54:57,920 --> 00:55:01,880
and other your colleagues that don't have leadership experience. And

1236
00:55:02,000 --> 00:55:03,960
if you find yourself lost and no one at the

1237
00:55:03,960 --> 00:55:07,400
company can help you. The community exists to be able

1238
00:55:07,440 --> 00:55:10,079
to ask questions to and get feedback and how to

1239
00:55:10,079 --> 00:55:14,559
grow in your career or just solve standard leadership manager

1240
00:55:14,719 --> 00:55:17,440
like questions. And I think out of every community I've

1241
00:55:17,440 --> 00:55:19,199
been in, it's for sure one of the best. You

1242
00:55:19,199 --> 00:55:21,239
do talk about leadership a little bit on this podcast,

1243
00:55:22,039 --> 00:55:23,639
because I do feel like it is in the back

1244
00:55:23,679 --> 00:55:25,400
of a lot of people's heads and there's a lot

1245
00:55:25,440 --> 00:55:26,320
of different things you can do.

1246
00:55:26,519 --> 00:55:28,840
Speaker 3: What's like an example of something you would talk about

1247
00:55:28,880 --> 00:55:31,880
in this community, Like I am struggling to think about

1248
00:55:31,880 --> 00:55:34,360
what is leadership on a lower level?

1249
00:55:34,440 --> 00:55:39,000
Speaker 1: Cus Well, I think it's it's not actually topics specific,

1250
00:55:39,000 --> 00:55:41,599
it's more of like how you approach any particular topic,

1251
00:55:41,719 --> 00:55:44,039
so like one of the most controversial things. And I

1252
00:55:44,039 --> 00:55:46,719
don't think I'm violating any rules of the community by

1253
00:55:46,719 --> 00:55:49,480
saying this that even outside the community is like micro

1254
00:55:49,519 --> 00:55:51,039
services versus modolists.

1255
00:55:51,559 --> 00:55:54,119
Speaker 4: And the interesting thing is when someone.

1256
00:55:53,920 --> 00:55:55,719
Speaker 1: Posts a question in the community like oh, I had

1257
00:55:55,719 --> 00:55:59,320
this problem, should we switch to micro services? You may

1258
00:55:59,320 --> 00:56:01,639
get a debate like which one's better, but you'll often

1259
00:56:01,760 --> 00:56:03,519
get a question like, well, why do you want to

1260
00:56:03,519 --> 00:56:05,920
do that? Like, what's the core problem you're trying to solve.

1261
00:56:06,440 --> 00:56:09,239
Is it a technical challenge or is it an organizational

1262
00:56:09,280 --> 00:56:12,880
issue or a culture issue or you know, an interpersonal one.

1263
00:56:13,400 --> 00:56:14,440
Speaker 4: Are the inssentives in line?

1264
00:56:14,480 --> 00:56:16,480
Speaker 1: You know what's going on there, and then the conversation

1265
00:56:16,559 --> 00:56:18,960
may pivot to actually talking about that, And so it

1266
00:56:19,039 --> 00:56:21,119
helps you see not just like whatever problem is in

1267
00:56:21,119 --> 00:56:23,239
front of you, but anything that could be happening. And

1268
00:56:23,280 --> 00:56:26,280
that's like on the technical side, there are like thousands

1269
00:56:26,280 --> 00:56:29,079
of channels on you know, whatever arbitrary topic you could

1270
00:56:29,119 --> 00:56:32,480
possibly imagine that could be relevant. So maybe you are

1271
00:56:32,559 --> 00:56:34,559
going through a reorg and you want to know how

1272
00:56:34,840 --> 00:56:37,760
like whether the messaging makes sense and you're not sure

1273
00:56:37,760 --> 00:56:40,079
how people will take it, or maybe you're dealing with

1274
00:56:40,280 --> 00:56:42,760
a boss or a manager who says, yes, you are

1275
00:56:42,760 --> 00:56:45,639
going to count your llm usages and you're looking for

1276
00:56:45,760 --> 00:56:48,000
arguments why that could be a good thing or how

1277
00:56:48,039 --> 00:56:50,519
to push back against it. And I feel like this

1278
00:56:50,599 --> 00:56:52,519
is a place where you can go and actually have

1279
00:56:52,599 --> 00:56:55,559
that conversation. And there may be people from other companies

1280
00:56:55,599 --> 00:56:57,000
that you have heard of or ones that you don't,

1281
00:56:57,000 --> 00:56:59,559
who have gone through a similar process and can provide

1282
00:56:59,559 --> 00:57:01,679
you inside into how they approach it or be a

1283
00:57:01,679 --> 00:57:03,000
thinking partner for how to solve it.

1284
00:57:03,039 --> 00:57:03,960
Speaker 2: Oh, that's very interesting.

1285
00:57:04,480 --> 00:57:08,360
Speaker 3: My pick for today is a book I am at

1286
00:57:08,400 --> 00:57:10,920
the moment thinking a lot about how to build great teams,

1287
00:57:11,599 --> 00:57:14,119
both in that you know, they high velocity, but also

1288
00:57:14,239 --> 00:57:16,119
just like they're a good culture to work in. People

1289
00:57:16,119 --> 00:57:18,440
are excited to be there, they're happy, you know, working

1290
00:57:18,440 --> 00:57:21,920
out what they're working on. So something A book that

1291
00:57:21,960 --> 00:57:25,320
I just finished is The Manager's Path by Camila Forearner,

1292
00:57:25,760 --> 00:57:28,840
and it talks a lot about like the transition from

1293
00:57:29,000 --> 00:57:31,039
you know, being a technical ice to a manager and

1294
00:57:31,079 --> 00:57:34,679
then like a leader within an organization. And there's a

1295
00:57:34,719 --> 00:57:37,920
lot of good topics everything from like how do you

1296
00:57:38,840 --> 00:57:41,840
step back in the technical strategy piece and like grow

1297
00:57:41,880 --> 00:57:44,840
people to play that role and like what your role becomes.

1298
00:57:44,880 --> 00:57:47,320
Speaker 2: So definitely would recommend.

1299
00:57:47,039 --> 00:57:49,559
Speaker 1: You're reading it and like preface for making sure that

1300
00:57:49,679 --> 00:57:53,119
your leaders are taking a path that you can approve of.

1301
00:57:53,440 --> 00:57:54,760
Speaker 3: Yeah, I also thing it's just like really good to

1302
00:57:54,800 --> 00:57:56,639
think through some of these topics, Like it just adds

1303
00:57:56,639 --> 00:58:00,440
different perspectives of how other companies do things. But also yeah,

1304
00:58:00,480 --> 00:58:01,920
even things like how do you run a one on

1305
00:58:01,920 --> 00:58:04,519
one and it gives you a framework of like here's

1306
00:58:04,639 --> 00:58:06,480
you know, a way to do it, and you might

1307
00:58:06,519 --> 00:58:08,519
not take all of it, but there's things, there's value,

1308
00:58:08,559 --> 00:58:11,000
there's nuggets all over to be able to to pick

1309
00:58:11,079 --> 00:58:11,639
up an adult.

1310
00:58:11,719 --> 00:58:13,519
Speaker 1: Yeah, I do think the book is pretty great in

1311
00:58:13,519 --> 00:58:15,960
that way that it's like if I've never done this

1312
00:58:16,000 --> 00:58:18,519
thing that, how would I even like what do I

1313
00:58:18,519 --> 00:58:21,360
even need to be aware of? And it's not true

1314
00:58:21,400 --> 00:58:22,800
that you need to be aware of all those things,

1315
00:58:22,880 --> 00:58:25,000
and but it's like here's a list and you can

1316
00:58:25,079 --> 00:58:27,480
ignore the list or you know, dive into it, and

1317
00:58:27,519 --> 00:58:29,199
then I think the most important thing, of course, is

1318
00:58:29,199 --> 00:58:32,719
adjusting to whatever situation that you're actually in. So I

1319
00:58:32,800 --> 00:58:35,519
think I think the Manager Paths great book, great pick.

1320
00:58:35,559 --> 00:58:37,039
Thank you for for sharing that.

1321
00:58:37,159 --> 00:58:40,199
Speaker 2: Yeah, thanks for having me on. Yeah, so debating m.

1322
00:58:42,559 --> 00:58:44,599
Speaker 1: You know, I worry sometimes that our podcast may go

1323
00:58:44,679 --> 00:58:46,840
too much in that in that direction, like there's a

1324
00:58:47,519 --> 00:58:49,880
this is a desire to either you know, jump up

1325
00:58:49,920 --> 00:58:53,719
and down and celebrate it or be critical of it.

1326
00:58:53,880 --> 00:58:55,719
So you know, we had a proponent on the show

1327
00:58:56,039 --> 00:58:59,440
this week. I think last week there was a fight

1328
00:58:59,480 --> 00:59:02,760
against it. So you know, everyone everyone can everyone can

1329
00:59:03,039 --> 00:59:05,639
can pick their their performance episode and with that I'll

1330
00:59:05,639 --> 00:59:08,800
say thank you Megan for coming into the show for us.

1331
00:59:08,800 --> 00:59:10,800
I think this has been a great episode, and uh,

1332
00:59:10,960 --> 00:59:14,880
thank you to all the viewers and listeners however you're

1333
00:59:14,960 --> 00:59:17,280
consuming this for for listening to this episode. And uh

1334
00:59:17,599 --> 00:59:18,840
we'll be back hopefully.

1335
00:59:21,119 --> 00:59:23,199
Speaker 4: H M.