1
00:00:00,960 --> 00:00:02,879
Speaker 1: Hey Richard, Hey Carl, what do you know?

2
00:00:03,120 --> 00:00:06,679
Speaker 2: Well, I know that our friend Michelle Rubusta Monte is

3
00:00:06,719 --> 00:00:08,880
with us to tell us about something that's going on

4
00:00:09,519 --> 00:00:11,199
adjacent to DEV Intersection.

5
00:00:11,240 --> 00:00:15,560
Speaker 1: What is it? It's cybersecurity Intersection. Let's let Michelle tell

6
00:00:15,599 --> 00:00:16,199
that story.

7
00:00:16,679 --> 00:00:21,320
Speaker 3: Hey Michelle, Hey Carl, Hey Richard, how are you.

8
00:00:21,000 --> 00:00:22,839
Speaker 2: Tell us about cybersecurity Intersection?

9
00:00:23,480 --> 00:00:26,920
Speaker 3: Well, so, Richard and I are partnering with the group

10
00:00:27,000 --> 00:00:30,320
that does DEV Intersection and next Gen AI, and we

11
00:00:30,600 --> 00:00:34,200
are putting on a new conference dedicated to one hundred

12
00:00:34,240 --> 00:00:40,000
percent security focused topics. And I mean, honestly, the lineup

13
00:00:40,039 --> 00:00:43,679
of speakers is incredible. We have Paula A. Jenis, who's

14
00:00:43,719 --> 00:00:47,600
here from Poland and does keynotes all over the world

15
00:00:47,719 --> 00:00:50,600
and is one of the top rated RSA speakers and

16
00:00:50,640 --> 00:00:53,159
black hat speaker. We're so lucky to have her. But

17
00:00:53,240 --> 00:00:56,000
she's not only keynoting, she's got a workshop teaches you

18
00:00:56,039 --> 00:01:01,039
about protecting your environments against hackers and shows you about

19
00:01:01,280 --> 00:01:03,719
how to you know, do attacks so that you can

20
00:01:03,759 --> 00:01:07,840
prevent them. It's pretty cool and sessions like that as well.

21
00:01:07,840 --> 00:01:10,799
But we also have speakers from Microsoft. We have we

22
00:01:10,840 --> 00:01:15,120
have speakers that specialize in you know secure coding practices,

23
00:01:15,640 --> 00:01:20,439
Azure security, Zuero, trust architectures on Azure UH and people

24
00:01:20,439 --> 00:01:23,519
who do decision maker tracks, so things around governance policy

25
00:01:23,560 --> 00:01:26,200
and you know how to how to manage and your

26
00:01:26,200 --> 00:01:29,480
production operations keep them secure. So it's an amazing group

27
00:01:29,519 --> 00:01:31,040
of speakers, really excited about it.

28
00:01:31,120 --> 00:01:33,560
Speaker 2: And I think I can count myself among the group

29
00:01:33,599 --> 00:01:34,599
of speakers there.

30
00:01:35,040 --> 00:01:37,120
Speaker 3: Well, yes you can. That is great.

31
00:01:37,480 --> 00:01:42,159
Speaker 2: Yeah, I'm doing a securing Blazer Server applications talk and

32
00:01:42,239 --> 00:01:45,959
also I think we're doing a Security this Week live

33
00:01:46,040 --> 00:01:48,480
show there somewhere that is correct.

34
00:01:48,719 --> 00:01:51,200
Speaker 3: Yeah, we'll be recording Security this Week Live. We're going

35
00:01:51,239 --> 00:01:54,719
to have a great panel with some folks. The interesting

36
00:01:54,719 --> 00:01:57,920
thing here is we don't really have a Microsoft and

37
00:01:58,000 --> 00:02:02,120
dot net and Azure focused toecurity conference yet, so that's

38
00:02:02,159 --> 00:02:05,159
the reason we're putting this on as well. You know

39
00:02:05,200 --> 00:02:08,280
there are other security conferences, but they have a spread

40
00:02:08,280 --> 00:02:10,439
of topics that maybe don't focus on the things you

41
00:02:10,520 --> 00:02:13,080
do day to day. And you know this overlaps with

42
00:02:13,199 --> 00:02:17,560
again our community of folks that specialize in again dot net,

43
00:02:17,680 --> 00:02:21,039
Azure and yeah, they need to keep it secure too,

44
00:02:21,280 --> 00:02:22,840
So with tons of talks.

45
00:02:23,719 --> 00:02:27,240
Speaker 1: Cyber Intersection is part of a trio of conferences we're doing.

46
00:02:27,280 --> 00:02:30,360
They have Intersection alongside the next Gen AI conference all

47
00:02:30,439 --> 00:02:34,680
in Orlando the week of October fifth through tenth. That's

48
00:02:34,759 --> 00:02:38,080
workshops and the main conference. And you can get a

49
00:02:38,120 --> 00:02:41,400
special registration code if you sign up through Cybersecurity Intersection

50
00:02:41,639 --> 00:02:42,360
dot com.

51
00:02:42,680 --> 00:02:47,400
Speaker 3: Yeah, so if you sign up at Cybersecurity Intersection dot com,

52
00:02:47,599 --> 00:02:52,520
then you put in this code so Alliance cyber three

53
00:02:52,599 --> 00:02:56,280
hundred and you'll get three hundred off the entry price.

54
00:02:56,560 --> 00:02:59,680
So that's a special code that only works at cybersecurity

55
00:03:00,599 --> 00:03:04,439
dot com. And then you have access to all the conferences.

56
00:03:04,479 --> 00:03:09,400
Speaker 2: Like Richard said, Wow, that's cool. Thanks Michelle. I'm looking

57
00:03:09,439 --> 00:03:24,120
forward to it and I'll see you there. Hey, get

58
00:03:24,159 --> 00:03:26,800
down rock and Roll. It's Carl Franklin and Richard Campbell

59
00:03:26,800 --> 00:03:27,759
for dot net Rocks.

60
00:03:28,000 --> 00:03:29,400
Speaker 1: Hey, Richard, how you do it, Bud?

61
00:03:29,479 --> 00:03:33,400
Speaker 2: I'm good, getting psyched up to go down to Orlando.

62
00:03:33,759 --> 00:03:37,240
Speaker 1: Yeah, it's almost time back to a new dev Intersection

63
00:03:37,400 --> 00:03:41,319
and next jen AI and the New Cybersecurity Conference side

64
00:03:41,360 --> 00:03:42,680
by side. Yep, yep.

65
00:03:42,879 --> 00:03:45,919
Speaker 2: Looking forward to doing a live security this week's show

66
00:03:46,120 --> 00:03:46,599
down there.

67
00:03:46,719 --> 00:03:50,479
Speaker 1: That should be fun, fun and you're crazy thing with Maddie.

68
00:03:50,560 --> 00:03:54,560
Oh god, you're going to aspireify dot net rocks here.

69
00:03:54,599 --> 00:03:56,639
Speaker 2: I have no idea what to expect. That could be

70
00:03:56,680 --> 00:03:57,439
a horror show.

71
00:03:57,879 --> 00:04:00,319
Speaker 1: This is you know, you love a good you know,

72
00:04:00,439 --> 00:04:02,960
trapease act, just going without a net.

73
00:04:03,080 --> 00:04:07,919
Speaker 2: Absolutely, as long as I don't you know, screw up

74
00:04:07,919 --> 00:04:10,000
too badly, it should it should work out fun.

75
00:04:10,039 --> 00:04:13,240
Speaker 1: You know, a good crash it burns fun too, but it.

76
00:04:13,199 --> 00:04:16,800
Speaker 2: Could be fun. Yeah, yeah, yeah, Okay, let's start with

77
00:04:16,879 --> 00:04:20,720
nineteen seventy. That's the episode number. Oh yeah, and a

78
00:04:21,000 --> 00:04:23,199
bunch of things happened in nineteen seventy.

79
00:04:23,480 --> 00:04:24,519
Speaker 1: Where do you want to start?

80
00:04:24,639 --> 00:04:27,279
Speaker 2: Well, the unhappy things, the Kent State shootings.

81
00:04:27,399 --> 00:04:28,399
Speaker 1: Yeah, it's terrifying.

82
00:04:28,639 --> 00:04:32,000
Speaker 2: On May fourth, National Guard troops killed four students during

83
00:04:32,000 --> 00:04:35,639
protest against the Vietnam War at Kent State University in Ohio,

84
00:04:37,519 --> 00:04:42,360
leading to nationwide outrage and the song what is it

85
00:04:42,399 --> 00:04:43,600
for Dead in Ohio?

86
00:04:43,800 --> 00:04:44,160
Speaker 1: Who's that?

87
00:04:44,319 --> 00:04:47,759
Speaker 2: Neil Young or Crosby Sills, Nash and Young? I'm not sure.

88
00:04:49,600 --> 00:04:55,399
Nigerian Civil War. The conflict ended in January when Biaffron

89
00:04:55,560 --> 00:04:59,160
forces the Affron forces surrendered after a thirty two month

90
00:04:59,199 --> 00:05:03,439
struggle for independ and it's the first Earth Day was observed.

91
00:05:03,480 --> 00:05:07,279
On April twenty second, the Beatles broke up and let

92
00:05:07,319 --> 00:05:10,439
it be. McCartney said he was leaving the band on

93
00:05:10,480 --> 00:05:13,199
April tenth. That was the end of that. But John

94
00:05:13,279 --> 00:05:17,040
Lennon instant karma. He wrote and recorded this hit song

95
00:05:17,079 --> 00:05:22,519
in a single day, showcasing his prolific creativity. Diana Ross

96
00:05:22,519 --> 00:05:25,920
and the Supremes gave their final concert in Las Vegas

97
00:05:25,920 --> 00:05:31,600
in January fourteenth. Back to the bad stuff the Tonguhai earthquake.

98
00:05:32,439 --> 00:05:38,120
Devastating earthquakes struck Tongue High County, China on January fifth,

99
00:05:38,160 --> 00:05:42,720
resulting in significant casualties, with estimates of up to fourteen thousand,

100
00:05:43,079 --> 00:05:47,480
six hundred and twenty one deaths. Yeah, and an avalanche

101
00:05:47,560 --> 00:05:51,000
in someplace that in France that I can't pronounce zi

102
00:05:52,680 --> 00:05:56,480
fool full, Sorry about that killed forty two people, making

103
00:05:56,480 --> 00:06:00,519
one of the worst disasters in French skiing history. You

104
00:06:00,560 --> 00:06:04,920
can talk about the science, yea science. Some things happened,

105
00:06:05,079 --> 00:06:05,959
well I was.

106
00:06:06,480 --> 00:06:09,360
Speaker 1: I mean the space one's the obvious one. After having

107
00:06:09,519 --> 00:06:12,759
both Apollo nine, Pollo ten, and Paula eleven and Apollo

108
00:06:12,839 --> 00:06:15,360
twelve all in nineteen sixty nine, there was only one

109
00:06:15,360 --> 00:06:19,319
Apollo mission in nineteen seventeen. That was a Poulla thirteen.

110
00:06:19,399 --> 00:06:23,120
It launched on April eleventh, and on April thirteenth they said,

111
00:06:23,720 --> 00:06:25,000
we've had a problem.

112
00:06:24,600 --> 00:06:28,279
Speaker 2: Here, Houston. We've got a problem. And we're a great

113
00:06:28,319 --> 00:06:28,879
movie too.

114
00:06:29,040 --> 00:06:31,680
Speaker 1: Yeah, and you've seen the movie, a beautiful rendering of

115
00:06:32,319 --> 00:06:35,720
more or less what happened. The HBO Earth of the

116
00:06:35,720 --> 00:06:38,839
Moon series, if you ever get a chance to watch,

117
00:06:38,959 --> 00:06:42,519
that does a version of Apollo thirteen, but from the

118
00:06:42,639 --> 00:06:45,480
view of the people on the ground, so you only

119
00:06:45,519 --> 00:06:48,720
ever hear the astronauts over the radio, which is how

120
00:06:48,800 --> 00:06:51,839
it was. Right. Sure, here's the crazy thing to realize.

121
00:06:51,920 --> 00:06:54,600
So the explosion in the tank happens on April on

122
00:06:54,680 --> 00:06:59,920
April thirteenth, the splash downy April seventeenth. It was four days. Wow,

123
00:07:00,160 --> 00:07:02,560
the whole thing's four days. I know, it feels like forever.

124
00:07:03,120 --> 00:07:03,800
It's four days.

125
00:07:03,800 --> 00:07:04,079
Speaker 2: Wow.

126
00:07:04,079 --> 00:07:05,959
Speaker 1: But it was four days of are these guys going

127
00:07:06,040 --> 00:07:08,800
to make it? You know, like four days of sheer terror. Yeah,

128
00:07:08,800 --> 00:07:12,759
it was. And of course they the lunar module Aquarius

129
00:07:12,879 --> 00:07:16,079
was turned into a lifeboat because the power systems, a

130
00:07:16,079 --> 00:07:17,639
little bit of battery that was left in the command

131
00:07:17,680 --> 00:07:19,199
module is going to need for re entry, so they

132
00:07:19,319 --> 00:07:22,279
basically powered down the command module and then use the

133
00:07:22,480 --> 00:07:26,600
Life Sports system for two four to three and just

134
00:07:26,720 --> 00:07:28,800
four days and they were able to get home amazing

135
00:07:28,839 --> 00:07:31,560
and survive. It's a great story. And of course the

136
00:07:31,600 --> 00:07:33,720
next Apollo mission would be delayed while they dealt with

137
00:07:33,720 --> 00:07:36,040
some of those issues, and in nineteen seventy one, you'll

138
00:07:36,040 --> 00:07:39,279
get Apollo fourteen. Talk about that next week apparently. Yeah.

139
00:07:39,319 --> 00:07:44,160
On the computer side of things, Nicholas Worth releases Pascal Woll.

140
00:07:44,560 --> 00:07:47,160
He worked previously on the language I'll Go sixty and

141
00:07:47,199 --> 00:07:49,680
there's some derivations therein he was trying to do a

142
00:07:49,680 --> 00:07:56,600
combination of sort of procedural and algorithmic programming. So Popular

143
00:07:56,680 --> 00:08:00,560
Language did some great things. But on the heart work side,

144
00:08:00,600 --> 00:08:04,040
for me, the show stealer is my you know, iba

145
00:08:04,439 --> 00:08:10,399
Intel's most important product, the eleven O three, the d ram. Okay,

146
00:08:10,519 --> 00:08:14,000
this is what Moore's law actually was about, was making

147
00:08:14,720 --> 00:08:18,120
RAM right based on a bunch of other developments to

148
00:08:18,160 --> 00:08:22,680
make a transistor based memory. They were able to make

149
00:08:22,720 --> 00:08:27,480
a silicon substrate for an eighteen dip pin dip can

150
00:08:27,920 --> 00:08:31,759
with one K of RAM in it for sixty bucks.

151
00:08:31,800 --> 00:08:34,399
Speaker 2: Wow, that seems cheap back then, and.

152
00:08:34,480 --> 00:08:37,480
Speaker 1: One cent per bit, and it was small because then

153
00:08:37,519 --> 00:08:40,759
they were largely using core magnet Ferris cores for memory.

154
00:08:40,840 --> 00:08:45,080
So this was very compact and it was adopted immediately everywhere.

155
00:08:45,120 --> 00:08:48,440
It's it's uptake. That's also the same year that the

156
00:08:48,480 --> 00:08:51,559
first version of the IBM system three seventy comes out

157
00:08:51,559 --> 00:08:54,519
with all semi conductor RAM, but that was not Intel's RAM.

158
00:08:54,559 --> 00:08:58,480
But shortly after that, Intel's RAM just dominates the market

159
00:08:58,519 --> 00:09:03,360
and sends Intel on its trajectory. Although nineteen seventy one

160
00:09:03,360 --> 00:09:06,039
they'll make arguably and even more in product important product.

161
00:09:06,440 --> 00:09:09,960
Tune in next week for nineteen seventy one, nineteen seventy one.

162
00:09:10,000 --> 00:09:13,600
But yes, the eleven O three was there, you know,

163
00:09:13,759 --> 00:09:17,320
definitive product. They were rammed digit you know, semiconductor ramming.

164
00:09:18,039 --> 00:09:18,799
And that's what I got.

165
00:09:18,879 --> 00:09:22,360
Speaker 2: All right, well, I guess we should carry on with

166
00:09:22,480 --> 00:09:23,960
better no framework.

167
00:09:23,600 --> 00:09:24,919
Speaker 1: Roll the crazy music possible.

168
00:09:32,360 --> 00:09:34,399
Speaker 2: All right, man, what do you got again? I looked

169
00:09:34,399 --> 00:09:37,360
for a trending repost on GitHub and I found MCP

170
00:09:37,720 --> 00:09:42,120
for Unity. Oh my, yeah, you know Unity Create create

171
00:09:42,200 --> 00:09:47,559
games with the Unity. It's a graphical tool that uses

172
00:09:47,639 --> 00:09:50,879
c sharp and JavaScript for scripting, but it also does

173
00:09:50,919 --> 00:09:54,039
all of the three D stuff. So here's what it is,

174
00:09:54,559 --> 00:10:00,399
proudly sponsored and maintained by Coplay, the best AI systant

175
00:10:00,440 --> 00:10:03,440
for Unity. There you go create your Unity apps with

176
00:10:03,639 --> 00:10:06,840
l l MS. M CP for Unity acts as a bridge,

177
00:10:06,879 --> 00:10:11,159
allowing a assistance like Claude Cursor to interact directly with

178
00:10:11,240 --> 00:10:16,080
your Unity editor via a local MCP model context protocol.

179
00:10:16,120 --> 00:10:20,320
We've been talking about those. A local MCP client use

180
00:10:20,360 --> 00:10:25,000
your lll M tools to manage assets, control scenes, edit scripts,

181
00:10:25,039 --> 00:10:27,320
and automate tasks within Unity.

182
00:10:27,639 --> 00:10:32,000
Speaker 1: Pretty cool. Interesting, Yeah, a good show to actually walk

183
00:10:32,080 --> 00:10:36,080
through the process of, you know, including making a game

184
00:10:36,080 --> 00:10:39,279
in Unity with with the MCPM, with l MS in

185
00:10:39,279 --> 00:10:40,000
the role. Yep.

186
00:10:40,759 --> 00:10:43,919
Speaker 2: Also code it with the AI. Dot com is up

187
00:10:44,240 --> 00:10:50,480
and the first episode is there and we're basically using

188
00:10:50,600 --> 00:10:56,679
playwright to with the code agent in the visual studio

189
00:10:56,720 --> 00:11:00,679
code nice and using clauds on it. And we basically

190
00:11:01,039 --> 00:11:06,720
one prompt told it to create a user documentation of

191
00:11:08,279 --> 00:11:15,120
Jeff Fritz's copilot do John dot com website, and it

192
00:11:15,159 --> 00:11:18,240
did a pretty good job. What we didn't show was

193
00:11:18,559 --> 00:11:22,879
what's involved in setting up the playwright MCP so that

194
00:11:22,960 --> 00:11:25,679
the agent can use it. Oh yeah, and it turns

195
00:11:25,720 --> 00:11:29,840
out that's pretty complex. You need node JS and NPM

196
00:11:30,120 --> 00:11:34,080
and all that stuff, and we're looking for a video

197
00:11:34,919 --> 00:11:37,159
on how to do that, so look in the show

198
00:11:37,200 --> 00:11:40,080
notes for that. Cool, but that's it for a better

199
00:11:40,120 --> 00:11:42,200
no framework. Who's talking to us? I have a common

200
00:11:42,200 --> 00:11:45,120
of a show nineteen sixty nine. Yes, that's last week's

201
00:11:45,159 --> 00:11:48,879
show with our friend James monte Magno.

202
00:11:48,639 --> 00:11:51,440
Speaker 1: And we talked a little bit about the AI tooling

203
00:11:51,480 --> 00:11:54,960
inside of Visual Studio code and its relationship with Visual

204
00:11:55,000 --> 00:11:57,399
Studio and so on. And our friend Richard Rukima, also

205
00:11:57,480 --> 00:12:00,440
known as Coputer, has this common but he says, I

206
00:12:00,440 --> 00:12:02,879
think Richard nailed it. Do you like the code or

207
00:12:02,919 --> 00:12:06,320
do you like a solution? I consider my expertise working

208
00:12:06,320 --> 00:12:09,120
with AI as a beginner, especially after listening to James,

209
00:12:09,159 --> 00:12:12,039
but I felt that vibe of joy in getting things

210
00:12:12,120 --> 00:12:15,960
done so fast. So do I like the if then else?

211
00:12:16,080 --> 00:12:18,080
Or do I like ask you for a future reviewing

212
00:12:18,120 --> 00:12:21,120
the result? I'm long past the joy of knowing how

213
00:12:21,159 --> 00:12:25,759
to write procedural code. Yeah. An interesting aspect of this is, like,

214
00:12:26,240 --> 00:12:28,159
is it the more experienced folks that are going to

215
00:12:28,159 --> 00:12:31,559
embrace these tools faster? Because it's typically the more junior

216
00:12:31,600 --> 00:12:33,799
people that tend to jump on the bandwagona new things,

217
00:12:33,840 --> 00:12:36,840
but I hear the same tone over and over again. Yep.

218
00:12:37,000 --> 00:12:40,279
Certainly in terms of respectful interaction with AI, I don't

219
00:12:40,360 --> 00:12:44,120
prescribe to the harsh language, as I feel it reveals character.

220
00:12:44,320 --> 00:12:48,039
It's an interesting statement rights in my character not to

221
00:12:48,159 --> 00:12:51,480
be harsh fully or and to focus on being respectful communication.

222
00:12:51,679 --> 00:12:54,000
I don't think AI should be treated any different, not

223
00:12:54,039 --> 00:12:57,120
for the benefit of AI or the benefit of myself.

224
00:12:57,200 --> 00:13:00,480
Speaker 2: Yeah, exactly. You're not going to feel good, you know,

225
00:13:00,759 --> 00:13:01,759
using harsh language.

226
00:13:01,799 --> 00:13:04,320
Speaker 1: Putting those mean words out there is as much impact

227
00:13:04,360 --> 00:13:06,240
on you as it is on anything else. And leave me.

228
00:13:06,320 --> 00:13:10,720
The software is not affected, that's thing, right.

229
00:13:11,000 --> 00:13:13,240
Speaker 2: The only thing left to be affected is you.

230
00:13:13,720 --> 00:13:18,320
Speaker 1: Yeah, so be kind to yourself. It's not necessary, right, Hey, Richard,

231
00:13:18,399 --> 00:13:20,759
I'm pretty sure you've got a copy of Music code

232
00:13:20,759 --> 00:13:23,200
By already, but thanks so much for your comment. But

233
00:13:23,240 --> 00:13:24,759
if you'd like a copy of music, Cobey, I write

234
00:13:24,759 --> 00:13:26,639
a comment on the website at dot net rocks dot

235
00:13:26,679 --> 00:13:28,399
com or on the facebooks to publish every show there

236
00:13:28,399 --> 00:13:30,200
and every comment there, and never reading the show, will

237
00:13:30,240 --> 00:13:31,159
send your copy of music Go.

238
00:13:31,399 --> 00:13:33,519
Speaker 2: Music to code By is still going strong after all

239
00:13:33,559 --> 00:13:36,399
these years twenty two tracks. You can get him in

240
00:13:37,279 --> 00:13:41,440
uh wave, flack or MP three and that's at Music

241
00:13:41,480 --> 00:13:46,080
to Code by dot Net. Okay, let's bring back our

242
00:13:46,120 --> 00:13:50,360
friend Joseph Finney. Joseph is a mobile product owner in

243
00:13:50,559 --> 00:13:54,000
MVP by day and he builds productivity apps for Windows

244
00:13:54,039 --> 00:13:58,399
by night. When he's not programming, he's burning running and

245
00:13:58,480 --> 00:14:00,919
enjoying tasty coffee and beer in Milwaukee.

246
00:14:00,960 --> 00:14:03,279
Speaker 1: Hey Joe, Hello, welcome back to having that.

247
00:14:03,440 --> 00:14:07,320
Speaker 4: Good to be back talking more about the hot topic

248
00:14:07,360 --> 00:14:09,039
of the day, AI.

249
00:14:08,840 --> 00:14:11,320
Speaker 1: With a Century. Yeah, but you've got you've got a

250
00:14:11,360 --> 00:14:13,240
cool angle of this. That's why I asked you that

251
00:14:13,399 --> 00:14:15,679
to come on. So what are you working on?

252
00:14:15,840 --> 00:14:18,679
Speaker 4: Well, one of my most popular apps that I make

253
00:14:18,799 --> 00:14:22,360
is text grab, which is pretty basic. It's also the

254
00:14:22,399 --> 00:14:26,840
basis for the Power Toys Text Extractor, which is basically

255
00:14:26,879 --> 00:14:29,720
select a region on your screen of somebody who sent

256
00:14:29,759 --> 00:14:32,919
you text that you can't actually select and put somewhere

257
00:14:32,960 --> 00:14:35,639
where you want it. And it does some on device

258
00:14:35,840 --> 00:14:41,960
local OCR. Pretty simple, and now with these new models,

259
00:14:42,960 --> 00:14:47,720
the OCR is getting better. But it does change compatibility

260
00:14:47,799 --> 00:14:51,279
and devices, but it's it's pretty interesting what we can

261
00:14:51,320 --> 00:14:54,000
do here now with these local models Microsoft's making it

262
00:14:54,039 --> 00:14:58,399
easier with some of their Windows AI APIs, and then

263
00:14:58,399 --> 00:15:00,919
there's it just gets more and more complicated from there.

264
00:15:01,320 --> 00:15:03,440
Speaker 2: Mm hmm. So I have an app that I'm running

265
00:15:03,519 --> 00:15:08,000
right here that does little OCR and I'm using Tesseract

266
00:15:08,399 --> 00:15:11,120
to read the text in a bitmap at a certain coordinate.

267
00:15:12,799 --> 00:15:16,000
That is that the sort of representing the state of

268
00:15:16,000 --> 00:15:17,759
the art before AI got into the mix.

269
00:15:17,840 --> 00:15:20,000
Speaker 4: Yeah, I would say it's it's similar. Tests React was

270
00:15:20,039 --> 00:15:23,159
the open source project that Google took over I think.

271
00:15:23,240 --> 00:15:26,200
I think actually HP started it way back there and

272
00:15:26,240 --> 00:15:29,440
then kind of Google took it over. Yeah, it's on GitHub.

273
00:15:29,480 --> 00:15:33,200
There's a lot of models. It's very widely used and loved.

274
00:15:35,200 --> 00:15:39,080
Text grab does enable you to download tests earect and

275
00:15:39,120 --> 00:15:41,480
then you can interact with it through the CLI. Well

276
00:15:41,679 --> 00:15:44,720
text grab will just interact with it directly, but there's

277
00:15:44,759 --> 00:15:46,200
a little bit of setting up. You do have to

278
00:15:46,200 --> 00:15:50,000
download it. It's a it's another installer. It's through ub

279
00:15:50,159 --> 00:15:52,639
Mannheim I think who does the installation. So there's definitely

280
00:15:52,720 --> 00:15:54,679
some hoops you have to jump through to get it working.

281
00:15:54,440 --> 00:15:57,200
Speaker 2: And there's a data set that goes along with it, right.

282
00:15:57,159 --> 00:15:59,519
Speaker 4: Yeah, Yeah, so yeah, you have to download the languages.

283
00:15:59,559 --> 00:16:02,200
There's a lot. One of the benefits there of Tessaact

284
00:16:02,320 --> 00:16:04,399
is that there's a lot of languages, and they have

285
00:16:05,480 --> 00:16:10,799
packages for scripts, and they have packages for like handwritten

286
00:16:11,120 --> 00:16:15,480
and so it's really high quality. Originally, Textcrab was built

287
00:16:15,559 --> 00:16:19,799
using the Windows ten ocr APIs, which are definitely older,

288
00:16:19,960 --> 00:16:22,480
not as good, but they're very fast. So that was

289
00:16:22,519 --> 00:16:24,759
kind of the nice thing there. They're built in, they're fast,

290
00:16:24,799 --> 00:16:26,919
they're quick for most stuff. It worked pretty well cool

291
00:16:27,039 --> 00:16:28,960
test erect was a bump up, but again you have

292
00:16:29,000 --> 00:16:31,480
that complexity where you have to download the models locally.

293
00:16:31,639 --> 00:16:35,440
But it's open source, it's available, it's free. And now

294
00:16:35,840 --> 00:16:39,960
there's these Windows AI APIs that Microsoft has released. I

295
00:16:40,000 --> 00:16:42,480
don't think we know exactly what those models are. I

296
00:16:42,480 --> 00:16:46,600
don't think they've shared. I haven't learned what they are exactly.

297
00:16:46,960 --> 00:16:49,600
Speaker 2: But what was the acronym that you used before we

298
00:16:49,679 --> 00:16:52,240
started recording for this new.

299
00:16:52,080 --> 00:16:56,360
Speaker 4: Wind WINML, Windows and machine learning.

300
00:16:56,720 --> 00:16:59,519
Speaker 2: Okay, and this is new, yeah, literally days old than

301
00:16:59,559 --> 00:17:00,799
we don't know anything about it.

302
00:17:00,840 --> 00:17:03,480
Speaker 4: Well, the win mL stuff is kind of a middle

303
00:17:03,559 --> 00:17:05,839
layer here, Okay. So I would say there's like three

304
00:17:06,400 --> 00:17:12,200
general levels of intensity. If you are a local Windows

305
00:17:12,240 --> 00:17:17,920
app developer and you want to get ocr image language

306
00:17:18,400 --> 00:17:20,039
models like all of that stuff. If you want to

307
00:17:20,079 --> 00:17:22,240
do that in your app. I would say there's like

308
00:17:22,240 --> 00:17:25,559
three different tiers of complexity that you can engage in,

309
00:17:26,000 --> 00:17:30,119
and the first one is the new Windows aiapis. And

310
00:17:30,160 --> 00:17:32,480
these were released kind of around the time the Copilot

311
00:17:32,480 --> 00:17:36,839
plus PCs were released, Okayne, and they've been rolling out. Yeah,

312
00:17:36,839 --> 00:17:39,880
they've been rolling out slowly. They were in experimental. You

313
00:17:39,920 --> 00:17:42,400
had to be on the insider preview to build them.

314
00:17:42,400 --> 00:17:44,839
To use them, you have to have a co Pilot

315
00:17:44,839 --> 00:17:48,720
plus PC. But you know, there's a higher bar kind

316
00:17:48,720 --> 00:17:51,839
of on the consumer side, but that means it's easier

317
00:17:51,880 --> 00:17:54,920
on the developer side. So they basically in the code

318
00:17:55,039 --> 00:17:57,680
when you're building, you just have to check does this

319
00:17:57,799 --> 00:18:02,559
device support these APIs? If so, do it very simple

320
00:18:02,640 --> 00:18:04,480
and like that's it. You don't have to manage models,

321
00:18:04,519 --> 00:18:07,319
you don't have to manage memory or downloading, and you

322
00:18:07,319 --> 00:18:10,200
don't have to worry about shipping. You know, a five

323
00:18:10,319 --> 00:18:14,000
gig model with your app. They're already on the device.

324
00:18:14,119 --> 00:18:16,920
If the device supports it, then you can kind of

325
00:18:17,000 --> 00:18:20,640
light up those features, turn on those buttons, show that capability,

326
00:18:20,720 --> 00:18:21,480
and boom, it's there.

327
00:18:21,559 --> 00:18:21,839
Speaker 1: Kelly.

328
00:18:21,920 --> 00:18:25,799
Speaker 2: My wife bought a new Copilot plus PC. She didn't,

329
00:18:25,839 --> 00:18:27,880
of course know it. We went to best Buy together,

330
00:18:27,960 --> 00:18:30,960
you know, and she picked it out. But the first

331
00:18:30,960 --> 00:18:33,839
thing I did is immediately turned off all this stuff.

332
00:18:34,519 --> 00:18:36,920
It's going to get in the way. The thing that

333
00:18:36,960 --> 00:18:41,200
takes screenshots all the time. I can't remember the name

334
00:18:41,240 --> 00:18:44,559
of it now, recall, recall, that's it. It was turned

335
00:18:44,559 --> 00:18:48,079
off by default. So that's good. That's good. I did

336
00:18:48,079 --> 00:18:48,799
not want that on.

337
00:18:49,279 --> 00:18:51,480
Speaker 1: It's a really powerful tool. People love it, you know,

338
00:18:51,720 --> 00:18:53,920
like because the bottom line is you can you can

339
00:18:53,960 --> 00:18:57,119
ask the machine, he where did I see such and such,

340
00:18:57,160 --> 00:18:58,480
and it'll find it for you. Yeah.

341
00:18:58,480 --> 00:19:00,680
Speaker 2: I just don't have that kind of problem, like I

342
00:19:00,720 --> 00:19:02,720
know where I saw stuff, and I keep good notes

343
00:19:02,759 --> 00:19:05,400
and dot your machine. Yeah, she didn't want.

344
00:19:05,279 --> 00:19:09,400
Speaker 4: It, so yeah, I also don't use it like I

345
00:19:09,559 --> 00:19:12,559
have AI features in well, AI, I should say, I

346
00:19:12,599 --> 00:19:14,839
know this show Richard has talked a lot about how

347
00:19:15,119 --> 00:19:18,000
you have these big amorphous buckets of AI, and then

348
00:19:18,039 --> 00:19:20,799
as soon as you start explaining it and giving a

349
00:19:20,839 --> 00:19:24,039
more clear, straightforward name to it, it stops really being AI.

350
00:19:24,640 --> 00:19:28,400
And that's kind of where the OCR and LLM and

351
00:19:28,799 --> 00:19:32,920
image segmentation and image detection. So those are all under

352
00:19:32,960 --> 00:19:36,960
this umbrella of AI, and it can be a little

353
00:19:37,519 --> 00:19:38,000
I don't know.

354
00:19:38,079 --> 00:19:40,680
Speaker 1: You left the impolite part, Joe, which is like, so

355
00:19:40,759 --> 00:19:44,799
for me, the term artificial intelligence means something that doesn't work. Yeah,

356
00:19:44,839 --> 00:19:47,599
there you go, because as soon as it does work,

357
00:19:47,839 --> 00:19:49,160
it gets a new name.

358
00:19:49,200 --> 00:19:52,519
Speaker 4: Software, right right, that's it's a module. Yeah, well, I

359
00:19:52,519 --> 00:19:56,160
should say, then the using name space in dot net

360
00:19:56,359 --> 00:19:59,920
is AI. But then after that there's always dot tech

361
00:20:00,279 --> 00:20:04,200
that imaging that image recognition. So there's a bunch of

362
00:20:04,920 --> 00:20:07,880
there's a bunch of APIs after the namespace that actually

363
00:20:08,039 --> 00:20:10,279
point to the real APIs, the real functionality of what

364
00:20:10,319 --> 00:20:14,799
you're actually trying to do. And I don't think you

365
00:20:14,799 --> 00:20:18,079
can easily turn all of that off. I would say,

366
00:20:18,200 --> 00:20:20,720
so there's a lot of experiences that are built on

367
00:20:20,799 --> 00:20:23,880
top of this technology that's already in these Copilot plus PCs,

368
00:20:24,440 --> 00:20:26,759
and you could turn those experiences off. You know, they're

369
00:20:26,799 --> 00:20:29,559
not going to run by default. But Microsoft does a

370
00:20:29,599 --> 00:20:33,319
pretty good job of managing bringing down the model, keeping

371
00:20:33,319 --> 00:20:35,440
it up to date, and making it really easy for

372
00:20:35,599 --> 00:20:38,480
developers to interact with, which is kind of what you want, right,

373
00:20:38,559 --> 00:20:42,000
You want something really simple easy. It's a super complex problem,

374
00:20:42,200 --> 00:20:44,440
but you could just say, you know, send this block

375
00:20:44,480 --> 00:20:46,400
of text, summarize it, and then get it back.

376
00:20:46,759 --> 00:20:48,960
Speaker 2: So in case anyone hasn't figured it out by now,

377
00:20:49,000 --> 00:20:53,200
the Copilot plus PC has a local LLM built into it.

378
00:20:53,319 --> 00:20:53,559
Speaker 1: Yep.

379
00:20:53,640 --> 00:20:55,759
Speaker 2: And you know, this is the kind of thing that

380
00:20:55,839 --> 00:20:57,720
you might think of if you were going to use

381
00:20:57,759 --> 00:21:04,319
OLAMA right and download models and you know, train it,

382
00:21:04,680 --> 00:21:07,519
run it on a laptop or something like that, the

383
00:21:07,559 --> 00:21:08,880
gaming PC or something.

384
00:21:09,079 --> 00:21:10,839
Speaker 4: Yeah, there's that's just kind of where I said, there's

385
00:21:10,839 --> 00:21:15,680
like these different layers of the complexity and the easiest, simplest,

386
00:21:15,720 --> 00:21:18,480
like lowest level, easiest for any developer out there to

387
00:21:18,519 --> 00:21:21,200
integrate into their Windows app. Any Windows app by the way,

388
00:21:21,519 --> 00:21:26,559
so WPF or when UI or wind forms you can

389
00:21:26,680 --> 00:21:29,160
or MAUI, you can do them all. It does have

390
00:21:29,200 --> 00:21:33,200
to have identity, some sort of identity because SB there's

391
00:21:33,400 --> 00:21:35,839
Microsoft doesn't want to just open up these APIs to

392
00:21:36,519 --> 00:21:39,960
any random raw ex But if you want to do

393
00:21:40,160 --> 00:21:44,279
some more maybe more niche stuff, maybe a little bit

394
00:21:44,279 --> 00:21:47,559
more complicated stuff, or you want to use this specific model,

395
00:21:47,720 --> 00:21:49,039
you can kind of use what I would call like

396
00:21:49,079 --> 00:21:52,960
the next step of complexity here, and that's win mL

397
00:21:53,000 --> 00:21:54,799
and that's there's a little bit of a middle layer

398
00:21:54,839 --> 00:21:56,799
there where you can go download your own on X

399
00:21:56,880 --> 00:22:00,400
models and run those and it makes it easy. There's

400
00:22:00,440 --> 00:22:04,240
like a basically a standardized interface and you say, run

401
00:22:04,279 --> 00:22:06,920
this model. You don't have to necessarily optimize it for

402
00:22:06,960 --> 00:22:11,279
the specific hardware and it can run CPU, GPU and

403
00:22:11,359 --> 00:22:15,079
PU and it's an easy way. But again, there you

404
00:22:15,200 --> 00:22:18,799
have to manage the model. So if you want that,

405
00:22:18,880 --> 00:22:20,559
if you need that in your application, maybe you have

406
00:22:20,599 --> 00:22:24,759
it specifically fine tuned for your application, or you have

407
00:22:24,799 --> 00:22:29,359
a model that isn't in the box, or I don't

408
00:22:29,359 --> 00:22:31,359
know if there are other legal or.

409
00:22:31,599 --> 00:22:33,759
Speaker 1: Hey, I'm just appreciating you're talking about something other than

410
00:22:33,759 --> 00:22:37,960
in the LLM, because it's just it's just overwhelming right now.

411
00:22:38,039 --> 00:22:40,480
So you know, clearly there's a bunch of other models

412
00:22:40,480 --> 00:22:42,519
out there and all of those infrastructure, and I'm including

413
00:22:42,559 --> 00:22:44,759
links to onyx and things like if you haven't looked here,

414
00:22:45,039 --> 00:22:47,920
there's lots of good work being done for specific tasks.

415
00:22:48,319 --> 00:22:51,960
Speaker 4: Yeah, and I think immediately people can kind of get

416
00:22:51,960 --> 00:22:55,240
annoyed by, oh, LM, why do I need an LLM

417
00:22:55,240 --> 00:22:58,160
in my model? I'll need AI and it definitely has

418
00:22:58,200 --> 00:23:01,720
become synonymous like AI and LLM. Yeah, but there are

419
00:23:01,799 --> 00:23:03,559
so many If you go to hugging face and you

420
00:23:03,599 --> 00:23:06,799
look at all the different categories, I mean OCR, image segmentation,

421
00:23:06,920 --> 00:23:11,920
image detection, object detection, huggy face, oh yeah, hugging face,

422
00:23:12,200 --> 00:23:15,640
hugging face, hugging face. Yeah, this is a I think

423
00:23:15,759 --> 00:23:19,920
Facebook is kind of backing it. And it's a big

424
00:23:20,440 --> 00:23:25,119
repository for models, so you can access models, you get

425
00:23:25,119 --> 00:23:27,319
download models, and if you're thinking.

426
00:23:27,839 --> 00:23:32,000
Speaker 1: Before the insanity of lllms, we had we had good

427
00:23:32,039 --> 00:23:36,279
tooling around just building machine models for object detection and

428
00:23:36,880 --> 00:23:39,960
recognizers and OCR all these good things, right, Like, it's

429
00:23:40,000 --> 00:23:43,160
just there was so much going on before chat GPT

430
00:23:43,279 --> 00:23:45,440
showed up and just overwhelm the message.

431
00:23:45,519 --> 00:23:47,599
Speaker 2: Wow, hugging face looks awesome.

432
00:23:47,720 --> 00:23:52,920
Speaker 4: Yeah, it's it is a huge, kind of big repository

433
00:23:53,000 --> 00:23:56,440
of models online where you can go download them. But

434
00:23:56,640 --> 00:23:59,319
if you're a normal person who's just curious and says

435
00:23:59,480 --> 00:24:01,319
I want to kind of to try some of these out,

436
00:24:01,680 --> 00:24:03,960
it's not as easy. You can't just download them and

437
00:24:04,000 --> 00:24:07,640
then run them. They are not programs, their models, so

438
00:24:08,000 --> 00:24:11,440
you need to interface with them somehow, and there is

439
00:24:11,680 --> 00:24:15,400
actually a way if you are inclined. You can download

440
00:24:15,440 --> 00:24:19,519
an app from Microsoft called the AI Dev Gallery app.

441
00:24:20,119 --> 00:24:22,880
And what this is it's kind of a playground for

442
00:24:23,039 --> 00:24:25,680
people who are curious about models and different models and

443
00:24:25,720 --> 00:24:28,480
how this all works. It's open source on GitHub, it's

444
00:24:28,519 --> 00:24:31,559
in the Microsoft Store and it is a really low

445
00:24:31,640 --> 00:24:34,400
barrier to entry if you are interested in trying some

446
00:24:34,440 --> 00:24:36,119
of these models out on your own device.

447
00:24:36,240 --> 00:24:36,519
Speaker 2: Wow.

448
00:24:36,599 --> 00:24:39,319
Speaker 4: So you can download models from hugging Face. You can

449
00:24:39,440 --> 00:24:42,400
run them. They're very limited, basic samples, so don't expect

450
00:24:42,400 --> 00:24:46,279
anything grandiose or chaining them together. But it's a great

451
00:24:46,279 --> 00:24:48,119
way to play with those Hugging Face models.

452
00:24:48,279 --> 00:24:48,599
Speaker 2: Very cool.

453
00:24:48,640 --> 00:24:51,920
Speaker 1: Did you ever play with Cagle, because we've talked about

454
00:24:51,920 --> 00:24:54,920
this on the show Ages Ago. Just like there is

455
00:24:54,960 --> 00:24:58,759
another playground for practicing your mL skills.

456
00:24:58,680 --> 00:25:00,680
Speaker 4: I've never tried. It is in a a website or

457
00:25:00,720 --> 00:25:02,400
a technology.

458
00:25:01,920 --> 00:25:06,599
Speaker 1: They actually run competitions for you know. The sort of

459
00:25:06,599 --> 00:25:09,640
famous one for them was the predict how many people

460
00:25:09,680 --> 00:25:13,240
survive the Titanic sinking. There was a bunch of different

461
00:25:14,920 --> 00:25:17,839
models or different competitions, and some of them have a

462
00:25:17,839 --> 00:25:20,440
lot of money in them because they're actually you know,

463
00:25:20,720 --> 00:25:25,119
organizations encourage folks to mature a model particular problem space

464
00:25:25,160 --> 00:25:27,640
that they can then use elsewhere. There was things like

465
00:25:27,759 --> 00:25:35,000
aneurysm detection and even sports predicting. So just again a

466
00:25:35,039 --> 00:25:39,720
reminder that there's things other than llms.

467
00:25:39,839 --> 00:25:43,279
Speaker 4: Right, And I would say that is the like the farthest,

468
00:25:43,319 --> 00:25:46,759
the highest tier of integrating AI models into your app,

469
00:25:46,960 --> 00:25:50,799
your local Windows app is making your own models, training

470
00:25:50,880 --> 00:25:53,960
your own models from scratch, So you can do that.

471
00:25:54,039 --> 00:25:57,799
I mean, you can ship models and integrate them directly in.

472
00:25:58,079 --> 00:26:01,920
It's again way more integration work, but it's way more

473
00:26:01,960 --> 00:26:05,000
fine tuned. So if you have a specific application where

474
00:26:05,039 --> 00:26:07,480
you need a model that can do very niche things

475
00:26:07,640 --> 00:26:11,839
or very specific data sets, it's possible. It's doable, and

476
00:26:12,319 --> 00:26:14,000
there's ways to do it. You should check it out.

477
00:26:14,519 --> 00:26:17,759
One of the nice things about this current age of

478
00:26:17,920 --> 00:26:20,480
programming is a lot of these big popular apps are

479
00:26:20,519 --> 00:26:23,160
open source, so you can just see how it's done,

480
00:26:23,440 --> 00:26:25,920
and you obviously read the license, but a lot of

481
00:26:25,920 --> 00:26:29,039
this stuff is available to see how other people are

482
00:26:29,039 --> 00:26:30,880
integrating these AI models.

483
00:26:31,039 --> 00:26:31,319
Speaker 1: Guys.

484
00:26:31,559 --> 00:26:33,799
Speaker 2: I know we've talked about deep seek a bit on

485
00:26:33,880 --> 00:26:37,759
this show, and Joe's nodding his head, so he knows

486
00:26:37,799 --> 00:26:40,759
about it, and this was the model that came out

487
00:26:40,759 --> 00:26:46,039
of China that uses a lot less resources and is

488
00:26:46,079 --> 00:26:50,319
therefore cheaper to run than you know, chat GPT was,

489
00:26:50,400 --> 00:26:52,839
and everybody was like, oh my god, open ai is

490
00:26:52,880 --> 00:26:56,359
going down, and it didn't. And then there were concerns

491
00:26:56,359 --> 00:27:02,279
about you know, if I use deep Seek, am I

492
00:27:02,319 --> 00:27:07,160
sharing data with you know, the country of China and

493
00:27:07,960 --> 00:27:10,039
is it safe in all of these things. But you

494
00:27:10,119 --> 00:27:13,160
can also I think, correct me if I'm wrong, but

495
00:27:13,799 --> 00:27:17,799
download it the app and run it locally like olama.

496
00:27:18,240 --> 00:27:18,759
Is that true?

497
00:27:18,920 --> 00:27:19,240
Speaker 1: Yeah?

498
00:27:19,400 --> 00:27:23,279
Speaker 4: That So one of the nice things about deepseek is

499
00:27:23,279 --> 00:27:27,359
how small it is. But they also have NPU optimized

500
00:27:27,599 --> 00:27:31,640
models which you can go download and there's also an

501
00:27:31,640 --> 00:27:33,279
extension for vs code.

502
00:27:33,440 --> 00:27:36,519
Speaker 2: Wait wait, go back to the is M or NPU

503
00:27:36,880 --> 00:27:38,160
and what is that?

504
00:27:38,160 --> 00:27:41,200
Speaker 4: That's the neural processing unit. So you kind of have

505
00:27:41,279 --> 00:27:44,400
your CPU, your GPU, and your NPU.

506
00:27:44,839 --> 00:27:45,519
Speaker 1: And this was.

507
00:27:45,480 --> 00:27:50,559
Speaker 4: The core the chip, the part of the CPU in

508
00:27:50,599 --> 00:27:54,000
these ARM devices that really made it easy to run

509
00:27:54,039 --> 00:27:56,200
these models locally and efficiently.

510
00:27:56,200 --> 00:27:59,000
Speaker 1: Okay, part of the requirement for a copilot plus PCs

511
00:27:59,000 --> 00:28:01,160
that it has an MPU of at least what is it,

512
00:28:01,240 --> 00:28:04,839
forty tops or trillion operations per second.

513
00:28:04,920 --> 00:28:07,440
Speaker 2: So if you have a copile plus PC, you can

514
00:28:07,720 --> 00:28:10,920
download deep Seek and use it even if you don't,

515
00:28:10,960 --> 00:28:13,039
and you're probably going to get good results.

516
00:28:13,519 --> 00:28:16,839
Speaker 4: Yeah, you don't have to have a NPU, but a

517
00:28:16,920 --> 00:28:20,759
lot of these models. So Microsoft makes a LM called

518
00:28:20,920 --> 00:28:26,480
five Silica, and this model they have they've been releasing three,

519
00:28:26,680 --> 00:28:29,920
three point five, they just released four. It's optimized for

520
00:28:29,960 --> 00:28:33,720
the CPU and the GPU and not the NPU right now,

521
00:28:34,000 --> 00:28:36,519
at least the models that they've released, and there are

522
00:28:36,599 --> 00:28:39,119
models out there that you can get that are optimized

523
00:28:39,119 --> 00:28:41,000
for the NPU. So if you do have a device

524
00:28:41,400 --> 00:28:44,400
that is OM device or low power device and you

525
00:28:44,440 --> 00:28:46,720
want more of an optimized model, you can find them

526
00:28:47,319 --> 00:28:50,039
and run them. And you can also do that in

527
00:28:50,160 --> 00:28:54,079
VS code. There's an extension called AI Toolkit for Visual

528
00:28:54,079 --> 00:28:58,920
Studio Code, and that's another kind of playground esque place,

529
00:28:59,039 --> 00:29:02,480
but you can also do the model refinement and fine

530
00:29:02,519 --> 00:29:06,119
tuning in there. So there's a lot of ways that

531
00:29:06,160 --> 00:29:09,440
you can experiment with these models without really being a pro.

532
00:29:09,920 --> 00:29:12,680
So if you're just curious and you have a lot

533
00:29:12,720 --> 00:29:14,359
of hard drive space, that is the one thing that

534
00:29:14,400 --> 00:29:18,119
I'll say, I recently upgraded my surface hard drive from

535
00:29:18,119 --> 00:29:21,160
a five to twelve to a two terabyte because these

536
00:29:21,240 --> 00:29:25,400
models are big and if you want accurate ones, they're

537
00:29:25,759 --> 00:29:26,359
very large.

538
00:29:26,440 --> 00:29:29,240
Speaker 2: I just saw Richard probably knows about this, but there

539
00:29:29,240 --> 00:29:34,319
are now twenty two terabyte SSD drives. Yeah, for like

540
00:29:34,359 --> 00:29:38,039
around five hundred bucks. Can you wrap your mind around that.

541
00:29:38,119 --> 00:29:39,279
Speaker 1: It's a lot of storage.

542
00:29:39,359 --> 00:29:43,240
Speaker 2: Oh my goodness, Like me know, Joe's like is shaking

543
00:29:43,279 --> 00:29:44,359
his head, like what.

544
00:29:44,759 --> 00:29:47,000
Speaker 4: One drive twenty two terramytes.

545
00:29:46,559 --> 00:29:49,279
Speaker 2: Twenty two terabyte SSD five hundred bucks?

546
00:29:49,359 --> 00:29:50,599
Speaker 4: You should that's not a typeout.

547
00:29:51,119 --> 00:29:53,359
Speaker 2: No, there's a couple of different brands.

548
00:29:53,440 --> 00:29:54,039
Speaker 4: That's amazing.

549
00:29:54,119 --> 00:29:56,880
Speaker 1: Yeah, ridiculous, Yeah, that really is. I think I should.

550
00:29:56,920 --> 00:29:59,200
I don't think they are sists. I think they're spinning drives.

551
00:29:59,279 --> 00:30:04,680
Oh really two terabytes? Yeah SSDs the solid state ones

552
00:30:04,720 --> 00:30:05,680
and aren't that big yet?

553
00:30:05,759 --> 00:30:06,119
Speaker 2: Okay?

554
00:30:06,240 --> 00:30:10,359
Speaker 1: The still twenty two terabytes is madness? Like that's just

555
00:30:10,400 --> 00:30:11,240
a lot of storage.

556
00:30:11,400 --> 00:30:15,960
Speaker 4: Yeah, it really is. And the AI Toolkit and vs

557
00:30:15,960 --> 00:30:20,559
code does allow you to interact with these llms through

558
00:30:20,559 --> 00:30:23,240
the web, and so GitHub will host some of these models,

559
00:30:23,279 --> 00:30:25,839
other providers will host them, and so you can kind

560
00:30:25,839 --> 00:30:31,480
of do comparisons. So there's the local foundry, and that's

561
00:30:31,759 --> 00:30:34,759
what Microsoft has branded there. You know, I've called it,

562
00:30:34,759 --> 00:30:36,880
I think the second tier kind of where you have

563
00:30:37,880 --> 00:30:40,240
win mL and you have your local models and you're

564
00:30:40,400 --> 00:30:43,799
doing that work. So you have your local models and

565
00:30:43,880 --> 00:30:46,839
you can compare those two cloud hosted models and test

566
00:30:46,880 --> 00:30:48,960
them because again, you know software, you have to be

567
00:30:49,000 --> 00:30:52,000
able to test it. So it is hard too with

568
00:30:52,119 --> 00:30:54,519
these how do you compare them? Like, which one's good,

569
00:30:54,519 --> 00:30:56,519
which one's bad? Is it good enough? Is it good

570
00:30:56,599 --> 00:30:58,480
enough in our use cases? And it can be tedious

571
00:30:58,519 --> 00:31:01,480
to test manually. But there are a lot of tools

572
00:31:01,480 --> 00:31:06,079
out there to experiment, get started, and if anybody's curious,

573
00:31:06,079 --> 00:31:09,799
I definitely you should check out the aidev gallery for sure.

574
00:31:10,200 --> 00:31:11,759
That is a lot of fun to play around with

575
00:31:11,799 --> 00:31:16,200
those different models and for a little bit more advanced scenarios,

576
00:31:16,480 --> 00:31:21,279
what more language focused. The AI toolkit in vs code

577
00:31:21,319 --> 00:31:24,160
is another really fun I'm looking at deep seak here

578
00:31:24,279 --> 00:31:27,279
right now. You can download it on your device and

579
00:31:27,359 --> 00:31:27,720
run it.

580
00:31:27,839 --> 00:31:30,039
Speaker 2: Wow, it seems like a pretty good place to take

581
00:31:30,079 --> 00:31:32,480
a break. So we'll be right back after these very

582
00:31:32,839 --> 00:31:33,839
important messages.

583
00:31:34,000 --> 00:31:34,559
Speaker 1: Stay tuned.

584
00:31:36,839 --> 00:31:39,559
Speaker 2: You know, dot net six has officially reached the end

585
00:31:39,599 --> 00:31:42,839
of support and now is the time to upgrade. Dot

586
00:31:42,880 --> 00:31:46,519
Net eight is well supported on AWS. Learn more at

587
00:31:46,559 --> 00:31:50,480
aws dot Amazon dot com, slash dot net.

588
00:31:53,440 --> 00:31:55,559
Speaker 1: And we're back. It's don that Rocks emergor Campbell. Let's

589
00:31:55,559 --> 00:31:58,359
call Franklin. You talking a bit to our friend Joe

590
00:31:58,599 --> 00:32:03,720
about work with local models and also and the non

591
00:32:04,039 --> 00:32:06,119
LLM stuff just sort of a good reminder there's been

592
00:32:06,160 --> 00:32:08,519
all kinds of cool stuff going on in the mL

593
00:32:08,599 --> 00:32:11,720
space that didn't necessarily have to do with language per se.

594
00:32:12,279 --> 00:32:15,160
But you know, you've you've hinted this a couple of

595
00:32:15,160 --> 00:32:17,640
times in the first half. It's like, if you want

596
00:32:17,680 --> 00:32:21,119
to own the model, you know, there's a lot of

597
00:32:21,200 --> 00:32:24,440
models available to download from hugey face and all these

598
00:32:24,480 --> 00:32:27,519
other places. Why would you want to own a model

599
00:32:27,559 --> 00:32:30,000
because it sounds like a lot of work. It's like

600
00:32:30,119 --> 00:32:31,039
owning a framework.

601
00:32:31,119 --> 00:32:34,119
Speaker 4: Yeah, yeah, it is like, don't trust somebody who says

602
00:32:34,160 --> 00:32:36,400
they can write their own language and write their own

603
00:32:36,440 --> 00:32:38,119
ide You're like, oh.

604
00:32:38,240 --> 00:32:41,880
Speaker 1: Their own garbage collector, you know, their own crypto library.

605
00:32:42,359 --> 00:32:44,480
Like these are all scary things to me. So when

606
00:32:44,480 --> 00:32:46,359
someone says I'll just make our own model, I'm like,

607
00:32:46,640 --> 00:32:47,799
why do we need to do that?

608
00:32:48,400 --> 00:32:52,160
Speaker 4: Well, if you're in the industry. If you have insane

609
00:32:52,200 --> 00:32:56,920
amounts of data and a niche in a specific industry,

610
00:32:57,960 --> 00:33:00,000
it might be worth it for you to look into

611
00:33:00,160 --> 00:33:03,240
doing this. And if you have a hard time processing

612
00:33:03,319 --> 00:33:05,960
large amounts of data to get insights and actions out

613
00:33:06,000 --> 00:33:09,039
of it, which is kind of the idea here, right,

614
00:33:09,039 --> 00:33:11,960
what you have an entire language that you have to

615
00:33:11,960 --> 00:33:14,319
train these models on, or you have an entire data

616
00:33:14,319 --> 00:33:18,079
set of images with boxes drawn around the dogs or

617
00:33:18,200 --> 00:33:22,599
dog breeds or very specific things like that. If that's

618
00:33:22,680 --> 00:33:24,799
what you need to do, is something where it's not

619
00:33:24,839 --> 00:33:29,519
available or it's not good enough, there's really no other

620
00:33:29,519 --> 00:33:31,680
way around it than to build your own model today.

621
00:33:32,319 --> 00:33:33,519
But it really is that data.

622
00:33:33,559 --> 00:33:37,119
Speaker 1: It's I mean that being said, this is all sort

623
00:33:37,160 --> 00:33:39,440
of non terministic thing, like you're never going to get

624
00:33:39,440 --> 00:33:41,799
one hundred percent out of a machine learning model.

625
00:33:41,839 --> 00:33:45,839
Speaker 4: It's probabilistic, right, absolutely, even maybe especially so some of

626
00:33:45,880 --> 00:33:48,519
the image detection ones, and a lot of times they'll

627
00:33:48,519 --> 00:33:53,000
give you back a number a fraction of confidence, and

628
00:33:53,039 --> 00:33:54,839
I think maybe this is why they don't get as

629
00:33:54,960 --> 00:33:58,000
much play as they're not as exciting for individuals to use.

630
00:33:58,519 --> 00:34:01,200
It's like the could take a picture of your cat

631
00:34:01,359 --> 00:34:03,759
and then your phone will draw a box around it

632
00:34:03,759 --> 00:34:06,279
and say that's a cat. Yep, that's a cat. So

633
00:34:06,640 --> 00:34:09,119
I think it's a lot less interesting. The language ones

634
00:34:09,199 --> 00:34:11,639
just kind of capture people's imagination and there's a lot

635
00:34:11,679 --> 00:34:14,079
more back and forth. But when you really think about

636
00:34:14,079 --> 00:34:16,920
building an application, like what are you doing? Maybe you

637
00:34:17,000 --> 00:34:20,760
have a you're playing around with your Raspberry Pie as

638
00:34:20,760 --> 00:34:22,519
a security system for your house, and you want to

639
00:34:22,519 --> 00:34:25,159
add a vision system and you want to do box

640
00:34:25,199 --> 00:34:28,239
detection and you have hours and hours and hours and

641
00:34:28,280 --> 00:34:31,320
hours of security footage. Or maybe you have a specific

642
00:34:31,440 --> 00:34:34,199
niche application where you're trying to, you know, detect a

643
00:34:34,239 --> 00:34:37,519
particular squirrel who's given you trouble. It's a fun you know,

644
00:34:37,559 --> 00:34:38,760
it's a fun experiment and you.

645
00:34:38,719 --> 00:34:40,599
Speaker 1: Can do a bear or a bear.

646
00:34:40,880 --> 00:34:43,480
Speaker 2: Joe, do you have a toi less squirrel bird feeder?

647
00:34:44,000 --> 00:34:44,239
Speaker 1: No?

648
00:34:44,320 --> 00:34:47,800
Speaker 2: I do not seeing this YouTube? Check YouTube for toil

649
00:34:47,840 --> 00:34:51,880
less squirrel terrible right. It's basically it goes between you

650
00:34:51,960 --> 00:34:54,119
know what you hang the bird feeder on and the

651
00:34:54,119 --> 00:34:56,320
bird feeder, so it's got a hook on either side.

652
00:34:56,840 --> 00:35:01,000
It detects weight and so when there's a squirrel on it,

653
00:35:01,000 --> 00:35:04,679
it just starts spinning and the squirrels go flying. It's

654
00:35:04,760 --> 00:35:06,280
hilarious to whirl the squirrel.

655
00:35:06,400 --> 00:35:10,000
Speaker 4: Yeah, that you could build an AI powered twirl a squirrel.

656
00:35:10,159 --> 00:35:12,199
Speaker 1: There you go, There you go. I don't think that's necessary.

657
00:35:12,239 --> 00:35:15,960
I am thinking about animal recognition this particular part of

658
00:35:15,960 --> 00:35:18,079
the world where you know. The one that would be

659
00:35:18,119 --> 00:35:20,960
tricky that I would really challenge myself would be whale

660
00:35:20,960 --> 00:35:23,440
detection because we've had you know, you don't have a

661
00:35:23,440 --> 00:35:25,679
lot of time to pick up on the fact that

662
00:35:25,760 --> 00:35:28,599
there's whale blow, like they're going by, and it could

663
00:35:28,599 --> 00:35:30,440
be orcers and it could be humpbacks, and it could

664
00:35:30,480 --> 00:35:32,559
be grays, and it could be porpoises, and it could

665
00:35:32,559 --> 00:35:34,960
be dolphins. Like you have to be a lot of

666
00:35:34,960 --> 00:35:37,599
stuff going on. You have to be on the surface.

667
00:35:38,159 --> 00:35:41,519
We hear no, no, we hear them like we hear

668
00:35:41,599 --> 00:35:44,519
whale blow before we see the whale because it travels

669
00:35:44,679 --> 00:35:46,599
like when they when they exhale its loud.

670
00:35:46,599 --> 00:35:48,960
Speaker 2: Well, you could identify a whale by the sounds it's

671
00:35:49,000 --> 00:35:49,480
making too.

672
00:35:49,559 --> 00:35:53,280
Speaker 1: Yeah, I wonder. Yeah, speaking of it still seems nuts

673
00:35:53,320 --> 00:35:55,199
to build your own model like that just seems like

674
00:35:55,239 --> 00:35:56,159
a thing I don't want to own.

675
00:35:56,239 --> 00:35:59,280
Speaker 4: Yeah, it's it's definitely the research side of things. And

676
00:35:59,480 --> 00:36:01,719
I know people have been saying for a long time

677
00:36:01,960 --> 00:36:05,199
that data is the new oil, right, this is the

678
00:36:05,199 --> 00:36:08,199
new black gold of do you have the data? Do

679
00:36:08,280 --> 00:36:12,920
you have the databases? Is it structured, is it consistent,

680
00:36:13,079 --> 00:36:16,000
is it clean? Is it real? Is it good? And

681
00:36:16,039 --> 00:36:19,119
if you have all that, I think we have a very

682
00:36:19,159 --> 00:36:21,119
small number of people who can say yes, we have

683
00:36:21,199 --> 00:36:22,840
that right and you don't have to spend all that

684
00:36:22,880 --> 00:36:25,760
time cleaning the data, which is such a challenge where

685
00:36:25,880 --> 00:36:29,280
you have so much noise in the data today that

686
00:36:29,320 --> 00:36:30,920
if you're trying to train a model, Yeah.

687
00:36:31,000 --> 00:36:32,760
Speaker 2: If how I was going to use a local LM,

688
00:36:33,039 --> 00:36:37,320
I would want it to understand C sharp, JavaScript, Blazer,

689
00:36:37,880 --> 00:36:42,320
you know, and CSS. That's and I don't know how

690
00:36:42,679 --> 00:36:46,599
realistic that is. Like I know that the current models

691
00:36:46,639 --> 00:36:49,519
like Claude's on it, and you know even chat GPT

692
00:36:49,800 --> 00:36:53,559
understand it. But for lack of a better word, sorry, Richard,

693
00:36:53,559 --> 00:36:57,719
didn't mean to offend you. There. They're programmed, you know,

694
00:36:58,000 --> 00:37:01,239
they're they're trained against it. But what does it take

695
00:37:01,679 --> 00:37:04,719
to do that locally, to train the models to train well,

696
00:37:04,880 --> 00:37:08,079
or to get a model that understands you know, programmers

697
00:37:08,079 --> 00:37:10,119
speak languages and stuff they do.

698
00:37:10,239 --> 00:37:13,360
Speaker 4: Yeah, local models will and they can write code. I

699
00:37:13,360 --> 00:37:17,079
think part of the challenge that you'll see if you

700
00:37:17,079 --> 00:37:20,000
start using them is speed. So the response speed of

701
00:37:20,039 --> 00:37:22,639
a local model is going to be much slower actually

702
00:37:22,679 --> 00:37:26,519
than a cloud hosted one because your computer cannot compete

703
00:37:26,519 --> 00:37:29,800
with a server with a rack of GPUs. Yeah, well

704
00:37:29,960 --> 00:37:31,559
maybe yours, Carl, not mine.

705
00:37:31,599 --> 00:37:34,039
Speaker 2: Oh, I don't know. I don't think so. But you know,

706
00:37:34,239 --> 00:37:37,239
I think if I had a great Copilot plus PC,

707
00:37:38,280 --> 00:37:41,679
you know, with a lot of RAM and a lot

708
00:37:41,679 --> 00:37:45,280
of storage, and I just set it over in a

709
00:37:45,280 --> 00:37:47,039
closet somewhere, I could probably use that.

710
00:37:47,199 --> 00:37:48,079
Speaker 4: Yeah, you should try it.

711
00:37:48,280 --> 00:37:48,639
Speaker 1: Yeah.

712
00:37:48,920 --> 00:37:51,800
Speaker 4: Another challenge is going to be context, which is how

713
00:37:51,840 --> 00:37:54,920
big of a context window can the model actually hold

714
00:37:54,960 --> 00:37:57,119
in the provider there's all of that, there's a lot

715
00:37:57,159 --> 00:38:01,280
of infrastructure in between the model and actually getting stuff out.

716
00:38:01,320 --> 00:38:03,239
So speeding context, I would say, are going to be

717
00:38:03,280 --> 00:38:06,039
your biggest risks where you don't necessarily just want it

718
00:38:06,039 --> 00:38:09,800
to give you new greenfield CSS. You want it to

719
00:38:09,800 --> 00:38:13,760
give you new CSS in the right spot for your codings.

720
00:38:14,000 --> 00:38:14,480
Which is that?

721
00:38:14,519 --> 00:38:15,840
Speaker 1: And I want a much harder question.

722
00:38:15,920 --> 00:38:18,239
Speaker 2: I wanted to remember everything we've said, Like I want

723
00:38:18,280 --> 00:38:21,079
as big a context as I can possibly get. So

724
00:38:21,320 --> 00:38:24,599
is that just a measure of more RAM or is

725
00:38:24,599 --> 00:38:27,880
it the more that context you have, the slower it's

726
00:38:27,920 --> 00:38:31,239
going to be to come up with a new answer.

727
00:38:31,320 --> 00:38:33,519
Speaker 4: Yeah, that's a good question. I would love to hear

728
00:38:33,679 --> 00:38:37,599
an expert who actually knows more about context and how

729
00:38:37,679 --> 00:38:39,840
that differs from the training data and how it differs

730
00:38:39,840 --> 00:38:45,159
from fine tuning, because in my experiences with local AI,

731
00:38:45,280 --> 00:38:47,800
I have a pretty narrow context window that you could

732
00:38:47,840 --> 00:38:50,480
basically feed it, Hey, here's everything I know, and you

733
00:38:50,519 --> 00:38:53,599
feed it with the prompt yeah, and you say okay,

734
00:38:53,719 --> 00:38:55,440
now do this and then give it back to me.

735
00:38:55,760 --> 00:38:57,880
But you're not feeding it documents.

736
00:38:57,960 --> 00:38:59,760
Speaker 1: The thing that's made a difference for me has been

737
00:38:59,800 --> 00:39:02,119
the video card and the amount of memory in the

738
00:39:02,159 --> 00:39:04,280
video card, Like playing with frame Pack and a couple

739
00:39:04,320 --> 00:39:07,559
of other models, and so I'm running a fifty eighty

740
00:39:07,559 --> 00:39:11,239
with sixteen gigs of v RAM, and that has made

741
00:39:11,239 --> 00:39:14,239
a huge difference for running bigger models. No, I'm not

742
00:39:14,239 --> 00:39:17,480
talking about building models, but actually executing a more complex workload.

743
00:39:18,239 --> 00:39:20,440
And if you have got the money to spend, because

744
00:39:20,480 --> 00:39:24,239
they're thousands of dollars like those top in RTX cards.

745
00:39:24,239 --> 00:39:26,559
Now you can get ninety six gigs in them. Jeez,

746
00:39:26,599 --> 00:39:29,559
it's a ten thousand dollars card. But you know that

747
00:39:29,719 --> 00:39:32,000
seems to be the thing that makes the most difference

748
00:39:32,320 --> 00:39:34,000
for a lot of these kinds of tools when you

749
00:39:34,039 --> 00:39:35,320
want to kindle a lot of contact.

750
00:39:35,400 --> 00:39:38,079
Speaker 2: What about an NPU? Is that gonna do it less

751
00:39:38,079 --> 00:39:40,199
than more than a ten thousand dollars video card.

752
00:39:40,360 --> 00:39:43,239
Speaker 1: No, because there's just no You know they talk about

753
00:39:43,320 --> 00:39:46,239
that Copilot plus PC has forty tops. I don't know

754
00:39:46,239 --> 00:39:49,199
what that means. Yeah, that's the trend trilling operation per second.

755
00:39:49,199 --> 00:39:52,920
It's the measure of its compute power for neural nets. Okay,

756
00:39:53,000 --> 00:39:55,760
my fifty eighty has thirteen hundred TOP. I see so.

757
00:39:56,320 --> 00:39:58,320
And when you look at what Nvidious selling the data

758
00:39:58,320 --> 00:40:01,360
centers and things, is their giant GPU like that with

759
00:40:01,519 --> 00:40:03,920
huge amounts of memory, this super fast memory and them

760
00:40:03,920 --> 00:40:04,960
for scale processing.

761
00:40:05,079 --> 00:40:07,119
Speaker 4: Yeah, the NPU, I think was more of a play

762
00:40:07,360 --> 00:40:12,320
for a continuous operation or in the background and on

763
00:40:12,599 --> 00:40:16,159
mobile devices where battery and power consumption is a much

764
00:40:16,199 --> 00:40:19,480
bigger concern for individuals, where they're thinking, well, I don't

765
00:40:19,519 --> 00:40:22,840
want this GPU chugging away in the background. Can I

766
00:40:22,880 --> 00:40:25,079
get something? Can I get something good enough, and that's

767
00:40:25,159 --> 00:40:28,400
kind of where that minimum bar is that doesn't absolutely

768
00:40:28,440 --> 00:40:30,840
consume my battery life. You know, you open your computer

769
00:40:30,920 --> 00:40:32,760
up and it's like, hey, I was working in the

770
00:40:32,760 --> 00:40:34,639
background seeing if anything was happening.

771
00:40:35,119 --> 00:40:37,880
Speaker 1: No, thank you. Yeah. Yeah. And it's been an argument

772
00:40:37,960 --> 00:40:42,280
now that you can jack up a PC enough with

773
00:40:42,400 --> 00:40:44,639
those with a couple of those big GPUs and run

774
00:40:44,679 --> 00:40:48,280
a mid size LLM on it. So you know, certainly,

775
00:40:48,320 --> 00:40:50,360
I've had conversations with folks where it's like, I am

776
00:40:50,400 --> 00:40:52,840
not prepared to send any of this data to the cloud.

777
00:40:53,280 --> 00:40:56,199
What can I do one hundred percent local? Yeah.

778
00:40:56,199 --> 00:40:58,400
Speaker 4: Another thing that you do have to consider if you're

779
00:40:58,400 --> 00:41:01,000
going to get into building and those apps are especially

780
00:41:01,039 --> 00:41:05,679
local apps, is the idea of multi modal. Yeah, these models,

781
00:41:06,000 --> 00:41:10,199
these local models, at least the Windows aiapis are not multimodel,

782
00:41:10,480 --> 00:41:11,519
so you will have to.

783
00:41:11,519 --> 00:41:14,079
Speaker 2: In other words, you can't talk to them and write

784
00:41:14,159 --> 00:41:16,039
to them exactly. Is that what you mean?

785
00:41:16,320 --> 00:41:16,559
Speaker 1: Right?

786
00:41:16,599 --> 00:41:18,360
Speaker 4: So you're going to have to build that. I mean

787
00:41:18,360 --> 00:41:20,400
you could, but you're going to have to put a

788
00:41:20,599 --> 00:41:24,519
speech recognition model in front of the LM or a

789
00:41:24,599 --> 00:41:28,559
object detection model plus an OCR model plus that you know,

790
00:41:28,599 --> 00:41:30,880
you have to maybe chain these models together and then

791
00:41:30,960 --> 00:41:35,000
you can get that multimodal experience where you can drop images,

792
00:41:35,039 --> 00:41:36,719
you can put PDFs in, but you have to be

793
00:41:36,760 --> 00:41:39,840
able to read the PDF. So these lllms don't read

794
00:41:39,880 --> 00:41:43,760
PDFs by default locally. You do have to get them

795
00:41:43,760 --> 00:41:46,800
into a text format. So if you're thinking about how

796
00:41:46,960 --> 00:41:48,639
you can apply this into your work, and I know

797
00:41:48,679 --> 00:41:51,440
a lot of enterprises, a lot of companies, a lot

798
00:41:51,440 --> 00:41:55,119
of their data is not in raw text format, so

799
00:41:55,239 --> 00:41:56,239
you do have to get it there.

800
00:41:56,320 --> 00:41:59,880
Speaker 1: Yeah, but there's an MCP for PDFs. So you know,

801
00:42:00,880 --> 00:42:02,480
glue these bits together.

802
00:42:02,440 --> 00:42:04,199
Speaker 4: Right, yeap, but you will have to do the gluing.

803
00:42:04,719 --> 00:42:05,800
Some assembly required.

804
00:42:05,880 --> 00:42:08,599
Speaker 1: This is the job, right, Like, this is not just

805
00:42:08,639 --> 00:42:12,480
an app you run, but we are assembling parts to

806
00:42:12,559 --> 00:42:15,159
try and get to a place where a model could

807
00:42:15,159 --> 00:42:15,559
be built.

808
00:42:15,639 --> 00:42:19,039
Speaker 2: So if you were going to build a local LLM

809
00:42:19,880 --> 00:42:25,679
Joe yourself using some existing technology, would you first reach

810
00:42:25,719 --> 00:42:28,320
for deep seek or would you go for just the

811
00:42:28,360 --> 00:42:30,760
stuff that Microsoft is exposing in Windows.

812
00:42:31,039 --> 00:42:33,760
Speaker 4: Yeah, I just reach for this stuff or a Microsoft

813
00:42:33,800 --> 00:42:37,800
is exposing in Windows and their five model. It's pretty good,

814
00:42:38,360 --> 00:42:42,679
it's pretty robust, and I would say it's a nice

815
00:42:42,880 --> 00:42:46,960
middle middle ground there for building on top of and

816
00:42:47,000 --> 00:42:51,239
fine tuning. I don't have enough time to be building

817
00:42:51,239 --> 00:42:54,079
all these applications and learn the APIs and learning the

818
00:42:54,719 --> 00:42:57,760
political history of where all these models come from. So

819
00:42:58,920 --> 00:43:02,920
it is a The benefit of Microsoft as a software

820
00:43:03,760 --> 00:43:06,599
provider is it's the one throat to choke, right, this

821
00:43:06,760 --> 00:43:10,239
is the one person you go to. They provide a

822
00:43:10,239 --> 00:43:12,199
lot of the tooling, they provide a lot of the models.

823
00:43:12,480 --> 00:43:14,159
Is it the best of any of the world's the

824
00:43:14,199 --> 00:43:17,079
absolute best. No, But when you're doing a lot of

825
00:43:17,079 --> 00:43:20,360
different stuff, sometimes you just have to have some heuristics

826
00:43:20,360 --> 00:43:23,360
here and just make the decision making. There's an infinite

827
00:43:23,440 --> 00:43:25,519
number of decisions that you have to make when you're

828
00:43:25,639 --> 00:43:29,719
picking all of these. So starting just with the built

829
00:43:29,719 --> 00:43:32,559
in tools, the built in APIs, it's a great easy

830
00:43:32,599 --> 00:43:36,039
way to get started. And if they don't work for you,

831
00:43:36,800 --> 00:43:40,760
then you can start making other questions and decisions. And yeah,

832
00:43:40,800 --> 00:43:43,119
but I would say start with the built in stuff

833
00:43:43,199 --> 00:43:44,199
definitely at first.

834
00:43:44,320 --> 00:43:46,760
Speaker 1: Okay, yeah, here I knew Ivious read I'd read this.

835
00:43:46,800 --> 00:43:51,639
I just looked at up again. Gptoss is a version

836
00:43:51,920 --> 00:43:56,440
of GPT three that can be run locally on a

837
00:43:56,480 --> 00:43:59,119
machine with sixty four gigs around and a fifty to

838
00:43:59,199 --> 00:44:02,280
ninety with twenty five gigs of v RAM. So that's

839
00:44:02,360 --> 00:44:07,559
roughly six or seven thousand dollars PC somewhere in that neighborhood,

840
00:44:07,559 --> 00:44:08,800
depending on how much you pay for the video car.

841
00:44:08,840 --> 00:44:10,480
The video cards can be driving around it. But that's

842
00:44:10,559 --> 00:44:14,519
running you know, GPT three, which is what the original

843
00:44:14,559 --> 00:44:17,280
GitHub copilot was built. Again, Like, that's a pretty torquy,

844
00:44:18,639 --> 00:44:21,400
pretty good little LM one hundred and twenty billion parameters.

845
00:44:21,800 --> 00:44:23,400
Like it's not GPT.

846
00:44:23,000 --> 00:44:23,639
Speaker 2: Four, but.

847
00:44:25,199 --> 00:44:28,639
Speaker 1: Especially in a narrow scope application like a NOME set

848
00:44:28,679 --> 00:44:32,239
of code, that's pretty robust. Man, you could do a

849
00:44:32,239 --> 00:44:32,880
lot with that.

850
00:44:33,280 --> 00:44:35,199
Speaker 4: Yeah, you could do a lot with that. And also

851
00:44:35,360 --> 00:44:38,440
you have to consider the big question of why would

852
00:44:38,480 --> 00:44:40,519
you build local ever, you know, why do it at all?

853
00:44:40,639 --> 00:44:43,719
Obviously privacy is a concern for a lot of people

854
00:44:43,760 --> 00:44:45,559
of why would you do this stuff locally on your

855
00:44:45,559 --> 00:44:49,039
own computer? If you have network concerns, if you don't

856
00:44:49,039 --> 00:44:52,639
have reliable or high quality or high speed internet, then

857
00:44:52,800 --> 00:44:56,239
obviously this is the only solution for you. But then

858
00:44:56,320 --> 00:45:00,280
also there's the cost concern and the cost question of yeah,

859
00:45:00,480 --> 00:45:03,320
you don't necessarily want to make some code that runs

860
00:45:03,320 --> 00:45:05,880
out and is running all these llms, and then you

861
00:45:05,960 --> 00:45:08,440
come back with a bill for you know, thousands of

862
00:45:08,519 --> 00:45:12,000
tens of thousands of dollars because your credits went crazy. Right,

863
00:45:12,159 --> 00:45:15,360
But when you have it local again, try There's so

864
00:45:15,519 --> 00:45:19,840
many cool tools, the AIDEV gallery, the AI toolkit, and

865
00:45:20,800 --> 00:45:24,039
then there's the APIs available already today. There's so many

866
00:45:24,039 --> 00:45:27,239
ways to get started and try and see. I you know,

867
00:45:27,280 --> 00:45:29,159
what is your application, what could it be? Try it

868
00:45:29,199 --> 00:45:31,199
out because you might not have to sign up get

869
00:45:31,199 --> 00:45:33,239
an API key at all. You could do all this

870
00:45:33,239 --> 00:45:36,000
stuff locally. And then if you want to do batch

871
00:45:36,119 --> 00:45:39,199
processing of again your own data, maybe you want to

872
00:45:39,480 --> 00:45:42,599
kind of use these models to put the data into

873
00:45:42,599 --> 00:45:45,719
a particular shape or clean it or work through it.

874
00:45:46,400 --> 00:45:49,360
But you don't want to pay tokens to do all

875
00:45:49,400 --> 00:45:52,079
that work. Well, do it locally, do it overnight. Build

876
00:45:52,119 --> 00:45:55,280
an app, your own app, not something you ship necessarily,

877
00:45:55,320 --> 00:45:57,840
but do it locally, you know, process that data locally,

878
00:45:57,880 --> 00:45:59,840
and then go from there. Maybe you're going to build

879
00:45:59,840 --> 00:46:01,440
your model, but first you have to get all the

880
00:46:01,480 --> 00:46:02,599
data in the right shape.

881
00:46:02,559 --> 00:46:05,920
Speaker 1: Right, and and you're trading time for money right right.

882
00:46:06,000 --> 00:46:08,559
Essentially the game you're playing here. It's like, Okay, if

883
00:46:08,559 --> 00:46:10,880
I run it on the cloud, it's going to cost

884
00:46:10,960 --> 00:46:13,280
me more, but I get it done less time, or

885
00:46:13,320 --> 00:46:15,920
I'm restricted to my own hardware so it may take longer.

886
00:46:16,679 --> 00:46:18,320
And then you start, you know, doing the economics. So

887
00:46:18,519 --> 00:46:21,280
just looking up the high end. Yeah, the ninety six

888
00:46:21,440 --> 00:46:26,400
gig in Nvidia RTX pro six thousand Blackwell, that's the

889
00:46:26,440 --> 00:46:28,400
big Box twelve thousands.

890
00:46:28,440 --> 00:46:30,039
Speaker 2: Well, you know, it's not only the money, but as

891
00:46:30,119 --> 00:46:33,880
Joe said, the security and the privacy that may trump

892
00:46:34,360 --> 00:46:37,639
any kind of money, and you know, and that may

893
00:46:37,639 --> 00:46:39,440
be the requirement you know.

894
00:46:39,719 --> 00:46:42,159
Speaker 1: Sorry that was Canadian dollars, just nine thousand Americans.

895
00:46:42,280 --> 00:46:44,760
Speaker 4: Ah, well that totally changes.

896
00:46:45,360 --> 00:46:51,320
Speaker 1: Yeah, everything's different. Now he just saved me two thousand

897
00:46:52,360 --> 00:46:55,039
three grand grand. But again, if I'm playing that game

898
00:46:55,079 --> 00:46:57,639
of the cost benefit, like what am I spending on

899
00:46:57,719 --> 00:47:02,039
tokens at that scale? True? And I really get the

900
00:47:02,119 --> 00:47:05,679
sense that as this sort of bubble starts to burst

901
00:47:05,679 --> 00:47:08,400
and people need to make money, like tokens ain't getting

902
00:47:08,480 --> 00:47:09,760
cheaper nowp.

903
00:47:09,760 --> 00:47:16,000
Speaker 4: Yeah, I have been using Claude and Codex and Copilot.

904
00:47:16,159 --> 00:47:21,119
There's definitely times where I have three computers running and

905
00:47:21,159 --> 00:47:23,639
they're I'm just kind of like telling them to keep

906
00:47:23,639 --> 00:47:27,119
going over. They're checking and building, but it's never going

907
00:47:27,199 --> 00:47:30,159
to be cheaper than it is now, Like this is

908
00:47:30,199 --> 00:47:31,679
the cheapest is going to be. They're trying to get

909
00:47:31,679 --> 00:47:35,039
as many users as possible, but that floor has to rise.

910
00:47:35,119 --> 00:47:38,519
I mean, I know Anthropic was having some issues a

911
00:47:38,559 --> 00:47:43,000
couple of weeks ago with limits and quality, and Codex

912
00:47:43,039 --> 00:47:45,400
I think had something a month or so ago where

913
00:47:45,599 --> 00:47:48,679
the limits. And again, if you're relying on these cloud services,

914
00:47:49,119 --> 00:47:51,440
not only are you relying on them to stay up

915
00:47:51,639 --> 00:47:54,400
and your connection to them to say live, but you're

916
00:47:54,400 --> 00:47:57,840
also relying on the model and the pricing and the

917
00:47:57,880 --> 00:48:00,679
availability at all from a business to point for them

918
00:48:00,719 --> 00:48:03,559
to stay up. Because it might make sense today, I was.

919
00:48:03,599 --> 00:48:07,000
Speaker 1: Talking to some folks abroad that are big, like running

920
00:48:07,039 --> 00:48:10,519
five sixty seven simultaneous instances because they're working that fast

921
00:48:10,639 --> 00:48:14,119
right tuned models, reaching these things, and they said that

922
00:48:14,199 --> 00:48:18,079
over July fourth everything got dramatically faster, like they got

923
00:48:18,079 --> 00:48:21,079
a ton of work going July fourth because Americans weren't

924
00:48:21,079 --> 00:48:24,440
working like these, like these cloud infrastructures are stressed to

925
00:48:24,519 --> 00:48:28,239
the limit and slowing performance as it is right and

926
00:48:28,280 --> 00:48:30,239
say and the only and the proof we've had is

927
00:48:30,280 --> 00:48:33,440
like when the stress isn't a high, things are better.

928
00:48:33,559 --> 00:48:36,519
So there is this interesting argument about at what point

929
00:48:36,559 --> 00:48:39,079
does this make more sense to be local versus remote?

930
00:48:39,320 --> 00:48:41,199
And this is going to be a shared resource too,

931
00:48:41,239 --> 00:48:44,000
like these big boxes don't have to be per dev

932
00:48:44,159 --> 00:48:47,239
They could be shared out again with potential performance issues

933
00:48:47,280 --> 00:48:50,280
like well, of course, I'm such a hardware geek, like

934
00:48:50,320 --> 00:48:52,440
I'd love to build out a rack of this stuff.

935
00:48:52,480 --> 00:48:53,679
Speaker 2: It would be fun, wouldn't it.

936
00:48:53,679 --> 00:48:55,960
Speaker 1: It would be and you know, and then now I've

937
00:48:56,000 --> 00:48:59,960
got the heat and power problems right.

938
00:49:00,039 --> 00:49:03,320
Speaker 4: To live it firsthand. Well to your point about shared resources,

939
00:49:03,559 --> 00:49:06,280
that is one of the nice things about win mL

940
00:49:06,519 --> 00:49:12,840
that just released Execution Provider that Microsoft announced making it

941
00:49:12,880 --> 00:49:17,000
easier for local devs to integrate models is if you

942
00:49:17,159 --> 00:49:20,599
have an application and you need a model, do you

943
00:49:20,719 --> 00:49:24,239
download it? And then every single one of your applications

944
00:49:24,320 --> 00:49:28,119
is downloading a five gig LM. Yeah, obviously that becomes

945
00:49:28,239 --> 00:49:31,280
untenable very quickly unless you have that twenty two terabyte

946
00:49:31,360 --> 00:49:35,719
drive in your computer. Solow you yeah, yeah, more than one.

947
00:49:35,880 --> 00:49:40,119
It does allow you to share models across application rich

948
00:49:40,159 --> 00:49:41,920
so you can have one machine install.

949
00:49:41,679 --> 00:49:44,039
Speaker 2: Richard, you were right. I thought they were SSDs.

950
00:49:44,119 --> 00:49:48,360
Speaker 1: They're not. They're HDDs ds. There are a few SSDs

951
00:49:48,400 --> 00:49:51,119
over eight terabytes, but most of them the line seems

952
00:49:51,119 --> 00:49:53,360
to be eight. By the way, the RTX six thousand,

953
00:49:53,679 --> 00:49:55,559
six hundred watts each.

954
00:49:56,400 --> 00:49:58,199
Speaker 2: That's why I have solar panels.

955
00:49:58,280 --> 00:50:00,960
Speaker 1: Yeah, that's it, you know, like oil BOYD. I'm just

956
00:50:01,000 --> 00:50:03,960
thinking about how much you remember in the end, this

957
00:50:04,039 --> 00:50:06,840
is moving electrons around and generating heat like you just

958
00:50:06,880 --> 00:50:09,599
made rocks make heat. Like that's saying time watts. You're

959
00:50:09,599 --> 00:50:10,880
gonna feel it. You don't want to sit in the

960
00:50:10,920 --> 00:50:13,159
room with that thing only man, No, it's going to

961
00:50:13,239 --> 00:50:16,880
be crazy. But it is an interesting point of view

962
00:50:17,760 --> 00:50:20,400
as we're still going through this to say, what are

963
00:50:20,440 --> 00:50:21,880
we going to shift local? What are we going to

964
00:50:21,960 --> 00:50:25,360
run remote? Like, what's feasible at what makes sense for

965
00:50:26,000 --> 00:50:28,880
folks here? And I think, you know, not everything has

966
00:50:28,920 --> 00:50:30,800
to be cloud and not everybody wants it there.

967
00:50:30,800 --> 00:50:35,119
Speaker 4: Right, And I think you just have to be you

968
00:50:35,159 --> 00:50:37,159
know wide. I'm not saying to get super deep on

969
00:50:37,239 --> 00:50:39,000
all of this stuff, but the tools for you to

970
00:50:39,039 --> 00:50:43,039
get your feet wet are available, and when you're CTO

971
00:50:43,320 --> 00:50:46,079
or more probably more likely, your CFO comes to you

972
00:50:46,159 --> 00:50:50,360
and says, hey, we can't afford this bill anymore. Your

973
00:50:50,519 --> 00:50:53,079
critical application can't use this LLM. You have to stop,

974
00:50:53,199 --> 00:50:56,440
or you have to change something because either somebody's prices

975
00:50:56,440 --> 00:50:58,000
went up or the business model changed.

976
00:50:58,159 --> 00:50:58,960
Speaker 1: Yeah, what are you going to do?

977
00:50:58,960 --> 00:51:00,639
Speaker 4: What are you going to reach for? And getting your

978
00:51:00,639 --> 00:51:03,440
feet wet in some of these local models, it's a

979
00:51:03,440 --> 00:51:05,880
great way to have an answer or have some sort

980
00:51:05,880 --> 00:51:08,039
of solution or see if that solution will work.

981
00:51:08,079 --> 00:51:11,320
Speaker 1: Now you're swapping op X for CAPEX and then, you know,

982
00:51:11,559 --> 00:51:14,800
using CFO speak like, we have two ways to solve

983
00:51:14,840 --> 00:51:17,199
this problem. We spend month over month on it, or

984
00:51:17,199 --> 00:51:20,159
we made a capital investment and spend less. You know,

985
00:51:20,760 --> 00:51:22,800
let's do the math. You know, if you want to

986
00:51:22,800 --> 00:51:24,480
talk to CFO, bring a spreadsheet.

987
00:51:24,599 --> 00:51:29,199
Speaker 4: Yeah, exactly. And it's as as we've said, as you've said,

988
00:51:29,480 --> 00:51:33,599
stuff is changing so fast. So if you get super deep,

989
00:51:33,800 --> 00:51:36,079
if you start training your own model, and then tomorrow

990
00:51:36,199 --> 00:51:38,880
somebody comes out with a model that just makes all

991
00:51:38,920 --> 00:51:42,119
that effort useless. This is again, this is like the

992
00:51:42,599 --> 00:51:44,920
sweet spot, right, Isn't this where the Windows developer has

993
00:51:45,000 --> 00:51:48,000
kind of always loved to live where they're like, yeah, yeah. Yeah,

994
00:51:48,000 --> 00:51:51,079
we're not like hardware level, we're not doing machine code.

995
00:51:51,079 --> 00:51:53,880
But then we're also not just like bleeding like the

996
00:51:53,920 --> 00:51:55,599
best of the best. It's like, okay, we're in the

997
00:51:55,599 --> 00:51:57,880
middle here where we got models, we got a local

998
00:51:58,239 --> 00:52:01,199
It's it's efficient, it's it's a good balance.

999
00:52:01,320 --> 00:52:03,119
Speaker 1: Yeah. Well, and I'm going to call back to Cagle

1000
00:52:03,159 --> 00:52:05,559
again because one of the other ways you can get

1001
00:52:05,559 --> 00:52:08,320
a model built is to put out a bounty on

1002
00:52:08,480 --> 00:52:12,559
Cagle in a competition to have someone build it for you. Effectively,

1003
00:52:13,159 --> 00:52:15,639
there you go. So you've got the data set, but

1004
00:52:15,719 --> 00:52:17,719
you don't want to actually do the construction. You can

1005
00:52:18,199 --> 00:52:22,880
host a competition and define your problem space and provide

1006
00:52:22,880 --> 00:52:24,920
the sample data, and a bunch of people compete for

1007
00:52:25,199 --> 00:52:28,320
to deliver you the best model. It's a weird world, man,

1008
00:52:28,519 --> 00:52:30,280
is like, if you want to go deep into mL,

1009
00:52:30,320 --> 00:52:32,880
there's so many interesting things to be done here. M hmmm.

1010
00:52:33,480 --> 00:52:36,119
Speaker 2: I had the weird meta thought that you could get

1011
00:52:36,159 --> 00:52:40,760
a model to build your model instead of you know,

1012
00:52:40,880 --> 00:52:42,239
farming it out for a bounty.

1013
00:52:42,320 --> 00:52:45,400
Speaker 1: Well, you're not wrong to interact with an LM to

1014
00:52:45,480 --> 00:52:49,039
start constructing a plan around how a model would get built,

1015
00:52:49,039 --> 00:52:51,119
because that you know, in the end, they are a

1016
00:52:51,159 --> 00:52:53,239
pretty clever search tool for best practices.

1017
00:52:53,639 --> 00:52:57,880
Speaker 4: Yeah, search and tokenization is a really nice thing that

1018
00:52:57,920 --> 00:53:00,639
you can do with your local LM of crunching some

1019
00:53:00,719 --> 00:53:03,800
of your data, your text, tokenize it make it easier

1020
00:53:03,800 --> 00:53:08,519
to search, have that more natural language available for your users.

1021
00:53:08,719 --> 00:53:10,880
It's a really hard thing to code, but if you

1022
00:53:10,880 --> 00:53:13,039
have local l MS, I can help you build that.

1023
00:53:13,199 --> 00:53:14,480
Speaker 1: Why not. Yeah, that's cool.

1024
00:53:15,280 --> 00:53:17,000
Speaker 2: Anything else on your mind that you want to touch

1025
00:53:17,039 --> 00:53:19,360
on before we call it a show?

1026
00:53:20,039 --> 00:53:22,280
Speaker 4: Not really, I mean we touched on a lot here. Yeah,

1027
00:53:22,280 --> 00:53:23,400
we just try it.

1028
00:53:23,679 --> 00:53:27,039
Speaker 1: We went we went on a ride today friend again. Yeah,

1029
00:53:27,079 --> 00:53:28,360
but this is the kind of deep.

1030
00:53:28,159 --> 00:53:31,639
Speaker 2: Dive into local lms and local AI that I really

1031
00:53:31,679 --> 00:53:35,400
wanted to get to. So I'm very very happy we talked.

1032
00:53:35,519 --> 00:53:36,199
Thank you, Joe.

1033
00:53:36,440 --> 00:53:37,559
Speaker 4: Yeah, happy to be here.

1034
00:53:37,840 --> 00:53:38,960
Speaker 2: I'm right and we'll.

1035
00:53:38,800 --> 00:53:58,880
Speaker 5: Talk to you next time on dot net rocks.

1036
00:54:03,079 --> 00:54:05,800
Speaker 2: Dot net rocks is brought to you by Franklin's Net

1037
00:54:05,880 --> 00:54:09,840
and produced by Pop Studios, a full service audio, video

1038
00:54:09,920 --> 00:54:14,000
and post production facility located physically in New London, Connecticut,

1039
00:54:14,239 --> 00:54:19,039
and of course in the cloud online at pwop dot com.

1040
00:54:19,239 --> 00:54:21,360
Visit our website at d O T N E t

1041
00:54:21,599 --> 00:54:25,639
r o c k S dot com for RSS feeds, downloads,

1042
00:54:25,760 --> 00:54:29,480
mobile apps, comments, and access to the full archives going

1043
00:54:29,480 --> 00:54:32,880
back to show number one, recorded in September two thousand

1044
00:54:32,920 --> 00:54:35,559
and two. And make sure you check out our sponsors.

1045
00:54:35,719 --> 00:54:38,760
They keep us in business. Now go write some code,

1046
00:54:39,079 --> 00:54:39,840
See you next time.

1047
00:54:40,760 --> 00:54:42,559
Speaker 4: You got jas.