Speaker 1: Hey Richard, Hey Carl, what do you know?
Speaker 2: Well, I know that our friend Michelle Rubusta Monte is
Speaker 2: with us to tell us about something that's going on
Speaker 2: adjacent to DEV Intersection.
Speaker 1: What is it? It's cybersecurity Intersection. Let's let Michelle tell
Speaker 1: that story.
Speaker 3: Hey Michelle, Hey Carl, Hey Richard, how are you.
Speaker 2: Tell us about cybersecurity Intersection?
Speaker 3: Well, so, Richard and I are partnering with the group
Speaker 3: that does DEV Intersection and next Gen AI, and we
Speaker 3: are putting on a new conference dedicated to one hundred
Speaker 3: percent security focused topics. And I mean, honestly, the lineup
Speaker 3: of speakers is incredible. We have Paula A. Jenis, who's
Speaker 3: here from Poland and does keynotes all over the world
Speaker 3: and is one of the top rated RSA speakers and
Speaker 3: black hat speaker. We're so lucky to have her. But
Speaker 3: she's not only keynoting, she's got a workshop teaches you
Speaker 3: about protecting your environments against hackers and shows you about
Speaker 3: how to you know, do attacks so that you can
Speaker 3: prevent them. It's pretty cool and sessions like that as well.
Speaker 3: But we also have speakers from Microsoft. We have we
Speaker 3: have speakers that specialize in you know secure coding practices,
Speaker 3: Azure security, Zuero, trust architectures on Azure UH and people
Speaker 3: who do decision maker tracks, so things around governance policy
Speaker 3: and you know how to how to manage and your
Speaker 3: production operations keep them secure. So it's an amazing group
Speaker 3: of speakers, really excited about it.
Speaker 2: And I think I can count myself among the group
Speaker 2: of speakers there.
Speaker 3: Well, yes you can. That is great.
Speaker 2: Yeah, I'm doing a securing Blazer Server applications talk and
Speaker 2: also I think we're doing a Security this Week live
Speaker 2: show there somewhere that is correct.
Speaker 3: Yeah, we'll be recording Security this Week Live. We're going
Speaker 3: to have a great panel with some folks. The interesting
Speaker 3: thing here is we don't really have a Microsoft and
Speaker 3: dot net and Azure focused toecurity conference yet, so that's
Speaker 3: the reason we're putting this on as well. You know
Speaker 3: there are other security conferences, but they have a spread
Speaker 3: of topics that maybe don't focus on the things you
Speaker 3: do day to day. And you know this overlaps with
Speaker 3: again our community of folks that specialize in again dot net,
Speaker 3: Azure and yeah, they need to keep it secure too,
Speaker 3: So with tons of talks.
Speaker 1: Cyber Intersection is part of a trio of conferences we're doing.
Speaker 1: They have Intersection alongside the next Gen AI conference all
Speaker 1: in Orlando the week of October fifth through tenth. That's
Speaker 1: workshops and the main conference. And you can get a
Speaker 1: special registration code if you sign up through Cybersecurity Intersection
Speaker 1: dot com.
Speaker 3: Yeah, so if you sign up at Cybersecurity Intersection dot com,
Speaker 3: then you put in this code so Alliance cyber three
Speaker 3: hundred and you'll get three hundred off the entry price.
Speaker 3: So that's a special code that only works at cybersecurity
Speaker 3: dot com. And then you have access to all the conferences.
Speaker 2: Like Richard said, Wow, that's cool. Thanks Michelle. I'm looking
Speaker 2: forward to it and I'll see you there. Hey, get
Speaker 2: down rock and Roll. It's Carl Franklin and Richard Campbell
Speaker 2: for dot net Rocks.
Speaker 1: Hey, Richard, how you do it, Bud?
Speaker 2: I'm good, getting psyched up to go down to Orlando.
Speaker 1: Yeah, it's almost time back to a new dev Intersection
Speaker 1: and next jen AI and the New Cybersecurity Conference side
Speaker 1: by side. Yep, yep.
Speaker 2: Looking forward to doing a live security this week's show
Speaker 2: down there.
Speaker 1: That should be fun, fun and you're crazy thing with Maddie.
Speaker 1: Oh god, you're going to aspireify dot net rocks here.
Speaker 2: I have no idea what to expect. That could be
Speaker 2: a horror show.
Speaker 1: This is you know, you love a good you know,
Speaker 1: trapease act, just going without a net.
Speaker 2: Absolutely, as long as I don't you know, screw up
Speaker 2: too badly, it should it should work out fun.
Speaker 1: You know, a good crash it burns fun too, but it.
Speaker 2: Could be fun. Yeah, yeah, yeah, Okay, let's start with
Speaker 2: nineteen seventy. That's the episode number. Oh yeah, and a
Speaker 2: bunch of things happened in nineteen seventy.
Speaker 1: Where do you want to start?
Speaker 2: Well, the unhappy things, the Kent State shootings.
Speaker 1: Yeah, it's terrifying.
Speaker 2: On May fourth, National Guard troops killed four students during
Speaker 2: protest against the Vietnam War at Kent State University in Ohio,
Speaker 2: leading to nationwide outrage and the song what is it
Speaker 2: for Dead in Ohio?
Speaker 1: Who's that?
Speaker 2: Neil Young or Crosby Sills, Nash and Young? I'm not sure.
Speaker 2: Nigerian Civil War. The conflict ended in January when Biaffron
Speaker 2: forces the Affron forces surrendered after a thirty two month
Speaker 2: struggle for independ and it's the first Earth Day was observed.
Speaker 2: On April twenty second, the Beatles broke up and let
Speaker 2: it be. McCartney said he was leaving the band on
Speaker 2: April tenth. That was the end of that. But John
Speaker 2: Lennon instant karma. He wrote and recorded this hit song
Speaker 2: in a single day, showcasing his prolific creativity. Diana Ross
Speaker 2: and the Supremes gave their final concert in Las Vegas
Speaker 2: in January fourteenth. Back to the bad stuff the Tonguhai earthquake.
Speaker 2: Devastating earthquakes struck Tongue High County, China on January fifth,
Speaker 2: resulting in significant casualties, with estimates of up to fourteen thousand,
Speaker 2: six hundred and twenty one deaths. Yeah, and an avalanche
Speaker 2: in someplace that in France that I can't pronounce zi
Speaker 2: fool full, Sorry about that killed forty two people, making
Speaker 2: one of the worst disasters in French skiing history. You
Speaker 2: can talk about the science, yea science. Some things happened,
Speaker 2: well I was.
Speaker 1: I mean the space one's the obvious one. After having
Speaker 1: both Apollo nine, Pollo ten, and Paula eleven and Apollo
Speaker 1: twelve all in nineteen sixty nine, there was only one
Speaker 1: Apollo mission in nineteen seventeen. That was a Poulla thirteen.
Speaker 1: It launched on April eleventh, and on April thirteenth they said,
Speaker 1: we've had a problem.
Speaker 2: Here, Houston. We've got a problem. And we're a great
Speaker 2: movie too.
Speaker 1: Yeah, and you've seen the movie, a beautiful rendering of
Speaker 1: more or less what happened. The HBO Earth of the
Speaker 1: Moon series, if you ever get a chance to watch,
Speaker 1: that does a version of Apollo thirteen, but from the
Speaker 1: view of the people on the ground, so you only
Speaker 1: ever hear the astronauts over the radio, which is how
Speaker 1: it was. Right. Sure, here's the crazy thing to realize.
Speaker 1: So the explosion in the tank happens on April on
Speaker 1: April thirteenth, the splash downy April seventeenth. It was four days. Wow,
Speaker 1: the whole thing's four days. I know, it feels like forever.
Speaker 1: It's four days.
Speaker 2: Wow.
Speaker 1: But it was four days of are these guys going
Speaker 1: to make it? You know, like four days of sheer terror. Yeah,
Speaker 1: it was. And of course they the lunar module Aquarius
Speaker 1: was turned into a lifeboat because the power systems, a
Speaker 1: little bit of battery that was left in the command
Speaker 1: module is going to need for re entry, so they
Speaker 1: basically powered down the command module and then use the
Speaker 1: Life Sports system for two four to three and just
Speaker 1: four days and they were able to get home amazing
Speaker 1: and survive. It's a great story. And of course the
Speaker 1: next Apollo mission would be delayed while they dealt with
Speaker 1: some of those issues, and in nineteen seventy one, you'll
Speaker 1: get Apollo fourteen. Talk about that next week apparently. Yeah.
Speaker 1: On the computer side of things, Nicholas Worth releases Pascal Woll.
Speaker 1: He worked previously on the language I'll Go sixty and
Speaker 1: there's some derivations therein he was trying to do a
Speaker 1: combination of sort of procedural and algorithmic programming. So Popular
Speaker 1: Language did some great things. But on the heart work side,
Speaker 1: for me, the show stealer is my you know, iba
Speaker 1: Intel's most important product, the eleven O three, the d ram. Okay,
Speaker 1: this is what Moore's law actually was about, was making
Speaker 1: RAM right based on a bunch of other developments to
Speaker 1: make a transistor based memory. They were able to make
Speaker 1: a silicon substrate for an eighteen dip pin dip can
Speaker 1: with one K of RAM in it for sixty bucks.
Speaker 2: Wow, that seems cheap back then, and.
Speaker 1: One cent per bit, and it was small because then
Speaker 1: they were largely using core magnet Ferris cores for memory.
Speaker 1: So this was very compact and it was adopted immediately everywhere.
Speaker 1: It's it's uptake. That's also the same year that the
Speaker 1: first version of the IBM system three seventy comes out
Speaker 1: with all semi conductor RAM, but that was not Intel's RAM.
Speaker 1: But shortly after that, Intel's RAM just dominates the market
Speaker 1: and sends Intel on its trajectory. Although nineteen seventy one
Speaker 1: they'll make arguably and even more in product important product.
Speaker 1: Tune in next week for nineteen seventy one, nineteen seventy one.
Speaker 1: But yes, the eleven O three was there, you know,
Speaker 1: definitive product. They were rammed digit you know, semiconductor ramming.
Speaker 1: And that's what I got.
Speaker 2: All right, well, I guess we should carry on with
Speaker 2: better no framework.
Speaker 1: Roll the crazy music possible.
Speaker 2: All right, man, what do you got again? I looked
Speaker 2: for a trending repost on GitHub and I found MCP
Speaker 2: for Unity. Oh my, yeah, you know Unity Create create
Speaker 2: games with the Unity. It's a graphical tool that uses
Speaker 2: c sharp and JavaScript for scripting, but it also does
Speaker 2: all of the three D stuff. So here's what it is,
Speaker 2: proudly sponsored and maintained by Coplay, the best AI systant
Speaker 2: for Unity. There you go create your Unity apps with
Speaker 2: l l MS. M CP for Unity acts as a bridge,
Speaker 2: allowing a assistance like Claude Cursor to interact directly with
Speaker 2: your Unity editor via a local MCP model context protocol.
Speaker 2: We've been talking about those. A local MCP client use
Speaker 2: your lll M tools to manage assets, control scenes, edit scripts,
Speaker 2: and automate tasks within Unity.
Speaker 1: Pretty cool. Interesting, Yeah, a good show to actually walk
Speaker 1: through the process of, you know, including making a game
Speaker 1: in Unity with with the MCPM, with l MS in
Speaker 1: the role. Yep.
Speaker 2: Also code it with the AI. Dot com is up
Speaker 2: and the first episode is there and we're basically using
Speaker 2: playwright to with the code agent in the visual studio
Speaker 2: code nice and using clauds on it. And we basically
Speaker 2: one prompt told it to create a user documentation of
Speaker 2: Jeff Fritz's copilot do John dot com website, and it
Speaker 2: did a pretty good job. What we didn't show was
Speaker 2: what's involved in setting up the playwright MCP so that
Speaker 2: the agent can use it. Oh yeah, and it turns
Speaker 2: out that's pretty complex. You need node JS and NPM
Speaker 2: and all that stuff, and we're looking for a video
Speaker 2: on how to do that, so look in the show
Speaker 2: notes for that. Cool, but that's it for a better
Speaker 2: no framework. Who's talking to us? I have a common
Speaker 2: of a show nineteen sixty nine. Yes, that's last week's
Speaker 2: show with our friend James monte Magno.
Speaker 1: And we talked a little bit about the AI tooling
Speaker 1: inside of Visual Studio code and its relationship with Visual
Speaker 1: Studio and so on. And our friend Richard Rukima, also
Speaker 1: known as Coputer, has this common but he says, I
Speaker 1: think Richard nailed it. Do you like the code or
Speaker 1: do you like a solution? I consider my expertise working
Speaker 1: with AI as a beginner, especially after listening to James,
Speaker 1: but I felt that vibe of joy in getting things
Speaker 1: done so fast. So do I like the if then else?
Speaker 1: Or do I like ask you for a future reviewing
Speaker 1: the result? I'm long past the joy of knowing how
Speaker 1: to write procedural code. Yeah. An interesting aspect of this is, like,
Speaker 1: is it the more experienced folks that are going to
Speaker 1: embrace these tools faster? Because it's typically the more junior
Speaker 1: people that tend to jump on the bandwagona new things,
Speaker 1: but I hear the same tone over and over again. Yep.
Speaker 1: Certainly in terms of respectful interaction with AI, I don't
Speaker 1: prescribe to the harsh language, as I feel it reveals character.
Speaker 1: It's an interesting statement rights in my character not to
Speaker 1: be harsh fully or and to focus on being respectful communication.
Speaker 1: I don't think AI should be treated any different, not
Speaker 1: for the benefit of AI or the benefit of myself.
Speaker 2: Yeah, exactly. You're not going to feel good, you know,
Speaker 2: using harsh language.
Speaker 1: Putting those mean words out there is as much impact
Speaker 1: on you as it is on anything else. And leave me.
Speaker 1: The software is not affected, that's thing, right.
Speaker 2: The only thing left to be affected is you.
Speaker 1: Yeah, so be kind to yourself. It's not necessary, right, Hey, Richard,
Speaker 1: I'm pretty sure you've got a copy of Music code
Speaker 1: By already, but thanks so much for your comment. But
Speaker 1: if you'd like a copy of music, Cobey, I write
Speaker 1: a comment on the website at dot net rocks dot
Speaker 1: com or on the facebooks to publish every show there
Speaker 1: and every comment there, and never reading the show, will
Speaker 1: send your copy of music Go.
Speaker 2: Music to code By is still going strong after all
Speaker 2: these years twenty two tracks. You can get him in
Speaker 2: uh wave, flack or MP three and that's at Music
Speaker 2: to Code by dot Net. Okay, let's bring back our
Speaker 2: friend Joseph Finney. Joseph is a mobile product owner in
Speaker 2: MVP by day and he builds productivity apps for Windows
Speaker 2: by night. When he's not programming, he's burning running and
Speaker 2: enjoying tasty coffee and beer in Milwaukee.
Speaker 1: Hey Joe, Hello, welcome back to having that.
Speaker 4: Good to be back talking more about the hot topic
Speaker 4: of the day, AI.
Speaker 1: With a Century. Yeah, but you've got you've got a
Speaker 1: cool angle of this. That's why I asked you that
Speaker 1: to come on. So what are you working on?
Speaker 4: Well, one of my most popular apps that I make
Speaker 4: is text grab, which is pretty basic. It's also the
Speaker 4: basis for the Power Toys Text Extractor, which is basically
Speaker 4: select a region on your screen of somebody who sent
Speaker 4: you text that you can't actually select and put somewhere
Speaker 4: where you want it. And it does some on device
Speaker 4: local OCR. Pretty simple, and now with these new models,
Speaker 4: the OCR is getting better. But it does change compatibility
Speaker 4: and devices, but it's it's pretty interesting what we can
Speaker 4: do here now with these local models Microsoft's making it
Speaker 4: easier with some of their Windows AI APIs, and then
Speaker 4: there's it just gets more and more complicated from there.
Speaker 2: Mm hmm. So I have an app that I'm running
Speaker 2: right here that does little OCR and I'm using Tesseract
Speaker 2: to read the text in a bitmap at a certain coordinate.
Speaker 2: That is that the sort of representing the state of
Speaker 2: the art before AI got into the mix.
Speaker 4: Yeah, I would say it's it's similar. Tests React was
Speaker 4: the open source project that Google took over I think.
Speaker 4: I think actually HP started it way back there and
Speaker 4: then kind of Google took it over. Yeah, it's on GitHub.
Speaker 4: There's a lot of models. It's very widely used and loved.
Speaker 4: Text grab does enable you to download tests earect and
Speaker 4: then you can interact with it through the CLI. Well
Speaker 4: text grab will just interact with it directly, but there's
Speaker 4: a little bit of setting up. You do have to
Speaker 4: download it. It's a it's another installer. It's through ub
Speaker 4: Mannheim I think who does the installation. So there's definitely
Speaker 4: some hoops you have to jump through to get it working.
Speaker 2: And there's a data set that goes along with it, right.
Speaker 4: Yeah, Yeah, so yeah, you have to download the languages.
Speaker 4: There's a lot. One of the benefits there of Tessaact
Speaker 4: is that there's a lot of languages, and they have
Speaker 4: packages for scripts, and they have packages for like handwritten
Speaker 4: and so it's really high quality. Originally, Textcrab was built
Speaker 4: using the Windows ten ocr APIs, which are definitely older,
Speaker 4: not as good, but they're very fast. So that was
Speaker 4: kind of the nice thing there. They're built in, they're fast,
Speaker 4: they're quick for most stuff. It worked pretty well cool
Speaker 4: test erect was a bump up, but again you have
Speaker 4: that complexity where you have to download the models locally.
Speaker 4: But it's open source, it's available, it's free. And now
Speaker 4: there's these Windows AI APIs that Microsoft has released. I
Speaker 4: don't think we know exactly what those models are. I
Speaker 4: don't think they've shared. I haven't learned what they are exactly.
Speaker 2: But what was the acronym that you used before we
Speaker 2: started recording for this new.
Speaker 4: Wind WINML, Windows and machine learning.
Speaker 2: Okay, and this is new, yeah, literally days old than
Speaker 2: we don't know anything about it.
Speaker 4: Well, the win mL stuff is kind of a middle
Speaker 4: layer here, Okay. So I would say there's like three
Speaker 4: general levels of intensity. If you are a local Windows
Speaker 4: app developer and you want to get ocr image language
Speaker 4: models like all of that stuff. If you want to
Speaker 4: do that in your app. I would say there's like
Speaker 4: three different tiers of complexity that you can engage in,
Speaker 4: and the first one is the new Windows aiapis. And
Speaker 4: these were released kind of around the time the Copilot
Speaker 4: plus PCs were released, Okayne, and they've been rolling out. Yeah,
Speaker 4: they've been rolling out slowly. They were in experimental. You
Speaker 4: had to be on the insider preview to build them.
Speaker 4: To use them, you have to have a co Pilot
Speaker 4: plus PC. But you know, there's a higher bar kind
Speaker 4: of on the consumer side, but that means it's easier
Speaker 4: on the developer side. So they basically in the code
Speaker 4: when you're building, you just have to check does this
Speaker 4: device support these APIs? If so, do it very simple
Speaker 4: and like that's it. You don't have to manage models,
Speaker 4: you don't have to manage memory or downloading, and you
Speaker 4: don't have to worry about shipping. You know, a five
Speaker 4: gig model with your app. They're already on the device.
Speaker 4: If the device supports it, then you can kind of
Speaker 4: light up those features, turn on those buttons, show that capability,
Speaker 4: and boom, it's there.
Speaker 1: Kelly.
Speaker 2: My wife bought a new Copilot plus PC. She didn't,
Speaker 2: of course know it. We went to best Buy together,
Speaker 2: you know, and she picked it out. But the first
Speaker 2: thing I did is immediately turned off all this stuff.
Speaker 2: It's going to get in the way. The thing that
Speaker 2: takes screenshots all the time. I can't remember the name
Speaker 2: of it now, recall, recall, that's it. It was turned
Speaker 2: off by default. So that's good. That's good. I did
Speaker 2: not want that on.
Speaker 1: It's a really powerful tool. People love it, you know,
Speaker 1: like because the bottom line is you can you can
Speaker 1: ask the machine, he where did I see such and such,
Speaker 1: and it'll find it for you. Yeah.
Speaker 2: I just don't have that kind of problem, like I
Speaker 2: know where I saw stuff, and I keep good notes
Speaker 2: and dot your machine. Yeah, she didn't want.
Speaker 4: It, so yeah, I also don't use it like I
Speaker 4: have AI features in well, AI, I should say, I
Speaker 4: know this show Richard has talked a lot about how
Speaker 4: you have these big amorphous buckets of AI, and then
Speaker 4: as soon as you start explaining it and giving a
Speaker 4: more clear, straightforward name to it, it stops really being AI.
Speaker 4: And that's kind of where the OCR and LLM and
Speaker 4: image segmentation and image detection. So those are all under
Speaker 4: this umbrella of AI, and it can be a little
Speaker 4: I don't know.
Speaker 1: You left the impolite part, Joe, which is like, so
Speaker 1: for me, the term artificial intelligence means something that doesn't work. Yeah,
Speaker 1: there you go, because as soon as it does work,
Speaker 1: it gets a new name.
Speaker 4: Software, right right, that's it's a module. Yeah, well, I
Speaker 4: should say, then the using name space in dot net
Speaker 4: is AI. But then after that there's always dot tech
Speaker 4: that imaging that image recognition. So there's a bunch of
Speaker 4: there's a bunch of APIs after the namespace that actually
Speaker 4: point to the real APIs, the real functionality of what
Speaker 4: you're actually trying to do. And I don't think you
Speaker 4: can easily turn all of that off. I would say,
Speaker 4: so there's a lot of experiences that are built on
Speaker 4: top of this technology that's already in these Copilot plus PCs,
Speaker 4: and you could turn those experiences off. You know, they're
Speaker 4: not going to run by default. But Microsoft does a
Speaker 4: pretty good job of managing bringing down the model, keeping
Speaker 4: it up to date, and making it really easy for
Speaker 4: developers to interact with, which is kind of what you want, right,
Speaker 4: You want something really simple easy. It's a super complex problem,
Speaker 4: but you could just say, you know, send this block
Speaker 4: of text, summarize it, and then get it back.
Speaker 2: So in case anyone hasn't figured it out by now,
Speaker 2: the Copilot plus PC has a local LLM built into it.
Speaker 1: Yep.
Speaker 2: And you know, this is the kind of thing that
Speaker 2: you might think of if you were going to use
Speaker 2: OLAMA right and download models and you know, train it,
Speaker 2: run it on a laptop or something like that, the
Speaker 2: gaming PC or something.
Speaker 4: Yeah, there's that's just kind of where I said, there's
Speaker 4: like these different layers of the complexity and the easiest, simplest,
Speaker 4: like lowest level, easiest for any developer out there to
Speaker 4: integrate into their Windows app. Any Windows app by the way,
Speaker 4: so WPF or when UI or wind forms you can
Speaker 4: or MAUI, you can do them all. It does have
Speaker 4: to have identity, some sort of identity because SB there's
Speaker 4: Microsoft doesn't want to just open up these APIs to
Speaker 4: any random raw ex But if you want to do
Speaker 4: some more maybe more niche stuff, maybe a little bit
Speaker 4: more complicated stuff, or you want to use this specific model,
Speaker 4: you can kind of use what I would call like
Speaker 4: the next step of complexity here, and that's win mL
Speaker 4: and that's there's a little bit of a middle layer
Speaker 4: there where you can go download your own on X
Speaker 4: models and run those and it makes it easy. There's
Speaker 4: like a basically a standardized interface and you say, run
Speaker 4: this model. You don't have to necessarily optimize it for
Speaker 4: the specific hardware and it can run CPU, GPU and
Speaker 4: PU and it's an easy way. But again, there you
Speaker 4: have to manage the model. So if you want that,
Speaker 4: if you need that in your application, maybe you have
Speaker 4: it specifically fine tuned for your application, or you have
Speaker 4: a model that isn't in the box, or I don't
Speaker 4: know if there are other legal or.
Speaker 1: Hey, I'm just appreciating you're talking about something other than
Speaker 1: in the LLM, because it's just it's just overwhelming right now.
Speaker 1: So you know, clearly there's a bunch of other models
Speaker 1: out there and all of those infrastructure, and I'm including
Speaker 1: links to onyx and things like if you haven't looked here,
Speaker 1: there's lots of good work being done for specific tasks.
Speaker 4: Yeah, and I think immediately people can kind of get
Speaker 4: annoyed by, oh, LM, why do I need an LLM
Speaker 4: in my model? I'll need AI and it definitely has
Speaker 4: become synonymous like AI and LLM. Yeah, but there are
Speaker 4: so many If you go to hugging face and you
Speaker 4: look at all the different categories, I mean OCR, image segmentation,
Speaker 4: image detection, object detection, huggy face, oh yeah, hugging face,
Speaker 4: hugging face, hugging face. Yeah, this is a I think
Speaker 4: Facebook is kind of backing it. And it's a big
Speaker 4: repository for models, so you can access models, you get
Speaker 4: download models, and if you're thinking.
Speaker 1: Before the insanity of lllms, we had we had good
Speaker 1: tooling around just building machine models for object detection and
Speaker 1: recognizers and OCR all these good things, right, Like, it's
Speaker 1: just there was so much going on before chat GPT
Speaker 1: showed up and just overwhelm the message.
Speaker 2: Wow, hugging face looks awesome.
Speaker 4: Yeah, it's it is a huge, kind of big repository
Speaker 4: of models online where you can go download them. But
Speaker 4: if you're a normal person who's just curious and says
Speaker 4: I want to kind of to try some of these out,
Speaker 4: it's not as easy. You can't just download them and
Speaker 4: then run them. They are not programs, their models, so
Speaker 4: you need to interface with them somehow, and there is
Speaker 4: actually a way if you are inclined. You can download
Speaker 4: an app from Microsoft called the AI Dev Gallery app.
Speaker 4: And what this is it's kind of a playground for
Speaker 4: people who are curious about models and different models and
Speaker 4: how this all works. It's open source on GitHub, it's
Speaker 4: in the Microsoft Store and it is a really low
Speaker 4: barrier to entry if you are interested in trying some
Speaker 4: of these models out on your own device.
Speaker 2: Wow.
Speaker 4: So you can download models from hugging Face. You can
Speaker 4: run them. They're very limited, basic samples, so don't expect
Speaker 4: anything grandiose or chaining them together. But it's a great
Speaker 4: way to play with those Hugging Face models.
Speaker 2: Very cool.
Speaker 1: Did you ever play with Cagle, because we've talked about
Speaker 1: this on the show Ages Ago. Just like there is
Speaker 1: another playground for practicing your mL skills.
Speaker 4: I've never tried. It is in a a website or
Speaker 4: a technology.
Speaker 1: They actually run competitions for you know. The sort of
Speaker 1: famous one for them was the predict how many people
Speaker 1: survive the Titanic sinking. There was a bunch of different
Speaker 1: models or different competitions, and some of them have a
Speaker 1: lot of money in them because they're actually you know,
Speaker 1: organizations encourage folks to mature a model particular problem space
Speaker 1: that they can then use elsewhere. There was things like
Speaker 1: aneurysm detection and even sports predicting. So just again a
Speaker 1: reminder that there's things other than llms.
Speaker 4: Right, And I would say that is the like the farthest,
Speaker 4: the highest tier of integrating AI models into your app,
Speaker 4: your local Windows app is making your own models, training
Speaker 4: your own models from scratch, So you can do that.
Speaker 4: I mean, you can ship models and integrate them directly in.
Speaker 4: It's again way more integration work, but it's way more
Speaker 4: fine tuned. So if you have a specific application where
Speaker 4: you need a model that can do very niche things
Speaker 4: or very specific data sets, it's possible. It's doable, and
Speaker 4: there's ways to do it. You should check it out.
Speaker 4: One of the nice things about this current age of
Speaker 4: programming is a lot of these big popular apps are
Speaker 4: open source, so you can just see how it's done,
Speaker 4: and you obviously read the license, but a lot of
Speaker 4: this stuff is available to see how other people are
Speaker 4: integrating these AI models.
Speaker 1: Guys.
Speaker 2: I know we've talked about deep seek a bit on
Speaker 2: this show, and Joe's nodding his head, so he knows
Speaker 2: about it, and this was the model that came out
Speaker 2: of China that uses a lot less resources and is
Speaker 2: therefore cheaper to run than you know, chat GPT was,
Speaker 2: and everybody was like, oh my god, open ai is
Speaker 2: going down, and it didn't. And then there were concerns
Speaker 2: about you know, if I use deep Seek, am I
Speaker 2: sharing data with you know, the country of China and
Speaker 2: is it safe in all of these things. But you
Speaker 2: can also I think, correct me if I'm wrong, but
Speaker 2: download it the app and run it locally like olama.
Speaker 2: Is that true?
Speaker 1: Yeah?
Speaker 4: That So one of the nice things about deepseek is
Speaker 4: how small it is. But they also have NPU optimized
Speaker 4: models which you can go download and there's also an
Speaker 4: extension for vs code.
Speaker 2: Wait wait, go back to the is M or NPU
Speaker 2: and what is that?
Speaker 4: That's the neural processing unit. So you kind of have
Speaker 4: your CPU, your GPU, and your NPU.
Speaker 1: And this was.
Speaker 4: The core the chip, the part of the CPU in
Speaker 4: these ARM devices that really made it easy to run
Speaker 4: these models locally and efficiently.
Speaker 1: Okay, part of the requirement for a copilot plus PCs
Speaker 1: that it has an MPU of at least what is it,
Speaker 1: forty tops or trillion operations per second.
Speaker 2: So if you have a copile plus PC, you can
Speaker 2: download deep Seek and use it even if you don't,
Speaker 2: and you're probably going to get good results.
Speaker 4: Yeah, you don't have to have a NPU, but a
Speaker 4: lot of these models. So Microsoft makes a LM called
Speaker 4: five Silica, and this model they have they've been releasing three,
Speaker 4: three point five, they just released four. It's optimized for
Speaker 4: the CPU and the GPU and not the NPU right now,
Speaker 4: at least the models that they've released, and there are
Speaker 4: models out there that you can get that are optimized
Speaker 4: for the NPU. So if you do have a device
Speaker 4: that is OM device or low power device and you
Speaker 4: want more of an optimized model, you can find them
Speaker 4: and run them. And you can also do that in
Speaker 4: VS code. There's an extension called AI Toolkit for Visual
Speaker 4: Studio Code, and that's another kind of playground esque place,
Speaker 4: but you can also do the model refinement and fine
Speaker 4: tuning in there. So there's a lot of ways that
Speaker 4: you can experiment with these models without really being a pro.
Speaker 4: So if you're just curious and you have a lot
Speaker 4: of hard drive space, that is the one thing that
Speaker 4: I'll say, I recently upgraded my surface hard drive from
Speaker 4: a five to twelve to a two terabyte because these
Speaker 4: models are big and if you want accurate ones, they're
Speaker 4: very large.
Speaker 2: I just saw Richard probably knows about this, but there
Speaker 2: are now twenty two terabyte SSD drives. Yeah, for like
Speaker 2: around five hundred bucks. Can you wrap your mind around that.
Speaker 1: It's a lot of storage.
Speaker 2: Oh my goodness, Like me know, Joe's like is shaking
Speaker 2: his head, like what.
Speaker 4: One drive twenty two terramytes.
Speaker 2: Twenty two terabyte SSD five hundred bucks?
Speaker 4: You should that's not a typeout.
Speaker 2: No, there's a couple of different brands.
Speaker 4: That's amazing.
Speaker 1: Yeah, ridiculous, Yeah, that really is. I think I should.
Speaker 1: I don't think they are sists. I think they're spinning drives.
Speaker 1: Oh really two terabytes? Yeah SSDs the solid state ones
Speaker 1: and aren't that big yet?
Speaker 2: Okay?
Speaker 1: The still twenty two terabytes is madness? Like that's just
Speaker 1: a lot of storage.
Speaker 4: Yeah, it really is. And the AI Toolkit and vs
Speaker 4: code does allow you to interact with these llms through
Speaker 4: the web, and so GitHub will host some of these models,
Speaker 4: other providers will host them, and so you can kind
Speaker 4: of do comparisons. So there's the local foundry, and that's
Speaker 4: what Microsoft has branded there. You know, I've called it,
Speaker 4: I think the second tier kind of where you have
Speaker 4: win mL and you have your local models and you're
Speaker 4: doing that work. So you have your local models and
Speaker 4: you can compare those two cloud hosted models and test
Speaker 4: them because again, you know software, you have to be
Speaker 4: able to test it. So it is hard too with
Speaker 4: these how do you compare them? Like, which one's good,
Speaker 4: which one's bad? Is it good enough? Is it good
Speaker 4: enough in our use cases? And it can be tedious
Speaker 4: to test manually. But there are a lot of tools
Speaker 4: out there to experiment, get started, and if anybody's curious,
Speaker 4: I definitely you should check out the aidev gallery for sure.
Speaker 4: That is a lot of fun to play around with
Speaker 4: those different models and for a little bit more advanced scenarios,
Speaker 4: what more language focused. The AI toolkit in vs code
Speaker 4: is another really fun I'm looking at deep seak here
Speaker 4: right now. You can download it on your device and
Speaker 4: run it.
Speaker 2: Wow, it seems like a pretty good place to take
Speaker 2: a break. So we'll be right back after these very
Speaker 2: important messages.
Speaker 1: Stay tuned.
Speaker 2: You know, dot net six has officially reached the end
Speaker 2: of support and now is the time to upgrade. Dot
Speaker 2: Net eight is well supported on AWS. Learn more at
Speaker 2: aws dot Amazon dot com, slash dot net.
Speaker 1: And we're back. It's don that Rocks emergor Campbell. Let's
Speaker 1: call Franklin. You talking a bit to our friend Joe
Speaker 1: about work with local models and also and the non
Speaker 1: LLM stuff just sort of a good reminder there's been
Speaker 1: all kinds of cool stuff going on in the mL
Speaker 1: space that didn't necessarily have to do with language per se.
Speaker 1: But you know, you've you've hinted this a couple of
Speaker 1: times in the first half. It's like, if you want
Speaker 1: to own the model, you know, there's a lot of
Speaker 1: models available to download from hugey face and all these
Speaker 1: other places. Why would you want to own a model
Speaker 1: because it sounds like a lot of work. It's like
Speaker 1: owning a framework.
Speaker 4: Yeah, yeah, it is like, don't trust somebody who says
Speaker 4: they can write their own language and write their own
Speaker 4: ide You're like, oh.
Speaker 1: Their own garbage collector, you know, their own crypto library.
Speaker 1: Like these are all scary things to me. So when
Speaker 1: someone says I'll just make our own model, I'm like,
Speaker 1: why do we need to do that?
Speaker 4: Well, if you're in the industry. If you have insane
Speaker 4: amounts of data and a niche in a specific industry,
Speaker 4: it might be worth it for you to look into
Speaker 4: doing this. And if you have a hard time processing
Speaker 4: large amounts of data to get insights and actions out
Speaker 4: of it, which is kind of the idea here, right,
Speaker 4: what you have an entire language that you have to
Speaker 4: train these models on, or you have an entire data
Speaker 4: set of images with boxes drawn around the dogs or
Speaker 4: dog breeds or very specific things like that. If that's
Speaker 4: what you need to do, is something where it's not
Speaker 4: available or it's not good enough, there's really no other
Speaker 4: way around it than to build your own model today.
Speaker 4: But it really is that data.
Speaker 1: It's I mean that being said, this is all sort
Speaker 1: of non terministic thing, like you're never going to get
Speaker 1: one hundred percent out of a machine learning model.
Speaker 4: It's probabilistic, right, absolutely, even maybe especially so some of
Speaker 4: the image detection ones, and a lot of times they'll
Speaker 4: give you back a number a fraction of confidence, and
Speaker 4: I think maybe this is why they don't get as
Speaker 4: much play as they're not as exciting for individuals to use.
Speaker 4: It's like the could take a picture of your cat
Speaker 4: and then your phone will draw a box around it
Speaker 4: and say that's a cat. Yep, that's a cat. So
Speaker 4: I think it's a lot less interesting. The language ones
Speaker 4: just kind of capture people's imagination and there's a lot
Speaker 4: more back and forth. But when you really think about
Speaker 4: building an application, like what are you doing? Maybe you
Speaker 4: have a you're playing around with your Raspberry Pie as
Speaker 4: a security system for your house, and you want to
Speaker 4: add a vision system and you want to do box
Speaker 4: detection and you have hours and hours and hours and
Speaker 4: hours of security footage. Or maybe you have a specific
Speaker 4: niche application where you're trying to, you know, detect a
Speaker 4: particular squirrel who's given you trouble. It's a fun you know,
Speaker 4: it's a fun experiment and you.
Speaker 1: Can do a bear or a bear.
Speaker 2: Joe, do you have a toi less squirrel bird feeder?
Speaker 1: No?
Speaker 2: I do not seeing this YouTube? Check YouTube for toil
Speaker 2: less squirrel terrible right. It's basically it goes between you
Speaker 2: know what you hang the bird feeder on and the
Speaker 2: bird feeder, so it's got a hook on either side.
Speaker 2: It detects weight and so when there's a squirrel on it,
Speaker 2: it just starts spinning and the squirrels go flying. It's
Speaker 2: hilarious to whirl the squirrel.
Speaker 4: Yeah, that you could build an AI powered twirl a squirrel.
Speaker 1: There you go, There you go. I don't think that's necessary.
Speaker 1: I am thinking about animal recognition this particular part of
Speaker 1: the world where you know. The one that would be
Speaker 1: tricky that I would really challenge myself would be whale
Speaker 1: detection because we've had you know, you don't have a
Speaker 1: lot of time to pick up on the fact that
Speaker 1: there's whale blow, like they're going by, and it could
Speaker 1: be orcers and it could be humpbacks, and it could
Speaker 1: be grays, and it could be porpoises, and it could
Speaker 1: be dolphins. Like you have to be a lot of
Speaker 1: stuff going on. You have to be on the surface.
Speaker 1: We hear no, no, we hear them like we hear
Speaker 1: whale blow before we see the whale because it travels
Speaker 1: like when they when they exhale its loud.
Speaker 2: Well, you could identify a whale by the sounds it's
Speaker 2: making too.
Speaker 1: Yeah, I wonder. Yeah, speaking of it still seems nuts
Speaker 1: to build your own model like that just seems like
Speaker 1: a thing I don't want to own.
Speaker 4: Yeah, it's it's definitely the research side of things. And
Speaker 4: I know people have been saying for a long time
Speaker 4: that data is the new oil, right, this is the
Speaker 4: new black gold of do you have the data? Do
Speaker 4: you have the databases? Is it structured, is it consistent,
Speaker 4: is it clean? Is it real? Is it good? And
Speaker 4: if you have all that, I think we have a very
Speaker 4: small number of people who can say yes, we have
Speaker 4: that right and you don't have to spend all that
Speaker 4: time cleaning the data, which is such a challenge where
Speaker 4: you have so much noise in the data today that
Speaker 4: if you're trying to train a model, Yeah.
Speaker 2: If how I was going to use a local LM,
Speaker 2: I would want it to understand C sharp, JavaScript, Blazer,
Speaker 2: you know, and CSS. That's and I don't know how
Speaker 2: realistic that is. Like I know that the current models
Speaker 2: like Claude's on it, and you know even chat GPT
Speaker 2: understand it. But for lack of a better word, sorry, Richard,
Speaker 2: didn't mean to offend you. There. They're programmed, you know,
Speaker 2: they're they're trained against it. But what does it take
Speaker 2: to do that locally, to train the models to train well,
Speaker 2: or to get a model that understands you know, programmers
Speaker 2: speak languages and stuff they do.
Speaker 4: Yeah, local models will and they can write code. I
Speaker 4: think part of the challenge that you'll see if you
Speaker 4: start using them is speed. So the response speed of
Speaker 4: a local model is going to be much slower actually
Speaker 4: than a cloud hosted one because your computer cannot compete
Speaker 4: with a server with a rack of GPUs. Yeah, well
Speaker 4: maybe yours, Carl, not mine.
Speaker 2: Oh, I don't know. I don't think so. But you know,
Speaker 2: I think if I had a great Copilot plus PC,
Speaker 2: you know, with a lot of RAM and a lot
Speaker 2: of storage, and I just set it over in a
Speaker 2: closet somewhere, I could probably use that.
Speaker 4: Yeah, you should try it.
Speaker 1: Yeah.
Speaker 4: Another challenge is going to be context, which is how
Speaker 4: big of a context window can the model actually hold
Speaker 4: in the provider there's all of that, there's a lot
Speaker 4: of infrastructure in between the model and actually getting stuff out.
Speaker 4: So speeding context, I would say, are going to be
Speaker 4: your biggest risks where you don't necessarily just want it
Speaker 4: to give you new greenfield CSS. You want it to
Speaker 4: give you new CSS in the right spot for your codings.
Speaker 4: Which is that?
Speaker 1: And I want a much harder question.
Speaker 2: I wanted to remember everything we've said, Like I want
Speaker 2: as big a context as I can possibly get. So
Speaker 2: is that just a measure of more RAM or is
Speaker 2: it the more that context you have, the slower it's
Speaker 2: going to be to come up with a new answer.
Speaker 4: Yeah, that's a good question. I would love to hear
Speaker 4: an expert who actually knows more about context and how
Speaker 4: that differs from the training data and how it differs
Speaker 4: from fine tuning, because in my experiences with local AI,
Speaker 4: I have a pretty narrow context window that you could
Speaker 4: basically feed it, Hey, here's everything I know, and you
Speaker 4: feed it with the prompt yeah, and you say okay,
Speaker 4: now do this and then give it back to me.
Speaker 4: But you're not feeding it documents.
Speaker 1: The thing that's made a difference for me has been
Speaker 1: the video card and the amount of memory in the
Speaker 1: video card, Like playing with frame Pack and a couple
Speaker 1: of other models, and so I'm running a fifty eighty
Speaker 1: with sixteen gigs of v RAM, and that has made
Speaker 1: a huge difference for running bigger models. No, I'm not
Speaker 1: talking about building models, but actually executing a more complex workload.
Speaker 1: And if you have got the money to spend, because
Speaker 1: they're thousands of dollars like those top in RTX cards.
Speaker 1: Now you can get ninety six gigs in them. Jeez,
Speaker 1: it's a ten thousand dollars card. But you know that
Speaker 1: seems to be the thing that makes the most difference
Speaker 1: for a lot of these kinds of tools when you
Speaker 1: want to kindle a lot of contact.
Speaker 2: What about an NPU? Is that gonna do it less
Speaker 2: than more than a ten thousand dollars video card.
Speaker 1: No, because there's just no You know they talk about
Speaker 1: that Copilot plus PC has forty tops. I don't know
Speaker 1: what that means. Yeah, that's the trend trilling operation per second.
Speaker 1: It's the measure of its compute power for neural nets. Okay,
Speaker 1: my fifty eighty has thirteen hundred TOP. I see so.
Speaker 1: And when you look at what Nvidious selling the data
Speaker 1: centers and things, is their giant GPU like that with
Speaker 1: huge amounts of memory, this super fast memory and them
Speaker 1: for scale processing.
Speaker 4: Yeah, the NPU, I think was more of a play
Speaker 4: for a continuous operation or in the background and on
Speaker 4: mobile devices where battery and power consumption is a much
Speaker 4: bigger concern for individuals, where they're thinking, well, I don't
Speaker 4: want this GPU chugging away in the background. Can I
Speaker 4: get something? Can I get something good enough, and that's
Speaker 4: kind of where that minimum bar is that doesn't absolutely
Speaker 4: consume my battery life. You know, you open your computer
Speaker 4: up and it's like, hey, I was working in the
Speaker 4: background seeing if anything was happening.
Speaker 1: No, thank you. Yeah. Yeah. And it's been an argument
Speaker 1: now that you can jack up a PC enough with
Speaker 1: those with a couple of those big GPUs and run
Speaker 1: a mid size LLM on it. So you know, certainly,
Speaker 1: I've had conversations with folks where it's like, I am
Speaker 1: not prepared to send any of this data to the cloud.
Speaker 1: What can I do one hundred percent local? Yeah.
Speaker 4: Another thing that you do have to consider if you're
Speaker 4: going to get into building and those apps are especially
Speaker 4: local apps, is the idea of multi modal. Yeah, these models,
Speaker 4: these local models, at least the Windows aiapis are not multimodel,
Speaker 4: so you will have to.
Speaker 2: In other words, you can't talk to them and write
Speaker 2: to them exactly. Is that what you mean?
Speaker 1: Right?
Speaker 4: So you're going to have to build that. I mean
Speaker 4: you could, but you're going to have to put a
Speaker 4: speech recognition model in front of the LM or a
Speaker 4: object detection model plus an OCR model plus that you know,
Speaker 4: you have to maybe chain these models together and then
Speaker 4: you can get that multimodal experience where you can drop images,
Speaker 4: you can put PDFs in, but you have to be
Speaker 4: able to read the PDF. So these lllms don't read
Speaker 4: PDFs by default locally. You do have to get them
Speaker 4: into a text format. So if you're thinking about how
Speaker 4: you can apply this into your work, and I know
Speaker 4: a lot of enterprises, a lot of companies, a lot
Speaker 4: of their data is not in raw text format, so
Speaker 4: you do have to get it there.
Speaker 1: Yeah, but there's an MCP for PDFs. So you know,
Speaker 1: glue these bits together.
Speaker 4: Right, yeap, but you will have to do the gluing.
Speaker 4: Some assembly required.
Speaker 1: This is the job, right, Like, this is not just
Speaker 1: an app you run, but we are assembling parts to
Speaker 1: try and get to a place where a model could
Speaker 1: be built.
Speaker 2: So if you were going to build a local LLM
Speaker 2: Joe yourself using some existing technology, would you first reach
Speaker 2: for deep seek or would you go for just the
Speaker 2: stuff that Microsoft is exposing in Windows.
Speaker 4: Yeah, I just reach for this stuff or a Microsoft
Speaker 4: is exposing in Windows and their five model. It's pretty good,
Speaker 4: it's pretty robust, and I would say it's a nice
Speaker 4: middle middle ground there for building on top of and
Speaker 4: fine tuning. I don't have enough time to be building
Speaker 4: all these applications and learn the APIs and learning the
Speaker 4: political history of where all these models come from. So
Speaker 4: it is a The benefit of Microsoft as a software
Speaker 4: provider is it's the one throat to choke, right, this
Speaker 4: is the one person you go to. They provide a
Speaker 4: lot of the tooling, they provide a lot of the models.
Speaker 4: Is it the best of any of the world's the
Speaker 4: absolute best. No, But when you're doing a lot of
Speaker 4: different stuff, sometimes you just have to have some heuristics
Speaker 4: here and just make the decision making. There's an infinite
Speaker 4: number of decisions that you have to make when you're
Speaker 4: picking all of these. So starting just with the built
Speaker 4: in tools, the built in APIs, it's a great easy
Speaker 4: way to get started. And if they don't work for you,
Speaker 4: then you can start making other questions and decisions. And yeah,
Speaker 4: but I would say start with the built in stuff
Speaker 4: definitely at first.
Speaker 1: Okay, yeah, here I knew Ivious read I'd read this.
Speaker 1: I just looked at up again. Gptoss is a version
Speaker 1: of GPT three that can be run locally on a
Speaker 1: machine with sixty four gigs around and a fifty to
Speaker 1: ninety with twenty five gigs of v RAM. So that's
Speaker 1: roughly six or seven thousand dollars PC somewhere in that neighborhood,
Speaker 1: depending on how much you pay for the video car.
Speaker 1: The video cards can be driving around it. But that's
Speaker 1: running you know, GPT three, which is what the original
Speaker 1: GitHub copilot was built. Again, Like, that's a pretty torquy,
Speaker 1: pretty good little LM one hundred and twenty billion parameters.
Speaker 1: Like it's not GPT.
Speaker 2: Four, but.
Speaker 1: Especially in a narrow scope application like a NOME set
Speaker 1: of code, that's pretty robust. Man, you could do a
Speaker 1: lot with that.
Speaker 4: Yeah, you could do a lot with that. And also
Speaker 4: you have to consider the big question of why would
Speaker 4: you build local ever, you know, why do it at all?
Speaker 4: Obviously privacy is a concern for a lot of people
Speaker 4: of why would you do this stuff locally on your
Speaker 4: own computer? If you have network concerns, if you don't
Speaker 4: have reliable or high quality or high speed internet, then
Speaker 4: obviously this is the only solution for you. But then
Speaker 4: also there's the cost concern and the cost question of yeah,
Speaker 4: you don't necessarily want to make some code that runs
Speaker 4: out and is running all these llms, and then you
Speaker 4: come back with a bill for you know, thousands of
Speaker 4: tens of thousands of dollars because your credits went crazy. Right,
Speaker 4: But when you have it local again, try There's so
Speaker 4: many cool tools, the AIDEV gallery, the AI toolkit, and
Speaker 4: then there's the APIs available already today. There's so many
Speaker 4: ways to get started and try and see. I you know,
Speaker 4: what is your application, what could it be? Try it
Speaker 4: out because you might not have to sign up get
Speaker 4: an API key at all. You could do all this
Speaker 4: stuff locally. And then if you want to do batch
Speaker 4: processing of again your own data, maybe you want to
Speaker 4: kind of use these models to put the data into
Speaker 4: a particular shape or clean it or work through it.
Speaker 4: But you don't want to pay tokens to do all
Speaker 4: that work. Well, do it locally, do it overnight. Build
Speaker 4: an app, your own app, not something you ship necessarily,
Speaker 4: but do it locally, you know, process that data locally,
Speaker 4: and then go from there. Maybe you're going to build
Speaker 4: your model, but first you have to get all the
Speaker 4: data in the right shape.
Speaker 1: Right, and and you're trading time for money right right.
Speaker 1: Essentially the game you're playing here. It's like, Okay, if
Speaker 1: I run it on the cloud, it's going to cost
Speaker 1: me more, but I get it done less time, or
Speaker 1: I'm restricted to my own hardware so it may take longer.
Speaker 1: And then you start, you know, doing the economics. So
Speaker 1: just looking up the high end. Yeah, the ninety six
Speaker 1: gig in Nvidia RTX pro six thousand Blackwell, that's the
Speaker 1: big Box twelve thousands.
Speaker 2: Well, you know, it's not only the money, but as
Speaker 2: Joe said, the security and the privacy that may trump
Speaker 2: any kind of money, and you know, and that may
Speaker 2: be the requirement you know.
Speaker 1: Sorry that was Canadian dollars, just nine thousand Americans.
Speaker 4: Ah, well that totally changes.
Speaker 1: Yeah, everything's different. Now he just saved me two thousand
Speaker 1: three grand grand. But again, if I'm playing that game
Speaker 1: of the cost benefit, like what am I spending on
Speaker 1: tokens at that scale? True? And I really get the
Speaker 1: sense that as this sort of bubble starts to burst
Speaker 1: and people need to make money, like tokens ain't getting
Speaker 1: cheaper nowp.
Speaker 4: Yeah, I have been using Claude and Codex and Copilot.
Speaker 4: There's definitely times where I have three computers running and
Speaker 4: they're I'm just kind of like telling them to keep
Speaker 4: going over. They're checking and building, but it's never going
Speaker 4: to be cheaper than it is now, Like this is
Speaker 4: the cheapest is going to be. They're trying to get
Speaker 4: as many users as possible, but that floor has to rise.
Speaker 4: I mean, I know Anthropic was having some issues a
Speaker 4: couple of weeks ago with limits and quality, and Codex
Speaker 4: I think had something a month or so ago where
Speaker 4: the limits. And again, if you're relying on these cloud services,
Speaker 4: not only are you relying on them to stay up
Speaker 4: and your connection to them to say live, but you're
Speaker 4: also relying on the model and the pricing and the
Speaker 4: availability at all from a business to point for them
Speaker 4: to stay up. Because it might make sense today, I was.
Speaker 1: Talking to some folks abroad that are big, like running
Speaker 1: five sixty seven simultaneous instances because they're working that fast
Speaker 1: right tuned models, reaching these things, and they said that
Speaker 1: over July fourth everything got dramatically faster, like they got
Speaker 1: a ton of work going July fourth because Americans weren't
Speaker 1: working like these, like these cloud infrastructures are stressed to
Speaker 1: the limit and slowing performance as it is right and
Speaker 1: say and the only and the proof we've had is
Speaker 1: like when the stress isn't a high, things are better.
Speaker 1: So there is this interesting argument about at what point
Speaker 1: does this make more sense to be local versus remote?
Speaker 1: And this is going to be a shared resource too,
Speaker 1: like these big boxes don't have to be per dev
Speaker 1: They could be shared out again with potential performance issues
Speaker 1: like well, of course, I'm such a hardware geek, like
Speaker 1: I'd love to build out a rack of this stuff.
Speaker 2: It would be fun, wouldn't it.
Speaker 1: It would be and you know, and then now I've
Speaker 1: got the heat and power problems right.
Speaker 4: To live it firsthand. Well to your point about shared resources,
Speaker 4: that is one of the nice things about win mL
Speaker 4: that just released Execution Provider that Microsoft announced making it
Speaker 4: easier for local devs to integrate models is if you
Speaker 4: have an application and you need a model, do you
Speaker 4: download it? And then every single one of your applications
Speaker 4: is downloading a five gig LM. Yeah, obviously that becomes
Speaker 4: untenable very quickly unless you have that twenty two terabyte
Speaker 4: drive in your computer. Solow you yeah, yeah, more than one.
Speaker 4: It does allow you to share models across application rich
Speaker 4: so you can have one machine install.
Speaker 2: Richard, you were right. I thought they were SSDs.
Speaker 1: They're not. They're HDDs ds. There are a few SSDs
Speaker 1: over eight terabytes, but most of them the line seems
Speaker 1: to be eight. By the way, the RTX six thousand,
Speaker 1: six hundred watts each.
Speaker 2: That's why I have solar panels.
Speaker 1: Yeah, that's it, you know, like oil BOYD. I'm just
Speaker 1: thinking about how much you remember in the end, this
Speaker 1: is moving electrons around and generating heat like you just
Speaker 1: made rocks make heat. Like that's saying time watts. You're
Speaker 1: gonna feel it. You don't want to sit in the
Speaker 1: room with that thing only man, No, it's going to
Speaker 1: be crazy. But it is an interesting point of view
Speaker 1: as we're still going through this to say, what are
Speaker 1: we going to shift local? What are we going to
Speaker 1: run remote? Like, what's feasible at what makes sense for
Speaker 1: folks here? And I think, you know, not everything has
Speaker 1: to be cloud and not everybody wants it there.
Speaker 4: Right, And I think you just have to be you
Speaker 4: know wide. I'm not saying to get super deep on
Speaker 4: all of this stuff, but the tools for you to
Speaker 4: get your feet wet are available, and when you're CTO
Speaker 4: or more probably more likely, your CFO comes to you
Speaker 4: and says, hey, we can't afford this bill anymore. Your
Speaker 4: critical application can't use this LLM. You have to stop,
Speaker 4: or you have to change something because either somebody's prices
Speaker 4: went up or the business model changed.
Speaker 1: Yeah, what are you going to do?
Speaker 4: What are you going to reach for? And getting your
Speaker 4: feet wet in some of these local models, it's a
Speaker 4: great way to have an answer or have some sort
Speaker 4: of solution or see if that solution will work.
Speaker 1: Now you're swapping op X for CAPEX and then, you know,
Speaker 1: using CFO speak like, we have two ways to solve
Speaker 1: this problem. We spend month over month on it, or
Speaker 1: we made a capital investment and spend less. You know,
Speaker 1: let's do the math. You know, if you want to
Speaker 1: talk to CFO, bring a spreadsheet.
Speaker 4: Yeah, exactly. And it's as as we've said, as you've said,
Speaker 4: stuff is changing so fast. So if you get super deep,
Speaker 4: if you start training your own model, and then tomorrow
Speaker 4: somebody comes out with a model that just makes all
Speaker 4: that effort useless. This is again, this is like the
Speaker 4: sweet spot, right, Isn't this where the Windows developer has
Speaker 4: kind of always loved to live where they're like, yeah, yeah. Yeah,
Speaker 4: we're not like hardware level, we're not doing machine code.
Speaker 4: But then we're also not just like bleeding like the
Speaker 4: best of the best. It's like, okay, we're in the
Speaker 4: middle here where we got models, we got a local
Speaker 4: It's it's efficient, it's it's a good balance.
Speaker 1: Yeah. Well, and I'm going to call back to Cagle
Speaker 1: again because one of the other ways you can get
Speaker 1: a model built is to put out a bounty on
Speaker 1: Cagle in a competition to have someone build it for you. Effectively,
Speaker 1: there you go. So you've got the data set, but
Speaker 1: you don't want to actually do the construction. You can
Speaker 1: host a competition and define your problem space and provide
Speaker 1: the sample data, and a bunch of people compete for
Speaker 1: to deliver you the best model. It's a weird world, man,
Speaker 1: is like, if you want to go deep into mL,
Speaker 1: there's so many interesting things to be done here. M hmmm.
Speaker 2: I had the weird meta thought that you could get
Speaker 2: a model to build your model instead of you know,
Speaker 2: farming it out for a bounty.
Speaker 1: Well, you're not wrong to interact with an LM to
Speaker 1: start constructing a plan around how a model would get built,
Speaker 1: because that you know, in the end, they are a
Speaker 1: pretty clever search tool for best practices.
Speaker 4: Yeah, search and tokenization is a really nice thing that
Speaker 4: you can do with your local LM of crunching some
Speaker 4: of your data, your text, tokenize it make it easier
Speaker 4: to search, have that more natural language available for your users.
Speaker 4: It's a really hard thing to code, but if you
Speaker 4: have local l MS, I can help you build that.
Speaker 1: Why not. Yeah, that's cool.
Speaker 2: Anything else on your mind that you want to touch
Speaker 2: on before we call it a show?
Speaker 4: Not really, I mean we touched on a lot here. Yeah,
Speaker 4: we just try it.
Speaker 1: We went we went on a ride today friend again. Yeah,
Speaker 1: but this is the kind of deep.
Speaker 2: Dive into local lms and local AI that I really
Speaker 2: wanted to get to. So I'm very very happy we talked.
Speaker 2: Thank you, Joe.
Speaker 4: Yeah, happy to be here.
Speaker 2: I'm right and we'll.
Speaker 5: Talk to you next time on dot net rocks.
Speaker 2: Dot net rocks is brought to you by Franklin's Net
Speaker 2: and produced by Pop Studios, a full service audio, video
Speaker 2: and post production facility located physically in New London, Connecticut,
Speaker 2: and of course in the cloud online at pwop dot com.
Speaker 2: Visit our website at d O T N E t
Speaker 2: r o c k S dot com for RSS feeds, downloads,
Speaker 2: mobile apps, comments, and access to the full archives going
Speaker 2: back to show number one, recorded in September two thousand
Speaker 2: and two. And make sure you check out our sponsors.
Speaker 2: They keep us in business. Now go write some code,
Speaker 2: See you next time.
Speaker 4: You got jas.