Speaker 1: How'd you like to listen to dot net rocks with
no ads? Easy? Become a patron for just five dollars
a month. You get access to a private RSS feed
where all the shows have no ads. Twenty dollars a month.
We'll get you that and a special dot NetRocks patron mug.
Sign up now at Patreon dot dot NetRocks dot com.
Speaker 2: Hey guess what. It's dot net Rocks Steve Sanderson Edition.
I'm Carl Franklin, the amateur Campbell And uh, what can
we say? Richard New Year?
Speaker 3: We're you're making a show and checking it twice? Oh wait, no,
Christmas is over? Yeah?
Speaker 2: Uh all right, Well we got a lot of sort
of pre show ramble to do, so let's get right
into a better no framework awesome. Well I don't have
a framework piece or anything per se, but I do
have some news about Blazer Train and Blazer Puzzle. So
Blazer Train I have done sparsely since we started the
Blazer Puzzle, just focusing on redoing old things. But I've
started a series on Blazer Train now that's going to
be weekly or every other week on What's New and
Dot at nine. So the first one is published maybe
even the second one by the time this comes out,
and the Blazer Puzzle is going to be a little
different than usual. Typically, what we've done with the Blazer
puzzles we give a problem and we say, okay, take
the week or a few days and find a solution
and email us the solution. Then we pick a winner
from all the right answers. And we're not doing that anymore.
We're going to give you a problem and then we're
going to take a little pause, give people a chance
to duck out or press the pause button on the video,
and then we're going to give the solution. So no
more winners, no more mugs, no more don't post this
on social media. We want you to engage with us more.
So it's kind of fun. It's more fun.
Speaker 3: That's cool, man.
Speaker 2: Yeah, So that's what I got. Just some updates from
my Blazer videos. And also we should probably just briefly
go over some things that happened in nineteen thirty six.
Speaker 3: This is episode ninety thirty six.
Speaker 2: Yeah, since it's episode nineteen thirty six and I want
to be labor stuff and we don't want to take
too much time. But there are a few important things
that happened. Of course, you know, Nazi Germany just kind
of ratcheted up in general. So there's a whole bunch
of things. But on February fourth, Keynes John Maynard Keynes
published The General Theory of Employment Interest in Money, a
revolutionary economic work that fundamentally transformed modern modern macroeconomic thought
and policymaking. Everybody knows about Keynesy and economics now. On
February nineteenth, Marion Anderson performed at the White House, significant
in the context of racial segregation, would later be remembered
as a pivotal, pivotal moment in the civil rights movement.
March first, the first p seventeen bomber was delivered on
March seventeenth, Saint Patrick's Day Flood in Pittsburgh, Terrible, terrible flood.
April thirteenth. Hitler appears on Time magazine cover. May seventh,
Italian annexation of Ethiopia, and they continued to do a
lot of damage in Ethiopia, the Italian military in On
May twenty eighth, Alan Turing submitted the groundbreaking paper on computability.
And he's a hero among us developers, of course, in
mathematicians everywhere, and you know, do you have anything else
you want to add? I mean, there's tons of things
that happened, but sure, well you know, I stick towards
the Zion stuff. So in nineteen thirty six was the
year that the last thylacine died, also known as the
Tasmanian tiger. Although it was more wolf like than anything else.
It had been hunted because they believed it had been
hunted for bounty because they believed it was killing sheep.
Turned out to not be true. The last one died
of neglect in the Tasmanian zoo. This is strictly a
Tasmanian animal, and coincidentally, there is an organization working hard
to de extinct the thylaccene. Wow like Jurassic Park style or.
Speaker 3: Well, no, his Jurassic Park is fiction.
Speaker 2: No, no, of course, but you know what I mean
to cloning DNA.
Speaker 3: But in Jurassic Park's approach was to find DNA of
the dinosaurs and then fill in the gaps of amphibian DNA,
which is you know, again it's kind of fictional. What
they're actually doing, these folks are doing, is they are
finding the closest living genetic relative because they have good
genetic material for the thylacine, and then they are changing
that DNA to match the thylacine. Okay, but they're also
working closely with Tasmani itself, like if they're going to
does it make sense to reinduce it all that sort
of thing, and there's a strong will to bring that
animal back. It'll actually helps stabilize the ecosystem in Tasmania.
Now that the thylacine has been gone for some ninety years,
Tasmani has been stuffing air effects for not having a
top tier predator, and so there's an idea that bringing
back the thylascene would help.
Speaker 2: See you get a mini geek out in every dot
net Rocks show.
Speaker 3: There you go. I mean, I've been keeping notes on
de extinction for the mammoth and a few others and
so forth. The thilastine has been one of them, and
it just happens to be that nineteen thirty six was
the year that the last thyloscene died.
Speaker 2: Wow, well, who's talking to us today?
Speaker 3: Richard Graudy comment off of show nineteen twenty two, the
one we did with our friend Dan Roth and the
Fall of twenty four talking about Blazer and dot net nine.
And this comment seemed particularly relevant to our conversation today
because it's from Brent from Atlanta, who said it was Grelway.
Was great to hear from Dan. He binds me of
a brilliant co worker of mine who just retired from
Mongo dB. I'd love to know if coding llm's running
locally like the Quinn two point five coder, will be
easily integrated individual studio. I'm looking to enhance my Blazer
coding with such tools without having to pay for Copilot,
which is preferred to be in verie to Quinn. Anyway,
I love the show. Well, hey get out copilot now free,
So so much for paying for that. B All the
copilots now have options for different back ends you can
run on, so you know, the default is the open
AI one, but if you want to switch to a
LAMA or cloud or anything like that, you can do that. Yeah.
Absolutely so there.
Speaker 2: Just as in a side I was talking to the
guys at dev Express and they have a control now
that tracks all of that stuff away and just gives
you like an input box and you can connect it
to wherever you want without any kind of nasty pain.
Speaker 3: Nice. Yeah, so yeah, all good things happening, good things,
and so Brent, thank you so much for your comment.
Copy of music Cobuy is on its way to you.
And if you'd like copy of music Cooba, I write
a comment on the website at dotn at Rocks dot
com or on the facebooks. We publish every show there
and if you comment there, we're read it on the show.
We'll send you copy of music Goba.
Speaker 2: And you can follow us on other social media. We've
been on ex Twitter forever. We're on mastadon and on
blue Sky, and I believe Richard's also on threads. I
have a Thread's account, but I don't watch it, and
you know, but you can find us out there.
Speaker 3: Yep, we're out there. Blue Sky seems to do the
fun one of these days.
Speaker 2: Yeah, just some variation of at Carl Franklin at rich Campbell.
So let's bring back our friend Steve Sanderson. This is
a very humble bio that Steve wrote for us. He's
a developer on the dot net team at Microsoft. He
focused as I'm making dot net better for application developers
with a focus on AI and the web. Yeah, he's
done a few things, a couple of things here and there,
not really mentioning in the bio.
Speaker 4: Welcome back, Steve, Hello, thank you for bringing me back.
Speaker 2: Good to have you.
Speaker 3: Yeah, absolutely, man, And you know I always lead this
off the same way. What are you working on?
Speaker 4: What am I working on?
Speaker 3: Well?
Speaker 4: I think, like quite a lot of us, I've started
to focus on the AI side of software.
Speaker 2: Oh wow, recently.
Speaker 4: Yeah. I think it's a really big opportunity, and there's
many different ways you can think about what it is.
Like on one level, it's like a tool that can
help you write your code, or it's just like a
chat interface or things like that. I think all those
things are fine, but that's not what I'm personally drawn
to as like an area where we can exploit opportunities.
Maybe it's my background, but I feel really drawn to
doing things around applications and doing things for application developers,
and I think about the you know, the many many
people I know who working companies that are producing software
for their customers or for their employees, and things like
how do you actually use AI within that to make
better software? How can you make your software have better
features than it would have had before?
Speaker 3: Well, and when you were thinking about that, like I
remember you thinking about trying to make it easier to
put data on web pages, and that made knockout. Yeah. So,
and then you were thinking about how to make it
easier to use c sharp on web pages, and that
made Blazer. So I kind of like it when you're
thinking about things, Steve, good things happening.
Speaker 2: I remember last year you did a demo at one
of the Dona comps or something like that. You did
a demo of some AI in AI chat wired up
to data. Yeah, it was in Blazer and it was
just mind blowing. So you've been doing it for a while.
Speaker 4: Yeah, that was our e shop support demo. I'm sure
we can add a link to that. Yeah. That was
some exploration around this of what you get when you
put Blazer and AI and typical application business app scenarios together.
And that's an exploration of like a customer support scenario
where people are submitting inquiries and there's some AI that
automatically classifies them as different types of thing, and then
the staff who's trying to deal with customer support gets
like a chat assistant that's able to look through corporate
or enterprise data to provide suggested answers. Fills it out,
you know, rates the customer's satisfaction automatically, all that kind
of state thing, and it just represents a whole own
of scenarios that we think people would probably actually have.
Speaker 2: Yeah, I'm looking forward to the day when I can
replace my report criteria selection UI with just a chat box. Yeah,
and you probably can do that already and I can actually,
but you know, it's going to take a little bit
of a paradigm shift for users to want to go
for that perhaps, But you know, nobody's saying you can't
offer it as an option.
Speaker 4: Yeah, totally. And you mentioned chat as an interface there.
I think chat is clearly something that people strongly associate
with AI, and it's a thing that we didn't really
have as a form of UI before AI came along,
and so that's kind of exciting and cool. But I
also whenever I talk about this, I really try to
emphasize the fact that chat is not the only way
to benefit from AI, And personally, in a lot of cases,
I don't actually like typing into chat boxes like it.
When I'm using my computer to do something productive, my
mind is focused on the task and I don't really
want to be like explaining stuff in English to the computer. Yeah,
and so I think a lot of the time when
we think about how we can add AI to our
applications and produce genuinely more productive apps, it's about using
AI on the back end just sort of like almost
like secretly, without involving the user. It's just predicting what
they want and doing parts of the job for them
ahead of the time so they don't have to do it.
Speaker 2: It does bring to mind a couple of really good
text interfaces where you're writing and then things are helping you.
Obviously get ub copilot and go back to intelligence and
statement completion and see sharp or even writing queries in
SQL management studio, you know when you say select star
from and you get dropped down. You know, you could
implement that in a chatbot where you could say I
want to see all the reports of type and then
you know, transaction type, whatever, whatever type of reports. You know,
those things will probably come in the in the AI
front end chat boxes.
Speaker 4: I'm sure they will. Yeah. And another good example of
that is, of course, as you already mentioned, visual studio Copilot.
And one thing that I really love about that is
the fact that I don't have to explain to it
what I want it to do, Like it's there while
I'm writing code and producing sensible suggestions for me without
me stopping to say I need to do X. It's
just you know, inferring that from what I've already typed
in code, which is I've.
Speaker 3: Seen this happen in Excel. I don't know if you've
played with Excel lately if you have that stuff turned on.
But if if you're doing a pattern of entry into
a series of a column of cells, it starts pre
populating it with the next one it thinks you want,
and it's I mean, the easy one is like one, two, three,
but you know, first of each month, it's figured that
one out, Like you're exactly right. It's not doing it
for you. You're not telling it what to do. It's
just looking at what you're doing and then shortcutting instead
of me having to type it out. Now I'm just
like tab yep, I'll keep that, tah that and so forth.
Like that's really subtle and clever.
Speaker 2: And it doesn't have a lobotomy like Clippy. But it's
the you know, tried to it's trying to solve the
same problem. Really, I mean, it's the whole idea of clippy,
you know, without a lobotomy.
Speaker 3: Well, when we've had smart populating and forms and things
like our browsers know our addresses and so if you
looks like you're starting a type of address, it finds
the rest of the boxes and fills them all in
for you. Like that kind of thing, and actually address
fill in is one of those ones I think people
have worked really hard on where it's like, hey, when
you seeing your postcode, okay, well now I know your
city in state right, Like.
Speaker 2: The country and postcode sort of reverse.
Speaker 3: The thing up. But now I start thinking about forms
over data, Like I'm really interested in the machine learning
models that study a user's workflow and start to tailor
their behavior to that user. Like if you do the
same thing every day in this app, how do I
make it quicker for you?
Speaker 4: Yeah, that's an interesting area. People have started speculating like
about really far off futuristic things which are interesting. I
don't know what the sort of short term thing is here,
Like there's already products and demos out there of AI
systems that generate a UI on demand, so you know,
while a user is doing some task, there isn't a
pre defined set of textboxers, a pre defined set of
wizard steps or whatever. But some AI is being asked
to make up a UI on the fly based on
what the user appears to be doing. That's tailored towards
that task, And I think that's pretty much still a
sort of proof of concept level right now. I doubt
that anyone's completely relying on that to ship and app with,
but it's conceivable that, you know, five years from now,
the way we interact with computers will be much more
sort of generated on demand for us rather than pre defined.
Speaker 2: Is that one of a good example of what you
mean by AI working on the back end without you know,
you really noticing it all that much, or were you
thinking of something else.
Speaker 4: I was thinking a much simpler thing. So the dynamically
make up a UI as the user is working is
like super sci fi futuristic stuff, which by which I
mean like it's not going to be several months away.
Speaker 3: But like you know with the pets of AI.
Speaker 4: But no, what I meant was something like really straightforward
stuff that you can code up in in ten minutes.
So the kind of like core features that you can
do with AI, stuff like being able to automatically classify text,
automatically extracting structured data from fuzzy inputs, whether that's plain
text or images, ability to summarize, translate, ability to detect
anomalies in data, the ability to score user sentiment. All
these kinds of things you can do programmatically on the
back end of your application and then use that to
you know, trigger workflows automatically. Like some message comes in
and it appears to be a complaint, Well you can automatically,
you know, send a message to the person who deals
with complaints or something like that. Or you get something
that you know, you get a large amount of text
in from something and you can summarize it down to
one sentence to display as like the heading within a
table or something like that. You know, there's so many
ways that you can use AI systems programmatically to make
your application just feel that bit more intelligent.
Speaker 2: It sounds like pattern matching on steroids.
Speaker 4: Yeah, in in some ways, you.
Speaker 2: Know the things that we loved pattern matching for.
Speaker 4: Well, arguably at the lowest level, that's what language models are.
Speaker 3: Yeah. Yeah, it is all pattern managing matching tokens effectively, but.
Speaker 2: It's not stuff that you necessarily have to do in
raw code. You know, absolutely the language has a pattern
matching feature and you're using that.
Speaker 4: Yeah. Yeah, there are tons of application features that people
could be using quite easily and cheaply, especially in terms
of the amount of developer effort. Something like semantic search,
for example, Like the naming makes it sound kind of
fancy and advanced or something like that, but it's really not.
You know, you could you can take any search feature
that you've got today and upgrade it to be semantic search,
like within an afternoon, and at least at a sort
of prototype level, and see whether or not that's going
to benefit your application, so that users don't need to
type things correctly and they don't need to phrase things
in a particular way, but it just sort of smartly
works out what they're trying to search for.
Speaker 3: Well. Semantic search to me seems like one of those
buzzword phrases of a promise of a better future, right,
I mean, we've been arguing about semantic search since the
web was invented, right. Even Tim Berner's lead was like,
this is what this is all about. It's just a
question of or we finally got and we finally had
a set of tools that can do it consistently take
the semantical information around it and say you make a
more informed search.
Speaker 4: Well, I personally think that when it comes to just
using embedding models, which are one of the two main
types of language models that people use today, to take
things like document titles or text from documents, convert them
into embeddings and then be able to search through those
in a way that matches on meaning rather than exact
string matches is a very straightforward thing to achieve, Like,
it's a really well trodden path by now, We've got
so many cheap ways of doing it that I think
that most people who have got any kind of application
and do as any kind of search feature ought to
be thinking about doing that.
Speaker 2: We think about it this way, like you make a
document a PDF document of rules that your software must follow, right,
and then just set the AI on it and say, Okay,
don't let me break the rules in my software. Okay,
there's a there's a sci fi for you.
Speaker 4: Yeah, it's it's a scary thing. So, I mean, AI
has so much promise, right, But at the same time,
it also kind of violates many of the assumptions that
we've had about how software should work, like, particularly when
you talk about rules that can't be violated or anything
to do with security. You know, as soon as you
bring AI into it and have a language model making
decisions about what a user kind of can't do, everything
becomes so fuzzy. Like you tell the language model a
certain user should or shouldn't be allowed to do something,
but if that user says that they want to do
it persuasively enough, then it might just let them do
it anyway.
Speaker 3: Persuading software, I said in air quotes.
Speaker 2: But what I would do in that situation is I
would want a tool that says, Okay, here's the document
of rules and business rules. Right, not necessarily you know,
thou shalt not or whatever, but and then say, and
here's my code and here's my rules. Now generate some
code from me that I can where I can plug
the holes. Yeah, okay, you know, and then I can
look at I can do that once, and then I
can look at that code those things and see if
they if that's a good idea to implement them. But
you know, that's how I want to use AI. I
wanted to either generate some code or give me some
suggestions and they'll let me write the final you know,
do the final cut and paste or whatever and testing.
Speaker 4: Yeah, I think that's reasonable. So that's using GAI as
a coding assistant, which is definitely a very very valuable
part of the whole thing. And a lot of people
are doing that and that's great. There are limitations to that,
you know, one of them is ultimately it might still
get it wrong and you may or may not actually
spark the fact that it's got it wrong. Like, there
could still be bugs, especially subtle security bugs. It's not magic.
And secondly, if you're constrained to only using GAI at
development time and not at run time, then that constrains
the sorts of features that you can ship. So I
don't think it's like the ultimate or only solution, but
it's definitely a valuable and valid way of thinking about
things in a lot of cases.
Speaker 2: Right, And I you know, just as an example, what
I talked about is just an evolution of what we're
already doing now with you know, Copilot and all of
that stuff. Yeah, so I can see those things happening,
you know, more discrete tools for helping us shore up
our code. Static code analysis another great way to use AI.
But yeah, I would, I would be skeptical to Here's
here's something that Jeff Fritz and I were thinking about,
and it's not necessarily AI, but it's calling translation services
on the fly so that somebody you don't have to
actually do different language versions of your application. You could
just you know, you're in Japan, you bring up everything
in Japanese and it goes and does a translation. But
now you're relying on that translation to be accurate in
real time. Not only that, but you're racking up all
the API calls that you're doing. So another idea would
be to just take a tool that would generate all
the text and all the resources in every language and
then boom, now you've got some Now you can actually
test it.
Speaker 4: And I'm sure and a lot of people are doing
that absolutely.
Speaker 3: Yeah. Yeah. The language tokenisation strategy that llm's brought to
the table made translation.
Speaker 2: A lot easier, yeah, and more accurate.
Speaker 3: And it does seem to be more accurate as well.
Speaker 4: Yeah. Well, modern large language models entirely come out of
machine translation. That's where this whole thing started, you know,
building mL models that could translate between human languages, and
the way that that was very successful through transformers became
you know what we have today.
Speaker 3: Sort of the key ingredient to what began the path
towards GPT was it's the translation problem first, and it's
just yeah, it's one of its superpowers. And what I
like about it is, for the most part, we haven't noticed.
It's just that every all translations seem to be better
and cheaper and more prevalent, just showing up in more places.
Speaker 1: Yeah.
Speaker 3: Absolute.
Speaker 4: In fact, it was just today that I don't know
if you saw VLC the media player. The oper was
media Player as a shipped to feature this morning, which
adds automatic subtitles to what you're watching, is controversial because
people are not sure if the quality is good enough.
But yeah, you're right, it is showing up all over
the place.
Speaker 2: So how likely are you to use a chat bot
on a website that pretends to be a representative of
the company. Oh?
Speaker 4: Usually, I mean I'm usually my general feeling when encountering
something like that is oh no, what have they done?
But I did have one example of a time quite
recently where I had to use a chat butt and
it actually worked, and I was amazed, Wow, I had
made a booking with like Expedia or something. I think
it was Expedia, and I couldn't find my booking number anywhere.
Like I went to every single email, there was no
booking number in anything I could find. I tried to
contact customer services. It was AI. I thought, oh, this
is stupid, this is never going to work. And I
asked it the question and it said, like what's your
name and where you're going and stuff, and then it
just immediately came back and said, this is your booking number,
and it was awesome and it was correct. Yeah of course,
yeah wow.
Speaker 3: And from there, like you then searched on that or
you know, opened something up like validated it elsewhere.
Speaker 4: Yeah, well it took it gave me a link to
something that I hadn't been able to find previously.
Speaker 2: Do you do you think that chat bots should be
required to tell you that they are a bot and
you're not actually speaking to a real person.
Speaker 4: I don't know. I mean within context, it can be
really obvious. Like if if it's literally a chat you
I and there's a little icon of a robot and
it says like assistant bot is his name, then you're
not going to think it's a person. Yeah, it would
be a terrible person.
Speaker 2: I've asked bots before and they have said, no, I'm
not a bot, but I knew it was.
Speaker 3: And you asked him to respond to you and iamic pentameter.
Speaker 4: Yeah, and then you do the the voidkamp test or something.
Speaker 2: Is that Okay, well I'm interested in that, but we
probably shouldn't go down that rabbit.
Speaker 3: Yeah, definitely trouble. I am thinking in terms of how
we use generative AI models to understand the user's interaction
with the app. I've been doing some work with some
neuroscience companies and they do a lot of additional instrumentation
on people. And one of the things they've said is like,
facial expressions are not a good way to measure someone's
response to things, and they're very cultural, they're very personal
individuals and so forth. But if you take that camera
and zoom in on a point of skin close enough,
you can actually see heart rate, and heart rate is neutral.
Like the fact that now you can measure that person's
heart rate relative to one step to another inside of
a piece of software and say is a heart rate
increasing or decreasing and that is a better indicator of
frustration or But.
Speaker 2: You have to have a baseline though you have to
know it when they're not excited.
Speaker 3: Yeah, yeah, I think it's very personal, right that you
could put someone through some paces and their cases. They
do initial testing for a kind of surveying, and so
they sort of look at what your measurements look like
on that and then go into more important ones and
see right through reaction.
Speaker 4: Yeah, well that's interesting. It possibly gets us into some
ethical and maybe even legal issues. Like so, you know
a lot of recruitment takes place through AI these days,
and people are often when applying for jobs as to
do video interviews that kind of like an AI video
AI assessed video interview. You know, you could it would
be really straightforward then to be using a technique like
that to try and assess do we think this person's
really being honest about what they're saying on that? But
if you're going to make decisions about someone on that basis,
then you know, as a software developer, engineer and manager
thinking about implement feature like that, you've got to think
very carefully about how reliable you can make this and
what the ethics of doing that really are.
Speaker 2: Because humans have trigger words, right that may not have
anything to do with what you're talking about But you
use a word or something and some of you know,
the heart rate will go up and they'll start sweating.
But has nothing to do with the context.
Speaker 3: Well, we've all interviewed developers. It turns out that doing
being interviewed as a developer increases your heart rate. Yeah,
they're not comfortable, the vast majority of the market. It's
almost strange to find someone who is comfortable.
Speaker 4: Yeah, that's even more suspicious.
Speaker 3: Yeah, what's wrong with that, psychopath? Why are you happy
being interviewed? There's something wrong here.
Speaker 2: There's no laughing in development, laughing in interviews.
Speaker 3: Nothing's fun. This is not fun.
Speaker 2: We'll be right back with more with Steve Sanderson after this.
Did you know that you can work with AWS directly
from your ide AWS provides toolkits for visual studio, visual studio, code,
and jet brains rider Learn more at AWS dot Amazon
dot com, slash net slash tools. And we're back. It's
dot net Rocks. I'm Carl Franklin, that's Richard Campbell hey,
and that is mister Blazer himself, Steve Sanderson. We're talking
about AI, his latest passion. We kind of stopped and uh,
let's open up the floor for a new topic. What's
what's something else that you're thinking about AI wise.
Speaker 4: Steve Well, I guess the main area is what do
we actually do practically in order to give better tools
and capabilities to dot net developers. You know, I'm on
the dot net team. That's that's what my jobs. And
so the main thing that we're looking at doing at
the moment is producing a new set of standard abstractions
for AI features in dot Net. This is a pattern
that you know we've we've used in all sorts of
other things. So you know, we've got packages like Microsoft
Extensions Dependency Injection. You'll know we've got Microsoft Extensions Logging
and all sorts of other Microsoft Extensions things that contain
standard representations of common features. So we're following that same pattern.
We're going to have already shipped a preview of a
package called Microsoft Extensions AI, and that contains some new
standard representations for things like language models which we call
i CHET client and embedding models or I Embedding Generator
as we call it, and all kinds of other standard
types and helpers that you can use when you want
to add some AI features into your application. And then
there are implementations for all of these with many different
implementations provided. So we've got the obvious ones like open
AI as your open AI. We've got ones for Olama,
there's ones for Gemini, ones for Anthropic. You know, all
kinds of different implementations.
Speaker 2: So both cloudy and local implementations of LLMS.
Speaker 4: YEP absolutely and then all from us as well. The
intention is that this is a shared community effort, so
we've been working with external projects like the Alama Sharp
project and others to make sure that they've all got
implementations of these same interfaces. And the payoff for all
this for dot net developers is that you have one
common program model that you can use whichever AI back
end you want to work with. And it's hopefully quite
a well thought out one because you know, we've had
people dedicated to thinking about this for months and it's
very flexible and you can plug in your own behaviors
at different stages in the pipeline and combine things and
you know, do all the stuff that you will want
to do, and yeah, we want to be able to
build an ecosystem around that.
Speaker 2: Are all these providers committed to adhering to the interfaces
that you have or is it up to you doing
the implementation to adjust if their APIs change.
Speaker 4: So yeah, So let's take an example. So we'll say
the Alarma on for example. If people don't know, Olama
is some software that you can run locally on your
developer machine as a way of using things like language
models locally. You probably wouldn't use it in production, but
it's great for development, Okay. So there are there are
a bunch of different client packages for that for dot net.
So Alarma sharp is the most well known one, and
you can use it's APIs directly, just like it's straight
concrete APIs a newer Alarma client or whatever it is,
and use it directly. But then that code's not interoperable
with anything else. So what you can do is use
the iChat client implementation that it provides. So you would
say something like Newolarma client dot dot, and then at
the end you put dot as chat client and that
gives you back and in an object that implements this
iChat client interface. And now you can use that in
the same way as every other AI service.
Speaker 2: So it's really Olama's responsibility to implement Yeah, iChat client,
it is totally, which is the dot Net interface.
Speaker 4: Yeah, that's right, in the same way that Sarah log
implements IE logging. And you know, that's what the whole
point of having these standard interfaces is that everyone can
make them work in whatever way makes sense to them.
Speaker 2: Yeah, and I'm wondering, like, how would that evolve.
Speaker 4: Well, yeah, so it evolves in that the industry itself
is evolving and we're getting new types of AI services
coming out. A good example of that at the moment
are the real time APIs. So at the moment, most
of our interactions with language models are a sort of
request response pattern where we send a bunch of text
to them and they come back with a bunch of text.
But there's a different paradigm that's emerging called real time
at the moment, where it's a bidirectional communication channel a
bit like a web socket or you know, just straight
TCP socket connection where you can just sort of fire
a load of bites across the wire to the AI
system and it can send bites back to you at
any time in any order, like overlapping or whatever, and
those bites could represent text, or they could be audio
data or images or whatever in either direction. So that's
the real time API, and our standard abstraction I chat
client doesn't represent that today. It represents the request responds
text based pattern. So, as you alluded to, we have
to evolve as time goes on. We will add further
abstractions to represent other AI patterns when they become common
across multiple providers.
Speaker 2: And these providers keep contexts too, right well, they can
do so that they can.
Speaker 4: The classic way of interacting with the language model is stateless.
So if you're having a conversation in a chatbot where
you say A and it says B, and U say
C and it says D. When you sen want to
send the message E, you don't just send E to it.
You also send abcd like that because it hasn't remembered
any of the conversation, so you literally have to repeat
the entire conversation. Now, the app developer doesn't think about
that now, because that's like inside the libraries. But that's
what's happening on the wire.
Speaker 2: Okay, So we've been spoiled by chat GPT. In other words,
it's it's doing that context stuff under the hood and
we don't we don't have to worry about it.
Speaker 4: Yeah, the WebUI does that and to be honest, they
probably do some service side caching of the conversations as well,
but that's you know, that would be specific to them.
Speaker 3: Yeah, okay, interesting I've been I've got an open AI
interface into my Home Assistant implementation, and essentially what it
does is prefixes whatever my request is with the model
of the house as part of the prompt. So it's
essentially saying, here's what's in the house, now, what would
you do with this sentence? And then it comes back
with us, you know, so I can say things like
turn on all the outdoor lights, turn off the lights
in the garage, and turn off the lights in the garage,
and it will change those statements. It really translates that
into it's this switch that's which they know does all
the things. Yeah, but it is literally manufacturing the context
each time you make a request.
Speaker 4: Yeah, that is right.
Speaker 3: Yeah.
Speaker 2: When's Alexa going to catch up to this?
Speaker 1: Hmmm?
Speaker 3: Yeah. I don't envy those folks. The yellms have sort
of caught them with their pants down, both the Google
Home folks and they and the Amazon folks. They've got
you respond right.
Speaker 2: But they didn't They wanted to kill the product, didn't They.
Speaker 3: Well, the product was losing money, losing lots of money,
like the you know, the purpose behind those the Amazon
device was to sell stuff on Amazon Sales tool, except
that nobody did that right because it correctly became a
joke about walking into the room and ordering you know,
a ton of soap on someone, right like or having
your children order stuff because they saw you do it.
So nobody did that part. And they sold all that
gear for cost. So it was a few years ago
they said, hey, we're cutting these teams way back, like
we spent billions on them.
Speaker 2: Has anybody heard about anything that they're working on in
the back end?
Speaker 3: They clearly are. Yeah, I mean that they haven't.
Speaker 2: They have to be.
Speaker 3: They have to be. The same time, they had just
cut their teams and within months of Amazon saying that
Google did the same thing for their devices as well.
Speaker 2: Well. That that brings me to the next question, Steve,
do you have an iPhone? I do, yep. And what
do you think of Apple Intelligence?
Speaker 4: Oh, I don't have one that's that new, you say, I.
Speaker 2: Do, and I don't think it's very intelligent. I would
thought I was going to get an upgrade to sirikay
that hasn't happened. And I have, you know, my Apple
car play, so anytime I ask it something that's Siri ye,
and I'll ask it stuff and it's just as dumb
as it was before. Yeah, so I don't see Apple Intelligence.
Speaker 4: I mean, I'd be happy with even if the basic
SERI worked. Like I tell my phone to like set
a reminder or something and it just gives me a
spinner and that's it.
Speaker 2: Wow.
Speaker 3: So yeah, this is very unusual of Apple. But the
best guess that the Apple you know, Digerati have said
is that Apple was afraid last year if they didn't
say something about AI at WWDC because they only do
one show here, if they waited till this year would
be too late, so they basically pre aound something like normally,
Apple already has all this stuff worked out, talks about
it in the show and its releases on schedule. But
that is not what happened with Apple Intelligence. They came
up with a term, they pitched it at the show,
and then they immediately pushed it back.
Speaker 2: Yeah, No, that was weird.
Speaker 4: I don't even know if it's shipped globally. Yeah, I
don't know if it's available in the UK. I haven't
heard anyone talk about it.
Speaker 3: Yeah, it's it is. I think you're seeing the closest
thing to Apple being scared that they're missing the boat
on this thing and they're behaving oddly because we would
expect them. And again, I don't use Apple products, but
I've lived alongside them long enough to know if Apple
says they're going to do something, they generally just do
it and have a good plan. And that's not happening here. Yeah,
it's an odd time, but I do think we're all
feeling around for the right thing to do in this space.
Like I'm excited that you guys are building tools because
that gives me a standard way to interact with it
in my apps. No, like for me as a dot
net developer. Thanks like the extensions all I understand Right
when I need this, I go, I go to new Get,
I bring it in in and off I go. So
it's a way to sort of create a common approach
to integrating stuff into yourself. Yeah, absolutely, So. Just I
grab the blog post that I'll put in the show
notes because it does give you sort of a walk
through of this. And it's just that you talked about
open AI. You talked about, was it.
Speaker 2: Lama O Lama O Lama olama, And.
Speaker 3: Then there's the inference engine.
Speaker 4: The inference I'm not sure.
Speaker 3: As you as your AI inference Yeah.
Speaker 4: Oh right, yeah, sorry, as your AI inference. Yeah, that's right.
That's that's almost like the cloud version of OLAMA. That's
a very simple mental model for it. It's a way
of running any kind of model that you want to
provide from us or once from a standard catalog in
a hosted service, as opposed to say, like as your
open air, which is literally just open AI models and
doesn't include other things like the LAMA models.
Speaker 3: Because at the same time, as you AI has opened
up to many more models, and just open AI you
have all theseries.
Speaker 4: That's correct, I.
Speaker 3: Mentioned to the listener at the beginning of the show.
Speaker 4: Yeah, yeah, I think in a lot of cases that's
going to be a much easier way of running a
wider range of models in production than you know, setting
up your own like containers and serving model output from that.
That's quite hard business to do because you know, these
things only work well if they're running on the right
kind of GPU hardware, and that can be expensive, and
you know, you really don't want to be managing all
of that stuff.
Speaker 3: If you can avoid that stuff is plumbing, right. You
just want to be able to send your your thing
out over an API and get a useful response back. Yeah, done,
it's just another API. No magic. Please just let my API,
that my API call work exactly. You're creating a cloud dependency.
But that's not that weird. Like there seems to be
a big push to run more of these models on prem.
While we're recording this, I think C is still going
on the Consumer Electronic Show, and Nvidia made this announcement
about this three thousand dollars supercomputer running what they called
the Grace Blackwell chip set. But they're saying, for three
thousand bucks, you will run a two hundred billion parameter
model on prem.
Speaker 4: Oh it's funny, isn't it the way this competition works.
So yeah, the current status quo is that you use
a combination of Nvideo provided hardware and then the software
from open A or another tech giant, Microsoft or Google.
And so we've got Nvidea trying to commoditize the tech
the other tech giants the model providers by saying, oh no,
you just need our hardware. You don't need any cloud.
And then you've got the tech giants trying to commoditize
n video by saying, oh, we've got our own custom silicon,
you don't need Nvidia. Yeah.
Speaker 3: Fun well, happily buying each other's products too.
Speaker 4: Yeah, see who's going to win.
Speaker 3: This is co oppetition, But it did what impressed me, Like,
GPT three is one hundred and seventy five billion parameters.
So in a matter of four years, we've gone from
a cloud problem that Mark Rosinovich talked about building the
fifth largest supercomputer in the world to build that model
back in the day, to something for three grand I
could sit on my desk. In theory it was CES,
which is basically a concept show.
Speaker 2: Right, and the stock continues to get by them. Yeah.
Speaker 4: Yeah, it's amazing enough us that grows. In fact, just
yesterday I was experimenting with what's the biggest language model
that I can train on my laptop in like one minute?
I thought it was going to be nothing like I thought.
You know, it takes months of GPU time to make
a sensible language model, Like in one minute, it's not
even going to produce words that you can read. Sure,
it turns out you can produce something sort of if
you take a GPT that sorry, GPT two, an implementation
of that on a single GPU on my laptop, I
can get something that produces pretty intelligible sentences within one
minute of training.
Speaker 3: Wo.
Speaker 4: So yeah, it's crazy how how quickly we've come on
from you know, it being a you know, industry defining
problem to produce anything like the English language to being
really quite straightforward.
Speaker 3: You know, they're used to be an XKCD graphic about
identifying birds in a picture, being needing a team in
years of work, and now we expect our phone when
we take a photo, like I've got the new Android phone,
the Pixel nine, and one of its features in Gemini
is when you take a photo, it writes a caption
of what's in the book, and that's included, Like just prettful?
Speaker 4: Is that on device that.
Speaker 3: The apparently it's on device. I suspect there's a round
trip to the cloud involved. Okay, pretty sure?
Speaker 2: What I what I want to do when I keep
threatening my wife, I'm going to do this is to
take the ten thousand photos that have accumulated since I
started saving them and upload them to a cloud service
and have it identify what's in the photos, and then
create a database that I can search so that I
can say, you know, a picture of Richard Campbell with
a glass of whiskey, and nine thousand of those ten
thousand photos come up and.
Speaker 3: List Yeah, it's just in state from me.
Speaker 2: Yeah. I think that's another little science fiction any thing too.
But I mean you can sort of do that now.
Speaker 4: But oh you can totally do that now.
Speaker 2: Yeah, maybe not.
Speaker 4: You could even just have it generate those pictures even
if they didn't exist.
Speaker 2: Well, there you go.
Speaker 3: There is that. Yeah, you don't have enough pictures me
holding a glass of whiskey. You can make them.
Speaker 2: By the way, I still haven't been able to send
you a Christmas present because your mail system is borked. Richard.
Speaker 3: Well, we had a six week strike, yeah, and it
kind of made a mess of absolutely Ever.
Speaker 2: Is it back in business now or yeah.
Speaker 3: The strike has been stopped till the spring, so a
window now, Okay, that's right done. They never resolved anything.
They just kicked it down the road to this.
Speaker 2: Steve, do you have your hands in Blazer much these days?
Speaker 3: Yeah?
Speaker 4: So definitely not as much as before, because you know,
I've my focus is broadened and spread out. But yes,
I'm still very involved in the Blazer team. You know,
I've probably liked ten Blazer meetings a week and yeah,
so that we're working towards clarifying a dot net ten plan.
At the moment for Blazer, we've got a whole bunch
of meetings going on around that.
Speaker 3: It's crazy to think about. Dot net ten just gives
me chills. Man. Yeah, I guess it's only a couple
months ago. You ship dot net nine, So I guess
that's only fair. There's going to be another one.
Speaker 4: Yeah, I know what you mean. It's a bit like
browser versions, like you know, Chrome one hundred and fifty
five or whatever by now. Yeah, time goes on. So yeah,
the Blazer team is We're actually all gathering in person
next week, which is going to be fun to see everyone,
and we will try and dig in to a lot
of this planning and working out what we want. I
think one of the big themes that we'll focus on
in dot net ten is the most core scenario of
Blazer usage, which is Blazer server and what we can
do to make that as bulletproof as possible. Like one
of the key things that people have struggled with Blazer
server from the beginning is the fact that you have
to have this constant connection. And yes, we've always had
a sort of reconnection mechanism, but it's always been a
little bit you know, suboptimal and like does it really work,
And even when it works, it's a little bit like
jarring to the end user and can result in them
losing some state. So what can we do to make
it as practical as possible for a Blazer app developer
to have a really bulletproof experience for the user so
that whether or not the connection is lost, the user
does not lose state.
Speaker 2: Well, you certainly laid the foundation really well in dot
at nine. I mean even right out of the box,
the experience is, yeah, your code kind of freezes for
a bit, Yeah, but it usually comes back.
Speaker 4: Yeah, Yeah, that's right. We did improve it a lot
in nine. In prior to nine, it used to be
a pretty awful user experience, if I'm honest, where it
would just sort of make it the user's problem to
work out whether to reload. You know, it wasn't great.
And in nine it's a lot more prone to like
figure out the problem and solve it. Itself, but that's
still not bulletproof. You can still lose data because it's
stored in the server memory, and if the server's gone,
well it's gone, Like what else do you expect to happen?
So we want to create mechanisms by which you can
persist that state in a way that's independent of maintaining
the connection to the server.
Speaker 2: At nine, there are also laser improvements for auto mode.
In general, it didn't really work all that well for
me before in dot Net eight and before, and in
nine it seems to really work well.
Speaker 4: Yeah, so when an eight first shift, we unfortunately had
quite a bad bug with the automotive that basically meant
it was never going to work, which was terrible, and
we patched it almost straight away, but are still a
lot of people got this very bad first impression of that,
and even then it was you know, still had constraints
and limitations, and so yeah, we really wanted to improve
that a lot in nine, make sure that not only
did it really work this time, but it actually works
well in practical situations.
Speaker 2: And that seems to be with along with some state
management and you know, shifting state from server to client
when we go to web assembly. That kind of stuff
seems to be a really good pattern. I think now
that I would actually recommend it, you know, because you
get the startup snappiness of server, but then you get
the you know, the independence of web assembly.
Speaker 4: I absolutely agree with you that if your application is
built in the right way and the developer knows what
they're doing, then you can make it work really great.
And that's that's excellent. But we kind of want to
raise the bar from that from Dottnut time. We want
to make it so that you don't have to be
someone who writes your code in a sort of elaborate
and carefully thought out manner. You don't have to be
an expert, but rather just the natural way of building
your code in a simple, obvious manner. We'll just have
all those benefits just innately. That's really how we want
to raise the bar and improve things.
Speaker 2: Right, so we can still use server, because using server
is just by itself a huge benefit because you don't
have to have an API layer and you don't have
to extra security precautions and things like that.
Speaker 4: Yeah, it's very very straightforwards. I really enjoy whenever I
get to write a bit of playser server code because
it just bypasses so many of the boring boilerplate tasks
that you have in web development and just means you
just get something running very quickly and it just works.
Speaker 2: Well, I'm excited. Let me know when I'll have FILS.
Speaker 3: A long way from those experiments you showed in twenty sixteen. Friend,
my goodness.
Speaker 4: Oh yeah, yeah, yes, indeed.
Speaker 3: Yeah.
Speaker 2: Is that the best talk you ever gave?
Speaker 4: Or well, it's definitely fun. It was really cool.
Speaker 3: Actually to do that, you hurt David Fowler's head and
that's not easy to do. Yeah, yeah, that's true.
Speaker 2: Yeah, very cool.
Speaker 3: It was a great time. It's good. It was cool
to be in the room. Definitely, moment, definitely.
Speaker 2: Well, I'm looking forward to it, and I guess you
know the repos and open books so you can just
go check the progress and are the early bits out
there yet for.
Speaker 4: At ten, No, not at all, because we haven't even
decided what's going to go into it. Sometimes we start
our planning a little bit earlier, but we were intentional
as a team to wait a little bit longer. I'd
let things settle a little bit more, be more thoughtful
about collecting feedback and about like creating clear consensus about
what would be wanted in this particular release. So yeah,
that's why it's it's brought us into January before we've
got a nailed down plan.
Speaker 2: Excellent, awesome, Well we're coming to the end here. Is
there anything else you want to talk about before we
wrap it up?
Speaker 4: Not specifically for me, all right?
Speaker 2: Best place in your neighborhood for fish and chips?
Speaker 4: Fish and chips? Wow, I don't know. Probably got a
couple of local chippies, but yeah, it's not really my thing,
not really.
Speaker 2: Okay.
Speaker 3: You in the South of England is that fish and
chip country?
Speaker 1: It is?
Speaker 3: It is you eat?
Speaker 2: You must be going for a chicken teaka masala then
or something.
Speaker 3: Better it's more for you, yeah, for sure.
Speaker 2: And where's your where's your favorite place in Bristol?
Speaker 4: Oh, in Bristol, We've got a really good one called
Urban tand or just a good tip for the for
the locals. Yeah, that's that's sort of classic in my area.
Speaker 2: Well, Steve, it's always great to talk to you, of course,
and we'll be speaking to you more, you know, when
things around dot ten get awesome. Ye okay, thanks again
and we'll talk to you next time on dot net rocks.
Dot net Rocks is brought to you by Franklin's Net
and produced by Pop Studios, a full service audio, video
and post production facility located physically in New London, Connecticut,
and of course in the cloud online at pwop dot com.
Speaker 5: Visit our website at d O T N E t
R O c k S dot com for RSS feeds, downloads,
mobile apps, comments, and access to the full archives going
back to show number one, recorded in September two.
Speaker 2: Thousand and two. And make sure you check out our sponsors.
They keep us in business. Now, go write some code.
See you next time.
Speaker 5: You got javans
Speaker 2: That means home, then my Texas in line d