Speaker 1: How'd you like to listen to dot NetRocks with no ads?
Speaker 2: Easy?
Speaker 1: Become a patron for just five dollars a month. You
Speaker 1: get access to a private RSS feed where all the
Speaker 1: shows have no ads. Twenty dollars a month, we'll get
Speaker 1: you that and a special dot NetRocks patron mug. Sign
Speaker 1: up now at patreon dot dot NetRocks dot com. Hey
Speaker 1: guess what it's dot NetRocks episode nineteen forty four. I'm
Speaker 1: Carl Franklin.
Speaker 2: At amaterid cap nineteen forty four. Richard, I'm looking forward
Speaker 2: to the end of World War Two. Yeah, it's the
Speaker 2: beginning of the end. So nineteen forty four, the Allies
Speaker 2: launched D Day, the largest amphibious invasion in history, landing
Speaker 2: troops on the beaches of Normandy, France on June sixth,
Speaker 2: marking a turning point. In August, the Allied forces liberated
Speaker 2: Paris from Nazi occupation. You're welcome. In December, here's an
Speaker 2: anecdote to go with D Day if you like. Yeah.
Speaker 2: In concern for the soldiers in D Day, they mass
Speaker 2: produced penicillin for the very first time. There were two
Speaker 2: and a half million doses of penicillin made for the
Speaker 2: D Day invasion. That is so awesome. So post World
Speaker 2: War two, the reason we have antibiotics was that preparation. Yeah.
Speaker 1: In December, the Battle of the Bulge, the Germans launched
Speaker 1: a major counter offensive in the Ardennes region of Belgium.
Speaker 2: Did I say that right? Arden in the Ardennes? Yeah?
Speaker 2: Yeap Neardens.
Speaker 1: But the Allied forces eventually repelled the attack and in Rome,
Speaker 1: three hundred and thirty five Italians were killed in the
Speaker 1: Here's another thing I had but pronounce correctly in high school.
Speaker 1: Are eighteen r D ten ardittin all right?
Speaker 3: Right?
Speaker 1: R D ten We're going with that ar d eat
Speaker 1: I n E massacre including seventy five Jews and over
Speaker 1: two hundred members of the Italian resistance, various from various groups.
Speaker 2: So yeah, it's sort.
Speaker 1: Of the beginning of the end, the unwinding and leading
Speaker 1: up to the following year, nineteen forty five, which ended it.
Speaker 2: Right. Yeah. It's also the year that the first plutonium
Speaker 2: has ever made in the Hanford site in Washington, will
Speaker 2: eventually lead to the bombit Nagasaki. Yeah. And the Harvard
Speaker 2: Mark one, the built by IBM based on a design
Speaker 2: from professor at Harvard thirty five hundred relays and a
Speaker 2: fifty foot long camshaft because computers were different back then. Yeah,
Speaker 2: they were, and famously because it's a relays based computer.
Speaker 2: The next version of this they call, cleverly, the Mark two. Yeah,
Speaker 2: we'll have a moth get trapped in one of the relays,
Speaker 2: which race Hopper will find and remove and call the bug,
Speaker 2: and that will be the first bug, first bug in
Speaker 2: the machine. Yeah. I don't use a lot of relays
Speaker 2: and computers anymore.
Speaker 1: Yeah, And before we get started with doctor Rachelle, I
Speaker 1: wanted to just have you comment on the amazing recovery
Speaker 1: of the astronauts and the space station that happened this
Speaker 1: past week.
Speaker 2: Really not that amazing. It was so perfectly you know,
Speaker 2: it was an unexpected things. Those Butcher and Sonny both
Speaker 2: very experienced astronauts. When there was concerns about Starliner, they
Speaker 2: sent up the next crew with only the next crew
Speaker 2: on a crew Dragon with only two passengers, so they
Speaker 2: had the two additional seats for them to come back
Speaker 2: at any time. Yeah. But since they had two extremely
Speaker 2: qualified astronauts already up, why pay to send them back
Speaker 2: down when you can put them to work and in fact,
Speaker 2: they put Sonny in charge of the mission. She took
Speaker 2: over as mission commander for the station for the duration.
Speaker 1: And she and Butch were happy to stay there. They
Speaker 1: were like, no, we don't want to come home.
Speaker 2: Come on. Totally. They were never going to get to
Speaker 2: fly again. Those are retired astronauts, right, Yeah, so they
Speaker 2: got a great gig. Now that's going to take them
Speaker 2: more than a year to recover, which is also normal
Speaker 2: for a six months day, and they had a nine
Speaker 2: months day. Mark Kelly did a year, and you can
Speaker 2: read his book on this, Like, recovery is not a
Speaker 2: trivial thing. Yeah, I was watching him being interviewed. You know,
Speaker 2: you haven't walked on your feet nine months, your vestibulous
Speaker 2: systems messed up, your eyes have been bent out of shape. Like,
Speaker 2: it's not a small problem, right to recover from this.
Speaker 1: Yeah, I watched being interviewed on the news when it
Speaker 1: was having it. It's just still amazing to see that
Speaker 1: falcon Booster land.
Speaker 2: Land on his tail perfectly.
Speaker 1: Always it always is just going to be amazing to me.
Speaker 2: Yeah, no, it's it's a miracle. The crazier thing is
Speaker 2: it really is that starship Booster being caught out of
Speaker 2: the air. It's literally a twenty story, two hundred ton
Speaker 2: building that flies, yeah, and they catch it out of
Speaker 2: the air. So yeah, we are in amazing time. So
Speaker 2: the space industry has been funnelingentally changed by this, right.
Speaker 2: The cost of flight is so much lower. It's hard
Speaker 2: to even get her head around what's actually going on
Speaker 2: up there right now. So it's very cool with the proliferation.
Speaker 2: That's a very good experience for me. This week is
Speaker 2: very I felt very good about it, all right.
Speaker 1: So yeah, so that's a cue for me to roll
Speaker 1: the music for better no framework.
Speaker 2: So that's awesome. All right, man, what do you got
Speaker 2: our good buddy, Simon Crop has the genius. Simon Crop
Speaker 2: the ge. This guy is just he's so brilliant. He's
Speaker 2: brilliant and he comes up with solutions for things that
Speaker 2: you didn't even know you need it. Yeah.
Speaker 1: But this one is called symbol. It's a new GET
Speaker 1: package and it's an MS build task that enables bundling
Speaker 1: dot net symbols for references with a deployed app.
Speaker 2: Nice.
Speaker 1: The goal being to enable line numbers for exceptions in production.
Speaker 2: Oh okay, that's interesting.
Speaker 1: Yeah, because I guess you don't get that. Yeah, yeah,
Speaker 1: and this is this is what it does. So if
Speaker 1: you're in production you have an exception and yeah, I
Speaker 1: guess you log it, you're gonna see line numbers, all right, Yeah.
Speaker 2: That's cool. You got to know he had that problem, right, like, yeah,
Speaker 2: this is clearly a guy who built the thing to
Speaker 2: fix a thing that he had, and now we all
Speaker 2: get to benefit.
Speaker 1: Another alternative, I guess is just deploying the debug symbols
Speaker 1: with it, and now you're slowing things down in productions.
Speaker 2: So yeah, it's a lot more weight than just yeah,
Speaker 2: you know, use this library.
Speaker 1: So thank you Simon and Simon. Crop slash symbol on.
Speaker 2: GitHub continues to be awesome.
Speaker 1: See y mba l Yeah, the musical thing, the musical thing,
Speaker 1: all right?
Speaker 2: Who's talking to us? Richard grabbed a comment off a
Speaker 2: show eighteen thirty five of them when we did with
Speaker 2: our friend mattz Targanson talking about the next C sharp
Speaker 2: because we've got a great comment LLM related. This is
Speaker 2: from Murray who said MADD's mentioned making sure language features
Speaker 2: work with the tooling, such as ordering and link syntax.
Speaker 2: Increasingly with Copilot and other lms, this is part of
Speaker 2: the tooling. Yes. True. Obviously this is a year ago
Speaker 2: this comment, so you know so much changes happen. It's challenging.
Speaker 2: So given a piece of code using a new C
Speaker 2: Sharp language feature, which is what Mads was talking about,
Speaker 2: have you tried asking chat, GPT or copilot or so
Speaker 2: the LM to describe how that code works. If it
Speaker 2: gets it right, does it mean it's intuitive. He's an
Speaker 2: LM's intuition and at least you put that in quote,
Speaker 2: because there is no intuition in software. There is a
Speaker 2: grood approximation for the one that human programmers have, or
Speaker 2: a bad approximation, and if programmers are using copil, it
Speaker 2: doesn't matter about the human's intuition or the LMS. Let's
Speaker 2: complicate this fact with next year's LM that would be now,
Speaker 2: which will probably be profoundly different. Yes, so, having said
Speaker 2: all that, it's probably best to just aim for the
Speaker 2: human and let the LM catch up. Yeah, no intuition
Speaker 2: in software. The reality is, of course you would expect
Speaker 2: it to not understand a new language feature. There has
Speaker 2: to be some time for that language feature to be
Speaker 2: documented properly. The good news being as they keep regenerating
Speaker 2: these LMS on a regular basis, and Microsoft builds these
Speaker 2: features in public view on GitHub even before it ships.
Speaker 2: It's likely in the knowledge base that is the al Yeah, curiously,
Speaker 2: you know, in my last trip to Microsoft talking to folks,
Speaker 2: so what they're using, they've been using Claude Sonnet three seven.
Speaker 2: That's their favorite for working in dot net, which isn't
Speaker 2: that funny? Fascinating, But you know that's where it's at.
Speaker 2: So Mary, you're right, let's focus on the human understanding
Speaker 2: the language the most, because the software is only going
Speaker 2: to generate what it's got in its model, and it's
Speaker 2: up to you to assess it, although admittedly the compiler
Speaker 2: has to say also yes, and a copy of music
Speaker 2: Cobey is on its way to unit. If you'd like
Speaker 2: a copy of music code by, I write a comment
Speaker 2: on the website at dot netroocks dot comment on the facebooks.
Speaker 2: We publish every show there, and if you comment there
Speaker 2: and everything in the show, we'll send you copy of
Speaker 2: music code By.
Speaker 1: And if you don't want to wait for that, or
Speaker 1: you have other ideas and you just want to buy
Speaker 1: music to code buy, you can go to music tocode buy,
Speaker 1: dot net and track twenty two is new ish and
Speaker 1: you can get the entire collection an MP three flacre
Speaker 1: wave for a very good deal. It's a very good price,
Speaker 1: So happy coding, all right, Well, let's bring on doctor Birchell.
Speaker 1: Doctor Jody Birchell is the developer advocate in data science
Speaker 1: at jet Brains and was previously a lead data scientist
Speaker 1: at Verve Group Europe. She completed a PhD in clinical
Speaker 1: psychology and a postdoc in biostatistics before leaving academia for
Speaker 1: a data science career. She has worked for seven years
Speaker 1: as a data scientist in both Australia and Germany, developing
Speaker 1: a range of products including recommendation systems, analysis platforms, search
Speaker 1: engine improvements and audience profiling. She's held a broad range
Speaker 1: of responsibilities in her career, doing everything from data analytics
Speaker 1: to maintaining machine learning solutions and production. She's a longtime
Speaker 1: content creator in data science across conference and user group presentations, books, webinars,
Speaker 1: and posts on both her own and jet Brains blogs.
Speaker 1: In other words, a slacker.
Speaker 2: It occurs to me, Jody, that you and I hang
Speaker 2: out several times a year of various conferences, But I
Speaker 2: don't know that Carl's had time with you since we
Speaker 2: did that show at Tekarama. Takarama was the last time
Speaker 2: I saw you, No a couple of years ago.
Speaker 3: Yeah, yeah, exactly, So it's been a long time actually, Yeah.
Speaker 2: Things have changed your jet brains now.
Speaker 3: I have, certainly I think changed a lot. Yeah, yes, yeah, yeah,
Speaker 3: I was a jet brains when we first met as well,
Speaker 3: but I think I had only been there just over
Speaker 3: a year and so I was still like, I don't know,
Speaker 3: a little bit more shy, I think, a little bit
Speaker 3: less opinionated.
Speaker 2: You've been hanging around with the troublemakers for a while.
Speaker 3: Now, yeah, you talking about you?
Speaker 2: Yeah?
Speaker 3: Actually, well, and we're going to be hanging out in
Speaker 3: my hometown of Melbourne next month.
Speaker 2: Yeah, we're excited about that, yeah, NDC. Yes, so, And
Speaker 2: of course I've got family in New Zealand, so I've
Speaker 2: got to do a little time in Sydney to see
Speaker 2: some folks there, and then I'll be in Melbourne for
Speaker 2: the show with you, and then a week on the
Speaker 2: farm hanging with the cows and the cousins and the
Speaker 2: sheep and the sheep, No sheep, the sheep, what sheeps?
Speaker 2: The South Island thing? No sheep on the farm. No, no,
Speaker 2: it's it's it's a dairy farm. Dairy farm. Yeah. And
Speaker 2: by the way, cows are awesome. Sheep are dumb, dumb,
Speaker 2: dumb dumb, holy cow dumb. But they're tasty. Like how
Speaker 2: Jody says they're cute. I say they're tasty tasty. Where
Speaker 2: my mind is at, that's in the cow. The cows
Speaker 2: are smart enough that if they're actually having distress, you know,
Speaker 2: in birthing or anything, they will come for help. Wow. Right,
Speaker 2: Like they're bright and they and they follow the they
Speaker 2: follow the gates of the packs where you want them
Speaker 2: to go. But it doesn't mean they don't know how
Speaker 2: to open them themselves if they really wanted to. I've
Speaker 2: seen them do it. Yeah, damn, they're just playing along.
Speaker 2: Cows are great, they really are. In lls are great.
Speaker 3: Right in the right settings. Yeah, they are great.
Speaker 2: Yes, But even that that show we did in twenty three,
Speaker 2: you know you were the grown up in the room there,
Speaker 2: it's just tired, Like listen, there were limits like that.
Speaker 2: We're so hype ish in twenty three, not that it's
Speaker 2: all common rational in twenty five, but it's so.
Speaker 3: Funny actually, because I remember I was this was the
Speaker 3: first talk I did on LMS, so that one at
Speaker 3: Techorama actually was the first one I ever did.
Speaker 2: No free lunch.
Speaker 3: Yeah yeah, yeah yeah, And I was I was actually
Speaker 3: really scared of getting up and giving my opinion, like
Speaker 3: being a contrarian. Obviously, I'm feeling so vindicated right now.
Speaker 2: But it's right, isn't it.
Speaker 3: It's great being right, but it's I will say, like
Speaker 3: the hype has died slower than I thought it would.
Speaker 3: So I think Deep Seek finally has spelled the beginning
Speaker 3: of the.
Speaker 2: End, but not the end of the business, but the
Speaker 2: end of the hype cycle.
Speaker 3: The end of the hype cycle.
Speaker 2: Okay, I appreciate that the approach.
Speaker 3: To how we're going to be I guess, manufacturing these models,
Speaker 3: deploying these models, and thinking about these models fundamentally changed
Speaker 3: with Deep Seeks. So m it sort of showed that
Speaker 3: this hyperinvestment in data centers, which was kicking off with
Speaker 3: the Stargate project in the US. To explain context to
Speaker 3: anyone in the audience who doesn't know.
Speaker 2: It, five hundred billion dollars.
Speaker 3: And intended five hundred billion dollar investment between Open AI,
Speaker 3: the US government, and I think Microsoft was involved so I.
Speaker 2: Think Microsoft pulled out of it.
Speaker 3: It was gorecle very okak got you Yeah, yeah, that
Speaker 3: just got announced.
Speaker 2: Yeah, there was a little political game here is that
Speaker 2: was also run around the town. They sort of announced this, Hey,
Speaker 2: you know, I know we had this deal with open
Speaker 2: Ai wherever there's going to run an azure, but we're
Speaker 2: ready to let that go. I think it was because
Speaker 2: of Stargate that. Yeah, you know, there was sort of
Speaker 2: this pressure on Microsoft. You have to keep growing, growing, growing,
Speaker 2: and they're like, this is getting irrational. So if you
Speaker 2: want to go play with someone else, you knock yourself out.
Speaker 2: So yeahing it back to deepseek for a minute. From
Speaker 2: what I understand, you know, the open Ai and all
Speaker 2: these other models are looking at that and learning from
Speaker 2: it and figuring out how to make their own models
Speaker 2: more efficient. And at one point I heard that the
Speaker 2: Chinese model is, you know, hey, let's spend a lot
Speaker 2: less money on these things so that they're less expensive.
Speaker 2: We don't have to use as many processors and all
Speaker 2: that stuff. And I think I heard that, you know,
Speaker 2: the response from the American companies was, oh no, we're
Speaker 2: just going to make it ten times more one hundred
Speaker 2: times more powerful, you know, so a different kind of
Speaker 2: mindset whereas but that was originally Now I think that
Speaker 2: there's more of a desire to make to get smaller lllms, right, yeah,
Speaker 2: that are more specialized.
Speaker 3: The new ones of the story is that basically we've
Speaker 3: known that there are ways to make neural nets more efficient,
Speaker 3: right like, there are ways of making the models smaller,
Speaker 3: or after you've trained them, actually trimming them down and
Speaker 3: getting the same performance or almost the same performance for
Speaker 3: much smaller number of parameters. We've also known for quite
Speaker 3: a long time, and this is true with any machine
Speaker 3: learning model, that the higher the quality of data the
Speaker 3: you know, the better the model can perform for much
Speaker 3: smaller number of parameters. So this was proven last year
Speaker 3: with the Falcon last year or the year before with
Speaker 3: the Falcon models, they were sort of the first big
Speaker 3: open source ones that were trained on higher quality data
Speaker 3: sets and got a lot more performance for less parameters.
Speaker 3: But the most reliable way to get better performance was
Speaker 3: to scale, and I think what happened. The story I've
Speaker 3: heard in China is that they just couldn't get access
Speaker 3: to the same size of GPUs because of sanctions. Not
Speaker 3: sanctions Basically they weren't being sold in China, and so
Speaker 3: they had to make do with older and much less
Speaker 3: efficient processes, and they had to do all these tricks
Speaker 3: to basically share the training across a bunch of smaller machines.
Speaker 3: So this meant that they just couldn't create absolutely massive models.
Speaker 3: And essentially this meant that, yeah, they were forced to
Speaker 3: create a smaller model. But you know, the thing is
Speaker 3: is the quality of AI researchers and AI engineers that
Speaker 3: are being employed at companies like Open Ai and Anthropic
Speaker 3: and companies like this. I'm sure that they knew it
Speaker 3: was possible. It was just as I understood it, a
Speaker 3: less reliable path to performance. And you know, the American
Speaker 3: companies had they had the money and they had the
Speaker 3: servers to train it, so why not go big?
Speaker 2: And they understand that race right like they understand build bigger,
Speaker 2: keep going like it's a very American approach to things. Yes,
Speaker 2: you can always tune later, right, do your land grab now, but.
Speaker 1: Also that there's a difference between having one huge model
Speaker 1: like you know, chat ept that knows everything as bazillions
Speaker 1: of nodes or whatever it is, and then can you know,
Speaker 1: can cross reference things right and put connect the dots
Speaker 1: very much in ways that humans do, but in even
Speaker 1: more broadly, Whereas if you have smaller, less expensive models
Speaker 1: that are just our lllms that are trained on specific data, right,
Speaker 1: you'll get probably get more accurate things out of them
Speaker 1: for that particular set you know, that particular context maybe
Speaker 1: and then be able to have many of those with
Speaker 1: that have different expertise, but you won't necessarily be able
Speaker 1: to it won't necessarily be able to connect the dots
Speaker 1: like a large, huge model can.
Speaker 3: Right. This can actually lead into a further discussion about
Speaker 3: measurement if we want. But basically, looking at the current
Speaker 3: benchmarks that they're using to assess performance of llms, Deepseeky
Speaker 3: and smaller models coming out of China are actually rivaling
Speaker 3: the performance of larger models. So basically the understanding seems
Speaker 3: to be is that a lot of the parameters that
Speaker 3: these big models have are not actually being used every
Speaker 3: single time you try to do like inference for a
Speaker 3: particular task. It's only a subset of the parameters. So
Speaker 3: the way to think about parameters is think about neural
Speaker 3: nets as like you have inputs and then you have
Speaker 3: a bunch of neurons that are connected by what are
Speaker 3: called weights. They're basically multipliers, and you can kind of
Speaker 3: think about inference as a path that you take through
Speaker 3: the neural net, where like, you know, the whole thing's
Speaker 3: going to be used, but only certain weights will actually
Speaker 3: have an impact for particular types of tasks. And it
Speaker 3: sort of seems that what's happened with scaling down these
Speaker 3: models is that because they learned on so much data,
Speaker 3: and so much of the data seems to have not
Speaker 3: been high quality, that they really, like a lot of
Speaker 3: the parameters were not really being used in the majority
Speaker 3: of cases, they were just I see dead weight.
Speaker 1: And so so if you wanted to translate parameters and
Speaker 1: neurons to language, we're talking about the probability of the
Speaker 1: next word exactly right that it spits out. Yeah, and
Speaker 1: what you're saying is that they're only choosing from parameters
Speaker 1: with higher weights.
Speaker 3: Yeah, it's it's like or words.
Speaker 2: With higher weights.
Speaker 3: Yeah. So basically the way it works is, like you
Speaker 3: think about the last layer of the neural net is
Speaker 3: basically like all the words in the vocabulary. So it's
Speaker 3: obviously really really huge, and so the whole neural net
Speaker 3: is trying to predict to the probability of which of
Speaker 3: these words is the most likely to come next. So
Speaker 3: it's basically saying that for a particular import only a
Speaker 3: subset of that, you know, the paths that go through
Speaker 3: the neural net are actually going to give good information
Speaker 3: about what the next word is. And so yeah, it's
Speaker 3: it's also like it's kind of fascinating because the models
Speaker 3: are such black boxes. No one fully understands how the
Speaker 3: decisions are being made. I'm putting decisions in air quotes.
Speaker 3: I want to make this clear because interpretability is hot,
Speaker 3: but this is actually interpretability is becoming a really hot
Speaker 3: area in twenty twenty five. So actually understanding how llms
Speaker 3: come to the conclusions they come to, or sorry, how
Speaker 3: the predictions being made. Let's put it in more clinical terms,
Speaker 3: and that's going to help firstly make the models more efficient,
Speaker 3: but also demystify a lot of the assumptions we make
Speaker 3: about the predictions they make. Like we look at the prediction,
Speaker 3: we're like, oh, it's solving problems because if a person
Speaker 3: did that, it would be showing problem solving. Or the
Speaker 3: model's more intelligent because if a person did that, it
Speaker 3: would be showing more intelligence, but that's just us projecting.
Speaker 2: Sure, yeah, anther of morphisation. Now you know, I'm maybe
Speaker 2: I'm thinking about this the wrong way, but you know,
Speaker 2: as soon as you say that, I'm like, hey, there's like,
Speaker 2: what six hundred thousand words in the Oxford Dictionary that's
Speaker 2: just English and most people use fifteen hundred of them.
Speaker 2: So oh yeah, yeah, yeah. You know here you've built
Speaker 2: this model that has this huge potential range of comprehension
Speaker 2: and you're using a tiny subsect of it depending on
Speaker 2: what you're doing. Especially when we're coming at this from
Speaker 2: the copilot part of you was like, I'm working on code.
Speaker 1: Yeah, every symbol in the language is a is a
Speaker 1: word essentially right.
Speaker 2: So, but you also talked about performance. In My immediate
Speaker 2: reaction was, so, what do we mean when we say performance?
Speaker 3: Yes?
Speaker 2: Is that speed? Is that a speed measurement or is
Speaker 2: that an accuracy measurement?
Speaker 3: Yeah? So to kind of put this in context, I
Speaker 3: gave a keynote to NBC Porto about all the hairy
Speaker 3: things that go along with assessing LLM. So I didn't
Speaker 3: get into speed. We can come back to that if
Speaker 3: we get time. But it's more about like how do
Speaker 3: people judge if these models are good? And last time
Speaker 3: we talked and you gave the episode this name, we
Speaker 3: talked about the concept of there's no free lunch in
Speaker 3: machine learning, and what this means is there is no
Speaker 3: there's no one model that will be best for every
Speaker 3: possible task you can do.
Speaker 2: Right.
Speaker 3: But what we've seen with the way people talk about
Speaker 3: llms is there advertised exactly like this. Like it's like, oh,
Speaker 3: open Ai just came out with the one model, and
Speaker 3: it is the best model on the market, right, right,
Speaker 3: And even if we're not, let's put like engineering considerations aside,
Speaker 3: let's talk about like, let's put cost aside, let's put
Speaker 3: speed aside. That's still not going to be true.
Speaker 1: It's like who's the best guitar player in the world?
Speaker 3: Yes, how do you measure this?
Speaker 2: That's an impossible question? Answered well, I think when they
Speaker 2: were saying best that time, we were talking the largest
Speaker 2: number of parameters, weren't they.
Speaker 3: Well, what they're talking about is there's this suite of
Speaker 3: benchmarks that are designed to assess LLM performance. And we
Speaker 3: talked about this last time. But llms were originally designed
Speaker 3: to be natural language processing task generalists. So they're good
Speaker 3: at doing a range of natural language tasks, often without
Speaker 3: further training out of the box, so they can do
Speaker 3: things like classification, summarization, they can do translation, things like this.
Speaker 3: So generally, when these models were first designed, they were
Speaker 3: benchmarked against how well they could do these natural language tasks,
Speaker 3: like specific things like question and answering, translation, blah blah blah.
Speaker 3: But as as the capabilities of the models have grown,
Speaker 3: or maybe they seem to have grown, we don't know.
Speaker 3: What we started doing is getting them to do things
Speaker 3: like grade school math problems, or we've gotten them to
Speaker 3: do suites of questions that are designed to assess problem
Speaker 3: solving or blah blah blah. And then what we do
Speaker 3: is we collate a bunch of these gold standard measures
Speaker 3: together and we combine them in such a way, and
Speaker 3: we create leader boards and we rank these models and
Speaker 3: we say, oh, Okay, this model is the best because
Speaker 3: it did the best at the MMLU, which is like
Speaker 3: a reasoning benchmark, or this one's the best because it
Speaker 3: did the best at like a collated collection of all
Speaker 3: of these benchmarks. So it's doing well on reasoning, and
Speaker 3: it's doing well on problem solving, and it's doing well
Speaker 3: on math, and it's doing well on coding. But this
Speaker 3: is the thing, like, firstly, a lot of these measures
Speaker 3: have been found to have serious problems. Then they've been
Speaker 3: found to really not measure what they said they claim
Speaker 3: to measure in a variety of ways. And the second is, Okay,
Speaker 3: I am an application developer. I want to design an
Speaker 3: application that uses an LM. Say I want to make
Speaker 3: a chatbot that can help people plan their holidays. What
Speaker 3: does it matter to me that an l ELM is
Speaker 3: really good at solving science problems, grade school math problems,
Speaker 3: Like is that going to be good for my application?
Speaker 2: So, got a calculator do you have, like, okay, gotta
Speaker 2: coverage and.
Speaker 3: It's probably going to do the math wrong anywhere because
Speaker 3: they're not symbolically simulating exactly they do that, But.
Speaker 2: Then you also have it return the response in the
Speaker 2: form of a limerick.
Speaker 3: That's fantastic. It's what our customers needed.
Speaker 2: That's it.
Speaker 3: So yeah, this is part of the problem. The way
Speaker 3: we talk about the way we talk about l elms
Speaker 3: is we talk about them like they are a thing
Speaker 3: independent of machine learning, but they are absolutely not. And
Speaker 3: part of the problem with that is it means that
Speaker 3: the way that we use them is we tend to
Speaker 3: trust their outputs too much, and we also tend to
Speaker 3: you know, not have scrutiny about like whether a model
Speaker 3: is the best fit for our use KSE, so we
Speaker 3: don't design assessments to see like is this actually doing
Speaker 3: what it's supposed to do, which we would absolutely do
Speaker 3: with traditional machine learning.
Speaker 1: I have had the experience of using lllms in you know,
Speaker 1: both chat, GPT and Copilot to help with coding things,
Speaker 1: and I found a situation where I asked it to
Speaker 1: do something, you know, to write something, and instead of
Speaker 1: pointing to something in the framework that already did that
Speaker 1: and say why don't you just use this, it just
Speaker 1: went ahead creating the thing, you know, reinventing the wheel.
Speaker 1: And then you know, an hour later, I've got something
Speaker 1: that works. But I'm like, hey, there's something in the
Speaker 1: framework that works just like this.
Speaker 2: Yeah.
Speaker 1: So it's that's why you need a human in the
Speaker 1: in the equation.
Speaker 2: Although although where's there a prompt there to say is
Speaker 2: there a class that does X?
Speaker 1: Well, that would have been yeah, that's the human error
Speaker 1: that because that should be the first question. It's like
Speaker 1: when somebody says, you know, I have an idea for
Speaker 1: an app, and my first question is, well, first of all,
Speaker 1: I don't I'm not going to write it for you
Speaker 1: unless you pay.
Speaker 2: Me in second of all, does it already exist? And
Speaker 2: the answer is usually yeah, if it's really that good
Speaker 2: of an idea, somebody else's somebody else has done it. Okay, well,
Speaker 2: why don't we take a break down. I want to
Speaker 2: dig into some of these evaluation strategies.
Speaker 1: All right, we'll be right back after these very important messages.
Speaker 1: Stay tuned. Do you have a complex dot net monolith
Speaker 1: you'd like to refactor to a microservices architecture? The micro
Speaker 1: Service Extractor for dot Net tool visualizes your app and
Speaker 1: helps progressively extract code into micro services. Learn more at
Speaker 1: aws dot Amazon dot com, slash modernize.
Speaker 2: Am We're bag. It's done at Rocks Amateur canvill that's
Speaker 2: called Franklin talking to doctor Jody Burchell. Hi. And if
Speaker 2: you don't enjoy those those ads and you'd like an alternative,
Speaker 2: we do have a Patreon that provides an ad free feed.
Speaker 2: Let's go to patreon dot com. Check it out Patreon
Speaker 2: dot dot NetRocks dot com. Yeah, so I found the
Speaker 2: deep avow site that talks about MMLU. But nearest I
Speaker 2: can tell this is just a set of questions in
Speaker 2: different topic areas.
Speaker 3: Yes, yes, so so let's talk about benchmarks. So there
Speaker 3: is a very famous leader board called the Hugging Face
Speaker 3: open Fellolane leader board something like that. Okay, So hugging
Speaker 3: Face is a company. They're based in France and basically
Speaker 3: what they do in their open source branch is provide
Speaker 3: access to all of the major open source what are
Speaker 3: called foundational models, so big l lams that are open
Speaker 3: source computer vision models, those that can generate audio, you know,
Speaker 3: do transcription, all these sort of things. And so Hugging
Speaker 3: Face take the open source models. They run these models
Speaker 3: against a suiteter benchmarks and then they call aid them.
Speaker 3: And they used to have a used to have a
Speaker 3: leaderboard up until June last year. This was the first
Speaker 3: version and it included scales like Hella, Swag and the MMLU.
Speaker 3: So it got retired for a couple of reasons. But
Speaker 3: one of the reasons that got retired is people started
Speaker 3: going through the questions and MMLU was bad. It had
Speaker 3: a few questions that literally were like I think one
Speaker 3: of them was something like the continuity of the theory.
Speaker 3: That's that's the full question. And then it was a
Speaker 3: bunch of multiple choice answers that were just lists of
Speaker 3: numbers like that was the question, and think about, Wow,
Speaker 3: the gold standard is a human, so humans meant to
Speaker 3: be able to answer this, And then you rank how
Speaker 3: well the LM goes, and I'm.
Speaker 2: Like, nobody can answer that.
Speaker 3: What does this mean?
Speaker 2: What does it even mean?
Speaker 3: What does this mean? But my favorite, my favorite, my
Speaker 3: favorite was Hella Swag. So apparently Hella Swag I think
Speaker 3: was made using mechanical turk, so they got people to
Speaker 3: generate the questions and then validate them. But clearly whoever
Speaker 3: picked up this task was like not particularly invested, Like,
Speaker 3: you know, they're not getting paid a lot of money,
Speaker 3: they probably didn't care, right, And I have actually an
Speaker 3: article with some of my favorite absolutely bizarre Hella Slag questions. Okay,
Speaker 3: now keep in mind I am reading this out as
Speaker 3: it's written. Okay, So we have a question, and we
Speaker 3: have a bunch of multiple choice answers, and what the
Speaker 3: LM is supposed to do is complete the scenario. So
Speaker 3: it's meant to pick the option that has the most
Speaker 3: you know, fitting scenario end. Okay, so I've got one
Speaker 3: for you. Man is in roofed gym weightlifting. Woman is
Speaker 3: walking behind the man watching the man. Woman is a
Speaker 3: tightening balls on stand on front of weight bar b
Speaker 3: lifting eights while he man sits to watch her cee
Speaker 3: doing mediocrity spinning on the floor, D lift the weight
Speaker 3: lift man.
Speaker 2: That doesn't make any sense.
Speaker 3: It doesn't and probably around a third of the questions
Speaker 3: in hellaswag with this garbage.
Speaker 1: I just want to know what mediocrity spins are. I
Speaker 1: want to do that, and I just don't know because
Speaker 1: I don't.
Speaker 2: Know the definition.
Speaker 3: That's every time I turn around and knock something off
Speaker 3: a shelf with my clumsy hair. That was twenty twenty.
Speaker 3: That was mediocol for the child.
Speaker 2: Yeah, there's been some times. So I mean this just
Speaker 2: seems lazy then, like the well, let's back up. Is
Speaker 2: asking questions of an LM actually a good way to
Speaker 2: measure its effectiveness?
Speaker 3: Now? Yes and no. So you can create well defined
Speaker 3: problem suites if you have a good idea of what
Speaker 3: you're assessing. So this is this is basic measurement theory, right,
Speaker 3: It's like we learned this in psychology. It's tricky with
Speaker 3: llms because we have a tendency to extrapolate too much.
Speaker 3: We try to project what their performance would mean if
Speaker 3: a human did that, and we can't do it because
Speaker 3: llms do not have what's called fluid intelligence or general intelligence, right.
Speaker 3: They have what you could essentially call crystallized intelligence, which
Speaker 3: is that they have a bunch of little templates of
Speaker 3: how things work based on scenarios they've seen before. They
Speaker 3: can patent match questions they see against this, So you've
Speaker 3: got to be really careful about how far you deviate
Speaker 3: from the doing patent matching to their showing intelligence, right,
Speaker 3: But it is possible. Let's say you want to assess
Speaker 3: how well they do specific tasks, like they can answer
Speaker 3: questions about history or whatever. That's fine. I think that's
Speaker 3: fine to assess. It gets tricky because there are two
Speaker 3: main problems with using questions other than the one I've
Speaker 3: just said. The first is is that the answer type
Speaker 3: that an LM is presented with actually impacts their performance.
Speaker 3: So most of these measurements use multiple choice questions, and
Speaker 3: the reason that they do that is because it's much
Speaker 3: easier to score because they're essentially ways of seeing, you know,
Speaker 3: the probabilities of words I was talking about. You can
Speaker 3: quite accurately tell what's the highest probability sequence that it
Speaker 3: would have ended up predicting based on, you know, the
Speaker 3: ones that is present with, So you know, it's much
Speaker 3: much easier to work this out. But you can also
Speaker 3: get them to generate answers, and generating free form answers
Speaker 3: is really hard to assess unless you're gtting humans to
Speaker 3: actually compare it to a gold standard because the statistical
Speaker 3: ways we have of comparing two sequences are imperfect. So
Speaker 3: most of the time people will use these multiple choice
Speaker 3: answer keys. But the problem is is that elms seem
Speaker 3: to do a lot better when they're presented with multiple
Speaker 3: choice answers compared to free form answers. Sure, and the
Speaker 3: reason it seems to be is because it's a lot
Speaker 3: easier to just pattern match to something they've already memorized
Speaker 3: as opposed to having to generalize a bit more. And
Speaker 3: then the second big problem is hell, elms are ridiculously
Speaker 3: sensitive to the format of the prompt template you use.
Speaker 3: We've already talked about this, like did you tell them
Speaker 3: to use a framework that already exists? But it's so
Speaker 3: much more subtle than that. So using a different placement
Speaker 3: of punctuation, using different spacing, this can impact the performance
Speaker 3: of LMS on task by like thirty fifty seventy percent. Wow, yes, yeah,
Speaker 3: and like why it seems to be again pattern matching.
Speaker 3: So if that like particular formatting is closer to something
Speaker 3: that it's seen already in training, it's more likely going
Speaker 3: to be able to get it right.
Speaker 2: So all I got to do is ee cummings my
Speaker 2: prompt and it just.
Speaker 1: Exactly it's just Richard invents a new ferbs.
Speaker 2: Yeah. But you know an interesting point, like anytime you
Speaker 2: want to remind a person that this software is not intelligent,
Speaker 2: it's that that recognize that this is pattern matching. The
Speaker 2: fact that that as a human, I can hand you
Speaker 2: only lower case there's no punctuation statement or a perfectly
Speaker 2: punctuated statement, and you'll see it as exactly the same,
Speaker 2: just one lazier than the other. But the software treats
Speaker 2: it completely differently.
Speaker 1: Exactly do you guys know the story of what the
Speaker 1: moment that Bill Gates went nuts over chat GPT and
Speaker 1: began to trust it and his mind was blown over it.
Speaker 1: So the Richard I sent you a link in the
Speaker 1: chat you can post it there. This is the story
Speaker 1: and I heard about this story on the on the radio.
Speaker 1: So the story from this CNBC dot com things. Bill
Speaker 1: Gates watched chat GPT asen ap bio exam and went
Speaker 1: into quote a state of shock. And this was August eleventh,
Speaker 1: twenty twenty three. So but what you don't know, and
Speaker 1: I don't even know if they say it in this article,
Speaker 1: I don't think they do. But a couple of months
Speaker 1: before is when Sam All actually showed Bill chat GPT
Speaker 1: and he added a couple of things and he said,
Speaker 1: you know what, you know, it would be a great test.
Speaker 1: Sam is if we could give it the ap bio test.
Speaker 1: And it aced it and then Sam goes home and
Speaker 1: two months later brings it back and it ass the exam.
Speaker 1: So what do you think happened in those two months?
Speaker 3: What a mystery.
Speaker 2: It's so strange, I can't imagine. I'd also point out
Speaker 2: that a GAP exam is largely multiple choice.
Speaker 1: There you go, Well, I mean I thought it was
Speaker 1: I thought they were essay questions. There were five questions,
Speaker 1: but I don't I didn't read that part, but I
Speaker 1: heard that they were five five essay questions. Anyway, they
Speaker 1: did not say that in the article, but that that
Speaker 1: apparently happened.
Speaker 2: It all depends on what you.
Speaker 3: Train it on, right, Yes, and this is actually a
Speaker 3: bigger issue. So this is an issue that's called data leakage, again,
Speaker 3: well known problem in machine learning. It's when your model
Speaker 3: gets access to the test set during training that it
Speaker 3: can basically learn the answers and well, the implication from
Speaker 3: Carl is that this may not have been an accident
Speaker 3: this time. But you know, we don't have a clear
Speaker 3: idea of what's in the training data for a lot
Speaker 3: of these models. Even open source models now are being
Speaker 3: super cagey about what's in their training data. So they
Speaker 3: say it's a competitive advantage. But we know from experiments
Speaker 3: people have done that even benchmarks have ended up at
Speaker 3: least partially leaking into the data. So we know that
Speaker 3: a lot of these companies will optimize for benchmarks. They'll
Speaker 3: keep training the models until or not keep training them,
Speaker 3: but they'll keep tuning them until they do well on benchmarks.
Speaker 3: But even accidentally, because they're just scraping the open internet,
Speaker 3: sure they've accidentally shoved a bunch of these questions.
Speaker 2: Which is probably where the benchmarks came from in the
Speaker 2: first place. Any so exactly eventually you're going to meet
Speaker 2: up with the data. It doesn't seem surprising at all.
Speaker 3: So yeah, the modern suite of like benchmarks that started
Speaker 3: being created last year, they started making them private reasons
Speaker 3: to mitigate this. But it doesn't mean that you know,
Speaker 3: you as a consumer, you're a lay user of an
Speaker 3: l l M. Maybe not a lay user. You might
Speaker 3: be a bit more technically advanced, but none of us
Speaker 3: here are. AI research is right, right, and so we
Speaker 3: might be not far from that to inform consumer. Let's
Speaker 3: put me that way.
Speaker 1: But you know, I just found it. I'm sorry to interrupt,
Speaker 1: I just found it. A story is that, you know,
Speaker 1: Gates issued what he believed to be a rather difficult
Speaker 1: challenge to Sam Oltman, bring chat GPT back to me
Speaker 1: once it could exhibit advanced human level competency by achieving
Speaker 1: the highest possible score on the ap by l exam.
Speaker 1: And so two months later, oh Magic, Open a Eyes
Speaker 1: developers came back and Gates watched the top score of
Speaker 1: five on the test. So so yeah, so there it is,
Speaker 1: right in black and white that actually happened. And as
Speaker 1: I was hearing this, I was like, you idiot, why
Speaker 1: didn't you just say give it to me when it
Speaker 1: can answer a test question and don't tell them what test? Yeah,
Speaker 1: you know a test question and then just do it.
Speaker 2: Try it.
Speaker 3: Yeah, I don't know.
Speaker 2: Far be it from me to call Bill Gates an idiots?
Speaker 2: Did I actually do that? But you know, there's this
Speaker 2: confirmation bias situation you can put yourself into. Yeah.
Speaker 3: And this is the thing too, Like I don't blame
Speaker 3: people for feeling enchanted by the models, like there is
Speaker 3: something so human feeling about them because they're echoing back
Speaker 3: our humanity. Yeah, but you need you always need to
Speaker 3: be CAUs and like we were trying to do at
Speaker 3: the beginning, like you see, we slip into anthromorphizing the
Speaker 3: models even though we know better.
Speaker 2: For sure, because it's easy.
Speaker 3: It is easy. But really, like even with the latest
Speaker 3: benchmarks trying to assess AGI, this one called the ARCAGI
Speaker 3: that Open Eyes three actually did very well on got
Speaker 3: seventy percent late last year. This is still just pattern matching,
Speaker 3: but pattern matching in a more organized way. It's basically
Speaker 3: the model has more of an ability to sort of
Speaker 3: sort through which patterns might be the best to apply.
Speaker 3: But again, we're just talking about a more systematic application
Speaker 3: of crystallized intelligence. We're not talking about generalizability yet.
Speaker 2: Yeah, And I mean the more I read, the less
Speaker 2: I'm concerned about the AGI side of the equation. It
Speaker 2: seems more and more like a marketing term to hire
Speaker 2: more people to work at open AI.
Speaker 1: Yeah, it's only a fluid term that keeps changing. The
Speaker 1: definition keeps changing.
Speaker 3: But how do you assess AGI? Like I don't know
Speaker 3: if we talked about this last time, because I had
Speaker 3: that in my first talk, But you know, how can
Speaker 3: you even assess the gap between what a model knows
Speaker 3: and you know, a task, so like the difficulty that
Speaker 3: a model would have doing that task based on what
Speaker 3: it already knows, and then standardized that across a bunch
Speaker 3: of different models that have potentially been exposed to very
Speaker 3: different tasks and knowledge like it. Sure, it feels it
Speaker 3: feels like such a difficult.
Speaker 2: Challenge and it's way too broad, and ultimately I feel
Speaker 2: like it's a distraction from the fact that we're just
Speaker 2: trying to be engineers making using a useful tool. And
Speaker 2: I mean I let off this conversation talking about the
Speaker 2: fact that I always ask folks like what one are
Speaker 2: you using right now? What are you enamored of? And
Speaker 2: the fact that you know, I had sort of a
Speaker 2: universal everybody likes cloud right now, it's like, why what
Speaker 2: do you What is your innate benchmark that made you
Speaker 2: switch to this or is it just a social pressure
Speaker 2: thing vibesmen because that smart person was using Cloud. Now
Speaker 2: I'll use claud and then there'll be some nice confirmation
Speaker 2: bias there. Well, yeah, no, it seems to be doing
Speaker 2: the thing. Is it actually better than the chat GPT information?
Speaker 2: I don't know how would you measure that? So we're
Speaker 2: in this loop and I don't feel like I can
Speaker 2: get a version, a new version of anything from any
Speaker 2: of these folks come out and you open AI a
Speaker 2: new cloud or any of these and say, okay, is
Speaker 2: it worth switching? Yeah? I mean I know they want
Speaker 2: me to. I know it's invariably more expensive, but is
Speaker 2: it better?
Speaker 3: Yeah? Look, I have a prediction that here we go.
Speaker 3: I'm going to do a prediction why not?
Speaker 2: Why not? Why not? Why not?
Speaker 3: I'm going to say probably in a year's time, the
Speaker 3: landscape of providers is going to look quite different. Oh, definitely,
Speaker 3: And it's because the advantages of using smaller models is
Speaker 3: just drastically outweighs using bigger one. They're cheaper, they're more
Speaker 3: momentally friendly, they're more more specialized, they can be more specialized,
Speaker 3: like it's easier to tune them so that you can
Speaker 3: focus them on specific tasks. And yeah, ultimately they're just
Speaker 3: you know, it's easy to control what happens to your data.
Speaker 2: So right, that's a big one.
Speaker 3: That is a big one.
Speaker 1: It's a big one, especially with something like deep seek.
Speaker 1: You know, the only way I'm going to run that
Speaker 1: is on my own network not connected to the internet.
Speaker 2: Well, they do often a local offer a local version, Yeah,
Speaker 2: they do, right, which n video has been benchmarking with
Speaker 2: a fair bit I noticed, like, which I thought was cool,
Speaker 2: like smart thing to do, not just to because there's
Speaker 2: lots of folks saying, no, don't use the Chinese LLM.
Speaker 2: But yeah, the fact that video just said.
Speaker 1: That's the same reason they're saying don't use TikTok, right,
Speaker 1: so they don't trust it more or less?
Speaker 2: What could happen I don't.
Speaker 3: Use Yeah, I don't use TikTok just because I'm deeply uncool,
Speaker 3: Like it makes with you.
Speaker 2: I'm so with you.
Speaker 3: I actually had to make a TikTok. I was at
Speaker 3: a workshop for my job like two months ago, so
Speaker 3: you know, I'm an advocate. So they're like, hey, let's
Speaker 3: teach these old people to make tiktoks nice. And I
Speaker 3: made this TikTok with Michelle you know, Michelle Richard, Michelle Frost,
Speaker 3: so yeah, yeah, yeah, she just started with jet brains.
Speaker 3: And so we made this TikTok with another of our
Speaker 3: colleagues and an involved Wilbur, her dog, and it was
Speaker 3: just like it was so bad. And then they're like
Speaker 3: it's awesome, like you should go on TikTok right now,
Speaker 3: and I'm like no, I'm deeply ashamed, like.
Speaker 2: I should see.
Speaker 3: This was bad.
Speaker 1: I do have to confess that I have a TikTok
Speaker 1: account Carlotphoenix dot com. I have not used it yet
Speaker 1: for anything more than hey, I'm here, and I certainly
Speaker 1: don't scroll TikTok. I have so many better things to
Speaker 1: do than to scroll inane, insane, crazy music videos of
Speaker 1: people doing stupid things, and dogs and.
Speaker 2: Cats are also. You know what it's interesting about TikTok
Speaker 2: is you're not really picking the content they're picking the content. Yes, yeah,
Speaker 2: they are watching your loiter time, so it's your behavior
Speaker 2: that's only selecting the content. But you know, it is
Speaker 2: a different mechanism there where you can't really curate a
Speaker 2: list or build a social graph. That's not up to you.
Speaker 2: And and I find that interesting, right, like that, were
Speaker 2: literally are handing over our attention to something else that's
Speaker 2: driving it.
Speaker 3: Yeah, I do have to admit I'm so curious about
Speaker 3: their recommended the.
Speaker 2: Yeah, well, as a technologist, right anything, like well, because
Speaker 2: that's the thing that they're all upset about. This is
Speaker 2: our secret sauce, and we'll be keeping it to ourselves,
Speaker 2: thanks very much. Oh, it's well, here's the thing. Is
Speaker 2: there a way when you see something that you don't
Speaker 2: like to say, give her to death. I don't want
Speaker 2: to see this again? No, because it's too late to
Speaker 2: scroll past it. You've already scrolled. The problem is that
Speaker 2: before you found out you didn't like it, you watched it.
Speaker 2: You know, it's the old old I'm trying to improve
Speaker 2: the quality of my diet by eating everything and deciding
Speaker 2: what I like. Yeah, there is no nutritional label on
Speaker 2: any of these things.
Speaker 3: Sorry, it's called democracy.
Speaker 2: Okay, so that's what you want to call it. The
Speaker 2: only person who doesn't get a vote is the viewer.
Speaker 2: I'm not cynical at all. I don't know what you're
Speaker 2: talking about.
Speaker 1: What you remind me of David Mitchell, the British comedian
Speaker 1: who just every once in a while will just go
Speaker 1: off on a rant.
Speaker 2: Just start and he'll keep me going. I'm fine, Everything's fine, Okay. Yeah,
Speaker 2: we're still at this core issue of how do I
Speaker 2: select an LM from my app? I mean, part of
Speaker 2: it is the running contact or I can I can
Speaker 2: go down the cost side and can go down the
Speaker 2: does it, you know, integrate well with my Do I
Speaker 2: any cloud access? Or can I run local?
Speaker 3: Like?
Speaker 1: There's all those decisions we need an LM to answer
Speaker 1: this question.
Speaker 2: I don't think.
Speaker 3: I have something even better. I've got a blog post, hey,
Speaker 3: are right? He so I will share this with you
Speaker 3: so you can share it with the audience. But I
Speaker 3: came across when I was writing my talk, I came
Speaker 3: across this absolutely phenomenal blog post. So guy's an AI
Speaker 3: engineer called Hassan Hussein. So this guy works in an
Speaker 3: AI consultancy. So exciting job these days goes out and
Speaker 3: he needs to basically build applications for companies that use AI.
Speaker 3: And one of the jobs that he talks about is
Speaker 3: he and his company were hired to build a chatbot
Speaker 3: for real estate agents. So basically, they wanted the real
Speaker 3: estate agents to be able to type in natural language,
Speaker 3: like give me the contact details for everyone in this area,
Speaker 3: whatever you know, and then the LM would generate a
Speaker 3: quer to a CRM something like that and return the information.
Speaker 3: So when they first started building the app, they like,
Speaker 3: they picked a good LM based on the leaderboard, good one,
Speaker 3: and then they wrote the initial prompt templates and then
Speaker 3: you know, everything looked good, and then things started not
Speaker 3: working on the edge cases, so they made the prompt
Speaker 3: a bit more elaborate, and then the prompt started getting
Speaker 3: really unwieldy, and then they realized the only evaluation metric
Speaker 3: they had was vibes and they were like, really, this
Speaker 3: is a mess. So he set out in this really
Speaker 3: interesting way how they actually went back to ground zero
Speaker 3: and they started again, and he said, like, basically, we
Speaker 3: realized we needed a tiered assessment. So he said, like,
Speaker 3: the first tier of assessment is unit tests, Like it
Speaker 3: seems really obvious, right, But he's like, the thing is
Speaker 3: is like, because it's nondeterministic, you're not going to have
Speaker 3: one hundred percent pass rate on your unit tests. So
Speaker 3: you need to determine what error rate you're happy with,
Speaker 3: and that's going to require a bit of experimentation.
Speaker 2: But you also have to accept the level of ur rate,
Speaker 2: like you're not getting all agree exactly.
Speaker 3: Exactly, so it might be like you just need ninety
Speaker 3: five or ninety nine percent or whatever to pass whatever
Speaker 3: looks realistic. But you know, an example of unit tests
Speaker 3: is let's say the query from the user was return
Speaker 3: me the phone number of you know, Jane Smith or
Speaker 3: you know someone like that, and then basically what you're
Speaker 3: going to expect from the CRM is a phone number,
Speaker 3: So you can write a unit test for that. You know,
Speaker 3: it's basic engineering. And then he said, you know, you
Speaker 3: can create a suite of manual evaluations, so you basically
Speaker 3: look at the traces how the LM is interacting with
Speaker 3: the users and the rest of the system, and you
Speaker 3: manually evaluate that. And you don't have to keep doing
Speaker 3: that forever because then you can use a new method
Speaker 3: called LM as a judge just where you get another
Speaker 3: LM to also do the same assessments and trying to
Speaker 3: get them to converge. And once you have a relatively
Speaker 3: strong sense that the LM is giving similar assessments to
Speaker 3: your human you can you know, you need to check
Speaker 3: in on it from nine to time to see if
Speaker 3: it's okay. But that you know, takes over that part
Speaker 3: of the assessment, and then you know, you can go
Speaker 3: up to your normal kind of higher level assessments like
Speaker 3: a B testing. You know, it's really it's just a
Speaker 3: normal engineering system, and you can create a feedback loop
Speaker 3: where you can you know, refine your prompts or fine
Speaker 3: tune models, or use different models that maybe are smaller
Speaker 3: or cheaper and see whether you can get the same
Speaker 3: sort of performance. So you know, obviously you're going to
Speaker 3: need to just pick a model to start with. You
Speaker 3: might be able to get a sense of whether it's
Speaker 3: good for chatbot applications in this language, you know, do
Speaker 3: your research on that. But this really shows me, like
Speaker 3: it's just it's so obvious, right, like this is how
Speaker 3: we do monitoring.
Speaker 2: Yeah, and it's I'm sorry, this looks too adult me.
Speaker 3: Well, I know, it looks like a lot of hard work.
Speaker 2: Literally, as you actually have to work at building a
Speaker 2: day decent testing framework specific to your your case. I
Speaker 2: know I wanted I wanted a happy button. Jody can
Speaker 2: be sad I want to button. Yeah.
Speaker 1: We recorded a show with Spencer Schneidenbach, which is actually
Speaker 1: next week's show.
Speaker 2: We recorded it a couple.
Speaker 1: Of days ago, so we have the benefit of future
Speaker 1: looking here, and we talked about some of these things
Speaker 1: with him, and uh, you know that just the comment
Speaker 1: came up and I think it was even mere Richard.
Speaker 1: I can't remember who, but you know, we we used
Speaker 1: to have we used to be programmers that you know,
Speaker 1: we have a bug, we fix it. Now the program
Speaker 1: is one hundred percent accurate. And now I mean it's
Speaker 1: even it's even more like we're a psychologist now instead
Speaker 1: of scientists. You know, we make some suggestions, we examine
Speaker 1: the output, you know, we think about it a little
Speaker 1: bit and it doesn't seem quite right. We ask some
Speaker 1: more questions, examine the behavior.
Speaker 2: You know.
Speaker 1: It's like, if these things are going into our software,
Speaker 1: I have a little trepidation about that just because of
Speaker 1: the inaccuracies. Even if it's even if something is ninety
Speaker 1: nine percent accurate. That's that's a bug. That's a one
Speaker 1: percent bug.
Speaker 2: Yeah, and when you probably can't pin down.
Speaker 1: And one you probably can't fix.
Speaker 2: Use probabilistic tools, get probabilistic results. Yeah, exactly.
Speaker 3: Look, it's funny because I'm probably so much more comfortable
Speaker 3: with this than any of you, because I'm like, hey,
Speaker 3: this is just how stuff works.
Speaker 2: That was how machine learning always work. When we talk
Speaker 2: to you in twenty three. You've been doing this for years,
Speaker 2: and it's like you do the testing and there is
Speaker 2: no one hundred percent exactly. Yeah, you get in the
Speaker 2: mid nineties. You should feel good.
Speaker 3: Yeah, yeah, well sometimes suspicious, it depends sometimes too quickly.
Speaker 3: But yeah, I think it's an uncomfortable new reality. And
Speaker 3: you know, it's something I've observed for years when you know,
Speaker 3: you bring engineers into the world of machine learning, and
Speaker 3: it is a deeply uncomfortable thing not knowing that something
Speaker 3: is one hundred percent deterministic. I think the main problem
Speaker 3: is is it's one thing to say have a system
Speaker 3: that otherwise works totally in a deterministic faction. So let's
Speaker 3: say you've got some sort of system that say, takes
Speaker 3: in queries or takes in numbers from a user, let's say,
Speaker 3: like nutrition numbers for a piece of food or something,
Speaker 3: and then you have a machine learning model that generates
Speaker 3: a prediction that maybe within a certain band of correct.
Speaker 3: It's more difficult when you're talking about an LM being
Speaker 3: an actor as part of that system and generating pieces
Speaker 3: of code that will then run that system and then
Speaker 3: generating error in that way is actually quite consequential.
Speaker 2: I just like that phrase certain band of correct.
Speaker 3: We call it, we call it a confidence interval. Actually,
Speaker 3: how confident am I that this is correct?
Speaker 2: But you know, I think as a developer, when you're
Speaker 2: talking to leadership that want you to use these tools,
Speaker 2: I think they're going to provide as advantage. Just like
Speaker 2: one of their educations is these are nondeterministic models and
Speaker 2: there will always be a certain level of uncertainty, and
Speaker 2: if you're not good with that, we don't get to
Speaker 2: use these tools.
Speaker 1: Yeah.
Speaker 2: Yeah, yeah, that's right.
Speaker 1: So you know, the I guess first analysis you should
Speaker 1: do in your business is what level of uncertainty are
Speaker 1: we comfortable with?
Speaker 2: Can we tolerate? You know, what is the benchmark that
Speaker 2: we're shooting for? Well? And then the other side of
Speaker 2: this is the consequences of the uncertainty of the mistake,
Speaker 2: like what has happen? Right, people gonna die? You know,
Speaker 2: I know, I see this over and over again, where
Speaker 2: like the first case of an LLEM in an organization
Speaker 2: is with an HR system. So it's totally internal. And
Speaker 2: part of that is because as the consequence omitting correct
Speaker 2: is minor. You know, yes, you're going to make somebody
Speaker 2: angry if you tell them they have more vacation days
Speaker 2: than they do, but you probably haven't cost a company
Speaker 2: a lot of money.
Speaker 1: Well, and also there's a human there to make sure
Speaker 1: that the you know, accurate information gets given to the person.
Speaker 2: I like your optimism, but yes, I would hope you
Speaker 2: would hope. I would hope. But you know, get the point, like,
Speaker 2: there is a bunch of ways to manage this uncertainty.
Speaker 2: So there's a going to be a new corporate title
Speaker 2: job and it's going to be a nondeterministic compensator. Oh,
Speaker 2: I think that's from I think I think that's from
Speaker 2: Back to the Future. You're thinking, CuO chief Certainty officer.
Speaker 1: Chief Uncertainly that's good. I think you're thinking of uncoupling
Speaker 1: the Heisenberg compensators.
Speaker 2: There you go, that's star Trek, Star Trek.
Speaker 3: Yeah, bouncing all over the place, such geeks.
Speaker 2: I found the blog post from Hassan and I'll include
Speaker 2: it in the show, or from Hammel, who's sing man
Speaker 2: who sings your AI product needs evals And it's exactly
Speaker 2: the way you describe it. Building unit tests, doing model evaluation,
Speaker 2: doing a b testing. This seems like a real concrete
Speaker 2: approach to just how do we at least be able
Speaker 2: to look people in the eye and say we've done
Speaker 2: our best to test this and have some certainty around it.
Speaker 3: Yeap, And well, I think what I like about it
Speaker 3: is it's not unfamiliar territory for engineers. This is exactly
Speaker 3: what you've all been doing for decades now, Like this
Speaker 3: is just monitoring well, or at.
Speaker 2: Least should have been doing. This is looking like the
Speaker 2: testing we do on software.
Speaker 3: But this is the thing. It's no one's fault. That's
Speaker 3: an on the ground developer, because the way these models
Speaker 3: are sold is that no their magic like they are
Speaker 3: different to everything else. They are certainly not. They are
Speaker 3: the same as any other machine learning model, except slightly
Speaker 3: more problem because you're probably involving them in critical parts
Speaker 3: of generating code.
Speaker 2: Just be careful and measure please be careful. Yeah. So so,
Speaker 2: doctor Burchell, what's next for you? What's in your inbox? I?
Speaker 3: As I said, I'm heading down to Melbourne in a
Speaker 3: month for NDC. I'm going to be giving this talk
Speaker 3: actually the one I gave it Porter and I'm going
Speaker 3: to be giving one that I gave in Oslo just
Speaker 3: about the psychology of llms. If you will not be
Speaker 3: with me in Australia, you can also watch that on
Speaker 3: YouTube on the NDC channel. The moment that I'm laying
Speaker 3: kind of low, I'm actually going for my German citizenship,
Speaker 3: so nice. Yeah, I gotta do my citizenship test in June.
Speaker 3: Just did my language test couple of weeks ago, and
Speaker 3: like everything in Germany, it takes months, so I may
Speaker 3: be able to apply by the end of the year.
Speaker 2: So you got to wratch it up your complaint too.
Speaker 3: I laughed, actually so hard, because a friend of mine
Speaker 3: did her exam and one of her writing tests was
Speaker 3: to write a letter of complaint.
Speaker 1: Well it started because before you came on, Richard, Jody says,
Speaker 1: how you doing, and I said, I can't complain, but
Speaker 1: I do anyway, She says free chairman.
Speaker 2: Awesome, all right, thanks Jody, really appreciate it. Yeah, thank you.
Speaker 2: What a great conversation, all.
Speaker 3: Right, Always always a pleasure, Okay, and.
Speaker 1: We'll talk to you next time on dot net rocks.
Speaker 1: Dot net Rocks is brought to you by Franklin's Net
Speaker 1: and produced by Pop Studios, a full service audio, video
Speaker 1: and post production facility located physically in New London, Connecticut,
Speaker 1: and of course in the cloud online at PWOP dot com.
Speaker 1: Visit our website at d O T N E t
Speaker 1: R O c k S dot com for RSS feeds, downloads,
Speaker 1: mobile apps, comments, and access to the full archives going
Speaker 1: back to show number one, recorded in September two thousand
Speaker 1: and two. And make sure you check out our sponsors.
Speaker 1: They keep us in business. Now, go write some code,
Speaker 1: See you next time.
Speaker 2: You got Jack Middle Vans and