Speaker 1: Hey, folks, welcome back to another episode of the Ruby
Speaker 1: Rogues podcast. I'm your host this week, Charles max Wood,
Speaker 1: and I am here with Carmine Paulino.
Speaker 2: And you said you're from Berlin, from Italy, Southern Italy.
Speaker 2: I always get excited when I have Italian song because
Speaker 2: I lived in Italy.
Speaker 1: For a couple of years, and I'm like, oh, all
Speaker 1: the good memories anyway, and the good memories are the
Speaker 1: good food and good people anyway. But yeah, you live
Speaker 1: in Berlin and you wrote the Ruby llm jem. I'm
Speaker 1: sure there's a lot more to know about you. What
Speaker 1: else should people know?
Speaker 2: And go, oh, Carmine is so cool.
Speaker 3: Well, thank you very much for the introduction, Charles. It's
Speaker 3: a pleasure to be here. So I started using technology
Speaker 3: when I was really, really young. So at the age
Speaker 3: of five, I started to write my first batch scripts
Speaker 3: to automates launching my favorite games, and damn, moment of
Speaker 3: making a computer do exactly what I wanted felt like magic.
Speaker 3: So that sparked lifelong fascination with technology. And at the
Speaker 3: moment I am building like AI tools, you know, I'm
Speaker 3: building chat with work, which is this retrieval advantage generation
Speaker 3: app that you can connect to your Google Drive and
Speaker 3: Slack Notion accounts essentially chat with all your documents, just
Speaker 3: as if Chgpt or Claud.
Speaker 4: Would know anything about them.
Speaker 3: I also co founded fresh Flow, which is also an
Speaker 3: AI based startup that helps supermarkets reduce food waste and
Speaker 3: maximize their profits. And yeah, I've done a bunch of things.
Speaker 4: Essentially.
Speaker 3: I graduated in computer science from University of Bologna, then
Speaker 3: did my master in AI from Free University of Amsterdam,
Speaker 3: and then I've been a data scientist ever since. But
Speaker 3: I think relevant to this podcast is that my last
Speaker 3: project at the Universe Still Bologna, which was for Internet
Speaker 3: and Web Technologies, was actually in Ruby. And the story
Speaker 3: goes that, you know, it was supposed to be in
Speaker 3: PHP and at the time, mind you, yeah, at the time,
Speaker 3: I was already.
Speaker 2: It's a great.
Speaker 3: Story because I was already looking at acer News and
Speaker 3: at the time, all my team, you know, we were
Speaker 3: four people, they all knew PHP and I didn't, right,
Speaker 3: But I followed after News and I saw Ruby on
Speaker 3: rails and this was, I believe, if I remember correctly,
Speaker 3: was twenty ten, so you know the era, right, And
Speaker 3: I was like, so intrigued by Ruby on Rails at
Speaker 3: the time that I convinced my entire team to switch
Speaker 3: to Ruby and rails and we all had so much
Speaker 3: fun that actually some of us went to work for
Speaker 3: other companies that use Ruby and rails. In fact, one
Speaker 3: of my friends I went to work for SoundCloud here
Speaker 3: in Berlin. I even thought about doing that, I even
Speaker 3: went through the application, but then I thought, hey, I
Speaker 3: want to have a master degree and you know, maybe
Speaker 3: look at something else as well. And that's where I
Speaker 3: started looking at AI. And I was a data scientist
Speaker 3: for many years. Whenever I could do anything that was
Speaker 3: related to the web, it was Ruby on rails anyway,
Speaker 3: but mostly Python, mostly machine learning engineering, you know, trying
Speaker 3: to scale also systems, so between you know, the DevOps
Speaker 3: to the modeling level, right, So it was kind of
Speaker 3: a full stack, let's say, machine learning person for a
Speaker 3: long time. And then well I decided to go back
Speaker 3: a little bit to academia. So I went to the
Speaker 3: volunteering Institute in the UK, mostly helping out with an
Speaker 3: organization called that as Signs for Social Good. And then
Speaker 3: I received a call from entrepreneur first and they told
Speaker 3: me like, hey, you have a profile of a potential founder.
Speaker 3: I was like, really, because that's that was my dream
Speaker 3: and I never thought I could do that. So I
Speaker 3: went for it and then I found it fresh Flow
Speaker 3: and now I'm actually.
Speaker 4: Doing my own thing.
Speaker 3: I'm a slopreneur and I'm building chat with work and
Speaker 3: so to connect back to rubylm since now I'm completely
Speaker 3: free and I'm doing you know, like a web app essentially,
Speaker 3: it was obviously done in Ruby and rails and at
Speaker 3: the time and maybe I'm jumping too far, but at
Speaker 3: the time, I wanted to do it because.
Speaker 4: It's all connected.
Speaker 3: I was searching for like libraries for building chat with
Speaker 3: work and I saw some of the libraries out there
Speaker 3: and I wasn't so satisfied with them, so I decided
Speaker 3: to build my own.
Speaker 2: Right Yeah.
Speaker 1: So yeah, it's it's interesting kind of getting into that
Speaker 1: a bit. And some of this reflects, you know, with
Speaker 1: my own background, right, you know, with you know, I
Speaker 1: was kind of a solopreneur. I had a couple of
Speaker 1: virtual assistants, but I mostly just you know, ran things
Speaker 1: myself with the podcast network for I don't know, like
Speaker 1: eight years, and you know, I built most of the
Speaker 1: software that the podcast run on and you know, just
Speaker 1: just on and on and on and on. I'm not
Speaker 1: going to get into kind of the emotional wreckage that
Speaker 1: I went through the last few years, especially with you know, now,
Speaker 1: I'm in a full time job and I love the
Speaker 1: people I work with, and I love the projects I
Speaker 1: work on. But there's just something about working on your
Speaker 1: own thing and having it, you know, come together in
Speaker 1: that way.
Speaker 2: So yeah, yeah, I really identify with a lot of
Speaker 2: what you're talking about. But yeah, I wanted to.
Speaker 1: I wanted to jump in and just start talking about
Speaker 1: Ruby LM, and I know that the like the AI
Speaker 1: space and the LLM space is pretty hot right now.
Speaker 1: And that's one of the things that I've been wanting
Speaker 1: to put together for a while. Is specifically you said, hey,
Speaker 1: you know, you can get in, you can chat with.
Speaker 2: Your dropbox or your Google Driver or whatever.
Speaker 1: And for me, I want the same thing, but I
Speaker 1: want to be able to do it with my podcasts.
Speaker 2: Right It's like, hey, you know, change the host out right.
Speaker 1: So like on Javascriptjbver, we just had somebody, you know,
Speaker 1: he's like, hey, I've got to take some time away
Speaker 1: and write, so things like that. You know, he's the
Speaker 1: last guy that's one of the originals besides me, Right,
Speaker 1: so chat with it about that, or Hey, how's the
Speaker 1: traffic look, or you know, publish this episode on this
Speaker 1: date instead of that date and that kind of a thing.
Speaker 2: And so yeah, I got excited about it.
Speaker 1: And then the local meetup the last like three months
Speaker 1: have all had presentations on how to through API's hit.
Speaker 2: LLM end points basically.
Speaker 1: And you know so, and I've been playing with some
Speaker 1: of that, and we've had Obi Fernandez on to talk
Speaker 1: about Ray and so I got everything hooked up and
Speaker 1: was figuring all that stuff out. Ray is our AIX
Speaker 1: for the people who are listening and trying to go
Speaker 1: find it, and he talked about it before, but I
Speaker 1: found that the documentation didn't give me clear direction on
Speaker 1: how to get everything together the way I wanted. And
Speaker 1: he kind of implies that you can stream the response back,
Speaker 1: but he basically says it's in my book, and I went,
Speaker 1: I have the book, and I went looked in the
Speaker 1: book and I got lost. So but but I did
Speaker 1: have a fundamental thing working to do the presentation at
Speaker 1: the meet up last night, and I was doing a
Speaker 1: chat bot and I just hooked it up to OpenWeatherMap
Speaker 1: dot org and so it was just you know so.
Speaker 1: But then I was showing him, Look, you don't just
Speaker 1: ask it, hey, what's the weather. Here's the latitude longitude.
Speaker 1: But I put in, is a good is it a
Speaker 1: good time right now to go have a picnic?
Speaker 2: And right?
Speaker 1: It came back and said the weather is right. So
Speaker 1: it figured out my cities at this latitude longitude and right.
Speaker 1: And this is the power of the AI, right, is
Speaker 1: that it's smart enough to figure out some of the context.
Speaker 1: And I'm being really long winded and I'm sorry, but anyway,
Speaker 1: what happened was, for some reason, the open router gem
Speaker 1: it was working Tuesday night, and then Wednesday morning when
Speaker 1: I was fiddling with it, it just wouldn't connect and
Speaker 1: it was given.
Speaker 2: Me a four or four error.
Speaker 1: Of course, the error also said that my JWT token
Speaker 1: was wrong, but I was just using an apikey and
Speaker 1: so I think something must have changed on their end
Speaker 1: or there was something funky going on between me and them,
Speaker 1: because it worked fine one day and was broken the next.
Speaker 1: I'm like, I really want to show those guys something working.
Speaker 1: And then I was like, but I'm talking to Carmine tomorrow,
Speaker 1: and so I just went and grabbed it and within
Speaker 1: an hour and a half I had the exact same
Speaker 1: thing working. And so, anyway, this is cool stuff, cool
Speaker 1: cool gem. Anyway, So I'm just curious. I mean, if
Speaker 1: you had any other people come to you and basically
Speaker 1: say this stuff is great, save my bacon. I don't
Speaker 1: know what's been your feedback on Ruby LLM.
Speaker 3: It has been absolutely overwhelmingly positive. I couldn't have imagined
Speaker 3: how good this would have been. So when I released
Speaker 3: the gem, I was like sitting on it for like
Speaker 3: two weeks and on you know zero point one point
Speaker 3: zero pre forty nine or something like that.
Speaker 4: Right, I had to release a lot of pre release.
Speaker 3: Versions for myself, and I thought, hey, maybe I should
Speaker 3: release it right now instead of waiting until I get
Speaker 3: everything downe with chatwood work and you know, doing it
Speaker 3: let's say the DHH way where you released it after base, Right,
Speaker 3: that was my original plan. I was like, well, what
Speaker 3: can go wrong? And so I released it and I
Speaker 3: made some write posts and immediately in two days we
Speaker 3: got to two hundred and sixty style and my goal
Speaker 3: was to get to three hundred, so already kind of
Speaker 3: achieved my goal in no time at all, and then
Speaker 3: someone posted it on Accre News and I remember that
Speaker 3: night I was playing in a club.
Speaker 4: So I'm also a DJ onlands, you know, my free time.
Speaker 3: And yeah, and I come back home, I go to sleep,
Speaker 3: and around like six am in the morning, I get
Speaker 3: so many notifications of discord, which for some reason I
Speaker 3: could hear. I didn't like mute my computer and I
Speaker 3: go and check and people were saying like, oh my god,
Speaker 3: you made it to a front page of Accre News.
Speaker 3: How amazing. And in one we got to one point
Speaker 3: seven thousand stars. Wow, which was absolutely overwhelming. Yeah. So
Speaker 3: the reception has been fantastic. I've made new friends. There's
Speaker 3: some people that I talked to now kind of regularly.
Speaker 3: I have an audience on Twitter, which I didn't really
Speaker 3: have before. So it has been absolutely fantastic. Just mostly
Speaker 3: I would say ninety nine point nine nine nine percent
Speaker 3: and everything has been super positive.
Speaker 2: Very cool, very very cool.
Speaker 1: Yeah, I have to say, yeah. I mean my feedback
Speaker 1: is is that it's really simple to use. I think
Speaker 1: a lot of people get worried that oh it's AI
Speaker 1: and I have to know all these things and.
Speaker 2: No, you just tell it. I want to chat, you know,
Speaker 2: you have to do a little.
Speaker 1: Bit of setup so that it knows how to connect
Speaker 1: open ai or something. But that was for me at least, literally,
Speaker 1: here's my API key go crazy so.
Speaker 2: Great.
Speaker 3: Yeah, that was exactly the type of design that I
Speaker 3: wanted to have. So basically the reason why I made
Speaker 3: Ruby l LAMB is because when I looked at the
Speaker 3: other libraries, they essentially fit in two different categories. Right,
Speaker 3: So you have the libraries that are mostly done by
Speaker 3: Alex Rudahl. Shout out to him, Oh.
Speaker 1: Basic, great guy, that his library is amazing. It does everything, yes, exactly.
Speaker 3: So his library is his main library is Ruby, open ai, right,
Speaker 3: and then he has also the Entropic gem. They are
Speaker 3: lightweight layers on top of the HDCP calls, right. So
Speaker 3: what that means is that it's really easy to support
Speaker 3: every single function from the l lamps, but it's also
Speaker 3: really hard to then switch between different l lamps.
Speaker 2: If you have to switch.
Speaker 4: And one of my use cases for chat with work.
Speaker 3: Exactly, and all of the response objects, right, they're all different.
Speaker 3: So when I was designing Chat with Work, I wanted
Speaker 3: to have something that could make it easy to switch
Speaker 3: between models and I was thinking either mid conversation as
Speaker 3: a you know, north star, and so I needed to
Speaker 3: have something that was a unified interface.
Speaker 4: So I looked at the other.
Speaker 3: Type of libraries, you know, so like lang chain RB specifically,
Speaker 3: and I saw that I wanted to have some other features.
Speaker 3: The design wasn't exactly what I would have done for
Speaker 3: a library, but I implemented it. But then I wanted
Speaker 3: to have the pricing information, the model information, and listing
Speaker 3: of the models. So I started modifying it, and I
Speaker 3: started to go into the code of lang chain and
Speaker 3: it is quite a big jem. It's not as it
Speaker 3: has a lot of code. Yeah, so already you know,
Speaker 3: it was quite difficult to orienter yourself around, and it
Speaker 3: is quite I would I want to say, it's a
Speaker 3: little over engineered for my taste, right, Okay, But I
Speaker 3: started modifying it, and also like, it depends on the
Speaker 3: Alex's gems, right, so in order to have that same feature,
Speaker 3: I need to modify the entropic jam. Then also lun
Speaker 3: chain and also lun chain or be rails right, because
Speaker 3: there's a separate JEM for the rails integration.
Speaker 4: I was like, is this really what I want?
Speaker 3: Also, I noticed other things that I didn't like about it,
Speaker 3: which is that it put the providers first. And I
Speaker 3: think philosophically speaking, we don't really care about providers, right,
Speaker 3: people care about models.
Speaker 4: Where do they come from?
Speaker 3: Maybe it's interesting in terms of pricing, But if you
Speaker 3: want to just chat with a model, you want to
Speaker 3: just go like Ruby lam or Chat model and just
Speaker 3: go with it, right. I think that's the that's the
Speaker 3: beauty and why people have been finding Ruby LM so interesting.
Speaker 3: So one question I have puts the models first?
Speaker 4: Yeah, one question I.
Speaker 1: Have on that real quick is so initially I was,
Speaker 1: like I said, I was using open router and open
Speaker 1: router it kind of does the you can switch models
Speaker 1: on their service, right, so you're you can pick them,
Speaker 1: but you don't have to set up all the different
Speaker 1: service providers. So why not go with something like that?
Speaker 4: That's a good question.
Speaker 3: Well, the open router basically locks you in into that environment,
Speaker 3: so you need to use open router. What if open
Speaker 3: router dies or if providers like alge my Little Story
Speaker 3: or you four four errors right exactly. So I didn't
Speaker 3: want to be locked in into anything. I wanted to
Speaker 3: use truly open source software just like you know, Ruby
Speaker 3: and rails are and that's why I decided, you know what,
Speaker 3: this is probably a good idea to do. And yeah,
Speaker 3: there were other reasons actually why I thought that the
Speaker 3: existing gems were not of my taste. For example, streaming,
Speaker 3: even though like line chain or b has, you know,
Speaker 3: I unified the interface for most things. Streaming is where
Speaker 3: it gets complicated. So whenever you get a chunk, you
Speaker 3: actually get a chunk that is exactly the object that
Speaker 3: is retrieved from the API itself, So you need to
Speaker 3: parse the chunks in a different way. If you have
Speaker 3: an entropic model, then you have an opening eye model.
Speaker 3: And so all of these problems combined made it so
Speaker 3: that I wanted to build my own.
Speaker 1: Nice Well I'm glad you did because it's it's freaking awesome.
Speaker 1: So yeah, let's let's dive in a little bit. So
Speaker 1: you mentioned models, and if you get into like the
Speaker 1: getting started, I mean, I only hooked up open AI,
Speaker 1: but it sounds like you want people to be able
Speaker 1: to switch seamlessly. So if I put in an open
Speaker 1: aike and an Anthropic key, and one or two other providers,
Speaker 1: maybe a Gemini key, I can switch between those seamlessly.
Speaker 4: That's right. That's right.
Speaker 3: And the way that this works is that Ruby ALAM
Speaker 3: essentially has two different parts of the code. So you
Speaker 3: have the clean interface with the user, right, So you
Speaker 3: have the chat object, you have the message object, you
Speaker 3: have the content object, et cetera, et cetera, and then
Speaker 3: you have the mess part of the code, which is
Speaker 3: the provider code, right, and what the provider code does
Speaker 3: I'm saying messy, but I think it's decent. But it
Speaker 3: has all the messy parts of actually communicating with the
Speaker 3: provider interfaces and translating between you know, the Ruby LM
Speaker 3: internal representation and the provider representation. It sounds pretty straightforward,
Speaker 3: and so I implemented it that way so that you
Speaker 3: can have a very simple interface but actually change between models,
Speaker 3: because then it's just a question of parsing and retrieving
Speaker 3: data and formatting it for their for the format not
Speaker 3: really about you know, a specific representation that only works
Speaker 3: for the specific provider. I definitely didn't want to do that,
Speaker 3: and the way to achieve that was to essentially study
Speaker 3: the the APIs from you know, open Ai and Entropic,
Speaker 3: because that's where I started, and figure out what is
Speaker 3: the best way that they implement things, and how would
Speaker 3: I implement it in terms of the objects, the Ruby objects.
Speaker 4: That I want to have in my library.
Speaker 3: And so I picked and choose, you know, how to
Speaker 3: represent tool calls for example, and that's always now represented RUBYLM.
Speaker 4: And then if the auto.
Speaker 3: Provider that doesn't represent it this way, you know, will
Speaker 3: get a different format method in their modules in order
Speaker 3: to do that.
Speaker 2: That's awesome.
Speaker 1: So one thing that I'm wondering about is do the
Speaker 1: different models have different capabilities that you have to adjust
Speaker 1: for or let people know, Hey, you can do this
Speaker 1: with I don't know, GPT and Anthropic, but not with
Speaker 1: Gemini or something like that.
Speaker 3: That's right, Yeah, that's a great question. So I implemented
Speaker 3: them all the registry. That's actually where this whole thing
Speaker 3: about RUBYLM started right when I wanted to implement the
Speaker 3: model registry for for line chain. And it basically has
Speaker 3: a bunch of different capabilities per each model. So for example,
Speaker 3: it would know if a model has a function support,
Speaker 3: if we would have a Jason mode soon when we
Speaker 3: will actually have structured output support, if it will not,
Speaker 3: if we will have structured output support, what is the
Speaker 3: context length? What is the maximum amount of tokens for
Speaker 3: the output, you know, and all that information is put
Speaker 3: together in adjacent file that we distribute directly with the library,
Speaker 3: because one of the design principles is to be a
Speaker 3: turnkey solution. So you just as you as you did,
Speaker 3: like you installed it, you put your opening ike and
Speaker 3: there you go. You have a chat interface if you
Speaker 3: want to write at least in the common line. And
Speaker 3: this is exactly the experience that I wanted to have
Speaker 3: and I want people to have. So if there is
Speaker 3: a model that doesn't have let's say, function calling support,
Speaker 3: we will throw an error because we will know it
Speaker 3: from the from the models Dore Jason. And the way
Speaker 3: that at the moment we make the models Dore Jason
Speaker 3: is to query the APIs from the listing model APIs
Speaker 3: from open EI, from Entropic, Gemini, et cetera, and then
Speaker 3: augmented with our own code that figures out based on
Speaker 3: the model family, like that's the model family has function calling,
Speaker 3: that's the small family has you know, Jason mode, what's
Speaker 3: the context length, et cetera, et cetera.
Speaker 4: And you will find that in the code in the.
Speaker 3: Capabilities dot r B files in the module in the
Speaker 3: model sorry, in the provider modules. Now, however, since I've
Speaker 3: grown a little bit tired of you know, having to
Speaker 3: maintain that code and have you know, also just parsing
Speaker 3: the provider's web pages, which is actually kind of costly,
Speaker 3: especially when you do it with lams. And there is
Speaker 3: some code in Ruby LAM to do that in the tasks,
Speaker 3: but I remember the last time I used it, I
Speaker 3: spent twenty bucks just to parse all of the new
Speaker 3: open Ai model pages. Well, since I grown tired of
Speaker 3: doing that, I actually contacted an ex employee of mine
Speaker 3: from fresh Flow who now is also an entrepreneur, and
Speaker 3: he has a company called Parsera and they're basically doing
Speaker 3: scraping of web pages using l lams and then they
Speaker 3: returned that as data in Jason and I thought that
Speaker 3: that would have been the perfect use case in order
Speaker 3: to showcase their product, and we would get like an
Speaker 3: API for free in order to just you know, query
Speaker 3: about the capabilities of any LM in existence, since unfortunately
Speaker 3: the providers they don't provide that at the moment. Right,
Speaker 3: Like when you get a list model endpoint response, you
Speaker 3: just have most of the time just the model ID
Speaker 3: and sometimes the capabilities. I think Google does this and
Speaker 3: some other small things, but not the full you know,
Speaker 3: model info object that we have inside Ruby ELM. So
Speaker 3: basically what Persera is now building and they are like,
Speaker 3: I wouldn't say ninety percent. There is this API that
Speaker 3: you can just with a get request query and you
Speaker 3: have a beautiful interface, you know, a beautiful Jason object
Speaker 3: with all of the different capabilities, all the prices, all
Speaker 3: well formatted for all the major you know, for providers
Speaker 3: that we have in Ruby l M. And so I
Speaker 3: will scrap that code. I'm really excited when I remove code.
Speaker 3: So I'll scrap that code and simply like called that
Speaker 3: that API from from then on.
Speaker 4: So I'm really excited for that.
Speaker 1: So you're not going to have a Jason in your
Speaker 1: in the gym anymore. It's going to be an API
Speaker 1: call or the APA call. API call will generate the Jason.
Speaker 4: Rather the second.
Speaker 3: Yeah, I still want to have the turnkey solution so
Speaker 3: that anybody that installs it, maybe doesn't have Internet, I
Speaker 3: don't know, doesn't want to do that API call of
Speaker 3: course the Internet, then it can still actually have that
Speaker 3: installed and without having an extra API call every time.
Speaker 3: And it's refreshed models for example.
Speaker 2: Yep.
Speaker 1: So yeah, let's dive into what Ruby LLM actually provides,
Speaker 1: because I think people can now kind of be comfortable
Speaker 1: with Hey, look, it does all these models. It knows
Speaker 1: which models will do which things, and so if you
Speaker 1: try and do something with one model that it just doesn't,
Speaker 1: you know, it's not compatible with the model, It'll tell
Speaker 1: you and then you can do something different.
Speaker 2: So let's start with.
Speaker 1: The chat because that's where I started, and I was
Speaker 1: just blown away that it was that easy, because you
Speaker 1: literally just do you instantiate a chat object and then
Speaker 1: you do it dot ask and then you get content
Speaker 1: back and you can post it. And what I found
Speaker 1: was that I had to pull in the common mark
Speaker 1: gem and tell common Mark to convert it to HTML
Speaker 1: and I was set. That was like, it is there
Speaker 1: anything more to it?
Speaker 2: Am I missing it? I mean, we get into tools
Speaker 2: in a minute.
Speaker 3: But so in the chat you can also specify contents, right,
Speaker 3: so you can send an image with it, you can
Speaker 3: send the audio file with it, you can send a
Speaker 3: PDF with it.
Speaker 4: And also there I tried.
Speaker 3: To make the design of the library feel super easy
Speaker 3: so that if you need to analyze an image it's
Speaker 3: just a chat dot ask what's in this image? Come
Speaker 3: up with colon open curly braces image and then the
Speaker 3: path through your image or the HCP link to your image.
Speaker 3: And that also I think follows the same philosophy or
Speaker 3: rubylm of convention over configuration and you know, just trying
Speaker 3: to make people's life easier. Then you can also stream
Speaker 3: the responses in real time, right, and you can simply
Speaker 3: use Ruby blocks to do that. So it's super simple.
Speaker 3: It's just a chat dot ask whatever you want to ask,
Speaker 3: do chunk and then print chann dot content and the
Speaker 3: chunk is a chunk object and it looks like a message.
Speaker 3: In fact, if you look at the code is pretty
Speaker 3: much exactly like a message. I think it's just inherits
Speaker 3: from message and that's it. And it will if you
Speaker 3: just print the contents, it will simply print that chunk.
Speaker 3: But then in the background we're actually accumulating the response
Speaker 3: and so the result of that chat dot ask would
Speaker 3: actually be a full, fully blown message and it will
Speaker 3: be also persistent in the background in the in the
Speaker 3: memory that Ruby LM do chat has about messages, Yeah,
Speaker 3: what are capabilities that have? So it can also generate images.
Speaker 2: Because I mean you have so you have the capability.
Speaker 1: Yeah, and it just does it in a do loop
Speaker 1: and yeah, it has the chunk content and it has
Speaker 1: like a print or puts.
Speaker 2: In kind of your just general Ruby application.
Speaker 1: But I don't know, people who listen to this show
Speaker 1: know that I'm totally in love with Turbo.
Speaker 2: It was like, where have you been on my life Turbo?
Speaker 1: But you have examples on how to push the stuff
Speaker 1: out to Turbo as well, right, So one one question
Speaker 1: that I have with this because it shows a job
Speaker 1: and the job uses the Turbo streams channel broadcast right
Speaker 1: to update the message, which is you know what you want?
Speaker 1: I'm just I'm trying to figure out exactly where this
Speaker 1: gets called from.
Speaker 2: Does it just run periodically? And then so.
Speaker 3: How it works in chatwood work is that there is
Speaker 3: a job. It's not called chat stream job, it's called
Speaker 3: slightly differently, but there is that job similar to the
Speaker 3: job in the guides, and what it does is to
Speaker 3: stream via Turbo, essentially appending the chunk content to a
Speaker 3: div that is hidden, and then I use mark dot
Speaker 3: j as in order to doing the rendering of the
Speaker 3: of the actual message. The reason why I do this
Speaker 3: is because I used to have everything server side and
Speaker 3: do all of the rendering of markdown server side and
Speaker 3: then send you know, the HTML over But the problem
Speaker 3: is that when you get to really big messages is
Speaker 3: starting to stream every single time you have a new chunk,
Speaker 3: which may be even every single word, or you know,
Speaker 3: even sometimes less than one word, And you can imagine
Speaker 3: that refreshing the message so fast and and doing it
Speaker 3: so many times it actually bogs down your connection. So
Speaker 3: at some point I noticed that my connection was downloading
Speaker 3: like multiple megabytes per second right of HTML. So it
Speaker 3: didn't make any sense to me. So what I'm doing
Speaker 3: is to simply append the chunk to that specific dive
Speaker 3: and then marked will look at that div and it
Speaker 3: would actually do the rendering in an udative.
Speaker 4: That I actually displayed to the user.
Speaker 3: And that also gives me the possibility to do a
Speaker 3: copy functionality in which the when you copy the message,
Speaker 3: it would actually copy the markdown version of the message,
Speaker 3: which I think is so much more useful than getting
Speaker 3: the full blown HTML from the page. And so how
Speaker 3: do I call that job? Well, I call it from
Speaker 3: the the actual controller. So whenever the user asks something,
Speaker 3: you know the controller, which is fire that job in
Speaker 3: the background, and the job will simply process whatever comes
Speaker 3: from the OLM.
Speaker 1: Okay, so in the controller then I have something that effectively,
Speaker 1: I mean, do I tell it I'm streaming it? I
Speaker 1: guess I just have the do block and so then
Speaker 1: every time gets executive executed, sorry, it calls the job
Speaker 1: or how I'm just trying to wrap my brain around this.
Speaker 3: Right, So basically the controller would simply you know, maybe
Speaker 3: you want to create a message in the controller, do
Speaker 3: whatever you want, but in the end it needs to
Speaker 3: call that chat stream job so that you can process
Speaker 3: the streaming in the background because you can't have that
Speaker 3: running in the controller because it's such a long running thing.
Speaker 3: And that's exactly how I do it in chatwood work.
Speaker 2: Okay.
Speaker 1: So basically what you've got going on is, yeah, you've
Speaker 1: got your and it shows it in the chat right,
Speaker 1: So you do the chat dot ask with the do
Speaker 1: inside the do you call the job the job is
Speaker 1: streaming stuff out through turbo and then you have marked
Speaker 1: basically putting it in. Yeah, that's cool, that's correct. When
Speaker 1: your controller action finishes, then it renders a turbo stream whatever.
Speaker 3: Yeah, so if you set up your turbus, your broadcasts,
Speaker 3: you know, inside your application record, you know, on your models. Basically,
Speaker 3: it would also update the whole message when it's finished,
Speaker 3: and you can also tell it to you know, update
Speaker 3: of typing indicators and perhaps the button you know that
Speaker 3: has to change from a send to a stop. All
Speaker 3: of that you can do actually really really easily with Rails,
Speaker 3: and no wonder because Rails is so fantastic for these things.
Speaker 3: One thing that I think is interesting about the jobs,
Speaker 3: if I may, is that what I noticed, especially using
Speaker 3: things like solid que which use threads, is that you
Speaker 3: have a certain number of threads that you can use maximum, right,
Speaker 3: So you need to set it up and you have
Speaker 3: one worker per process as far as I remember, and
Speaker 3: then you need to set up how many threads per
Speaker 3: process you have as workers.
Speaker 2: Right.
Speaker 3: So even if you set it up in a way
Speaker 3: that it will process many many connections at the same time,
Speaker 3: many many chat streaming jobs at the same time, you
Speaker 3: will always be limited by that. And the limitation is
Speaker 3: that every single thread in Ruby is eight megabytes. And
Speaker 3: also you're occupying a whole slot, right, So you can
Speaker 3: imagine an application like chat with work it has not
Speaker 3: only the chat streaming job, but it also has the
Speaker 3: process document jobs, right, and the discovery document jobs and
Speaker 3: all of these and there's many, many jobs running at
Speaker 3: the same time, but they're all very lightweight, right that
Speaker 3: they're not doing much CPU wise, they're mostly like sending
Speaker 3: things back and forth through APIs and then you know,
Speaker 3: the grant of the work is probably done in postcris,
Speaker 3: you know, because I use postgris also as a vector storage.
Speaker 3: So they're all io bound, not CPU bound, and they're
Speaker 3: consuming one thread every single time there is a job
Speaker 3: running at the same time, and so that was not ideal.
Speaker 3: And in the Python world and in the JavaScript world,
Speaker 3: you know, for io bound operations, everybody would use a
Speaker 3: syn ko for example in Python, right, or a sync
Speaker 3: wait in JavaScript. And so what I noticed is that
Speaker 3: now I came back to Ruby, there's a bit of
Speaker 3: a lack of understanding or you know, knowledge about using
Speaker 3: concurrency instead of using parallelism, well fake parallelism with threats,
Speaker 3: And so I delved deeper into it. And actually there
Speaker 3: is a beautiful ecosystem of asynchronous gems such as a sync,
Speaker 3: a sink cable, ACNK job, anything that is built by
Speaker 3: Samuel Williams I aquatics, you know, also called socketry, is
Speaker 3: absolutely fantastic. And what I also noticed is that Ruby
Speaker 3: is probably the best language to do concurrency because in
Speaker 3: most other languages you need to uppend these or prepense
Speaker 3: these keywords like a sync await every time you need
Speaker 3: to do such an operation, But in Ruby you don't
Speaker 3: need to because if the C level operation actually is
Speaker 3: fiber aware, then you get fiber awareness for free. So
Speaker 3: you just need to essentially use or wrap whatever code
Speaker 3: you have into let's say on a sync block, and
Speaker 3: if you do it right, it will work in parallel
Speaker 3: in concurrent.
Speaker 4: Actually awesome.
Speaker 1: So one question I have on that is, because I've
Speaker 1: never even used this before, can you run this and
Speaker 1: solid que side by side and then just throw the
Speaker 1: acinc jobs at one and any irregular stuff at the
Speaker 1: other or do you just put everything into an acin job.
Speaker 3: At the moment, how I'm using it is everything is
Speaker 3: in a saying job, but you can totally right random
Speaker 3: side side by side. I think there is a Q
Speaker 3: adapter that you can use in each single job, so
Speaker 3: you just you know, you have the que as the
Speaker 3: fault for example, right, so you can have a Q
Speaker 3: adapter and you just say solid Q for certain jobs,
Speaker 3: and you can do a saying job for your jobs.
Speaker 1: That's a good tip because yeah, if I'm going to
Speaker 1: build a podcast assistant, I'm probably gonna run into the
Speaker 1: same constraints, and so that is really helpful.
Speaker 3: Right, Yeah, Yeah, I'm no, it's just saying that I
Speaker 3: basically discovered this this fly, and I think that it's
Speaker 3: really a matter of communication to the Ruby community out
Speaker 3: saying like, hey, there's not just this way of doing
Speaker 3: concurrency in paralyism that we're all familiar with, you know,
Speaker 3: like threads, but there's also fibers, and you know, there's
Speaker 3: other software that you could use that even though you
Speaker 3: don't have io bound operations, you can simply replace Puma
Speaker 3: with Falcon and have you know, five x speed ups.
Speaker 3: Sometimes it's really so much faster it's noticeable. I didn't
Speaker 3: do any benchmarks, but it's really Yeah.
Speaker 1: I've talked to Samuel before and I got the impression
Speaker 1: that some workloads really lend themselves to it and some
Speaker 1: don't as much.
Speaker 2: But yeah, Falcon is really cool.
Speaker 1: One other thing that I'm curious about here, So when
Speaker 1: I built my app the other night, with the chat.
Speaker 1: I realized that in my controller, I was instantiating a
Speaker 1: new chat every time writers doing RUBLLM dot chat. And
Speaker 1: what I want is I want an ongoing conversation with
Speaker 1: you know, so that it kind of maintains the context
Speaker 1: and things like that. And there's an example in here
Speaker 1: where you're basically what I would do, like in a
Speaker 1: RAILS console or IRB or something where you know, it's
Speaker 1: like chat dot ask and then you send it another
Speaker 1: message chat dot ask and it looks like that maintains
Speaker 1: that context. But how do I do that between requests
Speaker 1: when I'm hitting the controller anew every time? But I
Speaker 1: still want it to be part of that same.
Speaker 3: Chat, right, I think this is really important to say.
Speaker 3: So the RAILS integration essentially persistent messages, Yeah, persistent messages
Speaker 3: inside active record models.
Speaker 4: So we have essentially some helpers which are called.
Speaker 3: Acts as chat, acts as message access tool call that
Speaker 3: you can drop in into your models. You know, if
Speaker 3: it's called chat the model for chats, then you just
Speaker 3: say class you know chat inheriting from application record and
Speaker 3: then just say access chat and you have essentially all
Speaker 3: the connection to also the message model. You have persistence
Speaker 3: and every single method that we have in the classic
Speaker 3: chat interface, you know in plain old Ruby objects.
Speaker 4: You would have it also in rails.
Speaker 3: So if you do chat dot ask, it will first
Speaker 3: create a message with the with the user role and
Speaker 3: save it and then also call the API for whatever
Speaker 3: LLLM you're using, retrieve that and then put it inside
Speaker 3: a new message anderson is that and then return that
Speaker 3: message at the end. So it looked like exactly the
Speaker 3: same interface for PLANEOAR Ruby objects, but you have the
Speaker 3: persistence done not just in memory but also in your database.
Speaker 1: Right, what if I'm not using rails or not using
Speaker 1: active record, is there a way to do this like
Speaker 1: in Sinatra or something like that and that anyway?
Speaker 3: Right, I haven't used Sinatra recently. The last time I
Speaker 3: used it was more than ten years ago. Maybe it's right,
Speaker 3: So I wouldn't know how to actually build that out,
Speaker 3: but it's possible to. If it's possible to make an
Speaker 3: adapter for whatever type of you know, storage you have,
Speaker 3: it should be pretty easy and people can just look
Speaker 3: at the access code and see how we've done it.
Speaker 2: Right, Yeah, that makes sense.
Speaker 1: You can't just look at his chat and then see
Speaker 1: what she did. Very cool let's talk about tools for
Speaker 1: a minute, because this was one of the other things
Speaker 1: I showed off the other night was essentially I created
Speaker 1: a tool that just checked the weather, and yeah, I
Speaker 1: didn't use the same service you did, but it was
Speaker 1: basically the same thing. I used Open weather Map instead
Speaker 1: of Medio or whatever you used, but yeah, it was
Speaker 1: the same deal. Do you want to explain to people
Speaker 1: what tools are and how they work?
Speaker 2: Right?
Speaker 3: So, I think tools, especially with MCP, they became the
Speaker 3: most hot thing talked about at the moment in AI. Right,
Speaker 3: So for people who are not familiar essentially with LLM models,
Speaker 3: you can create a tool in your own language and
Speaker 3: then tell them like, hey, this is the way that
Speaker 3: this tool works. So this is the description of the tool,
Speaker 3: and these are all the parameters that we need. These
Speaker 3: are the parameters that are optional, and these are all
Speaker 3: the descriptions for every single parameters. And then the LM
Speaker 3: will decide when to use that tool.
Speaker 2: Right.
Speaker 3: So let's say you created this tool for weather, but
Speaker 3: then the user talks about something else, wants to talk
Speaker 3: about MacBooks, right, the LLLM will not actually call the
Speaker 3: tool every single time. And I think the fundamental difference
Speaker 3: between tools and the classical RAG for example that we
Speaker 3: used to have, you know, even just a year ago
Speaker 3: at this point is that you don't always do retrieval
Speaker 3: advantage generation. So basically you don't always search, you know,
Speaker 3: the documents and then put them in the context of
Speaker 3: the message every single time you have a new message. Basically,
Speaker 3: tools enable agentic aid I because the AI can be
Speaker 3: an agent and really figure out if it needs to
Speaker 3: use this tool or not. And they are now like
Speaker 3: all the age with MCP, So MCP, for it's not familiar,
Speaker 3: is model context protocol, and it's a way to bunch
Speaker 3: up a bunch, you know, a bunch of tools into
Speaker 3: one single interface and give that interface to the l
Speaker 3: l M so that the l M can figure out
Speaker 3: you know, hey, I can now access the file system,
Speaker 3: I can now use VIM, I can now use vs code,
Speaker 3: I can now use I don't know whatever API. And
Speaker 3: it's essentially an implementation of tools in a way, right,
Speaker 3: but just in a different API.
Speaker 2: Yeah.
Speaker 1: Absolutely, So this is where like with my my thing
Speaker 1: with my podcast stuff is you know, it's not just
Speaker 1: which episodes were most popular or what other thing do
Speaker 1: I want you to look up, but it's hey, reschedule this,
Speaker 1: or hey, invite this other person or do this other thing,
Speaker 1: and it can through the API go and talk to
Speaker 1: the other system and make it do something for me.
Speaker 4: That's right.
Speaker 3: So it opens up a lot of possibilities for using
Speaker 3: lms for all kinds of disparate features.
Speaker 2: Right.
Speaker 3: You can save things, you can retrieve things from the Internet.
Speaker 3: You can imagine for example Google search or like document search,
Speaker 3: or creating new blog posts. It's just I think the
Speaker 3: possibilities are endless because you're letting.
Speaker 4: The AI use your code.
Speaker 3: Of course, you need to be aware that also ais
Speaker 3: are generally benevolent, but you know they could use your
Speaker 3: tool badly.
Speaker 4: They could cause wreckage.
Speaker 3: So don't use evolved, don't use system right, but try
Speaker 3: to be kind of careful about it. But in general,
Speaker 3: it's a fantastic way to make ELM super super powerful.
Speaker 1: Yeah, and I think tool design and security with the
Speaker 1: tools and things like that might be outside the scope
Speaker 1: of this episode, but yeah, I mean I've had people
Speaker 1: ask me too, it's, hey, well, how do you make
Speaker 1: sure that the tool is only doing this stuff within
Speaker 1: the scope you want? So, for example, in my case
Speaker 1: with podcasting, it's how do I make sure it's only
Speaker 1: doing stuff on one person's podcast and not another, or
Speaker 1: things like that.
Speaker 2: And so.
Speaker 1: Because you know, we've seen people do what looks like
Speaker 1: SQL injection attacks except as prompt stuff, and it says, well,
Speaker 1: but pretend you can do it essentially, right, they talk
Speaker 1: the LM around to Okay, I'm going to pretend I
Speaker 1: can do it, and so I'm going to act like
Speaker 1: this other user even though I'm not technically supposed to
Speaker 1: do that. And there are only so many safeguards you
Speaker 1: can put into place. But if you don't have a
Speaker 1: tool that allows them to impersonate anyone else, and you've
Speaker 1: locked down the access to the user and it's data
Speaker 1: to a methodology you can control, like find the current
Speaker 1: user uses the session to validate the user instead of
Speaker 1: looking the user up by email address, for example, and
Speaker 1: it doesn't have a tool to look up the user
Speaker 1: by email address unless you need that feature for admins.
Speaker 1: But then you use a different chat agent for that
Speaker 1: and give it those tools there Now you're getting into
Speaker 1: so you have to be careful how you use them.
Speaker 1: Like you said, for sure, try and think ahead on
Speaker 1: how people may trick your agent into doing something that shouldn't.
Speaker 4: That's right, That's exactly right.
Speaker 2: But yeah, the tools are cool.
Speaker 1: One question I had on this, so I built a
Speaker 1: tool for the weather, and then I was thinking about
Speaker 1: it and I was like, well, what if I wanted
Speaker 1: So then I was showing off, not what's the weather
Speaker 1: in Lehigh, Utah, which is where I live, but you know,
Speaker 1: is it a good time to go have a picnic
Speaker 1: in Lehigh, Utah?
Speaker 2: Right?
Speaker 1: And it's smart enough to go, Okay, I need to
Speaker 1: know what Lehi is, I need you know. I figured
Speaker 1: out the latitude and longitude. I sent it to the
Speaker 1: Weather Service. It told me what the weather was, and
Speaker 1: then it said, yeah, it's kind of a nice evening
Speaker 1: for a picnic, right because it was sixty degrees and
Speaker 1: you know, the wind wasn't too bad and it wasn't raining.
Speaker 1: But then I was thinking, okay, well, what if I
Speaker 1: wanted to ask it what day this week should I
Speaker 1: go for a picnic? Then it has to have another
Speaker 1: tool that will do the forecast look up. And so
Speaker 1: I can imagine that if you're creating tools, you could
Speaker 1: wind up with a lot of tools, right, collection of them.
Speaker 1: I'm wondering if there's a way to like group them together.
Speaker 1: So it's like, hey, here's all the calendar stuff, here's
Speaker 1: all the drop box stuff, here's all the drive stuff.
Speaker 3: I think that's exactly what Entropic was trying to do
Speaker 3: with MCP. It's probably exactly that use case. And essentially,
Speaker 3: as I said, MCP is this way of bunching up
Speaker 3: different tools into one single API, and it tells the LLM, hey,
Speaker 3: I have all these things that you can do now.
Speaker 3: And you can even run like MCP servers in your
Speaker 3: own computer for example, in order to you know, connected
Speaker 3: the LM with your files.
Speaker 4: Use it more.
Speaker 3: You know, cloud code Sorry, cloud code uses MCP as
Speaker 3: far as I know, and maybe wrong on this, but
Speaker 3: I think it uses MCP to actually connect to your
Speaker 3: files and your file system and to look up stuff
Speaker 3: and to change it. So it is a valid point,
Speaker 3: you know, and you can win up with a lot
Speaker 3: of tools and actually they consume a little bit of
Speaker 3: your context window. However, now with the recent models, you know,
Speaker 3: I don't know if you heard, GPT four point one
Speaker 3: just came out and it has a one million token
Speaker 3: context that.
Speaker 4: Is pretty big.
Speaker 3: Yeah, you can do a lot of stuff with one
Speaker 3: million tokens and I've been using Gemini two point five
Speaker 3: pro extensively lately and it's a fantastic model super bowlish
Speaker 3: on on Google lately, and it also has you know,
Speaker 3: million context window if not two million, I don't remember that.
Speaker 2: Yeah, that makes sense.
Speaker 1: The other thing that I've seen done and we talked
Speaker 1: to Obi before MCP was really a thing, and what
Speaker 1: he was doing was he actually had different.
Speaker 4: Agents that had different jobs, right, and.
Speaker 1: So then instead of providing this huge list of tools,
Speaker 1: he would he would have one tool that was essentially said,
Speaker 1: write a prompt for the calendar agent, right, and so
Speaker 1: then it would prompt the calendar agent and say, hey,
Speaker 1: can you invite so and so to this calendar event?
Speaker 1: And then on the calendar agent would then say, oh,
Speaker 1: this means I have to go find the event.
Speaker 2: And update it with you know, with the user or whatever.
Speaker 1: So that that was his way around it. I kind
Speaker 1: of liked that approach anyway, because then I really can
Speaker 1: kind of box in each of the agents with just
Speaker 1: a small set of responsibilities and capabilities. But yeah, that
Speaker 1: was his way around it, and I think that's kind
Speaker 1: of an elegant solution. But I can imagine if I
Speaker 1: have a app tools directory in my application, that I
Speaker 1: might have a lot of those in there, depending on
Speaker 1: how many capabilities I'm giving to my application right or
Speaker 1: to my agent.
Speaker 2: Right.
Speaker 4: However, I think that this.
Speaker 3: Is a totally fine design solution for the past models
Speaker 3: that we had, you know, even last year, where two
Speaker 3: hundred thousand tokens context windows were pretty big, right.
Speaker 2: Yeah.
Speaker 3: I think we'll probably see in the future, you know,
Speaker 3: just having one single model that would actually handle all
Speaker 3: the tools you have, because the context window is big enough,
Speaker 3: and it would be super powerful and it can do
Speaker 3: so many things, it will actually.
Speaker 4: Probably blow people's minds.
Speaker 1: Yeah, with a million tokens. Yeah, I can imagine that
Speaker 1: that's that's a ton. It's interesting because I was working
Speaker 1: with a model this earlier this year, and I think
Speaker 1: the context window had.
Speaker 2: Like two hundred thousand tokens. I think something like that
Speaker 2: one hundred and fifty two hundred thousand.
Speaker 1: And the issue that I was running into there was
Speaker 1: that I was pulling in bills during the Utah legislative
Speaker 1: session and getting summaries done and things like that with it,
Speaker 1: and there were like four.
Speaker 2: Bills that were too big. It blew, it blew past
Speaker 2: the context window, and so I.
Speaker 1: Tried breaking it up, but ultimately that's the other great
Speaker 1: thing about a lot of this stuff is that the
Speaker 1: context windows are getting bigger, and what that means is
Speaker 1: that the prices are coming down for models that have
Speaker 1: larger context windows that were kind of the run of
Speaker 1: the mill, state of the art ones before. And so
Speaker 1: you can get away with a lot more or you
Speaker 1: can just like in your what you're saying here with
Speaker 1: the chat is that I can go in in midstreams
Speaker 1: it comes back and says, hey, this was way too big.
Speaker 1: I can go Okay, just this time, go use bigger
Speaker 1: model and pay the extra money.
Speaker 2: Right Yeah.
Speaker 3: I think these are two trends that we've seen so far.
Speaker 3: So one is context window becoming bigger. And I think
Speaker 3: it's a question of just pure capabilities.
Speaker 4: Right, So, all of.
Speaker 3: The applications similar to chat with work, and there are
Speaker 3: a few, they all need to have sort of big
Speaker 3: context windows in order to function correctly, because you want
Speaker 3: to have as many documents, you know, describing things or
Speaker 3: chats or whatever else that is relevant for the topic
Speaker 3: that the user asked in order to be first of
Speaker 3: all correct in your answer. And Thori and I think
Speaker 3: there is always a balance in an information retrieval system
Speaker 3: like this, like how many you actually present even to
Speaker 3: the LLLM itself.
Speaker 4: But that's a whole other topic.
Speaker 3: But with the big context windows, you can kind of,
Speaker 3: you know, relax a little bit that balance. And then
Speaker 3: the second thing that you mentioned, which is getting the
Speaker 3: prices down, I think this is inevitable. The models are
Speaker 3: being commoditized more and more, and honestly, I am finally
Speaker 3: seeing what I thought Google would have done, which is,
Speaker 3: you know, putting the prices very very low for models
Speaker 3: because they have access to TPUs, right, so they make
Speaker 3: their own chips for AI. And one thing, you know,
Speaker 3: you can use Nvidia cards and they're fantastic, you know,
Speaker 3: for for EI usage. But another thing is you know
Speaker 3: TPU from from Google. It goes much faster, and you know,
Speaker 3: it's it's so much better. They make it in house,
Speaker 3: so it's probably cheaper for them. So yeah, I'm really
Speaker 3: excited to see where where things will go next with
Speaker 3: this in in AI because there seems to be so
Speaker 3: much going on at the same time it's kind of
Speaker 3: hard to follow.
Speaker 2: Yep, yep, absolutely well.
Speaker 1: Yeah, and every time I see a new model come
Speaker 1: out and people are like because it's felt like the
Speaker 1: last little while, you know, like Grock came out and everyone's.
Speaker 2: Like Grock is the best and it's doing the best
Speaker 2: and blah blah blah.
Speaker 1: And literally like a week later, like GPT four one
Speaker 1: was out or Anthropic, I can't remember, but one of
Speaker 1: them came out with something and it was like, Wow,
Speaker 1: this one has you know, this much more over Grock.
Speaker 1: And then it was the other one, whether it was
Speaker 1: GPT or Claude or something else, came out and it
Speaker 1: was like, oh, now this one right. And so we
Speaker 1: keep getting better and better and it feels like this
Speaker 1: arms race. But the flip side of it is is
Speaker 1: like you said, you know, the models that are powerful
Speaker 1: enough are being commoditized to the point where it's just like,
Speaker 1: you know what, just just pick the one that's going
Speaker 1: to do the thing for you. And I mean, granted,
Speaker 1: you know, the latent space some of them allow the
Speaker 1: different models to do better in certain contexts. Right, it's
Speaker 1: not just the token size, but holy crap, if they
Speaker 1: keep pushing it the way they're pushing it, I mean,
Speaker 1: we're just going to have some wild capabilities with what
Speaker 1: it can do.
Speaker 2: And I for one super excited about that.
Speaker 1: Yeah, we're getting a little bit close to time, so
Speaker 1: I'm going to push us to image generation and text
Speaker 1: embedding and then we'll wrap up with picks. We've got
Speaker 1: about ten minutes to talk about image generation and embedding.
Speaker 1: So how does that all work?
Speaker 2: Because I didn't dive into that so much.
Speaker 1: I've used other tools to do it, but I didn't
Speaker 1: dive into that so much with.
Speaker 3: Ruby LLM, right, So it's a simple call. It's at
Speaker 3: the same level as RUBYLM do chat, but since it
Speaker 3: doesn't need to have like the context of all the
Speaker 3: conversation that you have with the LLM, it's as simple
Speaker 3: as RUBYLM do.
Speaker 4: Paint and then you have your message there.
Speaker 3: And also for embedding, so you can just use RUBYLM
Speaker 3: dot m bed and just the text. You can also
Speaker 3: pass a raise so it will embed multiple texts altogether
Speaker 3: and batch them together. This is nice for you know,
Speaker 3: pricing and API usage. But that's pretty much all there
Speaker 3: is to it. You know, there's some little things that
Speaker 3: you can do depending on the model. For example, for
Speaker 3: I believe Dali has two different sizes outputs, so.
Speaker 4: You can configure that.
Speaker 3: But and you can choose obviously between all the models
Speaker 3: that we support in the providers that we support. But
Speaker 3: there's pretty much all there is to it about these
Speaker 3: two two things is just a very simple interface for that.
Speaker 2: Yeah, I have to say.
Speaker 1: One of the things that I'm excited about this as
Speaker 1: far as the image generation goes, is I want people
Speaker 1: to be able to say, hey, I want you to
Speaker 1: generate some album of art or some episode artwork or
Speaker 1: something like that. And like if I say, hey, use
Speaker 1: this episode that we recorded and generate some artwork for it,
Speaker 1: I mean I could see it basically using the chat
Speaker 1: prompt set up to say you know, and you could
Speaker 1: you said you could pass it attachments, so it could say,
Speaker 1: here's the audio for the episode, you know, give me,
Speaker 1: write me a prompt to get a good image, you know,
Speaker 1: cover art image for this episode. And then it turns
Speaker 1: around and it goes ruby LLM paint. Here's the prompt
Speaker 1: that I made up, right, and then you can see
Speaker 1: what it comes up with. And the other thing is
Speaker 1: is that on the chat models we didn't talk about temperature,
Speaker 1: but you could set the temperature a little bit higher
Speaker 1: and get some more creativity out of your prompt. Then
Speaker 1: the image generation may give you something a little bit different,
Speaker 1: And so I'm looking at changing some of these together
Speaker 1: and I'm really excited about that stuff too. But yeah,
Speaker 1: the image generation seem pretty straightforward to me.
Speaker 3: Yeah, what I'm excited about is to see GPT four
Speaker 3: all image generation in the APIs, and I know it's
Speaker 3: coming soon. I know it has been taking the world
Speaker 3: by storm lately. How many Studio Ghibli memes of your found, Charles, Oh.
Speaker 4: Yeah, you have them.
Speaker 1: Yeah, so you see wacky stuff with Donald Trump or
Speaker 1: Bernie Sanders here in the US, right, you know, not
Speaker 1: to pick on one side or the other, but both
Speaker 1: of those guys cartoon out really really well, and it's
Speaker 1: so fun to see some of them because it's like
Speaker 1: it's like, oh, yeah, that's the studio Ghibli that kind
Speaker 1: of emphasizes kind of the quirky aspects of either one
Speaker 1: of those guys.
Speaker 3: Yeah, the capability of JUBT for image and it is fantastic.
Speaker 4: But returning back to your question, I think the.
Speaker 3: Or rather what you said is to me, I'm really
Speaker 3: excited about that because it really would combine both the
Speaker 3: text interface and also like the image generation in the
Speaker 3: same model.
Speaker 4: So that we can have.
Speaker 3: This capability without even needing to create a new tool.
Speaker 3: So right now, how you would do it is that
Speaker 3: you create a tool and inside the tool you have
Speaker 3: RUBLM dot paint. Right with these types of models, you
Speaker 3: wouldn't need a tool anymore, and it would do it
Speaker 3: all in the same model, and it would do it
Speaker 3: so much better than the previous diffusion models because apparently
Speaker 3: they're using combination of diffusion and transformers architectures. So in
Speaker 3: the end, what comes down to is that is so
Speaker 3: much more beautiful, so much more precise. It also understands language, right,
Speaker 3: so it also understands out to make fonts so you
Speaker 3: can write you know, when it writes it, actually you
Speaker 3: can read it, and it's not just this garbage that
Speaker 3: we used to get like a year ago.
Speaker 1: Right, So I have to admit with the embeddings, so
Speaker 1: I was flirting with the idea of doing vector searches
Speaker 1: for top end devs, and I'm going to move there,
Speaker 1: but I'm just using what's built into postgress at this point,
Speaker 1: and that's what it pointed me too as one option, Right,
Speaker 1: I could hand off getting the vectors out of a
Speaker 1: text embedding, but beyond that are there are other things
Speaker 1: that people do with these because I'm I'm kind of
Speaker 1: lost as far as like how useful this is?
Speaker 3: Right, Well, you could do so many things, like you know,
Speaker 3: cluster things together, for example, and try to have different topics.
Speaker 3: So let's maybe start from what isn't embedding. I'll try
Speaker 3: to compress it because I know we don't have so
Speaker 3: much time. It's basically a very long array of numbers
Speaker 3: which represents a point in a space which is a
Speaker 3: multi dimensional space, and the dimensions are you know, the
Speaker 3: length of the array. Essentially, what that means is that
Speaker 3: if you projected in three D space, it's just this point, right,
Speaker 3: And let's say this point represents a word or a
Speaker 3: document or whatever, and the distance between another point is
Speaker 3: actually significant, and it is semantically significant.
Speaker 4: So, for example, if.
Speaker 3: You would have the word and let's let's bring it
Speaker 3: back to word t K. Right, it's a really really
Speaker 3: old model that used to represent words in this embedding space.
Speaker 2: Right.
Speaker 3: So if we say, if we take the embedding of king, right,
Speaker 3: and we take the embedding of queen, we can actually
Speaker 3: I believe, subtract from King the embedding of man and
Speaker 3: add the embedding of woman, and we will get very close,
Speaker 3: if not exactly perfect on the embedding of queen. Right,
Speaker 3: So they're semantically significant. So what that means is that
Speaker 3: you can start to you know, cluster similar data points,
Speaker 3: similar documents, similar movies, similar whatever you want. You can
Speaker 3: search in them. And searching is a kind of a clustering.
Speaker 3: So you're basically like taking the query and embed it
Speaker 3: and then see where it ends up in the space,
Speaker 3: and then you start to expand your your bubble to
Speaker 3: see who else is nearby, and then you just take
Speaker 3: a limit, you know, the top ten or whatever, because
Speaker 3: it could go infinitely. You know, you could get all
Speaker 3: the documents just ordered by out.
Speaker 1: Close, right you were just yeah, ordered by distance. Yeah yeah, Okay,
Speaker 1: So it really is.
Speaker 3: Kind of a search type use case, that's right, But
Speaker 3: it's more semantic than the typical keyword search.
Speaker 1: Right, because if you use the keyword for ruby, well
Speaker 1: that's not a great example, but if you make it
Speaker 1: you have two synonyms, yeah.
Speaker 2: Or yes, sing a TYPEO won't find it.
Speaker 1: Or if you have synonyms, right, And so somebody said testing,
Speaker 1: and somebody said, uh, tested, and you look for tested,
Speaker 1: it won't find testing because it's not doing semantic you know,
Speaker 1: it doesn't break it up into tokens.
Speaker 2: It doesn't do all of that other work where this does.
Speaker 2: And so yeah that makes sense.
Speaker 3: Yeah, a typical search engine would have stemming, so they
Speaker 3: would separate testing and tested and kind of matches anyway.
Speaker 4: But I that's exactly right. I think the genre idea
Speaker 4: is exactly right.
Speaker 1: Yeah, all right, cool, Well, I'm gonna go ahead and
Speaker 1: wrap us up and head us into picks. But this
Speaker 1: is terrific. I'm gonna point people to ruby l l
Speaker 1: M dot com. But if they want to reach out
Speaker 1: to you or connect in some other way, like how
Speaker 1: do people find you on the internet and say, dude,
Speaker 1: this is awesome.
Speaker 3: So my blog is at paulinot me and you can
Speaker 3: reach me on Twitter as well at x dot com
Speaker 3: slash Paulino.
Speaker 2: Mm hmm. And for the non Italians, Paulino is p
Speaker 2: a O l I n O.
Speaker 4: Yes, correct, p l O n I N.
Speaker 2: So yeah, it's it's the p a O that might
Speaker 2: lose people anyway.
Speaker 1: Thanks so much for coming. Let's let's jump in on
Speaker 1: the picks. I'm gonna be kind of short with mine
Speaker 1: just because I don't want to take a ton of
Speaker 1: time and I have to jump over to this other
Speaker 1: meeting at eleven. But so my first pick I always
Speaker 1: do a board game picks. Last week I think I
Speaker 1: might have picked this, but I'm going to pick it again.
Speaker 1: It's kind of like the game that we've been playing lately.
Speaker 1: It's called Cult Express, and basically you have a train
Speaker 1: on the table and you know, it's just made out
Speaker 1: of wood. It kind of all fits together. But it's
Speaker 1: unique for a game board, right because it's it's it's
Speaker 1: this three D train and your bank your train robbers,
Speaker 1: and so you're trying to gather as much loot as
Speaker 1: possible and whoever gathers the most loot wins. You also
Speaker 1: get you get one thousand dollars for being the person
Speaker 1: who shot the most bullets. And so you have cards
Speaker 1: that represent actions. You can pick up loot, you can
Speaker 1: move from the roof to the car, or the car
Speaker 1: to the roof. You can move one card to the other.
Speaker 1: You can shoot your gun at other robbers, and if
Speaker 1: you hit them, then your bullet card goes into their deck.
Speaker 1: And it's basically no op, no action card. And so
Speaker 1: when you drop up a new hand, if you have
Speaker 1: those in there. They're just kind of duds. You can
Speaker 1: punch other players, and if you punch another player, then
Speaker 1: they drop a piece of loot, and then you can
Speaker 1: pick up loot. And then all the players or all
Speaker 1: the characters have different abilities, right, And so the player
Speaker 1: that I played, if you punch somebody and they drop loot,
Speaker 1: if they drop a loot bag, you can catch it
Speaker 1: before it hits the floor and collect it without having
Speaker 1: to play a pick up loot card.
Speaker 2: Right.
Speaker 1: Some of the others were that you could you could
Speaker 1: shoot above you or below you if you were on
Speaker 1: the roof or in the car. There were some that
Speaker 1: had to do with a sheriff, right, because there's a
Speaker 1: sheriff on the train and if he winds up in
Speaker 1: the same car as you, then he puts a bullet
Speaker 1: in your in your deck anyway, So it was a
Speaker 1: lot of fun board game. Geek waits at one point
Speaker 1: eighty six, which means that it's pretty approachable for the
Speaker 1: average random board game player that doesn't play the heavy
Speaker 1: board games like I like. And uh yeah, it took
Speaker 1: us like forty five minutes or so to play, or
Speaker 1: maybe an hour, but we were learning the rules and
Speaker 1: so I could I could imagine you can play in
Speaker 1: forty five minutes.
Speaker 2: And yeah.
Speaker 1: Anyway, so the way you play your cards is you
Speaker 1: go around the circle and everybody plays a card onto
Speaker 1: the pile.
Speaker 2: Sometimes you play two cards.
Speaker 1: Sometimes you play him face down, but you can see
Speaker 1: what everybody else is playing on a regular play. And
Speaker 1: then you take the deck and you flip it over
Speaker 1: and you just play the deck. And so then you know,
Speaker 1: he played a shoot the gun card, so if he
Speaker 1: could shoot somebody, you shot him. And then the next
Speaker 1: guy he played a punch card, so he punched the
Speaker 1: guy next to him because he could. And if you can,
Speaker 1: then you can't write. So if you play it and
Speaker 1: there's nobody in the car with you to punch and
Speaker 1: you just don't punch. But if there is somebody there
Speaker 1: and they're not the person you wanted to punch, you
Speaker 1: still punch them.
Speaker 2: So that's kind of how it works.
Speaker 1: So anyway, it was really really fun, really really enjoyed it.
Speaker 1: It kind of reminds me of another game called robot Rally.
Speaker 1: The difference is is that with robot Rally, what you
Speaker 1: do is you put all of your action cards into
Speaker 1: your own pile and then you flip it over and
Speaker 1: play them in order. But you play them in order,
Speaker 1: everybody goes around and plays them. And so you know,
Speaker 1: a lot of times you set yourself up to push
Speaker 1: another robot into a hole or make them go off course,
Speaker 1: but they did something you didn't expect, and so then
Speaker 1: you wind up making moves that you didn't expect because
Speaker 1: you expected to hit them. And so anyway, it's it's
Speaker 1: a lot of fun. It's kind of a it's almost
Speaker 1: like mob programming a little bit. So anyway, a fun
Speaker 1: cult express it. It's really fun. Besides that, at work,
Speaker 1: we're doing a Ruby book club and we're reading Sandy
Speaker 1: Metz's oh what is it called. It's the Pewter Book,
Speaker 1: The Practical Object Oriented Design in Ruby, and it's a
Speaker 1: terrific book. I don't agree with everything that Sandy says
Speaker 1: in it, but a lot of it I do, and
Speaker 1: the rest of it every time I read it and
Speaker 1: I think about it for a while ago.
Speaker 2: Okay, yes, but with caveats.
Speaker 1: Right, so right, anyway, if you haven't read it, it
Speaker 1: was written in twenty eighteen, the second edition, but it's
Speaker 1: still pretty down good.
Speaker 2: So I'm going to pick that.
Speaker 1: And yeah, in a couple of weeks, we're going to
Speaker 1: be talking to the guys that built Ruby events dot org.
Speaker 1: It was originally Ruby videos dot dev and so it
Speaker 1: had all the conference videos. They were trying to collect
Speaker 1: all of those. But yeah, now they're pulling in like
Speaker 1: all the upcoming conferences and stuff like that, making it
Speaker 1: easy to figure all that stuff out.
Speaker 2: So I'm going to pick that too, Carmine, what are
Speaker 2: your picks?
Speaker 4: My picks? Can I go with music?
Speaker 2: Yeah?
Speaker 3: Yeah, yeah, yeah, all right cool. So I run a
Speaker 3: disco collective here in Berlin. It's called Floppy Disco Fun
Speaker 3: and what women trying to do is to bring back
Speaker 3: the funk a little bit in Berlin. As you know,
Speaker 3: it's probably as you probably know, it's a very much
Speaker 3: of a techno city. And I want to pick some
Speaker 3: friends that have been playing with us that have been
Speaker 3: absolutely fantastic. Cody Curry, he is a jazz house master.
Speaker 3: We play with him actually this Saturday, and he has
Speaker 3: releases on Spotify that you know, they have three million plays,
Speaker 3: So definitely hitting a spot there with people. Very jazzy,
Speaker 3: very groovy, very soft jazz kind of style, but in
Speaker 3: house music, so really really cool stuff, really chill guy,
Speaker 3: also really cool.
Speaker 4: RP.
Speaker 3: Brown from the same label. Toy Toonics is the label
Speaker 3: by the way, it's an Italian German label by an
Speaker 3: Italian German guy.
Speaker 4: And r P.
Speaker 2: Brown is.
Speaker 3: More on the UK side of the spectrum. I want
Speaker 3: to say, so he combines you know, disco with UK
Speaker 3: influences because he lived there for a little bit, uh
Speaker 3: and Delphonic different label. He posts on a bunch of
Speaker 3: labels actually, and he has been I think one of
Speaker 3: the biggest disco influencers here in Berlin and DJs and
Speaker 3: he plays a lot and he's uh.
Speaker 4: Just also just such a nice guy. So I want
Speaker 4: to pick these three.
Speaker 3: And in terms of books, uh, the Hot Wire book
Speaker 3: by Joe Oh, fantastic, fantastic book.
Speaker 4: Been reading this and now you're really enjoying it.
Speaker 2: Which one is that?
Speaker 3: I think it's called All to Our tur Ball? I sorry, Joe.
Speaker 3: Why are Native for rails developers?
Speaker 2: Oh?
Speaker 1: Hot Wire Native Yes by Joe mast Yeah, yeah, yeah, yeah,
Speaker 1: that's a terrific book on practprog. We talked to him
Speaker 1: a little bit ago and we're going to have him
Speaker 1: come back on.
Speaker 2: Because there was just more to talk about. So I'll
Speaker 2: look for that a month or so.
Speaker 4: Awesome.
Speaker 2: All right, good deal. Well, thanks for coming, Thank you
Speaker 2: for having me.
Speaker 4: It's really a pleasure.
Speaker 2: This was so cool. I may not sleep for a
Speaker 2: week because I'm playing with this stuff.
Speaker 4: I'll feel bad and happy at the same time.
Speaker 1: Yeah, all right, Well I'm gonna go ahead and push stop.
Speaker 1: Stick around because we got to make sure we get
Speaker 1: all your stuff uploaded. But to the listener until next time,
Speaker 1: max out. M m mm hmmm.