Speaker 1: Hey everybody, Welcome back to another episode of the Ruby
Speaker 1: Rogues podcast. This week, on our panel we have yush Nuwatya.
Speaker 2: Hello.
Speaker 1: No, I'm Charles max Wood from top End Devs. And
Speaker 1: this week we have a special guest. We have Mohammed Hassan. Mohammad,
Speaker 1: you built light stack, which is pretty cool. What else
Speaker 1: doul people know about you?
Speaker 2: Well, I've been doing Ruby for for a long one now,
Speaker 2: I guess, I guess two thousand and five or something.
Speaker 2: I've been been in this industry for for a long while.
Speaker 2: I go by the handle old Mo for a reason.
Speaker 2: I think, yeah, it's it's literally old Mo, and I've
Speaker 2: been I've been doing computers since nineteen ninety something, so yeah,
Speaker 2: I have have a lot of work that has been
Speaker 2: done in the Ruby land, especially in concurrency. I've released
Speaker 2: the library long ago if anyone remembers, was called never
Speaker 2: block once fibers started, and uh yeah, I got fascinated
Speaker 2: by by this idea of like high concurrency and and
Speaker 2: doing high performance stuff in Ruby. And then afterwards I
Speaker 2: also got into got in love with with embedded databases
Speaker 2: Siquo light to be specific. So basically I'm marrying the two.
Speaker 2: So like high performance Ruby, high performance embedded database siqual
Speaker 2: Light in particular, and I've been I've been doing that
Speaker 2: with my own work in my own startups and looking
Speaker 2: forward now to actually deliver something to the community that
Speaker 2: will you know, spread this, uh this value to everyone
Speaker 2: that is using both Ruby and siqual Light.
Speaker 1: Cool. So I looked at it and it and you
Speaker 1: can reckon me where I'm wrong. I didn't have a
Speaker 1: chance to try it because my life is absolutely nuts.
Speaker 1: You think the kids get out of school, your life
Speaker 1: gets lets the nuts, It doesn't. So yeah, I'm looking
Speaker 1: at it, and I'm sitting here thinking well, because I
Speaker 1: remember back in the day, sequel Light was kind of
Speaker 1: the not serious database. You know, you use it development
Speaker 1: and then you'd use a real database like post ris ql.
Speaker 1: But with light stack it looks like you provide the database,
Speaker 1: which is kind of obvious, but then you're also pulling
Speaker 1: in stuff like cashing and jobs, and yeah, do you
Speaker 1: want to just talk about all of this because yes,
Speaker 1: you pull in just one gem and it does all
Speaker 1: this stuff.
Speaker 2: Yes, I think. I think one one the biggest issue
Speaker 2: here was was the amount of dependencies you need when
Speaker 2: you start a new rails app like you need, you
Speaker 2: need like a database naturally, yeah, and then you need
Speaker 2: some some cash sover and and and then things go
Speaker 2: a little bit crazy and you make a cluster of
Speaker 2: those cash servers. And then you need a job processor,
Speaker 2: and you need a queue for that job processor. So
Speaker 2: you have read this for example, and then uh a
Speaker 2: process for sidekick or something like that. And then you
Speaker 2: need full tech search, so you use something like elastic
Speaker 2: or open search. And then you need like uh cable
Speaker 2: uh capabilities, so you need a pop sub server and
Speaker 2: you have usually read this for that as well, or
Speaker 2: postographs and and and you end up with so many
Speaker 2: dependencies and so many moving parts in your application. And
Speaker 2: I was I was thinking of like, yeah, like making
Speaker 2: making Siquo light your your database is nice, but like
Speaker 2: limiting it to just database means it's only limited to
Speaker 2: smallish applications that only require database. Actually, sequel Light, due
Speaker 2: to its latency characteristics, is capable of a lot more.
Speaker 2: And I went out to try and see if I
Speaker 2: can actually prove that it is capable of a lot more.
Speaker 2: And so yeah, database, cash, pops up, que, job processor,
Speaker 2: full tex search, everything you'd need for not everything, but
Speaker 2: almost everything. You'd need for your application, and currently in
Speaker 2: the rapport but not released is like ID which is
Speaker 2: greatest basically reported to sequel light, so you won't need
Speaker 2: for your key data another separate process running somewhere in
Speaker 2: your infrastructure.
Speaker 1: Very cool, I have to say, like I switched over
Speaker 1: to what is it the caching and queueing that? Yeah,
Speaker 1: solid cash and solid cute that came out with. Sorry, folks,
Speaker 1: I'm a little slow. I am under the weather big time,
Speaker 1: but this looks so cool. I just wanted to talk
Speaker 1: through it, and so I was super excited to get
Speaker 1: rid of rettis right, because I haven't been using action
Speaker 1: Cable a ton, so yeah, I kind of like the
Speaker 1: idea of And I've also been using Melee search and
Speaker 1: I've been trying to move that into my Postgress database
Speaker 1: as well, just use Postgress full tech search. So so
Speaker 1: this appeals to me a lot, because I, you know,
Speaker 1: my stack gets a lot simpler.
Speaker 2: Yes, definitely. I think I think there are two dimensions here.
Speaker 2: One of them is simplifying the stack, and one of
Speaker 2: them is actually realizing the use potential in having an
Speaker 2: embitted database. So one way to simplify the stack is
Speaker 2: to move everything to posgress posts is very capable, it's
Speaker 2: very solid, and it's been there since forever and you
Speaker 2: can rely on it. But what I'm my thing is
Speaker 2: there is a new dimension of performance when you go embedded,
Speaker 2: like a complete a completely different thing. I think in Brighton,
Speaker 2: ruge I presented what it takes for a query to
Speaker 2: run if you go through like client server stack versus
Speaker 2: an embedded stack, and what do that is? Basically you're
Speaker 2: you're doing for most treats, you're doing a memory call
Speaker 2: and and and that's that's a huge advancement in performance
Speaker 2: that you'll only appreciate it if you see it. Once
Speaker 2: you see that your Rail's application is able to able
Speaker 2: to deal with like potentially thousands of concurrent connections and
Speaker 2: then responding to them very quickly, you'll you'll, you'll you'll
Speaker 2: have to see it to believe it.
Speaker 1: Basically very cool. Are usually you played with this?
Speaker 3: I haven't actually had a chance to play with it.
Speaker 3: Sequel Light and Rails has been something that's been on
Speaker 3: my do list for a while, but well, life gets
Speaker 3: in the way. I think my next project is almost
Speaker 3: definitely going to be my next side project is definitely
Speaker 3: going to be a sequel light rather than postgress. But yes,
Speaker 3: I just I just want to get very clear in
Speaker 3: my head about on the gotcha's with sequel light. And
Speaker 3: let's start with just what light stock actually does in
Speaker 3: certain things like with light job and sorry, with light
Speaker 3: cable and like like Katie he said, like like like
Speaker 3: cable used like pub sub it's like a drop in replacement,
Speaker 3: right and like Katie replicates credits that are these building
Speaker 3: on like native sequel light features off like pub sub
Speaker 3: in a key value store or.
Speaker 2: Is a little blend then than that, so it doesn't
Speaker 2: have like something like notify in in postgress. And I
Speaker 2: like I had to build this functionality into Syqul light
Speaker 2: on top of Syqul itself, but on top of native
Speaker 2: sequel light. So by the way, the solid family of
Speaker 2: products like Solid cash, slid C they have they do
Speaker 2: have sequel light support, but the problem is they do
Speaker 2: deliver that on through active record, which is like an
Speaker 2: order of man nude more abstracted then going native with
Speaker 2: sycuolight like directly and hence the siqal light performance. If
Speaker 2: if you go directly to the metal, it's it's a
Speaker 2: much much faster and the nice thing about about what
Speaker 2: we have in in light stack is that we have
Speaker 2: the seql code completely the coupled from the from the
Speaker 2: Ruby logic, so you can optimize that independently and work
Speaker 2: on ensuring gas every call is through an index. You're
Speaker 2: not working, uh, you're not walking tables or doing any
Speaker 2: table scans things of that sort. So it's uh, I guess,
Speaker 2: I guess it's. It's it's implemented, but the functionality is
Speaker 2: not that hard. I guess it's it's really straightforward getting
Speaker 2: a cable and getting it to perform the problem that
Speaker 2: your core issue would be to get it to perform
Speaker 2: correctly under like concurrent operation, which is which is tricky
Speaker 2: and because you don't know whether the application would run
Speaker 2: in in threads, run fibers, multiple processes or not. And
Speaker 2: this is I think the core contribution of light stack,
Speaker 2: which it abstracts all this and and make sure any
Speaker 2: component can be brought up very quickly, like I built
Speaker 2: like cable in one hour. Oh wow, okay, because because
Speaker 2: all the machinery there for for like making sure that
Speaker 2: like uh, pulling of connections, threads don't go into these
Speaker 2: conditions and and proceeds are not affecting each other that
Speaker 2: was there already, so you only get to implement the
Speaker 2: logic of the higher level of course one hour with bugs.
Speaker 2: But but then I had a working for the type
Speaker 2: and and and unlikely you was not much harder. It
Speaker 2: was done in two days, so and then it was
Speaker 2: mostly the functionality, a lot of functionality, a lot of
Speaker 2: logic on top of the connection. So I guess, I
Speaker 2: guess that's that's what light stack is able to do,
Speaker 2: bring up those features quickly, and I implement as much
Speaker 2: of them as possible in sequel land rather than in
Speaker 2: rugby land, for for performance reasons from one side, and
Speaker 2: and also for portability and separation of concerns as well.
Speaker 3: Yeah, I think that that's a great approach. I've kind
Speaker 3: of been uh, blissfully unaware of sequel up to a
Speaker 3: point because Actor record kind of handles it for me.
Speaker 3: But for the client I'm working with at the moment,
Speaker 3: we've had to do some really advanced stuff and I
Speaker 3: have been getting my hands dirty with seql and the
Speaker 3: power that it has has blown me away a little bit.
Speaker 3: So yeah, I completely get your approach there. So yeah,
Speaker 3: just for personal curiosity, could you talk about how you
Speaker 3: how you did pops up with just like SEQL and
Speaker 3: database features without having something like notify at your disposal.
Speaker 2: Yes. So the idea is when when you do a
Speaker 2: database calls it in the traditional sense it goes through
Speaker 2: the network to the server and back, but in in
Speaker 2: in the sequel like sense, it's just a mess of coal,
Speaker 2: a mess a call to a local memory address, especially
Speaker 2: if you if you're looking at some page that is
Speaker 2: definitely in the cash because it is like access, it's
Speaker 2: hot in the hot section of your data. So basically
Speaker 2: it's a pulling it. It's pulling happening from from Ruby.
Speaker 2: Like each thread that is that is subscribed to the
Speaker 2: pops up would constantly pull the database and ask it
Speaker 2: for for data. And and in benchmarking like four proceeds
Speaker 2: constantly writing and constantly pulling. I'm able to get around
Speaker 2: like one hundred and twenty thousand messages retired per second.
Speaker 2: So yeah, this is this and that's not on very
Speaker 2: high end hardware, so it is capable. And now I
Speaker 2: was even able to do faster than some some IPC
Speaker 2: cross process solutions that are tried at first as an optimization,
Speaker 2: but then it ended up like siquolites faster.
Speaker 3: Still, Yeah, I think when you eliminate the network from
Speaker 3: the equation of the performance speed up you get is
Speaker 3: just hard to comprehend because you're any AUTHO removing so
Speaker 3: much complexity from the call. Right, You're not opening a
Speaker 3: network connection, you're not sending anything over a wire.
Speaker 2: You're not data from from buffers to like the network
Speaker 2: stack in the OS. There are so many, so many
Speaker 2: things that are eliminated, and your your life becomes a
Speaker 2: lot easier.
Speaker 1: I'm in some cases, I think I kind of feel this,
Speaker 1: you know, because I've used some of the cloud databases, right,
Speaker 1: so you know, even if I'm pulling a you know,
Speaker 1: a database on Linode and a server on Linode or
Speaker 1: Digital Ocean or whatever, you know, pick a poison like those.
Speaker 1: I feel like I felt that more than when I
Speaker 1: have you know, to virtual servers that I've set up
Speaker 1: in the same network, you know, on Digital Otion and stuff,
Speaker 1: because all of that stuff is it's I think, it's
Speaker 1: all software and it's fast, right, and.
Speaker 2: It is fast, but it's not free.
Speaker 1: No, that's fair, and it will never be free. You
Speaker 1: don't get that for free. But yeah, I think I
Speaker 1: think there are different levels to this, right, I don't
Speaker 1: know that, but completely convinced that you know, it's gonna
Speaker 1: be that noticeable depending on what your setup is. Yeah,
Speaker 1: you should be under the circumstances.
Speaker 2: Yeah, yeah, definitely you should. You should try actually to
Speaker 2: go through the Like I'm all the benchmarks that they
Speaker 2: have for light stack. By the way, on on the
Speaker 2: GitHub ripple, these are run against postglass on the same machine.
Speaker 1: Okay, so you put mostgress on the same machine and
Speaker 1: run not even on a versus I think.
Speaker 2: Okay, Like all these are run through against postgress on
Speaker 2: the same very same machine, on the same space, same everything.
Speaker 1: Well that changes my argument a little bit.
Speaker 2: Yeah, so I like the point is how how much
Speaker 2: of the overhead? How much is the overhead compared to
Speaker 2: the actual work that you're doing, and it there is
Speaker 2: a spectrum here, not not all queries would be much
Speaker 2: faster and sequal lighte Like, if you're doing heavy io,
Speaker 2: if you hit the desk, then they will both be
Speaker 2: both be a lot slower. Then in that case, the
Speaker 2: savings that you had by not going through the network
Speaker 2: stack will be very un noticeable. They will not exist.
Speaker 2: Basically because reading a single block from disk is much
Speaker 2: much more expensive, like orders of magnitude more expensive. So like,
Speaker 2: if you're talking about quis that are fulfilled from the
Speaker 2: page cash, then the difference would be really not If
Speaker 2: you have complex queries that will require disk access, then
Speaker 2: the difference will be less. But in no situation will
Speaker 2: siquo light bill will be slower than post aggress in
Speaker 2: reed scenarios, given, of course, the quity plan that was
Speaker 2: provided by both databases. Both have very advanced quity planners,
Speaker 2: and they should produce for most in the most case,
Speaker 2: similar plans for the quities. But in case they're producing
Speaker 2: the similar plan, definitely Sequolite will be either as fast
Speaker 2: or much faster than like in a spectrum between fast
Speaker 2: and as fast and much faster than prosrogress. That's three equities, right,
Speaker 2: It is a completely different story.
Speaker 1: Right. So my question then is, because yeah, I'm looking
Speaker 1: at these benchmarks and it's pretty impressive, what if you
Speaker 1: have to reach it over the network? Right, So, what
Speaker 1: if you've got enough traffic to where you've spread your
Speaker 1: application servers out over multiple servers, and then I'll have
Speaker 1: to hit the same source of truth, right, the same
Speaker 1: database that way, How does that change the equation.
Speaker 2: So siquolite is not designed for this scenario like scaling
Speaker 2: out and connecting from multiple application servers to the same
Speaker 2: node database node is not what sequal light is designed for.
Speaker 2: Sickle Light is designed generally for virtical scaling rather than
Speaker 2: horizontal scaling, so you get a better, bigger machine or
Speaker 2: a VM basically, and in that case you can serve
Speaker 2: more from the same resources. But there's many attempts to
Speaker 2: actually enable enable this level of scaling. But but then
Speaker 2: the point would be a little bit moved, like why
Speaker 2: not use postal glass. I'm working myself on a solution
Speaker 2: that that is midway, so it only transmits rights, but
Speaker 2: all reads are local, so you have multiple nodes, each
Speaker 2: one has a complete copy of the database and rights
Speaker 2: are distributed. I've written about that on my blog and
Speaker 2: still didn't open source it yet. But I'm not surely
Speaker 2: if I'm going to open source it, or maybe you
Speaker 2: just try to build the service around it. But but
Speaker 2: you get a copy of the database of full copy
Speaker 2: of database locally and you can do all reads locally,
Speaker 2: so you get those benefits for reds and you get
Speaker 2: slight extralatency for rights.
Speaker 1: Yeah, but yeah, I mean I have to admit most
Speaker 1: of the time when I'm scaling, Yeah, I just go
Speaker 1: into lin ode or Digital Lotion or wherever I'm hosting,
Speaker 1: and I say I want more RAM and a little
Speaker 1: more disk space, and it goes okay, and then my
Speaker 1: website's down for a minute. So what you're talking about
Speaker 1: that makes sense to me.
Speaker 2: Yeah, yeah, because it simplifies yourself, like because once you
Speaker 2: go out of that, you you have to consider, okay,
Speaker 2: what about like a little balancing, what if it note fails,
Speaker 2: what happens here, what happened there? Then then you have
Speaker 2: so many moving parts and they keep just growing exponentially
Speaker 2: as as you add complexity to resist them.
Speaker 1: Right.
Speaker 3: I think in twenty twenty four, the amount of traffic
Speaker 3: you can serve with one single beefy sell. Yeah, it's
Speaker 3: just bloody unreal. I think horizontal scaling, If you need
Speaker 3: horizontal scaling, you will have enough money to pay people
Speaker 3: to get you off sequel light and on the postgress exactly.
Speaker 2: That's I think. I think that the point is, or
Speaker 2: or the point that I'm trying to to spread, is
Speaker 2: it is unwise to scale more than what you need
Speaker 2: initially in your small or starting projects. You can get
Speaker 2: a lot more a lot more value in like eliminating
Speaker 2: overheads and increasing performance and eliminating like moving parts by
Speaker 2: moving to an embedded solution. And sycolitis is one heck
Speaker 2: of an embedded solution. It's an amazing solution. It just works.
Speaker 2: And Yeah, my advice would be, don't focus on your infrastructure,
Speaker 2: don't have DevOps, focus on your application, and it will
Speaker 2: carry you a long way. And once you're in a
Speaker 2: position where you need to, you really need to move
Speaker 2: out of this exactly what you said, Like, you have
Speaker 2: the money and the funds to move out of this.
Speaker 2: Either either you're getting a lot of money or you
Speaker 2: have enough external funding to get out of this given
Speaker 2: that you have that amount of user activity.
Speaker 1: Yeah, DHH actually said I'm trying to find the tweet,
Speaker 1: but he said in a tweet that if you bought
Speaker 1: like I think it was thirty two or sixty four
Speaker 1: gigabyte Hetzner level thing, that you at this point could
Speaker 1: have run the entirety of base camp a few years ago. Right,
Speaker 1: And so yeah, yeah, I'm really kind of yeah, I'm
Speaker 1: liking the way that this looks.
Speaker 2: And then look at at things now like this camp
Speaker 2: now is a beast. It's a huge application and so
Speaker 2: many subscriptions. But and they had to go through a
Speaker 2: lot of pain to scale that. But if you're starting
Speaker 2: something like that, now you've got like at least five
Speaker 2: to six years of growth within a single box.
Speaker 3: Yeah, I think, I said when they first started back
Speaker 3: in like the mid two thousands or whatever, for a
Speaker 3: number of years they were running on one physical server. Yeah,
Speaker 3: not even virtualization back then, because this is two thousand
Speaker 3: and four. So yeah, I'm fully on board with this.
Speaker 3: Just rent the VM approach.
Speaker 1: Yeah, he said, you're right here. He said, you can
Speaker 1: rent a dedicated forty eight cores two hundred and fifty
Speaker 1: six gig two terror by AMD epic server from Heatzner
Speaker 1: at two hundred and thirty six dollars a month or
Speaker 1: undred and thirty six euros a month. Yeah. I mean,
Speaker 1: I've I've never had an application that got so much
Speaker 1: traffic that that wasn't enough.
Speaker 2: Yeah, and those four the eight courses like translate to
Speaker 2: ninety six course in cloud speak. So yeah, that's that's
Speaker 2: a lot more than many many applications and and a
Speaker 2: lot cheaper, h Yeah, than than what what deployments we
Speaker 2: see on the cloud.
Speaker 3: Yeah, it's crazy how much money you can save when
Speaker 3: you cut out complexity. Well, let's let's dig into a
Speaker 3: little bit just about sequel Light and its use and
Speaker 3: in a web application. So putting light stack and all
Speaker 3: the amazing stuff that does, let's put that to one side.
Speaker 3: And if I wanted to use sequel light directly with rails,
Speaker 3: I know there are some gotchas backups being one main
Speaker 3: thing is how do I back up and not lose
Speaker 3: my file? And the second thing is I've been looking
Speaker 3: through the campfire a code base which is obviously sequel
Speaker 3: light ones product from Petty Sound Signals, And the only
Speaker 3: thing I could find in there that was like specifically
Speaker 3: to do with sequel light in terms of like initialization
Speaker 3: and set up with something called a busy handler. So
Speaker 3: could you talk what that is and why that's necessary.
Speaker 2: Okay, So, originally sequal Light is configured for like really
Speaker 2: really embedded use cases, So it is for usage on
Speaker 2: a phone and probably between the three of us, like
Speaker 2: we have like maybe at least one hundred sequel Light
Speaker 2: instances running at this very moment on our phones and computers.
Speaker 2: But but this use cases is completely different from the
Speaker 2: web application that requires like multiple users accessing it at
Speaker 2: the same time as much as possibly as fast as possible.
Speaker 2: So there are a few caveats and a few knobs
Speaker 2: that he that should be set. What light stack does
Speaker 2: it It completely does that by default when it's publishing
Speaker 2: its own like Siicula driver to active record and what
Speaker 2: there are other gems like fact un minded with Stephen Mark,
Speaker 2: he goes, yeah, he has a lot of effort done
Speaker 2: in that in that area, and and I think I
Speaker 2: think he is helping a lot like the Vanilla's equal
Speaker 2: light experience for rails. By the way, one one thing
Speaker 2: about let's stack light Steck is not just trailed. It's
Speaker 2: basically you can use light stack with Anami and use
Speaker 2: it with with Sinatra whatever bring more cute like. But
Speaker 2: let us get back to the configuration. So basically what
Speaker 2: you need is the following. You need to ensure that
Speaker 2: you can do rights while you can do reads. The
Speaker 2: default configuration can only either do rights or reads. So
Speaker 2: you change the journal in mode from the default which
Speaker 2: is delete, to well which is right ahead log. So
Speaker 2: in that case you're you can have a writer and
Speaker 2: either at the same time, actually an unlimited number of
Speaker 2: readers and a writer at the same time. That's the
Speaker 2: first configuration. The second thing is which is a little
Speaker 2: bit uh tricky, like when when when when connections are
Speaker 2: trying to capture the lock on the database, they will
Speaker 2: either succeed in getting the lock or fail because another
Speaker 2: process capture the lock. And if it fails, then it
Speaker 2: will return to you, like ah, siqual Light busy error,
Speaker 2: and hence we need to have a busy handler, and
Speaker 2: that busy handler what will will do It will capture
Speaker 2: that error and not propagated to the application and it
Speaker 2: will try again later to capture the lock, which is
Speaker 2: basically what what a central server like postrogress would do
Speaker 2: in in locking. Of course, its locks are a lot
Speaker 2: more fine grained than siqual Light, which is like a
Speaker 2: single lock for the all database. And and then you
Speaker 2: also have something in the in the transaction type. Sickle
Speaker 2: Light has multi transaction types. It has like the the
Speaker 2: default which is deferred transactions, and the there's the immediate
Speaker 2: and the exclusive transaction. What we do also for the
Speaker 2: driver in to run a web application properly is to
Speaker 2: make all transactions immediates. What that means is a deferred
Speaker 2: transaction is a re transaction that you can midway upgrade
Speaker 2: to a right transaction. And and the problem with that
Speaker 2: is that if you do this and another transaction has
Speaker 2: if you attempt to upgrade the lock and another transaction
Speaker 2: already has written to the database, this operation will fail
Speaker 2: with a busy like sickle Light busy error. But if
Speaker 2: you start with an immediate transaction, then you will have
Speaker 2: a right lock right from the beginning of the transaction,
Speaker 2: and then you will not have this error again anymore.
Speaker 1: So.
Speaker 2: So basically the for configuring Siquelight, I think there are
Speaker 2: like four main things to do. Change the journey motu al,
Speaker 2: set a busy handler, and also configure your memory map
Speaker 2: to have like a shared cash across all processes rather
Speaker 2: than a cash pair connection. And yeah, and and and
Speaker 2: that's it and and set your transactions to medio. These
Speaker 2: are the four things that need to be done. Let's
Speaker 2: Tack does that automatically. There is also the sequel light
Speaker 2: JAM sycal light adapter JAM from Steven also does that.
Speaker 2: And I think these some of these are being merged
Speaker 2: into into the upstream sickle Light driver for active records.
Speaker 3: How do you make these changes? Do you do it?
Speaker 3: At Ruby level to run some sequel or how how's
Speaker 3: this gon figuration done?
Speaker 2: Like in lightstack, it's it's done by by overriding the
Speaker 2: sequal light driver and applying these at initialization, making sure
Speaker 2: all connections share these attributes.
Speaker 1: So this is on the the rails end or the
Speaker 1: SEQL end or whatever, not on the sequel light end.
Speaker 2: It's it's on the sequel light adapter end or sickle
Speaker 2: light driver end.
Speaker 1: Basically, so in database to do anything different, you're telling
Speaker 1: the application it has to manage its connection this way.
Speaker 2: More or less. Yeah, but but at the end of
Speaker 2: the day, Raid itself is not aware of that. It's
Speaker 2: not it doesn't see that this is happening, that the
Speaker 2: changes that are we're going to go upstream will will
Speaker 2: bring that knowledge to the rails driver.
Speaker 1: Mm hm. That makes sense.
Speaker 3: And regarding backups, like is light stream power light stock
Speaker 3: or is that no?
Speaker 2: Light stream is not part of lightstack, you can use
Speaker 2: light stream aside from using lightstack, light Stream is a
Speaker 2: very nice tool you can you can basically back up
Speaker 2: the database like incrementally and put that on S three.
Speaker 2: Like I personally use a different method which I'm trying
Speaker 2: to spread as not just possible, but like in in
Speaker 2: in in most situations when when you have a VPS
Speaker 2: or like a VM on on something like line Ode
Speaker 2: or or Digital Ocean or anything, my accommendation would be
Speaker 2: to put the database on a replicated volume rather than
Speaker 2: on the VM directly. And in that case you have
Speaker 2: you have the whole database living in in on the
Speaker 2: network basically, and and then if you have a file
Speaker 2: system that supports copy on write UH semantics like like
Speaker 2: better S or XS, then you can have a backup
Speaker 2: in of the whole database in under too many seconds
Speaker 2: by just copying the file. And you can have as
Speaker 2: many backups of these as you want, like you can
Speaker 2: have like a chron that is doing a backup every second,
Speaker 2: or all you need to do is open a like
Speaker 2: read transactions, take a backup, and it will be consistent.
Speaker 2: I have also written about that. I have a blog
Speaker 2: post about backup strategies, and I explain some of these,
Speaker 2: including light stream, but I don't go deep in there,
Speaker 2: but I also explain some of the other options. If
Speaker 2: you have like a COW file system, so like if
Speaker 2: I was if.
Speaker 3: I didn't really care about having like every second of
Speaker 3: data like backup, if I wrote like a crown job
Speaker 3: to back up the entire database file like every hour
Speaker 3: or something like that and upload it to our S
Speaker 3: three or some remote storage. That's basically a viable back
Speaker 3: cup solution because it's just.
Speaker 2: Yes, yeah, but but like let's swim will be even
Speaker 2: better here because it does just an incremental update, so
Speaker 2: not every second. It just sends the the changes or
Speaker 2: as as I as I mentioned on a volume on
Speaker 2: applicated volume. Then you have your data applicated across the
Speaker 2: data center. You don't need even S three. You just
Speaker 2: make a copy a copy CP of the file and
Speaker 2: it will be a copy on right and it will
Speaker 2: it will, it will. If you have gigs of information,
Speaker 2: it will take like two to four milli seconds to copy.
Speaker 1: Wow, transactions have completed.
Speaker 2: No no, no, It then unrelated to transactions because you
Speaker 2: open you open a read transaction and you copy, you
Speaker 2: don't get you get snapped. Okay, you get a valid
Speaker 2: snapshot of the database.
Speaker 1: Okay.
Speaker 2: I'm planning to add some some functionality in lightstack that
Speaker 2: would do that for you, like open the transaction and
Speaker 2: issue the right copy command. But it will not work
Speaker 2: if you don't have it will not work efficiently if
Speaker 2: you don't have like an XS or better f s
Speaker 2: or zfs.
Speaker 3: Are there any like other gotchas that someone coming from
Speaker 3: a postgraph background. If I'm writing a rails app or
Speaker 3: any any web ap, let's not any web app with
Speaker 3: sequel Light as a database. Is there anything else I
Speaker 3: need to be aware of?
Speaker 2: Definitely? And and And that's the biggest concern when when
Speaker 2: you're dealing with a database that is not I like
Speaker 2: to call it a decentralized embittered database because it doesn't
Speaker 2: have a server to coordinate actions, which is basically right
Speaker 2: performance and specifically things like creating an index on launch
Speaker 2: table and things like that. Because of its nature, Sequelight
Speaker 2: will will always take like a lock on the whole
Speaker 2: database whenever it's doing the right operation, so only one
Speaker 2: writer at a time. Keeping keeping your rights small and
Speaker 2: untidy will mean that you get very fast rights like
Speaker 2: you can do like tens of thousands of rights per
Speaker 2: second and on a syquo light database, and it will
Speaker 2: beat any any other database. But if you have really
Speaker 2: large rights, any other database on a on a like
Speaker 2: a reasonable configuration on the same configuration like a single note.
Speaker 2: But if you have really large right transactions that do
Speaker 2: a lot of things, especially if you're trying to do
Speaker 2: things within the transaction like going to a server or something,
Speaker 2: then you're you're hurting the syco light performance a lot,
Speaker 2: and and that would while this will degrade other solutions
Speaker 2: as well, including postrogression by sequel, they will be a
Speaker 2: lot more graceful at providing a chance for other transactions
Speaker 2: to run at the same time. Siquo Light will not
Speaker 2: be able to do so. So if if you have
Speaker 2: like threally large transactions that are trying to do a
Speaker 2: lot in writing, you should hopefully avoid that and break
Speaker 2: it down, or you will suffer from like queueing right
Speaker 2: requests because only one writer at the time. That's one thing.
Speaker 2: The other thing is some some some of these are unescapable,
Speaker 2: like creating an index. When you create an index, you
Speaker 2: have to wait until the index is finished, and it
Speaker 2: has to have like a snapshot like a correct state
Speaker 2: of the database well being created, so it cannot allow
Speaker 2: other writers to write at the same time, even to
Speaker 2: other tables. So in that case, creating a creating an
Speaker 2: index on the large table, really large table, can actually
Speaker 2: take your to debase flying briefly, and is I.
Speaker 3: Think I don't know if I am completely making this
Speaker 3: up or if I read it somewhere, but is like
Speaker 3: N plus one query is a feature rather than a
Speaker 3: bug when it comes to sequel.
Speaker 2: Life, Yeah, it's it's it's not a feature, per see. Basically,
Speaker 2: here's the thing. The overhead of running equity on sequel
Speaker 2: light is way way less than running a call across
Speaker 2: the network stack, whether locally or remotely. So if you
Speaker 2: have en plus ones, they're not as expensive. Of course,
Speaker 2: having one running one query instead of like n plus
Speaker 2: one queries is cheaper, but the difference is not as
Speaker 2: big as with client server database. It's a lot less,
Speaker 2: like it could be like fifty percent more or something
Speaker 2: depending on end. So in that case, if doing an
Speaker 2: en plus one would actually make your call nicer, just
Speaker 2: go for it because because it's not as penalizing to
Speaker 2: your system as it is with something like uh, something
Speaker 2: like my sequel or postrogress. The authors of sequel Light,
Speaker 2: they use M plus one intentionally in some cases on
Speaker 2: their website and on the Fossil sem system because it
Speaker 2: makes things a lot easier and and it makes encapsulation
Speaker 2: a lot.
Speaker 1: Like I had that beaten out of me. I swear.
Speaker 2: Yeah, like I think about it, it's it's an optimization
Speaker 2: like eliminating M plus one is an optimization. It's not
Speaker 2: something that would make your code look better, right, Actually
Speaker 2: it's another way out.
Speaker 1: Well, and and it still works, right, it's just fast.
Speaker 2: Yes, the the idea in siquolizes is not that big deal.
Speaker 2: It's not an issue like of mice for for your
Speaker 2: code organization and installation better right.
Speaker 3: Yeah, just a different mental model, isn't it, Because yeah,
Speaker 3: suddenly database is local.
Speaker 2: Imagine if you cannot do partials in in in views,
Speaker 2: like you'll have to have those read and do all
Speaker 2: the partials inside in line. It's it's exactly the same.
Speaker 2: Bushis are more expensive, but but you don't go there
Speaker 2: because because then it's a nightmare. Good vice.
Speaker 3: True. Yeah, So let's let's pick up on something we're
Speaker 3: actually briefly discussing before the show started. But so I
Speaker 3: have done a fair amount of work on search and
Speaker 3: postgrass the postgas native full tack search, and for the
Speaker 3: client I'm working with at the moment, I've altered on
Speaker 3: quite elastic search. What what is the search story on
Speaker 3: sequel light and does light stock have anything kind of
Speaker 3: building upon that.
Speaker 2: Yeah. So like syco Light has its own native search
Speaker 2: implementation which is called FTS. There are multiple versions of
Speaker 2: that three, four, and five. Five is the currently the
Speaker 2: most maintained version. So cycle light does have an FDS story.
Speaker 2: The problem with the FTS story in Sequo light is
Speaker 2: that it is slightly rigid, so you have you have
Speaker 2: to have a table that maps to another table in
Speaker 2: the database, and the mapping has to be static. If
Speaker 2: you change a column you want to add it to
Speaker 2: the index, you have to rebuild the whole index. Things
Speaker 2: of that sort. In light Stack, we have a light
Speaker 2: search component which builds on top off the FTS five
Speaker 2: but it also brings a lot of dynamism to it,
Speaker 2: so it actually implements like a dynamic layer. It's a lot.
Speaker 2: The interface is a lot like mini search, so if
Speaker 2: you if you use that, you do the same, you
Speaker 2: do define your index almost the same. But then all
Speaker 2: this is going down into an FTS five database and
Speaker 2: FTI is five module. I'm sorry, so there is I
Speaker 2: guess there is a strong search story there and the
Speaker 2: performance is amazing. Like you have to see I have
Speaker 2: that blog comparing Mini Search to and then you have
Speaker 2: that in the benchmarks as well on the rippot Meli
Speaker 2: search to FTS five and I kept repeating the benchmarks
Speaker 2: because they couldn't couldn't believe the results. Like uh, Emali
Speaker 2: searches has a lot more functionality, I would say, so
Speaker 2: Vanilla FTS five doesn't have, for example, like something like
Speaker 2: type of tolerance, and and you need to add some
Speaker 2: modules in order to support multiple languages. But aside from
Speaker 2: multiple languages and type of tolerance, the performance difference is
Speaker 2: like it has to be seen to be.
Speaker 3: Believed, okay, uh And is that like just in like
Speaker 3: performance as in time, the time it takes. How the
Speaker 3: performance with regards accuracy accuracy.
Speaker 2: That's that's a different thing. Yeah, yeah, that's that's a
Speaker 2: lot harder to measure. But but at the same time,
Speaker 2: it's it's an own quantity. Like working in l P
Speaker 2: is has like this. The industry has been has been
Speaker 2: done that for long and then the tools to to
Speaker 2: produce those are already there. So while while type of
Speaker 2: toolerance is not there by default, you can actually implement
Speaker 2: that there is. There is stemming using the traditional stemmers
Speaker 2: from snowballs like porter and others, and and these are
Speaker 2: mostly the stammers that are used by almost all implementations.
Speaker 2: If we're talking about Latin based languages and Latin alphabets,
Speaker 2: basically and and and and and then aside aside from that,
Speaker 2: it's just typeotolerance that's the difference. So you get you get,
Speaker 2: you get near search, you get race search, you get
Speaker 2: like you can mix and match and or and and
Speaker 2: things like that. You have perfect search all all the
Speaker 2: all the known suspects basically, So from a search quality perspective,
Speaker 2: I believe it's a The biggest difference is type, type
Speaker 2: of tolerance and fuzzy support, which is basically are basically
Speaker 2: the same.
Speaker 1: This is not there.
Speaker 2: But aside from that, you get you get, you get
Speaker 2: everything that you would get with the typical search engine
Speaker 2: like plastic Mealy or swings or things like that.
Speaker 3: That's a pretty solid story, to be honest, I wasn't
Speaker 3: expecting it to be that solid.
Speaker 2: It is. It is a very it's strange like the
Speaker 2: sique Light team, all two of them, they produce very
Speaker 2: very solid software and components, and they build so many
Speaker 2: things from scratch it's it's mind boggling. And yeah, I
Speaker 2: would love more people to actually use these.
Speaker 1: So what I'm curious about is and and It's funny
Speaker 1: because you mentioned you've been doing this since two thousand
Speaker 1: and five, and I got into Ruby and Rails in
Speaker 1: two thousand and six, so I've kind of seen a
Speaker 1: lot of the same stuff come and go. What changed, right,
Speaker 1: because back then, sequel light, yeah, it was the default
Speaker 1: when you ran Rails without any flag for the database,
Speaker 1: but it was essentially understood that, yeah, you're probably going
Speaker 1: to replace this with another database. So what changed to
Speaker 1: the point where now people are realistically looking at this
Speaker 1: and going, huh, I could run this in production?
Speaker 2: Yeah, I guess. I guess this impression that sequel light
Speaker 2: is just like something to start with or even to
Speaker 2: use just for testing was the norm, and it was
Speaker 2: understandable because many of the features that we're talking about,
Speaker 2: they happened over the years. So, right, headlog mode was
Speaker 2: not there. I'm not sure if it was there on
Speaker 2: two thousand and five. I guess it was, but many
Speaker 2: of the other features around it were not. And now
Speaker 2: you see how how those features came together and made
Speaker 2: Siquolide that rich. By the way, it also when you
Speaker 2: build it, it comes with a geopoly database and like
Speaker 2: you can you can do like distance calculations, and stuff
Speaker 2: like that. And this is based indexes and many other
Speaker 2: features like it's it's it has grown a lot and
Speaker 2: is now capable a lot, and I think people are
Speaker 2: starting to realize that, yeah, I might not need a
Speaker 2: lot more than that, especially with the advancement in hardware
Speaker 2: that is happening in parallel. So you get these two.
Speaker 2: Now you can have like a single box that can
Speaker 2: do a lot and a piece of software that is
Speaker 2: capable of like managing all your data and your data requirements.
Speaker 2: And and what I'm trying to do with light stack
Speaker 2: is to build that layer between both, like your data
Speaker 2: layer and your application that is friction free, like no
Speaker 2: configuration required, you don't need to do anything.
Speaker 1: Yeah, that makes sense, and it's it's kind of exciting
Speaker 1: just to see that, Yeah, somebody could really reliably come
Speaker 1: in and and do a lot of this stuff. It's
Speaker 1: it's funny too because and then I'm working on a
Speaker 1: different layer than you are. But I've been working on
Speaker 1: pulling together rails Composer, which I actually got the domain
Speaker 1: from Daniel Keho, and I've been working on pulling this together.
Speaker 1: It's the same idea, right, It's I need these areas
Speaker 1: of functionality in my application, and so right, I pull
Speaker 1: in one gem and then I can just run generators
Speaker 1: and have it right, Yes, exactly, And so you know
Speaker 1: with this it's the same deal where it's I pull
Speaker 1: in light Stack and then I immediately have the capabilities
Speaker 1: I need to run a modern Ruby on rails app.
Speaker 2: Yeah, almost almost all your data requirements are already fulfilled.
Speaker 2: Google a great application. And don't bother about the infrastructure,
Speaker 2: don't pay for DevOps like eliminate all these points of
Speaker 2: like pain, headache and and and mon uh spending.
Speaker 1: So are there people out there currently running their production
Speaker 1: apps in with light stack?
Speaker 2: Some some I know, yes, uh, and and some are
Speaker 2: are running light stack on no staging like I know
Speaker 2: of a large, large Ruby app that uh their environment
Speaker 2: staging environment was switched to light stack overnight without even
Speaker 2: the developers knowing. And they they didn't know except after
Speaker 2: pondering why everything is much faster.
Speaker 3: That is a sales prige and a half.
Speaker 1: I was gonna say, you're making me feel stupid for
Speaker 1: not trying this. Yeah, I think.
Speaker 2: I think the idea is if if you're if you're
Speaker 2: if you're building something and starting, it's really unwise to
Speaker 2: add to to go on a more complex path, you
Speaker 2: should you should go with the least resistance path until
Speaker 2: it starts resisting you. And hopefully it will never resist you.
Speaker 2: We'll never show you that fiction.
Speaker 1: Yeah, you're speaking my language here. I mean, this is
Speaker 1: where and I picked some of this up from reading
Speaker 1: Ayush's book The Rails and hot Wire Codex, where essentially,
Speaker 1: right he walks you through I mean the first what third,
Speaker 1: almost half of the book is you know, building the
Speaker 1: authentication piece. But you know, I got through it and
Speaker 1: I was just like, this, this is not as hard
Speaker 1: as I thought. And right, I kind of get all
Speaker 1: of the niceties that I want out of it because
Speaker 1: I understand it. And you know, yeah, anyway it appeals
Speaker 1: to me in that same way because I'm looking at
Speaker 1: it and going, Okay, you know the underlying engine for
Speaker 1: this thing, which is essentially rack and the database, you know,
Speaker 1: that's stuff that I don't want to fiddle with, right, Yeah,
Speaker 1: and I can just rest everything on top.
Speaker 2: Of it, exactly pushing that complexity away from you and
Speaker 2: hiding it completely versus exposing every complexity. Like if you're
Speaker 2: trying to deal with the traditional app you'll have to
Speaker 2: think of many things, including like where should I put
Speaker 2: my cash, where should I put my Do I have
Speaker 2: a single radius incense for all of these or do
Speaker 2: I have to split them? And how now that I'm
Speaker 2: scaling anyways, how do I scale these components as well?
Speaker 2: Because you can add application servers, but then your database
Speaker 2: will need to scale. So the nice thing with with
Speaker 2: vertically scaling that you scale both together. You don't scale everything,
Speaker 2: but but with with like a client server architecture, you
Speaker 2: have to separately scale each one.
Speaker 3: Does so again coming from my postgrap postgrass brain, like
Speaker 3: that's what my background is in. So yeah, two questions again,
Speaker 3: trying to find equivalent for postgress. Is there is that
Speaker 3: like a Jason Bee column and sequel light or any equivalent.
Speaker 2: Yes, but it's a different formats. It's equal its own format.
Speaker 2: So you cannot get a jacon b record from here
Speaker 2: and just put it there. You have to convert it
Speaker 2: to jason first. But but yeah, it has it has.
Speaker 1: J befo serialize it and then serialize it in the
Speaker 1: other database.
Speaker 2: But but but almost all the jacent functions that you'd
Speaker 2: expect are there for both the adjacent text format and
Speaker 2: the JCNB format.
Speaker 3: Oh wow, so you can like query for stuff that's
Speaker 3: inside and you.
Speaker 2: Can create an index on an a adjacent field and
Speaker 2: you can Yeah.
Speaker 3: Nice. And another thing I use quite often, especially to
Speaker 3: like just like filter stuff in a UI is UH
Speaker 3: like and I like queries, and I use the program
Speaker 3: program indexes and UH in postgress speed those up. So yeah,
Speaker 3: we spoke about full texa. But like, is trigram a
Speaker 3: thing in one One of.
Speaker 2: The one of the UH tokenizers available by default is
Speaker 2: the trigram togonizer in FTS five. So you can have
Speaker 2: a trigram.
Speaker 3: Okay, and that will help it like with like, and
Speaker 3: I like es exactly. Why don't I you equal light again?
Speaker 1: So yeah, here's another Oh go ahead.
Speaker 2: Yeah, I wanted to mention that I also have an
Speaker 2: extension for sequel light that uses roaring bit maps, so
Speaker 2: you can create facets for your search results. Like you
Speaker 2: have like a search index, and you want to generate
Speaker 2: distributions of results. So you have like you get your
Speaker 2: social books, for example, and you want to see see
Speaker 2: in that search how many of those books are in
Speaker 2: watch contigorties and this this, this can be done using
Speaker 2: maps in like a blink of an eye, like under
Speaker 2: under ten milli seconds, you get get so many statistics
Speaker 2: stuff like you see on Amazon, like those things and
Speaker 2: how many of those are red, and how many are blue?
Speaker 2: How many are at that price, so many of that price.
Speaker 2: All these those things can be done like almost matterally
Speaker 2: in sequel light very cool.
Speaker 1: So I'm wondering. You mentioned that there was a company
Speaker 1: that switched over to light stack overnight, and I'm wondering
Speaker 1: what that process looks like. I mean, I would imagine that,
Speaker 1: you know, kind of the queueing and things like that,
Speaker 1: but you know, are probably pretty easy, right because the
Speaker 1: job's either been run or it's not, so you know,
Speaker 1: pull the job from something else. But I'm looking at
Speaker 1: like my data on some of my apps, and some
Speaker 1: of it's kind of hairy, and so I'm wondering, Okay,
Speaker 1: you know, what would it take for me to go
Speaker 1: in and do you mean my.
Speaker 2: Sequel in table like like table.
Speaker 1: Yeah, I guess it's the state of my models that
Speaker 1: I'm not happy with, not necessarily the state of the data.
Speaker 2: Yeah, that that's one one level above. Yeah, the problem
Speaker 2: we're dealing with. It's completely opaque for the models. So
Speaker 2: you just you that's an option, yes, somehow, and you
Speaker 2: can do a sequel lump and and and just copy
Speaker 2: that over, or you can do a What what I'm
Speaker 2: trying to work on is to do an import export
Speaker 2: at the active record level so that you can easily
Speaker 2: pointed to that database and that detabaate and then the
Speaker 2: data gets migrated with active record semantics, so any any
Speaker 2: potential compatibility issues are just washed away.
Speaker 1: Oh wow, that would be cool.
Speaker 2: Yeah, like like active record does have this machinery already,
Speaker 2: or just need to do it together, right, or you
Speaker 2: can dump. But the way you can dump in active
Speaker 2: record h format basically huh, and and the store after
Speaker 2: you change the driver.
Speaker 1: Yeah. The only other thing that I'm thinking about with
Speaker 1: my setup is so I've been using camal to deploy
Speaker 1: and it puts the traffic load balancer the way I
Speaker 1: haven't set up. I have a traffic load balancer in
Speaker 1: front of two application nodes, so effectively I would have
Speaker 1: to get rid of one of those. I wouldn't need
Speaker 1: traffic anymore.
Speaker 2: Right, yes, but but if if they are on the
Speaker 2: same machine, then you can have a volume.
Speaker 1: Right you volume for the doctor containers.
Speaker 2: Yeah, and they can share it but it's it's much
Speaker 2: much more straightforward to guys you just have one.
Speaker 1: Yeah.
Speaker 2: The nice thing about having two is it makes like
Speaker 2: zero down to deployment is easier, but it is double
Speaker 2: still with one, and that's something we Yeah, that's a
Speaker 2: topic that I want to add ass at one point.
Speaker 1: You can address it now if you want.
Speaker 2: No, I mean there's I mean condition yes, yeah, yeah, yeah,
Speaker 2: like maybe maybe a light deploy of something.
Speaker 1: That'd be interesting because.
Speaker 2: Almost almost almost all modern applications of support like zero
Speaker 2: down time deployments, right and zero downtime resorts. So you
Speaker 2: definitely lose some of the web socket connections, but that's it.
Speaker 2: Even other connections can can.
Speaker 1: Remain not well, especially if you're using something like camal
Speaker 1: with doctor base. Right, you just wait till the other
Speaker 1: the other I wanted the same machine the other container
Speaker 1: is running, yeah, and then it's okay, now everything new
Speaker 1: goes to this one and when the other one's done,
Speaker 1: you just kill it.
Speaker 2: And I think I think that's one of the nice
Speaker 2: things about this setup that yeah, many many of the
Speaker 2: issues that people thought they're gonna face with with similar
Speaker 2: setups are actually addressed. And yeah, I would definitely recommend
Speaker 2: giving giving that smallish stack a go and then see
Speaker 2: if if it works for you. And if you're starting
Speaker 2: something new, just just don't just don't bother, don't bother
Speaker 2: with the complex that is.
Speaker 3: There is there a use case where you would say,
Speaker 3: don't start it's equel light or light stack.
Speaker 2: Let me think, so, maybe if there is a use
Speaker 2: case where there's something that is someone that is writing
Speaker 2: huge records in a database, uh and and like we're
Speaker 2: talking gigabytes, maybe some people do that, some people and
Speaker 2: and and that's like the core of what they're doing,
Speaker 2: then definitely you need something that can spread that data
Speaker 2: over a larger surface in terms of files. If you're
Speaker 2: doing like a network or even a distributed database eventually,
Speaker 2: but syco light will not be the database for you.
Speaker 1: This is not.
Speaker 2: Or not necessarily gigabytes or you're like treading in that area,
Speaker 2: like eight hundred megabyte filees still very large. Other than that,
Speaker 2: I think, I think it it could be a case
Speaker 2: by case basis. But if you also have an application
Speaker 2: where so many people are writing at the same time
Speaker 2: something big relatively that or doing like complex right transaction,
Speaker 2: syco light is a NOG But if you have so
Speaker 2: many people doing smallest transactions. Cycle light is smare than enough.
Speaker 2: It would be even faster than the typical solutions. So yeah,
Speaker 2: I would. I would. I would say, like you'd need
Speaker 2: to have a strong case against using Syqule light to
Speaker 2: not use Siquo light.
Speaker 1: So do you see this becoming kind of the thing
Speaker 1: that people reach for in the future by default?
Speaker 2: I hope that would be the case, Like that's the
Speaker 2: that's the future. I see that. Yeah, Like why why
Speaker 2: go through the complex route if if I have something simpler?
Speaker 2: And I think it just makes sense to simplify things,
Speaker 2: like this industry has suffered a lot from breeding complexity,
Speaker 2: Like we've been doing really really complex setups, especially in
Speaker 2: in the front end. Then infrastructure domains just because people
Speaker 2: wanted to play with with different stuff. And I think
Speaker 2: it's it's about time we and it's it's it's helping
Speaker 2: that the VC money is drying, So I guess I
Speaker 2: guess it's it's it's about time to to realize that
Speaker 2: we need build applications, not not build resumes, and focus
Speaker 2: on on what we're trying to deliver the value that
Speaker 2: we're trying to deliver that rather than this is my stack,
Speaker 2: this is my front end like stack and things like that.
Speaker 2: This is not this is not what what people should
Speaker 2: be bothered with. And if your engineering team is actually
Speaker 2: focused on what type of stack they're building of consulted
Speaker 2: with a lot of teams, and I've seen teams that
Speaker 2: would would talk to me about the technologies they are
Speaker 2: using rather than what value they're they're delivering or building.
Speaker 2: Then then you're you're in the wrong place.
Speaker 3: Yeah, I've been there.
Speaker 2: Yeah, Yeah, it's it's it's it's not fun like seeing
Speaker 2: seeing really complex stacks and you ask how many users
Speaker 2: do you have and they tell you we have two
Speaker 2: hundred and and then why bother? Like why all this? Then?
Speaker 2: Why not focus on actually getting the business off the
Speaker 2: ground and then and running with it. And once once
Speaker 2: you have ten million users, maybe you can talk like
Speaker 2: I I hosted millions of users on the databases, and
Speaker 2: like I had a situation when a friend was asking
Speaker 2: me about their they're hosting costs because they ballooned they
Speaker 2: were paying at the time it was a lot like
Speaker 2: two thousand, five hundred a month at AWS and they
Speaker 2: had almost half our user base or even less, and
Speaker 2: they were asking it was gaming gaming sort of both
Speaker 2: of us. And they were asking how much we pay
Speaker 2: every month and I was really really like embarrassed to
Speaker 2: tell them that we're paying seventy five dollars. So, yeah,
Speaker 2: people need to realize they can they can get a
Speaker 2: lot of value by by removing complexity and removing redundancy
Speaker 2: and moving abstractions and indirection.
Speaker 3: Right, it's pretty mad that computers are like an order
Speaker 3: of magnitude faster than they were twenty years ago. But yeah,
Speaker 3: software seems slower than it's ever been.
Speaker 2: Yeah, it's it's it's stuck with it. Where like it
Speaker 2: because because computers are being faster, softlow is getting faster naturally.
Speaker 2: But it's it's it's not trying to do things differently.
Speaker 2: It's not No one has come up and said, like,
Speaker 2: let's make use of the capacity now that we have
Speaker 2: in a single machine.
Speaker 1: Yeah, I was going to say, and and this was
Speaker 1: kind of came to me when DHH was doing his
Speaker 1: keynote at Rails World last year and he points out,
Speaker 1: you know, we're we have solid Q and solid cash
Speaker 1: because disc speeds have you know, grown in this way, right,
Speaker 1: And he was basically pointing out, yeah, you know we're
Speaker 1: going to take advantage of this. And yeah, it drove
Speaker 1: home to me that the the limiting factor isn't the technology,
Speaker 1: it's the mindset, right, because I'm still stuck in you
Speaker 1: know when I started programming in two thousand and well,
Speaker 1: I started programming in high school in the nineties, but
Speaker 1: when I started doing rails apps into in two thousand
Speaker 1: and six, Yeah, you know, I'm looking at it and going, oh,
Speaker 1: I've got to account for all this stuff, and so
Speaker 1: I have to go wrangle all of these things to
Speaker 1: make it work. And and so the limitation was the
Speaker 1: way I thought about applications being built, not not the hardware,
Speaker 1: the technology that was available to me. And then I
Speaker 1: think that's what you're saying, Mohammad.
Speaker 2: Yes, exactly exactly like people, people like we have we
Speaker 2: have been conditioned to do things a certain way, like
Speaker 2: you need you need cash, you put in this inostance,
Speaker 2: you need job, you put the incense and the job
Speaker 2: runner and stuff like that, and people think this is
Speaker 2: this is the best they can get. And I would
Speaker 2: challenge anyone, anyone. I want to see a faster read
Speaker 2: cash reet performance from any any adapter for UH, for
Speaker 2: for rails UH then then light stacks cash. Of course
Speaker 2: solid cash is is fast as well, but it's not
Speaker 2: as fast because it relies on active record. But once
Speaker 2: you hit sequel light, there is nothing faster than this.
Speaker 2: Like you're reading literally from memory. It's an order of
Speaker 2: magnitude faster than a radios at.
Speaker 1: Oh hang on, hang on, time out. You're you're saying
Speaker 1: that you built the light cash and it is the
Speaker 1: queuing this way as well. Yes, so you're not relying
Speaker 1: on active record, You're you're just going straight to the
Speaker 1: database and going, huh, what's in the cash?
Speaker 2: Exactly exactly, because because the nice thing about RAILS is
Speaker 2: that it has these really nice interfaces for active job
Speaker 2: and for active source cash and for a light cable,
Speaker 2: for active action cable and sorry, and and what you
Speaker 2: do is you provide an adapter that does the job
Speaker 2: rather than going one level up and and relying on
Speaker 2: active records that you you you you just delivered the
Speaker 2: functionality directly. It ends up being very fast, like really
Speaker 2: really very fast. Not much faster than of course, having
Speaker 2: the active record abstraction top of you and actual records
Speaker 2: buys you a lot of things like like distributed databases
Speaker 2: and like you know, having multiple database for the same
Speaker 2: like the same application, talking to multiple databases and shouting
Speaker 2: and stuff like that. You don't need that to sique light. Yeah,
Speaker 2: it's it's one database on the same machine, so you
Speaker 2: don't need all of this.
Speaker 1: You're making me.
Speaker 3: Think, Yeah, it's really hard to kind of change your
Speaker 3: mental models once I get set. Like, like, my background
Speaker 3: is in uh was in mobile development when I started
Speaker 3: working as a professional software developer, started off as an
Speaker 3: iOS and Android developer, so was twenty fourteen and I
Speaker 3: only switched to RAILS in twenty twenty. And I remember, like,
Speaker 3: when you're in a mobile app, going to the disc
Speaker 3: is considered slow. So when you launch the app, you
Speaker 3: basically load up whatever you can in memory, and if
Speaker 3: you're going to go to the disk, you have to
Speaker 3: kind of really think about the coreer writing because you
Speaker 3: don't want to lock up the UI. So I have
Speaker 3: this drilled in me. Going to the discus slow. Going
Speaker 3: to the disc is slow. So when I came to
Speaker 3: RAILS and I was trying to understand Russian doll caching
Speaker 3: in RAILS, I couldn't understand why there was a benefit
Speaker 3: because we still had to go to the database to
Speaker 3: get the updated app value of the record. I couldn't
Speaker 3: understand that in a web context, rendering the HTML is
Speaker 3: the slow part. It took me months and I'm not
Speaker 3: exaggerating months to change that mental model. So I completely
Speaker 3: get that people have been doing this for decades, have
Speaker 3: this models set about how you structure and appy the
Speaker 3: database hour and stuff, and now they're just like, it's
Speaker 3: really hard to just change that.
Speaker 2: Yeah, I'm glad, I'm glad you brought that up. Because
Speaker 2: one of the really nice things about switching to light
Speaker 2: stack or not nice actually the bad things about switching
Speaker 2: to light stack that it mhm suddenly highlights how slow
Speaker 2: your front end is, like the rendering path, like the
Speaker 2: view layer. Basically it's really highlighted because you eliminate a
Speaker 2: lot of the buttonnecks at the database layer, and now
Speaker 2: your bottleneck, your core bottle neck is rendering whether you're
Speaker 2: rending the Jason or rending like htmail in your app.
Speaker 2: So it just highlights it very very fast, and you
Speaker 2: now have to do with this level if you want
Speaker 2: to keep up with the with how fast back and
Speaker 2: this has become I think I think the other the
Speaker 2: other thing here is like rains has a lot of
Speaker 2: or Ruby itself had a lot of progress lately in
Speaker 2: the Jet compiler. So Ruby code is is gonna run faster,
Speaker 2: and it's getting faster and faster, which is really nice.
Speaker 2: We're looking at like fifty percent plus yeah, speed.
Speaker 1: Improvement depending on what your app is doing, yeh, but
Speaker 1: but yeah.
Speaker 2: I'm looking at something like Lobster for example. It's almost
Speaker 2: fifty percent faster in the benchmarks that the Shop of
Speaker 2: five team is sharing, which is really nice. It's it's
Speaker 2: a full fledged trade application. Now imagine this is one
Speaker 2: botle neck computation. Now the other bottle neck is IO.
Speaker 2: Now imagine if you also get even an er boost
Speaker 2: in that that regard. So, actually, by the way, moving
Speaker 2: to light stack doesn't just improve performance like I think.
Speaker 2: Think about your situation where you have two vms talking
Speaker 2: to a single database. You'd probably end up with one
Speaker 2: VM the same size and getting almost the same performance
Speaker 2: from your application, especially if you don't have a huge
Speaker 2: bottleneck in the view layer. If you do, then yeah,
Speaker 2: you need basically as computational power as you had in
Speaker 2: the previous setup. But still you're saving a lot of
Speaker 2: cycles that you otherwise would have wasted in the communication
Speaker 2: over the network stack. You'd save that for other competition needs.
Speaker 1: Very cool. We're getting toward the end of our time.
Speaker 1: Is there anything else you want to add or maybe
Speaker 1: just summarize where we're at with sequal light and light Stack.
Speaker 2: Yeah, so I would say there is a big update
Speaker 2: coming soon, hopefully like the next version of light Stack,
Speaker 2: which will have the light kid officially released. And I'm
Speaker 2: hoping also to bring one other major advancement, which is
Speaker 2: concurrent queries in Ruby. One of the issues of siquilight
Speaker 2: that it's a Sea extension and see extensions. Basically, as
Speaker 2: you're running Sea code, it takes over the process. The
Speaker 2: Ruby process doesn't allow other things to happen why it
Speaker 2: is running. So that results if you're doing a red
Speaker 2: query or quity, whatever the type of the query you're doing,
Speaker 2: if it's a long one, it will lock the process
Speaker 2: until it's finished. And the thing that I'm trying to
Speaker 2: to bring forth in the current next version is the
Speaker 2: ability to run multiple queries even in a single threaded
Speaker 2: application if you're losing fibers, or in different threads if
Speaker 2: you're dosing threads, so they will run concurrently next to
Speaker 2: each other and allow the application to be a lot
Speaker 2: more responsive even if you hit a daily slow query.
Speaker 1: Cool, very cool.
Speaker 2: Yeah, and I'm hoping I'm hoping to also make the
Speaker 2: other aspects of the story solid deployment backups and if
Speaker 2: if that, yeah, well once we're there. I I really
Speaker 2: think it makes a lot more sense to go with
Speaker 2: the simpler solution.
Speaker 1: Mm hmm.
Speaker 2: Regardless of what you're doing, it makes a lot more
Speaker 2: sense to go with the simpler solution. And hopefully this
Speaker 2: will be the simpler solution.
Speaker 1: Nice, all right, Okay, on.
Speaker 2: One final thing I want to I want to say, like,
Speaker 2: I'm really I'm offering basically a free consultation for anyone
Speaker 2: who wants to migrate to light Stack. So just ping me,
Speaker 2: and I'm happy to to help and and and and
Speaker 2: help you like gain the performance benefits and and and
Speaker 2: an opportunity for me to learn also about what could
Speaker 2: happen the wild in the process.
Speaker 3: Nice.
Speaker 1: How do people reach it for that?
Speaker 2: I'm available on on X so you can d M
Speaker 2: me at any time and also you can you can
Speaker 2: send me a emails. I'm not sure if I can
Speaker 2: share my email here or not. Yeah, basically it's old
Speaker 2: mo same handle like x at Gmail.
Speaker 1: Look what were you saying, Ayush?
Speaker 3: I was asking the exact same question as you.
Speaker 1: Okay, good deal, All right, well let's go ahead and
Speaker 1: do our picks are usual? You want to start us off?
Speaker 3: I didn't thought about. Does it completely forgotten me?
Speaker 1: Do these?
Speaker 3: Do you want to go first?
Speaker 1: While I'm thinking sure, I'll throw a few out. So
Speaker 1: it's been a few weeks since we recorded. Just because
Speaker 1: life is insane. Yeah, So I played a game with
Speaker 1: some of my friends that I hadn't played before. It's
Speaker 1: a really really simple game. It's called six nim. It's
Speaker 1: got a bunch of other names too, so you might
Speaker 1: have played it. I think it's also called like Take five.
Speaker 1: Not sure, let me look it up on Board game Geek. Yeah,
Speaker 1: it's called Take five or Category five or anyway. So
Speaker 1: what you do is you have numbered cards. It's super simple.
Speaker 1: Board game Geek has a weight of one point one nine,
Speaker 1: which means that it's a casual game that is pretty simple,
Speaker 1: and they say that eight and older can play at
Speaker 1: night as probably fair. It'll play up to ten players.
Speaker 1: Two to ten players. I think it took us a
Speaker 1: half hour to play a full round. Anyway, the cards
Speaker 1: are numbered zero or one. I can't remember. Start at one,
Speaker 1: but it goes up to one hundred, No, it goes
Speaker 1: past one hundred, but anyway, So what you do is
Speaker 1: everybody plays a card out face down. You flip the
Speaker 1: card over, and then the lowest card gets put out
Speaker 1: on top of the stack that it most closely matches to. Right.
Speaker 1: So if the top card on any of the four
Speaker 1: stacks you've got in front of you is seven, and
Speaker 1: you put out ten, unless the eight or nine are
Speaker 1: on top of any of the other stacks, you're going
Speaker 1: to put it on the seven, right, and then the
Speaker 1: next one goes and so on. You take the stack
Speaker 1: if you play the sixth card on the stack, and
Speaker 1: then the rest of the game is is let's say
Speaker 1: that the lowest one is a seven and you play
Speaker 1: a three. Then you get to pick which stack you take,
Speaker 1: and you're trying to get the fewest points possible, and
Speaker 1: different cards different numbers are worth different number of points.
Speaker 1: Some of the cards are worth one, some are worth two,
Speaker 1: some are worth three, and some worth five, And so
Speaker 1: you just play it til all the cards are gone,
Speaker 1: and then you tally it to score. That's the whole game.
Speaker 1: It was really fun, just kind of a you know,
Speaker 1: an interesting filler game. And so my first pick is
Speaker 1: going to be six nim I'll put a board game
Speaker 1: geek link and an Amazon affiliate link into the chat
Speaker 1: or into the comments a couple of other picks. My
Speaker 1: son and I went and saw a quiet place day
Speaker 1: one last night, and it was good. I've only seen
Speaker 1: a quiet place. I haven't seen a quiet place too,
Speaker 1: but still it was good. It wasn't as good as
Speaker 1: a quiet place, but it was it was definitely, you know,
Speaker 1: something that was worth seeing in theater. And yeah, I
Speaker 1: guess the only other thing that I'm just gonna let
Speaker 1: people know about if you want to follow me on
Speaker 1: Twitter or Instagram or something like that, I'm gonna be
Speaker 1: posting videos about doing AI. So I've been pulling a
Speaker 1: lot more AI stuff into top end devs. I got
Speaker 1: really inspired by the episode we did with Obie Fernandez
Speaker 1: about building AI agents, and so this is this is
Speaker 1: not on the level of Hey, I'm going to train
Speaker 1: all this data into a model and then use my model.
Speaker 1: It's it's a level above that where it's okay, I've
Speaker 1: got a large language model. You know what can I
Speaker 1: make it do for me?
Speaker 2: Yeah, I'd love to. I'm sorry, go ahead, no, go ahead. Yeah.
Speaker 2: I wanted just to have a shout out for Obi,
Speaker 2: like he's he's been one of my hoovie heroes for yeah,
Speaker 2: since so long.
Speaker 1: Yeah, yeah, And I guess he's getting ready to release
Speaker 1: the Rails eight way whenever that comes out, Rails eight does,
Speaker 1: which should be in a couple of months, but yeah, I.
Speaker 3: Think I think Rails it's going to be early next
Speaker 3: year because I think seven point two is going to
Speaker 3: come out like now, or maybe it's just out.
Speaker 1: Yeah. I know that they were looking to try and
Speaker 1: get it out around Rails World, which is gonna be
Speaker 1: in September, but I don't know what the timelines look
Speaker 1: like now because I haven't talked to David in a while.
Speaker 1: But anyway, yeah, so he got me excited about that.
Speaker 1: I've kind of been looking at Olympia AI which is
Speaker 1: the company that he put together, but I'm looking at
Speaker 1: building an ai I assistant or set of assistants that
Speaker 1: effectively are focused on podcasters. Right, So it's you know,
Speaker 1: it does a lot of the work of well, I
Speaker 1: want so many of these kind of episodes and so
Speaker 1: many of those kind of episodes, or hey, go invite
Speaker 1: such and such a person, you know, go invite Mohammed
Speaker 1: to come talk about light stack. Right, so it is
Speaker 1: smart enough to go make some web calls, go scrape
Speaker 1: the internet, go find your email address and then send
Speaker 1: you an email, right kind of thing. Or hey, we
Speaker 1: didn't record this week, put out an episode that we
Speaker 1: haven't you know, re released within the last year. Right,
Speaker 1: And so then it can go and it can say, well,
Speaker 1: these are the top ones, and right, it's smart enough
Speaker 1: to figure all that stuff out, you know, or even
Speaker 1: to the level of, you know, I usually said this,
Speaker 1: you know, and then he he kind of thought better
Speaker 1: of it afterwards. Right, it's like I shouldn't have said
Speaker 1: that because my client doesn't want me to reveal these
Speaker 1: certain things about Right, so I can go tell the bot, hey,
Speaker 1: go take out when I usually talked about this, this
Speaker 1: and this, Right, and then it'll go in and actually
Speaker 1: have the tools to go and extract it, right, you know,
Speaker 1: and my editor does a terrific job. But it'd be
Speaker 1: interesting just to see how far we can take it
Speaker 1: with that stuff. Right where it's cleaning up that kind
Speaker 1: of a thing, right, just tell it, Hey, we recorded
Speaker 1: a new episode, so it logs into stream yard and
Speaker 1: downloads the files and kind of does some preliminary cleanup
Speaker 1: on them. Right. So anyway, it's it's that level of stuff.
Speaker 1: I think it'd be way fun. So I'm working on that.
Speaker 1: And then I'm pulling together because I know a lot
Speaker 1: of other people want to learn it. I'm looking to
Speaker 1: do a boot camp where we use a lot of
Speaker 1: these tools, right down to some of the lms like
Speaker 1: the Lama three LM or something right where it's okay,
Speaker 1: I've got a baseline thing that I have to train
Speaker 1: some or maybe I am just going straight to GPT
Speaker 1: four or Gemini or some other thing right and just
Speaker 1: having it do the work for me because it is capable.
Speaker 1: And so just just working all that in and you know,
Speaker 1: just over three months, just help people go from not
Speaker 1: knowing anything about AI all the way up to Okay,
Speaker 1: now I can add these features to my application, So
Speaker 1: keep an eye out for that, mostly on social media.
Speaker 1: And yeah, I'm also looking at putting together an AI
Speaker 1: summit in September.
Speaker 2: So that's all anyway.
Speaker 1: Yeah, and then the last pick I have, and I
Speaker 1: think I'm going to connect it to a GoFundMe or
Speaker 1: something like it is. I would very much Now that
Speaker 1: next year is going to be the last year for
Speaker 1: Rails Rails Comp, I would like to put something together
Speaker 1: like Rails Comp. And I know we have Rails World
Speaker 1: and they kind of move it around. I guess they're
Speaker 1: doing North America and then not North America, and so
Speaker 1: I would like to alternate opposite them, right, so that
Speaker 1: there's a conference in North America every year, and then
Speaker 1: there's a conference somewhere else out in the world, right,
Speaker 1: and just see if we can get more access. But
Speaker 1: there is no way I can afford to cover things
Speaker 1: like an ex bo hal or things on my own.
Speaker 1: And so if you all are interested in something like
Speaker 1: that and would like to contribute, I mean, obviously we're
Speaker 1: we're gonna you know, at certain levels of you know,
Speaker 1: of helping us out, you know, give you tickets or
Speaker 1: shout outs or you know, any number of other things.
Speaker 1: You know, and I'm thinking, you know, kind of base level,
Speaker 1: maybe you get a ticket, a little higher level, you
Speaker 1: get lifetime tickets, you know, another level, you get sponsorship
Speaker 1: at a certain level. So anyway, if you're interested, I
Speaker 1: did buy the domain. I think it was railsxpo dot com.
Speaker 1: So go to railsxbo dot com in like a week
Speaker 1: or two after we recorded this, and I should have
Speaker 1: something up where it tells you what we're looking at
Speaker 1: for contributions and things like that. So anyway, that's all
Speaker 1: the stuff. I kind of wanted to get those out
Speaker 1: there just because those are things that I want to
Speaker 1: put together for the community that I think people are
Speaker 1: interested in. Are USh, what are your picks?
Speaker 3: So I've got to both non technical picks. This week,
Speaker 3: I reread a book from my childhood a few weeks ago.
Speaker 3: It's called Havoka Rasa by Matthew Riley, and yeah, it's
Speaker 3: just one of those books that it kind of shaped
Speaker 3: my teens a little bit. Like it has a quote
Speaker 3: that I absolutely love and I loved at the time
Speaker 3: as well, which is two air is human to make
Speaker 3: the same mistake twice as stupid. And I've pret I've
Speaker 3: pretty much reread the book just to try and remember
Speaker 3: what the context of that quote was, because I remember
Speaker 3: the quote, but I couldn't remember any of the context
Speaker 3: around it. But yes, I enjoyed reading it again. It's
Speaker 3: it's a book aimed more at a like probably a
Speaker 3: teen audience, but even now in my thirties, I still
Speaker 3: quite enjoyed it, despite it being quite far fetched at times.
Speaker 3: But yeah, that's one pick. The other pick is a
Speaker 3: musical pick, which I don't think I've done for a while.
Speaker 3: It's a band called Solstice, who have been a little
Speaker 3: bit obsessed with, unhealthily obsessed with for the last couple
Speaker 3: of months. I basically had their two albums, the last
Speaker 3: two albums on repeat throughout my work day since about June,
Speaker 3: like start of June, which is just a bit mad
Speaker 3: and I haven't got bored of them yet. So yeah,
Speaker 3: if you're into any kind of rock music at all,
Speaker 3: I'd highly recommend checking out Solstice the last two albums,
Speaker 3: All Light Up and s S. I A, yeah, those
Speaker 3: are my two picks.
Speaker 1: Cool, how about you, Mohammad? You pick for us?
Speaker 2: Not not not really, I'm not sure if that is
Speaker 2: interesting enough. But as we were like arranging to to
Speaker 2: get into this discussion, I was working on concurrent index
Speaker 2: creation for sequal Light, which will allow indexes to be
Speaker 2: run in the background. I have I have now like
Speaker 2: a set of like features that are not open source yet,
Speaker 2: and I'm hoping i'll be able to build a service
Speaker 2: around that soonersh and allow you to host your Ruby
Speaker 2: siqual Light powered applications for a fraction of the current
Speaker 2: costs at a much higher performance than people are accustomed
Speaker 2: to without having to deal with with like Docker images,
Speaker 2: kemal or things like that, even even simpler more like
Speaker 2: hero gon Stroits very cool.
Speaker 1: All right, well we'll go ahead and wrap it up.
Speaker 1: Thanks for coming. People find you on socials as old mo.
Speaker 1: Just remind people of that, and yeah, until next time,
Speaker 1: max out. Everybody