Hey, welcome back to another episode of the Ruby Rogues podcast. This week,
on our panel, we have Valentino Stole. Hey. Now, I'm
Charles Maxwood from Top Endevs. We already have people coming in on the live
stream, which is awesome. I think they're all coming in on Twitter.
We have a special guest this week that is Stephen marg I'm Steven. Welcome
to the show. Do you want to introduce yourself for real quick? Yes?
Thank you for having me. As you said, my name is Stephen.
I am living here in Berlin, Germany, though I am originally American.
Just gotten married earlier this month. I saw that on Instagram. Congratulations.
Yeah, thank you and excited to be here and talk about sequel light
and Rails. Very cool. So where do we start. I'm just going
to give a little bit of context because I mean, I've used sql light,
but I mean everything I do I do on Postgress. I think most
Rails developers go that way, but I think seql light is also the default
if you start a new Rails project, or at least it used to be.
Anymore, I just tell it no, I want Postgress. So yeah,
if you know you're talking about seql Light, like it's a serious database
on Twitter. I know, weird, right, I know. I couldn't
think of a better way to put it. But the way you talk about
it and the way we talked about it on the Ruby Dev Summit, it
made it. You made it sound like you could honestly use this in other
places. And I've heard other people being interviewed. I think somebody from Planet
Scale or one of the Internet hosted databases was talking about how sequal light is
the real deal now to not just my SQL or Postgress. So yeah,
is it the real deal and why change our minds? M Well, it
is the real deal. But like with any database, and this is I
think the key point, there are trade offs. So at like a very
high level, just my sort of sense of things, right, Like,
Postgress is great because it comes with a lot of functionality out of the box,
and it has a really rich community of developers and set of extensions.
So you want to be doing stuff with vector databases. You've got pg vector,
you can plug that into your Postgress database. But compare that with my
sequel and the v tests sort of layer that can be put on top of
that, and the scaling that has been done in the last two decades on
my sequel, there aren't the same sort of solutions that are as battle tested
at the same level of scale for POSTCRIS right, so something like planet scale,
which is sitting on top of the test is a relatively unique offering for
my SQL. And then sqlight has I think a unique sort of value proposition
because it offers you, like the full range of sequel, Like there's really
nothing that you need to do with SQL, the language that sqlight doesn't offer,
you can if you want to run it in a client server architecture.
The libseql project, which is a fork of the cql I codebase by the
Turso guys, and the platform that they are building on top of that is
like breaking the client server architecture on top of seqlite for edge computing. But
you have the option, which is unique with SQL ie right to run it
on your application server machine right next to your rails process and just simply remove
network latency as a consideration. And I think that the major shift that has
been happening in web development lately has been a growing recognition that seqal light is
production grade software for web applications, with the caveat that you do have to
configure it to be ready to use for web applications. And where I think
there has been this odd disconnect is that there's been the sense that like,
oh yeah, postcress my sequel these are obviously web application databases. Just get
them up and you're ready to go. But of course you have to configure
those as well, Like there isn't really a database that is completely blind,
plug and play. And as people have started lowering that barrier to entry and
just started to demonstrate, hey, there is a lot of power here,
there's a lot of flexibility here. This is a model for building and deploying
applications that offers some unique value adds, whether it's to developer experience or to
performance, or to simplicity of deployment. And it I think it really is
just a question of education around what it takes to get a robust, resilient,
production grade deployment of an application backed by Sequelite out into the world.
And that education has been growing in the last couple of years. So yes,
it is the real deal. So I want you said production grade like
twelve times. I wasn't counting. I'm just throwing the number out there.
But what do you mean by production grade. Yeah, So I think that
there have been three major barriers to using Sequelite out of the box in a
web application, and I think it has led to the sense that Seqlite isn't
production grade. So the first barrier is concurrency. So everyone who has read
even a little bit about using c qlite for a web application will very quickly
come across people claiming, hey, you absolutely could never use this for a
web application, because in order to have a web server that can deal with
even just a reasonable amount of requests coming in, you're going to take advantage
of the stateless stature of HCTP and deal with multiple web requests concurrently with your
application process. But Sequelite running on that same machine is going to linearize your
interactions with the database. And by default, if you don't touch anything,
you're using the original model that they have for ensuring transactionality. You're going to
have linear operations, whether there reads or write. If you want to do
anything, you do them one at a time. So the fact that you
have concurrent web requests is negated. You everything bottlenecks at your database, and
people say, of course that's never going to scale. Now there are two
key points to make on this barrier. The first one is people really too
quickly forget to do like the basic math on if you remove latency and you
give yourself some head from so you go from talking to a postcos database over
a web connection which can take some number of milliseconds, to talking directly in
process to seq light, which can take some number of microseconds. You have
ten, one hundred up to one thousand more opportunities to do things right when
you move from milliseconds to microseconds. So even as you linearize things dealing with
different orders of magnitude of latency, and that gives you possibilities. And then
the second one is that you can configure sequlite to have concurrent reads, but
as of now there is no way to get concurrent rights. So out of
the box, sequlite is linear operations. Web applications tend to deal with concurrent
operations, and people think, well, that's a hard a hard no,
but it isn't. That's barrier one, barrier two, and I'll just stop
at barrier two is the main one or these are the main two? Would
be around backups, right, People are like, hey, this is ridiculous.
If you want to have something that's like key money on the internet,
or is in some way trying to be valuable, you need to make sure
that it's resilient. Something happens. We know how the world is. The
can get unplugged, or storms come in. Who knows what. If you
have everything running on one machine, including your persistent data, and that machine
goes down and your persistent data is now gone, well, it's easy to
spin up a new machine in a new application process, not so easy to
recover lost data. So the fact that seqlight is on that same machine that
you don't have network latency is a vulnerability for persisting your data and keeping your
system resilient to system failures. And in the last couple of years, there
have been a number of tools that have been built to back up seqallied databases
to some other place and to make it easy to take those backups and spin
up a new instance of that database somewhere else. Probably the most well known
one now, the one that I have used the most is a project called
light stream, which just streams rights to your sql ie database to an S
three bucket or really any S three compatible bucket service, and has a very
simple CLI. You can just pull it back down onto any machine. So
when I say production grade, I tend to mean these kinds of ideas where
people think about like, sure, I can do this like quick script on
my laptop, and it's nice to have some quasi persistent storage. But if
I'm trying to build a business, if I'm trying to build a SaaS,
if I'm trying to build an application that I want to have some meaningful presence
on the Internet, I'm going to need to have the ability to handle concurrent
web requests. I'm going to need to have the ability to spin up new
web servers with my application and keep my data persistent across them. And there
wasn't always a clear story for how simple or easy or possible that was to
do a SQL, and that story is being written much more clearly now.
First of all, you have a great serious blog series that we should link
to on uh, you know, successful usage of sequel light in rails and
configuration and performance metrics and all kinds of great stuff that I think people should
read and dive into. And I guess, like my biggest you know,
I've been on the same boat of incorrectly thinking for the longest time that sequel
light is not performant, and oh like don't use in production, but you
know, sure in development or whatever, you know, testing or you know,
even just like small like hobby apps, sure it could be. You
know, fine, but once you get above a certain level or scale or
whatever it may be, then it, you know, don't use it.
And I'm sure I don't. I'm not alone thinking this, right all.
So, right, and so you do shed quite a lot of light on
you know, Okay, there's a lot of butts in a lot of those
you know, ways of thinking, and I definitely agree with you there,
and I'm kind of curious, like to get some further insight there on well,
like what are the limitations? Right? Like, well, what point
do you hit SEQL light and then it no longer worked for your app?
Right? Because I think that's where a lot of people are still hesitant to
dive into it because they still don't know what that barrier is. And then
you it's one of those things where okay, you don't know it till you
know it, and then it's too late and then you're trying to scramble to
find another solution. Right, So, like what are those things right?
Like what should you be looking out for to know if sql light is the
right option for you. Yeah, it's a good question. It's the question
I get most often. And there are a few different angles to take on
an answer. One of them is where are you comfortable in investing your right
So if you're someone who has you're trying to start up a side project,
you're you're hoping to get some money with it or just sort of do something
of interest and value, and you have a decade of experience with Postcrest and
really feel comfortable, like with the path two, evolving a system over time
with postgris, you know, just stick with them. These days, it's
there's really not actually a lot of limitations with sequel I. So here's here's
like the first big one that used to be more of a limitation. Right,
So there's a major difference between seqlight and postcress, my sqel Oracle,
like pick any of the others. Most database engines follow the client server architecture.
So it's like you have you minimally have two different processes, and most
of the time you have two different machines. You've got a machine running your
application, you've got a machine running your database, and you most often I
have multiple clients, have multiple machines running application in one machine running your database,
and you talk to your database through web requests, and there are like
there's been decades and decades of exploration, are like different ways to scale that
kind of architecture, to make that kind of architecture resilient. And when you
go from an embedded database, there's one machine, there's one process. Everything
is just happening right there too. You know, something like get Hub scale,
Shopify scale, like there's no vertical scaling, will never be able to
handle that. You're going to have to horizontally scale. So then how do
you horizontally scale in your database is living on each machine? That was a
problem, that has been a problem for a long time, and so I
have said, you know, if you are starting a project, and the
way that it succeeds is it gets to that scale, right Like if you
go and get venture money, you probably should have chosen my SQL from day
one, right, Like you're shooting for planet scale, not TM from day
one. But that's actually not even a technical limitation anymore. So, whether
it is the predecessor to the light Stream project light FS from Ben Jonson at
fly Io, which is a tool to have a replicated logical database spread across
multiple machines Sequlite database obviously, or whether you're using a platform like Turso,
which, as I said, like brings that client server architecture but also allows
you to have read replicas on each of your machines and it'll you know,
manage consistency for you. Like there are now and there are others. There
are multiple tools now to allow you to horizontally scale the sequel lie database.
So is is that particular scaling path as battle tested as like the test on
my sequel. No, it's not been around for two decades. You don't
have a company like YouTube that's running it. So again it goes back to
like your your risk profile. If you think, on day one, I
really want to be at that scale, you probably should just use my sql
env tests, right because the why take the risk if you know that's where
you're going. But for someone who is like unsure of what the future looks
like, I really do think that there are so many wins that you get
on day one and week one and year one with the developer experience and the
performance characteristics and the power and simplicity of like leaning into vertical scaling for as
far as that'll take you, that you should really give SQL a long look
and you can really fall back on the security be like, if you really
do get very successful successful such that like vertical scaling can't work for you anymore,
you don't have to change databases necessarily. But the other thing that's true
is like, if you do get to that scale, you're probably making money,
and probably good money, and at that point you probably could change databases
if you want to. And going from a sequelite schema, which is you
know it's going to be a looser schema typically two postcress for my sqel,
I think it is generally easier than the reverse direction. So I talked around
your answer. The short answer is I really can't think of like really hard
limits, to be perfectly honest with you, but I do think that there
are different risk profiles and there are different reasons why you might or might not
choose to use it. But like as of today, well, okay,
I say that, of course I should stay out loud. There is the
one key limitation, right which is if you have like massive right through but
if you're building an analytics platform to compete with Google Analytics, and you're just
going to get like the core of your businesses, you need to deal with
massive amounts of rights. Don't choose sequal Light today. That would be a
hard limitation. They are working on concurrent rights, so that limitation might not
be true in one year, but for today, I would not build a
analytics platform on top of Sequlite. How about that? Yeah that makes sense.
I mean I'm thinking more. I mean we should also not build an
APM with sqlid. Yeah. Again, it does sort of like come back
to your your expected growth profile. I have built an APM with sqlight and
it's a very like it's an internal small one, right, But I was
like, I'm not expecting if you know, the thirty seven Signals crew have
talked about load testing their seql light back Campfire application and on the biggest machines
that they have, you know, with their cloud exit, they're they're running
their own machines in a in a data center and like their bfous machines,
they're getting fifty thousand concurrent requests. Are there APMs that could handle that?
Yes? Right? If you like, I want to build a business and
I am either the next app signal or I have failed. Yeah, I
probably would not choose eqlight on day one. I gain my sort of hot
take on this is and all of the recent Internet drama aside, I personally
would probably choose either sql or Planet scale on day one. Those are the
two options I would choose. It's like, I'm either going for big scale
and I know that, like that's the characteristics up the app I'm building,
or I'm just a regular scale and seqal light handles regular scale quote unquote right
like, good luck getting if you can build an application and you have fifty
thousand concurrent requests, like, good on you you have, you have definitely
succeeded. So sure answer is maybe, you know, honestly, maybe it's
it's just funny to me because seql light comes out of the box of the
reels and for the longest time, you know, you question it, like
everybody's using sql light, And the more you use reels and you the more
like one off apps that you start to make, right like, even just
like trying something out like it's great. It does most of what you need,
right like, And I feel like most people won't even hit the limitations
of it or see any side effects of that, you know, until much
much later to your point where there is an actual you know, limitation and
you have the knee the financial uh you know ability to navigate that, right
uh, And I feel like most people won't even get there, you know,
to be honest and uh, I mean and in all honestly thanks to
the shopify folks for all of their efforts with it as far as like a
you know, live stream as an example, but you know, all all
of the other integrations to just make it super straightforward to get set up and
operational. Like I don't know, I feel like, why bother? Why
bother having Like I'm definitely coming around to that off the cloud idea for for
it specifically of you know, why have all these extra services that just aren't
needed and it's just to help you scale maybe right? Yeah, I mean,
this has been one of I like major realizations and principles that drive a
lot of my personal interest in the topic is recognition that for me, myself,
I really am trying to be conscientious about fighting off unnecessary complexity, like
it's just swarming around us, and especially like for where we are at in
the evolution of the Internet and the economy, of the internet, right like,
everything is assas. Now everything is a product now like, hey,
you need authentication, here's the sass you buy. You need emails, here's
the sass you buy. It's and it is, you know it just like
greats on me to see how like common the I don't know, the mind
virus, I'll be sassy. Is that the correct way to build a web
application, a service, a business is to sign up for fifteen different SaaS
platforms, stitch your things together, have no real ownership or understanding of like
what your rankin app is doing. Then, like I mean, and we've
been seeing more and more of it, right like, and then you actually
get some success or more commonly, the actual noise of the internet and the
people who just like doing naughty things and having their little bot farms run and
you just like boom all of a sudden like whoops, like I have a
bill for ten thousand dollars from this one service for this month. And yeah,
as you said, I mean, it's like it's actual like statistics,
like most applications are not going to actually need like there's real value there.
I'm not saying that most of these businesses shouldn't exist. They should exist,
but most people shouldn't pay for them, right like we servers are cheap,
servers are easy. Like I'm no DevOps genius. I'm certainly not an obstenist.
I barely can use Docker, you know. But I have spun up
many many servers in my day, and it's not rocket science. I have
maintained them. That's also not rocket science. It's not incredibly stressful. It
can be when you make like mistakes, and that's why this concept of production
grade is important to have a sense of like what does it mean, you
know, to give to give one specific example, I had an application that
we use at work for business that provides profit running on sql I and I
had been thinking for a while, you know, I really need to go
in and install light stream and set that up. But I didn't get around
to it. And then one day I was cleaning up and renaming my applications
to make them more consistent, and I did not recognize that in the little
web platform that when I rename an application, what it's going to do on
the server? Is this going to do our MRF? And then redeployed and
it wiped my sequel like database, you know, and I spent like a
full day trying to recover it from memory and it did not work and I
had to go back to the business and yeah, you know, that was
a shitty day. And I to put in backups after that, Right,
So this is why I do bring up production. It's like, yeah,
you know, you can footgun yourself, but you can set up backups for
postcards or my syqul or anything. But like, none of these things are
difficult. And it just saddens me to think that so many developers and especially
newcomers like come into web development and think like a, you have to understand
docer and Kuber needes and server lists, you have to pay for fifteen SaaS
platforms. It's like, man, it's going to take a PhD in web
development and at least five hundred dollars a month to even like start to do
something. It's like that just isn't true. Like more people can get started
with less prep and less upfront money, and I really want to make it
as widely known as possible like this, there is this alternative path. You
know, you can just go get a digital Ocean droplet, put a rail
server in a sequel like database, like you have one machine and you can
grow that you're just in your little digital otion dashboard, Like you could just
bump it up over time, and you can do that for a long time.
Like go and look at the biggest machine on there. It's really big
and it can handle a lot of requests, and sql I can handle a
lot of requests. And these things like can be cheap and they can be
simple, and that opens up opens up so much opportunity for like experimentation and
more people coming in and trying things out. And that's like that's what drives
the passion behind it, right, It's like doesn't have to be this other
way, Yeah, I feel Yeah, it just reminds you of CGI,
right, Like back in the day, you had a directory CGI and you
uploaded a programmable file and it ran, and you know, it was that
easy. Yeah, everybody had FTP line or whatever. You know, you
just uploaded the file and then you were ready to go, and like people
ran like legit businesses off of it, you know where you know, guitar
tab sites or like you know, all of the original search engines. You
know. It was just like a handful of programmatical files in a directory somewhere
and it just ran and I feel like, you know, maybe maybe we're
like a little bit beyond that, right, but it should still be that
easy. You know. It's like the underlying technologies haven't really changed that much.
Uh, you know, like you still have servers and clients and a
browser, you know, and it's it's those three things. Uh yeah,
I why make it more busy? I was just going to add with all
the different SaaS products you can add, right, So top endevs for a
long time ran the authentication through one of the authentication services out there that you
pay for, and yeah, it was more work to set up and manage
that than it was for me. Eventually, I just switched it back over
to deviz, right because I set it up once and then yeah, occasionally
I have to help somebody reset their password or something stupid. Right, Hey,
oh yeah I jacked that up. Let me fix you, right,
And yeah, I feel this on the rest of the infrastructure too, right.
I Mean, one of the things I really like about camal is that
it kind of does that, right. So it pulls down the postcress doctor
image, it pulls down the red as a Docker image, it pulls down
you know, a Ruby Docker image and then runs my doctor file against it
and then it says, hey, that's over there and that's over there,
and that's it right, and load balancer. It does the load balancer too,
and that that's what I love about this. So yeah, having something
simple where it's just like, oh, yeah, it's just going to run
alongside the app, the app server, right, run alongside Puma or whatever.
Yeah, that is beautiful, right, I don't have to fuss about
it, yeah, exactly. And then on top of it, you get
like these really nice benefits of like it's I would recommend for developers to just
like spin up an app and throw it up somewhere, right, use camal,
use hashbox, like use whatever, like do it quick an do it
easy, and just like play around and get a feel for what it feels
like on like two fronts. And this is where it like really sold me
and I like became passionate about it. So the one is it is it's
just really nice to have a simple enough system that you can keep it in
your head. And so the fewer moving parts, the simple as the system.
And like I have these applications that I have built at works, you
know, sort of like these R and d applications that I am effectively the
sole developer and maintainer of them. And you know, I'm an engineering manager.
I'm in a lot of calls. I have like these thirty minute windows
sometimes between calls, and I have to debug production issues that come up,
and I can actually do that. I can go from like I haven't heard
anything, I get a Slack message, I read the problem from that moment
to I have checked that the fix is live and fixes the problem in production.
I have consistently been able to do that inside of a thirty minute window
for over a year, and that is like a magical feeling. I every
single time I do it, I'm like, Wow, this feels amazing compared
to most of my work experience. And that is like driven by I have
a rails app and see collide astaut That is it. I have nothing else.
I don't have Rettis, I don't have. I have nothing. I
have like four different sequent databases, right, I've got my Q database and
my cash database and my model's database or you know, production database is what
it's called. But I have this one piece of software, and that piece
of software I don't have to do anything weird or fancy it just runs inside
of my application process. There's nothing to like spin up. I don't need
to have a PROC file necessarily. It's just is there, and that like
developer experience is so nice. And then like on top of it, it's
just like a file. And so if I need to which I've done many
times, right, like a production issue comes in and I can tell,
uh, you know, some data has gotten out of sync here, but
I don't know exactly how or where or what I need to dig into the
data. And I can literally like do a safe backup. I can use
the fact that I have these light stream backers. I can just tell light
Stream on my laptop, hey, replicate the database to my laptop please,
as of right now, what is the current stay of the day. So
I get a mirror of the production database on my laptop, I spin up
the app on local host. I've got all the production data, and I
can mess around and figure out like where is the where did the data go
wrong? Where's the bug in the background job that messed this up? And
I know it's completely safe. I don't have to remember to do the dash
dash sandbox in my Rails console or any of that, And it's like,
you know, one CLI command to get it down there, and there's like
so many of these little things, you know, whether it's the one of
the first articles I wrote was on doing database branching locally with get and it's
like the simplest thing in the world to set up with Rails, and it
just removes an entire class of problem that has annoyed me for a decade of
web development when you're working across these different branches and your you know, your
schema gets borked and it's just like they stack, they stack all these things,
and then at the very end of it, you're like, oh man,
every single database operation I have done went from n number of milliseconds to
n number of microseconds. Like I just got one hundred x performance improvement,
you know, like action pack is my bottleneck for all of the Rails apps
that run the stack, you know, So that is awesome. Real quick,
just dive in deeper to the branch specific databases you're talking about, because
I thought this was fascinating reading through this. Uh, like how does that
set up work? And what are you saving there? Yeah, it's it's
a really nice feature. Let me start off by explaining, like the problem.
So here's the scenario that many of your listeners are likely to have experienced.
You're working in a feature branch, so you get check out B Feature
A. You get to work. It's a long running feature branch, kind
of complicated. As a part of the future work, you have to make
changes to your schema, so you generate let's say two different migration files.
You added a new table and you have renamed a couple of colmns on an
existing table. While you are in the middle of working on this feature branch,
a major production issue is reported and a bug fix needs to be made.
Someone has created a hot fix branch and they ask you to review it.
You're like, okay, I did the code review, but I want
to make sure before we push this live that like the fix is actually doing
what we wanted to do. So you get check out hot fix one and
run your application and things aren't working, maybe like your Rails app won't even
boot. Maybe as you boot into the app, things are there, and
you're like, what in the world is going on? What is like?
What is broken here? Something is very clearly broken. And what has happened
is that the schema changes that you made to your single only local. You
know, development database in future branch a are still there when you checked out
hop fix one, but hopis one doesn't expect those changes to be there.
You name some tables or some columns on some tables, and the code in
hotpics A expects the database to be in the schema that existed in main,
but your schema locally is different, and so database and the app have a
mismatch and things blow up. Ah, this is so annoying. So you
have to like get your database state back into a correct place and then be
able to once you're done reviewing that topics or however you know, you change
branches and you get back into the future work, you want to bring those
change. So that's like the problem, and it's an annoying problem, and
there are various gems out there to try to help you with that. Here
is the simplest, most resilient possible way to solve that problem. If you
could, in some magical way have every single git branch have a completely separate
database, well, then definitionally you would never have this problem. That would
be nice. Now, when you're running postcress, that might be tricky,
might be possible, or it might just be computationally expensive or space who knows.
I've not really honestly tried it, but with sqlight it is both cheap
and easy. And all that it requires is you take advantage of the fact
that the database EAMO file that Rails uses is processed by ERB, so you
can just throw some RB in there, and in that ERB you can run
shell commands, so you can just PLoP in there, Hey, what is
my current get branch name please, and you can make that the name of
your database. So now you have dynamic database names. Then the only other
thing you need. Sql I itself will automatically touch non existent database. So
if you say sql I, please open database A, that file doesn't exist,
QUL I'll just create an empty file. So all you need is to
ensure that when you start your Rails app, if your database hasn't been loaded,
run your migrations load the schema. However you do it, that it
is right. So you just want to say, take whatever the current encoded
version of my schema is in my repository and run that. So what I
typically do, and I think what's in the article is like it will run
your migrations and you can do an after prepare block in your development dot erb
configuration file. So with those two changes, what you have done is you
have set up a system where whenever you change get branches and in any form
you boot your rails app. So that can be starting the rail server,
that can be starting a rails console, that can be using the rails runner.
It will create, if it doesn't exist, a new database file.
It will run all of your migrations and get the state set up. If
a database file does exist but there are some few migrations, it'll just run
those new migrations and get the state to where it needs to be. And
you can then enhance that if you want it to. That's like where I
stop in the article. I need to write some more about it, but
it's very easy to enhance that in like you can do seating. You know,
you can have generic seeds that are always run, so you can have
a baseline set of data that's populated into all of your branches. You can
set up branch specific seating such that you and you can make that more generic,
like around scenarios. There's all kinds of fun things, but in general,
you have the ability to have completely isolated like actually separate databases for each
of your GET branches, and so it simply becomes impossible to run into the
super super annoying when it happens, because it's like, you know, you're
completely blocking me in the moment where I don't want to have to deal with
any of this these schema conflicts as you move across GIT branches. That's interesting.
I mean, how do you do you do you not mind data getting
blown away between branches at that point or have you solved that problem? And
like you know, stream sum updates. Yeah, so for local development,
I I've ended up with a mix of a couple of things, and I
have a couple of later blog posts about this, but I should honestly finish
up writing some of the other stuff I've done. But in general, I
tend to start off with just like a generic seeds file. It's like,
let me have some usable data, and yeah that seventy five percent of the
time that is sufficient. For some applications. I have ended up in a
place where it's like, well, I have kind of built up a whether
it's like pulling down a production database or pulling down a staging database, like
I want to keep this across branches and I have just written some Ruby scripts.
Basically, I have a I have a blog post about this. So
it's like you know dB C, and I just say, okay, I
want to take this database and I want to clone it into a new file
with this name, and then I am just wiring up the name that I
point to in my data saml to that, and there's some automations you can
layer on top to connect that to get if you want. For the most
part, what I have found is like when I want to control the actual
data, the schema and the data this I was like, I want this
file that I sort of move away from the automatic kit branching stuff, and
I just manually go in and say comment out my default database line and database
EAML, and just point it out a particular file. So if you were
to look at like my laptop for most of the apps I work in,
and you look in my storage directory, on average, you will probably see
at least twelve different seql I three files in there. So I just create
a bunch of these different scenarios. I pulled down production on this date,
I've pulled down staging on that date. I've created this Git branch and that
kit branch and they all live there, and sometimes I manually point my app
to one of them or branch off and just do manual branching as if it
were getting me, Like I want to sort of play with the schema and
the data in this direction, give that a name, and wire the app
up to that, and it's all. It's like quite simple, but it
is quite nice to be able to like have that flexibility or your schema,
your data. You have the full range of like logical branching that you could
do with get and text files with your database. Yeah, it's super cool.
Yeah, I mean, look looking more at looking more at the configurations
you've got, for like having multiple databases all SEQL light, like, do
you find that there's like a limitation point of like too many databases of the
same sequel light server like serving for uh or or where it's just like becomes
too unmanageable to like reason about all of the different things. Right, Because
I saw you have like a solid errors which is kind of interesting where you're
like, you know, soaring all of your errors off to its own database
cash I imagine too, right, like you start like stacking up all these
different databases, is there is there a limit to the madness or is it
like still pretty reasonable, you know, for everything that you'd need. So
if there is a limit, I haven't hit it. The most databases that
I have in production is like six, I think, and like operationally that
it's nowhere close to that. I'm not even sure if there is a limit.
But conceptually, no, it hasn't been that bad. What I have
found is like, you know, for the nature of the system, I'm
going to have this bucket no matter what. I'm going to have to do
some kind of IO or some for an object of this type, no matter
what. And for me it is a conceptual win to keep the actual operational
infrastructure there like the same. You know. I have been able to become
a master of rails and SQLI and that's it, and I can do everything
I need. So I don't know all of the reddest commands. I don't
know all of the reddest data structures. I have a passing awareness of them,
and I wouldn't know how to optimize a reddest instance for caching. I
don't know how to optimize I've read as instance for background jobs. I don't
know it would redd us be a great choice or a terrible choice for the
back end for something like solid errors, you know, storing exceptions that happen
in your application somewhere. So being able to leverage a flexible, powerful tool
across these different problem domains, and you know, give myself the challenge,
but also give myself the time to become a master to like, I know
how to optimize SQL queries for caching, you know, I know, I've
spent time and I've thought about that, and it's just SQL, and I
know that language and I know it well, and I know how to optimize
like out of structure tables and what kind of queries to run and make sure
I know how to optimize SQL and sqlight for a background job que and I
know how to do that for errors. Each one of those I am able
to leverage expertise that I have gained in the past one. You know,
like there's a lot of overlap. So if you want to have a good
schema for your active record models, a lot of the wisdom and principles that
are you're going to learn in that problem area are going to roll over when
you're like, well, what if what would it look like to back a
queue, a job queue with sql. Okay, let me think about how
to you know, schema that out, what kinds of queries? How do
I want to deal with this stuff it is and what's really nice in the
rails ecosystem. As thirty seven Signals has gotten into building new applications, has
gotten into Sqli, then has been exploring the benefits of reducing the number of
moving parts and publishing more of these solid our gem, solid Q, solid
cash, solid cable. I imagine is being worked on is that I don't
even have to be the only expert. You know, there is now a
community of experts, and I read through the like, for example, the
solid q source code, and Rosa did a way better job at it than
I did, you know. So it's like I just get to take advantage
of the fact that she is a really top notch developer who spent a lot
of time thinking very deeply about how to structure a queuing system backed by some
persistent SQL database, And I just get to run bund lad. You know.
So I haven't gotten anywhere close to a limit, and I have actually
found like every single time I do it, I'm like, you know,
I should have more databases, Like I'm right now working on another gem,
which is like the conceptual sibling to solid errors. I think I've named it
solid metrics, which is just let me keep track of the metrics. You
know, I want to know them. I don't have super high needs.
I don't have a lot of customizability. I just need to know the basics.
Rails instruments itself already. It's the same thing with airs. It's all
instrumented. I just want to put them somewhere and put a very basic WebUI
on top of it. And you know, I've got like three more ideas
on top of it. So probably, you know, come back to me
in two years. I'll probably have like twelve different databases and some application somewhere
in production and might not be ready to stop there either. Very cool.
So I guess my question is because you said I know how to optimize sql
light for a Q, I know how to optimize it for something like solid
air, know how to optimize it for you know, whatever else. How
do I figure that out? I mean, do I just stack overflow the
thing, or yeah, I know how to tweak sql light for my use
case. Well, I'll start by just distinguishing one one small point, which
just that you know, for those particular domains, that's just SQL not SQL.
You know, like what kinds of tables with what kinds of columns and
what kinds of indices you're structure if you know. So somebody Mike had to
think about which reddest data structures and reddest commands should I use to have a
resilient and performance job back end, and other people I've thought about if I
went to like have those same behaviors, but I wanted to back it with
a relational database, how should I structure my tables? What kinds of industries
should I have? What kinds of query should I write? How do I
do that resiliently and performantly? That's a sequel problem. But when it comes
to SEQL light, I would say there are sort of two parts to an
answer to your question. So one part is one of my goals is to
make the amount that you or anyone else would have to learn about seqlight to
make it work incredibly well for a rails application specifically and for a web application
more generally, as close to zero as possible. Right, So you know,
we that's what I want the user, I think, and it's what
I would have wanted where I not the person who got you know, sort
of infected with the bug of really getting into it. But one of the
great levers that is available in using a web application framework is that these kinds
of things can be pushed into the framework itself. So you know, we
made a number of changes to the default configuration for sequel Light as a part
of Rail seven to one, and that like just there were six changes.
I have a whole block was about it. Like those six changes alone make
a massive difference. And if you had taken if you take a default Rail
seven zero application and put it on the Internet, you're going to experience pain.
And if you put a Rail seven one applicate Sequelight application of the Internet,
you will experience notably less pain, but still some pain. And we
are solving the major pain points at the rails level itself. So whether it's
improving concurrency or improving the performance of talking to seqlite sort of like the Ruby
and c bridge, I really want to have as much of that live in
the framework as possible. But of course, as for anything like your app
is going to have some special use cases and things where you're going to have
to move beyond the framework and so like, how do you learn to take
advantage of Sequela? And I would say that there are like compleately three things
I would recommend. So the first one is the sequel Like docs are extensive,
and I personally find them quite approachable. I mean like it's not just
like here's all the C code, and it's not just like here's the the
interfaces. There's I mean it's been around for thirty plus years, and you
can see they've talked about just about everything that's in there. And it has
a nice little search bar in the top, and they also have a form
and you can search that there, and there's just a lot of valuable content
there and you can really find a lot of answers to many questions just search
the forums, search the docs. So reach for that first. Secondly,
for like situations where you know, I sort of have this unique problem or
I kind of want to do this somewhat unique feature, how can I do
that with SQL? I there is a quite I mean fairly robust. I
don't know numbers. I haven't dug deep into post chress, but there are
a lot of sequel at extensions and they add all kinds of different functionality.
So whether it's you know, the hot new stuff of storing vectors in your
database so that you can run some form of AI feature inside of your application.
There's a sequel aid extension for that. You know, there is the
equivalent of pg vector for sequel. You can add it in, you can
store vectors, you can query them and retrieve them, and many many others
start there, you know, see like, hey, is there an extension
to do this? And then the third one would be if those things fail.
There really has been a growing number of people interested in working with this,
and the online community in my experience, has been very welcoming and very
positive. And you can just throw something up on Twitter, on stack overflow
and I think you're very likely to get some useful and good responses. Super
cool. Well, we're kind of at the end of our time. Was
there anything else that you wanted to make sure that you covered? Valentino there
was something I'm trying to think about, but he had you at machine learning.
I know, yeah, I know, I got sidetracked in my head.
I have a lot of questions there, but yeah, I guess,
like, have you hit any limitations as far as like I think of like
as an example, you know, Jason Collumns and postgraz Or, doing you
know, database views or things like that where you know, maybe seql light
doesn't have the feature. Have you found many limitations in that regard that where
you had previously, you know, reached for a certain tool that you no
longer have available because of the the database limitation. So the short answer is
no. But let me just add like I really hope people can will just
believe me. I guess when I say that, I really I'm not like
killing right, sequel Light is not paying me any money. I am not
making any money by talking about it. I'm not trying to get you to
you know, buy my Sequelight sasas or anything. So this is I'm not
just trying to like be a hype man. There's there's no value or utility
in it for me. Right, it is interesting that like it I was
in the exact same book, Like, I don't know what it is.
There's just like something you know, a general myth on the innate. It's
like sequel Light is anemic, just doesn't have a lot of functionality. It's
small, it's weak, It's like nope. Jason Columns, Gottam Jason Operators
Gottam generated columns, Gottom views Gottom store generated columns Gottom. Honestly, the
there are really only two features I've been like, you know, it'd be
really cool if Sequelight added them. So one is GIN indexes from Postcress,
Like that's just a really useful, I mean, generic index. It's nice
to have. Sequel Light doesn't have it. So you can have SEQUL like
columns, but you can't in them as easily. And if you wanted to
do something like that, you do have a workaround, like you can make
a stored generated column sort of like extract out the parts that you want the
index as a as a store generated column and then index that store generated column
and query against that. But doesn't have GIN would be nice. And then
of course, right like it would be cool if you could do concurrent rights,
that would be great. You can't, but that would be nice.
Aside from that, I personally have not come across any feature that I have
needed to or wanted to build that sequel I didn't provide, and I was
in fact surprised actually as I dug into it, more like oh okay,
sequel I actually does support like basically the full you know, it supports the
full Sequel standard and a lot of the stuff that is in the standard that
you know Postcress or My Sequel, like, this would be a nice feature.
So I was like, sure, it sounds like a nice feature.
I'll add it. So I'm not just a blind hype man, and I
don't think syquolid is the solution to all things. But really, genuinely,
I think there are a lot of limitations, Like we need to start fighting
back against this myth that it is an anemic tool. It's the most widely
used database in the world. It's the best tested database in the world.
It's one of the it is older than the other two major databases, my
Sequel and post Chris like. It is resilient, it is powerful, it
is well tested, it is flexible, it is easy to learn. There
really aren't reasons to not use it unless you're building the next you know,
Fathom analytics and you really do need one million rites per second, then yep,
there's a reason not to use it. Just one last note, there's
a a famous, you know, learning experiment where if you wanted to like
learn a new language. As an example, there's a build your own SEQL
light database kind of tutorial that lots of people follow because it does, like,
you know, everybody wants like a rebel, right, Like everybody wants
to build a store data read data from files, right, and like seql
light does all of these things that are very just common patterns for languages.
You have to learn to do the core tenants of whatever it is to make
it be a database like from a file system, which is kind of interesting.
So speaking of picks, you know, I might as well pick that
it's like how to build a sql light database. But I've definitely started one
in Rust, which is just for fun and it's it's amazing how simple,
like as you mentioned and sequel light. You know, it really is under
the hood as far as storing things, and this is fun. It made
me think about it. Cool. All right, Well, I'm gonna push
just pics. Valentino, you got some picks. Yeah, I've been loving
a Lama, which I know has been mentioned on the show before and by
others. But it is a way to run local inference on your machine against
many different open source large language models. It's kind of fantastic. It runs
as a server, so you can serve many different models optimized automatically with whatever
GPU you're running, and you configure it to offload some of that to the
CPU, and it's pretty fast and just it has saved so much time,
just like investigating and evaluating other models that are out there, and you can
even build your own models and serve those locally, which is kind of wild.
So I highly recommend checking it out if you have awesome I'm gonna jump
in with my board game pick first, so I'm gonna I went to salt
Con, which is a local board game convention, and I really had a
good time, just hung out with one of my buddies the whole time,
and we ran into some other people we knew there, so anyway, it
was awesome. So shout shout out to salt Con if you're in or around
Utah. They're doing another one in June, and I'm hoping to get a
ticket to that one. I haven't bought one yet, but we're kind of
doing that thing. Also, before I do that, I should probably mention
that I am looking for work, so if you're hiring contract or full time,
I would love to hear it. Chuck at topendevs dot com. All
right, So the game I'm gonna pick is a game I played there.
It's called Apiary and uh so Board Game Geek. I'm just gonna warn you
this one's on the more complicated end. It was definitely fun. There were
some unique things. But Board Game Geek has has a weight of two point
ninety six on it. That's out of five, so you know there are
fours and fives, and usually those ones will run for hours. Apery did
not. I think it took us an hour to play it with four players.
But it's pretty involved. So if you don't like the games where you
have a ton of stuff to keep track of, then this isn't the one
for you. But if you want one that has some mechanics and you know
you have to kind of figure out which strategy you're going to jump on,
it's a terrific game. It was. It was a lot of fun.
It's kind of a blend between like worker placement and I don't know. There
were some pretty unique mechanics. So every time your every time you placed your
worker, if there wasn't room for your worker, then you would bump somebody
else off the board and that b would go back to the hive. But
it would get promoted, and then once you got promoted, if a four
got promoted at Hibernate, and then go back to one right instead of level
four, and so anyway, but the number on the B and the number
of the other b's on a particular spot also determined what you could do there
and so anyway, and the number four is also have unique abilities at each
space. So anyway, lots of stuff, just really really interesting game.
It was almost too much. There were almost too many things going on to
make it just super laid back fun. But anyway, I did really enjoy
the game, and it's on my list is one that I may eventually go
by, so I'm gonna pick it now. If you didn't pick up on
the theme, Apary is like an av areas for birds, Apire is for
bees, and it's space bees, so you're adding on to your spaceship and
stuff like that. So anyway, so yeah, so that's my board game
pick. And uh yeah, I'm I'm actually just exhausted because I've been working
on been working on a bunch of stuff for Super Tuesday because I'm involved in
the state politics stuff here. So I'm just gonna end my picks there and
just remind you if you want to hire me to let me know, Steven,
what are your picks? Yeah, so I am friends with bull Drapper
who built and maintains the Flex jump. If anybody has heard of that,
and if you haven't, let me give you a quick intro there. Flex
is a view component gem for Ruby, not the view component gem, but
a gem for writing views in Ruby. And specifically it does everything in Ruby
right, so it maps elements to method names, and attributes to keyword arguments
and children to blocks. It's a really great gem and specifically, lately the
core team has been working on a really awesome feature that I want to make
sure as many people as possible know about because I actually think it can you
know, I hate to even say these words, but I think it could
possibly revolutionize how we do our Rails applications. And so what is the feature.
The feature is selective rendering on the server side. So conceptually it is
very similar to how turbo frames work. Right, So you have a turbo
frame on your page, it has id turboframe one, and you click a
link inside of that turbo frame the way that would be managed by Rails presently,
It's like that link gets sent up to your application. The controller processes
it, finds the appropriate view, renders that view, sends that view back
down to the client. On the client side, there is javascriup that says,
oh, you know, this is a response from a turbo frame request
that we made. We made that request from turbo frame one. Let's parse
this view and find where turbo frame one sits inside of it, will pull
it out, throw all the rest of it away, and replace the contents
of turbo frame one with the new contents of turbo frame one. So you
on the server, you're rendering everything, so you have some waste, but
you have the benefit that you get to use your existing rails routes. You
get to keep all of your existing controller logic, rendering logic, database access
logic, so you treat it as if it's just a regular page visit.
But you have excess content that is sent in the response, and then on
the client side you have to take all of that content. You have to
parse it and find the subsection to pull it out and replace the frame.
The feature that flex is adding now allows you to do that exact same thing,
but all on the server side. Right, So you send a request
up from turbo frame one and you say, hey, I want to reframe
one. It's just a regular link to a regular normal route. Controller receives
it, goes to render it. It's rendering a flex view and not an
action pack view. And that flex view is past the content that that or
that context that hey we want turbo frame one. And so now all of
your flex views it'll start rendering, and it's just gonna no op all of
those methods. You know, so your HTVL, your body tag, your
H one tag, your all of that. Okay, if those are methods,
we're just going to no opt them until it finds the method that has
id turbo frame one. It's like, oh okay, now it's time to
actually render. And so then flex will render all of that content, package
it up and send back just that give or section frame whatever you know.
Uh, So you get two or I guess three big benefits. So the
one is you get to keep the same simplicity of your rail application getting to
be structured as if all of the interactions you're doing are standards stateless HTTP interactions,
full page refreshes. You don't have to set up a whole sort of
like separate components name space where you render specific things, but you get the
second benefit of sending the minimally small payloads over the wire only the content that
you need, with the third benefit being that you only have to render the
content that you need, so you're rendering times actually speed up as well.
So this is available now in like the alpha release of Flex, it will
be generally available when they least version one ten. And we're doing a lot
of thinking and a lot of work now to really make the integration with hot
wire and turbo as seamless as possible. But I think that this kind of
general shift in thinking and leveraging these kinds of tools is going to be really
useful and valuable for rails applications. So that is my pick. Black selective
rendering awesome, very cool. All right, one last question. If people
want to find you on the internet, where are you? Yeah? So
I have fractled Mind on Twitter and get hub. My blog is Fractledmind dot
GitHub dot io. And if you want to meet up in person or you
just want to get more confident and comfortable with building rails applications with sqlight,
I am going to be at Rails comp in Detroit in May and I am
hosting or leading a workshop there on the second day on building rails applications with
people, and we're going to cover everything that I know and as much as
that as possible. So the goal is that coming out of that everyone will
feel comfortable and confident with using this stack and using these tools. So a
lot of exercises and content. There will be a fun to our workshop and
otherwise happy to meet up and have a soda with anyone in Detroit as well
if you're going to be there for Rails comp awesome. All right, Well
let's go ahead and wrap up. Thanks for coming Steven, This was awesome,
Thank you, all right, till next time, folks, max out
in the in