Hey everybody, and welcome to another episode of Ruby Rogues. This week counter
Panel, we have Nate Hopkins, Hello everybody, Andrew Mason. Hello.
I'm Charles Maxwood from dev chat dot TV. And this week we have a
special guest and that's kier shatdrop here. Do you want to say hi?
Let us know who you are. Hi, my name is Keir. I'm
a production dinin. You're at Shopify where I work on the scalability in the
platform, and I'm based in London. A cake nice. Now, Shopify
doesn't have to deal with any scalability, right, I mean they only run
like half the shopping carts on the web and things like that. Right.
Oh yeah, So I'm curious as we dive into this. You know,
you gave us a couple of articles. One was on the state of background
jobs. The other one was on like capacity planning for web apps. I
kind of want to start with this and dive mostly into when should I start
caring about this? Right? Because if I have a small app, it
matters a lot less for a while, and then eventually I'll get enough users
or enough people using the capacity to actually go all right now, I really
need to start thinking about this, So, Yeah, where do you find
that the cutoff point is for this kind of thing? Definitely, there is
a lot of talk and technologies that it's natural for engineers to be super interested
in it. But the price over engineering things and choosing some solutions that are
maybe too complex at the stage where your project is right now, that price
can be too high, and often the most resourceful thing you can do is
just deployed on Heirocu and let it run, and it will cost a few
hundred dollars for your Hiroco bill. For me, I think the cutpoint is
around the time when you start losing the control of maybe you're hosting costs or
you noticing that whatever scalability promise you have start hurting your customers and you start
losing money, either as a result of your customers being unhappy or as a
result of the thing costing to run a lot more than a company kind of
work to run the business in a reliable way. Yeah, that makes sense.
It's interesting too that you've kind of tied it to those two practical breakpoints,
right. A lot of people they try and tie it to well,
I have a certain number of users, or I have a certain size of
an app or I have you know, a certain amount of server capacity or
you know, stuff like that, and it's it's interesting to me that a
lot of this, you know, you've tied it back to oh, it's
impacting the customers or oh, you know, it's it's impacting my bottom line,
and then it's like, oh, okay, how do I deal with
this? I also think it's interesting that you mentioned that, you know,
it's easy to do if you just hand it off to Roku and let them
handle it. And I know that I haven't heard it as much from Nate,
but I've definitely heard it from Eric over at code Fund that that's kind
of his approach. He doesn't want to deal with DevOps. He just wants
to push it to the cloud and then, you know, let them handle
it, and he's willing to pay for Heroku to do it. Yeah,
that's that's our philosophy right now. But I mean we're also short staffed,
right Yeah, so we've got two well really, we're just one and a
half developers on the project. Other than we've got plenty of contributors that help
us fix bugs and things like that, but there's only two of us that
are full time. You know, looking at code, and Eric's really only
about halftime looking at code, if that right, So we don't have the
time of the bandwidth to really delve deep into into you know, the ops
story. That makes a lot of sense. So I'm curious, Nate,
at what point would you guys consider moving off of Heroku? I mean,
would it be a cost thing or would it be something else? You know,
we're still we've found product market fit and we are trying to scale it
now. We're trying to scale on the sales side. So as soon as
we have enough customers and enough consistent revenue flowing in to allow us to kind
of back off and look at our operations story, that's probably the time.
So I would say we're probably maybe six months away from you know, having
the luxury being able to look at that. Yeah, that makes sense.
So Keer as somebody gets to that point, you know, and I think
this might be a relevant conversation then for Nate. But you know, when
they get to that point and they're thinking, Okay, we're going to scale
this, maybe they move it off of Heroku and onto you know, a
Kubernetes cluster, or they move it on to you know, a virtual private
server, something like digital lotion or something. What things should they be looking
at then to scale their their stuff up. For any hosted services, like
for instance, it's common to use hosted database as a service, I think
it's important to look at whatever limitation that service provides, because any hosted service
would have some kind of those I remember read a blog post where an app
had a very specific requirement for some postgrous extension that they've been using, and
they switched i think three three providers that gave them Postgrass's service, and they've
been unhappy with each and they obviously spent a lot of efforts, and finally
they got to run postgrass on their own because having that very extension and requirement
that was a huge point for them when choosing a provider like that. It's
important to understand any limitations and and from another angle, I think there is
there is so many scalability related problems that you can run into that usually it's
you start looking at the one that's most critical right now. Like I've I've
been part of projects where they've run into scalability issues with the database layer with
my sequel or with progress and as they fixed it and iterated it on it
and their database could accept a lot more load. They came to another bottleneck,
and that bottleneck is different every time, depending on the business, depending
on your patterns of the usage that's coming from your customers. So it's fixing
one thing at the time, one by one, and sometimes that's a never
ending story, especially if the company grows large and there is a team works
just on scalability, which is currently the case for my team of Shopify.
Yeah, that's a terrific point in terms of really, this is not a
job that ever completes, right, It's something that you're always having to stay
on top of it, especially if the company is enjoying any level of success.
One cool thing about code fund is we are even though we're on Heroku,
we're able to leverage some of the postgress at more advanced postgress features like
table partitioning and things like that, which has enabled us to continue to scale
on that platform. We're hosted on one hundred and sixty plus sites right now,
and so we're seeing between two and a half million and three million requests
a day pipe through the server. Now. We are paying a premium for
Heroku, but we're still I think we're under eight hundred a month on our
on our production setup, and we're probably a little over provisioned in anticipation of
spikes and things like that, and so we don't quite have the fine tuned
control that we would like to have. Your point on postgress, as you
want to customize that and install your own plugins and things like that into the
database players, that would be something that would be fantastic because since we are
using table partitioning, I know there's some plugins that just are not broadly available
on the Heroku platform that would be kind of a luxury to use for us
that we've kind of had to work our way around some of those things.
I'm curious about your experience and time with Shopify. How long have you been
with the team and what types of changes have happened since you've been at the
company. I've been a Chopify for almost for years, and I've always been
part of the production engineering department, which deals with the infrastructure and is less
exposed to the product. And just that department grew so much from maybe while
I've been here, from maybe thirty people to now more than one hundred,
and all of those people are working on the infrastructure and reliability, and with
the motto of that, our job is to keep the site up. There's
another aspect of scaling here, going from forty to one hundred people, Like
how has the team scaled? Like what's the dynamic been? Like, Yeah,
it's interesting to follow dynamics in terms of team scaling in every organization,
and I imagine it's a different story. It affected so many things. Like
for instance, at the time when I joined, our Shopify is based in
Canada and most of infrastructure engineers were just one office. Now people who work
on the infrastructure are based in three offices, and there is also a lot
of remote people like me. And then as you grow, you end up
investing into some of the things that you would never invest before and have teams
who work just on one part of development environment for instance, or just on
background jobs infrastructure, something that I wouldn't have imagined three years ago. So
what is the technical portfolio for Shopify around and like how has it changed since
you join? Obviously that's a great question. There's been a lot of new
tools and techniques and stuff that have come out, but you know, just
over the last four years, and so I'm curious with the evolution of tooling
has looked like, Yeah, that's a great point of discussion. So I
think first there is something I wanted to give the context to our listeners.
First is that when Shopify was founded about twelve years ago by Toby Lutke,
Toby was one of the first contributors to Rails and he knew David djh and
they exchanged some emails and around the time when he started company, when he
started Shopify on rails, rails was just a ZIF file that they exchanged over
an email. It wasn't even some specific version published on a GEM server because
I'm not even sure there was if there were any GEM servers at that point.
So from that day when he started on rails, that app still exists.
It was never rewritten. It's a monolith that has been around for more
than a decade. We tend to put a lot of love into it to
make sure that developer experience stays great. Unlike it often happens that a monolith
is just too slow and too hard to work with that developers get so much
friction and decide to go splitting or calling the monolith a legacy. It never
happened for us. I've got to interject and just ask a question on your
monolith in terms of, like I know Shopify is a very large company,
how many developers have their hands in the monolithic code based my rough guess would
be from one hundred to two hundred people, given that R and D in
total is a lot more because there would always be people working on other part
of stack, also mobile developers and so on as you can imagine. So
back to your point about how has the stack changed in terms of tools that
are familiar to listeners of our podcast, it's still pretty much a classical rails
up with all the things that come with it. In terms of the infrastructure,
I think the biggest shift that I have observed of the company was moved
from physical data centers to the cloud to Kubernetes. And that's another who interesting
story because we were able to move to Kubernatus in cloud one shop at the
time, Given that we have millions of them, we wanted to make this
process as continuous and find control as possible, so we just took one shop,
moved it to cloud and progressed and we were able to control that.
It's fascinating to me that you have upwards of two hundred developers working on a
monolithic RAILS code base. Like some conventional wisdom that I've heard in other circles
and certainly bumped into in my career has been that if you're going to scale
your organization, you apply conways and break out into micro services. In the
conventional wisdom seems to be that that's really the only way to do it,
and you, guys are a terrific counterpoint to that. What are some techniques
you've used to facilitate it. I think one of the biggest has been adopting
domain driven development development and splitting that monolith into I would not call them name
spaces, but it's kind of components at least that's how we call them.
There is nothing very secret or special about it. It's basically just a way
to structure your app directory so that each team, each component gets their part.
Therefore, it helps a lot to establish the ownership because, for instance,
as soon as you see an exception in production in some of the exception
tracking service that you use, you see that exception is coming from components Slash
support, Slash app, Slash model, slash something. You immediately know that
a support component and you have all the metadata to find people who can help
with that, even a non call escalation or a Slack channel where you can
chat and point out. And we started leveraging that for some of the to
automate some other things like, for instance, if exception within one app happened
in that component, will send a notification to their Slack channel, not to
some generic Slack channel with tons of exceptions from all over the company. Establishing
those ownership is I would say, the main technique. Okay, so domain
kind of a domain driven design, and then you give a team like full
stack responsibility or at least all the areas of the stack that that particular domain
piece may touch, right, so that could slice all the way through front
end, all the way down into the model layer. Yeah, it's not
as strict as you can imagine, and there would always be cases of reaching
out directly from one active record model to another through components, through different domains,
and that's not great. We try to build tools to discourage people from
doing that and for them to know what are the right patterns. Like for
us, it's mostly entry points that are well that are typed and declared and
documented. So this is kind of shifting gears a little bit. I'm really
curious about the database infrastructure because I know on Shopify, essentially you've sharded the
database or maybe not sharter, but there's multiple instances of the database, right
that are all that backs this. How is that structured? And how do
you manage that from an OPS perspective? Oh yeah, that's also a great
discussion point. So also to give some of the context to the listeners.
For all well known rails companies like Shopify, Gethub, based Camp name a
few that's been founded around ten years ago. At that time, my sequel
was that best known database that everyone knew how to run and operate. People
were the most familiar, and some other like posgress were not maybe as good
or as established at that point. So that's one huge reason why this subset
of companies, including US, are all based on my sequel. And yeah,
at I think it was around twenty fourteen twenty fifteen when we realized we
can no longer fit everything into one dB. We figure out we have to
find a way to scale horizontally, and for a multi tenant SaaS application,
there is a great way to do that. Since your tenants are always isolated,
you don't have to. You don't have any joints between multiple tenants,
so you can put tenants through different charts, through different partitions and manage those
independently, which also reduces the blest radios. If you have hundred charts,
one is down for whatever reason, only one percent of your customers are getting
some negative experience, and you go and fix that as as soon as possible.
But it's not all of the platform. So we invested a lot into
charting. In terms of application logic, it's it's mostly done on rails layer.
We have a rails team at Shopify that that helps to steer that into
the best direction possible, at least from the rails point of view and from
the opps point of view, it's it's just a lot of charts that that
can be located even in different regions, and which also can allow to isolate
some tenants geographically. So let me just recap to see if I've got the
picture in my mind correct. So we've got a rails monolith that's kind of
structured with kind of these domain areas of responsibility. That's how you structure your
teams and the way you've scaled this at least up to this point in the
conversation is you're just dealing with gut like just mountains and mountains of data,
So you've sharded your multi tenancy across different database nodes. For the developer,
it can just look like a typical rails application, correct, And something to
add is that we our goal is to make that all that starting complexity gidden
away from developers who right product features for them. It may feel like there
is just a database with a lot of tables that represent the business model,
but underneath there would be some smart sharp selection that would happen at the beginning
of the request, for instance, that would select the right database. And
I mentioned this just for my sequel for relational database, but we've realized that
it makes no sense to have shared it my sequel, but just one global
redditis because regardless of how well you shared that one global redis or that one
global memcash would still be a single point of failure. And as you can
imagine, we learned that lesson by experiencing those single point of failures. So
our philosophy is that every resource would be sharded, so there would be a
smaller instance of shopify that has its own My sequel that has its own raddits
that has its own memcash that helps with this isolation. So with each web
server essentially or maybe partition of web servers the scale horizontally, all of those
would not necessarily have a local copy of them cash and read us, but
maybe just a shared one that cluster of web servers. One thing I should
note is that stuff like web servers, it's still all shared capacity, and
it's mostly it's only resources that are isolated. So any web server can talk
to any to any partition or any like smaller instance of Shopify, it's mostly
the matter of selecting the right path depending on what's the customer. So now
I'm a little curious in terms of because there's obviously a pretty significant coordination piece
there. You know, when the request initially comes in and then you assign
the correct mem cash server, the correct redit server, and the correct my
squel server. How much of that infrastructure did you guys have to build Shopify
and how much are you leaning on the database providers for those things? Honestly,
I think it's mostly all in house built. And to give a bit
of context about that, it's mainly a component called sortine hat I like the
name that is using sound the sortine hat is using a global lookoff table to
find which which domain, which shop is on which partition. It gets the
partition and then it goes to the location of that partition can be US West,
Central, use East, somewhere else, and then it just hits the
right database located in that region, and the right through all through rails and
mostly through HDP headers with And what's what I find very interesting is that we
were able to build all of that on top of Engine X since Engine X
allows you to write scriptible LUA modules where you can implement any kind of logic
in those local modules. In Engine X, you can query your database to
look up something where that tenant leaves, and then you just proxy that through
Engine X and you manipulate the headers and just make this work. So it's
quite a lot of infrastructure that we had to write. But at the same
time, as I talked to call different companies, it's all custom tailored and
there is no there is rarely a same stack, same use case. So
that's also that would be a bit hard, maybe a bit hard to share
and abstract. So yeah, how much of that infrastructure tooling is open sources
that all secret sauce internal stuff, or have you open sourced some of it
which try to open source quite a few things. There is also a lot
of conference tocks that will link to show notes that give way better over of
the architecture. Then I just explained the routing layer itself. I wouldn't say
it's open sourced, but there is lots of information out there for someone who
who would want to build and use same techniques. So that's probably a good
segue into you know, additional scaling aspects. So you've you've addressed a lot
of the persistence layer pretty much the entire persistence layer horizontal scalability, but you
still have response times to deal with, right, And so one way to
make response times fast is through background jobs. And I know you've got quite
a bit of expertise there. What is the approach and architecture of Shopify's background
job system. Well, and just to pile on here real quick, it
seems like when people start talking about scaling ruby at or rails apps or sondraps
or whatever, this is one of the first things people reach for, right
because any long running task they just you know, shunt it off to background
job and you know, report errors back to the user if they have to,
and it shortens the response time because then it's hey, go do this
job instead of I'm going to grind through the work of doing this job.
Yeah, and before you jump in with an answer too, I mean one
thing to bear in mind is like some of the stuff is just it's baked
into rails with active job. But you don't even have to set up redd
us or anything like that to support it, right, It'll run it on
a background thread out of the box. So what is the path for developer
kind of chucks lead in question? You start on a small project that's maybe
a little hobby thing, and it starts to get some traction and then maybe
it turns into a business. What does the evolution of kind of evolving that
background job handling look like over time? Oh yeah, And to note that
like myself or some of the byprojects, I run background jobs exactly in the
background thread in those uma processes. Yeah, just because it makes no sense
to pay for extra for instance, kick down as on Hierroco for those bad
projects. And exactly as you pointed out, it makes sense to start with
something as brutal as a background thread, and then I'm really happy that Ruby
community has a project like Sidekick and Mike Perham who is behind that project,
who has pushed the community to adopt some beast practices around background jobs, and
also offers nineteen nine percent of what community needs as an open sound project,
and for the remaining of one percent, when you get to that point,
you can buy a pro or an enterprise edition, and I'm pretty sure that
when anyone is at that point, that's actually quite an affordable software to buy.
As a company, and just like most of the community who is using
Sidekick, Shopify is very similar in terms of setup. Because we've been around
for so long time, such a long time. We've started with Rescue if
anyone remembers, that was a pre Sidekick era library to basically achieve the same.
So we still run Rescue, we run reddits. We got to rewrite
most of Rescue internals because we're multi tenant and we want to share some of
the capacity and reuse that between tenants, which we can dive into if if
you say later, I guess the first question from you and from some of
the listeners could be why we're not on Sidekick, And the answer I would
say is mostly the legacy part and also how much we know the stack and
how much we customize it for us at this point. But we're all so
starting some smaller apps at the company, some smaller rails apps. In fact,
in addition to the Monoli, you probably have a couple hundred other smaller
rails services for something very specific or maybe something just employee facing, and all
of that would use the recommended set of libraries that includes Psychic. Yeah,
that makes sense. I'm also working on a software as a service. I'm
sponsoring one of the bigger conferences that serves that niche podcasting in August, and
so I anticipate that things you're going to grow. And yeah, I have
a lot of things that I am pushing into the background jobs right now just
because you know, I want to get the response times down. But one
thing that I'm wondering about, and I'm kind of tempted to go with Heroku,
but part of me, I don't know, I have this mental block
about paying for something that I could probably figure out the scaling on myself or
at least do some you know, a couple of minor things to help with
the performance and scaling that way. So what should I be looking at next.
It seems like you all have kind of gone toward the cloud, and
I'm wondering if that's the right answer, or you know, beyond background jobs,
what's the next step? A step to reduce response time? No more,
it's more a step to just get it to scale, you know,
get that you know, be able to handle more traffic without having the site
slow down. Right, there would always be some kind of bottleneck, which
is depending on if you have a good setup of tools, should be possible
to find. And for us, that bottleneck has changed through the time,
And I would guess there is no single answer because maybe there is something in
a web server, in a controller still spending quite a lot of time which
which slows down the response time. Or maybe it's all database that's a bottleneck,
or maybe it's it's reddis or maybe the rails reaches out to some external
service that is not located too close to it, which increases latency and also
impacts response time. Yeah, that makes sense. I'm curious what criteria you
use to determine what should move into a background job. Obviously you may hit
some latency on a particular request and see something that is kind of low hanging
through to move to a background job. But just because you moved it to
a background job doesn't mean you've actually addressed the root of the problem. You've
just moved it out of the request flow, right. Oh yeah. And
a very common batter that I see in people do with jobs is, for
instance, you want to iterate over all users in your app and do something
about each of them, maybe remind them that they need to add a credit
card or maybe something expired, or you want to send them an engagement email.
When you start, you have just one hundred users, so that job
works off pretty quickly under a minute, maybe depending on what kind of work
that is. You grow to thousands, hundred, thousands, to millions,
and a job to iterate over a million users and to check balance of each
of them, that job starts taking days or weeks. And how do you
solve that? And it's just so easy to introduce that problem. You just
do user dot find each in a job and it works, but until the
point when it stops. So the way how we solved it, and that's
actually all open source. We'll also linking a show note. We've solved that
by making every job interruptible and preserving a cursor so that a job would progress
for a bit and then maybe it would get restarted for some reasonasically, this
allows us to iterate over really long collections and do some work with them and
never lose the work that has been done. Nice. Yeah, that's really
cool. I'm gonna check out the Shopify job iteration. That sounds really really
interesting. One of the things that we've done a code fund is when we're
iterating across of course, we'll do like a find in batches, and then
we will just in queue the smaller work, so when the large job fails,
it's essentially item potent and can be just rerun again without without impacting things
that may have been half processed or halfway chunked through. Yeah, that's the
approach that I take as well. An interesting side effect of that could be
that, again, if this leads to a fin out of a million jobs,
because if you have ten million users and each batch is side of ten
for instance, like the numbers don't really matter, but the point is that
if the fen out of so many the jobs, we need to remember that
something like credits is always limited in memory, and there's been so many times
across every I would say across every organization where I worked, that people would
push reddits into out of memory state, and unfortunately there is no I would
love to have a great solution for that. But every time we want to
do something like you describe, iterate in batches, thank you something, we
have to be mindful about what's behind that. And yeah, I've been that
as well. You start dropping jobs because there's no memory left. Certainly happens
at times when there's when jobs might be failing. Right sidekick for it gives
you some pretty nice failsafe capability where it will reattempt those jobs. But if
you've got a bug and not a lot of memory dedicated to your reddits,
instance, then of course you may start losing work that may be critical to
the business. Yeah I could see that. I haven't run into that myself,
but I could definitely see that happening. This is a great reminder about
all sorts of data databases that exist there, and maybe push push someone to
learn about that, because at the end rad as so reddits is in memory
database which is bound by some ram that you give. It can be gigabyte,
can be four, can be sixteen, and that backlock of jobs would
not be backed by something that's that can be written on storage that's bigger than
RAM like like which would be DISC if it's, for instance, my sequel
progress. So something that we would really like to find is a store that
could persist those things on disc with a performance not too far and features not
too far from radits. Reddis does have the capability to push right to disk,
right to flush itself out to disc. Yeah, So that only helps
to have a snapshot in case the computer where REDDITS is running rebootst but it
still doesn't allow you to store more than you have than the RAM that you
have. Yeah. I mean, that's probably a great argument to move to
cloud, right because on Heroku, it's just one button click when you see
the memory filling up to scale out or scale up your REDD storage capacity.
Yeah. And a lot of cloud databases or cloud instances they have methods were
compensating for that, and so they will just migrate you to a bigger instance
or you know, basically allocated to allocate it new memory without you even having
to click it. As far as that workflow is validated and people are certain
that it will work. That's a great feature of cloud providers. One of
the thoughts that i've I've had architectural which would be kind of neat on the
background processing would be some jobs obviously are a bit more ephemeral and less critical,
and they could be handled in a little bit more localized fashion, So
it'd be neat to build a routing layer that was intelligent where you maybe had
three stages of reddus or just background job storage. Right. One could be
this is very ephemeral and not very important, so we'll just let it be
handled in process on a separate thread, so we'll route that job over there.
Or it may be that this web server the job is still kind of
ephemeral, but a little bit more important, So we could have a dedicated
redd instance sitting on the web server that has just a small set of dedicated
memory for that, and you could push those jobs there to handle some of
that back pressure, and then for the really important stuff, you could hef
to those off to like your appliance tre of reddis storage that gives you the
full capacity across the entire application. Oh yeah, we haven't done something like
this for jobs though, I think it could help a lot. But in
general, like in terms of building systems, I think This is a common
case of defining priority for different workloads, which also allows you to shed some
of the load. So, for instance, you would have it doesn't have
to be jobs. It could be something as basic as web requests. And
there are requests that go to something that's very important to the business, maybe
checkouts, which has the highest priority. Then you have something medium priority that
may be browsing just the admin, and then you have something low priority like
checking out robots X or checking out site map or hitting an API. And
by declaring priorities to those requests when you're at the load, you can shed
some of those that you don't need. And this idea comes mostly from the
largest companies in the industry, Like Google has lots of papers and books how
they do it, and as you can imagine, every request to Google service
would have some kind of priority and they actually shared those like I'm pretty sure
that mail is higher priority than watching videos on YouTube. It's really interesting and
one of the neat things about sidekick is it provides like in terms of if
you couch that in terms of background jobs, sidekick provides some of that facility
just out of the box, even for a simple deploy right because will you
can you can prioritize. You can say this is in the critical queue,
this is in the default queue, is in the low priority queue, and
Sidekick will drain the higher priority queues first. Now you could start there and
then and then eventually expand out and say, well, I'm going to give
a set of dedicated worker virtual machines or dinas or whatever to process a particular
queue. And I may even give us up dedicated reddis instance or tier for
that particular cueue. But you can start with just a simple Reddus instance and
the default Sidekick configuration. Say just for anyone listening, because when we're talking
about like scaling large systems, right like Shopify. But if you're starting a
rails app, for me, the go to is pretty much I always reach
for Rettus, Postgress and Sidekick, along with everything else that comes out of
the box with Rails. That's pretty much what I always go for when I
start a new project. Yeah, I mean, I use I've used Rescue
in the past for a lot of projects, and then yeah, I've moved
into Sidekick for my newer stuff. But yeah, when is it too much
to background something? Right? So I wrote a gym that allows me to
essentially background every or any method that hangs off of an active record model,
which is really convenient, but what I've found is it makes it almost too
convenient, where if something seems to be slowing down a request, you can
just do it dot defer to the method name and it would stick it into
the background, which is great, but it got abused and we ended up
with far too much running in the background, hitting those problems you're talking about,
like exhausting memory and stuff. So how do you how do you determine
what should be backgrounded? That's a good question, and frankly, as someone
who's spent quite a lot of time on that part of stack, I'm not
sure there is a single answer, and I think it's somewhat related to how
For instance, if it's active record and sequel quarius, how heavy are those
quaris? If your request i'me out is thirty seconds and just one sequel query,
that's for some reason heavy some kind of aggregation takes ten and you need
maybe to run a few of those. There is no way to fit that
into a bub request, And of course it might not make a lot of
sense to do the premature optimization, and it can be fine to just start
with everything in a web request in a controller, and then you find out
that's the thing where your apps spends most of the time in a web request,
and you just move that to a job because for simple apps, that's
maybe it will be part it will never be a job, and it will
scale fine for the next few years. Yeah, I wonder if a good
approach would be to first This probably very much depends on if you've got paying
customers that are being impacted, right, So, if paying customers are being
impacted and you've got just some inefficiency and a query or some aspect of a
web request, maybe you background that, but you also set you put it
in some type of planning process where you revisit that job and try to actually
optimize the real root of the problem. Yeah. I tend to use the
background jobs when I have a performance issue in the request pipeline, like we've
talked about before, and then if there's problem with running it in a background
job, you know it's timing out or you know something's breaking or something like
that, you know, then I revisit it from there. I don't know
if there's a silver bullet. I think a lot of times it's context specific
and you just have to Okay, I'm moving this out of the request pipeline.
Okay, now it's having a problem here, So now I've got to
address the issue there. And yeah, you know, eventually it kind of
bubbles itself up to the top of your tech det queue and you address it.
So one thing before we wrap up, do you have like some favorite
tips or tricks or approaches that you do it shopify or have done at other
employers that make this easier, or you know something that you just feel like
is something that you did that you're proud of. Yes, For someone who
is curious about performance and fixing those kind of bottlenecks, my best advice would
be to study all the set and variety of tools that you can use.
These tools can be as high level and web based and simple as muralk and
some of the similar services that you can connect to your app and see insights.
Two more system level tools like for instance, as trays. The amount
of times where as trays saved me or and some of my colleagues at the
middle of the of the service disruption just it's so hard to count those And
my advice is not necessarily about as trays, but knowing the wide variety of
tools that you can use. Some of those tools are very Linux specific and
system level. Some of them are Ruby level, like arbispy, a great
tool by Julie Evans, or arbitrays, and then there are some services that
offer that those kinds of things. So if you know that range of tools
and you know which one is the best for something that you're looking for,
you pick it up and fix the thing. Anyway, You've got to wrap
up soon. I've got a couple, just a couple of questions to put
you on the spot here. One is, do you know what the request
volume that chaff of hy does per second? The public number that I can
say is about eighty thousand requests per minute. And what about background jobs?
How how many background jobs you are being processed per minute? That's a great
question, and to be honest, I don't remember those numbers just out of
my head. Yeah, yeah, I have probably suff It's a lot,
right, Yeah, it's a lot, and it can be very spiky.
And there is a huge difference from steady state and spiky state. Because shopify
is also hosting some of the words largest sales, sometimes for celebrities, sometimes
it's worldwide cups and some special sales that where millions of people try to crash
Superfest stores. Yeah, I can imagine code fund is tiny in comparison.
Since January we've done over three hundred million. Wow, that still feels like
a lot to me. Yeah. We keep changing what's in the background,
what's not in the background, so that we've had that number kind of artificially
inflated at times. But still, yeah, that's a lot of background work.
Yeah, makes sense. All right, Well, I'm going to push
us to picks, Nate, do you want to start us off with the
picks? Sure? So I guess one pick for me today is open source.
How fantastic open sources are good. A thing on the side that I'm
doing for my brother in law and it's basically a CRM. So I went
kind of diving around for open source tools that I might be able to use
to set up for him, and I found fat free CRM, which is
a rails based CRM. It's a bit antiquated on the uh you know,
the way it looks in terms of the UI and UX, but it's pretty
fantastic that data models solid and it meets all of his needs. Which is
terrific. The other pick I've got is cats. So we've got a Maine
coon in a Russian blue and they just provide so much joy for my girls
and for the family in general. So highly recommend getting a pet, and
especially a cat. Nice. I'm gonna step in here with a couple of
picks. The first one that I have is a challenge that I've been doing.
This is a challenge that has been less fun with a broken arm,
but it you know, I started it because I just I really want to
prove to myself that I can do this, and yeah, doing it with
a broken arm, it just I wasn't gonna wait to heal because it's several
weeks to heal a broken arm. Anyway, the challenge is called seventy five
Hard. It comes off of the mf CEO Project podcast with Andy Frizella,
and I've picked that on the show before his podcast, but anyway, it's
basically a challenge that he made up. But it essentially is a challenge to
prove that you can, you know, do what you've got to do for
seventy five days. So there are five rules and if you violate any of
the rules, then you have to start over the seventy five days. And
the first rule is you have to work out twice a day for at least
forty five minutes each time, and one of the workouts has to be outside.
So if it's raining, if it's cold, if it's hot, if
there's a hurricane, you know whatever, you're going to work out outside.
And basically the he says that that's just a you push you through the you
know what. Sometimes you have to do stuff when the conditions aren't ideal.
The other rule, you have to drink a gallon of water every day.
You have to read ten pages of a book every day. You have to
choose a diet and stick to it. No cheating every day for seventy five
days, So a lot of diets. You know, people are like,
well, I take a cheap day every week, no cheat days, no
cheat days on seventy five hard. And then the last one is you have
to post a status photo to social media. And so yeah, I've restarted
twice so far. The first time I forgot to read the ten pages,
which was dumb. It was the one thing I kind of took for granted
that I do and I didn't do it. The other one, I got
a salad from Coasta Vida and I didn't realize that I hadn't told them to
take the rice out of it. And I've been doing a Kido diet,
so yeah, so I started over. I felt really dumb about that.
I was like, I know they put rice in it. I don't know
why I didn't ask them to take it out. So yeah, So it's
just kind of learning to adapt to some of this stuff. But I'm definitely
enjoying the pros. And incidentally, just to throw it out there, so
I've I've been doing the the challenge for about a week and a half and
you know, and I'm currently on day two. Just to throw that in
the right because I had to restart. The flip side is is that I've
lost ten pounds and know we can that's a serious program, like you're gonna
be committed. Yeah, but he says it's a mental toughness challenge. Right,
You're going to go and some days you're just gonna have to push through
do some stuff but you really don't feel like doing. Yeah, like the
run to that I have scheduled today, it'll probably beat both of my forty
five minute workouts together. Because it's it's one of my longer training runs for
the marathon I'm gonna run in October. And yeah, I'm really feeling it
today, especially with my arm and everything else. I do not want to
go out there and do it, but you know, I've got to suck
it up and go do it, so anyway, but yeah, you know,
I've got to go do two workouts tomorrow, and tomorrow's a holiday,
so yeah. Anyway, so that that's my pick. If you want to
go follow me on Instagram, I think my handle is Charles max Wood.
Then I've been posting my uh my social media posts there. I tend to
try and post them to Twitter and Facebook as well, but I'm not always
great about that. I'm pretty consistent on Instagram. So anyway, here,
do you have some picks for us? To be honest, I'm not I
don't know the like the format very well. If you can, just are
there one or two things that you think everybody in the world should know about
that way, right this one? I think it would be interesting for the
main audience like Ruby developers. A couple of weeks ago, I followed a
hecking guide from MRI committers that shows you how to build Ruby, how to
change some simple source and see how to rebuild it again and see how it
works. Which also allows you to try all the new features that are coming
with Ruby two point seven because you build it from the master branch, so
you can go and try stuff like batter and matching. It's something that you're
excited about. And the reason why it can be interesting for any roupe developer
to try is because you get to see all the magic behind it, just
all the ce code, and it's becomes no longer just a thing that some
room committers that I have no idea about build, and it becomes something that
you can understand a little bit better maybe. And I think that a haicking
guide was also made to reduce the bearrier to start doing that open source.
So I think this point falls back to the pick that need brought up about
open source being awesome. We'll link that to very cool, yeaph cool.
One more question. If people want to find you online see what you're working
on these days, how do they find you? Yeah, it's a Katrov
on Twitter or Kres on GitHub. Awesome. All right, well, thank
you for coming. This is really interesting. I want to ask like a
dozen more questions, but we just don't have times, so maybe we'll have
you come back. Thanks for inviting. We'll be happy to come back,
all right. Well, let's go ahead and wrap this one up, folks,
and we'll come back next week with another episode. Thanks lot, Bye bye,