Speaker 1: Hey, folks, welcome back to another episode of the Ruby
Speaker 1: Rogues podcast. This week, on our panel we have Ayush
Speaker 1: Nowatya Hello. Hello, I'm Charles max Wood from top End Devs.
Speaker 1: This week we have a special guest. We have David Henner. David,
Speaker 1: you gave a talk at Rails World in September talking
Speaker 1: about performance, and so we reached out because we thought,
Speaker 1: you know, hey, it's it's interesting to talk about. You're
Speaker 1: on the Zendesk Ruby scaling team, so it sounds like
Speaker 1: you're working on I guess a non trivial application. I
Speaker 1: don't know if it's just the traffic, but it seems
Speaker 1: like it's it's an involved app. So yeah, do you
Speaker 1: want to just give I guess any more introduction that
Speaker 1: you want us to have and then we can dive
Speaker 1: in and talk about how to do performance with Ruby.
Speaker 2: Yeah.
Speaker 3: I gave that talk, and the ironic part of that,
Speaker 3: I gave that same well, I gave almost the same
Speaker 3: talk within Zendesk, and it was it was originally an
Speaker 3: hour and a half and then at Zendesk I had
Speaker 3: to break it down to forty five minutes. And then
Speaker 3: I thought I was dumb because I thought it was
Speaker 3: a forty five minute talk for Raals World. I found
Speaker 3: out it was a thirty minute talk. It's all so
Speaker 3: I've got a lot more content, but yeah, thirty minutes
Speaker 3: and that's normal. Probably, Wow, thirty minutes, it's not much time.
Speaker 1: It's so funny to me because I talked to people
Speaker 1: who haven't spoken at a conference or haven't you know,
Speaker 1: given a talk in other settings. And you know, so
Speaker 1: for example, at church, a lot of times, you know,
Speaker 1: they'll have us give like a fifteen minute talk on
Speaker 1: a topic.
Speaker 2: But it's like, man, fifteen.
Speaker 1: Minute talks are hard, and they're like, what, you only
Speaker 1: have to come up with this much content, And I'm
Speaker 1: like I always come up with an hour's worth of
Speaker 1: content and have to cut it down.
Speaker 3: Yeah.
Speaker 2: Right, there's always more I could put in it.
Speaker 1: And so it's knowing what to take out more than
Speaker 1: knowing what to put in and then turning that into
Speaker 1: a I guess, a cohesive narrative.
Speaker 2: And yeah, anyway, having the flow is the hardest part. Yeah.
Speaker 3: Today it's like, oh my god, I need to have
Speaker 3: this and the original flow is so perfect that I
Speaker 3: have actually tweaked down at my company. Then to asking
Speaker 3: was like, yeah, I got this, And it wasn't until
Speaker 3: two months before the conference talk when I had to
Speaker 3: submit my papers. Oh thirty minutes. Oh oh no, let's
Speaker 3: get that done. I think it turned out fairly well
Speaker 3: though at the end of the day. I was the
Speaker 3: last talk on the last day, right before Aaron Patterson talked, though,
Speaker 3: and I will say that is the worst time slot
Speaker 3: to have, like you're worried about your talk the whole time,
Speaker 3: and first talk that I actually paid attention to I
Speaker 3: was still kind of little amped up, and that was
Speaker 3: Aaron's say goodbye speech. Basically, it's like right, dang.
Speaker 1: Yeah, so yeah, I've been there too. I want to
Speaker 1: just dive in. So you know what, what's kind of
Speaker 1: the low hanging fruit in the rails performance world.
Speaker 3: I think the big thing that people like, at least
Speaker 3: I don't hear people talking about. And it's like everybody
Speaker 3: focuses on. First off, a lot of people focus on
Speaker 3: really hard things, and it's like, hey, let's change it
Speaker 3: to a micro service. It's like, oh, yeah, that might
Speaker 3: make it faster, but probably is just going to hold
Speaker 3: a bunch of crust. But the big thing people don't
Speaker 3: advertise in the community is just observability and like trying
Speaker 3: to actually like find the things you need through. Let's
Speaker 3: say I use data Dog as endsk, but everybody can
Speaker 3: also complains about the cost of the observability and it's like,
Speaker 3: right now, you can actually make your costs lower by
Speaker 3: turning on software switches. You don't the observabilit one hundred
Speaker 3: percent of time, for example, and you can actually have
Speaker 3: like tricks around, like the observability layer just literally going
Speaker 3: out and like Okay, if we monitor something and something
Speaker 3: goes awry, turn on the observability layer. But don't turn
Speaker 3: it out all the time, because if you do, you're
Speaker 3: going to be paying half of your infrastructure talks or
Speaker 3: or infrastructure costs to Data Dog. So I started to
Speaker 3: talk with that and that was I hope a lot
Speaker 3: of people got valuable as an dollar figures stuff out
Speaker 3: of that.
Speaker 1: But then like, hang on, because I have a question
Speaker 1: about that. Yep, Because you know, I've used Century, I've
Speaker 1: used roll Bar, I've used a new relic. I'm trying
Speaker 1: to think of all the different ones that I've I've
Speaker 1: used over the years. I mean most of them they
Speaker 1: sponsor the show, and so I feel like I have to.
Speaker 1: I have to feel good about it being a good
Speaker 1: product before I go Okay, you can sponsor the show
Speaker 1: and I can go out and say you should try
Speaker 1: this one. And so I've tried a whole bunch of them,
Speaker 1: but most of them you just install the gem and
Speaker 1: then it just kind of, you know, behind the scenes,
Speaker 1: does the work. And so my question there is how
Speaker 1: do you put a switch around that? Do you just
Speaker 1: put it into your initializer? And then so, lawyer, how
Speaker 1: do you handle it?
Speaker 3: In the talk I gave the actual code, but there
Speaker 3: is a general. There is a general like, hey, it's
Speaker 3: on or off, and we always have that general thing on.
Speaker 3: So there's like the general. But then let's say in
Speaker 3: the traces that you get a stack trace and the
Speaker 3: default stat trace. I believe it is just one bar,
Speaker 3: nothing extraordinary, but you can actually turn a whole bunch
Speaker 3: of traces on within that trace.
Speaker 2: Oh okay, break.
Speaker 3: Down to see everything that's going on, and then you
Speaker 3: can actually identify, oh, this part of the code, here's slow,
Speaker 3: so bye by turning that on and off, like all
Speaker 3: of it on or all of it off. If you
Speaker 3: try it all on, all all of it off. I
Speaker 3: don't know what our costs that data dog would be,
Speaker 3: but I'd be willing that data dog would even say, hey, stop, uh,
Speaker 3: you're overloading us, right, But no, that's the big thing
Speaker 3: is like turning not the base layer on, it's all
Speaker 3: the other little things they're optional. And then what those
Speaker 3: optional traces are where the valuable data is though, Like
Speaker 3: it's the stuff that you can figure within your own
Speaker 3: application that gives you way more insight. It's like, hey,
Speaker 3: I want to put a trace around this certain method
Speaker 3: because I think it's slow. And then sometimes you do
Speaker 3: that and you're like, oh, that's fast. Where's the slum part?
Speaker 3: But if you put if you put that around enough thing,
Speaker 3: enough things, and we basically our main application, at least
Speaker 3: the ticketing inside of the application, we've we got the
Speaker 3: observability layer, like just it's covering almost everything. So if
Speaker 3: a certain customer starts to have slow response times, we
Speaker 3: just turn it on. Really, oh, we can focus on
Speaker 3: this part of the code and make the customer faster.
Speaker 2: I got you. That makes sense.
Speaker 3: But let's see what what am I doing now? Right now?
Speaker 3: I'm doing something that's not in my talk. It wasn't
Speaker 3: my talk, but it's also interesting. I think if you're
Speaker 3: a big company or you're a small company, you might go, well,
Speaker 3: why is that hard, And that would be we're changing
Speaker 3: the default scope for our main ticketing object. And the
Speaker 3: reason why we're doing that we're adding new columns to
Speaker 3: and it just seems like, okay, we just change the queries.
Speaker 3: Well you change every single query. You also have to
Speaker 3: change every index from out from under the hood, and
Speaker 3: you have to never have downtime and adding aex alone
Speaker 3: on our ticketing tables, they're just so huge. You can't
Speaker 3: just say, hey, DBAs add this. They'll say no to you.
Speaker 1: So you can lock that table for I don't know
Speaker 1: a few days exactly.
Speaker 3: You can't do. So it is a juggling act that
Speaker 3: luckily I don't have to deal with the DBA side
Speaker 3: of it. I don't know what the problems are. And essentially,
Speaker 3: when we move from a raw two to a RORA three,
Speaker 3: it did two things. First off, it made some problems bigger,
Speaker 3: so it made this a direction we needed to go.
Speaker 3: But also have you heard of invisible indices? No, So
Speaker 3: invisible indices is kind of what sounds like. Essentially, you
Speaker 3: have an index and a lot of times you're scared,
Speaker 3: too scared to delete any index because if you delete
Speaker 3: an index, it's gone and maybe you actually affected your application.
Speaker 3: So invisible basically says make it invisible. It's still there.
Speaker 3: The sequel is still populating that index to make it
Speaker 3: actually work, but you can see are there performance problems
Speaker 3: with it? So we have not deleted an index for
Speaker 3: fifteen years of our application, and now we're actually able
Speaker 3: to go in and say, hey, if we actually delete
Speaker 3: these induses, we can add new ones because we the
Speaker 3: memory profile won't blow up. So that is the the
Speaker 3: saving grace of allowing us to actually do this whole project.
Speaker 3: Without that, we'd be too scared because well, having things
Speaker 3: break is scary, a lout of money.
Speaker 2: So I'm curious.
Speaker 1: I mean, I've deleted indices off of databases that I've
Speaker 1: worked on, but i haven't done it on like very
Speaker 1: large tables. I've added into ses on very large tables
Speaker 1: and watched the thing go ah, But deleting them doesn't
Speaker 1: seem to have that same effect. So is there a
Speaker 1: downside to deleting them other than some queries that use
Speaker 1: those indicies might slow down.
Speaker 3: Right, and you don't know what, like like when your
Speaker 3: application is I think we have over I'm guessing to
Speaker 3: have over a thousand repos for example.
Speaker 2: Uh huh.
Speaker 3: You can't figure out whether you have beat down every
Speaker 3: location where every query that actually hits this. There's actually
Speaker 3: I had to go through every query in our main
Speaker 3: API and we separate our queries by the API queries,
Speaker 3: and we separate it by non API queries. Let's just
Speaker 3: that way. And the API queries is a manageable list
Speaker 3: of things that it's like a couple hundred queries to
Speaker 3: the ticketing table that actually happens. You can wrap your
Speaker 3: head around that. But the non to API queries, I
Speaker 3: think it was over forty thousand different unique queries. Oh wow,
Speaker 3: go through each query and say would this index work
Speaker 3: with this index work with no? No, no, no no.
Speaker 1: So so it is complex.
Speaker 2: So what's your approach to that then?
Speaker 3: Uh? So that luckily the things we care more about
Speaker 3: are the API queries, so we absolutely make sure that
Speaker 3: those actually are working, and the non API queries. We're
Speaker 3: actually moving a lot of that stuff to elastic search anyways.
Speaker 3: So our long term fix is actually all those things
Speaker 3: are never actually going to need a query. So, and
Speaker 3: I don't know how the magic of that works, Like,
Speaker 3: oh wow, you don't even need a query to get
Speaker 3: the data. That's awesome. Okay, I'm guessing somebody's looking at
Speaker 3: en logs or something like that and they're throwing that
Speaker 3: information rate from the bin logs into seql. But I
Speaker 3: have That's that's magic to me.
Speaker 2: So I used, do you have anything you want to
Speaker 2: jump in with?
Speaker 4: Uh, haven't seen your talk. Probably probably some stupid questions. Really, Uh,
Speaker 4: what kind of like where do you normally find performance
Speaker 4: bottlenecks in like apps? You see? Because I know you
Speaker 4: primarily are you're working for zenders, you've probably worked on
Speaker 4: that product for quite a while. But I'm a freelancer,
Speaker 4: So I look at I look at a lot of
Speaker 4: code bases, and what I see and it's all it's
Speaker 4: all rails. And what I see most of the time
Speaker 4: is people just haven't thought through the database design properly
Speaker 4: and they're doing all these funky joins or even worse,
Speaker 4: getting stuff out of the database and doing things in
Speaker 4: memory that you could quite easily do in the database.
Speaker 4: And it's like it's quite common mistakes like that is
Speaker 4: where I usually find performance bottlenecks. What's your experience in
Speaker 4: that regard.
Speaker 3: So the worst areas are where they add funky joints
Speaker 3: because sometimes you can't solve the is very easily. The
Speaker 3: talk was actually more focused on the coding aspect, like
Speaker 3: just simple things that people do that they should know
Speaker 3: about that they just don't do. So, for example, just
Speaker 3: memorization and a little different memorization techniques. And one of
Speaker 3: the cool things I brought up a zendes that we
Speaker 3: do is we have what I it's kind of like
Speaker 3: a singleton, but it's not. It's so our account object.
Speaker 3: If you get an account, it doesn't matter how you
Speaker 3: get that account. In the request cycle, you only need
Speaker 3: one account object. So if you call it tickets dot account,
Speaker 3: or if you call user dot account, or if you
Speaker 3: call just get me the account through this method called
Speaker 3: fetch account that we have, any way you get the account,
Speaker 3: it is the same exact object. So one of the
Speaker 3: cool things you can do based on that. So if
Speaker 3: you get the same object, if you memoize your whatever
Speaker 3: you're trying to memorize based on that one object, you
Speaker 3: can instead of actually have a request where hey listen,
Speaker 3: I'm going to memorize it on this object. But then
Speaker 3: someplace else in the application, your memoir. You want to
Speaker 3: use that, but you can't. You can't get to that
Speaker 3: original object. We can actually memorize it on this god object,
Speaker 3: and wherever we call that method again, it is memorized.
Speaker 3: So that was I thought, a really cool way of
Speaker 3: actually going about like memorizing something is having that god object,
Speaker 3: and then you don't have to worry about Okay, and
Speaker 3: the presenter we're calling it this way. In the controller
Speaker 3: we're calling it this way, and the models we're calling
Speaker 3: it this way. But we cannot tie those spaghetti pieces
Speaker 3: of code together. But because we have that god object,
Speaker 3: we can memorize it in one place and always access
Speaker 3: the same data throughout and then never have to worry
Speaker 3: about That was a cool thing that actually, well that
Speaker 3: was in the talk.
Speaker 2: But really, what are you use it for that or
Speaker 2: using current attributes or something else.
Speaker 3: Far as making it a god object. I wish I
Speaker 3: knew that was well before I came into But I
Speaker 3: think we're basically we created a method called fetch account
Speaker 3: and one we call fetch account outside there, and then
Speaker 3: in probably active record we override the account method an
Speaker 3: active record, not even every one of our models has
Speaker 3: an account method, so I'm guessing we override that and
Speaker 3: just memoize the account in the instant variable and it
Speaker 3: always goes back to that incident. That's just a guess.
Speaker 3: I've not actually looked at that code because I haven't
Speaker 3: needed to. And I was actually asked that at Rails World.
Speaker 3: I should actually really get into it, but the person
Speaker 3: who knows that best just left send us, so I'll
Speaker 3: have to dive into code instead of ask questions.
Speaker 4: Yeah, I thought current doctributes are probably a bit new.
Speaker 4: Came and five or something that we.
Speaker 3: Just upgraded to. Did we hit Rails eight yet seven?
Speaker 3: Either way? Like, it's so good to be closer to
Speaker 3: the edge. We're definitely on Rails seven for application. We're
Speaker 3: on Rails four for like six years or something like that.
Speaker 3: Who's so good to be closer to the edge.
Speaker 1: Yeah, I've experienced that too. For a long time, I
Speaker 1: was working on like Rails four five apps, and yeah,
Speaker 1: the last year or so, I've been able to work
Speaker 1: on Rail seven and Rails eight apps now. And what's
Speaker 1: fun is you get to go back now and figure
Speaker 1: out all the stuff that's been added in since. So,
Speaker 1: you know, I got into like stimulus and Turbo, and
Speaker 1: then figured out that oh look, Turbo actually will work
Speaker 1: through your what's it action cable channels and so you
Speaker 1: can you can push updates when you have sort of
Speaker 1: out of band anyway, just all this stuff that it's like, okay,
Speaker 1: I don't now need a full cycle to get this
Speaker 1: these features to work.
Speaker 3: So, being at a bigger company and we started on
Speaker 3: Rails one or something like that, upgrading through there's obviously
Speaker 3: monkey patches like throughout the years because well rails didn't
Speaker 3: have certain things and now they do, and now taking
Speaker 3: those monkey patches away is hard, which also means that
Speaker 3: really adding the newest features of Rails is hard, in
Speaker 3: fact almost impossible. But what we can do is like,
Speaker 3: for example, is like, yeah, this week we upgraded to
Speaker 3: Rails three to three, and all of a sudden, the
Speaker 3: performance got a lot better. But also, like let's say.
Speaker 2: In Ruby three yeah, or Ruby three three.
Speaker 3: Sorry, Rails three three yeah.
Speaker 2: I do remember upgrading to Rails three three back in
Speaker 2: the day, but.
Speaker 3: I remember upgrading the Rails three from two. That was hard.
Speaker 2: But that was brutal. That was brutal.
Speaker 4: That was the mob.
Speaker 2: Yeah, it was yeah, it is the herb Any long before.
Speaker 4: I don't have those, cause.
Speaker 3: Oh I haven't.
Speaker 1: Who'da did a ton of work on that? Yeah, it
Speaker 1: was it was brutal.
Speaker 3: But uh, where was I going with this? Oh? I
Speaker 3: forgot my train of thought.
Speaker 2: Upgraded to Ruby three.
Speaker 3: Upgraded Ruby three. I don't think I'm going to remember
Speaker 3: what I was thinking about. Oh, and like, for example,
Speaker 3: with Ruby three, Uh, we had in our application. We
Speaker 3: were very scared because we allowed, we had and still
Speaker 3: do allow customers to add their own rej axes. Well,
Speaker 3: if you had your own rejects, you can really slow
Speaker 3: down and stop you're you can just kill an application
Speaker 3: if you allowed the end users. So we had some
Speaker 3: weird way of actually making it so they couldn't just
Speaker 3: kill our system. But it spun up a process, and
Speaker 3: that's spinning up a process took about a half second second.
Speaker 3: Let's just say to actually spin up so it would run,
Speaker 3: and it make sure that rejects would never actually just
Speaker 3: be infinitely running. It would time it out at a
Speaker 3: certain point. But doing that was well added a half
Speaker 3: second every request. For example, Ruby three allows you to
Speaker 3: actually by default have a timeout in your rejexes. So
Speaker 3: now were in that extra process up. So now if
Speaker 3: you're all most cases you're just running in rejects, that
Speaker 3: takes a millisecond or two. Those other cases that actually
Speaker 3: do time out, those do time out just by default.
Speaker 3: With Ruby, we didn't have to actually do this weird
Speaker 3: fancy trick to actually make it happen. So that was
Speaker 3: actually a very very big upgrade for us because we've
Speaker 3: had those rejects, allowed people to put rejects's in forever,
Speaker 3: and we've had to protect against it forever, and that's
Speaker 3: just just bad. I think I would have if I
Speaker 3: was around way back then, it probably was like, no,
Speaker 3: don't let them put their own rejectses and give them
Speaker 3: an array of things that they can do. Don't give
Speaker 3: them an infinite array, right yeah, but but yeah, of
Speaker 3: our upgrades, it is just like looking at the code
Speaker 3: that's old, and people don't when it is an application
Speaker 3: is as big as it is. People cannot wrap their
Speaker 3: head around the whole ticket, save request of the cycles.
Speaker 3: Because we have everything gets into the ticket, there's triggers,
Speaker 3: there's not just every layer, and essentially people can't wrap
Speaker 3: their head around that. So once you put traces around this,
Speaker 3: you find blatant like obvious mistakes and you just tackle
Speaker 3: them one at a time, and most of the time
Speaker 3: it's coding mistakes. It's not and it's not even a mistake.
Speaker 3: It was I got to get this feature out fast.
Speaker 3: People don't think about performance when you have to get
Speaker 3: it out within a week. Right. But it's been good, actually,
Speaker 3: like the amount that we've saved and the ticket cycle
Speaker 3: and just not even ticket now we're actually since we
Speaker 3: did such a good job, we've been given more and
Speaker 3: more freedom to actually work in random areas of the code,
Speaker 3: and just the amount of improvements that we've made. You
Speaker 3: don't think of little coding things like, for example, scopes
Speaker 3: versus conditional associations, like a scope. Every time you call
Speaker 3: a scope, Rails makes the query. If you have a
Speaker 3: conditional association, it makes a query and Rails by default
Speaker 3: has that query cashed up. Just making that switch will
Speaker 3: reduce the number of queries, and sometimes queries aren't very free,
Speaker 3: so all of a sudden, you've just saved time. There
Speaker 3: so many N plus one queries where that so?
Speaker 2: So were you advocating for scopes or for.
Speaker 3: Both? Most of the time conditional queries when when when
Speaker 3: you're trying to like tweak the last bit of performance.
Speaker 3: I would be advocating for conditional association. Okay, scopes are
Speaker 3: not evil, it's just sometimes people don't realize. You know,
Speaker 3: I'm using this in the scope within a loop for example.
Speaker 2: Uh huh, And.
Speaker 3: All of a sudden, I'm getting like like just query
Speaker 3: after query after query, where if you don't put it
Speaker 3: in a loop, and you if you use a conditional association,
Speaker 3: that loop makes one query and you're you're.
Speaker 1: Done right, So it automatically eliminates the N plus one query.
Speaker 3: Is what you're saying, Yeah, well, yes, yes, but N
Speaker 3: plus one query. I always think of n plus one
Speaker 3: queriers is the things that you can add just includes
Speaker 3: in and it just goes away. Yeah generally, yeah, but
Speaker 3: this is an N plus one query that generates itself
Speaker 3: a different way.
Speaker 4: Yeah, if it was to do like some if I've
Speaker 4: just built like a new feature and I just want
Speaker 4: to reassure myself that I haven't made any stupid performance
Speaker 4: mistakes or or anything like that. Like because again, like
Speaker 4: with the work I do, it tends to perform, Like
Speaker 4: performance doesn't really come into it because I tend to
Speaker 4: work with startups and stuff, like what how do I
Speaker 4: just go about like even testing like certain things in
Speaker 4: my local dev environment in terms of like what do
Speaker 4: I put in my Ruby good to check how long
Speaker 4: stupp is taking? I don't even know that. That's how
Speaker 4: stupid this question is, because you know, back to the basics.
Speaker 3: You know, we have this debate all the time with
Speaker 3: internally because essentially the only way to find out is
Speaker 3: to have our huge customers. I don't name the customers,
Speaker 3: but we have a few huge, wile customers, and you
Speaker 3: put it on and you're going to find the problem
Speaker 3: very quickly. But then there's also the aspect of a
Speaker 3: new feature. Even when you deploy it looks great because
Speaker 3: nobody's using it right. Then two years later somebody uses
Speaker 3: it in a way that you don't anticipate. Reactive development
Speaker 3: isn't terrible. I would say, at the end of the day,
Speaker 3: your objective when you first get out is if you
Speaker 3: write the code clean, you can clean it up later.
Speaker 3: If you don't write the code clean, cleaning it up
Speaker 3: later is almost impossible. So I would say for feature developers,
Speaker 3: focus on keeping it clean, because once it's clean and
Speaker 3: it's very easy to read, somebody else can maintain it later.
Speaker 3: Then I would say that that's the case for like
Speaker 3: any small startup. Just focus on cleaning and a lot
Speaker 3: of different just review code and actually have an old job.
Speaker 3: I had. There's a guy he he hated me. He
Speaker 3: loved me, but I was just me because I would
Speaker 3: always review code and I would be like a stickler
Speaker 3: on making it really easy to read and maintainable and
Speaker 3: stuff like that. He was hated me, and then he
Speaker 3: eventually left that company, and I left that company, and
Speaker 3: I'm talking to old people and I've heard him say
Speaker 3: I miss Dave. Oh my god, I hated when he
Speaker 3: reviewed our code. But now I'm at a company that
Speaker 3: doesn't review code, and the crap that comes in here,
Speaker 3: we can't maintain it. Yeah, maintainability is ninety percent of
Speaker 3: getting your speed out of it, because if it's easy
Speaker 3: to maintain, it's easy. He don't want to add the
Speaker 3: the best trick for speed day one anyways, because if
Speaker 3: the code is never used, who cares if it runs
Speaker 3: a little, if it runs a two milliseconds slower, who
Speaker 3: cares care about the features that all of a sudden
Speaker 3: get like traction to make performance improvements? And then somebody,
Speaker 3: a team like mine is probably always required a big
Speaker 3: company is because eventually you get big enough where all
Speaker 3: of a sudden, hey ten years ago, people weren't paying
Speaker 3: attention to performance and now needs to and well, yeah, focus,
Speaker 3: done well.
Speaker 2: To your point on this.
Speaker 1: So we're actually running into performance issues on this application
Speaker 1: that I'm working on now for client. And what it
Speaker 1: does is it it does this gigantic calculation across a
Speaker 1: whole bunch of transactions on a hedge fund. And the
Speaker 1: people that wrote it, I mean, it's impossible to read,
Speaker 1: and so you know, you're trying to just parse through
Speaker 1: it and go.
Speaker 2: Okay, what what is it doing?
Speaker 1: And it's super hard, right, But the thing is is
Speaker 1: that it's it's in a psyche job. So it gets
Speaker 1: kicked off and you know, it runs, and you know,
Speaker 1: if it's a large fund, it can run forty five minutes,
Speaker 1: an hour, two hours, and you know, I'm looking at
Speaker 1: it and what we're you know, we had the conversation
Speaker 1: where we're saying, well, it'd be really nice to break
Speaker 1: this up, right so that you can parallelize some of
Speaker 1: the work because you have multiple workers, and so you
Speaker 1: know that would make it faster. And just getting to
Speaker 1: the point where we can actually figure out where to
Speaker 1: break it apart. Has been just I mean, it's been ugly,
Speaker 1: and you know, yeah, if it had been written in
Speaker 1: just a maintainable way, like you're talking about, where it's like, okay,
Speaker 1: I see where this kind of breaks down and you know,
Speaker 1: where it kind of goes into these methods and comes
Speaker 1: back out, it'd be way easier to do because it's like, Okay,
Speaker 1: we can turn these things into these things, and we
Speaker 1: can do these.
Speaker 2: Things over here.
Speaker 1: We can put these safeguards in place to make sure
Speaker 1: that it's not changing anything it shouldn't. But yeah, it's
Speaker 1: going to be a major overhaul, but it's like the
Speaker 1: core of the app, and so it's something that we're
Speaker 1: going to have to do. And so it's just been
Speaker 1: interesting to sit there and think about, Okay, yeah, if
Speaker 1: this had been written in a maintainable, readable, understandable manner, right,
Speaker 1: we're not even talking about how it executes or how
Speaker 1: fast it is in any other way, it would have
Speaker 1: made the fixes that we have to make to it
Speaker 1: now so much, so much easier. Yeah, And ninety percent
Speaker 1: of the problem is is that since it's all in
Speaker 1: one job, it's it does everything serially right. It does
Speaker 1: the whole process in one run without outsourcing any of
Speaker 1: the work.
Speaker 3: Yeah, everybody's ran into that application where they have a
Speaker 3: twenty six thousand line file and you're.
Speaker 1: Like what, Yeah, yeah, I don't think this one was
Speaker 1: twenty six thousand, but it's a few thousand at least.
Speaker 3: I'm thinking about a very specific contract they had a
Speaker 3: long time.
Speaker 2: Oh I'm sure. Yeah.
Speaker 3: Oh god, my one of my co workers worked at
Speaker 3: that job too. So if he listens to this, which
Speaker 3: I'm sure you will, he'll get a good laugh out
Speaker 3: of my comment right there. Yeah.
Speaker 1: But but it's interesting because, yeah, when we talk about performance,
Speaker 1: we don't often talk about clean code maintainability, you know,
Speaker 1: and and those kinds of things, because we're just talking about, yeah,
Speaker 1: what are the tweaks we can make?
Speaker 2: And then when we get into it.
Speaker 1: This is the other thing that I'm kind of pulling
Speaker 1: out of what you've been talking about, is that a
Speaker 1: lot of times the things that we're doing are not
Speaker 1: just simple fixes, right, because you know N plus one queries,
Speaker 1: it's like, okay, drop it includes.
Speaker 2: On it, right.
Speaker 1: You know, you want to make your query faster at
Speaker 1: an index, right, And most of the time. Those simple
Speaker 1: fixes will get you a long way, but you know,
Speaker 1: you get too far down the line, and yeah, all
Speaker 1: of a sudden, it's I have to restructure this to
Speaker 1: run a different way, or it's not gonna write in
Speaker 1: order to speed it up, or you know, there's gonna
Speaker 1: be some some massive infrastructure changes in order to make
Speaker 1: this faster, or you know, I have to change the
Speaker 1: way I'm doing caching or something like that, where it's
Speaker 1: not just this straightforward Oh okay, here's a shortcut. You
Speaker 1: got some speed out of it, and there are those
Speaker 1: I'm not saying they're not, but at a certain point,
Speaker 1: you get to that place where it's like, okay, all
Speaker 1: the all the quick fixes on this are done, and
Speaker 1: it's still not good enough.
Speaker 2: So now what?
Speaker 1: And if your code is maintainable, if you know, if
Speaker 1: you can approach it and understand it and know how
Speaker 1: to change it, it makes that job a lot easier.
Speaker 1: But it's it's also interesting because I've worked contracts where
Speaker 1: it's like I'm not I'm not going to pay you
Speaker 1: to write tests, and I'm not going to pay it
Speaker 1: a refactor, and I'm not going to pay it right
Speaker 1: and and I'm looking at them and going you realize
Speaker 1: that I'm going to be faster for the first like month,
Speaker 1: and then after that I'm going to have to refactor
Speaker 1: or write test in order to understand what the hell
Speaker 1: what this even does so that I can make it
Speaker 1: do what you want or change what it does.
Speaker 3: M Oh yeah, yeah. Without tests, I wouldn't be able
Speaker 3: to do any of my work right. That that is
Speaker 3: the key. And we don't have perfect test nobody probably does,
Speaker 3: but we have enough tests where all of a sudden,
Speaker 3: you change something and you can feel have a relative confidence.
Speaker 3: And I think that is another thing about performance work
Speaker 3: that you can't be afraid to change things and break it,
Speaker 3: which means you need to have software switches everywhere where
Speaker 3: you're turning it on and you're seeing if you broke.
Speaker 3: We've had extremely good luck within my UH group of
Speaker 3: turning these switches on and not breaking things. But if
Speaker 3: without those switches and while the monitors that you set up,
Speaker 3: you can't change anything because of fear, fear develop your
Speaker 3: driven development. It's like for it talking about before, it's
Speaker 3: like if you don't have like the indices that we
Speaker 3: took away, we didn't take it away because we're scared
Speaker 3: and was actually that was that before the show started.
Speaker 3: There's during the show that I talked about the indices
Speaker 3: forget anyways.
Speaker 2: I don't remember, yeah.
Speaker 3: Anyways, like yeah, so anyways, feared of a driven development
Speaker 3: will like slow down performance too because you don't, as
Speaker 3: you just said, you don't have tests. You're too scared
Speaker 3: to make any changes, like U will work And I
Speaker 3: don't know. My first couple jobs didn't have any tests.
Speaker 3: And then ever since then, I've had several jobs where
Speaker 3: I came in like nope, we're writing tests. No, I
Speaker 3: don't get No, we're not. No, yeah, we were writing
Speaker 3: tests and people are but how it's like, well, let's learn.
Speaker 2: Yeah.
Speaker 1: Well that's the other piece on the thing that I'm
Speaker 1: working on is there aren't tests and so it's like
Speaker 1: it's this mission critical thing that does all the valuations
Speaker 1: for all of the hedge funds and all of your customers.
Speaker 1: God and yeah, I mean I was just like you
Speaker 1: you want us to refactor this, but you know, but
Speaker 1: then I'm going putting a.
Speaker 2: Test on this is it's ugly, is so ugly.
Speaker 1: But the flip side is is that making it testable, right,
Speaker 1: writing the test will force us to refactor some of it.
Speaker 3: And when you wrote code without tests, typically methods all
Speaker 3: of a sudden go from I like to keep my
Speaker 3: methods within five lines of code and Ruby and Joba.
Speaker 3: I'm sure it would be different, but five lines of code,
Speaker 3: now that's not a rule, but you try to keep
Speaker 3: them small. Whereas if somebody wrote the original code without tests,
Speaker 3: usually they have one hundred let's just say, hundred line
Speaker 3: methods are not uncommon. Trying to test a hundred line
Speaker 3: method is like I felt, right, Uh, there's too many conditions.
Speaker 3: There's like seven ifs here I do.
Speaker 1: Yep, yeah, I I've heard that. I've heard that rule.
Speaker 1: I've heard that rule all the way up to like
Speaker 1: ten lines of code. Some people are a little bit
Speaker 1: more rigid on it than others. My rule of thumb
Speaker 1: is is if I'm talking to somebody and I'm explaining
Speaker 1: how something works, and I try and everything, that kind
Speaker 1: of happens at the same level, at the top level,
Speaker 1: right it it retrieves the you know, or maybe not
Speaker 1: even that right. That might be part of a lower step,
Speaker 1: but usually it's it makes sure that these conditions are
Speaker 1: met right, and then if the conditions are right, then
Speaker 1: it makes sure that you know it locks the table, right,
Speaker 1: does this other stuff in a transaction?
Speaker 3: Right?
Speaker 1: So there's this step right, Maybe the transaction is an
Speaker 1: implementation deal further down. But if I can say these
Speaker 1: are the five steps to do a thing right, and
Speaker 1: they're all mostly at the same conceptual level, then that's
Speaker 1: what my.
Speaker 2: Method should look like.
Speaker 1: And then underneath that it's like, okay, how do I
Speaker 1: do this stage of the theme right? And again at
Speaker 1: the conceptual level. And so once you start getting lower
Speaker 1: into the code or when I've written it, what you
Speaker 1: wind up seeing is is the bottom levels of the methods,
Speaker 1: they'll have some things that look like their implementation right
Speaker 1: where it's like I'm actually doing the query, and then
Speaker 1: maybe there's a and then call this to do the
Speaker 1: rest of the work because I feel like the retrieval
Speaker 1: and the work are at the same level. But you know,
Speaker 1: and then at the very bottom, yeah, it's just logic
Speaker 1: and just just being able to approach that and say, Okay,
Speaker 1: I see what the process is. Okay, I see what
Speaker 1: the process for this part of the process is. That's
Speaker 1: how I like to approach it. So yeah, so sometimes
Speaker 1: I wind up with a twenty or.
Speaker 2: Thirty line method.
Speaker 1: But at the end of the day, I'm looking at
Speaker 1: it and going, I understand, I can look at this
Speaker 1: and know what it does.
Speaker 3: I think if you can't look at the method and
Speaker 3: understand what a piece of it does, it's time to
Speaker 3: create another method name.
Speaker 2: Yeah, well that's another one.
Speaker 1: When I first got started, I kind of had this
Speaker 1: ethos from Rails where it was like, you know, they
Speaker 1: have convenient methods called like fine or you know, update
Speaker 1: or things like that, and I was like, okay, so
Speaker 1: I've got to come up with this short, you know,
Speaker 1: concise name nowadays is like updates the thing on the thing,
Speaker 1: because then you look at it and you're like, oh,
Speaker 1: I know what this does.
Speaker 3: I'll just make a guess on how many methods there
Speaker 3: are that we've created just on our ticket object, and
Speaker 3: it's got to be at least five hundred.
Speaker 2: I don't know whether that's time.
Speaker 3: Yeah, a thousand. You can't create little things, little update
Speaker 3: method and know what the hell it's doing. You need
Speaker 3: a bit Well.
Speaker 1: It's it's also interesting because you're working on an application
Speaker 1: that mostly focuses around the ticket so everything everything touches
Speaker 1: this one core.
Speaker 3: Piece absolutely, and we created several teams around it, right.
Speaker 1: You know, I've worked on other projects that didn't have that,
Speaker 1: but then they had like this core two or three
Speaker 1: objects that were the same thing. I've almost never seen
Speaker 1: an app where, you know, everything's spread across everything. The
Speaker 1: only exception I can really think of is the way
Speaker 1: that Top Endevs is put together. But that's because I
Speaker 1: deliver several different types of media and so those kind
Speaker 1: of get the between those and like the membership and
Speaker 1: role management, you know, it kind of gets spread across
Speaker 1: all of that because it's do you have access to it?
Speaker 2: And then how do you deliver it?
Speaker 1: And so you know, I get a wide swath of
Speaker 1: things that people are going to be generally touching. But yeah,
Speaker 1: I mean even within their little areas, right, it's podcast
Speaker 1: episodes have a ton of stuff on them, right, course,
Speaker 1: lessons have a ton of stuff on them. And and
Speaker 1: it's for that same reason, right, because within that little
Speaker 1: focused area, I've got that core piece and so understanding
Speaker 1: that and then recognizing, okay, so I've got to optimize
Speaker 1: and make it as easy to understand as possible.
Speaker 3: Around this thing, there's some some somebody else is going
Speaker 3: to maintain it and Chris, and you're.
Speaker 2: That's right, but I'm the boss, so that laughs Purson
Speaker 2: it quietly.
Speaker 3: Granted I worked at a company, it will go nameless
Speaker 3: that I before I got there. There's another topper that
Speaker 3: only named this also, and they didn't have everything was
Speaker 3: mixture between tabs and white space. There's always extra white
Speaker 3: space at the end. Nobody actually had any cleanup around that.
Speaker 3: So I wanted to want to created a script that
Speaker 3: basically made it so everything spaces and there's no extra
Speaker 3: white space at the Eddybody method. So you do it,
Speaker 3: get blame in that and everybody thought I wrote terrible code.
Speaker 3: I'm like, no, this is not my code right what
Speaker 3: it is, but get blame wise.
Speaker 1: Yeah, we've also, Yeah, I want to bring it back
Speaker 1: to performance a little bit. Kind of hit the maintainability
Speaker 1: idea pretty hard, and I think it's it's a critical
Speaker 1: piece of this, right because you're going to suffer trying
Speaker 1: to make performance happen with poorly maintained, poorly maintained or
Speaker 1: hard to maintain code. But when you're so you were
Speaker 1: talking about observability, I want to go back to that idea,
Speaker 1: like what are you actually looking for? Do you do
Speaker 1: you are you looking for kind of the like new
Speaker 1: relic for example, We'll break it down and say you
Speaker 1: spent this much time on your query, and this much
Speaker 1: time rendering the view, and this much time working in
Speaker 1: the controller, and this much time doing these other things.
Speaker 1: Is that what you're looking at when you when you
Speaker 1: see something slow or are you looking at some other
Speaker 1: breakdown or some other numbers, or how do you approach
Speaker 1: something like that?
Speaker 3: Before we started the team, it was basically, this is
Speaker 3: how long the presenter is taken, this is how long
Speaker 3: the controller is taken, and various little methods throughout saying
Speaker 3: they maybe validations. When we started this team, our first
Speaker 3: objective was, hey, listen, we need observability everywhere. So almost
Speaker 3: I'm not going to say every method, but every validator,
Speaker 3: for example, is wrapped in a data dog optional data
Speaker 3: dog tag where you can actually show really so literally
Speaker 3: every not every method, but a significant portion of the
Speaker 3: methods are actually showing. You can see and you can
Speaker 3: narrow down exactly in the code very fast, because all
Speaker 3: of a sudden, it's like you turn it down really
Speaker 3: there it is, it's right there, and you might have like, okay,
Speaker 3: it's within this section and there's three things going on
Speaker 3: here one of these three, and there's sometimes what happens
Speaker 3: is a lot of times it's one of the three.
Speaker 3: We have the option of trying to figure out where
Speaker 3: in the code. The slow part is where we have
Speaker 3: the option of just deploying more code that will tell
Speaker 3: us exactly where it is. And we've gotten to the
Speaker 3: point where there is some empty gaps, especially with the
Speaker 3: new features. When people add new features, they're not thinking
Speaker 3: about adding traces and stuff like that. But we can
Speaker 3: just really just go down and target exactly the method
Speaker 3: that we're looking for. And we've had things like you
Speaker 3: look at the code and you're like, oh, it looks fine,
Speaker 3: but you don't see a loop. And then way over
Speaker 3: here there's another loop, but that's in there. And then
Speaker 3: the way over here there's another loop and that's in
Speaker 3: there too. So you got I think they call that
Speaker 3: ncube or whatnottens, like all of a sudden, like, oh,
Speaker 3: obviously we need to do something with this, and then
Speaker 3: you add some for that specific one. I just added
Speaker 3: some memoization where all of a sudden, this last loop
Speaker 3: was just literally just it wasn't even a loop anymore,
Speaker 3: just calling it into a hash table look up and
Speaker 3: it's just a fan So all of a sudden turned
Speaker 3: it into an NQ but is still not good. But
Speaker 3: the performance of NQ versus and squared or whatever is huge.
Speaker 3: We got a big enough performance game there. All of
Speaker 3: a sudden, I could run away from that problem.
Speaker 2: Kick it down the road a little bit.
Speaker 3: Oh my god. Some of the times you see the
Speaker 3: code and you're like, in it's business decisions that people
Speaker 3: make too. Do they make business decisions like yeah, I
Speaker 3: can see why the developer wrote it like this, But
Speaker 3: why didn't the developer tell you, no, no, we can't
Speaker 3: do that. It's going to not perform in five years.
Speaker 1: Yeah. Sometimes you don't know though, at the end of
Speaker 1: the day.
Speaker 3: The one thing that I didn't talk about today or
Speaker 3: on might talk that I think is really important is
Speaker 3: when you first create an application and you deploy it,
Speaker 3: you can always set tight limits and the amount of
Speaker 3: data that you have, so product limit limits. Say, hey, listen,
Speaker 3: you can only have ten options in your drop down.
Speaker 3: That's obviously exaggeration, but you're allowed to put any ten,
Speaker 3: but you only get ten. You can always give more
Speaker 3: and eventually say I'll give you more. I'll give you more,
Speaker 3: I'll give you more. But going from hey, you get unlimited.
Speaker 3: I remember the first thing I worked at at Zendesk
Speaker 3: was you can have unlimited brands. Well, now the limit
Speaker 3: is three hundred because we realize that unlimited brands means
Speaker 3: almost well, unlimited complexity. And you can give me one
Speaker 3: hundred thousand brands because and that's one hundred thousand domains
Speaker 3: and that's the complexity around that is just too big.
Speaker 3: So yeah, adding limits to your product. And as a developer,
Speaker 3: I think that our responsibility to tell the product people
Speaker 3: that there's areas in the code. You go, if we
Speaker 3: have a hundred million of these, what does it? What's
Speaker 3: the performance? It's sometimes very obviously we wouldn't be able
Speaker 3: to have one hundred thousand of these right when we
Speaker 3: first developing a product. Go back to your product manager
Speaker 3: to say, hey, that feature, if somebody actually uses it
Speaker 3: and use it a lot, it's not gonna work. And
Speaker 3: they might come back and say, nobody would ever do that.
Speaker 3: It's like, so, what's the problem with actually having a
Speaker 3: limit if nobody's ever going to run into it? So
Speaker 3: I think that's also very key, and that's not programming
Speaker 3: per se. But I think ninety or a huge percentage
Speaker 3: of our job as a programmers to go back to
Speaker 3: the product managers and telling them, hey, listen, in three
Speaker 3: years from now, we wouldn't want a drop down. We
Speaker 3: had dropdowns for example that had over one hundred thousand
Speaker 3: options in them.
Speaker 2: Oh wow.
Speaker 3: First off, the UI isn't very good, but just loading
Speaker 3: them up and for example, for this one, this one
Speaker 3: was cool and it wasn't actual talk because one of
Speaker 3: the things that we did well that customer that had
Speaker 3: one hundred thousand options, they churned because well, we're too
Speaker 3: slow for them, right. We actually we probably would be
Speaker 3: able to support maybe even one hundred thousand in the
Speaker 3: options now because what we did is we changed our queries.
Speaker 3: And our original queries was grabbing one hundred thousand active
Speaker 3: record objects. And you can have one hundred thousand active
Speaker 3: rectcord objects without a serious time penalty, right. But what
Speaker 3: we do now is we pluck for the data, so
Speaker 3: we get the data fields that we need, and we
Speaker 3: only need like two or three fields out of records,
Speaker 3: so you're not initializing one hundred thousand active record objects,
Speaker 3: so you grab one hundred thousands options right in the array.
Speaker 3: And then you populate it via the the erase slash
Speaker 3: hash that you get back, and all of a sudden,
Speaker 3: this is very fast now. And all of a sudden,
Speaker 3: that customer that had one hundred thousand options that churned
Speaker 3: five six years ago, we might be able to maintain
Speaker 3: them now, but they're not coming back right. And that's
Speaker 3: another reason why you have a performance teams. All of
Speaker 3: a sudden, once you start have enough churn, people are like, oh,
Speaker 3: customers are leaving because of performance.
Speaker 2: Yeah.
Speaker 4: Do you have any opinions around using things like five
Speaker 4: ars of falcon in front of reels to improve performance?
Speaker 4: Do you have any experience with that?
Speaker 3: I wish I did. No, Yeah, that's good. Of the
Speaker 3: answer as I can give you is yeah, no, I
Speaker 3: have not used fibers at all.
Speaker 4: So yeah, I just started a new side project and
Speaker 4: I've put falcon and not because it's the performance intensive project.
Speaker 4: It's just a side project that lets me play around
Speaker 4: with new stuff with no risk. I was like, why not,
Speaker 4: it'll be fun.
Speaker 3: I did that for what I worked for the UFC
Speaker 3: one time in the UFC thought they were gonna get
Speaker 3: a lot of different but it just they thought day
Speaker 3: one is gonna be bam. It's just an e commerce
Speaker 3: store too, so it's not going to get that much traffic.
Speaker 3: But UFC is like, but we're UFC. It's like yeah,
Speaker 3: but like it's still like only very small part. Anyways. Yeah,
Speaker 3: I remember I was able to play in that and
Speaker 3: it's like, hey, I'm just going to tweak every part
Speaker 3: of this e commerce app so you can go as
Speaker 3: fast as possible. I think they launched and had almost
Speaker 3: no traffic, but it was fast for the no traffic.
Speaker 2: Yeah.
Speaker 1: One thing that I'm running into with top end devs
Speaker 1: is the search takes like five seconds to run.
Speaker 3: So search is a very completely different team for us.
Speaker 3: For us, it's the exact opposite. So our search team
Speaker 3: is better than my sequel. Like whatever they're doing, they've
Speaker 3: made it index things quite fast, so we've done. There's
Speaker 3: APIs that sometimes you need. They needed to be like
Speaker 3: no most current data, so you were going after SQEL,
Speaker 3: But there's APIs where it's like, hey, if it's delayed
Speaker 3: by two hundred milliseconds, is it that big of a
Speaker 3: deal that we don't return all the tickets. We minus
Speaker 3: all the tickets except for the one last one that
Speaker 3: they put in right, and in those cases we've found
Speaker 3: the performance of a LASS search is so much faster, like,
Speaker 3: well we'll get our response times response times, we're just
Speaker 3: dropping half for example.
Speaker 1: Yeah, that's the thing that I've been looking at lately.
Speaker 1: Is I put in PG search and yeah, it's just
Speaker 1: you know, doing a multi search across all the models.
Speaker 1: I mean, it will give you back a result, but
Speaker 1: I had to tell the proxy server that sits in
Speaker 1: front of it, come all proxy, not to time out
Speaker 1: so fast in order to get those results. And yeah,
Speaker 1: I've been looking at possibly moving to elastic search because
Speaker 1: I hear it's really just stinking fast. I was using
Speaker 1: another search engine, mealy Search, but the problem I had
Speaker 1: there was that they I couldn't I could only search
Speaker 1: one type of object at a time, and so I
Speaker 1: had to go and say, okay, give me all the episodes,
Speaker 1: give me all the podcast and it was fairly fast,
Speaker 1: but I hated making six queries to it to get
Speaker 1: all the different things because I kind of want just
Speaker 1: the most relevant thing to show up first. And so yeah,
Speaker 1: that's where I'm looking at elastic search, because I think
Speaker 1: elastic search doesn't make you do that.
Speaker 3: So yeah, we have a differ team that maintains that. So
Speaker 3: that's another place where I just use plastic search and
Speaker 3: I was fast in our system, but we have That's
Speaker 3: the thing about being at a big company. There's areas
Speaker 3: where you know how to use it within your company,
Speaker 3: but to set it up or anything to be an
Speaker 3: expert on it. No, Yeah, that's where I missed the
Speaker 3: being a startup get you get to be an expert,
Speaker 3: or at least you get to play with more tools.
Speaker 2: Right, So, are there any other big secrets.
Speaker 1: That you want to share with us or thoughts before
Speaker 1: we head into picks.
Speaker 3: Nothing that's coming in to mind. But I mean, as
Speaker 3: I said, we have. I mean I'm creating a whole
Speaker 3: new talk. For example, I think I told you before
Speaker 3: we started, it's like how graph ql is going to be,
Speaker 3: uh speed up a whole bunch of things for us,
Speaker 3: and then just there's so many different areas that I
Speaker 3: was not able to hit that now I'm going to
Speaker 3: I'm submitting cocks right now to actually talk about those.
Speaker 3: But those are not well thought out in my brain yet,
Speaker 3: So I don't think I should be a talking about
Speaker 3: right now.
Speaker 1: Yeah, you mentioned graph ql, and I have to say
Speaker 1: my experience implementing things in graph Ql has been not
Speaker 1: my favorite thing that I've ever done. But you're talking
Speaker 1: about a performance aspect of it is. Yeah, I'm a
Speaker 1: little curious what you're looking at there. I know that
Speaker 1: with graph ql, the thing that people tend to like
Speaker 1: is that you make one query and you get all
Speaker 1: the data you need one go.
Speaker 2: So is that where the speed up comes from or
Speaker 2: is it something else?
Speaker 3: No? No, So we don't use Ruby every place, and
Speaker 3: our graph Ql endpoints we use Java. Then we're actually
Speaker 3: only using them internally. We're not using them externally yet,
Speaker 3: but internally it hits our Java service. Java does an
Speaker 3: amazing job of paralyzing things. So then Java makes like
Speaker 3: fifty ten probably, but let's say fifty different rest requests
Speaker 3: to gain application gets them all back way faster than
Speaker 3: if they was compiling up, and then that comes back
Speaker 3: to the Java app. The Java app knows how to
Speaker 3: configure and put it all in the way it's supposed
Speaker 3: to put it back, and then serializes it and sent
Speaker 3: back to the requester of the actual appeatt And because
Speaker 3: of the paralyzation of that, it's so much faster because
Speaker 3: rest requests right now in our application, we have something
Speaker 3: what we call internally side loads, where you can basically
Speaker 3: say I want anything I want. It's almost like a
Speaker 3: poor man's graph QL. Well, if all of a sudden
Speaker 3: you're doing that serially, it's just the response times here,
Speaker 3: so graftl speeding us up because it's basically forcing us
Speaker 3: to come back to true RESTful responses that are really fast.
Speaker 1: All right, cool, Well, let's go ahead and do some
Speaker 1: picks before we get to those.
Speaker 2: If people want to follow you online, where they find you, I.
Speaker 3: Think get hub or LinkedIn, or you'd find me. Dr
Speaker 3: Henter is usually what you find me. If you look
Speaker 3: for Dr Henter, get Hubb or Dr Henter. I think
Speaker 3: get LinkedIn. Even Dr Henner in my Gmail would work
Speaker 3: too awesome.
Speaker 2: All right, well, let's go ahead and do some picks.
Speaker 2: Are you sure? Do you have some picks for us?
Speaker 3: Yeah?
Speaker 4: Hang on one second, I have broken broad for my client,
Speaker 4: which is all the fun.
Speaker 3: I'm flaxing.
Speaker 2: I'm on it.
Speaker 3: I'll give you five minutes. I'm on it.
Speaker 4: Perfect timing, isn't it shouldn't I shouldn't have imrged. I
Speaker 4: merged up on requests just before coming onto this collision
Speaker 4: to done that.
Speaker 1: Yeah, so I used when he got real quiet over
Speaker 1: there that that's.
Speaker 2: What was going on.
Speaker 4: It was like, yeah, yeah, sorry about that.
Speaker 2: Sorry picks.
Speaker 4: I'm gonna start with a very selfish pick, which is
Speaker 4: I just relaunched one of my side projects called Skatagun
Speaker 4: dot Email, which has been in hibernation for like two
Speaker 4: and a half years because of the joys of sales
Speaker 4: tax and EU V eight. I was using Stripe for
Speaker 4: payments on my accounters Like, yeah, you know, you can't
Speaker 4: do that because you'll have to collect V eight for
Speaker 4: everyone in the EU. And I'm like, okay, thanks for
Speaker 4: telling me that now. So I've moved payments over to paddle, uh,
Speaker 4: which took a lot longer because I got distracted by
Speaker 4: this stuff, but it's it's back now. So Skatagun dot
Speaker 4: Email has relaunched it Tuesday. So that's one pick.
Speaker 3: The I.
Speaker 4: Usually do like a music or a TV show pick,
Speaker 4: So I'll go to the TV show this time. Severance
Speaker 4: on Apple TV season two just started. It's uh weird,
Speaker 4: and I like it. I like it very much. It's
Speaker 4: been a very strong start. I think it was quite
Speaker 4: a cliffhanger ending in season one, and the show has
Speaker 4: started off really strong, so if you haven't watched it,
Speaker 4: i'd recommend that.
Speaker 2: Just getting the links here for that.
Speaker 1: I've got a couple of picks I'm gonna put out there, so,
Speaker 1: as most of you probably know, I tend to do
Speaker 1: the board game picks as part of my picks, So
Speaker 1: I'm going to put out The game is called Cascadero,
Speaker 1: and it's.
Speaker 2: It's got some.
Speaker 1: Elements that are similar to other board games that I've played.
Speaker 1: So there's kind of a there's a track that you
Speaker 1: advance up, and you get rewards for hitting certain milestones,
Speaker 1: and you also get points for hitting certain milestones. The
Speaker 1: way that you advance up the track is you so
Speaker 1: there's a kingdom and the king has dyed, and so
Speaker 1: you're basically you're out there connecting the cities and things
Speaker 1: like that and spreading the word on the new king
Speaker 1: or whatever. And so you put little horsemen down on
Speaker 1: the on the board around these towns and you're connecting
Speaker 1: up the towns, and you get bonus points for say,
Speaker 1: connecting two towns to the same color. So the first
Speaker 1: time you do that for any given color there are
Speaker 1: five colors, then you.
Speaker 2: You get points you advance up the board.
Speaker 1: If you are the first person to that city and
Speaker 1: it's not the first horseman that you've put down in
Speaker 1: the group, then you get one point. And if you're
Speaker 1: the second, third, fourth, I guess you can do fourth
Speaker 1: because other people can connect onto it as well, right,
Speaker 1: but it's it's per group. So when you connect a
Speaker 1: group to a city and you're the second or third,
Speaker 1: or fourth person, you move your thing up the advanced
Speaker 1: track or advance it up the track of that color
Speaker 1: two spaces instead of one. And then if there's a
Speaker 1: herald on the city, then you add one to whatever
Speaker 1: you get to advance it. And then there's a bonus
Speaker 1: for having connected two cities of all five colors with
Speaker 1: any you know across any of or all of your groups.
Speaker 1: And then there are also point bonuses for getting all
Speaker 1: five of your markers past the first it's kind of
Speaker 1: a finish line three on the second finish line, or
Speaker 1: getting being the first person to get one all the
Speaker 1: way to the end of one of the lines. And
Speaker 1: the way that you win is you have the most
Speaker 1: points of the people that get there their piece all
Speaker 1: the way to the end of whatever color they're playing.
Speaker 1: So when I was playing, I was playing as Red,
Speaker 1: and so I made sure I got my piece all
Speaker 1: the way to the end of Red, and then I
Speaker 1: tried to get it all the way to the end
Speaker 1: of yellow as well, but I didn't have to do that.
Speaker 1: And then one of the other guys that I was
Speaker 1: playing with got his all the way to the end
Speaker 1: of blue, which was the color he was playing, but
Speaker 1: he did that on his last turn, and if he
Speaker 1: hadn't gotten it there, even though he had more points
Speaker 1: than I did, because I got mine to the end
Speaker 1: of the track, I won.
Speaker 2: And so what your.
Speaker 1: Goal is is you want to be the person who
Speaker 1: gets it all the way to the end and has
Speaker 1: the most points. But the way you end the game
Speaker 1: is by getting more than fifty points. As soon as
Speaker 1: somebody crosses that fifty point threshold, all the other players
Speaker 1: get one more turn and then it ends. Or when
Speaker 1: somebody runs out of horsemen and can't place another horseman,
Speaker 1: that's the other way it ends. And that's the way
Speaker 1: I ended for us, because we had never played it
Speaker 1: before and so we weren't racking up the points the
Speaker 1: way that we should have been.
Speaker 2: But anyway, it was pretty fun. And then they're like.
Speaker 1: I said, there are different rewards up the track that
Speaker 1: you know, so you can advance another marker up the track,
Speaker 1: or you can place another horseman, or you can move
Speaker 1: a horseman or just stuff like that.
Speaker 2: So anyway, it was pretty fun Cascadero.
Speaker 1: It came out in twenty twenty four, so it's a
Speaker 1: relatively new game and board game Geek gives it a
Speaker 1: weight of two point five to three, which means that
Speaker 1: if you're kind of a casual player, this might be
Speaker 1: a little bit complicated, but not so complicated that you're
Speaker 1: not going to enjoy it. I have to say that
Speaker 1: the complicated part is just keeping track of what you've
Speaker 1: connected with your groups, and then you know, keeping track
Speaker 1: of how to move stuff up the track, and you
Speaker 1: get used to that in the first time minutes or so,
Speaker 1: and then I would probably wait it once you understand
Speaker 1: those two things at like one point nine. I mean,
Speaker 1: it's it's just those couple of concepts you have to
Speaker 1: get used to and then you're good to go. It
Speaker 1: took us about there were three of us playing. It
Speaker 1: took us what forty five minutes maybe an hour, and
Speaker 1: that was with us learning the game. So I would,
Speaker 1: I would assume that it probably takes you know, if
Speaker 1: you have four or five people playing, you know, I
Speaker 1: thank you forty five minutes. So anyway, it was fun.
Speaker 1: I definitely enjoyed it. One of my buddies is teaching
Speaker 1: the hot games at salt Con in March, which is
Speaker 1: the big board game convention up in Layton, Utah.
Speaker 2: And yeah, anyway, so that was one of the games
Speaker 2: he had to learn.
Speaker 1: We'll probably pick up another one later, but yeah, I
Speaker 1: really enjoyed it, So I'm gonna pick that. My wife
Speaker 1: and I for TV shows, we just started watching season
Speaker 1: two of The Night Agent and we enjoyed the first
Speaker 1: season a lot, so I'll pick that as well. Let
Speaker 1: me see if I get a link for that.
Speaker 2: And then I am starting to.
Speaker 1: Put together meetups and stuff for ruby Geniuses, so if
Speaker 1: you go to rubygeniuses dot com. I kind of so
Speaker 1: it took me a little longer to get it launched
Speaker 1: than I wanted to. And the reason essentially is that
Speaker 1: the stuff that I'm doing for top end devs as
Speaker 1: far as like delivering courses and podcasts and stuff like that,
Speaker 1: I've been talking to other people who want to use
Speaker 1: the system.
Speaker 2: I'm using and so rather than do the smart thing.
Speaker 1: And just deploy them a copy, I made it a
Speaker 1: multi tenant app so I can charge them for it.
Speaker 1: So that that took a bit of work to kind
Speaker 1: of get everything figured out and learn how to do
Speaker 1: multi tenancy with Stripe connect on you know, on my app,
Speaker 1: make sure I'm getting all the right web hooks and
Speaker 1: all that stuff. But it's basically done. So if you
Speaker 1: want to go sign up, I'm going to change the
Speaker 1: price today. I'm going to move it down to forty
Speaker 1: seven dollars just because I'm still pretty happy about the inauguration.
Speaker 2: Of Donald Trump and he's president at forty seven.
Speaker 1: But yeah, so if you want to go and sign
Speaker 1: up for forty seven dollars, it will go back up
Speaker 1: to ninety seven dollars in a few weeks and that'll
Speaker 1: give you access to I'm essentially I'm putting together videos kind.
Speaker 2: Of like railscast was or go Rails or Drifting Ruby.
Speaker 1: My focus is going to be essentially I'm going to
Speaker 1: say here's my project, and then you're gonna get a
Speaker 1: video of me working on the project every week, and
Speaker 1: it might be interspersed with and this is something that
Speaker 1: I fixed on top end devs. Right, So if I
Speaker 1: get elastic search working for the search for example, I
Speaker 1: might work that in the first app I'm building is
Speaker 1: actually to help track bills through the Utah legislature, and
Speaker 1: so it's going to be hitting their APIs and working
Speaker 1: around the gaping holes in their APIs.
Speaker 2: Because they have gaping holes in their APIs.
Speaker 1: So there may be some screen scraping and things like
Speaker 1: that to get all the data in and then yeah,
Speaker 1: putting together the alerts and stuff like that, so you know,
Speaker 1: just you know, you get a notice that says, hey,
Speaker 1: your bill went to committee, or you know, your bill
Speaker 1: had its second reading, and so if it has a
Speaker 1: third reading and passes, then it passes, you know, and
Speaker 1: just kind of working through all that stuff and then
Speaker 1: helping people connect with their legislators because I feel like
Speaker 1: if we have more people involved in the legislative process
Speaker 1: and talking to their legislators, regardless of which side of
Speaker 1: the aisle you're on, I think.
Speaker 2: We generally get better.
Speaker 1: Legislation because I think most of the things when we're
Speaker 1: talking about roads and schools and things like that, we
Speaker 1: generally agree on what we want, whether you're right or left,
Speaker 1: and So what I want is I want moms and
Speaker 1: dads talking to the legislators and saying you're passing, you're
Speaker 1: pushing this education bill, but that's not what we want,
Speaker 1: you know, or things like that. So so that's what
Speaker 1: I'm going to be working through and just showing people
Speaker 1: how to do that. It's probably going to have some
Speaker 1: premium features too that involve AI and doing bill summaries
Speaker 1: for AI. I'm already doing this for another project, and
Speaker 1: so yeah, you're going to get kind of a wide
Speaker 1: breadth of things out of this. Also creating groups so
Speaker 1: people can discuss bills together. I want to give organizations
Speaker 1: an option for putting together their slate of bills and
Speaker 1: where they stand on it, and give them some way
Speaker 1: of posting that on their websites. So those are kind
Speaker 1: of the features that I'm going to be putting in
Speaker 1: so you know, imbedible stuff. So anyway, if you're looking
Speaker 1: to learn any of those things, then then you want
Speaker 1: to sign up for that. We're also doing the meetups
Speaker 1: and we'll be going through some of the technologies, maybe
Speaker 1: a little more in depth, so where in the videos
Speaker 1: I'll be showing you I'm building this feature using Turbo,
Speaker 1: I may go through, Hey, here are all the features
Speaker 1: that turbo offers in this way or comal or I've
Speaker 1: had people ask me for all kinds of stuff. If
Speaker 1: you connect with me on LinkedIn too, I'll actually get
Speaker 1: on a call with you and ask you what you
Speaker 1: want to learn. So anyway, I've been way more long
Speaker 1: winded on that than I want. But yeah, so that's
Speaker 1: all going to be coming out through Rubygeniuses dot com
Speaker 1: and yeah, so hopefully that's in the wheelhouse of things
Speaker 1: you want. But yeah, we're going to be doing meetups
Speaker 1: and tutorials and screencasts and courses in the whole nine
Speaker 1: yards and you get access to all of that through
Speaker 1: the membership.
Speaker 2: So anyway, David, what are your picks?
Speaker 3: Yes, So, I have a coworker actually that I work
Speaker 3: side by side with that's publishing a book soon and
Speaker 3: it's called rail Scales. He actually gives more talks about
Speaker 3: scaling rails than I do. But Christian Polantis is his
Speaker 3: the author's name, book name, Ral Scales. I'm actually very
Speaker 3: interested in to see what it looks like and I
Speaker 3: can't wait to get myself a copy. I hope I
Speaker 3: get myself a free copy.
Speaker 2: Yeah, become a technical reviewer. They give those out for free.
Speaker 3: Well, I mean he works with me and he's yeah,
Speaker 3: I'll tell him. I gave him a shout out. There
Speaker 3: you go, you go. What else? I actually have one
Speaker 3: other coworker that he wrote an article on AWS about
Speaker 3: scaling Zendesk And if you do a search for the
Speaker 3: guy's name is Anatoly aws Zendesk Performance, So Anatoly Aws
Speaker 3: Zendesk Performance, you will find his article in Google and
Speaker 3: he talks about some of the things that we've been doing.
Speaker 3: Much more on the infrastructure side. He's a much more
Speaker 3: of an infrastructure than a Ruby guy, but I work
Speaker 3: side by side of the He's absolutely brilliant. Also, if
Speaker 3: anybody's hiring product manager or managers in general, I know
Speaker 3: several that got laid off, So feel free to ping
Speaker 3: me on LinkedIn because I feel like there's some good
Speaker 3: people out there that I know that I'd love to
Speaker 3: connect to other people. And lastly, you guys probably don't
Speaker 3: know Swift, but Swift is essentially an interactive game that
Speaker 3: you connect to a bicycle a bicycle trainer, and I
Speaker 3: absolutely love it. I'm I mean, I every night like
Speaker 3: I do races sometimes and the last few nights I've
Speaker 3: been doing races. I am not a racer. I am
Speaker 3: not that fast, But at the end of the day,
Speaker 3: I'll still go in and do the races. And I
Speaker 3: think the category ds I'm taught. I'm I'm the best
Speaker 3: of the worst category.
Speaker 2: Oh, there you go.
Speaker 3: So as soon as I get the next category, I'll
Speaker 3: get destroyed. Yeah.
Speaker 1: I got a membership to Swift when I bought my
Speaker 1: bike trainer because I have a Wahoo trainer and or.
Speaker 3: The is it the bike? Okay, I got the Kicker bike.
Speaker 2: Okay, Yeah, I've only used it once or twice. In fact,
Speaker 2: I think it expired.
Speaker 1: But yeah, it's cool because it's yeah, it's almost like,
Speaker 1: well I guess it's not almost like, but it's it's
Speaker 1: kind of fun to just ride your bike.
Speaker 3: And yeah, if you're.
Speaker 2: Harder up the hills and all that stuff.
Speaker 3: And exactly, and then that you're in Utah, so you
Speaker 3: you suffer from the coldness and you don't want to
Speaker 3: be biking.
Speaker 1: Oh yeah, I don't train outside. I'm training for a
Speaker 1: triathlon now this year. I want to do either an
Speaker 1: iron Man or half iron Man.
Speaker 3: I haven't decided I'd do that with you, but I
Speaker 3: think my knees would say no on the running. In fact,
Speaker 3: I got an MRI scheduled for next week from my shoulders,
Speaker 3: so man, I'm breaking down. Yeah, and last I.
Speaker 1: Watched my dad fall apart, and I just I feel
Speaker 1: like I need to keep moving, so I don't.
Speaker 3: So the last day I was going to give this
Speaker 3: my last thing to saying go Bills because I'm hoping
Speaker 3: for the Super Bowl. But well I still say go
Speaker 3: Bills because I love them.
Speaker 2: Okay, so have you decided if you're rooting for the
Speaker 2: Eagles or the Chiefs?
Speaker 3: Have I decided whether I'm watching the Super Bowls?
Speaker 2: Fair enough?
Speaker 3: You know, I don't hate the Chiefs, but at the
Speaker 3: same time, I got to root for the Eagles just
Speaker 3: because they I don't want the Chiefs to win three
Speaker 3: times in a row. They're not bad, They're not the
Speaker 3: Patriots where I hated them, but still I don't want
Speaker 3: to win three times in a row. And I think
Speaker 3: that their fan base understands that too.
Speaker 2: Yeah.
Speaker 1: I like both teams, so I'm not sure or which
Speaker 1: way I'm swinging yet, but.
Speaker 2: A right good deal?
Speaker 3: What was that was amazing? This year?
Speaker 1: All right, well, I'm gonna go ahead and end the stream,
Speaker 1: but thanks for coming, thank you, and until next time,
Speaker 1: folks max out