Speaker 1: Hey, growl. Welcome to another episode of Ruby Ropes. I'm
Speaker 1: David Kumia and today on our panel we have Luke
Speaker 1: Sutter's Stutters, and we have John Epperson everybody, and today
Speaker 1: we have a special guest, Jesse Speedback.
Speaker 2: Great to be here.
Speaker 1: So, Jesse, would you mind explaining just a bit about
Speaker 1: who you are, some of the things that you're doing,
Speaker 1: who you work for, and what are you're famous and
Speaker 1: all that good stuff.
Speaker 2: Yeah. Absolutely, so. My name is Jessin Speedback. I'm a
Speaker 2: senior engineer at a company called Ibata, which is a
Speaker 2: cash back for shopping app based in Denver, Colorado. I've
Speaker 2: been working there for about three and a half years.
Speaker 2: We are to do some hiring, so check out our
Speaker 2: cruise page. I guess I'm famous as it were, because
Speaker 2: I gave a talk at the first Remote Rails comp
Speaker 2: this past May, and I talked about kind of how
Speaker 2: crummy of a developer I am.
Speaker 3: Yeah, I think we can all relate to that on
Speaker 3: the daily basis sometimes, So would you might have given
Speaker 3: a bit of a you know, highlight talk about what
Speaker 3: you covered the conference and stuff, so we can just
Speaker 3: kind of pick it up from there.
Speaker 1: We'll link to the in the show notes a link
Speaker 1: to the conference, but just for those who maybe didn't
Speaker 1: see it.
Speaker 2: Sure, and there's no substitute for actually watching this fantastic talk.
Speaker 2: But more more seriously, the talk is really about my
Speaker 2: experience as a tech lead and I bought a working
Speaker 2: on a pretty critical project over the course of about
Speaker 2: six six months or so, and over that time I
Speaker 2: made four very big mistakes that put the project in jeopardy.
Speaker 2: And hopefully there are mistakes that I will learn from
Speaker 2: and not make again as I continue to lead projects
Speaker 2: that I bought in the future. And my hope is
Speaker 2: that by sort of articulating these mistakes and what I
Speaker 2: learned from them, other folks can benefit. And so the
Speaker 2: four mistakes that I made are first, we picked the
Speaker 2: wrong technology. Can get more into that. We also, as
Speaker 2: a team we siloed work, so work was divided up
Speaker 2: in not the best way. We fell into the premature
Speaker 2: optimization trap. And then maybe worst of all, we made
Speaker 2: way too many changes at one time. So I can
Speaker 2: go into detail on any of those.
Speaker 1: Yeah, and I think it's fair to say that our
Speaker 1: listeners other members on the panel have been there before.
Speaker 4: Yeah, I've worked with loads of people who made those mistakes.
Speaker 4: Obviously I've never made them myself, but oh my word.
Speaker 5: Luke is the perfect developer. Let's be fair.
Speaker 6: No, I was actually going to jump in and say that, like,
Speaker 6: I feel like I don't feel like I'm alone, but
Speaker 6: usually when you make a mistake, it cascades into more
Speaker 6: for whatever reason. I generally feel like you you either
Speaker 6: kind of a rolling along and you feel pretty good
Speaker 6: about stuff, or you're just like in a bottomless pit
Speaker 6: and there's very little in between. Time you just suddenly
Speaker 6: notice that you're at the bottom. So yeah, just saying.
Speaker 2: And absolutely, and I just want to kind of call
Speaker 2: out that there's like a certain amount of privilege that
Speaker 2: comes with being able to talk about our mistakes. Right,
Speaker 2: I'm not worried that my boss is going to fire me,
Speaker 2: and I'm also not worried that folks won't take me
Speaker 2: seriously for getting this talk. If anything, it probably improves
Speaker 2: my reputation as evidence by getting to talk to three
Speaker 2: of you gentlemen. So I just want to call that
Speaker 2: out because I think it's important.
Speaker 6: I think that, Okay, So speaking of that, I think
Speaker 6: there's a give and take there too, So I think
Speaker 6: this is one of this will get into a thing
Speaker 6: that I care a lot about this particular subject because
Speaker 6: for most of my shoot, I still feel it today
Speaker 6: or whatever, it doesn't really matter. I still always feel
Speaker 6: this pressure to not let people know about the mistakes
Speaker 6: that I made, right, because letting people know about the
Speaker 6: mistakes that I made makes me more vulnerable to them
Speaker 6: not wanting me to work for them, right, like fire
Speaker 6: me whatever. The bad thing is that it's a boogeyman,
Speaker 6: it's not real. It's just a pressure, right, And so
Speaker 6: I guess what I'm trying to get at is like
Speaker 6: I kind of feel like one of the things that
Speaker 6: I always care a lot about, what mistakes is telling
Speaker 6: people about your mistakes, I actually have discovered in reality
Speaker 6: actually makes people respect you more, but it doesn't change
Speaker 6: the fact that it's like really hard, and I still
Speaker 6: feel that pressure to this day.
Speaker 2: Yeah, John, Yeah, I agree with what you're saying. I
Speaker 2: think that for a lot of us, talking about mistakes
Speaker 2: openly and honestly and what we learn from them actually
Speaker 2: builds our credibility. But that's not always the case, depending
Speaker 2: on sort of how you present what you look like.
Speaker 2: I think that you guys recently did an episode. I
Speaker 2: think you talked about issues of hiring and getting equality
Speaker 2: in a workplace, and I think that that kind of
Speaker 2: plays in here for sure.
Speaker 1: And I would go as far to say that it
Speaker 1: depends on what the mistake is, what kind of mistake
Speaker 1: it is. If it is just utter and complete negligence,
Speaker 1: then those aren't really the kind of mistakes I would
Speaker 1: really want to be forthcoming and outright about, you know.
Speaker 1: But like if I went in and always had a
Speaker 1: VP and tunnel into my production environment and then we
Speaker 1: got malware in our local work environment and that transferred
Speaker 1: over to production and encrypted all of our production data,
Speaker 1: I don't know if I would really say like, oh, yeah,
Speaker 1: you know, that was a silly mistake of mine. No,
Speaker 1: it's like, okay, not only of our customers effective, but
Speaker 1: now you know, my jobs in jeopardy because I decided
Speaker 1: to always take shortcuts. But something like and what I'm
Speaker 1: really interested in is your technology framework or the use
Speaker 1: the wrong technology? Can you explain a bit more about
Speaker 1: the scenario?
Speaker 2: Sure? Absolutely, so. This is the high level we had
Speaker 2: ibought it. We have just a wonderful majestic monolith rails application. Actually,
Speaker 2: originally the application we've written in Scala, and then after
Speaker 2: about a month of that, they switched it over and
Speaker 2: rebuilt the whole thing in Rails and it's been Rails
Speaker 2: ever since. So it's going on close almost ten years
Speaker 2: at this point. So it's a it's a large application,
Speaker 2: it serves millions of users. Very cool, and about three
Speaker 2: years ago when I joined, we started thinking. We started
Speaker 2: growing the team and thinking about how we could, like
Speaker 2: many people have pull pieces out of the monolith into
Speaker 2: micro services. So this project in particular was about taking
Speaker 2: a piece of billing logic from the system, from the
Speaker 2: monolith and pulling it out into a micro service. The
Speaker 2: hope was to make it better, encapsulated, easier to iterate on,
Speaker 2: not you know, isolate dependencies, every reason that you'd think
Speaker 2: to build a micro service. So we chose Actually, before
Speaker 2: I say the technology, because before I get trolled by
Speaker 2: all the lovers of this technology, I'm going to preface
Speaker 2: this by saying I don't think that this technology is wrong,
Speaker 2: and I don't think it's bad in and of itself.
Speaker 2: I just think it was not the right technology for
Speaker 2: the problem.
Speaker 4: And the team hold on a second hold on the
Speaker 4: second Jesse because the slide I'm looking at says big mistakes.
Speaker 4: It doesn't say small mistakes. It said big mistakes. So
Speaker 4: let's just keep this in mind, beful, what we reveal,
Speaker 4: what this technology is, What a big mistake.
Speaker 2: Was absolutely this. Whoever uses this technology is definitely making
Speaker 2: a big mistake. So you spoiler. We weren't building like
Speaker 2: us a side rails app micro service, which would probably
Speaker 2: would not have been as big a mistake. But the
Speaker 2: issue really was that our team, the small team of
Speaker 2: developers and then a larger team of engineers in the company,
Speaker 2: really did not have a ton of experience with the
Speaker 2: framework that we chose, and as a result, we ended
Speaker 2: up having to do a lot of plumbing and reinventing
Speaker 2: the wheel and just not benefiting from the institutional experience
Speaker 2: that exists at Ibottom. And unfortunately, right this this could
Speaker 2: work if you're doing kind of like a proof concept,
Speaker 2: like let's show what this technology can do. Let's pick
Speaker 2: a pretty isolated use case. But the building logic that
Speaker 2: we were pulling out about Monolith was basically like do
Speaker 2: or die. Right, if it did not work, it costs
Speaker 2: millions of dollars to fix, or you know, it ends
Speaker 2: up costing company millions of dollars. So it really we
Speaker 2: were walking on a tide rope and there was no
Speaker 2: net underneath us, and unfortunately we decided to I guess
Speaker 2: like walk on our hands instead of go across normally.
Speaker 2: So the framework that we used is called and I
Speaker 2: think for a team that knows AKA, this probably would
Speaker 2: have been really a perfect tool for the job. But
Speaker 2: our team and our company really did not have a
Speaker 2: ton of experience with Akka, and so unfortunately we weren't
Speaker 2: able to sort of take advantage of it and use
Speaker 2: it in a way that sort of professional AKA developers
Speaker 2: likely can.
Speaker 4: So this is kind of like a functional programming thing, right.
Speaker 2: It deals with data streams and passing data along in
Speaker 2: a functional paradigm, and it's meant to accommodate high volume
Speaker 2: data across highly parallelized system. So you know, at the
Speaker 2: time we went there, well I'll talk about when we
Speaker 2: went there in a second, but in retrospect, it was
Speaker 2: it was something that could handle basically ten thousand x
Speaker 2: really what we needed in terms of what it was
Speaker 2: designed to handle. So just sort of on paper, it
Speaker 2: probably wasn't the best the best move in that respect,
Speaker 2: but I could also talk about the team as well
Speaker 2: and why it wasn't a great a great bit.
Speaker 6: Yeah, can you can you talk for just a minute
Speaker 6: about like what was the problem that you were trying
Speaker 6: to solve, why did you choose?
Speaker 5: Like what did you need akup for sure?
Speaker 2: Yeah? Perfect. So the problem that we're trying to solve,
Speaker 2: and you have to know a little bit about the
Speaker 2: Ibota app. So I assume all of you, all of
Speaker 2: you guys have downloaded it and are actively using it.
Speaker 4: Absolutely it's the best Apple that they used.
Speaker 2: Good.
Speaker 6: I've definitely looked at it and I was like, nope.
Speaker 2: That's funny. We did a cert. Well I'll get to
Speaker 2: that later, But basically, I bought it as a way
Speaker 2: for you to get digital coupons. You can get brands
Speaker 2: put offers in the app. You click on the offer,
Speaker 2: you show us evidence that you bought the thing that
Speaker 2: is on offer, and then I bought it. Will pay
Speaker 2: you cash back, they'll put me send it to your
Speaker 2: PayPal account, give you a gift card to Amazon, whatever
Speaker 2: you want. So the problem that we're trying to solve
Speaker 2: is how do we make sure that the offers in
Speaker 2: the app don't exceed the budget that is allocated to
Speaker 2: them by the brands that put those offers in the air.
Speaker 2: And that sounds maybe like an easy problem, like there's
Speaker 2: an easy way to just say, Okay, there's five hundred
Speaker 2: thousand dollars in budget for Oreo coupons. Just divide five
Speaker 2: hundred thousand dollars by how many how much money we're
Speaker 2: giving out per coupon, and then you know, there's actually
Speaker 2: much Like it's obviously much harder than that, And in
Speaker 2: order to preserve a good user experience, we need to
Speaker 2: make sure that we're not yanking content and surprising our users.
Speaker 2: Like you would be really upset if you went to
Speaker 2: the store specifically to buy Oreos to get the coupon
Speaker 2: and then by the time you checked out, the coupon
Speaker 2: was no longer in your application. So we have to
Speaker 2: run some predictive algorithms to basically guess when we're going
Speaker 2: to run out of money and kind of slow the coupon,
Speaker 2: slow the velocity down as we approach that point.
Speaker 4: Oh dear, that's that's that's that's a dangerous recipe, I
Speaker 4: must add to think about. It is a vague in
Speaker 4: the UK at the moment. Is it available in Canada.
Speaker 2: We are only in the United States right now. Sorry.
Speaker 4: Yeah, So in my defense, this looks like the perfect
Speaker 4: storm for me because you've both got the kind of
Speaker 4: strong financial components. So if you get it wrong, you
Speaker 4: lose money. And if you get it long, then people
Speaker 4: have wasted their time going to the shop. You know,
Speaker 4: in Britain it's so small. I can walk to the shop.
Speaker 4: You know, sometimes I just open the window and shout
Speaker 4: at them and tell them what I want. But you know,
Speaker 4: I know when I was living in the States and
Speaker 4: people maybe drive for two hours to go to Walmart.
Speaker 4: So this is this is this is quite a quite
Speaker 4: a problem you've got there.
Speaker 2: Yeah, it was. It's interesting that you say that, because
Speaker 2: right when I joined the company, I was kind of
Speaker 2: put on the team. I was actually giving you my
Speaker 2: tech body, my mentor. When I joined the company, this
Speaker 2: was his project, and as someone knew in the company,
Speaker 2: it was it was very it was really overwhelming problem
Speaker 2: space because basically, these campaigns and application almost have like a
Speaker 2: physical momentum to them. So if you imagine trying to
Speaker 2: stop a moving train, you can't just hit the brakes
Speaker 2: and expect it to stop on a dime. You have
Speaker 2: to apply the brakes over some distance to slow the
Speaker 2: train down. And that's really how the content in the
Speaker 2: application is modeled. And it's I'm not a physics person,
Speaker 2: and so it's really confusing.
Speaker 6: So all right, so this seems a slightly I mean,
Speaker 6: it's a it's a twist right on your classical inventory problem, right,
Speaker 6: is what it kind of sounds like. So the way
Speaker 6: that AKA came into this is is what you were taking,
Speaker 6: like what all these requests from users and trying to
Speaker 6: decide whether or not you're going to I guess give
Speaker 6: them the coupon or not or something.
Speaker 2: Yeah, that's that's that's sort of right. So we have
Speaker 2: an event based architecture where our system and for the
Speaker 2: for folks who are familiar with that, I mean is
Speaker 2: that basically your system publishes events, which is data that
Speaker 2: signified that something has happened of interest in the system.
Speaker 2: So maybe, like you know, shopping cart loaded is an
Speaker 2: event that you might have in like a typical inventory
Speaker 2: space or something like that. So we have events that
Speaker 2: we're interested in, like content awarded, which means that John
Speaker 2: went to the store, submitted or received through the app,
Speaker 2: and got cash back. So the content has been awarded,
Speaker 2: So we listen for those events in order to keep
Speaker 2: track in real time of how much budget is being used,
Speaker 2: and we basically track that over time to make a
Speaker 2: rough prediction about how fast things are moving. And so AKA, sorry, John,
Speaker 2: to get back to your original question, AKA is really
Speaker 2: good at streaming data, at streaming large amounts, high volume data.
Speaker 2: And I bought a we'll get on the order of
Speaker 2: several hundred thousand of these content aword events per day,
Speaker 2: which seems like a lot, but it's actually much lower
Speaker 2: than I think what AKA can kind of deal with
Speaker 2: or is designed to deal with out of the box.
Speaker 5: Yeah did you so?
Speaker 6: Did you consider alternatives? Did you reject them for various reasons?
Speaker 6: And and I guess so. Obviously AKA didn't work out
Speaker 6: for you, so you must have picked something else that
Speaker 6: you like better. And how did you arrive at that
Speaker 6: new thing? And what was that? If that makes some sense?
Speaker 2: Yeah? Perfect, So yeah we we Basically this comes down
Speaker 2: to some team issues again and not not an issue
Speaker 2: with OCCA. So the team issue was basically that at
Speaker 2: the start of this project, we scaled up our team,
Speaker 2: like this is a lot of work we need, we
Speaker 2: need to bring in some artillery, and we brought in
Speaker 2: a new developer from outside of the company who is awesome.
Speaker 2: She's a rock star and had it was from the
Speaker 2: ad product space and with dealing with volume, much volume
Speaker 2: of streaming data at a scale much higher than than
Speaker 2: what we needed or we're going to be dealing with
Speaker 2: in it in any near near future. And you know,
Speaker 2: she was coming from I believe in AKA shop and
Speaker 2: so she joins the team. We're excited to have her.
Speaker 2: We think she's a rock star. I mean, she's a
Speaker 2: rock star. And she's like, this is a perfect use
Speaker 2: case for ACCA. We're like, okay, never heard of that,
Speaker 2: but you know, we trust you. You crushed our interview
Speaker 2: and we think you're amazing. So yeah, that's that sounds
Speaker 2: pretty good. And again this isn't a knock against Akka.
Speaker 2: I think this is just that in our company, we
Speaker 2: have a ton of infrastructure setup to support the tools
Speaker 2: that our company has sort of blessed that that are
Speaker 2: kind of frequently in use. In fact, we have like
Speaker 2: a name for that. We call it the paved road,
Speaker 2: which is like, if you're an engineer, you have a
Speaker 2: lot of autonomy and abada belt what you use but
Speaker 2: if you stay on the paved road, it should be
Speaker 2: an easy path. And akka, which is a jam you know,
Speaker 2: which used with Java or Scala typically is not on.
Speaker 2: I bought a paved road. So we had to kind
Speaker 2: of have a contentious meeting, a contentious conversation with the
Speaker 2: architects at Abada and say, no, we really we believe
Speaker 2: in in our developer and we think that she knows
Speaker 2: what she's doing, and we're ready to sort of make
Speaker 2: our bed and lay in it. And they were like, well,
Speaker 2: you know, as long as you know that, so they
Speaker 2: let us kind of they gave us just enough rope
Speaker 2: to forget how the rest of that goes to hang
Speaker 2: ourselves with.
Speaker 6: Okay, So so it sounds like you had an advocate
Speaker 6: and that's kind of how you landed on it. Where
Speaker 6: did you go after you decided that ACA was the
Speaker 6: poor choice? Like what did you end up with?
Speaker 2: Yeah? So this and this kind of gets into the
Speaker 2: next problem, which was the siloed work. So we're starting
Speaker 2: to work on building this AKA powered race car of
Speaker 2: a micro service to pull biling logic out of our
Speaker 2: rails monolith. And there's sort of two streams of work.
Speaker 2: There's the development of the ACA micro service, which is
Speaker 2: we were writing the reason Cotland, which is the JBM.
Speaker 2: It's kind of a nicer job. It's what Android apps
Speaker 2: are written in. It's actually pretty nice. So we have
Speaker 2: that work going on, and then we have to integrate
Speaker 2: that micro service sorry into the Rails monolith. So I
Speaker 2: ended up taking on a lot of the integration work
Speaker 2: and the other developer took on most of the Akka
Speaker 2: and Cotland work. So we have these sort of two
Speaker 2: very isolated pieces of work that are siloed. And then
Speaker 2: something terrible happened. And I don't blame this person at all,
Speaker 2: because I wouldn't want to be on my team anyway.
Speaker 2: She decided that she was much more interested in data
Speaker 2: engineering and she moved to a different team in the company.
Speaker 2: So and that's actually, you know a great thing about
Speaker 2: working at Ibada, Like we've got tons of really cool problems,
Speaker 2: different spaces, and people are almost encouraged to move around
Speaker 2: to find things that they're that they like and are
Speaker 2: good at. So she made this move, and now now
Speaker 2: the the problem of using a framework that none of
Speaker 2: us had experience with and that the general company didn't
Speaker 2: have a ton of experience with really became self evident
Speaker 2: because now there was there was still work left to
Speaker 2: do on the micro service. And I'm just I'm just
Speaker 2: a rails developer, Like I had to go into there
Speaker 2: and start reading Cotlan, started reading the ACA documentation and
Speaker 2: try to wrap my head around what what this whole
Speaker 2: you know, actor system meant. And it was. It was tough,
Speaker 2: and that's why I call it a big mistake. So sorry, Dave.
Speaker 1: You know from my experience too, is that, yeah, Ruby
Speaker 1: is slow, you know, there's no getting around that when
Speaker 1: you compare to a compiled language. But holy crab, is
Speaker 1: it fast too. You know, I have a production application
Speaker 1: which processes over five hundred thousand active jobs every single day,
Speaker 1: and that it does it extremely quick. I don't need
Speaker 1: it to be faster. I mean that's plenty fast on
Speaker 1: the current setup that it's on, which is two cores
Speaker 1: and four gigs of RAM, and we have two servers
Speaker 1: dedicated to the background job, so two vms. It's able
Speaker 1: to handle that load and we don't have to worry
Speaker 1: about it crashing or anything like that. So I mean
Speaker 1: that's good enough for us. Now I imagine that it
Speaker 1: would be able to double that workload before we ever
Speaker 1: ran into any kind of performance issues where we needed
Speaker 1: to start scaling up.
Speaker 2: Absolutely, Davi, and we were originally in the monolith. The
Speaker 2: way the system worked was it ran on a background
Speaker 2: scheduled job using rescue, and the scheduled job based took
Speaker 2: roughly forty five minutes to run, So it was running
Speaker 2: like kind of like a waterfall, like every ten minutes,
Speaker 2: we'd start a new job that would take forty five
Speaker 2: minutes to run. And so going from that to basically
Speaker 2: completely real time is a huge improvement. And if you know,
Speaker 2: real time for five hundred thousand events per day versus
Speaker 2: five million events per day are different things, but if
Speaker 2: you're at real time, it's already this just enormous improvement
Speaker 2: over what we were working with, which is like this
Speaker 2: forty five minute loop. So at that point, and I
Speaker 2: guess I'll say another thing before I get into like
Speaker 2: how we fixed how we fixed my mistake, This is
Speaker 2: actually getting to what Dave said, this is a case
Speaker 2: of premature optimization. We sort of didn't do the back
Speaker 2: of the envelope math well enough or didn't have a
Speaker 2: clear picture of like, okay, this is really like designed
Speaker 2: to handle like literally a thousand times more traffic than
Speaker 2: our best day. So you know, we didn't that was
Speaker 2: that was definitely a mistake, like we should have asked
Speaker 2: that question and realized, Okay, maybe maybe this is a
Speaker 2: little too much horsepower. We don't need this. It's too
Speaker 2: much trouble for what it's for, what's buying us. And
Speaker 2: on my end, also, I was imagining, you know, getting
Speaker 2: all these events we call it, So the micro service
Speaker 2: was producing like prediction events, Okay, so predicting every time
Speaker 2: a piece of content should come down and making an
Speaker 2: adjustment about when it thinks that that should happen. And
Speaker 2: so I'm imagining the monolith is listening to these prediction events,
Speaker 2: subscribe to an SNS topic or sqsq and I imagine,
Speaker 2: you know, thousands and thousands and thousands of events every second,
Speaker 2: which is way way too much. And so I was
Speaker 2: thinking of like all these clever ways to do caching
Speaker 2: and like to try to figure out when can I
Speaker 2: drop events so I don't need to hit the database,
Speaker 2: like when can I? When can I? How can I
Speaker 2: come up with these clever ways to only make round
Speaker 2: trips to the database when I really need to, And
Speaker 2: that added time to the DEVI, and it also added
Speaker 2: a ton of complexity so that when our two systems,
Speaker 2: when we were like, okay, let's turn them on, let's
Speaker 2: see what happens. The first error that comes out, because
Speaker 2: of course there's going to be an error when you
Speaker 2: first try it out. That was really hard to debug.
Speaker 2: It was really hard to understand, like is this a
Speaker 2: caching issue? Is this actually an issue with the data
Speaker 2: coming from microservice? Like it was really hard to isolate that.
Speaker 2: And then that obviously moves into the third problem. The
Speaker 2: third mistake was that we made too many changes at
Speaker 2: one time. So all these were ways that I tried
Speaker 2: to shoot myself in my foot, in my own foot
Speaker 2: while working on this very important project.
Speaker 4: Yeah makes sense. I'm not sure if I understand it.
Speaker 4: So you're saying there's kind of two separate things going
Speaker 4: on here. Firstly, you're moving to a completely new technology,
Speaker 4: so you're taking a large part of the app out,
Speaker 4: you're putting it in a micro service. It's not very
Speaker 4: micro if you're doing what seven hundred thousand things today,
Speaker 4: that's not a micro service. That's ah, this is this
Speaker 4: is this is this is a macro MicroB or medcro
Speaker 4: and the second thing you're doing is you're actually changing
Speaker 4: the whole into face. So you're saying you're going from
Speaker 4: you previously had a rescue batch job and now it's
Speaker 4: saying no, no, no, we're not going to do batching. We're
Speaker 4: going to do it all immediately.
Speaker 2: Yeah, that's a really a student observation. So we were
Speaker 2: changing structure and you were changing behavior at the same time,
Speaker 2: which is something since that point I've really tried to
Speaker 2: avoid doing, even at like the micro PR level. If
Speaker 2: I have a PR, then I'm going to push up.
Speaker 2: It's either going to add behavior, change behavior, or it's
Speaker 2: going to change the structure of the code, and I'm
Speaker 2: going to try to not do the two things at
Speaker 2: the same time. So in this case, we did the
Speaker 2: two things at the same time, and at multiple points
Speaker 2: we should have known, or we should in retrospect, we
Speaker 2: should have known to pause and verify, right and if
Speaker 2: if we couldn't verify, take that failure as feedback and iterate.
Speaker 2: And I think that that's really what really strong engineers,
Speaker 2: experienced engineers know to do, make one change at a
Speaker 2: time and verify like I have. It's I guess it's
Speaker 2: kind of like when I run our respect right or
Speaker 2: ourspect feels like supernatural to me. So every time I
Speaker 2: hit my test suite, I'm like, Okay, I expect this
Speaker 2: to happen, and like sometimes I'm right and then sometimes
Speaker 2: I'm wrong, and that's like really interesting too. But making
Speaker 2: that initial guess of what's going to happen is super important.
Speaker 2: And I think we didn't slow down and stop and say, Okay,
Speaker 2: here's our guesses to what's going to happen. Here's how
Speaker 2: we're going to verify it. And if we can't verify it,
Speaker 2: here's like the corrective action we can take.
Speaker 6: Yeah, so I think you're hitting it, like one thing
Speaker 6: that's like really important, right, So one of the things
Speaker 6: I think that make that really kind of makes you
Speaker 6: a strong engineer of sorts. Right, It is not necessarily
Speaker 6: like to avoid mistakes because shoot, for whatever reason, they
Speaker 6: keep happening to me, Right, it's are you good at
Speaker 6: cleaning up?
Speaker 5: Right?
Speaker 6: Are you good at figuring out that something's not right,
Speaker 6: something smells funny? Whatever it is? Can you go back
Speaker 6: and you know, sweep up your mess? You know, either
Speaker 6: push it across the finish line because you're close enough,
Speaker 6: or say, actually, this is the wrong path. We should
Speaker 6: actually back up, go the last intersection and go down
Speaker 6: the other way. So I mean it's definitely yeah. I
Speaker 6: mean I can speak to experiences too where I was
Speaker 6: just like, man, I really like missed all the flags
Speaker 6: and then I kept missing more flags and just kept
Speaker 6: going and and why are we surprised that we were
Speaker 6: in a bad place? Because I like ignored everything. So
Speaker 6: I hear you meant it, and and kudos for recognizing
Speaker 6: it at some point and and uh and doing something
Speaker 6: about it.
Speaker 2: Yeah, so what do we do about it? We? Unfortunately,
Speaker 2: I well, not unfortunately. It was a good It was
Speaker 2: a good learning experience. But obviously my happy place is
Speaker 2: deep in our legacy rails application finding all the pathways there.
Speaker 2: But I had to get really pushed myself yet out
Speaker 2: of my comfort zone. I had to pick up Colin. Luckily,
Speaker 2: Colin is like a super friendly language I think to
Speaker 2: get into, especially for folks who are coming from Ruby,
Speaker 2: because I think that there's like kind of like an
Speaker 2: emphasis on syntax, on like syntactic sugar on, like making
Speaker 2: the code actually like readable and look nice, whereas that's
Speaker 2: not always the case with all JBM languages. So in
Speaker 2: that case. In that sense, it was kind of friendly. Also,
Speaker 2: at I bought a we were able to like organize
Speaker 2: kind of a learning group. So there are lots of
Speaker 2: folks who are new to Ibata or new to Colin
Speaker 2: who were kind of trying to solve these similar problems.
Speaker 2: So we made a study group, we found cool online coursework.
Speaker 2: We held each other accountable for making sure that we
Speaker 2: were making progress on those things. And I started writing
Speaker 2: Colin in the micro service to try to, like you said,
Speaker 2: push across the line, like we need to get this
Speaker 2: across the line, and then we can kind of circle
Speaker 2: back and deal with some of the underlying problems. And
Speaker 2: that's important because like as engineers, like at the end
Speaker 2: of the day, we're trying to deliver value to the
Speaker 2: businesses that we that we work for and not just
Speaker 2: like trying out a new technology or like optimize what technology,
Speaker 2: what what technological solution we're we're we're implementing.
Speaker 6: So to just to clarify with what you're saying, you kept,
Speaker 6: you kept oka you just you just pushed it across
Speaker 6: the finish line despite all your your problems.
Speaker 5: Yeah, that's okay.
Speaker 2: So we had we had problems with it, and we decided,
Speaker 2: let's get this thing. We're like, we're close enough, even
Speaker 2: with the problems, even with our lack of understanding, we're
Speaker 2: close enough that a total re right right now would
Speaker 2: put us back a long way and upset the business.
Speaker 2: So let's push it across the line and then we
Speaker 2: can circle back and figure out what to do. So
Speaker 2: that was that. Delivering that value, even though it kind
Speaker 2: of felt yucky, even though we knew that there were
Speaker 2: things that were going to need to change over the
Speaker 2: long swer bought us some time and brought us some
Speaker 2: credibility in the organization.
Speaker 1: And I want to circle back to the point you
Speaker 1: made about moving too quickly or too many changes happening
Speaker 1: at one time, And I think that's something that a
Speaker 1: lot of developers might start to experience soon with the
Speaker 1: whole removal of j Query from Bootstrap five. So no,
Speaker 1: not my jQuery, no.
Speaker 5: Sorry.
Speaker 1: So the idea is that Bootstrap five no longer has
Speaker 1: a jQuery dependency. So you might plan to upgrade your
Speaker 1: rails application the CSS framework from bootsrap four to Bootstrap five,
Speaker 1: and you might say like, hey, well, why we're making
Speaker 1: this change, Why don't we go ahead and rewrite a
Speaker 1: lot of our JavaScript that was also jQuery dependent, and
Speaker 1: so I would say, instead of doing that, do one thing.
Speaker 1: Either first remove all your jQuery dependency within your application
Speaker 1: and just have jQuery be a dependency of Bootstrap, and
Speaker 1: then in another iteration, upgrade your Bootstrap version, removing then
Speaker 1: jQuery entirely. Or do advice versa where you do your
Speaker 1: Bootstrap upgrade first and then you do your j Query
Speaker 1: removal from your application as a dependency. But trying to
Speaker 1: do both side by side it's too big of a task,
Speaker 1: you know, for you know, one person or one team
Speaker 1: to do right away. I would just handle one thing
Speaker 1: at a time and moving slower. You're going to say, okay,
Speaker 1: what broke this? Was it the new Bootstrap framework.
Speaker 5: That broke this?
Speaker 1: Or was it our jQuery rewrite? So that way you're
Speaker 1: gonna be able to identify a lot more problems quicker
Speaker 1: before they are reported to you by the customer.
Speaker 2: Absolutely, I think the most experienced engines of the best
Speaker 2: engineers I've worked with, his name is Justin Hart. He
Speaker 2: was the like did the first commit on the on
Speaker 2: the monath that I work on, and every time I
Speaker 2: work with him, that is always my experience like is like,
Speaker 2: we're making this change, here's what we expect, and we're
Speaker 2: going to do it, and then we're going to verify
Speaker 2: it and then we're gonna move on. And I'm always like, oh,
Speaker 2: why don't we do this, this and this, and he's like, no, no, no,
Speaker 2: just this one thing. And it's illuminating, right, it's illuminating.
Speaker 2: I think it's helped me as a developer in the
Speaker 2: projects that I'm doing now, realizing like, Okay, this thing
Speaker 2: that I that I want to do, it's actually like
Speaker 2: four different things, and I'm going to do the first
Speaker 2: the first thing, first, verify it, and then move on
Speaker 2: from there.
Speaker 4: If you all your loved ones are affected by the
Speaker 4: removal of j Query from boots Chap, why not check
Speaker 4: out the excellent call from Jef dev chat dot tv
Speaker 4: you don't know JavaScript yet thirty day challenge available with
Speaker 4: the link in the episode showness.
Speaker 5: Were scheduled for advertisement there.
Speaker 4: Just trying to just trying to just trying to keep
Speaker 4: a ship afloat.
Speaker 6: Man, come on, uh, I was gonna so this goes
Speaker 6: along for me with.
Speaker 4: No no, no, no no no no no no no no no.
Speaker 4: We have to do jQuery now because this is really
Speaker 4: affecting me. I mean, this is this is terrible news.
Speaker 4: I literally can't write JavaScript without putting a dollar sign
Speaker 4: in front of everything.
Speaker 5: You wanted to have a.
Speaker 6: Dollar sign it you just need to assign something to
Speaker 6: the dollar sign that's off.
Speaker 4: One of the reasons I started using maybe not jQuery
Speaker 4: was because it has a little dollar sign in front
Speaker 4: of the data structure, and that kind of really makes
Speaker 4: me feel at home. Plus, you know, it's if it
Speaker 4: feels like money, you know, when I tip the dollar
Speaker 4: sign in, I feel like, yeah, this is going to
Speaker 4: make me a millionaire.
Speaker 5: Oh, I'm sorry.
Speaker 1: So I guess while we're on JavaScript, I think another
Speaker 1: premature optimization, or rather I like to call them premature
Speaker 1: de optimization is creating a new Rubyon Reil's application with React.
Speaker 1: The dashat web back equals React. Just thought through that
Speaker 1: there for the oblutory bash on React.
Speaker 5: I do like stimulus.
Speaker 6: I do feel like I do feel like we can
Speaker 6: we can let React go now.
Speaker 2: Yeah, I mean, I don't know. Have you all checked
Speaker 2: out the base camp's email Hey heay dot com.
Speaker 4: Mm hm, oh yeah, yeah, what do you think?
Speaker 2: Yeah? So I've been I've been using it, and obviously
Speaker 2: I'm a base Camp fanboy, but I think it's it's
Speaker 2: pretty amazing what what they're able to do with just
Speaker 2: HTML over the wire for the most part, and not
Speaker 2: not relying on some of the frameworks that have come
Speaker 2: out recently. Also think it's super interesting, Like the twenty
Speaker 2: twenty rebown Rails community survey, if you look the one
Speaker 2: put out by Planet are Gone. If you look at
Speaker 2: like what jobscript libraries people are using. So I was
Speaker 2: expecting that number one would be React and maybe number
Speaker 2: two would be I don't a view, or like Ember
Speaker 2: or maybe, but number one on there is jQuery, and
Speaker 2: it's like we're all in this, in this jQuery world,
Speaker 2: and it's not bad. I love your Query.
Speaker 6: So so for all of you who, along with Luke
Speaker 6: are mourning j Query, I'm telling you Stimulus go check
Speaker 6: it out.
Speaker 4: All of them don't understand it.
Speaker 5: I don't have an It's that's fair. I thought I
Speaker 5: found it pretty easy.
Speaker 6: I thought it gave me everything that I felt like
Speaker 6: I was getting from j Querry before, which is a
Speaker 6: thing happened on the page. And because that thing happened,
Speaker 6: I mean Jesse talk about event based architecture here. Come on,
Speaker 6: this is exactly what JavaScript is, right and jQuery especially right.
Speaker 6: Stimulus is exactly that event happens. Do something else because
Speaker 6: this thing happened, right, Just saying if you love that
Speaker 6: event based architecture style stimulus is that it's just way
Speaker 6: prettier and I found it a lot easier to use,
Speaker 6: and because I don't like dollar signs, I was happy
Speaker 6: about it.
Speaker 1: Yeah, And Luke, if you want a bunch of different
Speaker 1: stimulus JS tutorials, check out your the Ruby man, I
Speaker 1: have a whole bunch on there.
Speaker 4: And it's a big plug episode. Everyone quickly plugged their thing.
Speaker 4: Do you know what, Actually, Dave, I have already checked
Speaker 4: that out and I still don't understand it, so I
Speaker 4: need to check it out again. It's definitely me, I
Speaker 4: tell you, I tell you what the problem is, right.
Speaker 4: I've been leaning on j Query for so long and
Speaker 4: I'm talking about ten years, right, And it's not just
Speaker 4: that I've got my whole arsenal of weird front end
Speaker 4: stuff that I can pull in and replacing that big
Speaker 4: long list of handy widgets I know will solve this problem.
Speaker 4: Is the is the is what I'm lacking. So yeah,
Speaker 4: basic gomb stuff. Fine, you know, yes, six all away.
Speaker 4: But if I want to do something weird, if I
Speaker 4: want to do something fancy, the whole point of stimulus
Speaker 4: is it doesn't have weird fancy stuff. It's clean.
Speaker 6: So so, now that we've decided that Jaquer is eyeing
Speaker 6: and the hopefully our morning period is almost over, it
Speaker 6: is dead.
Speaker 4: I know it's dead. I'm just finding it hard to cope.
Speaker 6: Okay, fair So, Dave, you earlier had me a little
Speaker 6: bit on this train and I kind of wanted to
Speaker 6: go back to it or whatever talking about like changing
Speaker 6: multiple things because there's so many places where we can
Speaker 6: find ourselves tricked or just like seduced into it right
Speaker 6: where it just seems like the right thing to do.
Speaker 6: So so my the thing that like I like to
Speaker 6: tell people because it made total sense to me when
Speaker 6: I first heard it, was, you know, Kep Beck's like,
Speaker 6: first you make the change easy, then you go and
Speaker 6: make the easy change, right. And the important thing about that,
Speaker 6: right is there are steps here, right, like I Yeah, dude,
Speaker 6: engineering is all about taking those like tiny steps and
Speaker 6: like you said, testing right, and we all know what
Speaker 6: happens when we change a bunch of things at once
Speaker 6: and then we all do it and then we Yeah,
Speaker 6: that's how we go down the road of mistakes.
Speaker 5: Engineering is this.
Speaker 6: It requires self discipline and whenever we don't have self
Speaker 6: discipline or whenever we you know, just break it. Because
Speaker 6: for human beings that stuff happens.
Speaker 5: Yeah.
Speaker 2: I almost feel like it's I go into like meditative
Speaker 2: state almost. It's kind of like the flow state kind of.
Speaker 2: And there are times where it's like I pick up
Speaker 2: a story and it say, Okay, yeah, this is easy.
Speaker 2: I can just kind of like half think about it
Speaker 2: and do it. And when that doesn't work, which it
Speaker 2: almost never does, I have to like get into this
Speaker 2: like very focused. I'm making this one change verifying state.
Speaker 2: It's almost like like the cartoon Avatar going into your
Speaker 2: Avatar state. If you guys know that amazing cartoon.
Speaker 5: Yes, I'm familiar with it, Luke, I.
Speaker 2: Highly recommend it.
Speaker 4: Well, it's a cool Avatar and.
Speaker 2: The Last Airbender. I think it came out like ten years.
Speaker 4: Ago, fifteen hours to go up with the face the
Speaker 4: face tattoo.
Speaker 5: It's got narrow yeah, exactly, you know exactly.
Speaker 4: What right, I will check it out.
Speaker 5: I've seen that art too, he looks like he's really angry. Exactly.
Speaker 4: He's got no hair, right, yeah, bald head. Yeah that's
Speaker 4: the guy. Yeah, all right, all right, I'm a huge
Speaker 4: anime watch too. That's a that's a dangerous gap in
Speaker 4: my knowledge.
Speaker 6: Yes, So anyway, it's like you go into the state
Speaker 6: just oh Jesse, you were you were saying, you go
Speaker 6: into the state when when the first.
Speaker 2: And unfortunately for me maybe it's because of my age
Speaker 2: or whatever, but getting into that really focused state, actually
Speaker 2: that costs something, right, It takes like a little bit out.
Speaker 2: You can't maintain that state forever. There's like a manipool
Speaker 2: almost of how long you how long for me? I
Speaker 2: can be in that extreme focused state. And I think
Speaker 2: that when there are times where I look at a problem,
Speaker 2: I'm like do it Like I almost like ask myself
Speaker 2: do I need to be in this highly focused date
Speaker 2: to get this thing accomplished? And more often than not
Speaker 2: the answer is yes, But my initial answer is no,
Speaker 2: And so I end up wasting time by not doing
Speaker 2: things as systematically as they need to get done. Am
Speaker 2: I am I one on this? Nobody else? When they
Speaker 2: when they call.
Speaker 4: I find that, Yeah, people talk about this flow state
Speaker 4: and I was.
Speaker 2: I was.
Speaker 4: I was not a believer for a while until I
Speaker 4: got something really hard to work on, you know. And
Speaker 4: it's those problems where you're the kind of limit of
Speaker 4: what you can do, when someone you're thinking, can I
Speaker 4: actually write this? That is when I find you get
Speaker 4: this kind of periods of intense concentration, and obviously you
Speaker 4: don't want to be, you know, at the edge of
Speaker 4: your ability all the time. You want to kind of
Speaker 4: line up the nice easy jobs or things tend to
Speaker 4: go disastrously wrong. But one thing I have found is
Speaker 4: that you know, when you're doing your really hard problems
Speaker 4: and you're like, can can we do it? You know,
Speaker 4: can is it possible? You always have to come back
Speaker 4: and redo it. Once you prove the as possible, then
Speaker 4: you have to hit it again, and then you come
Speaker 4: up the one. So all of the stuff I've done
Speaker 4: in the kind of state intents concentration tends to get
Speaker 4: thrown away, but it moves you forward down the old
Speaker 4: down the road, down the road.
Speaker 6: I always cause those my naive implementations, and it's totally
Speaker 6: cool to write really terrible code during naive implementation.
Speaker 5: That's not the point of it.
Speaker 6: The point is to get from having nothing to having
Speaker 6: a working thing. The important thing is that after your
Speaker 6: naive implementation, you decide, you know, decide whether it's worth
Speaker 6: refractoring when you're going to refactor all that kind of stuff.
Speaker 2: Yeah, I love that idea of a naive implementation. And
Speaker 2: I think the Aka micro service was our naive implementation.
Speaker 2: We didn't consider it a scale at all. We didn't
Speaker 2: consider the makeup of the team changing. And as a result,
Speaker 2: when we went back to fix the naive implementation, right
Speaker 2: when we wanted to get rid of the double loops
Speaker 2: or whatever, we ended up moving to Java. So from
Speaker 2: Cotton to Java and from Aka to Camel. And the
Speaker 2: reason that those tools made sense is because they were
Speaker 2: very common in our company and what we didn't have
Speaker 2: to reinvent the wheel. We weren't seeing errors for the
Speaker 2: first time in the entire company. It was sort of
Speaker 2: a knowledge base to draw from. So the current state
Speaker 2: is that we have Java, Camel, a Spring micro service
Speaker 2: talking to the rails monolith. It's very stable, it's performing
Speaker 2: super well now and yeah, I mean getting I guess
Speaker 2: a question I have is like, how can we speed
Speaker 2: up the process of getting from our naive solution to
Speaker 2: the more learned solution.
Speaker 6: I think, in my opinion, if I had an answer
Speaker 6: for that, I would be I would be a super
Speaker 6: wealthy person because in my experience, usually what happens is
Speaker 6: when you get to the point that you're calling something
Speaker 6: a naive solution, right, it's usually because you recognize that
Speaker 6: there's a problem with it, right. And one of the
Speaker 6: things that's going on, I feel like is that you're
Speaker 6: sort of you're kind of burning yourself out on the problem.
Speaker 5: At that point when you when.
Speaker 6: You sort of have this realization, you're also at the
Speaker 6: point that you're kind of burning yourself out in this problem.
Speaker 6: So if you as an organization don't have the resources
Speaker 6: to sort of like swap out people, you know, bring
Speaker 6: somebody in it's fresh things like that, right, or or
Speaker 6: kind of go back to a huddle and you know,
Speaker 6: somehow become re energized. Like all the things that we
Speaker 6: can do are almost all like social interactions, not engineering interactions, right.
Speaker 6: And so if you have a culture where you can
Speaker 6: kind of deal with that, I feel like people can
Speaker 6: kind of turn around and refactor and make reasonable choices
Speaker 6: or whatever. But if you don't, like I feel like
Speaker 6: it's really hard. It's really hard for you, as somebody
Speaker 6: who just burned yourself out pushing something across the finish
Speaker 6: line that was really hard, to then turn back around
Speaker 6: and be like, I'm throwing you out and I'm going
Speaker 6: to redo this really hard problem again, right like. That's
Speaker 6: just a really hard thing to do.
Speaker 2: I think it's interesting that you say that, because at
Speaker 2: the point that we made that decision, the problem didn't
Speaker 2: seem hard anymore. At the point that we were like,
Speaker 2: you know, we can just do this in Job and Camel,
Speaker 2: we had wrapped our heads around the problem enough that
Speaker 2: we really felt like it wasn't hard anymore. And maybe
Speaker 2: that's what it takes.
Speaker 6: You also said a really important word there, which is
Speaker 6: we right. So one of the things that you did
Speaker 6: as a company to recover from this problem is you
Speaker 6: went back to this huddle and you said, hey, I
Speaker 6: made some mistakes, and then your team said that's okay, Jesse,
Speaker 6: we as a team are taking ownership for this, right like,
Speaker 6: and we as a team are going to fix this
Speaker 6: right like that. That's what a lot of that communicates
Speaker 6: right here. And when you do that, like, that's one
Speaker 6: of those re energizing moments or you know things, and
Speaker 6: then those decisions become a lot easier.
Speaker 4: In my experience, it breaks my heart every time, every
Speaker 4: every time I have a have a get commit with
Speaker 4: a kind of more lines removed than added. Oh man,
Speaker 4: those are the proudest, is right. I know some people
Speaker 4: like to take the code out and they're like, oh,
Speaker 4: I go knocked out. You know, loads of that repeated code.
Speaker 4: I just think, oh, why couldn't I get it right
Speaker 4: the first time?
Speaker 2: That's interesting. The person who taught me how to code,
Speaker 2: one of the first pieces of advice that he gave
Speaker 2: me was that you should write a lot of code
Speaker 2: and throw a lot of code out, and that's how
Speaker 2: you'll get better at coding.
Speaker 4: That is very very sensible advice, very sensible advice, and
Speaker 4: by that standard, I've got amazing coding over the years.
Speaker 4: Do I The context is a big mistake. Now, just
Speaker 4: to be clear, this big mistake has had a happy ending.
Speaker 4: So this isn't This isn't the kind of mistake I
Speaker 4: got fired. This is big mistake. We pulled through it
Speaker 4: and it'll work down, Am I right?
Speaker 2: Thankfully? Yeah? So I think so we did a couple
Speaker 2: of things. I think that helped us. So the first is,
Speaker 2: as John said, we were open about these things. We
Speaker 2: didn't try to hide that things weren't going as well
Speaker 2: as we had wanted them to. And I think that
Speaker 2: Ibota has a pretty strong culture in the sense that
Speaker 2: we're not trying to throw people under the bus. In engineering,
Speaker 2: if something crashes, it's not who's going to get fired,
Speaker 2: It's like, Okay, how do we learn from this this
Speaker 2: this is a mistake that costs us some money. How
Speaker 2: do we make sure that that money is actually teaching
Speaker 2: us something. So that was part of it. And then
Speaker 2: I think also we did a good job of communicating,
Speaker 2: like to external stakeholders. We communicated to the finance team,
Speaker 2: who were kind of one of the main main consumers
Speaker 2: of the data that we were producing, and you know,
Speaker 2: really went through in detail, here's where we're at, here's
Speaker 2: the timeline, like up them. We were checking in with
Speaker 2: them all the time and just keeping expectations in line.
Speaker 2: I think really helped us out. So even though we
Speaker 2: delivered a little bit later than I think we thought
Speaker 2: we would at the onset of the project, because we
Speaker 2: were able to communicate that we were not fired. And yeah,
Speaker 2: I mean not only that we've hired more people, we're
Speaker 2: still hiring. And if you're thinking about getting into the
Speaker 2: mobile coupon space, there's a ton of really cool problems
Speaker 2: even if you're not passionate about mobile coupons, and you
Speaker 2: might get to talk to me in the interview process,
Speaker 2: which will be fun.
Speaker 4: Speaking of interviews, I see a smooth transition there. Speaking
Speaker 4: of interviews, I understand that you have a project which
Speaker 4: you call meanas one interviewing that is I have no
Speaker 4: idea what minis ones aunds well, I see a lot
Speaker 4: of people saying it in a rib community, but I
Speaker 4: just just not It's one of these weird in jokes
Speaker 4: I think they have in a Rube community.
Speaker 1: It's so nice, I know, I know, I know, try humor, Luke,
Speaker 1: never pick up on your sarcasm.
Speaker 5: Good lord, I'm just over here like like dying everything.
Speaker 4: That's the What was it, Dave?
Speaker 1: Sorry? It was seriously? Yeah, go on, Matt's is nice
Speaker 1: and storial.
Speaker 5: We Yeah.
Speaker 4: What the best thing about Rabius community, it's a it's
Speaker 4: a really it's a really great language, and that's a
Speaker 4: really strong community, really great events, even though obviously the
Speaker 4: events this year have been a bit difficult. So how
Speaker 4: are you carrying that culture, that tradition over into the
Speaker 4: interviewing process. What's your gambit.
Speaker 2: Yeah, so maybe maybe everybody has experience or a lot
Speaker 2: of folks haven't have an experience in let's just say
Speaker 2: a not mean us one interview an interview, maybe it's
Speaker 2: a little more hostile than we'd like, And I think
Speaker 2: I think a lot of us have experienced kind of
Speaker 2: broken interviews where it feels more like the person on
Speaker 2: the other side of the table is trying to prove
Speaker 2: how much smarter they are or how much better they
Speaker 2: are coding than I am. And that's not nice. Another
Speaker 2: thing that's not nice is asking someone to do an
Speaker 2: inordinate amount of work outside of work that's not paid
Speaker 2: in the form of like a take home project. So
Speaker 2: I've done take home projects that have taken me in
Speaker 2: entire weekends, multiple days, and that's uncompensated work, and that
Speaker 2: that can bias your process against people who have outside
Speaker 2: of work commitments like families and I just you know,
Speaker 2: who don't want to be working all the time. So
Speaker 2: I thought it would make sense to kind of take
Speaker 2: the best part of the Ruby community. This idea that
Speaker 2: Matt's is nice, and so we are nice and apply
Speaker 2: it to interviewing. Let's like actually be nice to the
Speaker 2: people that we potentially could be working with. And you know,
Speaker 2: the pandemic has been terrible in so many ways, but
Speaker 2: it did offer us this opportunity to kind of dramatically
Speaker 2: rethink what our interview was.
Speaker 4: Going to look like.
Speaker 2: Because we're not coming into the office. Everything has to
Speaker 2: be remote. And basically, our HR team and our leadership
Speaker 2: were like, how how can we do this? We've only
Speaker 2: been accustomed to bringing people in, asking them super tricky
Speaker 2: things that they have to wipeboard. How can we translate
Speaker 2: this to a remote interview? And this is what I proposed,
Speaker 2: And this is like what we landed on, which is
Speaker 2: an interview not meant to trick the interviewee. It's an
Speaker 2: interview meant to simulate what the first couple of days
Speaker 2: of work is going to look like. And it's supposed
Speaker 2: to give give us as an organization, a sense of
Speaker 2: how much we would enjoy this person as a colleague,
Speaker 2: how successful they'll be. And the message that we're always
Speaker 2: trying to send is not hey, I'm so much smarter
Speaker 2: than you, because I understand ourcourage because I understand how
Speaker 2: to do whatever, this type of model, this type of
Speaker 2: data structure. The message is you're going to be successful
Speaker 2: on our team, and you're going to like working with us,
Speaker 2: and we're going to like working with you. So the
Speaker 2: way that we do that is basically by giving the
Speaker 2: person a sample of code from our domain and it's
Speaker 2: highly simplified and it's not and we ask them to
Speaker 2: just read the code and we say what is this doing?
Speaker 2: What do you like about this code? What do you
Speaker 2: not like about this code? And it's not a bug
Speaker 2: finding adventure, and we're not asking them to find where
Speaker 2: where an error is going to be secretly raised or
Speaker 2: why it tests this failing that's that's kind of beyond,
Speaker 2: like you can't really do that in a twenty or
Speaker 2: thirty minute conversation. We want to hear how this person
Speaker 2: would approach an alien code base, which is what their
Speaker 2: first task is going to be on the job. Then
Speaker 2: we present them with some data, you know, some kind
Speaker 2: of some award events from our system, and we ask
Speaker 2: them to manipulate that data and with a pretty simple
Speaker 2: alberta them. We even tell them what the algorithm is
Speaker 2: and we ask them to code it. And we say specifically,
Speaker 2: we don't care about the answer here. The answer is
Speaker 2: not interesting. We just told you what the answer is.
Speaker 2: We actually want to see what it looks like when
Speaker 2: you code. How what's your approach? Are you systematic right?
Speaker 2: Are you making guesses about what should happen and checking yourself?
Speaker 2: Those are the things that we're looking for. We're not
Speaker 2: looking for some for to see. You know, do you
Speaker 2: know this random algorithm from your computer science education that
Speaker 2: you'll never use as a as a web developer at
Speaker 2: I bought it? So those are those are the big pieces.
Speaker 2: And I'm stoked because I got invited to talk at
Speaker 2: Ruby KMF which is coming up in November, where I'm
Speaker 2: going to be kind of outlining this. You all got
Speaker 2: a preview of the content there exclusive.
Speaker 5: To Ruby Rhodes Awesome.
Speaker 6: I'm definitely curious as you guys implement this, like when
Speaker 6: you kind of come back and say, well this works
Speaker 6: and this didn't work, or or this is like this
Speaker 6: is how we sort of had to come on with
Speaker 6: a way to you know, because this is a subjective thing.
Speaker 6: This is this is how we came up with a
Speaker 6: way to to make this more objective than it originally was,
Speaker 6: you know, and that helped us like definitely interested in
Speaker 6: seeing this. I mean all of you, all of the
Speaker 6: pain points that you're talking about, right, all this homework
Speaker 6: that we have, all these you know, coding challenges that
Speaker 6: we do. I mean, we were very I feel like
Speaker 6: as a community, we talk about how where we are
Speaker 6: that they don't work, but we don't have alternatives yet,
Speaker 6: and so people continue to use them regardless because they're like, well,
Speaker 6: I don't know what to do, and the people who
Speaker 6: are lost, you know, just run back to where they
Speaker 6: came from a lot of times, right, It's like the squirrel.
Speaker 6: It's like in the middle of the street and the
Speaker 6: car's coming and it's like, shoot, I gotta go all
Speaker 6: the way back. So I kind of feel like we
Speaker 6: do a lot of that. Still, Yeah, there have been
Speaker 6: there have been various things like talked about. I remember
Speaker 6: I think I've heard various people from thomp Bottom various
Speaker 6: podcasts like talk about the fact that like they they
Speaker 6: have you come and spend like I don't remember if
Speaker 6: it was like a day or whatever, half day, but
Speaker 6: like a lot of time with them pairing, right. I
Speaker 6: know that boot camp does that as well, or base Camp. Sorry, wow,
Speaker 6: whatever base Camp does that as well or something similar.
Speaker 6: They'll they'll actually I remember hearing they had an article
Speaker 6: about they did some homework or something recently. Whatever they
Speaker 6: pay for the homework though, Yeah, that's okay. Yeah, I
Speaker 6: mean there's there's people that are like exploring around these edges,
Speaker 6: but we don't really have like a good way forward,
Speaker 6: I feel like yet.
Speaker 5: Still, so I'm super interested to see how this turns out.
Speaker 2: Yeah, I mean so far, you know, obviously hiring has
Speaker 2: been slower than typical for for Ibata, but we are
Speaker 2: still hiring, and so far, to feedback from candidates has
Speaker 2: been really interesting because you know, we'll get unsolicited emails
Speaker 2: about how much people like the process and how they
Speaker 2: hope that this is the direction that the industry goes in.
Speaker 2: A lot of folks who are interviewing with us are
Speaker 2: interviewing at other places as well. And I see this
Speaker 2: as like a competitive competitive advantage for us, Right, does
Speaker 2: this by us like ten thousand dollars in salary of
Speaker 2: five thousand dollars in salary or something like that, Like
Speaker 2: does this make our company more? You know, once you
Speaker 2: get to a certain point, there's like diminishing returns on
Speaker 2: all these different levers that a company can offer to developer.
Speaker 2: And I think that this is maybe an unexplored lever
Speaker 2: or a lever where there's a ton of root to gain.
Speaker 2: And when someone goes through our process and comes out
Speaker 2: feeling like, hey, I'm gonna kick butt there the people
Speaker 2: that are super nice, they didn't make me feel like
Speaker 2: an idiot, and they can trast that with other interview
Speaker 2: experiences that they've recently had, hopefully that will play in
Speaker 2: our favor.
Speaker 4: That is certainly not the current approach to coding interviews.
Speaker 4: I've been hearing recently a mill more people talking about
Speaker 4: make how to make coding interviews harder. If you go
Speaker 4: on something like hack and news, you know, they discuss
Speaker 4: which framework they need to insert onns of the fingernails
Speaker 4: of potential applicants. And I hear if you go to
Speaker 4: Facebook now and apply for a job, then you have
Speaker 4: to bring along your PhD in computer science. They then
Speaker 4: burn it in front of you, mix it into old coffee,
Speaker 4: and make you drink it as part of the interview process.
Speaker 4: So it's really I think it's really something that the
Speaker 4: tech community needs. We have been doing a few episodes
Speaker 4: about trying to make community more inclusive, trying to bring
Speaker 4: in people, trying to keep people in, and that sounds
Speaker 4: like a really great idea for talk. You know, why
Speaker 4: don't we be be nicer to people, try and get
Speaker 4: the most out of them during interviews and stead of
Speaker 4: subjecting them to torture.
Speaker 2: I love that image of shoving a framework under someone's fingernails.
Speaker 2: Luke Right. The thing that I noticed, and I've done
Speaker 2: a lot of interviews. I have done I would say
Speaker 2: easily two hundred interviews over the past two years. We
Speaker 2: did a ton of firing, so it's crazy. And the
Speaker 2: thing that I noticed is that when we were asking
Speaker 2: people to wipeboard right to answer these kind of tougher
Speaker 2: algorithm algorithm questions, I didn't I didn't always see a
Speaker 2: one to one correlation between the people who crush that
Speaker 2: kind of question and the people who I really enjoy
Speaker 2: working with in either direction. Right, they're false positives and
Speaker 2: false negatives, and I just found there to be way
Speaker 2: too much noise in that type of diagnostic tool for
Speaker 2: me to really make a strong decision that I felt
Speaker 2: confident with that proved out over time, and I'm hoping
Speaker 2: that this is a remedy for that.
Speaker 6: Yeah, I mean you always have to check to make
Speaker 6: sure that whatever test you're giving, right, like is that
Speaker 6: you have to decide what you're testing for and you
Speaker 6: have to say, is the question that I'm asking actually
Speaker 6: testing for the thing that I'm testing for? And too
Speaker 6: often we jump to this conclusion that, oh, yeah, this
Speaker 6: sort of this sort of is related. Therefore it'll let
Speaker 6: me know when really you're just you're making a subjective
Speaker 6: judgment call again and calling it objective.
Speaker 5: So yep, it is.
Speaker 6: It is a thing that's very painful, and maybe maybe
Speaker 6: we should we should focus a little bit on this
Speaker 6: in the future and have a more in depth chat
Speaker 6: about this. I feel like that would be an interesting subject.
Speaker 2: Yeah, there was a rails comp talk last year. I believe.
Speaker 2: The guy's name is Eric. I'll throw it in the
Speaker 2: chat in the second. He's from test Double and he
Speaker 2: talked about test doubles like a rail when you consultancy
Speaker 2: based on Columbus, I believe, And he talked about their
Speaker 2: hiring process and more in the sense of our goal
Speaker 2: is to build a process that lets candidates show off
Speaker 2: their strengths and as opposed to helping us find their weaknesses.
Speaker 2: And that's how really informed our process that I bought it.
Speaker 2: We're trying to find people's strengths and trying to figure
Speaker 2: out where they're going to be able to make the
Speaker 2: most impact in our company.
Speaker 5: Yeah, I have.
Speaker 6: I have gone for the past, like I don't know,
Speaker 6: four or five rails comps. I've gone to basically every
Speaker 6: single talk that was sort of like related to the
Speaker 6: subject I do. I care about it quite a bit,
Speaker 6: and I just kind of hope that we get to
Speaker 6: a better place. It's Yeah, like I said earlier, my
Speaker 6: feelings on this I think are basically what I said
Speaker 6: earlier that I think that a lot of people are
Speaker 6: thinking about this and I just don't think there's a
Speaker 6: clear answer yet. But I'm super interested at like everything
Speaker 6: that people are trying experiment. It's kind that's how I
Speaker 6: think we're going to get a little better.
Speaker 1: Yeah. Well, hey, it looks like we're coming up on
Speaker 1: the hour, so we need to move things along. Jesse.
Speaker 1: If people want to get in touch with you, where
Speaker 1: should they go online?
Speaker 2: And look, I would say you can find me on Twitter.
Speaker 2: I tweet from Planet Efficacy, So if you can, if
Speaker 2: you can spell that you can find me and you'll
Speaker 2: find a lot of interesting movie content, political outrage, and
Speaker 2: Vermont progressive rock on that Twitter stream.
Speaker 1: Awesome. Well, let's go ahead and move over into Look
Speaker 1: you want to kick.
Speaker 5: Us off, I would.
Speaker 4: My first pick is a version of Windows ten. A
Speaker 4: bit of a strange thing to pick on a podcast,
Speaker 4: but this is called Windows ten Ameliorrated Edition. I found
Speaker 4: it featured on a popular YouTube channel not long ago.
Speaker 4: And this is a version of Windows ten called a
Speaker 4: spy aware taken out, which also incredibly boosts the responsivity
Speaker 4: and performance because it's not doing any acinc network calls
Speaker 4: back home in the background, if you know what I mean.
Speaker 4: This is the first operating system I've ever found where
Speaker 4: to download it. I had to get a bit torrent
Speaker 4: link from a Telegram channel. So if you want some
Speaker 4: excitement in your operating systems, check out Windows ten Ameliorated Edition.
Speaker 4: Never seen anything like it.
Speaker 6: Just just wanted to confirm something here really quick. Look,
Speaker 6: is this a legal version of Windows ten.
Speaker 4: It's got quite a big section on legality on the website. Okay,
Speaker 4: the website does make it clear that you do need
Speaker 4: to have a Windows ten license in order to legally
Speaker 4: use the Windows ten and Media rated edition, but I
Speaker 4: mean come on bit torrently through a Telegram channel. Wow,
Speaker 4: that's that's quite anyway, It's it's quite. It's a cool
Speaker 4: little project. There's lots in the faq about their rationale
Speaker 4: behind it, and it does fly when it runs, it's
Speaker 4: really quick.
Speaker 1: I heard there were some issues with it with not
Speaker 1: being able to do certain things because it can't call
Speaker 1: home to Microsoft even some like simple rudimentary things that
Speaker 1: you would think that would be possible but just kind
Speaker 1: of breaks it.
Speaker 4: Yeah, I mean you could call it Windows ten bottomized.
Speaker 4: I mean, there's barely anything there at all.
Speaker 5: But it's it's.
Speaker 4: Oh well, it's a lot of fun.
Speaker 5: It flies doing nothing awesome.
Speaker 1: Well, John, you want to do some picks?
Speaker 6: Yeah, so I have a different pick this week too.
Speaker 6: You may by the time this this comes out. I
Speaker 6: have no idea if people are even gonna still be
Speaker 6: playing this game, but I have been playing and streaming
Speaker 6: and absolutely having a blast playing.
Speaker 5: I guess that's redundant among us.
Speaker 6: So if you're not familiar with it, if you're familiar
Speaker 6: with like Werewolf or Mafia or kind of games in
Speaker 6: that nature where you there's another game that people have
Speaker 6: like compared this to or whatever, where you like sort
Speaker 6: of have like like townsfolk, and then you have like
Speaker 6: people that are pretending to be townsfolk that are like
Speaker 6: trying to kill everybody off, and your goal is a
Speaker 6: townsfolk are to try and find the well. In this game,
Speaker 6: the imposters among you. So this is like set in
Speaker 6: space or whatever. But yeah, it's an absolute blast. I
Speaker 6: really am not sure what to say about it other
Speaker 6: than it's like kind of a great social game. Yeah,
Speaker 6: it's it's pretty awesome, so I highly recommend checking it out.
Speaker 6: I definitely have been doing a lot of streaming of
Speaker 6: it lately, so yeah, I'll.
Speaker 1: Jump in with a few picks. First pick bamboo flooring,
Speaker 1: so I love bamboo. And instead of the crummy old
Speaker 1: chair mat, the little plastic, thin plastic that I was
Speaker 1: using for a long time, I finally went to home
Speaker 1: Depot and got some bamboo flooring and I put glued
Speaker 1: that to a piece of plywood and I now use
Speaker 1: that as my floor mat on my carpeted floor, so
Speaker 1: my chair can just slide around on it. It's really
Speaker 1: really cool and it's been a life changing event here,
Speaker 1: so it's pretty awesome. And the second thing that I
Speaker 1: pick is the Elgato key light, So I got those
Speaker 1: for mass streaming set up and they make a huge difference.
Speaker 1: So you won't be able to tell on the podcast,
Speaker 1: but this is with my light's on and this is
Speaker 1: what the like it's off. Makes a really big difference
Speaker 1: in quality. So having a proper lighting on doing any
Speaker 1: kind of video work is an absolute must.
Speaker 4: Can you can you do it that again, Dave? Just
Speaker 4: so everyone can see on the podcast. Yeah, I got
Speaker 4: to say that is quite striking. Is that the same
Speaker 4: company that did the shortcut bomb?
Speaker 1: Yes, the stream deck.
Speaker 4: Yeah, that's a pretty cool thing as well.
Speaker 1: Big al galto f and Jesse, do you want to
Speaker 1: jump in with some picks?
Speaker 2: Yeah, I've got a couple of picks different categories. So
Speaker 2: first is a talk that I watched recently that maybe
Speaker 2: you all have seen already, but I recommend revisiting it.
Speaker 2: So Sandy Metz is rails com twenty nineteen keynote which
Speaker 2: is called Lucky You, and I think it's especially timely
Speaker 2: at the as we approach election season and thinking about
Speaker 2: equality and issues of inequality in our country. So I
Speaker 2: highly recommend that talk. It's amazing, as all Sandy Metz
Speaker 2: talks tend to be. Then I have when the pandemic starts,
Speaker 2: we started working remote, I made a small investment in
Speaker 2: my work from home setup that I highly highly highly recommend.
Speaker 2: I went out and I got an Ergo doos split keyboard,
Speaker 2: having some risk pain on the keyboard I was using,
Speaker 2: so I know keyboards can be a whole other podcast,
Speaker 2: But check out ergodocs. They make a really, really, really
Speaker 2: good product that's worth the price and I've been a
Speaker 2: huge fan of it since getting it. And then my
Speaker 2: final pick, there's a book that just came out. It's
Speaker 2: a guilty pleasure. It's called The Trouble with Peace by
Speaker 2: Joe Abercrombie, British gentleman, and it is in the grim
Speaker 2: dark genre of fantasy literature and I highly recommend it.
Speaker 1: Awesome. Well, thank you for those picks, and just be
Speaker 1: sure to post them into our chat section here so
Speaker 1: we can include them on the show notes. Well, Jesse,
Speaker 1: thank you for coming on today. It was a lot
Speaker 1: of fun. I love these kind of talks where we
Speaker 1: can just humble ourselves and talk about past mistake weeks
Speaker 1: and you know what we learned from them, so thanks again.
Speaker 4: This was awesome, really interesting.
Speaker 2: I listened a long time listener, first time guest, and
Speaker 2: this was This was awesome. I appreciate it.
Speaker 5: Yeah, thank you for coming right.
Speaker 1: Well, that's all for this episode everyone, Thanks for listening.
Speaker 5: Take everybody,