Speaker 1: Hey everybody, and welcome to another episode of the Ruby
Speaker 1: Rokes podcast. This week on a panel, we have Luke Stutters. Hello,
Speaker 1: we have Dave Kamia. Hey everyone, I'm Charles Maxwood from
Speaker 1: dev chat dot TV.
Speaker 2: Quick shout out about most valuable dot depth. Go check
Speaker 2: it out. We have a special guest this week, and
Speaker 2: that is Paul Zeich.
Speaker 3: Zeich, well done, thank you.
Speaker 2: Now you're here from Checker.
Speaker 1: You gave a talk at rails komf about how you
Speaker 1: broke stuff or somebody broke stuff.
Speaker 2: Do you want to just kind of give us a
Speaker 2: quick intro to who you are.
Speaker 1: And what you do, and then we'll dive in and
Speaker 1: talk about what broke and how you've figured it out.
Speaker 3: Sure.
Speaker 4: So, I've been a software engineer for about ten years. Recently,
Speaker 4: in the last year or so, transitioned into an engineering
Speaker 4: management role. But I've worked at a number of different
Speaker 4: small startups. I and joined Checker in twenty seventeen when
Speaker 4: the company was at about one hundred employees thirty engineers.
Speaker 4: Contributed as an engineer for a couple of years to
Speaker 4: our team, and then have recently transitioned, like I said,
Speaker 4: into a jerry management role at the company.
Speaker 2: Very cool.
Speaker 1: I actually have a Checker T shirt in my closet
Speaker 1: that I never wear. It's check R for those that
Speaker 1: are listening and not reading it. Yeah, So why don't
Speaker 1: you kind of te us up for this as far
Speaker 1: as yeah, what happened, what broke? Yeah, give us a
Speaker 1: sort of a preliminary timeline and explain what Checker does
Speaker 1: and why that matters.
Speaker 3: Sure.
Speaker 4: So checker Checker was founded in twenty fourteen. Daniel and Jonathan,
Speaker 4: our founders, had worked in the on demand space another
Speaker 4: company and had discovered that is very difficult containing great
Speaker 4: background checks into their onboarding process. Background checks tend to
Speaker 4: be a very important final safety step for a lot
Speaker 4: of these companies to make sure that their platform is
Speaker 4: going to be safe and secure for their customers, and
Speaker 4: so in twenty fourteen they started an automated background check company.
Speaker 4: And initially the biggest selling point was that Checker abstracted
Speaker 4: away a lot of the complexity of background check process,
Speaker 4: collecting candidate information and then executing that flow and exposing
Speaker 4: that via an API that was developed in a sinatrap.
Speaker 4: And three years later, in twenty seventeen, I just joined
Speaker 4: about four or five months, four or five months before
Speaker 4: this particular incident happened. Fast forward to that point. We're
Speaker 4: running i'd say, a few million checks a year for
Speaker 4: a variety of different customers. Most of those customers use
Speaker 4: our API, like I said before, to manage that process,
Speaker 4: and they do most of the collection and interface with
Speaker 4: the candidate on their side in their own application.
Speaker 2: Oh that's interesting.
Speaker 1: Yeah, I think a lot of the background check portals
Speaker 1: that I've seen they yeah, they're like the fully baked
Speaker 1: portal instead of yeah, being a background service that somebody
Speaker 1: else can integrate into their own app.
Speaker 5: Did yeah exactly that ground check on me?
Speaker 4: Before this episode, I did not that there are a
Speaker 4: lot of very important guidelines in stipulations governed by the
Speaker 4: Federal Credit Reporting Act that make sure that you have
Speaker 4: to have a permissible purpose for running a background check.
Speaker 4: So in this case, most of our customers are using
Speaker 4: the permissible purpose around employment as the reason for actually
Speaker 4: running that check.
Speaker 1: Well, that's no fun, I know, right, I want to
Speaker 1: know everybody's dirty secrets. Interesting, So, yeah, why don't you
Speaker 1: tell us a little bit about what went down with
Speaker 1: the app?
Speaker 2: Right?
Speaker 4: So, like I said in twenty seventeen, Checker at this
Speaker 4: point was a pretty important component of a number of
Speaker 4: customers onboarding process. But we'd started off small and things
Speaker 4: grew quickly in a lot of ways. We were just
Speaker 4: trying to keep the lights on and scale the system
Speaker 4: along with our customers as they continued to grow. On
Speaker 4: demand is growing a lot this time as well. So
Speaker 4: in twenty seventeen, we were doing some fairly routine changes
Speaker 4: to a data model. I wasn't directly involved with that,
Speaker 4: but we were changing something from an eteger ID to
Speaker 4: UID and the references, and there were some backfills that
Speaker 4: needed to happen, And so an engineer executed a script
Speaker 4: on a Friday afternoon, which is always a great idea,
Speaker 4: and they executed a script at about four thirty pm,
Speaker 4: probably went and grabbed something, had a little happy hour,
Speaker 4: and then headed home. And about an hour later we
Speaker 4: started to receive a few various different pages to completely
Speaker 4: unrelated teams that didn't really know what was going on
Speaker 4: in terms of this backfill, and it didn't look like
Speaker 4: anything too serious. It was just an elevated number of
Speaker 4: exceptions in our client application that does some of the
Speaker 4: candidate PII collection, and so we just decided to that
Speaker 4: team decided to snooze that decided to just kind of
Speaker 4: ignore it.
Speaker 1: Yeah, so people that aren't aware. PII is an acronym
Speaker 1: for personally identifiable information and is usually protected by a law.
Speaker 2: Thank you anyway, go ahead.
Speaker 4: So come Saturday morning, this has been going on for
Speaker 4: about twelve hours now, this exception comes in again, and
Speaker 4: at that point someone on our team actually decided to
Speaker 4: escalate that and get more stakeholders involved. We had some
Speaker 4: variety of other issues going on. We just migrated from
Speaker 4: one deployment platform to Kubernetes, and so we had some
Speaker 4: issues getting onto the cluster. There are too many of
Speaker 4: us trying to get on at the same time, so
Speaker 4: we ended up all having to actually go into the
Speaker 4: office to the physical internet to finally get in debug
Speaker 4: the issue. So we had a couple of other confounding
Speaker 4: issues come up at the same time that made the
Speaker 4: process of response even worse. So finally, this is maybe
Speaker 4: ten o'clock in the morning, ten or eleven in the morning,
Speaker 4: we finally, after being able to take a look at that,
Speaker 4: identified what they issue was and we were starting We're
Speaker 4: responding to about fifty to sixty percent of the one
Speaker 4: of the most critical end points on our system, which
Speaker 4: is to actually create requests to make a report. So
Speaker 4: after you've collected the candidate's information, you say, please execute
Speaker 4: this report so we can get that back, and that's
Speaker 4: a synchronous request that you make using our API. And
Speaker 4: when that request was failing, it was failing about forty
Speaker 4: to fifty percent of the time with a four or
Speaker 4: four response, which isn't really expected. So at that point
Speaker 4: we were finally able to pin down the issue and
Speaker 4: it came back to this script, and it turned out
Speaker 4: that when you went to create this report, we would
Speaker 4: look for create these additional subobjects called screenings, and due
Speaker 4: to the script, we had actually created an issue where
Speaker 4: validation would cause.
Speaker 3: The reports to fail to create in this EDU.
Speaker 4: Case, So there's some confounding issues with the way that
Speaker 4: we had set up the data modeling to begin with
Speaker 4: that we were trying to work around and this acception happened.
Speaker 3: But when we.
Speaker 4: Finally fix the issue, that where we shifted more into
Speaker 4: what could what actually went wrong and what were the
Speaker 4: real issues that caused us this outage gotcha.
Speaker 1: So I'm curious as you work through this, what did
Speaker 1: you add to your workflow to make sure that this
Speaker 1: doesn't happen again, because I mean, some of it's going
Speaker 1: to be technical, right, it's testing or you know, maybe
Speaker 1: you set up a staging environment or something like that,
Speaker 1: and some of it is going to be hey, when
Speaker 1: this kind of alert comes up, do this thing right,
Speaker 1: because it sounded like you did have some early indication
Speaker 1: that this happened, right.
Speaker 4: So I think the first most important thing that we
Speaker 4: did was that are really from the beginning, we have
Speaker 4: had what we call a blame this culture. I think
Speaker 4: it's a common term now in the industry. But the
Speaker 4: idea there is to really focus on learning from issues,
Speaker 4: not trying to find who made the particular mistake, and
Speaker 4: trying to look at what processes you're missing and what
Speaker 4: changes you need to make in just your code base
Speaker 4: as well that would have prevented the problem from happening,
Speaker 4: so not trying to focus on the individual mistake.
Speaker 3: That was made. So as part of that, we did a.
Speaker 4: Post mortem doc and we went through and identified things
Speaker 4: like one we should really have like a dedicated script
Speaker 4: repository that goes through a code review process. So that's
Speaker 4: that's one thing we implemented, and we made some safeguards
Speaker 4: and to address this particular issue with the data models
Speaker 4: as well. But I think for everyone that the bigger
Speaker 4: issue was really the fact that we missed the outage
Speaker 4: for so long. And we did actually have some monitoring
Speaker 4: in place for this particular issue that would have that
Speaker 4: should have page for the downtime that we were experiencing
Speaker 4: for some report creation, But it turned out that our
Speaker 4: monitors were really just not set up in the most
Speaker 4: effective way to trigger for that particular type of outage
Speaker 4: in this case is a partial outage, and that's that
Speaker 4: requires a much more sensitive monitor in order for us
Speaker 4: to detect everything. We designed the porehand was much more
Speaker 4: targeted towards a complete failure of our system.
Speaker 6: And so was this something that could have been caught
Speaker 6: by automated tests.
Speaker 4: This particular issue most likely could not have been caught
Speaker 4: by an automated test because it was a it was
Speaker 4: so so outside of the norm of what we expected
Speaker 4: data to look like. So we had particularly I mean,
Speaker 4: we had of course unit tests for everything that we
Speaker 4: were running, and we had requests specs as well. We
Speaker 4: did not have like an end to end environment set
Speaker 4: up for like a staging environment where you could run
Speaker 4: these tests and to end.
Speaker 3: But again, the data in that this particular.
Speaker 4: Case was very old, and it was essentially doing a
Speaker 4: migration where that data was in a state that wasn't
Speaker 4: anywhere in our code base at this point, So I'm
Speaker 4: not sure we could have anticipated this particular issue.
Speaker 7: What was the Fordland? Did everyone like phone up and
Speaker 7: get really angry?
Speaker 3: Oh? From a customer's perpective.
Speaker 7: Yeah, yeah, this is the the best bit of outage
Speaker 7: stories is the kind of the human cost of whoever
Speaker 7: has to answer the phone the next week code drama.
Speaker 4: Right, That's that's always one of the especially as your
Speaker 4: as your application becomes more important to customers and what
Speaker 4: your service, the impact to customers is more and more extreme.
Speaker 4: And so in this case, piezas a Friday night, it
Speaker 4: wasn't something where a lot of our customers were actively
Speaker 4: monitoring on their end as well. Fortunately, we were able
Speaker 4: to see that retries were happening, and many of our
Speaker 4: customers use a retry fallback mechanism, so they were able
Speaker 4: to just allow those to run through. But this is
Speaker 4: particularly tricky in this case because there wasn't actually like
Speaker 4: a record idea for many of these these particular responses. Fortunately,
Speaker 4: we did have a. We do keep API logs, so
Speaker 4: we were able to see exactly which requests failed for
Speaker 4: each of our customers, and so we were able to
Speaker 4: then reach out to our customer success team and they
Speaker 4: were able to start to share the impact with each
Speaker 4: of those customers pretty quickly. I will say that we've
Speaker 4: done a lot of work to make our customer communication
Speaker 4: a lot, a lot more polished since then, and that's
Speaker 4: something that we're really focusing on now as well, and
Speaker 4: just being able to give more visibility customers sooner. And
Speaker 4: one of the most important things there is when it
Speaker 4: comes to monitoring, is that you really want to be
Speaker 4: able to find the issue and be able to start
Speaker 4: to investigate it before you You don't want a customer to
Speaker 4: identify it first. You should really understand what's happening in
Speaker 4: your system before anyone else detects that issue.
Speaker 7: And I guess, but this specific not this specific product,
Speaker 7: but kind of product where your customers are consuming your API.
Speaker 7: You're also at the mercy of their implementation too, so
Speaker 7: you know, making a kind of called against you, and
Speaker 7: if that call is failing, you know, you've got to
Speaker 7: hope that their system can cope with that as well.
Speaker 4: Exactly if some of these requests are happening in the browser,
Speaker 4: or we're not set up to automatically really try, that
Speaker 4: could be a much worse impact on the customers.
Speaker 7: Can we talk about the blameless culture for a bit.
Speaker 7: This is this is a new idea. And when I
Speaker 7: was managing engineering teams, I used to have what I
Speaker 7: called the finger of blame. So I used to around
Speaker 7: I would hold up my finger in the meeting and
Speaker 7: I'd introduce the finger as the finger of blame, and
Speaker 7: then we'd work out who the finger of blame should
Speaker 7: be pointing to. Now, more often than not, of course
Speaker 7: it was me. So the finger of blame was a
Speaker 7: double edged finger, but it was it was a kind
Speaker 7: of way of you know, people take it very seriously
Speaker 7: when they mess up the kind of stuff, so you
Speaker 7: kind of have to get your get your team back
Speaker 7: on board. So it's a way of kind of lightening
Speaker 7: the mood after after that week's disaster. But a blameless culture,
Speaker 7: as you said, is a kind of more more sophisticated
Speaker 7: way of doing it instead of pointing a jovial finger
Speaker 7: at the person who messed up. What's what does that
Speaker 7: look like? I mean, you know, do you just go
Speaker 7: around telling people it's not their ful or you know,
Speaker 7: how do you implement a blameless culture in what sounds
Speaker 7: like quite a big engineering team.
Speaker 4: I think I think it starts for us with it
Speaker 4: really started with our CTO Jonathan and co founder making
Speaker 4: that a priority from pretty much day one, basically from
Speaker 4: the beginning of our process. When we've had issues or incidents,
Speaker 4: we've done a post mortem doc, we've had a process
Speaker 4: around that, and it's always been very forward facing, very
Speaker 4: much about what could we have done better, what can
Speaker 4: we improve, what are the things we should be doing
Speaker 4: going forward. So I think having that first touch point
Speaker 4: and really having that emphasis from the beginning was really
Speaker 4: important and cascades down. I think as you're building out
Speaker 4: a bigger engineering team, that's critical is to be able
Speaker 4: to just continue to build keep that culture going. And
Speaker 4: I think that's that's something a challenge to continue doing.
Speaker 4: But I think as we grow and we've been able
Speaker 4: to do that so far, so I think that was
Speaker 4: step number one. I think a second piece of it
Speaker 4: is understanding and trying to understand when it's more of
Speaker 4: a process issue versus something that someone particularly did wrong.
Speaker 4: And I think a lot of the time. I think
Speaker 4: a lot of incidents do occur because you're trying to
Speaker 4: you're trying to make different prioritization decisions, and you're trying
Speaker 4: to make sure that you anticipate things in advance or
Speaker 4: failure ferialments, and sometimes you just miss those. And those
Speaker 4: are particular cases where I think the management team needs
Speaker 4: to really take responsibility for it. It's not an individual
Speaker 4: issue that caused that particular downtime or that that was
Speaker 4: necessarily that one piece of code, and so it could
Speaker 4: be just an example. Is I mean, this is an
Speaker 4: example I think actually where we had some technical debt.
Speaker 4: We were trying to clean it up, and that was
Speaker 4: a good thing, but I think we didn't necessarily have
Speaker 4: everything in place to be able to address that technical
Speaker 4: debt effectively, and that's not necessarily one engineer's responsibility to
Speaker 4: be out in front.
Speaker 1: Of Yeah, one thing I just want to add is
Speaker 1: that I like the blameless culture just from the sense
Speaker 1: of unless somebody is either malicious, which I have never
Speaker 1: ever ever encountered, or is chronically reckless, which I've also
Speaker 1: never encountered. Right, everybody is usually trying to pull along
Speaker 1: in the same way you know, if somebody has that issue,
Speaker 1: you identify it pretty fast and you usually are able
Speaker 1: to counter it before it becomes a real problem. But yeah,
Speaker 1: just to put that together then you know, yeah, the
Speaker 1: rest of it, it's, hey, look, we're on the same team.
Speaker 1: We're all trying to get the same place. So let's
Speaker 1: talk about how we can do this better so that
Speaker 1: doesn't happen again, because next time it might be me, right,
Speaker 1: that misses a critical step. And I don't want you
Speaker 1: while fingering me either. I mean, I want to learn
Speaker 1: from it, but I you know, we don't want people
Speaker 1: walking around in fear. Instead, if somebody screws up, we
Speaker 1: want them to come forward and say, hey, I might
Speaker 1: have this up before it becomes an issue next time.
Speaker 3: Absolutely.
Speaker 4: And I think one other thing to highlight here is
Speaker 4: that when you don't have a blameless culture, folks are
Speaker 4: going to be very afraid to speak out when they
Speaker 4: do soon an issue, whether it was there they think
Speaker 4: it was their mistake or someone else's, They're not going
Speaker 4: to want to escalate that issue and make sure that
Speaker 4: it gets attention necessarily. And so one of the best
Speaker 4: side effects of having a blameless culture is that you
Speaker 4: get really engaged response and everyone's going to work together
Speaker 4: to try to address the issue. I think that even
Speaker 4: cascades down to customer communication as well, because when you're
Speaker 4: really engaged in trying to do that, then you're doing
Speaker 4: the best thing for the customers as well, because you're
Speaker 4: trying to address these issues head on and not try
Speaker 4: to find ways to kind of smooth them out under
Speaker 4: the surface.
Speaker 1: Yeah, it also and this is important, and sometimes I
Speaker 1: think people hear this and they're going to go that
Speaker 1: sounds a little scary, But you want people to take
Speaker 1: chances sometimes, right, you want people to kind of take
Speaker 1: a shot at making things better. That opens it up
Speaker 1: to them to do that, right, it's oh, well, you know,
Speaker 1: I tried this tweak on the Jinkins file, or I
Speaker 1: tried this tweak on the Kubernetes setup, or I tried
Speaker 1: this tweak on this other thing, and a lot of
Speaker 1: times those things pay off. But if you don't give
Speaker 1: people the freedom to go for it, a lot of
Speaker 1: times you're going to miss out on a lot of
Speaker 1: those benefits. And again, as long as they're not being
Speaker 1: reckless about it, right, so they're taking the steps, they're
Speaker 1: verifying it on their own system and things like that.
Speaker 1: Then you benefit much much more from people being willing
Speaker 1: to take a shot. So yeah, so with the blameless culture,
Speaker 1: I'm curious. So you get together and you start identifying
Speaker 1: what the issue is. So what does that look like
Speaker 1: then as far as figuring out what's going on, because
Speaker 1: you're not pointing fingers, but you are looking for the
Speaker 1: commit that made the problem, right you are.
Speaker 4: I think at the end of the day, you're going
Speaker 4: to try to find the root cause. Right, You're going
Speaker 4: to look for that commit, You're going to look for
Speaker 4: the law. Maybe it was a script that was logged
Speaker 4: into your logging system, whatever it is, You're going to
Speaker 4: look for that and look for the root cause. So honestly,
Speaker 4: a lot of times, you know, maybe what caused the
Speaker 4: issue from whether if it was something that was specifically
Speaker 4: run by a specific person, and they probably feel a
Speaker 4: little bit of guilt there, but there's no reason.
Speaker 3: To lay on more there.
Speaker 4: And I think everyone, like you said, feels a lot
Speaker 4: of responsibility around the work that they're doing already, so
Speaker 4: there's no reason to overemphasize that. So what that looks
Speaker 4: like is typically the team that is impacted is really
Speaker 4: going to own that post mortem, and that's one way
Speaker 4: for you to feel like you're resolving the incident or
Speaker 4: that they issue that the cost incident. So this is
Speaker 4: a definitely become a different a bit of a different
Speaker 4: process as the team is growing.
Speaker 3: When we're at thirty.
Speaker 4: I think it's a little bit easier just to know
Speaker 4: exactly who should work on those types of mitigations. It
Speaker 4: doesn't typically it's pretty isolated to a specific team. As
Speaker 4: the team has grown, growing and the system is growing,
Speaker 4: that's definitely become more of a challenge because sometimes incidents
Speaker 4: happen because different issues that multiple teams have introduced, or
Speaker 4: maybe there's multiple teams that need to be involved in
Speaker 4: the mitigation and for that in that case, we've definitely
Speaker 4: been trying to involve our post mortem process and the
Speaker 4: action items. So we have a program manager that one
Speaker 4: of her responsibilities is specifically around making sure that we
Speaker 4: are coordinating some of those efforts and meeting some essays.
Speaker 4: So we had to some additional rules and coordination around
Speaker 4: the process as we've as we've started to grow, a
Speaker 4: lot of it was just on the individual teams initially,
Speaker 4: and now as we've grown again, there's more processes involved.
Speaker 4: I think that's pretty common thing that you have to
Speaker 4: introduce as teams grow.
Speaker 7: I will say that if you've got relatives who are
Speaker 7: in the medical profession, especially if they're pathologies, even the
Speaker 7: use of the term person posts more toem makes me
Speaker 7: uncomfortable because those are no fun at all. But yeah,
Speaker 7: it's also a word that we use. So yeah, it's oh,
Speaker 7: it just makes me. Oh, it's creepy. It's all zombies.
Speaker 2: I don't know.
Speaker 7: Yeah, the post mortem brings me flashbacks to episodes of
Speaker 7: The X Files in the nineties when Dana Scuddy was
Speaker 7: taking a Navy in apart.
Speaker 1: Yeah, but it does give you a little perspective too, right,
Speaker 1: because usually in our post mortems, we're talking about what
Speaker 1: went wrong with the system, not that somebody actually died
Speaker 1: because of this, Right, I.
Speaker 5: Just got a weird brain a Right, that's what my
Speaker 5: brain thinks.
Speaker 1: That, well, some software it is life supporting, you know,
Speaker 1: a lot of the medical equipment and stuff out there.
Speaker 1: But you know, in this case, yeah, we all want
Speaker 1: to keep our jobs as well, so I mean, it's
Speaker 1: not like we can just blow it off either. So yeah,
Speaker 1: So I want to get back to the topic at hand, though,
Speaker 1: and talk a little bit about what kind of monitoring
Speaker 1: did you have before and what kind of monitoring do
Speaker 1: you have now in order to catch this kind of thing.
Speaker 4: So we use a number of different types of monitoring.
Speaker 4: At the time, we used a lot. We were pretty
Speaker 4: heavily reliant on exception tracking, and we also had some
Speaker 4: application and performance monitoring as well, commonly called APM. A
Speaker 4: couple examples of that would be something like new Relic
Speaker 4: or data Dog as a product as well. Now and
Speaker 4: then we did also use a stats D cluster that
Speaker 4: sent metrics over to data Dog, and I think we
Speaker 4: just had started using that maybe just a few months
Speaker 4: before this particular incident occurred. So, like I alluded to before,
Speaker 4: we had some We had some monitors for this particular issue,
Speaker 4: but they were pretty simplistic. They basically just looked for
Speaker 4: a minimum threshold of the number of reports that we're creating,
Speaker 4: and we had to set that threshold.
Speaker 3: To be very low over like an hour period.
Speaker 4: Because traffic is variable, you never know exactly how many
Speaker 4: reports you're going to get created. There's times a day
Speaker 4: where we received very few requests, and then there's other
Speaker 4: times where we see large spikes. So we just had
Speaker 4: very simplistic monitoring in place for some of these key
Speaker 4: metrics at that point, and at that point we were
Speaker 4: still very heavily reliant on, like I said, exception tracking
Speaker 4: using systems bug trackers like Centry that then could then
Speaker 4: alert if you had certain thresholds of number of errors
Speaker 4: over a period of time. In this particular case, exception
Speaker 4: tracking isn't very useful because we were responding with a
Speaker 4: four or four, so there wasn't actually there was an
Speaker 4: exception in the system.
Speaker 2: It was just.
Speaker 4: Automatically active record not found something like that that was
Speaker 4: then handled automatically and then responding with the four or four.
Speaker 4: So it wasn't expected behavior, but there wasn't an exception
Speaker 4: that could have been caught.
Speaker 1: Yeah, that makes sense. Somebody typed this question in it
Speaker 1: was one of the penelists. Did you get that answered?
Speaker 1: I don't know if it was Luc Dave.
Speaker 5: It was me. Just be clear, was this instant Was
Speaker 5: it a monitoring problem all an alerting problem because it
Speaker 5: sounds like an alert did go off at some point.
Speaker 6: Sounds like it was a people problem because they snooze
Speaker 6: the alert.
Speaker 4: I think this was more of a monitoring problem overall.
Speaker 4: As Dave mentioned, there there was a component where.
Speaker 3: A page was met was snoozed, But I.
Speaker 4: Think that was still a failure on our on our
Speaker 4: monitoring because in this case that was just a signal
Speaker 4: of what the true issue was. It was a downstream
Speaker 4: client application that had had a page earlier on and
Speaker 4: it wasn't It wasn't clear at all what the issue was.
Speaker 4: And I think when you're when you're developing a system
Speaker 4: for alerting, you need to have clear action items. So
Speaker 4: you need to have and that's where custom metrics, building
Speaker 4: application metrics as you as you grow become really important,
Speaker 4: having the having clear signal what wrong so that that's
Speaker 4: someone knows where to investigate. In this case, it was
Speaker 4: a client application and browser. There's a lot of noise
Speaker 4: there and I can easily understand why someone would just
Speaker 4: snooze something like that. In my opinion, it wasn't really
Speaker 4: a people issue in this particular case.
Speaker 6: Yeah, I think we've all been there before where we
Speaker 6: get an alert from whatever monitoring that we're doing and
Speaker 6: the error looks serious, but you kind of read it
Speaker 6: and like, oh, you know what, this is probably just
Speaker 6: a one off situation, and then turns out it is
Speaker 6: actually a big deal that needs to be addressed as
Speaker 6: soon as possible. So I no, I've been there before,
Speaker 6: and you know the hard times to really track this.
Speaker 6: I use Century for my air tracking and so I
Speaker 6: get email text notifications with that. And one of the
Speaker 6: nice things about it is that it'll show the number
Speaker 6: of occurrences, whether they are unique or not, so I
Speaker 6: can see if okay, this particular error is only coming
Speaker 6: from one user, or I could see we're getting one
Speaker 6: hundred errors that's coming from one hundred different users, so
Speaker 6: there's a more widespread problem. So I think, you know,
Speaker 6: definitely getting the notifications, but then having proper analytics on
Speaker 6: your errors so you can actually see the scope of
Speaker 6: how big this is can really kind of weigh in
Speaker 6: on the importance.
Speaker 5: Yeah, makes sense, I imagined, Dave.
Speaker 7: You've been through, like me, many different monitoring platforms data Dog,
Speaker 7: you said, new relics. You know what which the good
Speaker 7: Which are the good monitoring platforms for which one? So
Speaker 7: you're like, this is the platform that works really well
Speaker 7: for this API situation.
Speaker 6: I think it all depends on what you're doing. So
Speaker 6: if you have a heavy jobscript front in kind of deal,
Speaker 6: and if you also have a lot of rev backing code.
Speaker 6: I know Centric you can handle both of those situations.
Speaker 6: Other people will go with another solution. So I personally
Speaker 6: found centriy to be math flavor of choice, but you know,
Speaker 6: mile edge will vary based on what other people have.
Speaker 4: It also depends on where where you are in terms
Speaker 4: of your applications, use cases, what customers, what the customer
Speaker 4: profile looks like, how large the company has gone, how
Speaker 4: many people are supporting it. When you're early on, when
Speaker 4: you're building a new application, new product, by definition, the
Speaker 4: developers on that are going to really understand the full
Speaker 4: system very well. So cent exception tracking probably is going
Speaker 4: to be able to give you most of what you
Speaker 4: need to know in terms of understand what's going on.
Speaker 4: As the system starts to grow, and especially as you
Speaker 4: have more great teams, I think that's where things like
Speaker 4: stats D become more useful because you need to be
Speaker 4: able to set up specific use cases for core parts
Speaker 4: of your application. And I would maybe say that the
Speaker 4: bar there is maybe when you start to hit the
Speaker 4: point where you start to have a significant number of
Speaker 4: pain customers using specific features, maybe you need to start
Speaker 4: to hone in on one or two key processes that
Speaker 4: they break, it's absolutely critical that you know immediately. That's
Speaker 4: kind of the point that Checker was out in twenty seventeen.
Speaker 4: We really needed to have high intelligence, our very clear
Speaker 4: intelligence and visibility into specific parts of our system, and
Speaker 4: we're trying to move in that direction. When the sincident happened.
Speaker 4: We've continued to invest in that area going forward. I
Speaker 4: think it's become even more important as we're getting larger
Speaker 4: because there's just so many different systems that are interacting
Speaker 4: together that no one really understands the whole system at
Speaker 4: this point. And the only way to really know how
Speaker 4: the different systems are working together is maybe make sure
Speaker 4: everything's working properly. Is to have some of these custom
Speaker 4: metrics to find for specific key processes.
Speaker 5: Do you find that putting really large screens on the
Speaker 5: office wall helps make your application more reliable?
Speaker 3: That's a good question. We don't.
Speaker 4: We're all remote now, so at this point, having had
Speaker 4: an experiment with that, we did have some of those
Speaker 4: in our office. I think I've been trying to find
Speaker 4: ways to make that more visible and make metrics more
Speaker 4: visible to our team as we've been and shifted to
Speaker 4: one hundred percent remote due to the pandemic. There's also
Speaker 4: a challenge for our business in particular where sometimes things
Speaker 4: are very many of our processes are very asynchronous and
Speaker 4: they could take hours to date to fully execute, and
Speaker 4: so finding ways to short circuit and know that those
Speaker 4: things are broken can be challenging at times. So one
Speaker 4: of the things we have to do is we have
Speaker 4: to look at the data over time as well and
Speaker 4: not just look at real time metrics. So one thing
Speaker 4: I've been experimenting with is trying to create more automated
Speaker 4: reports that go into sort of a Slack channel that
Speaker 4: we can look at and so people can review that.
Speaker 4: And we've also implemented a basically a bi weekly review
Speaker 4: during our retro where we just look at our metrics
Speaker 4: and some of the longer, longer running trends so that
Speaker 4: we can see if those look correct, is there anything
Speaker 4: that's wrong, We can talk about it, see if there's
Speaker 4: things that we want to actually action on based on
Speaker 4: that review. So we're trying to find some ways to
Speaker 4: do check ins that don't require us to be all
Speaker 4: in office.
Speaker 7: The Slack channel truly is the Giant Performance monitor of
Speaker 7: twenty twenty that is that is literally what tells me
Speaker 7: whether stuff is working in a moment. I'm thinking a
Speaker 7: lot of people in the same boat. So it sounds
Speaker 7: like that you were saying that once you get to
Speaker 7: a certain stage, then the office shelf monitoring isn't really
Speaker 7: going to cut it. So you have written custom monitoring
Speaker 7: for your application.
Speaker 5: Is that correct?
Speaker 4: We have implemented what i'd consider customer tricks. We use
Speaker 4: Data Dog, so a lot of this is out of
Speaker 4: the box. You can use their implementation, but you're you're
Speaker 4: adding some code just the parts of your application. Maybe
Speaker 4: it's a maybe it's a callback on your active record model.
Speaker 4: When something is created, you send a message to a
Speaker 4: queue and then that triggers over a message into stats
Speaker 4: D that goes to data Dog. Anyways, you can do.
Speaker 4: It's a pretty lightweight to implementation in terms of what
Speaker 4: you can do. But you're adding specific events that you
Speaker 4: want to track, and then you can you can create
Speaker 4: your own monitors and alerting around those or correlations between
Speaker 4: different different events in your system. So you could potentially
Speaker 4: look at a custom metric and then look at that
Speaker 4: compared to HTTP statuses that are coming through or the
Speaker 4: latency of an endpoint and then you could correlate those
Speaker 4: two metrics as well. So there's there's some more advanced
Speaker 4: things you can do there as well if you need to.
Speaker 4: But again it's not really a lot of custom work.
Speaker 4: Is just adding some specific points in your code bas
Speaker 4: that you feel like are really important to truck. And
Speaker 4: one example of this for rails users is I believe
Speaker 4: there's something like this already set up for data Dog
Speaker 4: for sidekicks.
Speaker 3: So we instrument on a lot of our.
Speaker 4: Psydekick jobs and we can see when the log is
Speaker 4: growing on one of those cues, we can see what
Speaker 4: the average completion time is the p ninety completion time
Speaker 4: for different types of jobs. So you get a lot
Speaker 4: of visibility into your ssidechick workers and processes very easily,
Speaker 4: basically for free.
Speaker 6: And if you're going to use Slack for your error notification,
Speaker 6: now I'm not dossing that at all. No, I have
Speaker 6: a few applications that actually do that. It just triggers
Speaker 6: a Slack notification. But if you're only capturing the error
Speaker 6: message and not a stack trace along with it, then
Speaker 6: that error message is pretty much useless because it tells
Speaker 6: you you have a problem somewhere in your millions of
Speaker 6: lines of code, but we're not going to tell you
Speaker 6: where set.
Speaker 4: Just to be clear, we capture all of our our
Speaker 4: errors in Century. We do have some alerting because of Slack.
Speaker 4: But I would also want to emphasize that anything that's
Speaker 4: truly has any chance of being a serious issue should
Speaker 4: never be like an either an email or a Century
Speaker 4: alert or sorry, a Slack alert. You really should have
Speaker 4: some kind of escalation via either maybe it's text, maybe
Speaker 4: it's an actual incident response system like Peter Duty where
Speaker 4: you can have an escalation policy.
Speaker 3: For us, that's what we're using.
Speaker 4: It should have this synchronous alerting that really forces someone
Speaker 4: to look at it. You can't rely on something asynchronous
Speaker 4: like Slack in this case for serious response on issues.
Speaker 2: There's a little off topic.
Speaker 6: But you know what issue I found with that is
Speaker 6: I use my cell phone for everything. It's where I
Speaker 6: have my email, get my text messages, phone calls and
Speaker 6: all that stuff. So I would like to keep it
Speaker 6: on full volume late at night when I'm sleeping, so
Speaker 6: if a critical does arrive, then I can get notified.
Speaker 6: But my issue is that I would never get any
Speaker 6: sleep because my phone would just go off so I
Speaker 6: need to figure out some way that I can set
Speaker 6: up for a particular phone number or something to override
Speaker 6: any kind of sleep mode or whatever that I have
Speaker 6: on my phone right now, or get a different phone
Speaker 6: for that purpose. That seems a bit overkill.
Speaker 4: Doesn't You can actually do that. You can do that,
Speaker 4: I believe with at least with DIOS. You can set
Speaker 4: up an override where you snooze everything else, and then
Speaker 4: you can set up and you have to just put
Speaker 4: it in your personal contacts, whatever numbers you think you're
Speaker 4: going to receive critical notification from, and then that'll actually
Speaker 4: ring through it.
Speaker 6: All right, I need equip being lazy then and just
Speaker 6: do that.
Speaker 7: Back in twenty fifteen, I was working in the States
Speaker 7: and due to various issues, I was still responsible, thankfully
Speaker 7: for a bunch of service in the UK, and I'd
Speaker 7: gone to see a film and put my phone on silent,
Speaker 7: and of course all the servers melted halfway through Skyfall
Speaker 7: or whatever movie it was. Tom Cruise did not alert
Speaker 7: me of the impending server disaster while he was dealing
Speaker 7: with the aliens. So I came out and everyone was
Speaker 7: very upset. So I ended up writing custom alerting with
Speaker 7: a custom app. Were using the Android Automator that when
Speaker 7: it received a text message with the magic string in
Speaker 7: it would actually like manic turn turn up volume and
Speaker 7: then play the Beatles help at full volume. And that worked.
Speaker 7: That worked very well. But what didn't have, which I
Speaker 7: like on the page of duty system, is the acknowledgement
Speaker 7: so you can see, you know, yeah, I've sent the message.
Speaker 7: Has that person seen that message? And you know tapped
Speaker 7: the yes I am aware server as a melting button.
Speaker 1: Yeah, I've got I think it's the bedtime settings in iOS,
Speaker 1: and yeah, I've just told it. If it's a number
Speaker 1: in my contacts, then ring and if it's not, then don't.
Speaker 1: So yeah, it'll go off, but it'll only go off
Speaker 1: if it's yeah, if it's in my contact So yeah,
Speaker 1: then i just add whoever or whatever to my contacts
Speaker 1: and I'm set.
Speaker 6: Yeah that should work well for my use case because
Speaker 6: no one never calls me.
Speaker 2: Yeah.
Speaker 5: Right, So that's a tragic thing to say to you.
Speaker 3: Now.
Speaker 6: I had the Verizon call filter, which actually works pretty well.
Speaker 6: It's reduced the fifteen to twenty phone calls. I will
Speaker 6: get a day down to like one.
Speaker 2: Yeah, the iPhone has that feature.
Speaker 1: To where you can essentially tell it don't ring unless
Speaker 1: the numbers in my contacts.
Speaker 2: Yeah, I got.
Speaker 6: Burned by that pretty bad. One time. My wife was
Speaker 6: over the polls. She had forgotten her phone or she
Speaker 6: had lost its phone there, and because that random person
Speaker 6: wasn't in my contacts, I never got her phone call.
Speaker 2: My phone just stayed silent.
Speaker 6: So I had to disable that pretty quick.
Speaker 2: That'll teach you.
Speaker 7: Can I ask you about composite monitors, because that is
Speaker 7: a phrase I haven't heard before. I'm submiled with a
Speaker 7: rate monitor, and my understanding that is if it drops
Speaker 7: really quick, it goes off, but if it drops slowly,
Speaker 7: it doesn't come off. But what is this composite monitor?
Speaker 4: So composite monitor is basically a combination of several different
Speaker 4: metrics that you're measuring using, chained together those with and.
Speaker 3: Or or statements.
Speaker 4: So maybe referencing what I was talking about before, or
Speaker 4: you might want to have a custom metric that you're
Speaker 4: looking at and you want to look at how many
Speaker 4: of those are coming through, how many events are coming through?
Speaker 3: And then you might also want to look at, in.
Speaker 4: This case, the air rate for HTTP status, maybe how
Speaker 4: many four hundred errors you're getting relative to two hundreds.
Speaker 4: You could basically do something where you have an end
Speaker 4: statement between those two different measures and those bullying evaluations,
Speaker 4: or you could do something where you have an ore
Speaker 4: so you can say, these are basically signaling for the
Speaker 4: same type of issue that I want to alert on,
Speaker 4: but I'm going to look for these different conditions all
Speaker 4: in the same monitor.
Speaker 7: So you look at multiple different things at once. Is
Speaker 7: that so that you could combine those to kind of
Speaker 7: set effectively a much lower threshold and get higher signal
Speaker 7: to noise. So you say something like, you know, well,
Speaker 7: well allows some number of four row fast, this number
Speaker 7: of server load, this number of other errors. But if
Speaker 7: you get all three at the same time, then it
Speaker 7: triggers something different, or does it use a lower number?
Speaker 5: What's the what's the result of that?
Speaker 7: The advantage of using that logic instead of just saying,
Speaker 7: here is the minimum number of four row flaws.
Speaker 5: Here is here's the minimum, here's.
Speaker 7: The maximum number of fours, here's the maximum number of errors.
Speaker 7: How does that actually translate into a better metric?
Speaker 3: Right?
Speaker 4: So, I think I think it gives you the ability
Speaker 4: to tune things to make the potentially make something have
Speaker 4: a higher fidelity of when it alerts, so you're not
Speaker 4: getting one. You can set the thresholds actually higher and
Speaker 4: keep things. It depends how you want to use it,
Speaker 4: but you can. In this case, you could set the
Speaker 4: thresholds higher, but you could have something where it's like, well,
Speaker 4: if it's all there aren't any errors coming through, then
Speaker 4: maybe we're okay with that even though the numbers are
Speaker 4: a little bit lower, or you can do things where
Speaker 4: you can be more and again you can also tue
Speaker 4: this to be more sensitive. In this particular incident, if
Speaker 4: we had had some air monitoring around four hundreds in
Speaker 4: addition to the threshold that we had that was pretty low,
Speaker 4: I think we would have been triggered on we would
Speaker 4: have been alerted on that within maybe an hour. So
Speaker 4: you can do things there that give you more sensitivity
Speaker 4: without necessarily causing a lot more false alarms.
Speaker 3: And that's something that.
Speaker 4: You have to just be really careful with any kind
Speaker 4: of monitor on a team. Is you really need to
Speaker 4: make sure that you are not creating false alarms. I'd
Speaker 4: say it's almost as important or equally important to the
Speaker 4: sensitivity of the alarm as well, because if you're creating
Speaker 4: false alarms all the time. It's just human nature to
Speaker 4: basically start to ignore those or not really give them
Speaker 4: the review that they need. So if you're doing that
Speaker 4: all the time, you're probably going to miss something inevitably
Speaker 4: when there's actually a real issue.
Speaker 1: Makes sense, All right, we're getting close to the end
Speaker 1: of our time. Are there any other stories or examples
Speaker 1: or lessons that you want to make sure somebody listening
Speaker 1: to this gets.
Speaker 4: I just want to emphasize that this is a growing
Speaker 4: process that I think every team should go through. It's
Speaker 4: something that is going to evolve over time, and as
Speaker 4: your product becomes more important to customers and can use
Speaker 4: and grow, you need to just be constantly revealing what
Speaker 4: your approaches to this. What's going to work for brand
Speaker 4: new product, brand new startup, brand new company isn't necessarily
Speaker 4: going to be the right fit.
Speaker 3: As you continue to.
Speaker 4: Grow and something that you need to evaluate and as
Speaker 4: your product starts to be something that's really a critical
Speaker 4: service for your customers or for other teams at your company,
Speaker 4: you just need to continually set the bar higher and
Speaker 4: make sure that you're continuing to grow observability across the stock.
Speaker 1: All right, Well, one more thing before we go to picks,
Speaker 1: and that is if people want to get in contact
Speaker 1: with you, how do they find you on the internet.
Speaker 4: You're welcome to reach out to me on Twitter at Kyzeitch,
Speaker 4: or you can reach out to me on LinkedIn as well.
Speaker 2: Awesome.
Speaker 1: Yeah, we'll get links to those and we'll put them
Speaker 1: in the show notes. Let's go ahead and do some
Speaker 1: piccks then, Dave, do you want to start us off
Speaker 1: with the picks?
Speaker 2: Yeah?
Speaker 6: Sure. So went to the doctor the other week and
Speaker 6: they said I had high blood pressure, which I attribute
Speaker 6: to raising kids and them stressing me out. So I
Speaker 6: got this blood pressure monitor that syncs up with my
Speaker 6: iPhone so it keeps a historical track of it. And
Speaker 6: it's been really nice, and I guess it's accurate. I
Speaker 6: don't know. It says it's highs so I guess it's
Speaker 6: doing something. So it is the withings and it's a
Speaker 6: wireless rechargeable blood pressure monitor.
Speaker 2: Cool, Luke, how about you.
Speaker 7: Us as a really interesting Is this something you wear
Speaker 7: all the time day?
Speaker 6: No, it's just like the doctor's one where they put it,
Speaker 6: roll up your sleeve, put it on your arm and
Speaker 6: you know it starts to squeeze your arm. It's not
Speaker 6: like a wristwatch or anything. So I do it a
Speaker 6: couple of times today.
Speaker 7: Blood pressure just kidding, yeah, just just checking it, just
Speaker 7: obsessing about it. I suppose that's that's good. It's not
Speaker 7: real time. Other always, that'd be even more stressful because
Speaker 7: you'd be sitting here and it go off and say, yeah,
Speaker 7: blood pressure is going up. Get caught in the feedback loop.
Speaker 2: Cool? How about you, Luke? What are your picks?
Speaker 5: I've been fighting the code this week, Chuck.
Speaker 7: I've been building strange command line in the faces in
Speaker 7: a ruvie, and I've been using a little application which
Speaker 7: is installed by default on most de Bunty based systems
Speaker 7: called a whiptail. This is an old school text style
Speaker 7: interface so when you can't put a guy on it
Speaker 7: for various reasons, so this is kind of like it
Speaker 7: makes makes it look more professional, you know, it makes
Speaker 7: it like a real piece of software. And using this
Speaker 7: from Ruby has been a real pain because you need
Speaker 7: to do funny things with filed as scriptors to get
Speaker 7: the user data out. So it turns out a very
Speaker 7: unnice man by the name of Felix C. Stiguman has
Speaker 7: written a gem has written a gem to do it
Speaker 7: all for you in Felix. So yeah, you know, all
Speaker 7: of that work I did was totally unnecessary, and you
Speaker 7: too can build amazing old school asci looking interfaces using
Speaker 7: the gem. It's called ef and it's on GitHub on
Speaker 7: the the od fask and there's loads of really interesting
Speaker 7: utilties on the odd fast gub. If you dig in,
Speaker 7: there's an interesting low level stuff for when you want
Speaker 7: to kind of rudy yourself off on the commonline, say well,
Speaker 7: well look awesome.
Speaker 1: All right, I'm gonna throw out a couple of picks.
Speaker 1: The first one is I'm still working on this, so
Speaker 1: keep checking in most Valuable dot dev and Summit Dot
Speaker 1: Most Valuable dot Dev. I think I've mentioned it on
Speaker 1: the show before, but I'm talking to folks out there
Speaker 1: in the community. We've talked to a number of people
Speaker 1: that you've heard of, that you know well, that you're
Speaker 1: excited to hear from. But yeah, I'm going to be
Speaker 1: interviewing them and asking them what they would do if
Speaker 1: they woke up tomorrow. Was a mid level developer and
Speaker 1: felt like they didn't quite know where to go from there.
Speaker 1: So a lot of folks that's where they kind of
Speaker 1: end up right, they get to junior or mid level developer,
Speaker 1: and then it's okay, I'm proficient.
Speaker 2: Now what.
Speaker 1: Yeah, there are a lot of options, a lot of
Speaker 1: ways you can go. I'm hoping to have people come
Speaker 1: talk about blogging, podcasting, speaking at conferences and all the
Speaker 1: other stuff, and then just how to stay current, you know,
Speaker 1: how they keep up on what's going on out there.
Speaker 2: So I'm going to pick that.
Speaker 1: I've been playing a game on my phone just when
Speaker 1: I have a minute, and you know, I want to
Speaker 1: sink a little bit of time into it. It's called
Speaker 1: Mushroom Wars two. It's on the iPhone. I don't know
Speaker 1: if it's on the Android phone. Yeah, liking that, and
Speaker 1: then yeah, I'm also putting on a podcasting summit, So
Speaker 1: if you're interested in that, you can go to Podcasts
Speaker 1: podcast Growth Summit dot co and we'll have all the
Speaker 1: information up there if you listen to the Freelancer Show.
Speaker 2: The first interview I did was with Petromanos.
Speaker 1: She's in Australia, so I was talking to her in
Speaker 1: the evening here in the morning there, which is always
Speaker 1: fun with all the time zone stuff. But she talked
Speaker 1: about basically how to measure your growth and then how
Speaker 1: to use Google's tools not just to measure your growth,
Speaker 1: but then to figure out where to double down on
Speaker 1: it and get more traffic.
Speaker 2: So it was awesome.
Speaker 1: I'm talking to a bunch of other people that I've
Speaker 1: known for years and years in the podcasting space, and
Speaker 1: I'm super excited about it too. And I should probably
Speaker 1: throw out one more pick. So I'm gonna throw out
Speaker 1: gmailias that's g M E L I U S. And
Speaker 1: what it is is it's a tool. It's a CRM,
Speaker 1: but it also has like scheduling, so like schedule once
Speaker 1: or what's the other one. It allows you to set
Speaker 1: up a series of emails. It'll do automatic follow up
Speaker 1: for you and stuff like that, and so it just
Speaker 1: does a whole bunch of email automation, but it runs
Speaker 1: out of your email account, your Gmail account. That's the
Speaker 1: big nice thing about it is that you don't get
Speaker 1: downgraded by send grid or something if your emails aren't landing.
Speaker 2: And so that's another thing that I'm just really digging.
Speaker 2: So I'm going to shout out about that, Paul, what
Speaker 2: are your picks?
Speaker 4: I really enjoyed something that was in the Ruby Weekly
Speaker 4: newsletter this last week. There's a Ruby one liner cookbook,
Speaker 4: so it has a bunch of different one liners. You
Speaker 4: can actually just shout it out to and make those calls,
Speaker 4: and it explains how you can do a lot of
Speaker 4: things they do with a shell script very easily with Ruby.
Speaker 2: Awesome. Have to check that out.
Speaker 1: Sounds like a decent episode too, whether we just go
Speaker 1: through some of those and pick our favorites or whether
Speaker 1: we get whoever compiled it on. Thanks for coming, Paul.
Speaker 1: This was really helpful, and I think some folks are
Speaker 1: probably gonna either encounter this and go, yeah, I wish
Speaker 1: we were doing that, because the last time we were
Speaker 1: ended something like this it was painful, or some folks.
Speaker 1: Hopefully we'll be proactive and go out there and set
Speaker 1: things up so that they're watching things and communicating about
Speaker 1: the way that they handle issues and the way that
Speaker 1: they avoid them in the first place.
Speaker 3: It's a pleasure, all right.
Speaker 1: We'll go ahead and wrap this up and we will
Speaker 1: be back next week. Until next time, max out, everybody,