Speaker 1: What's up born, How you doing.
Speaker 2: I'm doing I'm doing pretty well. Thanks, Thanks, well cool.
Speaker 1: I cut my intro a little shorter this week because
Speaker 1: I just kind of assume, and maybe this is a
Speaker 1: bad assumption on my part, that people know which podcasts
Speaker 1: are listening to because they had to click on it,
Speaker 1: and so I felt like it was kind of redundant
Speaker 1: to say, welcome to the Adventures in DevOps podcast.
Speaker 2: You just you just did it right.
Speaker 1: That was subtle though, right, that was clever. I'm gonna
Speaker 1: just pat myself on the back for that one.
Speaker 3: Yeah, I've got I've got actually an interesting fact that
Speaker 3: I can share, so you know, we can jump into
Speaker 3: that there was an OTP provider that actually changed hands
Speaker 3: similar to the XC vulnerability and compression and Linux not
Speaker 3: too long ago. And for those of you that did OTPs,
Speaker 3: that's one time passwords, so you can think.
Speaker 2: Of like an off path that you got installed in
Speaker 2: your phone.
Speaker 3: And it's sort of ridiculous that this even happened, because
Speaker 3: if you think about how bad it is for a
Speaker 3: open source library to get co opted by a militias attacker,
Speaker 3: to have an application on your phone that is also
Speaker 3: responsible for security for two factor off codes to change hands.
Speaker 3: That provider now has access to every single one of
Speaker 3: those users two factor off and it could be even
Speaker 3: primary factor if it comes to password resets and whatnot.
Speaker 3: So that's a great way of stealing credentials. And I
Speaker 3: don't think it's an attack factor that a lot of
Speaker 3: people think about. I think they, you know, it's like whatever,
Speaker 3: it just stores my two factor codes, doesn't really matter.
Speaker 2: But now there's actually a huge problem that could come
Speaker 2: up because of that.
Speaker 1: Nice I look forward to it. I'm just excited about that.
Speaker 3: I think everyone really has to switch over to weboth
Speaker 3: and that's the truth secret. And if you don't know
Speaker 3: what that is, come and talk to me after the show.
Speaker 3: I'm happy to give everyone an earful about that.
Speaker 1: Or can we just give up on it and just
Speaker 1: everyone uses password all lower case letters for their password.
Speaker 1: I mean there's some credibility to that approach.
Speaker 2: Right, there's a whole episode there.
Speaker 1: Maybe maybe, But today's episode, we're talking about one of
Speaker 1: my favorite topics, incident response and on call management. And
Speaker 1: to chat through that topic with us, we've got Felipe Jane,
Speaker 1: the CEO of pager Ley, joining us today. Felipe welcome.
Speaker 4: Thanks.
Speaker 5: So i'mon guys, happy to join on this and you
Speaker 5: have a biggest psion response and ons right on.
Speaker 4: Cool.
Speaker 1: I feel like incident response is a learned skill, you know,
Speaker 1: and it's learned on the job under pressure when everything's
Speaker 1: going to hell, And prior to your first incident, you
Speaker 1: never even thought through that this is where your life
Speaker 1: was going to lead. So how did you end up
Speaker 1: starting a company dedicated to this incident response?
Speaker 5: It began with like when I started out my first
Speaker 5: job basically, and I was part of Amazon and in
Speaker 5: the dtail page TAM, which is friendly one of the
Speaker 5: highest sort of pages in terms of traffic difficulsider, And
Speaker 5: that was like the first hand experience. I like, my
Speaker 5: manager put me on oncoholic within sort of months of joining,
Speaker 5: and he told me like, hey, this is like the
Speaker 5: best way to learn about things, And I kidnam it.
Speaker 4: That was some serious pressure, say in.
Speaker 5: The initial initial stuff and I and since then I've
Speaker 5: always knie, yeah, this is the best way to sort
Speaker 5: of learn things because you are absolutely learning each and
Speaker 5: every bit of things.
Speaker 4: In the shortest possible amount of time.
Speaker 5: So I think that that was my first I would
Speaker 5: say interaction with the incidents and the on cord world.
Speaker 1: Well, let's be realistic there. Your manager's thought process was, actually,
Speaker 1: if this guy's going to quit, I'm going to make
Speaker 1: him quit sooner rather than later, So I'm putting him
Speaker 1: on call.
Speaker 2: Right at the beginning, it could be.
Speaker 4: I think it was probably you know, just titting.
Speaker 5: This is how you get waters And yeah, it was
Speaker 5: pretty pretty sort of a new world. Before that, I
Speaker 5: always thought like, yeah, software development is more about building
Speaker 5: stuff and you know, maybe designing it. This part is
Speaker 5: truly what you see a management or managing your maintaining
Speaker 5: your product.
Speaker 4: And that was like the first kind of interaction I had.
Speaker 1: Right and cool. So then you went from Amazon. After
Speaker 1: that you went to Disney, right yeah.
Speaker 4: Yeah, so in Amazon it was pretty interesting.
Speaker 5: The dal page and like there are they It was
Speaker 5: sort of quite a few days, especially during Fine Days
Speaker 5: and cyber Mondays and the Christmas week is always like
Speaker 5: pretty high pressure stuff and I remember each and every
Speaker 5: one of them, like in terms of the events, like
Speaker 5: even if there's like a small blade, there's like I
Speaker 5: think so many teams on a single bridge in even
Speaker 5: in different locations, and everyone just giving their status up
Speaker 5: dates each and after time and it was like it
Speaker 5: was almost like i'd say, like a bar room.
Speaker 4: Uh.
Speaker 5: People I think have now started to tagg in on
Speaker 5: called rooms as barrooms. Like everyone givings the startus updates
Speaker 5: and see update and go.
Speaker 4: On for quite a few nights. So that was I
Speaker 4: think like pretty interesting in the Disney.
Speaker 5: It was on the other way around, like our major
Speaker 5: events in Disney for the live stream. So in India
Speaker 5: we had a cricket as the major school and we
Speaker 5: used to live stream cricket and we had around twenty
Speaker 5: million concurent.
Speaker 4: Viewers also at some point.
Speaker 5: And with that scale, each and every bit of system
Speaker 5: you know, from starting from where to CDN even to
Speaker 5: a load balancer, even to a small humanities file to
Speaker 5: even to maybe our cashing systems, everything gets tested a lot.
Speaker 5: So for us in Disney, uh, that was our major priority.
Speaker 5: How to you know, manage our on calls response instead
Speaker 5: of responding on calls during the live streams because we
Speaker 5: cannot even afford to go for a minute down because
Speaker 5: we know how much takes the life events.
Speaker 4: Are for the company.
Speaker 1: Well, especially when you're talking about like live streaming cricket
Speaker 1: to Indians, because y'all take that seriously.
Speaker 5: Yeah, yeah, yeah, we you know the you know, the
Speaker 5: broadcast contracts are for you know, billions of dollars and
Speaker 5: let's say we go for Doubt for a couple of minutes.
Speaker 5: We're really using losing money at every point in time,
Speaker 5: and we can see the Twitter. You know, we're just
Speaker 5: praending on trat real like your app is down on
Speaker 5: what's happening and what's not. So you need to be
Speaker 5: very very careful you know, how to respond publicly also,
Speaker 5: and how to you know, quickly bring things up in
Speaker 5: a way that it can last at least during the live.
Speaker 5: So so those were like I'd say, the most you know,
Speaker 5: the closest I can be a customer at that point
Speaker 5: of time, and the most engineer can be at the
Speaker 5: highest pressure point. So yeah, so that was my Disney
Speaker 5: stint and those that I've started out with patiently.
Speaker 1: Right. Yeah, So I think after that you're kind of
Speaker 1: committed at this point, you yeah, you're just your career
Speaker 1: path is now incident response after those two stands.
Speaker 4: Yeah yeah, yeah.
Speaker 5: So interestingly, I think, uh, the two companies had slightly
Speaker 5: a different way of handling response, maybe because.
Speaker 4: Of the company size or team sizes.
Speaker 5: But overall, I think that the concept way like everyone
Speaker 5: was leading to a single sort of a role where
Speaker 5: we wanted to reduce the same incidents again and at
Speaker 5: least that that's.
Speaker 4: Our primary rules.
Speaker 5: And I see at what I saw in the two things,
Speaker 5: like the primary part of the incident response is the process,
Speaker 5: Like how do you sort of you know, set up
Speaker 5: the processes, how do you enable your engineers to you know,
Speaker 5: follow these processes?
Speaker 4: Uh?
Speaker 5: And and assistant I think like as an engineer, as
Speaker 5: a developer, ah, that's why we call on call as
Speaker 5: an operational part. Nobody wants to you know, spend a
Speaker 5: lot of time on it. Like everyone wants to maybe code,
Speaker 5: develop features, maybe even design, architect even blog nowadays, but
Speaker 5: on call is the last part, Like everyone wants to
Speaker 5: spend time, and so most grinted work, especially if they're
Speaker 5: like you know, work after work related to burgs or incidents.
Speaker 5: So that's where I saw a lot of common patterns
Speaker 5: across you know, incident response, even.
Speaker 4: On call management.
Speaker 5: UH is like like we there can be a lot
Speaker 5: of tools or automations or even.
Speaker 4: Assistant agents which can help the engeneers.
Speaker 5: So that's why I kind of you know started started
Speaker 5: with ag which is helping the teams to assist the
Speaker 5: incident and management.
Speaker 1: Yeah, I think that's a solid point that's often overlooked.
Speaker 1: I work a lot with early stage startups, and it's
Speaker 1: a pattern I've seen over my career, like the biggest
Speaker 1: part of incident response happens before you ever have your
Speaker 1: first incident, but because you have to talk through like
Speaker 1: what are we going to do when this actually happens,
Speaker 1: who are we gonna bring on, how are we going
Speaker 1: to carry out communications? And so Yeah, I think that's
Speaker 1: a good solid point. I like the fact that you
Speaker 1: mentioned that there's multiple ways of doing that. You know,
Speaker 1: there's not one right way.
Speaker 5: Right right, So there, I would say, like the first
Speaker 5: part is like you need to sort of realize, yeah,
Speaker 5: the time has come in my organization that we need
Speaker 5: to have this set up. I what I've usually seen
Speaker 5: it all ninety percent times it comes down from the
Speaker 5: top leadership with if you have like cetos or epes
Speaker 5: depending on the organization size, If if those books have
Speaker 5: come from a place where incident response or oncal processes had.
Speaker 4: Been in place, they bring that culture into.
Speaker 5: The company because they they have realized value over the
Speaker 5: time of these processes in companies where they have not
Speaker 5: like usually or they take a longer time to realize, yeah,
Speaker 5: we need such processes. So that's the first part is
Speaker 5: to realize like, hey, these are the processes we need
Speaker 5: so that we can at least, you know, radiuce are
Speaker 5: on our issues in a longer.
Speaker 4: Frame of time.
Speaker 5: So that's the first part. The second part is them
Speaker 5: to set up that on called roster. So on the
Speaker 5: rosters is something like uh, now that is something that
Speaker 5: which is very dependent on our too work. Some emanations
Speaker 5: wants to have a centralized on call team which kind
Speaker 5: of handles everything like let's say, even if it's like
Speaker 5: a re issue, they do it, even if it's like
Speaker 5: an infra situation, they handle it. And people have you know,
Speaker 5: different ways of setting up maybe like one one person
Speaker 5: from each team or just one person every week, and
Speaker 5: they do some some omnations have theredd on called team
Speaker 5: for each of their separate teams, so uh, and they
Speaker 5: kind of rotated weekly by I becieve monthly man.
Speaker 4: So that's the other part.
Speaker 5: The second part is to set the oncle on roster
Speaker 5: and I think then pergnission takes some time until the
Speaker 5: oncle rosters get setted and.
Speaker 4: They start you know, debugging tickets, and.
Speaker 5: After some certain amount of time then they go into
Speaker 5: the setting up the incident response part, which is the
Speaker 5: post mortems as well as you know, figuring out Hey, like,
Speaker 5: these are the sort of our general kind of you know,
Speaker 5: steps we take to solve an issue. These are the
Speaker 5: certain workflows that we do. Now let's try to streamline
Speaker 5: this both in the response as well as in the
Speaker 5: post mortem process as on the post modern analysis.
Speaker 4: So that's how that's what.
Speaker 5: We have seen orgnisation go from step A to step
Speaker 5: to last parties of post models.
Speaker 3: You said something really interesting I think, which is I
Speaker 3: haven't worked at any company so far, and even my
Speaker 3: own authors here we have process. It isn't like we
Speaker 3: don't do anything when that happens. Maybe it's because we're
Speaker 3: a tech focused or a software focused company, and that's
Speaker 3: pretty much where I've worked. But the part that you
Speaker 3: said that was really interesting for me is that software
Speaker 3: engineers don't like on call and you know, I have
Speaker 3: I want to challenge that or like, you know, I want.
Speaker 2: To live in the world where it's not a problem.
Speaker 3: It's like, why why do people not like it so much,
Speaker 3: any thoughts about that.
Speaker 5: Because people as engineers or developers, they don't consider as
Speaker 5: part of the building process. We love to build, we
Speaker 5: love to you know, architect things, but once we have
Speaker 5: done that, then we don want to sort of you know,
Speaker 5: going and you know, fix out just one tiny part
Speaker 5: of it which is actually causing the major issue, but
Speaker 5: going there fixing part of it and probably you know,
Speaker 5: just taking a blame or so people kind of have
Speaker 5: that kind of bias. Also, Hey, my my product is
Speaker 5: like a bug three.
Speaker 4: My product is you know, super gid so so.
Speaker 5: And going there fixing this as well as you know,
Speaker 5: you already have a lot of other work going on
Speaker 5: the strains in this agile world. So that's where people
Speaker 5: don't defini sort of difference, don't want to spend a
Speaker 5: lot of time on it.
Speaker 4: So that's what you think.
Speaker 3: It's not like well prioritized or rewarded. Like if you
Speaker 3: do on call work, you're not rewarded for it. If
Speaker 3: you write buglass code, you're not rewarded for it. So
Speaker 3: you know, whatever I don't want to do it, it's
Speaker 3: going to happen, and then I have to pay the
Speaker 3: I have to pay the fine because of it, and
Speaker 3: I don't I don't get the benefit.
Speaker 5: Yeah, I think like benefits and all, like cans probably
Speaker 5: be sort of be defined by the engineting managers or
Speaker 5: team leads if they want to sort of reward or
Speaker 5: they want to highlight maybe you know, like if the
Speaker 5: person has solved these many decads and these incidents, or
Speaker 5: maybe find a better way of you know, rewarding rewarding uh,
Speaker 5: developers who actually solve a lot of incidents.
Speaker 4: But yeah, I think like in.
Speaker 5: General sense, like it's not part of the building, it's
Speaker 5: only maintain, but that's the major.
Speaker 2: Like, no, no, I totally got it.
Speaker 1: Yeah, I don't like on call because it's never my code,
Speaker 1: it's always something else.
Speaker 3: Yeah, but I mean that makes me think there's something Yeah, no,
Speaker 3: I totally get I mean I feel like there's something
Speaker 3: fundamentally broken there.
Speaker 2: Like I've seen that where I worked at one.
Speaker 3: Of the previous previous jobs, that was twenty fifty engineers
Speaker 3: that were all rotating through all the same on call schedule,
Speaker 3: as if somehow code just because it was all in
Speaker 3: the monolith, if if something was broken, I somehow would
Speaker 3: magically know what was going on and say I don't
Speaker 3: know products or logistics code when I had nothing to
Speaker 3: do with the development of it, Like I like it
Speaker 3: might as well be like some for it, like I
Speaker 3: don't know Aramaic or you know, uniform to me, like
Speaker 3: I have no idea what that was going on there
Speaker 3: at all, and yet somehow I have to come in
Speaker 3: and debug or find out what the problem.
Speaker 5: Was, right, Yeah, I think in Amazon this was a
Speaker 5: case like the entire I the de deal page was
Speaker 5: kind of at that point of time like that, and
Speaker 5: and that particular sort of service the page is like
Speaker 5: maintained by more than one fifty software engineers. So like
Speaker 5: most of the times you are debugging something that you
Speaker 5: have not ruled, so and you're evening, it's a three
Speaker 5: in the night, and you're already frustrated, like you don't
Speaker 5: know what's happening. So that's why I think some bit
Speaker 5: of uscision do come from. And that's why I think
Speaker 5: like good processes can sort of somehow mitigate some of
Speaker 5: the pain points, especially like institutions.
Speaker 3: Like was the expectation at three am that software engineers
Speaker 3: should be able to log on and identify the problem
Speaker 3: and push out a fix like that seems like there's
Speaker 3: something that would never actually have like actually worked out
Speaker 3: in practice.
Speaker 5: It did like something. So like it kind of depends
Speaker 5: so what kind ofs that you have in place. So
Speaker 5: one past one part is to maybe mitigate. Sometimes medication
Speaker 5: can be done just to give a revered last running.
Speaker 5: That's one medication people don't have. But that's even that
Speaker 5: is not done. You probably need to sort of page
Speaker 5: the person who has probably added that line of code
Speaker 5: and take hit from him or her, or you maybe
Speaker 5: have certain like a time of team or something like
Speaker 5: that which can sort of you know, uh orches straight
Speaker 5: and collaborate with a lot of different on calls for
Speaker 5: different developers.
Speaker 4: To sort of mitigate and put up put a patch
Speaker 4: or fix the issues.
Speaker 5: Right, So this kind of like depends on you know,
Speaker 5: how you have set up that inst response forrocess itself.
Speaker 1: Yeah, I think that's a really good distinction to make
Speaker 1: there that like during an incident, oftentimes the primary goal
Speaker 1: is to mitigate the problem, which is different than solving
Speaker 1: the root cause of the problem. So like if if
Speaker 1: we're run an outage or an incident, like we might
Speaker 1: mitigate the problem by launching you know, fifteen more Kubernetes
Speaker 1: pods with just insane amounts of memory, just so we
Speaker 1: can ride through the problem too, we're able to figure
Speaker 1: out the root cause and test that theory and then
Speaker 1: deploy a fix for it.
Speaker 3: So just so I got you right, Well, your strategy
Speaker 3: for incident management is turning it off and back on.
Speaker 1: Again absolutely three times. Always reboot three times.
Speaker 4: Yeah, I think like legging ice stream.
Speaker 5: You know, one of the major kind of our last
Speaker 5: heart was to just put the live stream on and
Speaker 5: don't maybe don't have a pay on or something like that.
Speaker 4: So that can be one even in.
Speaker 5: Different scenarios that you can sit like, okay, even if
Speaker 5: fixing is stacking time even.
Speaker 4: Immedi maybe do something. Maybe just put that I took there.
Speaker 5: There was a certain uh forcessiarity had in peace, so
Speaker 5: I just said, like, just increase the community spots and
Speaker 5: at least that your customer is not facing that issue
Speaker 5: for the movement, then they use use that time to
Speaker 5: to actually fix the issue.
Speaker 3: How do you decide what mitigation strategy makes the most sense, Like,
Speaker 3: if you like, I feel like we're in the case
Speaker 3: of the world now where we're going to automate whatever
Speaker 3: it is. So if we have some number of failures,
Speaker 3: Do we just immediately start deploying extra pods? Do we
Speaker 3: immediately try to roll back to a previous code version?
Speaker 3: Like can we even know upfront what the right approach
Speaker 3: is there to automate? Because the last thing I want,
Speaker 3: I feel like, is someone to get online and after
Speaker 3: half an hour be like, you know what, maybe we
Speaker 3: should deploy some pods with an insane amount of memory
Speaker 3: that will solve all of our problems.
Speaker 4: Yeah, yeah, it's interesting.
Speaker 5: It was like like as an engineer, like even if
Speaker 5: you have some similarity, you would know this is the issue.
Speaker 4: Let's see, if you're seeing them mentally, then part increase
Speaker 4: makes a sense.
Speaker 5: But let's say if there's a something else you're seeing
Speaker 5: a lot of five texis, probably the part increase might
Speaker 5: not make a sense.
Speaker 4: They're probably reverting to a previous version might make a sense.
Speaker 5: So I think you need to have some bit of humanity,
Speaker 5: you know, with whatever system that you are hiding it
Speaker 5: in case you're not, then it becomes.
Speaker 4: A huge challenges. Then you probably don't know you know
Speaker 4: what to do. So I think, like I think some
Speaker 4: hulmarities need at least to have that first modication.
Speaker 1: Still, I think that alludes to an entire skill set
Speaker 1: in software engineering of how to troubleshoot.
Speaker 4: You know.
Speaker 1: It's because like debugging when you're writing code is completely
Speaker 1: different I think than troubleshooting a live system and all
Speaker 1: of its different dependencies and trying to figure out where
Speaker 1: the potential problem might be and how do you how
Speaker 1: do you get some faith in that theory and then
Speaker 1: do something to mitigate it.
Speaker 4: Yeah.
Speaker 5: Yeah, I think like a couple of techniques that we
Speaker 5: have seen and we had to kind of deployed. One
Speaker 5: is maybe do like a shadow on care like you
Speaker 5: can you do on calls, like someone is handing the
Speaker 5: on care, but let's say if they have major incidents,
Speaker 5: you shadow them. So you know, these are tools like
Speaker 5: even even I've seen sometimes people don't even know like
Speaker 5: these are dashboards that you can refer to and which
Speaker 5: will probably help you more so, uh, reaching out to
Speaker 5: people shadowing them definitely hips, especially when whenever there's an
Speaker 5: incident or an issue in a system which you have
Speaker 5: not yet touched. So that is one other bit is
Speaker 5: what we had done was we had done chaos monkey
Speaker 5: a lot. So chaos monkey like like a concept I
Speaker 5: think probably generated in Netflix engineering.
Speaker 4: That uh you kind of have like game days, uh
Speaker 4: where were what you do is like certain infrastructures, you
Speaker 4: just put it down.
Speaker 5: Let's say you put it down like a replica of
Speaker 5: let's say post is early and and see how your
Speaker 5: engineing team is performing after that.
Speaker 4: What's how much time they are just.
Speaker 5: Taken to mitigate or at least at least mitigating what's
Speaker 5: their mptity, What's how they're fixing it, how they're communicating,
Speaker 5: how they're collaborating, They're putting.
Speaker 4: The right communication to the right stakeholders at the right time.
Speaker 5: So those kind of events, those kinds of practices have helped,
Speaker 5: especially when you have not done.
Speaker 4: For a long time. So chaos lunkey help does a.
Speaker 5: Lot, especially tripping for you large events, and especially having
Speaker 5: like a proper collaboration sync with other other teams, because
Speaker 5: that's also what is needed. You do it within your team,
Speaker 5: but you also have to do it with let's say
Speaker 5: your maybe front time team or maybe your DevOps team.
Speaker 5: You need to do in sync to mitigating issue. So
Speaker 5: there are techniques to do that. So that't like right
Speaker 5: takes and right information also is like like the right
Speaker 5: education is also done for the for whoever is coming on.
Speaker 3: I often wonder how how much these things provide value.
Speaker 3: Like way along the spectrum is the right time to
Speaker 3: start implementing, say a game day where you're taking your
Speaker 3: own stuff down, or the Simian army to inject faults
Speaker 3: into your architecture or infrastructure. Like I see a lot
Speaker 3: of companies that I'm I could say, hey, you know what,
Speaker 3: that's probably not the highest value day. Like they're like,
Speaker 3: they have so many other problems that I think they
Speaker 3: should tackle first before they're ready to do that. But
Speaker 3: then on the opposite side, I'm thinking, wait, like if
Speaker 3: they did this, they may actually identify critical problems within
Speaker 3: their infrastructure that could cause them multi day downtimes or
Speaker 3: multi week downtimes, which you would have more catastrophic impacts
Speaker 3: in the long term. Uh, Like, I don't know, is
Speaker 3: that interesting? And do any thoughts of that?
Speaker 5: And like like, of course, like your company at a
Speaker 5: startup stage or initial stages.
Speaker 4: Where they maybe don't have a lot of customers or.
Speaker 5: That's uh, they don't, they won't be doing this even
Speaker 5: if I'll say, I think once you have started having
Speaker 5: multiple teams, multiple engineering teams with say different different powers,
Speaker 5: kind of a system where sometimes the information is scattered
Speaker 5: between teams and you don't know, you know, like when
Speaker 5: a when a fire is there, you don't know who
Speaker 5: to who to say, like who put that?
Speaker 4: And that's the beginning of it.
Speaker 5: And as slightly, I think the team's start to mature,
Speaker 5: and the mature I think, I think that's the right
Speaker 5: time to sort of sort of start these processes.
Speaker 1: Yeah. I think maturity is really the key word there
Speaker 1: because it takes you know, you have to have multiple
Speaker 1: layers of maturity there. You have to have a product
Speaker 1: that's mature enough to be tested, but you also also
Speaker 1: have to have maturity in your leadership team where they
Speaker 1: recognize and understand the value of saying, hey, we're not
Speaker 1: shipping new features this week, We're not shipping shiny new buttons.
Speaker 1: We're actually going to take the time and effort to
Speaker 1: see what it takes to break our system.
Speaker 5: Yeah, I think probably, But a company having a launch,
Speaker 5: launching new things or launching a new product, and maybe
Speaker 5: a week so I think people do dog footing. They
Speaker 5: can add this maybe instead of response or as a
Speaker 5: part of it, so that you know how your team
Speaker 5: would be reacting on day one days ago one in two.
Speaker 5: So I think I think it like generally sometimes even
Speaker 5: the managers or the management kind of starts to realizing,
Speaker 5: now we are spending a lot of time on these
Speaker 5: incidents itself, like our delivery for other important stuff is
Speaker 5: also getting impacted, so now we.
Speaker 4: Should find time to set some process time for.
Speaker 5: This or that, like we get you know, these incident
Speaker 5: it is so we can have a longer time for
Speaker 5: our whole features.
Speaker 4: So it's always, you know about finding that right balance.
Speaker 5: Even engineering managers or I believe have a tough time
Speaker 5: to sort of something justify a lot of spending a
Speaker 5: lot of time for these kind of things.
Speaker 4: Like it's always I think.
Speaker 5: That's I think that's probably there always conundrum they are in,
Speaker 5: you know, which which part to spend time?
Speaker 3: So uh, they have to take you I'm like stifling
Speaker 3: maybe laughing here because I feel like I have so
Speaker 3: many previous traumatic experiences of some sort of on call event.
Speaker 3: You know that's on one side the other and the
Speaker 3: other side. You said, it's like, oh, well, you know,
Speaker 3: the product manager needs to prioritize the factor, but like
Speaker 3: I want to hire that PM that actually is like
Speaker 3: you know what our insids, our incidents are are impacting
Speaker 3: our you know, future profitability, so we should actually take
Speaker 3: a look at it, improving ourselves.
Speaker 2: Like I've never heard that. I've never heard.
Speaker 3: Anything like anyone on that side say that, you know,
Speaker 3: like it's always the other way, like, oh, we don't
Speaker 3: need to worry about that is done right.
Speaker 2: We didn't we finish that coverage to push it.
Speaker 4: We don't.
Speaker 2: We don't need to improve it anymore, and think like
Speaker 2: I think the.
Speaker 5: First part is always about you know, having that right
Speaker 5: report or having some sort of information so that you
Speaker 5: could add like maybe you know if these are the
Speaker 5: these are incidents, there are recreatable incidents, these are the
Speaker 5: probably if you have some sort of a business impact
Speaker 5: to it, we show them their numbers and see like
Speaker 5: this is an impact and if you.
Speaker 4: Want to sort of reduce.
Speaker 5: That numbers of business impact, then we need this to
Speaker 5: I like, I think, I just think it's always a
Speaker 5: hard time to justify spending time on the instance.
Speaker 4: But if you have that data, that data would be
Speaker 4: any use.
Speaker 3: I mean, this is where like Dora is super successful,
Speaker 3: where we come in with meantime to resolution and change
Speaker 3: failure rate and so falling back on those statistics can
Speaker 3: be really helpful in the conversation to convince people that
Speaker 3: these aren't industry standard, that we have every single pot
Speaker 3: request we push out results in a bug in production.
Speaker 4: That's right, right right.
Speaker 1: I would imagine that most companies onboarding experience to incident
Speaker 1: response is a result of hitting a breaking point where
Speaker 1: they've had just outage after outage after outage, and finally
Speaker 1: they're like, Okay, we have to do something different, which
Speaker 1: is probably what leads them to you would that be
Speaker 1: a fair statement.
Speaker 5: Yeah, yeah, I think like one is what you say,
Speaker 5: like having outages outages, and the other part is even
Speaker 5: if let's say, if they want to sort of stream
Speaker 5: into some process, usually they see like maybe oncoll is
Speaker 5: confused what to do, or maybe they are the OnCore
Speaker 5: is need to react, or the manager doesn't know what's happening,
Speaker 5: or someone someone doesn't even know how.
Speaker 4: To report an for example. So there are different.
Speaker 5: Different aspects to it. I obviously like like the entire
Speaker 5: incident responses part of two bars, what is you know
Speaker 5: the trigger or you know, how are you creating the incident?
Speaker 5: Like what's the trigger for that? And then how are
Speaker 5: you responding to that, which is like debugging, communicating, and then.
Speaker 4: The post modern movement.
Speaker 5: So so that's where we kind of try to come in,
Speaker 5: like sort of you can stream at the entire pipeline
Speaker 5: of it, like make it as quick as possible, make
Speaker 5: it visible across maybe stakeholders, maybe support across engineering teams
Speaker 5: and having the post modern analysis processes in the least.
Speaker 5: So it like I think, like like we come in
Speaker 5: when people when teams recognize too many repetated incidents or
Speaker 5: too many of these.
Speaker 4: Stuff, and whoever is the on call is kind of feeling.
Speaker 5: Very confused for a state of things. So that's where
Speaker 5: we have seen a lot of competition onto this.
Speaker 1: Yeah, you mentioned the stakehold was there, and I think
Speaker 1: that's a really cool thing to dive into for a minute,
Speaker 1: because communication is one of the key things of incident response,
Speaker 1: and it's the one I always hated the most early
Speaker 1: on in my career because I would be in an
Speaker 1: incident and then everyone wants to know what's going on. Well,
Speaker 1: I'm working on it, damn it. But I can't work
Speaker 1: on it if you're sitting here hounded me with questions.
Speaker 1: And so I think a key part of a solid
Speaker 1: incident response plan is having a communication plan so that
Speaker 1: you can relay that information out and free up the
Speaker 1: people actually working on the incident to continue working on it.
Speaker 1: How do you recommend addressing that?
Speaker 5: Like I say, like an on call is a person
Speaker 5: who's always on fire who has to you know, mitigate
Speaker 5: the issue.
Speaker 4: I think that's the namone everyone Connie, but because of the.
Speaker 5: Environment, he needs to do a lot of things also
Speaker 5: communicated to them support also, you know what message of
Speaker 5: what's the estimated time or resolution from we communicate to manager, Hey,
Speaker 5: this is probably the impact these many users, this much
Speaker 5: subscriptions are being impacted right now. So so I that's
Speaker 5: a major pain point the on call person has, Uh,
Speaker 5: what's the way The best way is to you know,
Speaker 5: delegate a lot of these stuff or maybe have a
Speaker 5: have a system which is you know, like which is
Speaker 5: visible to the stakeholders so that they don't ping.
Speaker 4: The on call or they don't kind of you know,
Speaker 4: ask them again and again.
Speaker 5: What's one of the ways we do via page leg
Speaker 5: is we live with Slack as a major part of
Speaker 5: the incident response. So let's say we created channel Slack
Speaker 5: channel for each incident and in the Slack channel, you
Speaker 5: can see you know what's the eating Uh, what's.
Speaker 4: The business impact?
Speaker 5: Or maybe some bit of information is like something some
Speaker 5: bit of it is a by the on kore, but
Speaker 5: nobody is like asking on again in the.
Speaker 4: Game, you know what's the ETU? What's the impact? Let's
Speaker 4: say aug much wants to see they can go to
Speaker 4: the channel and see it.
Speaker 5: Customer support can see you so like like like whatever,
Speaker 5: Let's say if someone wants to send an email, no
Speaker 5: one likely they can just send all that information to emails.
Speaker 5: What if the stakeholders that the company has so whatever
Speaker 5: kind of the you know, actions on cool has to
Speaker 5: do apart from mitigation is an additional effort. And whatever
Speaker 5: tools and resources they can utilize to sort of you know,
Speaker 5: delegate an automatated would be much more helpful. Uh, And
Speaker 5: so that they so that his major sort of brain
Speaker 5: focuses always on mitigating the issue as quickly as possible.
Speaker 3: Yeah, I mean, I think having those additional things in
Speaker 3: place once you identify them to help streamline the process
Speaker 3: are super important. Like we've got uh status dashboards that
Speaker 3: we can point customers to immediately, so rather than trying
Speaker 3: to explain where the updates are going to be or
Speaker 3: how they're going to happen, and you just go to
Speaker 3: the Zurel and stuff is there. But I mean, I
Speaker 3: think also as customers of SaaS solutions, we have like
Speaker 3: an opportunity to even be nicer to companies that are
Speaker 3: having incidents. I mean, I think there's an emoji dedicated
Speaker 3: to this, hug ops right, you know when something's happening,
Speaker 3: you know, pass pass on the empathy a little bit.
Speaker 3: Like I care way more about as a customer that
Speaker 3: you tell me that you know that there's a problem
Speaker 3: that someone's looking at rather than being like, oh, we
Speaker 3: don't know, I don't I don't know what's going on,
Speaker 3: or you know, even someone's looking at it, Like I
Speaker 3: much prefer to be told oh, yeah, like we'll have
Speaker 3: an update in an hour, then oh, this is exactly
Speaker 3: what's happening at.
Speaker 2: This moment, Like I don't care about that.
Speaker 3: I want to know, you know, when's the next update
Speaker 3: going to be happening, more so than Play by Life.
Speaker 5: Right like there too also always there are two types
Speaker 5: of communication, one internally and externally.
Speaker 4: Both has to be I think that.
Speaker 5: Suddenly more because you have the state like you have
Speaker 5: the ultimate stakedness, but like like both needs to be
Speaker 5: you know, always updated.
Speaker 4: Both needs to be you know, always to the point
Speaker 4: so that like because.
Speaker 5: In any of these conditions there's any miscommindation happening, then
Speaker 5: it will you know, just prolonged instead much wrong.
Speaker 1: So it reminds me of the AWS status pages early
Speaker 1: in the days of AWS, like was always green, Like
Speaker 1: I would I would have put money that it was
Speaker 1: just a green icon there and there were no other
Speaker 1: options available because it was always green.
Speaker 5: Right, I remember at that time, I think like, uh,
Speaker 5: I usually didn't sort of had a lot of confidence
Speaker 5: on that, I like down these some other or even
Speaker 5: Twitter was much more sort of a better way to
Speaker 5: you know, there's like actually a major and those status
Speaker 5: pages were like not at all.
Speaker 4: I think, I think things have changed.
Speaker 1: But you bring up Twitter though, and that's a really
Speaker 1: good point. I mean, I think for a lot of
Speaker 1: tech oriented companies that's a primary communication channel, you know,
Speaker 1: sending out notifications on Twitter or x and relaying information
Speaker 1: that way. And also like it's kind of sad to say,
Speaker 1: but that's also a good notification method of whenever your
Speaker 1: customers think something's going wrong.
Speaker 3: Was I mean enough enough that I saw some products
Speaker 3: that specifically like we go around to social media and
Speaker 3: get the up real time status from potential users complaining
Speaker 3: because it's another source that you're not tapping into to
Speaker 3: actually let you know if customers have a problem, you know,
Speaker 3: they're not necessarily reporting it back to you.
Speaker 2: This is the report mechanism, right.
Speaker 4: I think these two kind of work.
Speaker 2: That's a good point.
Speaker 5: Yeah, And I think the companies, I think I've started
Speaker 5: to put artists feeds also, like for a longer time,
Speaker 5: and they have integrations with those feeds to their Twitter
Speaker 5: accounts or maybe some of their complements discord if they're
Speaker 5: doing a sas kind of a product or something like that,
Speaker 5: so that their customers are also updated by these platforms.
Speaker 3: I mean accuracy, though, is what you're bringing up will
Speaker 3: And I feel like there's a huge challenge there realistically
Speaker 3: to like what do you what like what makes sense
Speaker 3: to even talk about and what should be intermittent hidden
Speaker 3: failures from an internal company standpoint, Like I don't want
Speaker 3: to see Amazon just being read all the time because
Speaker 3: some node in cloud front failed one request because the
Speaker 3: connection didn't go through.
Speaker 2: Like how does that help?
Speaker 3: So I mean I feel like or yellow all the
Speaker 3: time because there's always something that's probably impossibly problem. I
Speaker 3: think a single color there is is always wrong.
Speaker 4: Right, and and that's why I didn't.
Speaker 5: I think if you see AWS, they have although the
Speaker 5: period of time they have evolved their status page earlier
Speaker 5: like now, they have actually region wise. Also I think
Speaker 5: they have also started to do for so for some
Speaker 5: of the services, they have started to do more grummar
Speaker 5: scooping as much as possible, so like, uh, that's even
Speaker 5: for Slack. Also, like earlier they used to do only
Speaker 5: for messages, you know, if you have work spaces working fine.
Speaker 5: They have not started to do for APIs. And you know,
Speaker 5: like even logging has been different, so every bit of
Speaker 5: different they have started to do so that like you
Speaker 5: don't have like a yellow for maybe just a small
Speaker 5: issue in or maybe just a small service in a
Speaker 5: small region. So if if that day page is more,
Speaker 5: if that status page is more sort of detay, then
Speaker 5: I think it probably helps to sort of give the
Speaker 5: right information.
Speaker 3: I mean, I actually think AWS went a lot further here.
Speaker 3: They have something called the Health Dashboard, which figures out
Speaker 3: what services you're actually used, I think, and how that
Speaker 3: could be impacted you and then actually have messages there,
Speaker 3: which I mean is really what we all actually care about, right,
Speaker 3: you know, is there something happening at this moment which
Speaker 3: actually affects us that could be interesting realistically if we
Speaker 3: saw a problem, does this explain it?
Speaker 4: Right? Right? Absolutely?
Speaker 1: So. One thing we haven't talked about a lot is
Speaker 1: the post mortem, and I feel like that's all just
Speaker 1: like that is as much work as doing the incident
Speaker 1: response itself, but sometimes it gets overlooked because it's no
Speaker 1: longer a priority. Like once the incident is no longer
Speaker 1: an incident, you have to just be disciplined enough to
Speaker 1: run through the post mortem process. How do you how
Speaker 1: do you approach that?
Speaker 4: I think the post mortem is like I'll say, like
Speaker 4: a chain your top in terms of the look like
Speaker 4: you have.
Speaker 5: You know, you're doing us keep less right, You're going
Speaker 5: to the incident and maybe fix it also, but now
Speaker 5: you need it like a Disney we used to have
Speaker 5: like a day Maximat's idea or even there's a need
Speaker 5: to you know, come with that most modern document because
Speaker 5: they were kind of very bullish on that, like we
Speaker 5: want to know what cause issues, soctly we can fix
Speaker 5: it r tomorre itself. So that like, like I would say,
Speaker 5: like that's where that's where the gap is, Like, that's
Speaker 5: where a lot of people drop where they don't want
Speaker 5: to do that work work, especially after a grueling period
Speaker 5: of you know, incident resolving process.
Speaker 4: So but I think it's just about.
Speaker 5: More of an education part or more of you know,
Speaker 5: realizing what you have learned from your incident resolving partly,
Speaker 5: h you have probably a tea has resolved a lot
Speaker 5: of incident, but if they have not learned anything from them,
Speaker 5: then's pretty much beastful because tomorrow similar or probably the
Speaker 5: same incident would occur, probably a different team of.
Speaker 4: You know, in your team itself. So like, I think
Speaker 4: the value of the postem needs to be told pretty
Speaker 4: you know, clearly, and it's a very clear poposition.
Speaker 5: I always feel like if you tell the engineer if
Speaker 5: you don't like, you know, hey, what we don't want
Speaker 5: is to you know, you spending this much amount of
Speaker 5: time again on a similar thing, you know, next week.
Speaker 5: So that's where postpartant can help. So I think think
Speaker 5: that value is pretty much it's I think it's it's important.
Speaker 1: Yeah, And I think that's one of the really big
Speaker 1: values of using an incident response tool is it it
Speaker 1: will collect all of those data points and help you
Speaker 1: more easily see that you're having this common failure.
Speaker 2: Yeah, over and over again.
Speaker 1: That otherwise, if you're just tracking this in like Google
Speaker 1: docs or whatever, you wouldn't actually see that correlation.
Speaker 5: Yeah, I think, like I think it needs to start
Speaker 5: with what information you are feeling. So like, like even visually,
Speaker 5: what we kind of help is to do the five buys.
Speaker 5: You know, what what went well? You know what's first
Speaker 5: of all, what happened, then what went well?
Speaker 4: What can go with?
Speaker 5: What we can do to you know, mediate or in future.
Speaker 5: So like having those information in places pretty much is
Speaker 5: the first tip. So do you have the right way
Speaker 5: to analyze stuff in the timeline spart so you know
Speaker 5: if you have you know, if you want to do
Speaker 5: the slack conversations or if you want to do you know,
Speaker 5: want to see what happened from when the incident was
Speaker 5: triggered puill the result that timeline also helps you a lot,
Speaker 5: so that you know where a lot of time was
Speaker 5: being spent or if there's a miss there is like
Speaker 5: a gap in the communication process. That is also that
Speaker 5: is also kind of visible from them or you know,
Speaker 5: what are the tickets or the action items that you
Speaker 5: have created out of it. So there are like I'm
Speaker 5: sell a lot of information in that postpotum document that can.
Speaker 4: Help you to you know, analyze a lot of things.
Speaker 5: And most of the times we have seen it are
Speaker 5: usually a communication you know, uh communication error that is happening. Generally,
Speaker 5: let's say you didn't sort of you didn't tell the
Speaker 5: team to you know, he ep vision celebrated, so you
Speaker 5: need to update. Things like that are the most common issues.
Speaker 5: So from you set up a process around that too.
Speaker 5: You know, next time, if you're great a version that
Speaker 5: that modefiction is tent to different teams. So but avery
Speaker 5: bit of this is always and always, you know, you
Speaker 5: get these results or these jwills only after you have.
Speaker 4: That document which has that entire information place.
Speaker 5: So so and you can sort of you know, add
Speaker 5: those action items alves maybe like a short term actional item,
Speaker 5: a long term action item, and that.
Speaker 4: That really helps. And the other part is to follow
Speaker 4: this up.
Speaker 5: So like you create these documents, you probably have meetings also,
Speaker 5: but what after that, we need to sort of follow
Speaker 5: those action items to the last brick until those tickets
Speaker 5: are closed. You need to follow that up because otherwise
Speaker 5: this entire process becomes useless. So following that part to
Speaker 5: the very end is also pretty much important.
Speaker 3: I heard a spicy take recently, and I want to
Speaker 3: I want to lay this on you. Every incident could
Speaker 3: have been prevented if you just had the right test.
Speaker 5: Like I think every instance, as I'll say, like most
Speaker 5: of the cases that we have seen is usually the
Speaker 5: communication part, Like that's the most common thing that we
Speaker 5: are also, uh like like I think like like I
Speaker 5: remember case like terraform has a lot of issues. Everyone
Speaker 5: kind of has a different story.
Speaker 2: To it, but no argument here.
Speaker 5: So we like one time what we saw is like
Speaker 5: if we update the security groups via terra form, what
Speaker 5: CBS was.
Speaker 4: Doing was like it removes the security groups first. Like
Speaker 4: let's say, if I want to add a security group, it.
Speaker 5: Removes the security group first, the existing one, and then
Speaker 5: it adds the list even though I'm adding a just
Speaker 5: one exact Now, what happened was in the meantime and
Speaker 5: it's removing those security groups. So we was down, like
Speaker 5: the service was down for let's say two three minutes.
Speaker 4: Now, this is something that we kind of.
Speaker 5: Like that happened to our two and that's a potentially
Speaker 5: like you know, ticking bomb, which can actually you know,
Speaker 5: happen any time to across any games. So even if
Speaker 5: we just communicate like, hey, this is what we have seen,
Speaker 5: this is what we had in the experience, and if
Speaker 5: we just relate to the all the engineering teams, that
Speaker 5: issue would not occur. So that's usually the case since
Speaker 5: we have seen Like if even if the communication is proper,
Speaker 5: i'd stay like excu s differicental times of incidents wander.
Speaker 3: I still I still can't get over the fact that
Speaker 3: terrorform does that by default, Like it seems like something
Speaker 3: that no one in their right mind would have designed
Speaker 3: to have the default be first delete all of the
Speaker 3: resources and then recreate them. I mean that just seems
Speaker 3: like it just backwards to be like, isn't isn't the
Speaker 3: common wisdom in in operations to okay, first we'll create
Speaker 3: the new things, make sure that it works, and then
Speaker 3: switch over to it. Why is the default delete?
Speaker 4: I don't know.
Speaker 2: I I maybe maybe I just need enough.
Speaker 3: Coffee or something and someone and it will just magically
Speaker 3: insight will come to me.
Speaker 4: Oh maybe it's that's a a W is seeing. We
Speaker 4: don't know. I don't remember, no, because.
Speaker 3: Like cloud cloud formation and CDK and everything like that,
Speaker 3: it's not it's just the order in which the s
Speaker 3: K is being executed. There's no fundamental reason why it
Speaker 3: has to be that way.
Speaker 1: I want to come back to your your spicy take Warren.
Speaker 1: Every outage could be eliminated or avoided with the right test.
Speaker 1: I mean, I think in theory that's true, but the
Speaker 1: like the practical steps of executing that make it not
Speaker 1: the right answer for a lot of people. But I
Speaker 1: think it does highlight something that I don't think I've
Speaker 1: ever talked about in terms of incident response with anyone before,
Speaker 1: and that's identifying what your risk tolerance is. Because for
Speaker 1: a lot of companies, having some downtime is really not
Speaker 1: a big deal. In other companies, it is a big deal.
Speaker 1: Like I worked for a while in a medical company
Speaker 1: where downtime for us meant that patients could potentially die,
Speaker 1: So we were kind of risk averse there. But in
Speaker 1: other places, you know, I worked for a company that
Speaker 1: built a fitness app. You know, if we were down,
Speaker 1: somebody had to figure out how to use the treadmill
Speaker 1: on their own, I think they're going to be okay, But.
Speaker 4: Like in those.
Speaker 1: Yeah, but in those two extremes, you know, there's like
Speaker 1: there's a different risk tolerance for how much downtime you're
Speaker 1: willing to take. And I think that is probably something
Speaker 1: that maybe needs to be talked about more by companies
Speaker 1: when deciding how much downtime we want, Yeah, is down
Speaker 1: every weekend?
Speaker 5: Right, I think both I think even in the downtime
Speaker 5: as well as I think it brings to a point
Speaker 5: also about alert fatigue or on call forty Like people
Speaker 5: kind of have very lower thresholds for a lot of
Speaker 5: things and over the time to realize like probably we
Speaker 5: don't need a lot of lower thresh shoes, so uh, Like,
Speaker 5: I think alert generally happens when people, when teams are
Speaker 5: starting to have their on call process in place, they
Speaker 5: put alerts on a bunch of things and over the
Speaker 5: time for we don't need this alert or that is
Speaker 5: probably what we or we can raise the thresholds, so
Speaker 5: like like like click really be also provide these values
Speaker 5: like you can innotate alerts to sort of analyze and
Speaker 5: probably reduce some of the alerts that you don't even.
Speaker 4: Need or you can probably increase trasuments.
Speaker 5: So similarly for formably incidents, also you can define or
Speaker 5: change your you know, sexual.
Speaker 4: Values over the period of time.
Speaker 5: Uh some some some companies can afford to have incident
Speaker 5: response only during let's say business hours, they don't probably
Speaker 5: they can afford to maybe don't do it during weekends
Speaker 5: or night times. But some companies can't afford for even
Speaker 5: a firm minute. So absolutely depends on completely company, type
Speaker 5: of product or type of service detment.
Speaker 3: Yeah, I mean, I think the same thing with like
Speaker 3: the dependent on alerts from a security standpoint, which in
Speaker 3: my domainment we.
Speaker 2: Talk about a lot like how how much do you want?
Speaker 2: Right like how much is important? How much is relevant
Speaker 2: for you?
Speaker 3: And and maybe the you know pajorly and at all
Speaker 3: you know, help you actually identify after the fact how
Speaker 3: much you have. And then point the ROI is super
Speaker 3: critical to actually evaluate because you know, trying to actually
Speaker 3: sort of duplicate production in a way to actually test
Speaker 3: to see what happens at that scale at that moment,
Speaker 3: and there's no way with cloud providers to uh, well
Speaker 3: practice what does capacity constrained look like? And then if
Speaker 3: I mean your capacity constrained because there isn't another bare
Speaker 3: metal device available. There's no there's no alternative. Oh well,
Speaker 3: you know, well we should be you know, multi cloud provider.
Speaker 3: Like it is never the answer.
Speaker 5: I mean you can have you know, backup das also,
Speaker 5: you can have as much as possible, but like there's
Speaker 5: no sort of answer.
Speaker 3: H Yeah, there's some things like actually pick what your
Speaker 3: solo is going to be, what your objective is going
Speaker 3: to be for uptime or incidents, and then make sure
Speaker 3: your strategy actually includes that and handles it, and then
Speaker 3: measure it based off the number of incidents you get
Speaker 3: rather than saying, oh, yeah, we should know when the
Speaker 3: memory goes about ninety because then it's it's bad.
Speaker 2: Apparently.
Speaker 5: Yeah, I mean it always you know, gets updated, it
Speaker 5: always gets you know, with the time it's probably from
Speaker 5: your your port is growing, your customers are doing to
Speaker 5: kind of get every passage.
Speaker 1: Of switch your top tips for someone who is not
Speaker 1: satisfied with their current incident response uh program or or software,
Speaker 1: I would.
Speaker 5: Say, like, like I think the entire fighting I think
Speaker 5: like you can always see different parts to it. The
Speaker 5: first part is the you know how easily your team
Speaker 5: or anyone is able to report the incident. So, yes,
Speaker 5: you have automated alerts on TV's or on from easy
Speaker 5: tools and all those parts, but they'd say not everything
Speaker 5: can be automated.
Speaker 4: You need to have you know, correct way of identify
Speaker 4: of you know, reporting issues.
Speaker 5: So if you have customers support or if you have
Speaker 5: a product team of let's say someone wants even if
Speaker 5: even if let's say, if you have you know their
Speaker 5: environments or three fraud environments you want to report issues there,
Speaker 5: you need to have a good, good way of reporting
Speaker 5: that UH and hope and have a process that the
Speaker 5: correct uh the issue is reported to the correct team
Speaker 5: as quickly as possible. So time to trigger the incident
Speaker 5: or time to you know, you know you call that
Speaker 5: on call should be as quickly as possible. So identify
Speaker 5: the blockages in that there there can be blockage is
Speaker 5: in you know, in set uping this process office.
Speaker 4: And the other part is let's say, once the incident
Speaker 4: has been triggered or created, what to do? So if
Speaker 4: you if if if you.
Speaker 5: Feel like if the calls feel like a lot of
Speaker 5: work outside mitigation, uh, if they like you know, if
Speaker 5: every time you are having an incident, you if you're
Speaker 5: just running around and probably adding you know, calling people
Speaker 5: and just figuring out, you know, what's the stackers.
Speaker 4: Or adding other team on calls all the time.
Speaker 5: Figure out these kind of blockages in your processes and
Speaker 5: try to streamline as as much as possible so that
Speaker 5: like on call or whoever is other stakeholders, can you know,
Speaker 5: focus on solving as much as possible if you want
Speaker 5: to have.
Speaker 4: Like ah, if that is still like.
Speaker 5: Taking a lot of time, maybe set up like a
Speaker 5: team of an eyework person who is actually handling all
Speaker 5: the incidents and he's he or she's actually dispatching the
Speaker 5: incidents to a correct team.
Speaker 4: And doing like a supervision of the entire process.
Speaker 5: And the last one is to see whether are you
Speaker 5: doing the post modems correctly, like uh, you know, are
Speaker 5: you doing it all or not? Are you actually learning
Speaker 5: the you know, uh learning from your incidents? Have your
Speaker 5: repeated incidents have reduced over their time online? I say,
Speaker 5: like that's the most between a lot of a lot
Speaker 5: of companies kind of focus on entity. I think that's
Speaker 5: not probably the right metric. The right metric is to
Speaker 5: see how many unique incidents you're getting. I think if
Speaker 5: if if if that's if that is fine, that's fine.
Speaker 5: But if you're getting the repeated incidents time of her time,
Speaker 5: something you could would have over then your incident responded
Speaker 5: process is like the first model processes.
Speaker 4: I think that that's interesting.
Speaker 1: That's a good point.
Speaker 3: I mean, I really wonder how many, like are people
Speaker 3: hitting the same incident over and over again? Like I
Speaker 3: my my guess would be probably not exactly, but maybe
Speaker 3: correct categorization would really help spill it. Like, you know,
Speaker 3: is it is it the same part of your framework,
Speaker 3: code based or or component? You know, if you have
Speaker 3: even a monolith and not micro services, you still have
Speaker 3: broken out components. You can at least target it down
Speaker 3: to is it the same component that's causing the problem
Speaker 3: all the time? Uh, as far as a place to
Speaker 3: look and invest in rather than oh, you know, it's
Speaker 3: just something happening within our whole system.
Speaker 5: Right, Like I think, so maybe you know that the
Speaker 5: on call have the run book for when he was
Speaker 5: solving an incidents? Is do they had the right tools
Speaker 5: for you know, just seeing or for the service the
Speaker 5: particular serve it doesn't even have a dashboard, but doesn't
Speaker 5: have any playbook, uh, which can help on So there
Speaker 5: are like I think a lot of learnings, uh, which
Speaker 5: any or or any engine team can do. Uh and see,
Speaker 5: you know how much like I think at the end,
Speaker 5: it's all good how much we can help the on
Speaker 5: hulls as much as possible.
Speaker 1: So yeah, I mean, if we're seeing the same incident
Speaker 1: over and over again, we should at least be able
Speaker 1: to brag about our meantime the resolution decreasing because everybody
Speaker 1: instant service to restart.
Speaker 2: But maybe that's a good point though, right maybe that
Speaker 2: maybe that's the whole point.
Speaker 3: Right, like you you don't want to have that actually
Speaker 3: decreasing because right then then there's like, you know, it
Speaker 3: really points to a different problem. It's like if you
Speaker 3: have run books, you must be because you hit the
Speaker 3: same problem over and over again. And so rather than
Speaker 3: having the run book, it'd be better to eliminate where
Speaker 3: the source of the problem is coming from.
Speaker 4: And that's why I didn't.
Speaker 5: Like not like I think it's always divided for you
Speaker 5: on you know, with how good does a MPDIA matrix
Speaker 5: actually want to like trust on it? Like, because it's
Speaker 5: usually counter intuitive if you see, like as as you
Speaker 5: rightly say, say, let's say let's say a company has
Speaker 5: dissolved most of the incidents, like they have resolved it
Speaker 5: to the correct points in six months of time. In
Speaker 5: the seventh and in the seventh month, the engineer team
Speaker 5: doesn't have the same incidents or similar incidents, but they
Speaker 5: have only one new incident which kind of took a
Speaker 5: long time because that was like a unique se So
Speaker 5: in that case, the MPDR is too big because a
Speaker 5: new incident came with a lower frequency, lower number of times,
Speaker 5: but it took a longer time. But can we say
Speaker 5: that that the engineering team had a you know, bad
Speaker 5: state of incidents hygien, No, because they had kind of
Speaker 5: resolved most of the incidents that have occurred in the
Speaker 5: past and those are not occurring now this is like
Speaker 5: a new one. So that's why I think like MDA
Speaker 5: is always not the right victory to see in terms
Speaker 5: of incident hygiene.
Speaker 3: Yeah, I think the ERA budget, according to your SOLO
Speaker 3: is a much better one in this regard unfortunately. Yeah,
Speaker 3: but I'm I'm totally I mean, I think that all
Speaker 3: the door metrics sort of have that problem in a
Speaker 3: way if you measure them purely or just people in general,
Speaker 3: rather than how they're actually relevant. Like I I remember
Speaker 3: working with one company that they were measuring even in
Speaker 3: cycle time, but they were using feature flags and not
Speaker 3: including that in the cycle time. So I'm like, yeah,
Speaker 3: your code is going to production, but no one's using it,
Speaker 3: So what's the point.
Speaker 2: What's the point?
Speaker 3: I mean, yeah, I mean measure then also measure the
Speaker 3: cycle like the cycle time on feature flag removal. That's
Speaker 3: going to tell you a lot more about your success.
Speaker 5: Right, I think, like you know, we have seen a
Speaker 5: lot of tools on so we have seen a lot
Speaker 5: of tools maybe just you know how many commits you
Speaker 5: have pushed, So I think everything has to be you know,
Speaker 5: read with a lot of context, with a lot of corns.
Speaker 5: Is also not just because it can change based on
Speaker 5: a different kind of uh things that are happening.
Speaker 3: Yeah, I know we're getting close to the limit on
Speaker 3: the time that you've got with us today. Uh maybe
Speaker 3: there's some last words and then we can move over
Speaker 3: to picks. Anything you want to share.
Speaker 4: No, I think like this was pretty cool. Uh.
Speaker 5: Like I think like with with pajor L also we
Speaker 5: have we have seen a lot of different and unique
Speaker 5: cases h and it's good, like this is something that
Speaker 5: which is very close to you know what we have
Speaker 5: seen like what we have failed, and like mean, we're
Speaker 5: like to help companies to sort of stream and this
Speaker 5: entire process as much as possible.
Speaker 3: It's got to be super interesting too to see how
Speaker 3: companies incidents actually look like Oh for sure, Yeah, I.
Speaker 5: Think like what we have realized is like every company
Speaker 5: needs their kind of like every company has different processes,
Speaker 5: probably because of the state of their product, the state
Speaker 5: of all size. And what's what we have always ensured
Speaker 5: is like whatever your process or whatever you feel like
Speaker 5: is the most app will not force a tool of
Speaker 5: that will adapt to your kind of processes. We'll just
Speaker 5: make it more automate in most because we know, like
Speaker 5: you have set up some big even your set and
Speaker 5: we know that you know that uh sort of some
Speaker 5: of the big spieces of it. So there's no one
Speaker 5: particular way of doing things, but whatever it may will.
Speaker 2: Hit well said, well said, So what do you think, Well,
Speaker 2: should we should we do the picks?
Speaker 1: Let's do some picks.
Speaker 2: Okay, I know you put me on the spot anyway,
Speaker 2: so I'll just go first.
Speaker 3: Uh my, My my pick for day's session is a
Speaker 3: book called Radical Focus by I think it's a Christina
Speaker 3: Vodka it's a it's actually fantastic. It's a hypothetical story
Speaker 3: about how to actually uh set priorities using okay, r
Speaker 3: S or KPIs or whatever, MBI whatever you want to
Speaker 3: call them, honestly, and how not to do it and
Speaker 3: lessons learned from that. It's it's super relevant no matter
Speaker 3: what level you're at, realistically, like even at the team level,
Speaker 3: it's super interesting to think about, like how many priorities
Speaker 3: and what should our focus be on? How to think
Speaker 3: about that because I've seen so many teams, so many
Speaker 3: companies have like, oh, yeah, we have ten initiatives for
Speaker 3: this quarter, and I'm like, you can't. I bet your
Speaker 3: engineers couldn't even tell you five of them, Like it's
Speaker 3: just too many. And I think it's a great story
Speaker 3: about how to actually think about this and what's relevant.
Speaker 2: So highly recommended.
Speaker 1: Dude, your picks are always so relevant. I feel like
Speaker 1: mine are just like better, but yours are like, oh wow,
Speaker 1: that could actually work and be helpful.
Speaker 3: I mean, I don't know, maybe being I'm being lazy
Speaker 3: by picking easy things. Well, you know, I'm when I'm
Speaker 3: year two of a host here, maybe I'll have run
Speaker 3: out of.
Speaker 2: Things and then I'll be onto the I don't know
Speaker 2: my weights that I've got in my other room that
Speaker 2: I'm using.
Speaker 1: All right, Fali, what you bring for us for a
Speaker 1: pick today?
Speaker 4: I that's a little thing is one. I think.
Speaker 5: One I've seen like a documentary recently which was how
Speaker 5: Toyota Big Stuff and a lot of things was very interesting.
Speaker 5: I'm forgetting what what they call it, but essentially what
Speaker 5: they they is. And the third is like known for
Speaker 5: it's like you know, building bug.
Speaker 4: Free products manufacturer. Yeah for sure, yeah. Yeah.
Speaker 5: And one of the one of the things that they
Speaker 5: have always is like no matter where their manufacturing nits are,
Speaker 5: no matter where the what they if there's an issue,
Speaker 5: it gets reported to the topmost year like immediately, like
Speaker 5: with with proper clarity and and and that's how I think,
Speaker 5: like communication becomes so much important and like that kind
Speaker 5: of solves a lot of things.
Speaker 4: So like it, I think, and it's just very fascinating
Speaker 4: how kind of.
Speaker 3: They actually they actually have these like cords on the
Speaker 3: manufacturing for called and onlines that helped. Yeah, they stop
Speaker 3: the whole manufacturing line at once. You know, hey, you know,
Speaker 3: no more pull requests at this moment. For the whole
Speaker 3: company because there's something critically wrong going on, Like could
Speaker 3: you imagine.
Speaker 4: Yeah, yeah, I think I think on line was I
Speaker 4: was different to him. That's super super.
Speaker 5: I think that's such a simple kind of technique that
Speaker 5: any company can sort of have, Like I make such
Speaker 5: a simple forceses but very.
Speaker 1: Yeah. One of my customers was a company that provided
Speaker 1: seats to Toyota, and it was wild because they would
Speaker 1: get orders like Okay, we need three hundred and seventeen
Speaker 1: seats that are beige delivered at ten twenty seven am.
Speaker 1: You know, like the level of specificity because they have
Speaker 1: that that just in time manufacturing, like we need these
Speaker 1: at ten twenty seven am. And and this was a
Speaker 1: smaller company, so like the level of pressure for them
Speaker 1: to meet those requirements was just through the roof.
Speaker 3: I mean it makes a lot of sense too if
Speaker 3: you think about it, because they see inventory storage as
Speaker 3: a waste, as a cost to them, and so they
Speaker 3: don't want to have it stored at You're the inventory
Speaker 3: for them. You know, they're they're going to the shelf
Speaker 3: and they're pulling it and you are that shelf for them.
Speaker 2: Yeah. No, it's awesome.
Speaker 1: Yeah, I feel like, you know, we were talking about
Speaker 1: this before we started recording, about how my picks are
Speaker 1: just kind of out there. I feel like this one
Speaker 1: is going to be unlike the the crazy scale, this
Speaker 1: one's going to be hard to top. And yeah, I'll
Speaker 1: just get to it. So I read this book. I
Speaker 1: just finished it up a couple of days ago, called
Speaker 1: The Sacred Mushroom and the Cross by a guy named
Speaker 1: John Marco Allegro, and I would be tempted to call
Speaker 1: bullshit on the book right away, except for the fact
Speaker 1: that this guy spent fifteen years deciphering the Dead Sea Scrolls.
Speaker 1: And so if you're not familiar with the Dead Sea Scrolls,
Speaker 1: it's a set of scrolls that were found in Egypt,
Speaker 1: I believe in the nineteen forties that were thousands of
Speaker 1: years old, and they contained some parts of the New
Speaker 1: Testament Bible, but they also had other stories in there
Speaker 1: as well that weren't included in there, and so he
Speaker 1: deciphered them. But this book, The Sacred Mushroom in the Cross,
Speaker 1: he basically goes through this book showing or arguing that
Speaker 1: a lot of the stuff written in the Old Testament
Speaker 1: and the New Testament and some other religious books as
Speaker 1: well were not factual base, but they were actually like
Speaker 1: a play on words referencing psychedelic mushrooms, and that the
Speaker 1: whole religion is based on an ancient cult or ancient
Speaker 1: culture that worshiped psychedelic mushrooms. And it's a wild read. Man,
Speaker 1: It's very hard to read because of all the references
Speaker 1: he makes to like the Aramaic and the Semitic languages.
Speaker 1: But the big takeaway for me was, you know, you're
Speaker 1: reading through this and he's like, oh, so, well, they
Speaker 1: said this thing and that's actually the you know, the
Speaker 1: ancient Sumerian word for this psychedelic mushroom, and like everything
Speaker 1: points back to being the name of a psychedelic mushroom.
Speaker 1: And I was like, dude, how is it that we
Speaker 1: know so little about the ancient Sumerians, but you know
Speaker 1: the four hundred different words they had for psychedelic mushrooms.
Speaker 1: But then at the end of this book there's like
Speaker 1: a chapter. I can figure out who wrote this last
Speaker 1: chapter because it wasn't John o'leegertt was someone else. But
Speaker 1: the guy was talking with his wife, she was from Russia,
Speaker 1: and they came across a field with a bunch of mushrooms,
Speaker 1: you know, and he's like he was an American and
Speaker 1: he's like, no, don't eat those, they're all poisonous and stuff.
Speaker 1: And she's like, no, this one is, this, this and this,
Speaker 1: and so it turns out in Russian they have like
Speaker 1: an endless number of words from mushroom, but in the
Speaker 1: US and in Western cultures, we have, you know, like
Speaker 1: three like toadstools, mushrooms and and whatever else.
Speaker 3: So it's a yeah, that's a European thing because people
Speaker 3: actually go pick out pick mushrooms here, and so knowing
Speaker 3: which ones are poisonous the same thing.
Speaker 2: You know, once I've moved here, I learned all about that.
Speaker 3: But I think you've meaned yourself well, because now I
Speaker 3: see instead of you know, just this, but instead of aliens,
Speaker 3: now it's just it's mushrooms.
Speaker 4: Right, aliens? The last one, well, no, it's the the
Speaker 4: guy from.
Speaker 1: The Ancient Aliens TV show with the big hair. If
Speaker 1: you've ever seen that meme.
Speaker 3: It was the History Channel, there was Ancient Aliens and yeah,
Speaker 3: it's like the pyramids are landing platforms for aliens, and
Speaker 3: you know, well's here, you know, trying to.
Speaker 2: Sell us on the fact that the religious cult is
Speaker 2: of mushrooms.
Speaker 1: I mean, yeah, good reading.
Speaker 2: I added to my list, So thank you for that.
Speaker 1: Yeah, let me know when you when you get through it,
Speaker 1: I'd be interested to talk through that with you, because
Speaker 1: there's like some parts of which you're like, okay, I
Speaker 1: see how you can get to that conclusion, and there's
Speaker 1: other parts who're like, come.
Speaker 4: On, is it like pretty famous kind of book it
Speaker 4: was written?
Speaker 1: I think it's recently. It was recently. It was published
Speaker 1: in nineteen seventy, so it's an older book. And then
Speaker 1: he got a lot of hatred and supposedly it was
Speaker 1: very detrimental to his career. Go figure, who would have
Speaker 1: thought that, you know, claiming Jesus was a psychedelic mushroom
Speaker 1: with detrimental to your career. But anyway, but I think
Speaker 1: it's gained in popularity over the last couple of years
Speaker 1: just because of the shift in the things that we're seeing,
Speaker 1: at least here in the Western world, where people are
Speaker 1: kind of changing their opinion and approach to things like
Speaker 1: to natural medicines like psychedelic mushrooms, and you know, the
Speaker 1: legalization of pot, and now in Oregon and Colorado there's
Speaker 1: actually decriminalized centers for using mushrooms to treat like PTSD
Speaker 1: and memory issues and things like that.
Speaker 3: You're really close to Canada where that's been a huge
Speaker 3: topic in the last years.
Speaker 4: Yeah.
Speaker 1: Yeah, so I think that's been a key to the
Speaker 1: book gaining new popularity. Yeah all right, so there you go.
Speaker 1: So now the challenge is on next week? What am
Speaker 1: I going to come with a pick that tops Jesus
Speaker 1: as a mushroom?
Speaker 4: I actually saw one movie which is on my mind
Speaker 4: is Inside Out too. I saw, I think.
Speaker 5: Last night, and I think it, Uh, if someone wants
Speaker 5: to make highly recomm I think you can sort of
Speaker 5: feel a lot of emotions, uh for for.
Speaker 4: That's that is something which is like just all my
Speaker 4: mind and what was the name of that one? Inside
Speaker 4: Out the second part?
Speaker 1: Yeah, awesome, And with that done, I think we have
Speaker 1: an episode for Thanks for joining us, man, This has
Speaker 1: been a blast. Really appreciate having you on the show.
Speaker 4: Great, thank you.
Speaker 5: Yeah, I think I think, uh, it's been really great,
Speaker 5: had you know, wonderful time just to chat about incidents
Speaker 5: and a lot of other things and sharing each other's
Speaker 5: you know, personal experiences. I think this is something like
Speaker 5: every OnCore and every or even every developer has their
Speaker 5: own personal experience what they want to share.
Speaker 4: So it's been a really good.
Speaker 1: Child, awesome cool. Thank you again, and to all the listeners,
Speaker 1: thank you for listening. Appreciate y'all and be sure and
Speaker 1: hit us up if there's anything we can do for you,
Speaker 1: and we'll see y'all next week