Hey, what's going on, everybody? I am the host of Adventures and
DevOps and wait, yeah, that's the right channel name. Sorry. You
know sometimes I get that mixed up, like almost every time I think,
so if you're a frequent listener to the podcast, you know that I messed
that up just about every time. So thanks for bearing with me on that.
But today I'm not going to miss this part up because today I have
as our guest of Drew Stokes. He's the senior manager of software engineering for
Page Your Duty, and we're talking about incident management. And that's one of
my favorite topics because it just goes so deep and it crosses so many disciplines
across your infrastructure and your application teams and marketing and sales and the executive suite.
Like depending on the level of the incident, you're just all overboard,
all over the board with this. So Drew, welcome to the show.
I'm excited to have you here. Thanks, I'm excited to be here.
It's going to be too well right on, So tell us a little bit
how you got into the field of incident management or incident response. Oh that's
a good question. Okay, Yeah, So I've been in tech for a
while, like most people here, I think it's been something like sixteen years,
and I think originally I was kind of trying to figure out my way
helping folks out with technology and networks, and then I got into front end
development and moved into back end and then dropped into SRE and that's when I
kind of really got familiar with not just the process of mitigating incidents, but
actually managing them and trying to learn from them. So I did that for
a while, and then I think for something like the last eight years,
I've been primarily focused on people manager role. And there's a lot of ways
in which you know, people managers are involved in incident management as well,
both as stakeholders but also you know, facilitators and folks who are playing a
supportive role for people who are responding. So kind of been in that space
for a while now, and back in I think it was May of twenty
twenty one, I joined a startup called Jelly, which was founded by Nora
Jones, who's the author of the Chaos Engineering book and the founder of the
Learning from Incidents community, and that was kind of where I really dropped into
you know, incident management in general, but specifically this opportunity to kind of
not just resolve incidents, mitigate the issues, but also to learn from them
in order to improve future response and organizational performance. So there's a lot of
really interesting ways to think about the space, and you mentioned at the beginning
it's really important. And part of the reason is because it's so cross cutting,
right, because incidents or a lens through which you can see the way
that your organization and your people operate, and that applies to customer service,
that that applies to executives and to the folks actually responding to the incidents.
It's a really interesting space with a lot of opportunity, which you'll you'll hear
that word a lot in this conversation. We refer to incidents as opportunities.
Oh for sure they are because you know, one of the things that I
think about a lot is just because we're in tech. You know, we've
all done the Google search for is such and such service down because you're having
problems and you're like, did I do something wrong? Or are they actually
dead in the water right now? And I think that's like, to me,
that's one of the hallmarks of highlighting that your incident response plan is really
really well done whenever your customers know that you're having an incident incidents because you
told them versus them discovering that something was broken. Yeah, there's a there's
a level of well. So, so one interesting aspect here is you mentioned
another cross cutting function there, right, which is you have internal stakeholders and
external stakeholders for these types of things. But there's also this layer I think
that you're referring to here of like operational excellence and observability. Right, do
you know that the system's broken before someone tells you that the system is broken,
And a lot of the ways in which you can improve that process is
through the learning process after the incident. Right, So, if you have
an incident, for example, where a customer reports an issue, looking at
the details of that timeline and what actually happened can help you figure out where
you need to add additional instrumentation or alerting, or how to adjust your team's
processes, you know, your software development life cycle or your release process to
better account for those kind of unpredictable behaviors in the system. So really interesting,
like complicate, you know, when you're dealing with not just complex software
systems, but also complex organizations and groups of people, right, really interesting
opportunities to figure out how do we kind of iteratively approve improve our understanding of
the system and our understanding of failure mode so that we can kind of inspire
customer confidence and trust, right, letting them know that there's an issue before
before they don't. Yeah, for sure. So you early early on in
this you mentioned something I want to highlight, mitigating an incident versus managing an
incident. Can you elaborate on the difference between those two, Yeah, that's
a that's a great question. So there are a lot of different aspects of
incident management in general, and I'll try to like decompose them in a way
that makes sense here. So I think when you just reference detection, right,
so there's a there's a phase there of like understanding whether or not there's
an incident and trying to do something about it. And I think when we
talk about managing incidents, what we're talking about is providing information and coordinating folks
in incident response. Right. Mitigating an incident is doing something to address the
issue and get the system back to a stable state or you know, performing
in a way that's expected with regard to external stakeholders. But I think for
us, managing an incident is really about investigating what's going on, getting the
necessary folks with the subject matter expertise into the room to contribute to that,
coordinating that group of people in you know, large organizations are really complex incidents.
Sometimes you have multiple work streams of investigation within an incident, and then
communicating status out to stakeholders, your customer success team, your executives in a
way that allows them to stay informed but does not have them jump in and
start, you know, trying to get involved in the process in a way
that can you know, add additional complexity to the overall incident managed. So
from my perspective, I think management is a lot more about the process of
coordinating and communicating during an incident, and mitigation is about that moment when you've
kind of identified and addressed the issue to stop whatever impact is associated with the
incident, Right, that's your signal to your external stakeholders that we are in
a stable state, we've seen things are good, and there are various other
steps after that, But for me, that's the primary difference. Yeah,
Yeah, I think that's really important for someone who's not done a lot of
incident responses to understand that the management of it is equally important as the mitigating
of it. And in many of the environments I've worked in, those are
actually two key roles for any incident. You have the first responder who's trying
to find the cause and restore the service. But then you know, alf
what have your primary communications individual who is getting the information from that first responder
and relaying it out and doing still in a way so that everyone feels like
they're in touch with what's going on and they aren't going around the back door
sending DMS to the first responder to get status updates. Yeah. Yeah.
One thing we talk a lot about is kind of this this incident management maturity
model, and we think about different buckets of you know, engineering teams or
organizations with regard to kind of how they approach this. And I've been in
you know, multiple layers of the lower maturity model, and sometimes it can
be really difficult, yeah, to even understand who's doing what and who do
I ask for an update? You know, I've got a customer who needs
an update now, and we have an SLA in the contract, what's going
on? It can be really difficult to even know who's doing that. And
I think you find that in you know, incident response tooling like Jelly,
those roles are actually codified in the process. You're assigning an incident commander,
you're assigning a communications lead to try and take care of that external communication of
here's the person to you know, connect with if you need an update,
or here's the person responsible for managing this incident, so that if you join
in, you can say, hey, I'm here and I know about X,
you know, can I help that sort of thing? Right? And
so that's one of the things that Jelly does for you. If you need
to improve the majority, improve the maturity of your incident response playing, using
something like Jelly can kind of help you say, hey, here are the
here are the people and the processes you need in place, and provide a
framework, right. Yeah. I think I think like every small organization goes
through a phase where someone opens a Google doc and writes down a run book
for how to run incidents, right, And so what we wanted to do
is to provide some of that for you in a way that didn't get in
your way. So we've got a bot in Slack right that you can use
to declare incidents as sigence, stakeholders, set stages, communicate status, all
that sort of stuff, so that you don't really have to go in and
kind of trial and error that Google doc and try to get folks enrolled in
the process. There's just a thing kind of nudging you along the way and
helping to offload some of that cognitive burden. When you're in the middle of
managing an incident, right or typically as an incident commander, you're thinking about
a lot of things. Sometimes you're also trying to mitigate the incident. Right
if it's two am, you may have a stretch of time where you're doing
everything on your own. And so I think the more folks can find mechanisms
and processes that help them reduce the number of things they're doing during management so
they can focus on getting the right folks in the room and finding the means
to mitigation, the more successful the response process becomes, which results in better
data for your post incident analysis, and then you're you know, cross the
incident learning over time. Yeah, it's one of those things that like we've
we've all done incident response wrong enough time enough times that we we kind of
know, So it's I think it's one of those things like you know,
like in software engineering, like writing logs has been done for decades now,
so you don't write your own logging engine. You just pull in a logging
library because you don't need to reinvent that wheel. And I think incident response
is one of those we don't need to reinvent this will we can just buy
a wheel that's already built. Yeah, we've we've we actually have a couple
of customers of Jelly who are trying to replace their wheels, right because you
know, some of some of the large organizations who started this process ten years
ago had to make their own I used to work at New Relic and we
had a slock bot we called nerd bot, which was our incident response you
know, facilitation tool. But there's a cost associated with those things, right,
You have to maintain them over time. Oftentimes they kind of fall to
the bottom of the priority stack, and so iterating on your internal process becomes
really hard. And I think that's where if you go with something you know,
like Jelly's incident a response spot, which is you know, fairly opinionated
but narrow in scope, right, it's just here are the set of criteria
that we use for this thing. With some customizable features like automation, then
you don't have to kind of invent that wheel and then reinvent it iteratively for
all time. And you also don't really have to, you know, answer
a lot of those questions when your incidents become more complex. There's like different
phases of your incident response process. When you're a five person team, you
jump in a zoom call, right, and you fix it. When you're
fifty people in a major incident room, it's a very different experience and requires
a different set of skills and supporting tooling. So, yeah, cool,
you mentioned a couple of times the post incident response plan, so elaborate on
that a little bit for me. Yeah, this is another area where I
think everyone kind of starts with a recognition that there's more that can be gleaned
from these experiences. Right early on, you have an incident, you respond
to it, you fix it, maybe you shoot an email off to folks
saying what happened, and you know, here's what we're going to do to
address in the future. But as your system complexity grows and as your organization
grows, there are you know, many more opportunities to figure out how to
change not just the system itself right to you know, write better logs or
increase visibility into the system's behavior, but also to change how the organization is
structured around those systems. Right. So, one anecdote I like to share
is at my time in a previous company, we had this custom feature flag
system that had been around for I don't know, it was like eight or
nine years or something. Everybody wanted to get off of it. It wasn't
great, and every time there was an incident with that system, someone from
the network engineering team would be pulled in because they were one of the original
authors. They had nothing to do with this system anymore, but no one
else knew how it works. And so if you're just responding to and mitigating
incidents and not looking any further, you don't see those types of organizational misalignment
right where you've got a primary owner or subject matter expert that is, you
know, accountable for a whole slew of things that have nothing to do with
this foundational service that's critical for business function. If you've got a feature flag
system in you know, a fourteen year old code base it's got to work.
So I think when we talk about post incident learning, this is this
is the next phase in maturity. Right, you figured out your response process,
you know how to get the right folks in the room, you know
how to move toward mitigation, and you're starting to capture some of the you
know, follow ups that you want to take. Maybe we need more ossability.
Maybe this library and our services out of date, and if we updata
we'll get better performance. Like that, But it goes beyond some of those
follow ups, and as you start to cultivate a process around this, and
there's a lot of different ways that folks do this. You know they're refer
to on this post mortems or learning reviews, or you know, sometimes you're
just getting in a room and talking about the incident without the structure, you
start to uncover all of these really interesting aspects of not only the responding team,
but the organization overall. And so some of the things that we're most
interested in learning is, you know, what did folks know when they responded
to the incident and what did they not know? Right? What are the
ways in which the folks involved communicated successfully and maybe not so much? How
did the organization's processes contribute to or prevent aspects of a specific incident. It's
all kinds of interesting stuff to dig into, and you can look at it
from a bunch of different angles. So we have, you know, a
lot of examples of our customers creating multiple investigations on an incident where a person
A and person beat both investigate and then you see like where the differences are,
and I think that turns up a lot of interesting stuff. We've taken
the approach in Jelly of writing incident narratives, so you know, post learning,
review, post mortem, whatever you want to call it. Our feeling
is that incidents are stories and the way that people connect with information and learn
is through storytelling. And so we've taken the approach that, you know,
we want to provide folks with a tool to tell a story backed by evidence,
right, what was actually said during the incident, what you know,
metrics or data we were looking at, but to kind of nudge folks in
the direction of sharing their perspective and their assertions about what it means. Right,
when these two folks were talking, they were talking about different aspects of
the system, and they didn't realize it what does that mean, right,
what's the opportunity there to improve the incident management and the way that these teams
are connected and communicating those sorts of things. Yeah, you see that a
lot whenever you have people with different disciplines or different backgrounds, you know,
a networking background versus a software engineering background. And I think that highlights one
of the one of the arts of post incident response is creating those follow up
items and getting those the right people engaged to recognize, prioritize, and address
the things that you learned from that incident. Yeah, and you know that
you mentioned like different disciplines. There are different different disciplines within the responding team,
but there are also incidents provide this really unique opportunity to consider the different
disciplines across an organization. Right, So for your major incidents, it's not
just your you know, senior engineers from a specific team. It's also your
customer support support folks on critical accounts. It's also your group leads and your
executives. All of these people have different priorities and perspectives and understanding with regard
to the impacted systems and the impact on the business. Right, if I'm
responding to an incident, my goal is to make the chart go down,
whereas my executive or salespeople's goal is to minimize the costs associated with customer impact.
Right, We've got slas with our customers for uptime, and we need
to keep that in line. And I think the different perspectives and priorities there
result in that same kind of differing perspective that I mentioned earlier, where I
may look at an incident and think it means one thing, but my group
lead or you know, my sales associate may look at it and think another
thing. And that opportunity with you know, incident narratives or post incident learning
is to try and bridge that divide between those different perspectives and help everyone cultivate
a shared understanding of what it means across those dimensions. Right, this is
what this incident meant for business impact and process, for customer satisfaction, and
for the you know, sustainability of our you know, critical services something like
that. Yeah. I've even worked in organizations where it involved the marketing team
because they were out scrolling Twitter, you know, catching tweet going on about
the incident and responding those and trying to do trying to minimize the blast radius
there. Yeah, this is a whole other aspect that's really interesting, which
is like where do incidents come from? Right? Who says what an incident
is? We've taken the approach that anyone can declare an incident. Some organizations
we've worked with are very narrow in terms of who can declare them. But
yeah, customer success marketing, you know, random person from the internet.
There are all sources of potential incidents, you know, automation and observability,
those sorts of things, and so it's you know, the the once you
start thinking about this space and you start exploring ways of benefiting from these lenses
on current state of systems and organizational process, you start to see like there
are opportunities everywhere. Right at Jelly internally, we create incidents for things that
are not incidents. If we have a release going out that we think might
be you know, impactful to customers because it changes some aspect of the user
experience, that's an incident. If we're trying to better understand database failover in
RDS, for example, we run a game day as an incident, and
doing that gives you this repository of information that you can use again to build
that narrative and make those assertions about where are we and where do we want
to be with regard to how we're operating and the health and stability of our
systems. So that's a really interesting anecdote about marketing. I love when those
things come in from places you don't expect, right, You just kind of
get a message from someone that you haven't met before and they're like, hey,
there's something going on yet we'd better declare Yeah, yeah, you see
someone from marketing enter in one of the tech Slack channels and that this is
not going to go well. So I think one of the cool one of
the cool types of companies I like to work with fit the model of Jelly
because you actually use your own product, you know, like when you build
and release it, your team actually uses it to manage your own incidents.
And I think that is really really cool because you get firsthand experience of what
it's like to be your own customer, and you can understand what your customers
are actually seeing when they're trying to use your tool. Yeah, one thing
that was really interesting thing in the early days about working with our customers.
It's interesting now as well. We'll have to talk about page duty at some
point later. But one thing that was really interesting is that the customers that
we work with are really passionate about their process and those opportunities to learn,
and so we get to work really closely with them on you know, understanding
their process and building tooling it works for them. We work with F five
and Indeed and Honeycomb and Zendesk. These are like, you know, large
influential organizations who are kind of at the cutting edge of this process. So
there's this bi directional information share where you know, we can build features that
support those organizations processes, but then we can also adopt some of those organizational
processes because they make a lot of sense and they work well for us.
I was we were doing a product demo for an important group of people the
other day and we noticed some lag in one of our features and I actually
declared an incident with Jelley about the performance of the incident was cons tol jelly,
and we ran that in parallel during the demo, and it was there
was this moment where I was just like, this is so cool running an
incident with the tool that we're demoing to people, and there wasn't actually an
issue. It was a Wi Fi lag you know thing, So everything was
good and that's okay. That's also a learning opportunity. But yeah, it's
been really exciting to kind of watch things evolve over time and be a you
know, benefactor of that system as well as trying to evolve it for our
customers and find that alignment across across orgs, which is really unique. Most
of your incident response and post incident learning is within an org. We've had
the unique opportunity to kind of extend that outward, so fun right on.
One of the things I'm interested to get your opinion on is I over the
years, I've developed the opinion that there's a difference between mitigating the issue and
resolving issue. And I refer to that in in terms of, like during
the incident, you know, you have you know, to say your API
service is slow, it's okay during the incident to throw more servers at it.
You know, we're going to we're going to mitigate the issue by adding
more servers or adding more memory, or do something to make the symptoms of
the problem go away. But then there's this like defining moment of okay,
customer impact has been resolved, but now we've got to go back and find
the root cause because adding the additional servers did not fix the issue that fixed
the symptoms. And I'm interested to get your opinion on that. Yeah,
it's a really good distinction that you're making there, and I think it has
a lot to do with prioritization and understanding. Right. So oftentimes, especially
in major incidents, there's a priority involved there to minimize customer impact, right,
because customer impact means lost revenue. Incidents are expensive both in terms of
time and you know, customer satisfaction and trust. And so I think there
are kind of two ways in my experience that you mitigate before resolution. And
the first i'm mentioning now is about minimizing the impact in favor of kind of
getting things back on track. And so, like you said, throw some
additional servers at the API and that'll address the symptom, but we still don't
understand what's going on in the hood, right, And so I think the
second reason, sometimes you can choose not to mitigate an issue. I've been
in situations where we've had customer impact, but the priority of understanding what's going
on has exceeded the priority of needing to address that impact, maybe because it's
like, you know, one user at a customer rather than all customers in
a major incident. And so that second bit I think is really interesting because
you can use the incident and the the levers you can pull during the incident
to create the conditions for learning while it's happening. Right, So if you
mitigate the incident with the API, it means that you have an opportunity to
explore what was actually going on. Maybe you isolate one of those servers and
you start to dig into you know which function calls. If you've got distributed
tracing, which is amazing, you know which specific function or endpoint is causing
delay in the response, right, that's causing a delay across all responses,
And you can kind of take advantage of that system state, which you know,
if you reboot the servers, if you add a ton of them,
those conditions go away and you lose your opportunity to understand what's going on.
And so there's a lot of different ways to look at it. I think
mitigation and resolution for folks outside of incident response, that's a mental framework for
understanding are we good now and are we good for the long term? Right?
But as a responder, those two events are really key in terms of
communicating within the response group what our level of understanding and what priority decisions we're
making with regard to customer impact or you know, system stability or what have
you. Sometimes incidents are not resolved for days after you know the actual incident.
I've especially for for large, complex incidents. Sometimes you just have to
get things to a steady state and let them stay there until you have chance
to enroll more folks or get a deeper understanding of what's going on. And
sometimes those fixes are not things you can roll out, you know, as
one hot fix. Sometimes they are major upgrades or major changes to kind of
foundational business logic. So I'm glad you made that distinction because they're they're really
important, and I think oftentimes folks outside of the incident are just like,
when are we mitigated? When we mitigated? But you can't you can't lose
sight of that, that time frame between mitigation and resolution, because that's where
a lot of the you know, exploratory understanding comes out for sure. And
one of the things that I try to insist on is that mitigating the issue,
were allowed to make live changes in production, but the actual root cost
fixed has to go through our normal development cycle of making the changes in DEV,
pushing the changes to a staging environment, validating them, and then promoting
those changes to production. So it has to follow that flow. Yeah,
and that's that goes back to that prioritization opportunity. Right. So once you've
kind of addressed the business impacting issue, then you've got to get back to
your fundamentals, right, and your business processes and compliance and all of that.
And so detangling those two things allows you to respond in a way that
helps the business, and then address the issue in a way that helps the
business, and do those in different ways, because especially when you're when you're
further along in your maturity model, when you're a large organization, there's a
lot of things that can hands stand in the way of quickly addressing an issue.
Right. If you don't create a path for doing that, then incidents
end up taking longer and having a lot more impact. So yeah, and
the other thing we've learned in all of this is that every organization is different,
Right. Some organizations have response processes that specifically call out different ways of
mitigating impacting issues and different ways of capturing follow ups for those. Right,
Sometimes the incident's not closed until you've resolved it, and sometimes it's closed at
the point that it's mitigated and you've captured the follow ups you want to take
action on. You know. As a result, sometimes folks keep talking about
the incident after it's been closed and they want all of that for their post
incident learning review as well. There's just so many different ways to tailor this
whole incident management process to help an organization be more successful. Yeah, one
of the places I worked years ago was is that a healthcare provider, and
we did we provided medical services for hospitals across the US for trauma patients,
and so every incident that we had, whenever we broke out an incident room,
we actually had a person from our quality team who would join the call
as well and let us know, like every five or ten minutes, how
many patients across the United States couldn't receive life saving healthcare because our stuff was
broken. And so we had a very unique incident response model there that doesn't
really apply anywhere I've been since then, but there were still lessons that I've
taken away from that, you know, number one is mitigate the issue as
possible. Right, I'm so interested to hear how how did that information help
or hinder mitigation for your teams. It really set the priority and kept us
focused, you know, because as that number went up, you started to
understand, you know, the impact that this was having. And this was
not a other development team sucks or their network is terrible, or and many
many of our incidents it was because of user error at one of the trauma
centers. But it's still not okay to say, oh, well, they're
just doing it wrong, because you have to realize at the same time,
you know, while you're on the phone with that person, they're up on
a table in the emergency room doing chess compressions on this patient. So they're
going to give it their best shot, but they may not be the most
attentive user at that time, and you just got to work with that.
Yeah, you're you're highlighting like a perfect example of I think why we are
so focused on post incident learning, and it's because the most important aspect of
these complex technical systems that we're all building and maintaining are the people involved,
Right, and when you're in an incident response room, a major incident room,
whatever, and you've got someone reminding you of the impact, especially when
that impact is you know, not just on dollars, but also on people's
lives. You create the conditions for this like profound human creativity, right in
terms of figuring out, you know, what can we do as a team
to kind of we're back to the incident management space, what can we do
as a team to kind of come up with a creative solution here and get
us back to good, you know, temporarily. And I think if you're
not reflecting on and talking about those moments in incident response and your you know,
postings in a learning review or narrative review, whatever you call it,
and you're missing out on all of those examples of the ways in which the
people are helping support the system and keep things moving. You know, we
hear a lot in tech and DevOps and elsewhere that like automation is the key
to sustainability and more reliable systems. And there are things that we can automate,
you know, especially assigning roles during incident management and response. But there's
a lot of you know, human involvement tweaking the system and adding you know
capacity, not you know, technical capacity in terms of number of network requests.
You can handle things like that, but adding capacity in terms of the
system's adaptability. And I just like, I would love to be a fly
on the wall for one of those incidents that you mentioned, because I imagine
folks really came together and came up with some creative solutions to find a way
to mitigate those incidents and get things back to good so that they could figure
out, you know, what the long term solutions were. That's such an
exciting like space, Yeah, for sure, and it's you know, it
was a majority of the role was communication. Like all of all the my
coworkers there had exceptional technical skills, but their communication skills were just a plus
one on top of that. And I think that's what made it work so
well. And I still say that to this day. You know, DevOps
is not a technical world. There's a technical comm component, but it really
is communications in building the technical framework, but then communicating that out to your
customers, the engineers that you support, and getting the feedback from them to
understand what's the difference between what I built and what they thought I built.
Yeah, it's It's really great when you have those folks who kind of know
how to be in a critical situation and maintain you know, effective communication and
find a solution to the issue. One thing we talk a lot about is
like how do you scale that, how do you how do you externalize those
skills? Oftentimes we find that the folks who are most effective and inti and
don't have the capacity or time to help up level or train folks into that
discipline. Right. It kind of requires a lot of different skills. You
need a technical expertise, you need experience with the systems involved, and you
need a good handle on like effective communication, not just for communicating the status
of the incident, but also communicating with the folks that you are directing if
you're in an incident commander role for example. There's another area where if you
invest in learning from these things, you can create artifacts that folks pick up
when they join the organization. Right in almost every large org I've been,
there's a confluence space or Google drive folder is something full of post incident reviews.
Sometimes I'll just go in and read those, right, and you start
to learn, you know, who are the folks who demonstrate an ability to
kind of respond to some of the most significant incidents and what are they doing,
how are they doing that right? What skills or actions have they taken
that stood out in the learning room review should I try and cultivate as a
responder? And so that that can be a really interesting space too, is
not just learning about the system and what things we can change to improve performance
of at the time, but how are we leaving breadcrumbs for the new folks
coming into the org who are growing into that discipline, because trial by fire
during a major incident can be a really stressful, kind of terrifying experience,
and so the more you can kind of give, you know, these these
anecdotal or story based accounts of how things go in your organization, more comfortable
folks and feel when they step into that role. Yeah, I think it's
one of those areas where there's like a mentoring path there. And as I
have gotten older and been doing this for a while, I've realized that that's
that's a larger part of my job is sharing that that context because you can
put the documentation, but then there's also like the unspoken or the unwritten part
of that. You know, there's the mood, the field the context of
the situation. And I think that's been a problem for you know, far
beyond my lifetime, and the only way we've been successful at solving it now
up to this point is just through that mentoring type role where you bring people
in even though you know that they aren't ready to be the lead in this,
you bring them in just so that they can can witness it and start
making notes for themselves. Yeah, and that's where a process or a policy
around incident response and incident learning that is based on transparency can be really helpful.
Right, Sometimes you get a lot of folks joining the major incident room
that are trying to contribute in ways that may not actually you know, help
with mitigation. But a lot of times we find in large organizations that have
you know, policies angle toward transparency, folks just joined to kind of understand
and learn in the moment and also after the fact. So, you know,
the the incident learning review calendar is always a place that I go and
try to figure out, you know, which which of these incidents are going
to be most helpful for me understanding the way this organization operates and the critical
systems. Right in the past, role we had a COFCA platform that was
you know, involved in a lot of incidents, not because the COFCA platform
was a problem, but because everything was built around it, right, So
every time there was an issue with any system, that kind of tied back
to there. And that presents a really interesting lens for you know, how
do these folks communicate with the low broader org and what changes are we making
to shore up some of those critical dependencies, And you know, just being
able to join a conversation about that, not having been involved in response or
having anything to do with the teams involved, can be a really powerful opportunity
for you to kind of learn about the team that you're working with and the
underlying technologies. Especially for folks like me, it's been eight years since I
was you know, maintaining those types of platforms and so picking up on some
of that nuance so that I can support the folks who are around those systems
can be really helpful. There's a line there, though, You've got to
make sure that expectations are clear, right. If you're participating in something for
the purpose of learning, you're kind of a sponge rather than someone who's bringing
opinions, you know, not having understood the circumstances of the specific incident.
So you need a healthy kind of culture and set of expectations around this.
But I've seen a lot of orgs that do it well, and it is
a game changer, you know, for for helping to provide you know,
scalable mentorship and opportunities for folks to get a better understanding of the details.
Yeah. One of the things you commented on that I think just can't be
elaborated enough is transparen see. And I've worked in multiple places, and when
I first started my career, it was it was in many instances a fireable
event if you created an incident, and for that reason, people would try
to hide and cover up their incidents, which led to no one learning from
that. And these days, you know, I almost paraded around you know,
hey, I broke this because there's a learning opportunity there. And I
think it's really important to be open and to build the environment where people aren't
afraid to say that they made mistakes, and even the dumb mistakes, we
all do them, you know, you learn from it. And I actually,
at some point in my career a boss of mine told me, and
it's an anecdotal story, but it's still effective. Someone created an incident cost
several hundred thousand dollars and said, oh am, I going to be fired
now, and the responded, no, I just spent two hundred thousand dollars
on your education. Why would I fire you now? Yeah? And this
is where I think, like, it's really difficult to build trust, right,
It's really easy to damage trust, it's really difficult to build it.
And so if you're approaching your your incident management, you know, life cycle
and process from the perspective of trying to support folks doing what they can to
help the business be successful, you get a lot of really impactful contribution and
collaboration with regard to you know, keeping systems healthy and things like that.
But if you over index on you know, the measurable metrics. We're humans,
right, every every human will gain a measure Right, you start to
cultivate some of those types of environments where you know, what's the consequence of
me doing the right thing here? Is is it going to reflect poorly on
me? Is it going to cause an issue? And so, thankfully,
I think every organization that the Jelly has worked with over the past two and
a half years since I joined two and a half plus years, they've taken
the approach that, yeah, these are these are blame aware learning reviews.
Right. We know that folks make mistakes, that they don't have sufficient context
in the moment, and that they can learn from those experiences and change their
approach next time, versus this kind of you know, older model we'll say,
of prioritizing the the the you know, public visibility of how things are
going, and maybe like maybe we don't declare an incident for that one,
we just try to fix it quickly. Early in my career, I was
learning how to use Microsoft SEQL databases and we had a large share point site.
It was another medical audit company, and I learned what drop database commands
do, and I did the hire production database and fortunately I had enough experience
to quickly restore it before anyone noticed. But that was an environment where I
didn't feel comfortable, you know, broadcasting that I had just seen in the
process of learning some new commands stopped the entire database. So yeah, it
can be a tricky balance, but you know, some light is the best
medicine, right. Transparency in these types of environments allow folks to do what's
necessary to get things back to good And I think the more you can kind
of socialize and demonstrate that transparency, the more effective your organization is going to
be, and the more folks are going to want to contribute to that mission,
whatever it is. Yeah, yeah, absolutely agreed. So let's talk
a little bit about what's going on with Jelly these days. Yeah, So
Jelly has been like the most interesting experience of my career. I think I
mentioned I joined in twenty twenty one. I think it was Jelly was just
a post incident analysis tool at that time. So we had this notion of
building narratives and not much else, and we recognized that part of the post
incident learning process involves having good data, and the way that you get good
data is you get consistent in your process. And so we ended up building
this incident response bot and we also went to the other end of the spectrum
and started introducing features for cross incident analysis. And so this is, you
know, after an incident, let's spend some time learning, but then how
do we roll up those learnings into themes across incidents that help the organization make
decisions around growing teams to support services or changing direction with regard to build versus
buy those sorts of things. And so we've been working on a lot of
cool stuff for the last two and a half years. And then in what
was it, I think November seconds the public announcement that we were merging with
Patrie Duty went out, which has been like really exciting and also a crying
experience has been a month, right, And so page Duty is something like
eleven hundred employees as of January of this year, we were twenty one.
We're kind of in the process of figuring out how to bridge those two divides.
And one thing that I'm really excited about is, you know, Jelly
has spent a lot of time differentiating itself as a product in the postings and
learning area, and I think we've brought a lot of kind of novel approaches
and opinions students that response in general. Patri Duty has been doing this for
fourteen plus years, right they and they created the category within which Jelry could
become a company, which is pretty cool. And so what we're looking to
do now is to take that practice, you know, post incident learning really
get folks from the earlier phases of the maturity level where they're just doing incident
response and maybe they're doing a post incident learning review on a Google doc,
and bring them into the modern right and start creating incident narratives and doing learning
reviews. Page of Duty has something like twenty seven thousand free and paid customers.
There's a huge opportunity there to help folks understand a better way of kind
of benefiting from all. So that's my focus right now is figuring out how
do we bring those two worlds together while keeping an eye on preserving that kind
of post incident learning tooling and opportunity. But yeah, a lot a lot
of exciting stuff on the horizon we are. We are going into a new
year, so I think things will look very different on the Page of Duty
side and probably also improve on the Jelly side as well. It's going to
be it's going to be really interesting. Yeah. I think it's a natural
fit, you know, because Page your Duty is hands down a great tool
for notifying people that there's something requesting their attention, but what you do after
that is kind of up to you, and so it seems like a natural
fit to just roll that right into into Jelly and and help help people like
just from a business perspective, take this huge page your Duty customer base and
just guide them into the thing that they thought they were doing all along.
Yeah, one one focus for us has always been, you know, how
can we improve the quality of our customers' postings and learning reviews, and how
can we allow the folks conducting those investigations to focus on what matters. We've
talked to organizations where, you know, there was one problem manager at a
a company that used Microsoft teams, and part of their job was to go
through every team's channel and find transcripts associated with an incident and put them in
service. Now, nobody should be doing that, right, That's just toil.
That's that's not productive. And so one thing I'm especially excited for with
this partnership with page Duty is or this this acquisition by paye Duty, is
they've got a ton of data, right, And so when you're building your
post incident narratives, your timeline, and you're adding evidence and you're trying to
help folks understand the details of an incident, the more data you have to
substantiate those claims and those events that you're highlighting in the incident, the more
folks can learn from you know, the not only the overall shape of the
incident, but the systems involved and how they're used to understand you know,
the underlying technology. And so there's an element there that's really exciting, which
is just we have a lot more data to allow our our users to work
with. But I also think, like I said, Page Duty has been
known for a really long time, UH as kind of an industry leader in
scheduling and alearning. Right Uh act and bail I got paged. I'm gonna
go fix it. Uh. There is a better way, right, Like,
there are ways to tie that process into the incident response process and the
postings of the view and I think that's that's going to be our focus over
the next you know, several months, is figuring out how do we give
pager Duty more mechanisms for supporting responders throughout the entire incident management life cycle,
not just the detection phase, which a lot of folks know and they're familiar
with, but you know, Page Duty's full operations cloud, which most folks
I've talked to don't even know exists. Uh. And and this is you
know, the the AI automation for reducing noise to signal when it comes to
events. This is all of the mechanisms around running actual incidents, and then
this is the post incident as well. Pad has a feature today called post
mortems, which is fairly straightforward. It's your your Google post mortem doc.
But we think there's a lot of opportunity to not require that folks are going
and creating these data sets manually, but just kind of provide that information so
they can use it to better narratives that are living all the things. And
yeah, I think I think it's a natural fit too. I mean,
I've been using page of duty for basically my entire career, right and being
able to bridge that gap between that paging and scheduling and then the things that
I need to do to help my team be successful, it's going to be
huge, you know, for from my experience. Yeah, I think having
access to that data is going to lead to better collaboration after the fact,
because that's for me, I've always struggled with that. You know, after
the incident's over and you're trying to do the review of it, trying to
remember what things happened in what order and remember all of those steps that you
took. So if you've got something that can can prompt you with reminders and
kind of pre populate that narrative for you. I think it's just going to
lead to better, better results at the end. Yeah, there is nothing
better than having a starting point when you are trying to investigate an incident.
Right, If you open an empty Google doc, it's a hard time.
But if you can start with you know, in Jelly today, you start
with the incident transcript, all the conversation that happened in slack and data about
who is involved, so much better than starting from nothing. And that's especially
true when you know your incident response process uses multiple data sources like multiple incident
response channels or your data dog charts or what have you. So we're not
really looking to do the post incident narrative for you. We're looking to give
you a point to start from because that saves time, it saves energy,
and let's you focus on the things that only you can create within your post
incident narrative. Right, the investigator is a conduit through which the folks who
are involved in the organizational miscellany kind of come together into a coherent story about
what happens and what it means. So we really want to like provide a
foundation on top of which folks can have these conversations. And I think there's
a lot of opportunity there with this kind of broader spectrum of data and integrations
within customer's existing processes. Yeah, it reminds me a lot of like a
there's a like a a people skill there. You put two people who don't
know each other in a room and anything could possibly happen. They could strick
up conversation, they could sit there in silence. You know, there's just
no way to gauge it. But then if you give them a conversation starter,
then you can sort of like guide the results from there. And I
think I think that's the real value of what the post incident narrative does,
is it's that conversation starter. Yeah, that I mean certainly for us,
as you mentioned, like we use Jelly internally and we do our own learning
reviews. I think the exercise of you know, mitigating the incident, putting
together the learning review, those are valuable experiences for the folks involved. But
getting everyone in the company, because we can do that at twenty one people
into a room to talk about what happened, to ask questions to figure out
what did you know? What did you not know? You know? What
did I know? And I wasn't involved those those sorts of things. That's
where you get really interesting kind of exponential increases and understand And it's not just
the thing that most excites me about these learning reviews is it's not just the
understanding of the technical or the organizational process. It's the understanding of each other.
Right, How how I communicate in these environments? How you communicate what
your expectations are, what sorts of things I need to be better about informing
during response. It's a it's a retro right, and the software can't operate
itself. And so if the people are working effectively together, then the software
is working effectively and if they're not, then it's not. And I think
that's that's one of the really big opportunities, especially for you know, the
large complex organizations in novel economic environments, to figure out, you know,
how do we improve our efficiency in our collaboration so that we can do what
needs to be done? Really exciting, Oh, it is really exciting.
I'm looking forward to seeing how this plays out for y'all. Yeah, I'll
have to let you out there's you know, we're in a phase right now
where there are too many good things for us to do, so we got
to figure out the next best thing and focus on that. But yeah,
that's that's the spot I want to be in. Endless opportunity ahead of us.
We just got to figure out how we're going to get that to our
customers as quickly as possible. Yeah, for sure. Yeah cool. So,
anything else you'd like to share with us about incident response, Jelly,
page of Duty, any topics at all. Yeah, if you're not already
using Page of Duty, take a look. It's the best thing for paging
that I've ever found. And if you want to give Jelly a try,
there's a free trial on the site and we start you off with some pre
built learning reviews so you can see what they look like. Start playing around
in there, and if you have any questions, you know, I'm sure
you'll be able to find me in the show notes here. But it's been
really great to meet you, Will and thank you so much for opportunity to
chat. No, thank you, it's been a great conversation. I've enjoyed
it. And uh, if you're up for it, I would love to
have you back on the show'd that'd be great? All right? So much?
All right? Cool? Well, thanks for listening everyone, and I
will see y'all next week. M