Speaker 1: And welcome back to another episode of Adventures and DevOps.
One of the things that I've talked with many of
my colleagues about is just how it seems like there's
a ramp up in the number of incidents, production or otherwise.
Speaker 2: They've had to deal with.
Speaker 1: So today I've brought in an expert in the industry,
Lawrence Jones, founding engineer at Incident IOW and before that,
if I'm right, principal site reliability engineer at go card List,
which probably explains a lot about how you end up
building Incident IOWE in the first place.
Speaker 3: Yeah, No, that's absolutely it. Yeah, So I think we
always we always choke it Incident about just how many
of our early founding team were sourced from fintech. Well
that might mean about the experience of working at a
fintech company, but no, I think.
Speaker 4: Yeah.
Speaker 3: I joined Incident about four years ago. I was the
first hire and joined alongside Pete, Stephen and Chris who
are the founders. Between the five of us, we've seen
love variety of incidents, both big and small, in financial
regulation and then also infrastructure and normal technical incidents, and
that serves you very well. It turns out when you're
trying to build an incident response platform.
Speaker 1: Do you think there's something specific about the financial industry
that lends itself more to having incidents that cause bigger problems?
Or is it everyone experiences incidents, no matter what vertical
or area they're in, and it's just something that stuck
for you as a problem that needed to be solved.
Speaker 3: So I think in infintech you have an extremely pressing concern, right,
which is the money, and managing finances is something that
is both regulatorily governed, so you have laws that tell
you how you should respond in certain situations, and you
have obligations to certain regulatory bodies that you respond in
that way, and that brings a level of rigor and
discipline that you need for your incident response that you
might not find in other companies. I remember the running
joke with my SA team at the time, many years
ago now at go cutas was the next job that
we would be doing is a social media network for cats,
because they presumably wouldn't care quite as much whenever we
had downtime. When you can trust that with like a
payments gateway where every minute of downtime is literally all
of your customers can't take payments from their customers. The
type of stress that goes into those incidents, it can
get quite big.
Speaker 2: From experience.
Speaker 1: The types of people that maybe on the cats and
would report to your support, you know, could be much
more angry, like magnitudes more angrier than than your customers
who aren't able to you know, build a couple of
their customers.
Speaker 3: I mean, given that I ended up going straight from
goat cutlers into incident I OWE, and now I'm on
the pager for our on call system, which is just
as if not more critically availability sensitive than a payment skateway,
I think maybe the cat social media network can be
my next venture.
Speaker 1: Yeah, a couple more years, and this is you'll do
that adventure and hopefully retire from there more of a
hobby project, right.
Speaker 3: I think fintech is just one of those areas where
not just like incidents in another in another environment, and
maybe the website has gone down and after it's resolved
and come back up, the incident is broadly okay, but
there's a huge amount of background and other work that
comes out of an incident that happens in fintech, and
I think that requires you to build up a lot
more in terms of your incident process, so that you
can track work even after the initial impact is over,
after you've stopped the bleeding. Yeah, you need to go
chase all your customers, you need to go inform people.
You need to do all of this, and the penalties
if you don't do it are quite harsh. So you
end up building good muscles around how to run incidents
as a result.
Speaker 2: So I have to ask this incident.
Speaker 1: I like, you're not focused on only financial customers though, right,
I assume there's no branding around that, but maybe I'm no. No.
Speaker 3: So incident is a generic incident response platform, so you
can think of us as so are like most most
probably recognizable customers are people.
Speaker 4: Like Netflix, etc. Skyscanner.
Speaker 3: They use our tool so that they get paid when
something goes wrong, and then when something goes wrong, they
end up running their incident through our system. So the
whole value of incident is that we're allowing our customers
to encode their incident response process into the tool so
that we can help them run the process without skipping
a beat. Basically, and then, at least most recently over
the last year in a bit, we've been looking at
how we can use AI to help our customers actually
solve the incident for them more debugging what's actually gone on.
When you receive an alert that says, hey, you've got
some network failure out over here, but we can appropriately
explore all of the different parts of your infrastructure and go, hey,
it's not actually this service that's going wrong. Maybe the
database over here is out of resource due to a
bad quick plan, and that's the reason that you're seeing
requests go wrong over here. That's the journey that we've taken.
And it's not just that there are financial companies using us,
so we do have a lot. It's mostly people who
care a lot about what happens when an incident happens,
so making sure that they cut their down time as
much as possible, and people who have like concerns such
as we need to be really openly transparent with our
customers and make sure that we communicate with them promptly
and run the incident in a way that they would
enjoy if the or maybe enjoys or word a way
that they would appreciate if they were themselves their customers.
Speaker 1: The reason I asked about the customer distribution is because
I'm curious whether or not you see patterns by industry,
or whether or not, say, handling incidents is something that
becomes more undifferentiated work, sort of like doing software development
at companies, Or whether or not the amount of regulations
or the complexity of the vertical segment, or even the
types of customers that a business has to deal with
changes the way in which they handle incidents.
Speaker 3: Yeah, so I think the answer is both yes and no,
perhaps unsurprisingly. I think as you become more mature as
a software engineer, you learn how to engage with an
incident response process more effectively. But each organization has very
different requirements or needs when it comes to incidents. If
you work in fintech or a similar area, then you're
often engaging with your legal team to try and figure
out what your obligations are if you ever have a
financial breach or something like that. But equally, as you
start scaling into like really really large large enterprises, so
people who have hundreds of thousands of cuss some of
which themselves are large enterprises, and they have slas with
them that are really stringent, and those customers care a
lot about them. At that point, you're engaging with different
parts of the business. Again, so maybe your GTM team
and your customer success team, and just like in fintech,
you end up with financial penalties if things go wrong.
So it depends on your context as a company, but
definitely as you scale upwards, you get much more serious
in terms of how you approach these problems, and even
across industries you see kind of broad patterns played out right.
So I think everyone, as an example, I think, starts
off with a status page where just transparently talking about
every incident as soon as it happens on a status
page is really appreciated from your customers and something good
to do. Gradually, as you broaden your customer base and
hopefully as you build resilience into the system, every incident
does not affect every customer and that can complicate your
relationship if you're publishing everything, so you look for more
sophistication and how you're running your response. That is equally
the same type of challenge that you have when you
have a regulatory breach and you need to inform a
certain subset of customers.
Speaker 1: A challenge to find the balanced area of transparency and
also solving the problem effectively. I know you're not specifically
in this space, but for us, the challenges obviously if
there is a problem with one of our services, and
where I don't want to say, we're in the security space.
We're providing logging and access control. So if there is
a problem, then making sure that we're rolling out solutions
or fixes or communication to just the customers that are
trustworthy is more important than just rolling it across, you know,
exposing that problem and then potentially having malicious customers of ours,
you know, attempting to do something with that malicious data.
And I know you probably don't have that particular scenario,
but it's an easy way of seeing that. You want
to focus the communication to the audience that makes the
most sense, and at scale, it just doesn't make sense
to send the same message to everyone.
Speaker 3: Well, if you just want to inform people as soon
as you possibly can, if you're not considerate with who
you're informing, as your customer list grows much much larger,
informing everyone about an incident where only ten percent of
the people you're informing maybe impacted, arguably you can cause
far more harm than good, like literally looking at like
the social calculus of worrying ten times as many people
as you need about it. That's not relevant to them
is also not good. So yeah, it get it gets
a lot more complicated as you scale up it.
Speaker 2: As you say, are you in an AWS GCP asure?
Speaker 3: So we run across a couple of different providers, but
primarily inside of GCP.
Speaker 1: So I actually don't know how a lot of experience
with the operational support of running technology and GCP. But
in AWS we get a lot of email saying like
there is a required action to be executed, and you
go into an email and the requirement is, oh, we're
some setting this product which you have never used in
any account in your entire organization, and so it definitely
has like a negative impact on the brand even or
you just send out these communications which aren't valuable and
they waste cycles on having someone have toil even filter them.
But read the email, understand that has I like to
do with them before even jumping into it.
Speaker 3: Be a hit and miss on this. I mean famously,
Google love deprecating things. I argue whether not that's good
or bad. But one thing that they do do in
their communication are they usually quite good at going through
all the different services that you're using and they'll only
notify you if they're pretty confident that they've seen activity
on the API that they think that they're going to
be changing. So often, Actually, my experience with GDP is
it's usually very actionable. They can go if I'm telling
you about this, it's because I've seen that you used
it in the last seven thirty days, and that usually
clears up a lot of these communications. But I have
much less experience running in EDIS. We were GCP primarily
inside of Goat Carlis, and we've been primarily GDP in incidents,
so they're my Yeah.
Speaker 1: I assume that since you brought that over and you're
the founding engineer, that was pretty much your decision.
Speaker 3: When I originally turned up, it was Stephen and Kris
have been paying together the MVP of what incident IOWA was. Honestly,
our architecture hasn't changed so much in that time. It's
evolved and become a lot more robust and mature. But
even now we are one big go modolithic binary that
we end up deploying in what you term a modular
monolith set up. But we're doing that inside of Cubernettes
now where when I first turned up it was just
one go binary but running as a single process with
all of the workers, all of the web stuff inside
of Hiroki, which was quite uncomfortable when like we had
our first obviously very predictable incident of like a worker.
Speaker 4: Going wrong that would bring down the entire.
Speaker 3: Service, and we were very quick to adjust as we went.
But I guess this is actually one of the benefits
of having seen the growth of a company like Cardlists
before and then coming in being the person kind of
responsible for revolving the infrastructure inside of the incident. It's
that you can go, well, I know that this thing
is going to happen at a certain point, and you
can be very ready for those adaptations. And I think
that's part of what helped us scale very effectively in
the first two years. It was making sure that we
only applied the right level of process and the right
level of sophistication when we thought that we needed it.
Speaker 1: I want to dive into that if something your at
stack was using Heroko. My experience in the past has
been at like deep levels. It's always surprised me how
more successful companies can buy in a utilization going forward,
because often it's missing some of those edge case services
which then become critical. I see these things like secrets management,
especially like anything related to cryptography. I just haven't seen
Heroku be able to support that. Well, so you were
on there, how did you actually decide, like, how did
what was the turning point to actually make the decision
to switch over and really go for one of the hyperscalars.
Speaker 3: I think exactly what you what you've mentioned is honestly
the thing that pushed us the most.
Speaker 4: Well.
Speaker 3: We were running in Heroku. The application at time was
this Go monolith and it was running on a Heroku
Service Postgress instance. And that hasn't I mean that architecture
hasn't changed. We still have one big postgress database that
we look after in a much more mature way now,
but that's in cloud sequel. But even when we first
started pops up was the way that the application did
asynchronous message processing, and that that actually.
Speaker 4: Worked really really well.
Speaker 3: But it meant that we were already kind of half
and half in both Heroku and in GCP. And I
think that the thing for me that made it very
obvious that we were going to move was what you
said about Heroku lacking various primitives that you want when
you want to become more mature, I mean, especially when
you consider we have some horrible percentage of the world's
like GDP is indirectly locked up in the companies that
use us. At the moment, we've got some really really
big customers and they're running their incidence in our platform,
so we have a huge obligation to make sure that
those things are secure and running. In Heroku, where you
have just standard like to our factor author and like
you're setting up environment variables, it's not the level of
sophistication and security that you would want. So gradually, I
think we were piecemeal moving things into what I had
spent the first couple of weeks at incidents setting up
as a very sensible GCP environment with all the right
security primitives and things like that. But we ended up
gradually moving to GCP's secret Manager, where even though we
were running in Heroku, we would have our app using
a security perimeter and service accounts that were authorized just
from the Heroku environment decrypt the secrets on the fly
as and when they wanted to use them, which was
a nice way of us grafting ourselves to a much
more secure GCP primitive without abandoning all.
Speaker 4: The stuff that you would normally get inside of Hiroki.
Speaker 3: And we did the same, gradually moving our work clothes
over from Heroki into Kubernetes inside the GCP and then
finding a way to split the traffic across and moving
over the workers and then eventually moving over the database too.
But yeah, I think there's always a push if you're
ever in a Heroku environment or something like that, where
you're just going I want a bit more control over this,
or I want a bit more visibility, and I'm willing
to pay the cost now because we're much larger, having
a dedicated infrastructure engineers, so you can look after this
and make a nice like paved path to production so
that people can deploy things effectively.
Speaker 1: I actually see it as a cost reduction mechanism. When
you are given the primitives that work out of the box,
you don't have to build complicated technology to say cross clouds,
because even in your example using the GP Secret Manager
and having service clients that have direct access there, you
still have to deploy this secrets. Whether I think in
GCP there are databut's or they're Jason blobs that have
the certificate embedded in order to actually generate databut's to
be sent to the server. But how do you even
secure those and get those deployed into the rocal environment
because if I recall and still true, maybe recently, there's
no workload entity identifiers for the individual workloads that are
running in Heroku, so there's no way to really secure
those payloads locally to just the machine that's actually relevant.
Speaker 3: Yeah, absolutely, and I think that this was part of
the migration path that we came up with. So the
first thing that we did was we created a security
perimeter and GCP that allowed secret Manager API access only
from a specific group of ips in Heroica. Interesting, that
is not anywhere near as secure as workload identity, for example,
But if you're looking at a halfway house, where you go, actually,
I want to be able to pull this stuff into
a different environment that's more secure and kind of like
oh my way gradually into more of the sophistication that
you get from a hyperscala that's a legitimate pathway to
getting there. And now we're in a be an environment
that is properly locked down and like, as you say,
using all the primitives that allow you to best leverage
the security tools that you get from the provider.
Speaker 1: There was a little hesitation I think early on when
you were answering which EXTAC you were using. And I
don't know if that was because you have workloads running
in the other our providers. Is that as backup strategy
or something.
Speaker 3: Like that, we want to prioritize on using few tools
and knowing them extremely well. And for us, that means
our disaster recovery plan has like a couple of different
phases to it. But what we do is we end
up running all of our infrastructure inside of primarily on region,
which involves several different data data centers inside of Google,
So all of our workloads in like the hot primary cluster,
they're spread across three different data centers in one region.
We then have a disaster recovery plan that has another
region that we will fall back to if that ever
goes wrong. That region is also within GCP. But we
also have various different other requirements. For example, some of
our work with customers in China require us to use
different types of infrastructure to send title of communications over
to them, So we have a ton of different constraints
that mean that we want to be running with backups
that span outside of just GCP now directionally izink. Over
the next year, we'll end up running our work clothes
across a variety of different providers. Again, I think that
we'll be prioritizing GCP and a multi regional backup inside
of GCP.
Speaker 4: Is the way that we want to run things.
Speaker 2: Now, based on the name of the company and the product.
Speaker 1: I'm guessing you're and what you've said so far, it's
really a focus on the incident management and I'll call
the standard run books organizational run books for.
Speaker 2: How to deal with that.
Speaker 1: But I feel like you've expanded outside of that, and
now I would see it as maybe else just for context,
I say things like fire Hose and Pajer Duty. Are
you would you say you're like directly in competition with
those because I know early on companies like Page of Duty,
they really focused on handling the event and alerting or
when there was an incident, but not necessary handling the flow.
And I don't remember ever coming up with a run
book and throwing it in you know, third party other tools.
Speaker 2: All our run books.
Speaker 1: Would be in a and I'm sure I'm going to
get some angry emails for this and question of my
life choices, but we were using confluence.
Speaker 2: I think I think what what I'll think about that is,
have you hat to find a better tool?
Speaker 1: It's the least bad one, but maybe you can add
some clarity to sort of how you're thinking about it,
because I know for bigger companies, as you said, if
you're trying to apply a policy because of either compliance
or because of regulation, then following the run books that
you're creating it at the organizational level is critical for
resolving the incident, which is of course separate from whether
or not how you know there's an incident and what
you communicate to your customers.
Speaker 3: The full focus of the company and why incident came
to be was it fell to us that the providers
that already existed to provide incident response and tooling were
focused kind of almost to the exclusion of anything else,
on just how to tell you there is something wrong.
And it's kind of this horrible sense that the efforts
start to the point that your phone went off, And
for people who've been paged for hundreds, maybe thousands of
incidents at this point, the reverse could not be more true,
which is that all the work starts at the point
that your phone actually starts ringing. So incident Io was
focused on initially focused on the attempt to help you
beyond the point where you initially got paged, and at
the start of the company was to plug into providers
like page duty or upstene, so that at the point
where you got Paige, we pull you into a Slack
channel and it was at that point that we take
you through the journey of actually trying to resolve it
and caring about things like tracking all the actions that
are going on on the incident, making sure that when
you're coordinating with people there's just one place to go
to talk about what's going on in the incident. As
the company matured, we've kind of built out various different
other kind of adjacent services that we think our core
to incident response but weren't always packaged with those providers
out there. So things that we've built so status pages.
One of the biggest customers using our stats of page
at the moment is open Ai. So if they ever
have an outage, open Ai going to post an incident
on those station's page, which ends up informing their customers
about we've got some issues with this model, and then
they can prioritize what they're doing. So that's a lot
about helping walk your customers through the coordination of an incident.
But we've also built out the on call aspect of
incident response to so it's now the case that you
don't need a page duc, you.
Speaker 4: Don't need an opstiny.
Speaker 3: We have the mobile app, we have all the telecommunications,
so will help you set up your schedules and make
sure that you're doing all the things that we think
are really healthy, such as if you've had a really
busy night on the pager, we'll propose to people on
the schedule that maybe they should take your shift the
next day so that you can get some rest and
recover and recuperate, rather than just keeping the same person
on throughout the week until they're run ragged. Most recently,
the thing that has been interesting in this area for
me is so I lead AI Incident. So we have
a team working on how to leverage AI to best
help people resolve their incidents, and we have a product
called Aisri which is aiming to help in that space.
And so what we do now whenever you have an
incident is we will look the alert and we'll set
our system off to go crawl a bunch of different things,
say looking at all of your GitHub pull requests to say,
i've found this alert, does anything.
Speaker 4: Look like it might?
Speaker 3: Of course, this alert based on the diffs of the
code that we can see in the timing around when
it was deployed, but we'll also check stuff in your
sack work space, and we'll also look at all of
your past incident data. And one of the most useful
things of this product is that we actually pull together
organically from the history of all your old incidents, how
you've responded to issues like this in the past, and
what we do is we end up combining this ephemeral
run book that say it's actually, given what I can
see that you've done in the past and what worked
and what didn't, and even looking at your post mortems
on what you said worked and what you missed, this
is actually what we think that you should do right now,
which can include things like, oh, this looks like a
data breach. You should be contacting your DPA, do you
want us to page them? Or maybe it's just by
the way, this service is really flaky. I'm pretty sure
this alert is actually not legit. You can run this
script to try and verify whether or not the thing
is actually true, and you can maybe even just ignore
this and.
Speaker 4: Go back to bed.
Speaker 3: But yeah, I thought that the run book stuff is
interesting is we have people who source their run books
in all sorts of places, and while they may give
a different answer on where they put their run books,
there is one consistent message with them, which is always
that the run books are always outdate, no matter what
you do.
Speaker 1: I think we've seen a lot of companies get wrong
the reason why they're creating a run book, and I'll
say that that reason is often I want to tell
someone else what to do when there is an incident.
Speaker 4: Gotcha.
Speaker 1: I think the problem there is the people with the
knowledge are trying to explain to someone who doesn't have
the knowledge in a critical or emergency situation. But the
correct thing to do is that's where the run book
has zero value. And the value you get out of
making a run book is making the run book and
understanding how to actually communicate you're really onto something they're
really critical, which is the fact that it's not about
what you think you should do.
Speaker 2: In that incident.
Speaker 1: It's learning more about the system, and you're doing that
programmatically through pulling in information from any number of sources
you mentioned. I'm really curious about the technical challenges in
actually achieving that. How do you actually go and pull
the information out of there?
Speaker 3: No, so, no, I mean, they're all really good questions.
So what we do is we connect to a variety
of different systems. That's kind of the prerogative of an
incident response still, because everything in the company kind of
falls downstream into an incident, so we already have access
into a lot of different places. Slacker is just one example,
And what we do is we allow you to connect
various channels into our system that say, hey, by the way,
here's a channel that will often contain interesting things that
might be relevant to you in an incident. So the
one that is often a really good example is like
an engineering channel. If you just have a shared public
SLIXE channel called engineering where you post, hey, by.
Speaker 4: The way, we've changed how we do things about deployments.
Speaker 3: We'll be watching that channel if you add it into
the system to go see, there's the thing I've learned.
So if I start seeing an incident that looks like
it's something to do with deployments, and it's happened just
a couple of hours after someone's gone and post it
into engineering. Then that's actually very relevant. So I'm going
to incorporate that into my findings and use it to
guide me. But there's a ton of other tools that
we connect to you. So we connect to get help,
like you said, so we'll look at recent code changes.
We'll also connect into telemetry, so we'll connect via Graffana
into your metrics logs and traces, and what we end
up doing is through a history of all the incidents
that you've had before, and also through a variety of
like background processing, we're continually trying to learn more about
how you as an organization have built your kind of
incident immune response, Like what are the things that you
do when you have an incident like this, so that
we can quickly guide an investigation to find or the
dashboards that you normally look at surface all the relevant
information inform you of anything someone said had changed recently.
And we do that through a combination of like honestly
a lot of quiet advanced like AI RAG.
Speaker 2: So yeah, no, absolutely, I just even want to go deeper.
Speaker 1: So the data is in my slack channels or in
my I don't know previous incidents wherever I'm recording that,
How does it get into a place that you can
actually utilize to identify what the new epheneral rom book
should be? Like, how do you decide what's relevant? Are
you a canonical thing as you said RAG? Are you
copying first level like what data seems like it could
be relevant into a set of RAG databases and then
using that at say runtime, whenever an incident happens to
query it for those pieces of data and use that
to then power some sort of search mechanism back on
the original source of data.
Speaker 4: Yeah, so it is exactly that.
Speaker 3: So we have this product called Catalog, which is your
service catalog and a load of other different organizational resources,
and so people use that to model their organization. So
we have a picture of all of your teams, all
of your services, all their dependencies. We also have a
picture for many people, like if you're B to B
maybe they'll they'll connect their CRM, so we'll know about
all the customers that you may have in your system.
So what this means is we can use that as
a knowledge graft to try and assemble all of the
resources that we know about your organization and then use
that to guide the investigation, where like you say, we're
continually background indexing the resources that we might need.
Speaker 1: Are you utilizing the APIs that are provided by a GitHub,
get lab, whatever it is to funnel their data in
or are you getting data from customers via I don't
know any number of xyz pipelines and are utilizing that.
For instance, I can imagine a lot of customers, especially
at scale, may already have all of that data having
been replicated into your redshifts or your snowflakes of the world.
Speaker 3: We have a variety of both native connections, so we
can connect directly to Salesforce or something like that, but
we also have this kind of universal adapter that I
wrote a while ago and has since been evolved a
lot that we call the Catalog Importer, and people can
run this and connect to their custom service catalog, maybe
if they've built it themselves, and they run it periodically
on a chrono and will pull down all their data
and sync it across. And that's how we end up
keeping our platform in sync with whatever your internal in
house solution might be. But when it comes to something
like GitHub and all the poor requests that you might
be making. That's us connecting over a GitHub integration listening
for webooks and going cool, we have a pr here.
Speaker 4: I'm going to pull that in.
Speaker 3: I'm going to have a look at the diff I'm
going to analyze it and figure out what are the
key things that have kind of changed. I'm going to
tag it keywords so that I can quickly find it
if something goes wrong. And we end up doing that
for all of the different types of data that we index.
But the cool thing is, I guess like this is
a bit that is very interesting to our platform is
that incidents themselves are amazing resources of this information. Are
probably one of the best logs that you have when
you end up taking the post mortims that people are writing,
and you take all of the activity that happened during
the incident channel and you try and join that up
with everything that you can then see in maybe Jira
or linear when you're opening follow up actions and things
like that. You get a really complete picture of everything
that's happened before and the state of your organization.
Speaker 1: There must be such a challenge to identify every single
source of data that could be utilized and correctly understand
the semantics of the data that's there so that it
can be utilized in the right way by I assume
you're using some sort of LAM how do you overcome
that challenge or you're just throwing all the data blog
you know, through your LM embedding model and just putting
in a RAG database and calling it done.
Speaker 3: So the answer to that is, like, yes, it is
a challenge, and if you run the naive approach, then
it isn't good enough to provide the level of accuracy
and actionable feedback that you want for an incident response
product like this. So the way that we look at
it is it's extremely harmful for you to turn up
at the start of an incident response channel and claim
that this is due to a problem that it is
then not the idea of like actual like inaccurate assertions
are really really distracting. It breaks trust in the system
that you've built. And it can also just be highly
negatively impactful send someone off on a real like wild
goose chase that doesn't return anything of value. Whilst if
we are who is improving this system for and whereas
it got worse because often you can't really change this
without at least something's getting worse at the same time,
there's a lot of other things getting better. So this
is everything from like having decent ev ol suites to
building a system of data sets of old incidents so
that we can rerun investigations on them, which also comes
with a bunch of challenges, like how do you run
an investigation as if it happened at a particular time
with only the information that you had back then, so
that you can then grade it to see if the
new system is better, but without you kind of accidentally
leaking information that happened after the incident back into the
back test.
Speaker 1: It's really interesting you're bringing that up, because actually I
don't It wasn't the most recent episode, but we had
Andrew Morlan on from chalk Ai and he was exactly
talking about what do you call time traveling where for
fraud detection actually, and it's a really interesting episode on
that topic, So we don't need to dive into it here,
but yeah, I totally get you have a really useful
set of information there, and I think they were doing
something similar basically in the fraud space in their database.
Speaker 2: What are you doing with this data that's coming in?
Speaker 1: Is it going into some proprietary RAG database or using
something off the shelf or a third party provider.
Speaker 4: Or something like that.
Speaker 3: So we're mostly indexing this into our primary POSTG database.
And what you can do is you can get a
really long way by just indexing this data with pre
process attributes that you can either use, so you can
use vector embeddings if you want. They have several advantages
and disadvantages. Postgress has PG vector that allows you to
look for vector similarity in your results set. Actually, we
found the embedding vectors aren't as easy to use and
aren't as reliably consistent as us just using tags. But yeah,
we do a ton of indexing things continuously attagging them
some vector embeddings, and then when we fetch all the
information back at the start of an investigation, we end
up unpacking all of that information and passing it back
through what we call a rerankor, which is a concept
that's quite familiar for people working with AI, but you
essentially unpack all of your long list of results and
then you pass them back into an LM to gradually
shortly to find the most compelling results from the.
Speaker 4: Longer list that you produced.
Speaker 3: And it's through doing this that we're able to search everything,
can return a pretty decent aggregate of all the targeted
resources within about a minute after the alerts fired, even
though that might be hundreds of thousands of requests we're
searching across.
Speaker 1: But you're solving a critical emergency moment problem for people.
But you're actually able to spend longer on that analysis
and almost generate something similar to the existing reasoning models
from the providers that are out there. They're running lms
that may take a longer period of time to generate stuff,
and I can imagine maybe you have some sort of
strategy for generating a first pass answer and then spending
more time in the background compiling longer challenged answers that
come up with using more tokens or pulling more data.
Speaker 4: Yeah, it's exactly that.
Speaker 3: So I think we put a pretty high price on
this idea that we want you to turn up in
this incident channel. Having been paged and we've got a
fairly substantial preliminary estimation of what's going on in the incident. Now,
we immediately after go back another itseration and we go
target like other resources, and we go work our way
through it and go do we really think that this
actually means the thing that we've gone and claimed, So
we might have a preliminary message that goes low confidence.
It might be this pr that we then actually we're
cloning down the codebase in the background, and we're double
checking all of the assumptions that were built into us
thinking that this was the cause. So yeah, over time,
I think we go maybe anywhere up to five or
ten turns of that cycle. Most people will create an incident,
especially a customer facing one. Someone else might create it
and it will go, hey, website's broken, and for us,
that's like, it's not very useful, right, how are you
going to.
Speaker 4: Find something that's going to help you with that?
Speaker 3: Arguably anything that you have merged to the website or
anything could cause that to be broken. So what we
do is we end up pausing until someone provides enough
information in the channel. Maybe they take a screenshot off
the page that's broken and they drop it in.
Speaker 4: So then we process the.
Speaker 3: Image with some multimodal models, and at this point, now
we know exactly what the website is and exactly what
the path is because we can see it in the browser.
We've got enough information now, so that's when we hit
like all the heavy gucy searches and then we start
putting things in.
Speaker 1: I mean, that's that's pretty amazing what we're doing there,
especially even pursing the images for pulling out the your
art texts, the ad in information.
Speaker 2: Of the pipeline.
Speaker 4: You know.
Speaker 1: I wonder if there's any sort of stat that's like
when someone says, oh, the website is down, they usually
mean this particular website in the past. The type of
people that report that often are you know, in a
particular area, And so there is still some sort of
corollary that and ELM would be able to you know,
pull out automatically via vector search.
Speaker 4: So absolutely, and that is that is what we will do.
Speaker 3: So even if we don't have enough information to go
searching through your get up prs, what we can do
is we can look for other incidents that look like
this one. So if there's another incident in the past
that went website is broken, we'll collate together all of
the information that we think was relevant from those incidents
and then we'll use that to guide our search. So
it might be that we can bootstrap ourselves to a
position where we go we're pretty sure we know what
this problem is, and then that allows us to engage
the other searches, but otherwise will wait until we have
a bit more clarity, because again, what we don't want
to do is send out a very generic query, get
tons of data back, and then misguide people.
Speaker 1: It's sobering almost that you said, you know, vector embedding
or using a betting model and storing a vector database
isn't the end all of optimized search results.
Speaker 4: Here.
Speaker 1: There was another episode in the recent past where we
were actually talking about how whether or not semantic search
for everything should just move to using in a betting
model and staff developer relations expert from pine Cone so, actually,
well maybe, but the keyword search is also still incredibly valuable.
Speaker 2: We can't get rid of that. And you know, you're
basically saying, yeah.
Speaker 1: Actually tagging human tagging on data is still the most
valuable thing that we could do be doing for some regard.
I mean, of course you combine it with the appropriate format,
but it's good. I feel like that multiple experts from
you know, different domains are sort of, you know, really
focused on getting down to the point here, which is
actually we still need things that we've developed in the
past to be successful it's not just like the new
model is better than the one before.
Speaker 3: Yeah, And I think this applies to so much in
the AI engineering discipline at the moment. I think when
we were first starting in ernest to do this maybe
a year and a half ago, there were a ton
of people whispering about how fine tuning was the way
to get yourself to a system that is like the
one that's so much better.
Speaker 1: You really nailed it there, specifically, I've seen it similarly,
the fine tuning is just prohibitively expensive, and so I'm
not going to say, you know, rule out all the
new technology options for sure, but you really have to
be doing something special where it perfectly matches the model
that you've got for it to really work effectively when
you're in that situation to actually be able to utilize it.
And I want to ask you a little bit about
your evolution here because the company started at I feel
like a really unique time for solving these things. Basically,
when you started, the models available were just basically being born,
and you've grown at the same time which the models
have been really significantly refined and curious what the impact
has been both internally in Incinnio but as well as
the product that you thought you were building and the
one that you ended up building.
Speaker 4: So I think you are absolutely right.
Speaker 3: And actually the models that we had maybe a couple
of years ago, they were so so so much different
than the ones that we have right now, and I
think that ends up impacting both the scope and ambition
that you have of the product that you want to build.
But like it's entirely changed, honestly the direction that we
would like to take the company as well. So I
think we first started thinking about how we were going
to use AI to really push the product forward, and
maybe about two and a half years ago, and we
started with very basic things. So we started with how
would we automatically summarize incidents or how would we generate
incident updates so that people who were responding to incidents
wouldn't have to actually type them out themselves. But gradually,
as we started becoming more AI literate, I guess, and
the models started improving, and we saw a gentic tool
being released that seem to do a lot more than
we've ever seen before the scope of what we thought
was possible to try and help our customers with the
product that we were going to build. Has just increased
probably every three months. And I think the biggest thing
that kind of changed my mind is the journey that
I've been through over the last year and a bit,
where we've gone from you know what, we can just
give someone a really great summary of everything that's going
on and examine all the dashboards and just go these
are the useful things to eventually going well. Actually, like,
we're pretty sure we can find a narrative about exactly
how this stuff has worked, and we can reason about
it and rule out the things that we think are
irrelevant and the things that we think are really relevant,
and we can propose next steps to the people to
eventually where we've got to now, which is we're able
to in some cases, if we can identify the part
of the code that's gone wrong, we'll actually create a
code change that we'll try and fix it for you.
So we have a virtual machine that sits there or
several of them, and we end up communicating from our
app and going, hey, please load up the codebase over there,
and then we have an agent that sits inside the
code base and tries actually debugging what's going on, and
then our investigation system is using a combination of that
coding agent, along with our telemetry agent and everything else
that we've got in our system to poke and prod
and build up it's understanding of what's going on, and
so eventually we go, hey, can you actually fix this?
And then we'll try and use the coding agent to
end up building a fix which we end up pushing into,
pushing in to get up. Wow.
Speaker 2: Okay, that's pretty intriguing.
Speaker 4: Really quite crazy security challenge.
Speaker 3: If you've I remember coming to Ben who's like lead,
and going, hey, how do you feel about us pulling
down other people's code and then running an agentic piece
of software in there? And like the agent has to
be able to run arbitrary shellle commands and stuff like that.
The amount of work that we did on trying to
just secure that and make it a secure platform for
us to run was kind of nuts.
Speaker 1: Were you able to take inspiration from like Aws's firecracker
or associated technologies or did you really have to spend
this up yourself? I mean, I asked, because we're in
a similar situation where sometimes not anything close to as
ridiculous as you're doing and trying to run basically the
customers code itself, but realistically, we let customers configure stuff
with maybe a programmatic point extension and write some code,
and one of the best security mechanisms I've found is
to just give it to a cloud provider to execute,
because their whole business is based on like isolation, and
so if there's an isolation problem there, we're not going
to be the only ones with an issue.
Speaker 3: So I think, like absolutely, yes, the problem that we
have is the latency, so being able to to connect
really quickly and issue arbitrary code queries. Some of the
repos that our customers have can be like many gigabytes
large or small, depending on how you feel. And being
able to pull that down and have that code base
available so that you can then run queries against it,
and trying to do that in ephemeral kind of isolated
environments is quite difficult.
Speaker 2: So complex.
Speaker 1: I just would never imagine anyone trying to actually make
that happen programmatically. So has hove to you on that
particular challenge, overcome and execute it, and it would be
really interesting to see how that evolves over time.
Speaker 3: Yeah, no, I think yeah. I think Ben has plans
to share a lot more about it soon. It's been
a lot of adventures in isolation, and well, I think
the key thing for us is this actually allows us
to power some really interesting product experiences, and that's always
the motivation for us here.
Speaker 1: I think when you've identified something that is so novel
in this way, it does create its own sort of
competitive advantage, which I can see is like not something
you ever necessarily wanted to start, Like you didn't start outside.
Speaker 2: You know what we're going to do.
Speaker 1: We're going to figure out that like all this infrastructure
as code, you know devlops world stuff, where you build it,
you run it. Now we're going to just figure out
how to run other people's code programmatically on the fly.
Speaker 2: And but once you've done that, which.
Speaker 1: Arguably is the hard part, there's a lot of like
that empower a lot of things, Like in one of
these previous episodes, we were talking about how the company
had to figure out how to dynamically run customer code.
And they're not as they're not gigabyte repositories. But if
you know what the code is and you convert it
into an AST, you can then prove that you understand
what the code actually does, and then run it in
whatever infrastructure you want, which is really interesting because it
puts all the effort on building the perfect AST generator
from programming and one particular language and then being able
to execute it. But you get around a lot of
the security concerns because you've proven that you understand how
the code is supposed to work, and if it violates
your whatever invariant you have on your code, they always know.
This actually is like an extra instruction to an ALM
to do something we don't actually want to run that.
Speaker 3: Even if you go into how do you get a
code query to execute over a very large code base
and give back responses that are both correct and also
do it quite quickly, there are a couple of ways
that you can do. You can either try and shave
time off how fast you're going to get that code
based locally.
Speaker 4: That's one thing that we've done.
Speaker 3: But there's actually a way more impactful thing that you
can do, which is pre analyzing a code base so
that you have a map of what that codebase looks like.
So for us, that like actually crawling codebases and building
up a bit of a map and an understanding of
what they are so that when someone has an issue
in an incident, we boost up our LLM with some
like cheat sheet notes about this is how you should
browse this thing, and that was probably the most significant
impact on a very large co base. It would go
from being like four or five minutes to answer a
question to if you could seed it with this analysis,
Suddenly it's like thirty seconds because it goes, oh, well
that bit's over there.
Speaker 4: Yeah.
Speaker 1: And that's the most ridiculous thing is that in order
to solve your problem, you designed this technology which basically
puts a whole fleet of other products like out of
business because you had to do the thing that they've
been trying to do, which is it like usually look
at it indexa all of your source code to be
able to just find things that you're looking for in
that source code with poorer class, et cetera. That's often
a challenge that large companies have tried to approach, and
in the past they've gone for tools that specifically offer
that functionality. But in order to deliver your solution, you
had to design a solution from that from the ground up.
Speaker 2: And that means you can actually.
Speaker 1: Target code searching for humans, not just for your agent,
to be able to identify what has changed and what
could it be impacting or creating the incident in the
first place.
Speaker 3: Yeah, absolutely, And I think the same goes for honestly
understanding people's telemetry architectures.
Speaker 4: Is the same deal.
Speaker 3: If you want to understand how to browse someone's looks like,
you need to first understand what looks there are and
try and build up a map of like what lugs
are even useful to you, and then you Whilst you
get these lms in their huge hammers for many different tools,
they also come with some pretty obvious and kind of
like crippling limitations context win those being one of them.
Speaker 1: Yeah, I don't want to undersell that in any way.
The context model problem is never going away. We found
out that just you will never be able to fit
the whole context in your window.
Speaker 2: And actually the larger the more.
Speaker 1: Tokens you add in, the less value each of those
individual tokens have. So you will always have this problem
of I have too much data to utilize how do
I deal with it?
Speaker 2: And the interesting thing, and I think there's may be a.
Speaker 1: Pattern for the episode, is that it turns out we've
sort of solved this problem many times before without using
an LM, and if you first, you know, take that
step to I don't want to say, like sanitize the
data or clean it in some way before passing it in.
Speaker 2: Then you're ending you're going to end up with a.
Speaker 1: Much better result in a much faster time period, rather
than trying to wait for Gemini to push out the
next you know multimillion token context window.
Speaker 4: Yeah, co signed that has absolutely our experience with it.
Speaker 1: Well, I like it that it's like a logical argument
against and alms get better. Yeah, probably, but increasing the
context window isn't isn't really going to be one of them.
Speaker 3: I think the thing that stands out to me that
I think was not true two years ago, maybe was
becoming true a year ago, but it's definitely true now
is I think that the models that we have out
there are no longer the limiting factor to us building
these systems. So when I think about the ais RE
product that we have and getting it from where it
is at the moment where it is like delivering value
to customers, but the scope of the incidents that it
can deal with really like small to moderate, and we
want to get it up to even the highest level
of very complicated incidents and dealing with this so that
it's active almost all the time. I do not think
that it will be an upgrade at the frontier models
that allow us to get from that moderate to high.
Almost all of the value that we've managed to deliver,
all the improvements, have been down to being more structured
in how we think about the problem, teaching us or
breaking our system up so that it's more modular so
that prompts can be more focused, and then figuring out
how we can actually make better use of the models
that we already have.
Speaker 4: Rather than waiting for the next.
Speaker 1: Upgrade, maybe we will schedule off fall up for this
episode a year from now. We can see whether or
not that promise is held true or not.
Speaker 4: I think you won't be the only one holding me
to that, so that seems okay to me.
Speaker 2: He's repeated this promise on multiple podcasts.
Speaker 3: So if anyone is exploring this space, I think the
key things to take away figuring out how to actually
objectively track whether or not what you're building is doing
a good job, and then hopefully once you have that
objective measure, you can focus on trying to keep things
as simple as you possibly need them to be up
until the objective measure proves that you actually need the
additional complexity. And if you're doing that, then I think
you can't really go far wrong.
Speaker 1: That no one wants listen to is that you still
need to do the actual hard work like that that
hasn't gone anywhere. You still need to you know, know
exactly what you're doing and do it the right way
and think through that problem. And it's not going to
magically land on your lap somewhere and you'll be able
to push it out.
Speaker 3: Oh yeah, And like there is, there is just a
huge amount of time that you need to spend using
these models and building systems like these to build the
intuition that allows you to see, actually there's this evolution
to what we're doing at the moment that can get
us to the next step. But yeah, there is no
shortcut that I know of yet to get yourself to
the place where you have that intuition other than just
share hard work and time thinking about it.
Speaker 2: And here I was thinking I could retire soon. Okay,
So with that, that's when we were to picks.
Speaker 1: Well, I'll go first what I brought it today, And
obviously you have to be watching the YouTube channel in
order to see this.
Speaker 2: It's a it's an anchor powerport ADAM three.
Speaker 1: It's got some USB USBA ports, and it's got this
nice little adapter in the back for our two pronged
wall socket I don't carry. The reason I really like
this is besides it's durable, it's light, and it's small.
Travel a lot, and I don't like bringing around like
the plugs to stick in to charge my USB stuff
and having to plug them in, and after doing that
for a lot of years, they're always like, really crap,
are really expensive, like it's fifty one hundred dollars francs
whatever for actually good quality ones that are now able
to charge my laptop.
Speaker 2: And it's just a real waste. And I found like
the cheaper ones they just break all the time. They're
so unreliable.
Speaker 1: And I don't want to be on vacation or traveling
for work and have my USB power AC adapter break
on me. So what I started doing is I just
carry this thing around and I buy the cord that
actually fits for the country that I'm going to, so
I have like a whole bunch of chords that match,
rather than having to play around with like wall socket
adapter AC adapters for the different sizes and it you know,
there's a transformer in here, so it's like really just
a cord. And like I was in it was in
Thailand a couple of years ago and we were on
vacation and the power socket didn't hold the plug in
like you're The adapter was just falling out of the wall.
We had to keep it to it in order to
actually make a charge anything. And after that, I'm just like,
I'm done. You have a chord that comes out of it.
It's like really light and it stays in no problem.
So I absolutely love this thing.
Speaker 2: It's fantastic great.
Speaker 4: I mean, I'm going to buy one. So I came.
Speaker 3: I came. I came fairly unprepared for this when I
first jumped into the podcast. So I think I've got
I've got two picks. One of them is mostly City,
which is that we have a team here of people
who love or like fidget toys, and one of the
most popular has been this Roctopus, which is a three
D printed model of the Rock with Octimus legs, which
you can't on Etsy. That is endlessly entertaining for half
the team going through this, So you check that out.
And the other pick that I have is a lot
more serious or a lot more relevant to the discussion
that we had, which is if you haven't ever read it,
it's a book called The Checklist Manifesto, which is about
how checklists have been adopted in areas of medicine and
also other areas such as aviation, and kind of the
gradual realization of how to build a good checklist and
what's required to make it good, which has some direct
relevance to everything that we were talking about with rum books,
particularly around the idea that until you have written the
checklist and then run it yourself, the checklist is probably wrong.
And yeah, I got recommended this by VP of Engineering
and Niberte, who've worked with for many years, who used
to be an esserit Blizzard and love this book just
because of how how relevant it was TOESRI and everything
else that you might do in DevOps. So you can
get that on Amazon. I would check it out, The
Checklist Manifesto.
Speaker 2: Yeah, the links will be in the down below.
Speaker 1: In the episode description, I will ask about the repridit octopus.
Speaker 3: Is it is it like?
Speaker 2: Would it didn't look like Maybe.
Speaker 4: It's just some sort of it's actually it's just it's
just just plastic, I see. And is the rock too
pust because it is the rock's head on top. It's
just it's just endlessly entertaining. You'd be you'd be really surprised.
Speaker 1: Is there like a particular model that you can go
and download that from the internet to go and run
with it, or there's like it's gone and there's like
lots of different versions available.
Speaker 3: I would imagine it's gone viral at this point, but
I will find out and make sure it pays in
the licks.
Speaker 1: That's a plug for if you are designing swag and
you want to hand stuff out at conferences, don't go
to a third party manufacturer. Just go and buy a
three D printer and your employees will just go make
stuff and you can go. It will be really unique, right,
That's a unique swag to give away or just use
around the office.
Speaker 4: It really is really unique. Whether or not you say
that as a plus or fine is really up to you.
Speaker 1: Well, thank you, Lawrence for this awesome episode. It's been
great so far, and I hope that we'll see you
again back on this show maybe in the years to come.
Speaker 2: With the updates on a giantic evolution in the industry.
Speaker 4: Yeah.
Speaker 3: Well, thank you very much for having me on and
hopefully it was interesting to anyone listening.
Speaker 2: Yeah, and all our listeners, we'll see you again next week.