Speaker 1: Welcome back everyone to another episodes of ventures and DevOps.
Speaker 1: And I am really excited today because we're going to
Speaker 1: jump into one of the areas that I find personally
Speaker 1: really interesting, but also our guest has worked at a
Speaker 1: number of companies in areas that I feel like lots
Speaker 1: of companies get wrong. So I just want to welcome
Speaker 1: to the show. Sylvan from from Rutley, who is the
Speaker 1: head of veloper Relations. Hey, how are you doing?
Speaker 2: Thank you for having me. We're in all good.
Speaker 1: Good to hear, so, I said, head of velper Relations,
Speaker 1: And I got to be honest, I feel like a
Speaker 1: lot of companies have started to over utilize this term
Speaker 1: to mean a wide variety of different roles and responsibilities.
Speaker 1: Can you give me like a breakdown like what it
Speaker 1: means for you today?
Speaker 2: Yeah? Absolutely, Actually to a great question because I have
Speaker 2: just puts it on LinkedIn two days ago when I
Speaker 2: was higher and historically the role of developer relations is
Speaker 2: to empower developers to use a product developer tool by
Speaker 2: providing education resources, by answering any questions they may have,
Speaker 2: and then just overall marketing you know, the product in
Speaker 2: a way that fits engineer, which is not product marketing, right,
Speaker 2: it's all about what you get out of it. Don't
Speaker 2: don't tell, show me, So it's tutorials, talks, you know
Speaker 2: in the former article, YouTube, video, and so on and
Speaker 2: so forth. But as I joinedly, it was clear that
Speaker 2: AI and more specifically here llms are the future of
Speaker 2: incident management. For those who don't know, rutly Routly is
Speaker 2: and on call and incident management platform. So when something
Speaker 2: breaks and you have people on call we need to
Speaker 2: respond to the incident, that's where they go to manage
Speaker 2: the incident up to it being solved. So I played
Speaker 2: a big role in that, and we'll speak about it
Speaker 2: during this episode and most of my time actually at
Speaker 2: truly most of my AERG, I would say like maybe
Speaker 2: seventy five percent has been dedicated to agent AI agent
Speaker 2: relations because we can see II agent as just another
Speaker 2: member of the team, and this agent also need to
Speaker 2: be taught and onboarded, just in a way that's different
Speaker 2: from humans. So I would say, while I'm the head
Speaker 2: of developer relation, can also say I'm the head of
Speaker 2: AI agent relations.
Speaker 1: Well, that's that's definitely a wide area.
Speaker 2: You know.
Speaker 1: I'm all interested because I found you mentioned this that
Speaker 1: you're not doing product marketing, you're still marked in some
Speaker 1: way to engineers. Uh, they're notoriously the biggest challenge to
Speaker 1: get engineers on board with whatever you're trying to sell.
Speaker 1: I mean I found of all groups of people, uh
Speaker 1: even ones in the technology space, I feel like engineers
Speaker 1: always want to do things themselves, right it.
Speaker 2: Is and and with already we are targeting SARES psyche
Speaker 2: truly ability engineers who are even more skeptical and hard
Speaker 2: to convince because and for good reason. Right, their job
Speaker 2: is to ensure that the infrastructure is rolling smoothly and
Speaker 2: in optimized fation. And so you want to be careful
Speaker 2: with the tool and the frame. A new framework or
Speaker 2: new tool or methodology that might include might bring chaos
Speaker 2: or instability. And uh, you know, so I used to
Speaker 2: be an SARE myself. Back then SARE was not truly
Speaker 2: a thing yet. I was working for flight Chair as
Speaker 2: a develops engineer. We were in the top fifteen months
Speaker 2: visited website in the world back then, displaying about one
Speaker 2: point five billion slide a day, which you know was
Speaker 2: definitely large volume. And we got acquired by LinkedIn, where
Speaker 2: I work as a senior SR. This time for three
Speaker 2: year and you can imagine the volume. So I've been
Speaker 2: on the side, on the engineering side, and so I
Speaker 2: completely you know, get the personal and I think at
Speaker 2: the end of the day, it's just that these people
Speaker 2: they don't want to waste time with marketing copies. They
Speaker 2: want to understand what's in it for them and their job. Right,
Speaker 2: So for me, it comes, you know, very naturally. I
Speaker 2: don't think it's a challenge, but I think for someone
Speaker 2: who does not come from an engineering background and doesn't
Speaker 2: have as good of an understanding as you know, an
Speaker 2: engineer may have, it may be hard to communicate to
Speaker 2: this audience where you know, I think this thing might
Speaker 2: come from.
Speaker 1: Yeah, no, I totally got it. I know I'm gonna
Speaker 1: I really want to dive into the core topic, which
Speaker 1: is self healing systems. And we were talking a little
Speaker 1: bit before the episode started about like this has been
Speaker 1: your like a lifelong project area. How did you get
Speaker 1: into this? I would say it's the first thing, like
Speaker 1: was this, Like you always knew this is this was
Speaker 1: the thing you're going to go into, and like how
Speaker 1: long you've been doing this and what does that really
Speaker 1: mean to work in self healing systems?
Speaker 2: Yeah, so it goes back to actually I started the
Speaker 2: project when I was working for a slight share as
Speaker 2: a develops engineer slate slash slate site reabity engineer, and
Speaker 2: I was on call and I had to you know,
Speaker 2: manage outages back then who are using puppet as a
Speaker 2: way to ensure that unfraustructure was as it should And
Speaker 2: you know, ultimately there were a lot of repeat incident
Speaker 2: or a lot of outage or you know issues that
Speaker 2: were coming from the same type of problems. I think.
Speaker 2: You know, as you as engineer grow in their careers,
Speaker 2: they kind of know what are the main failure types
Speaker 2: and so and so, you know, like I think it,
Speaker 2: it gets repetitive, and I think a great engineer wants
Speaker 2: to automate itself and not do the same thing over
Speaker 2: and over again. And so yeah, that's where the idea
Speaker 2: of building a self feeling system came about. So maybe just.
Speaker 1: Sort off like you said, you would see the same
Speaker 1: sort of regressions over and over again. Was there like
Speaker 1: one in your mind that was just like the most
Speaker 1: common where it was like every single time it happened. It
Speaker 1: was like the driver for you to make a real
Speaker 1: change in an organization.
Speaker 2: Yeah, you know, I think. I mean, I know it's
Speaker 2: been like more than a decade, so you know, I
Speaker 2: won't have like super sharp example, but I think the
Speaker 2: classic ones are, you know, issue with lack of resources,
Speaker 2: whether it's you know, storage or CPU or memory, and
Speaker 2: you need either to increase decrease the lots somehow or
Speaker 2: distribute or scale. Could be like a service that misbehaving
Speaker 2: and you need to restart. Could be a lot of
Speaker 2: things that I think, So I think the industry took
Speaker 2: different throughout. I think now with Kubernetists, what we do
Speaker 2: is that if if this is misbehaving, we just shut
Speaker 2: it down, right, we get rid of it and we
Speaker 2: start a new one. And obviously kubernets is great at scaling,
Speaker 2: so I think this tool to like the self feeling system,
Speaker 2: it still works in this way, right like by new
Speaker 2: king thing or scaling thing, you can heal a system.
Speaker 2: I think in my mind I wanted to take a
Speaker 2: different approach where I was training. I was envisioning a
Speaker 2: system that would actually address the root cause I think
Speaker 2: in some of the cases, you know, instead of just
Speaker 2: new kings, this thing like really try to mimic what
Speaker 2: a human engineer would do. So, Yeah, that's that's kind
Speaker 2: of the philosophy that I had back then.
Speaker 1: I mean, there's definitely a huge population of engineers who
Speaker 1: think that the what they would do in those examples
Speaker 1: would be for sure to restart the machine or the
Speaker 1: container or the node if it started to run out
Speaker 1: of memory or processing power. And I feel like that's
Speaker 1: sort of the crux of one of the issues that
Speaker 1: I've seen over and over again is that we do
Speaker 1: build those systems that I say we as collective humanity
Speaker 1: and not at my current company, that automatically restart or
Speaker 1: you know, allocate more memory or processing power. And I
Speaker 1: feel like the automatic scale scale out or scale up
Speaker 1: for resources can make sense if it doesn't create a
Speaker 1: negative impact on the feedback loop that you have to
Speaker 1: solve the problem. And I feel like this is one
Speaker 1: of the problems with automatic restarts is that it doesn't
Speaker 1: really solve the problem. It's still is persistent there, It's
Speaker 1: going to keep happening, and also you're delaying actually doing
Speaker 1: the investigation, and you're also eliminating some of the evidence
Speaker 1: that would allow you to identify the problem there. So
Speaker 1: it's really great to hear that you know, you thought
Speaker 1: that the appropriate process was, you know, go out like
Speaker 1: why is there extra memory usage? Why is you know,
Speaker 1: the machine getting stuck, et cetera, et cetera. And I
Speaker 1: feel like that's sort of the thing that sets apart
Speaker 1: the best esses from the ones that are just coming
Speaker 1: into quote unquote do the job it is.
Speaker 2: I think then you need to strike the right balance
Speaker 2: between achieving the end results, which is capability, stability, and
Speaker 2: if restarting is the way to go, and you know,
Speaker 2: you don't need to spend engineering resources. And obviously, I
Speaker 2: mean it's walking, so you know, I don't think it's
Speaker 2: it's a valid issue, but yeah, as you said, you
Speaker 2: need to find a strike the right balance between just
Speaker 2: doing this over and over and if it's repeat like
Speaker 2: out edsure issue, you know might need investigation. And so
Speaker 2: the idea that I had back in two Southern was
Speaker 2: I think I started in twelve thirteen was too really
Speaker 2: ingest as many data as you can from a distributed system,
Speaker 2: you know, whether it's any lugs, any metrics application, lugs, traces,
Speaker 2: and ingest all of this in the dallabas. Back then
Speaker 2: we were using fluend, which is an open source message,
Speaker 2: but we still exist and actually it's very popular. Back
Speaker 2: then I was we were one with lecture of the
Speaker 2: first main user big use there actually shout out to
Speaker 2: the team if they are listening. And now they went
Speaker 2: very far with this technology and still all of this
Speaker 2: in unstructured database. Back then we I bet on on
Speaker 2: Mongo DIB. It doesn't really matter the technology, but that's
Speaker 2: what we use for the prototype. And then based on
Speaker 2: that come up with like a state of a system
Speaker 2: and and use and then like try to resolve this
Speaker 2: issue by throwing at the system a bunch of actions
Speaker 2: that would be safe. You know, I'm speaking like any
Speaker 2: action like a like AREM action or something like you know,
Speaker 2: drop database like you need to be careful. But but
Speaker 2: but other set of safe action and then use machine
Speaker 2: learning as an engine to learn basically what, depending of
Speaker 2: the state of the system, what could solve the issue.
Speaker 2: And so we designed this for a way distributed infrastructure.
Speaker 2: Actually continued this work at LinkedIn and they asked me
Speaker 2: to write a patent, which eventually was accepted. But yeah,
Speaker 2: it never got a chance to build it, unfortunately, because
Speaker 2: then I left to become an entrepreneur. But that's why,
Speaker 2: you know, I was telling you have been swimming in
Speaker 2: this topic for a little while.
Speaker 1: Yeah, I mean I still want to go further into that.
Speaker 1: So just to summarize a little bit, the strategy is,
Speaker 1: we're collecting tons of logs, maybe metrics, et cetera. We
Speaker 1: maybe have access to the source code, and we train
Speaker 1: on that data to identify based off what sort of
Speaker 1: errors we're actually seeing, how to pinpoint potentially is it
Speaker 1: a part of the source code or the infrastructure which
Speaker 1: could be problematic, and then utilizing that on errors that
Speaker 1: actually do come out of the system to help dive
Speaker 1: in to identify the cause or does it go further
Speaker 1: than that?
Speaker 2: Yes, I think that's a good point if we speak
Speaker 2: about I think I think back then I was I
Speaker 2: directly dive into resolution. I was a young engineer, you know,
Speaker 2: I was like twenty five, twenty six, maybe even younger
Speaker 2: than this, so I was not really mature. But I
Speaker 2: think I think starting with the root cause analysis is
Speaker 2: the right approach. Obviously, you know now with insight it
Speaker 2: makes sense, but but my goal was really resolution, which
Speaker 2: ultimately you know where you want to go. So yeah,
Speaker 2: I was really really focusing on on direct resolution, and
Speaker 2: I would be a mix of kind of run books
Speaker 2: that that you know, we could feed and then but
Speaker 2: then more interestingly set of safe action safe commands that
Speaker 2: the system could run and could see if this solved
Speaker 2: the issue, and then kind of do like a learn
Speaker 2: from it, you know, maybe try something. It doesn't work,
Speaker 2: it's fine, you know, we just ditch this kind of
Speaker 2: pass instruction instruction set as an option, but sometimes it
Speaker 2: will succeed and it will use this for the next incident.
Speaker 2: And all of this would be based on machine learning obviously,
Speaker 2: like you know, like the more success you have with
Speaker 2: an instruction, the more you are likely to use it
Speaker 2: next time. Back then, machine learning was nearly not as
Speaker 2: advanced as it is today, you know, so it's it
Speaker 2: was hard like to achieve this goal. And I think
Speaker 2: the industry, I don't think anyone build this type of
Speaker 2: system until like now you add one player that did
Speaker 2: something similar, which is Facebook. They build a system that's
Speaker 2: called f bar F B A A R that the
Speaker 2: definers self feeling it was to manage Facebook that center racks,
Speaker 2: so it was not system but tracks where it would
Speaker 2: auto automatically perform action to solve some production issue. But
Speaker 2: it was deterministic, so it was not the well, no,
Speaker 2: none of machine learning used in it. And then Dropbox
Speaker 2: in twenty sixteen presented no rout Aserican and this was
Speaker 2: a self feeling system but for distribute distributed systems, this
Speaker 2: time for web infrastructure, but same year it was deterministic.
Speaker 2: So I you know, I think this system I've been
Speaker 2: around for like more than a decade, and I've been
Speaker 2: produce producing value. I know they've been in production, and
Speaker 2: I think as area generally not up with having mechanism
Speaker 2: or system working on their behalf and kind of in
Speaker 2: the shadow. But the truth is that they've been around
Speaker 2: for a while. I think. Now the main difference, the
Speaker 2: big difference, which is a huge difference, is that we
Speaker 2: are including this machine learning LLM part which is non deterministic.
Speaker 2: And that's a big deal.
Speaker 1: So let's just dive into that for a second. When
Speaker 1: we say deterministic in the history of self healing systems,
Speaker 1: we're talking about like auto scaling groups or identifying specifically
Speaker 1: based off of rules that some engineer wrote, what they're
Speaker 1: seeing and then how to handle the situation very concretely.
Speaker 1: Is that accurate?
Speaker 2: It's a great and also a one that was called
Speaker 2: earths and it's exactly how you describe it, like a
Speaker 2: runt book that's threaten by a human and then this
Speaker 2: rent book is a trigger based on a specific signal.
Speaker 2: But it's absolutely deterministic.
Speaker 1: Yeah, And I think the interesting thing with the deterministic
Speaker 1: systems is that really require you to do the root
Speaker 1: cause analysis so that you could write like if no
Speaker 1: run book applied or the run or a run book
Speaker 1: applied either by a human or through automation, didn't have
Speaker 1: an effect that actually resolved whatever incident you had, you
Speaker 1: had to actually still do the root cause analysis. And
Speaker 1: now I feel like we're getting into you know, I
Speaker 1: think everyone's waiting for us to talk about this how
Speaker 1: to apply Uh, I hate to say AI to this concept.
Speaker 1: So now that we're now that it's twenty twenty five,
Speaker 1: what does it mean to deploy LM to be able
Speaker 1: to self heal a system? What does that actually look
Speaker 1: like in practice?
Speaker 2: Yeah? So, you know, I think you hit the nail
Speaker 2: on the head with speaking about the root cause analysis.
Speaker 2: That's obviously the first step that this system need to
Speaker 2: do is to understand what's up. I think in the
Speaker 2: past it was a mix of for this self feeling
Speaker 2: system I think if part of this runt book where
Speaker 2: automatically based on a very specific maybe anologue or you know,
Speaker 2: something like very trivial, and then it would be automatically applied.
Speaker 2: In other cases, a human would need to do the
Speaker 2: root cause analysis and then you would push this rund book,
Speaker 2: which would still save a ton of time, right because
Speaker 2: this system would orchestrate all the action that needs to
Speaker 2: be done. And when we are speaking about very complicated
Speaker 2: infrasecure like at Google, Facebook or LinkedIn, you know, it
Speaker 2: can be a lot of work. But here the idea
Speaker 2: is that we can throw whatever broken system to this
Speaker 2: LLM and it should understand what's up or at least
Speaker 2: come up with hypothesis. And I think the hypothesis is
Speaker 2: very trivial, very important. Sorry, I don't think we need
Speaker 2: We should not consider this LM to be God or
Speaker 2: to be the silver bullet. We should consider them just
Speaker 2: as another human which can make mistakes. We make a
Speaker 2: lot of mistakes, and so I think one key element
Speaker 2: is to understand this that this ipothesis also kind of
Speaker 2: a degree of certainty, and so I think a great
Speaker 2: ais are will provide as part of the diagnosis what
Speaker 2: the degree of certainty that it has about the diagnosis,
Speaker 2: so human can say, hey, if it's fifty percent, maybe
Speaker 2: I should not pay too much attention attention to it.
Speaker 2: If it's ninety five percent, okay, maybe I should really
Speaker 2: look into it.
Speaker 1: So what sorts of source data are you utilizing in
Speaker 1: order to feed into the l I'm going to ask
Speaker 1: you questions about about that afterwards, but specifically right now,
Speaker 1: like is it you know a list of like source
Speaker 1: code and some other things like what does the sourt
Speaker 1: settle look like?
Speaker 2: M So I think with as with many tools in
Speaker 2: the LM air space today, context is king, and I'm
Speaker 2: going to speak about what we are building at Rutely,
Speaker 2: which again is an incident management platform. And why I
Speaker 2: repeat this because it really matters in the sense that
Speaker 2: engineering teams will all the signals that are associated with
Speaker 2: an incident will flow through their incident management platform. It
Speaker 2: just makes sense, right, And so these platforms such as
Speaker 2: routely have pretty much all the context that is available
Speaker 2: generally to solve this incident. So it can be monitoring,
Speaker 2: logging traces, it can be more than this. It can
Speaker 2: be slight conversation, right, it can be a zoom call
Speaker 2: here you know, you can do a transcript as a
Speaker 2: zoom call, so you can fit this in the l
Speaker 2: l M. But I think there also are very important
Speaker 2: data such as the history of post mortem incident resolution
Speaker 2: report right where everything is documented from what happened, how
Speaker 2: it happened, how did we solve the incident, how this
Speaker 2: incident like how it do I solve? Who solved it?
Speaker 2: And all this data is like super important for for
Speaker 2: the AI agent to to to find a root cause.
Speaker 2: And and the last one I I forgot you mentioned
Speaker 2: is obviously anything that's linked to changes, which you know
Speaker 2: takes the form of cod So the list of commits
Speaker 2: is often you know where you'll find the issue.
Speaker 1: So you've actually got a system that and justs all
Speaker 1: this information and that's out. You know, here's a here's
Speaker 1: some action items take And then I imagine some companies
Speaker 1: are actually automating based off of that to remediate the
Speaker 1: problems in production. Or is this like you need a
Speaker 1: human to review this before you do anything else.
Speaker 2: Yeah, I think the safe this field is extremely like
Speaker 2: the auto healing mechanism around a LAMB is extremely young.
Speaker 2: You know, the oldest company in the field maybe two
Speaker 2: year old, perhaps even less, there is a lot of
Speaker 2: competition in the space, seen at least twenty five products,
Speaker 2: and I've spoken to a lot of engineers who are
Speaker 2: building this internally at large companies, so you know, I
Speaker 2: think everybody's doing differently. The maturity of the product is
Speaker 2: also very different. But I think for s ARE is
Speaker 2: obviously a really ability is ultimately the goal, and not experimentation,
Speaker 2: right its secondary goal definitely Right, starting with just investigation
Speaker 2: is the right way to go. And then I think,
Speaker 2: as the space meteor and perhaps the model, what's work
Speaker 2: great with this technology? You can teach it right, It
Speaker 2: becomes better over time, so you can train models on
Speaker 2: your data, you can tune it right. So as the
Speaker 2: technology mature and the model is learning, and perhaps we
Speaker 2: are learning as humans, we can go more towards allowing
Speaker 2: this tool to do the resolution. But I think the
Speaker 2: first step in is just fun our root cause analysis.
Speaker 1: Yeah, I mean, at least from my personal standpoint, I'm
Speaker 1: scared to hand over the tools to make changes to
Speaker 1: production infrastructure automatically without involving some sort of review process.
Speaker 1: And I mean, I guess, I guess it's fine to
Speaker 1: have like another LM review. The first llm's work in
Speaker 1: some way, but I don't know if the direction matters.
Speaker 1: Like I think you need someone to review the context
Speaker 1: of what's happening, just like you probably want multiple engineers
Speaker 1: on call to actually validate any sort of code changes
Speaker 1: that would have to go into production, because I mean,
Speaker 1: otherwise you're in a situation where there's a critical event.
Speaker 1: It's three, something's going wrong with the database. You log
Speaker 1: in and accidentally drop the production dB. I mean, I'll
Speaker 1: pause there because that this has actually happened to more
Speaker 1: than one company, But I think there's one in our history.
Speaker 1: I think it's almost ten years old now, a major
Speaker 1: source code get server company had a production incident, very
Speaker 1: famous with their I think it was Postcress at the time.
Speaker 1: So engineer is definitely not infallible when it comes to remediation.
Speaker 1: But I guess my question is going to be do
Speaker 1: you find with all the data that you're collecting that
Speaker 1: the set of incidents all point back to some like
Speaker 1: as far as you're concerned, repeatable or already seen problems
Speaker 1: like oh yeah, this sort of software development issue or
Speaker 1: some syntax problem like no reference exception or dynamic module
Speaker 1: loading or you know, memory exhaustion or something like that,
Speaker 1: or are there like minor differences as time goes on,
Speaker 1: like oh well it used to be this said, but
Speaker 1: the next thing is sort of something that we haven't
Speaker 1: discovered yet, and so you're still discovering sort of new
Speaker 1: failure modes.
Speaker 2: Yes, just picking like very briefly on what you said before.
Speaker 2: I totally agree with you that ELMS should you be
Speaker 2: considered as another gument So can review, doing canary deploy,
Speaker 2: you know, passing the change through the CD basically making
Speaker 2: sure that the change is safe is just a must do, right.
Speaker 2: I don't think we should treat treat what the AI
Speaker 2: say as the resolution pass as any different as a
Speaker 2: human would say.
Speaker 1: I mean it goes further than that though, right, Because
Speaker 1: if we were able to confidently take the output from
Speaker 1: LLMS and feed it back in, LMS should be able
Speaker 1: to develop increasingly large solution of any size. And we
Speaker 1: see that no company has a software an automated software
Speaker 1: development or agent engineer that can just continually push out code.
Speaker 1: Even ones operating of very small scopes have utterly failed
Speaker 1: in their release and their push out of their products,
Speaker 1: let alone larger companies that have been trying to build
Speaker 1: stuff up. And the recent craze on vibe coding. Yeah,
Speaker 1: I mean, and for anyone who's not aware, it's this
Speaker 1: idea where you just you don't even look at the code.
Speaker 1: You just have the LM produce all the output, and
Speaker 1: whenever there's a problem, you just say, hey, here's the issue,
Speaker 1: try to fix it. The problem is that the context
Speaker 1: window will have to keep growing indefinitely. Every new feature
Speaker 1: you add will continue to grow. And so as long
Speaker 1: as we have these two failure modes A the LM's
Speaker 1: finite context W and B, companies who have made it
Speaker 1: their sole goal to make money off of automated software
Speaker 1: development aren't making money off of that, you know, aren't
Speaker 1: like wildly successful. The likelihood of you being able to
Speaker 1: do it, lust being able to trust them fundamentally tells
Speaker 1: us that, you know, we're not at that point yet.
Speaker 2: For sure. I think the this vibe coding is variable
Speaker 2: in many situations. If you want to prototypes, maybe if
Speaker 2: you are very young startup, you know, I think it
Speaker 2: makes a lot of sense. But when you get to
Speaker 2: the stage where you hire a necessary or you need
Speaker 2: stability in your product, or you are pushing a product
Speaker 2: that is crucial for your customer system. You know, I
Speaker 2: don't think this type of engineering practice, if we can
Speaker 2: call it this way, makes sense, but I think this
Speaker 2: technology can bring a lot of value. You mentioned do
Speaker 2: we find patterns in the type of incidents that with
Speaker 2: see through the system? And that's a really great question.
Speaker 2: So one of the initial leading at rutely is the
Speaker 2: rootly Air Labs. It's a community driven initiative where we
Speaker 2: hire software engineers. We have the head of platform engineering
Speaker 2: at Venmo and the former head of AI at Video
Speaker 2: and other very smart student PhDs from Stanford and whatnot,
Speaker 2: and we pay them to create open source prototype leveraging
Speaker 2: the latest air innovation to see how can this be
Speaker 2: applied to the world of free the ability and system operation.
Speaker 2: And one of the projects that we're working on is
Speaker 2: exactly what you mentioned, is to create a graph of
Speaker 2: the incidents and see if we can find patterns. So
Speaker 2: it could be an area we are infrastructure, or a
Speaker 2: part of your application, or perhaps a type of failure.
Speaker 2: You know, let's let's say we speak about we spoke
Speaker 2: about resources. Is the resources often something you know that's
Speaker 2: that's fading our system and maybe because our skating rules
Speaker 2: are not aggressive in US, and and alms are helping
Speaker 2: us to to create this graph because they are you know,
Speaker 2: they are great interesting unstructured data and make sense of it.
Speaker 2: And so then we can create this graph that can
Speaker 2: and power a sorry team to understand where and stability
Speaker 2: come from.
Speaker 1: I mean, that's something I'd be super interested to find out,
Speaker 1: like where statistically are the most problems coming from, and
Speaker 1: how that maps are, or like what the confounding variables
Speaker 1: are between maybe the culture of the company or software
Speaker 1: languages that they're utilizing, or the frameworks or the industries. Right,
Speaker 1: you know, maybe these industries have these common incidents. Like
Speaker 1: I think that'd be super interesting to say.
Speaker 2: Well, yeah, so we we're building it. You can check
Speaker 2: it out. We have a gid up GitHub space if
Speaker 2: you look for rootly AI labs. Everything is open source
Speaker 2: and we're always welcoming people to to join, just giving
Speaker 2: ideas or or contributing. Again, we're paying people to do that,
Speaker 2: and so it's kind of a side job. But yeah,
Speaker 2: I think AI is breaking like and and you know,
Speaker 2: I would say it's kind of a side thing. You know,
Speaker 2: it's it's not it's not as an ambitious goal as
Speaker 2: like self ining system. But I do think that's where
Speaker 2: you see that l M can can allow you to
Speaker 2: do other things that are interesting. I know there are
Speaker 2: prototype that I think it might be interesting in two
Speaker 2: other prototy, but speak about it very briefly. One of
Speaker 2: them is to create a diagram out of a post
Speaker 2: more tem showing where things went wrong. And post more
Speaker 2: TEMs are actually kind of painful for engineer to write
Speaker 2: like you know, no one wants to do that. You
Speaker 2: need to remember what happened and bring all of this together. Actually,
Speaker 2: that's what's good with lelamps and that's something we have
Speaker 2: in Rutley. Rutley will draft a pastmare tem for you
Speaker 2: and then you just have to review it and chances
Speaker 2: are that the post more time is going to be great.
Speaker 2: And then the next step that we we tried with
Speaker 2: the Routly air lab is how about trying to offer
Speaker 2: another way to consume a postmare tem and a visual
Speaker 2: way may help, especially I think non engineering audiences to
Speaker 2: understand where the failure happened and why the other service
Speaker 2: that may seem totally enerated was done as well. So
Speaker 2: the way it works is that it will ingest the
Speaker 2: post mortem makes sense of it as a geson and
Speaker 2: then ingest you know your your code base, infrastructure and
Speaker 2: code and make a gson out of it and then
Speaker 2: merge this to and and create a knockdown graph. So yeah,
Speaker 2: that's you know, another way to leverage LMS, which which
Speaker 2: can ultimately help a very team to do their job
Speaker 2: more efficiently.
Speaker 1: I like how you called out that after you are
Speaker 1: you've pushed out a post mortem that someone actually has
Speaker 1: to review what way you've created, Like, no, don't just
Speaker 1: don't just take that and you know, start sending it
Speaker 1: to people as the official thing are Like, if you
Speaker 1: take an LM generated post mortem and you put that
Speaker 1: up publicly, you will for sure get harassed on Blue
Speaker 1: Sky and asd on very quickly about how you spend
Speaker 1: zero effort and then making sure that that was accurate.
Speaker 1: It's very easy to identify LM generated stuff like that.
Speaker 2: And the second thing that we've built that maybe of
Speaker 2: interest to the audience is is an on called Burnard Detector.
Speaker 2: I think that's particularly interested for companies that are distributed
Speaker 2: where manager may not be in touch as much with
Speaker 2: what the team is doing, and especially for large companies.
Speaker 2: So what we do is that we feed all the
Speaker 2: associated data about incident responder can be how long was
Speaker 2: there shift over the last week, how many incidents they
Speaker 2: had to travel, shoot, what was the severity of this incident,
Speaker 2: how long were they working during the night, and so
Speaker 2: many things you know that are all instructured data. Right,
Speaker 2: So we again like Elem's are great at this, And
Speaker 2: then from this an elelant can come up with kind
Speaker 2: of a burnout level, you know, and see, hey, like
Speaker 2: you know, this person was like smashed very hard with
Speaker 2: a bunch of hard incident, like you may consider giving
Speaker 2: them a break. I'm sorry, So as.
Speaker 1: Long as it doesn't also suggest the therapy that should
Speaker 1: be necessary and try to provide that. I think you're
Speaker 1: on the right track there, Yeah, I mean, yeah, it
Speaker 1: can be. It can be difficult to see the differences
Speaker 1: between individuals. Like some of them are way more interested
Speaker 1: in actually jumping in and you know, diving in and
Speaker 1: trying to identify those problems and solve them, and others
Speaker 1: are you know, care more about the routine. But I
Speaker 1: don't think in my in the history of my engineering career,
Speaker 1: I ever saw someone jump up and down and say yes,
Speaker 1: I would love to be woken up at three a m.
Speaker 1: And jump on a call with other people and try
Speaker 1: to justify what was what was happening. So, you know,
Speaker 1: I think you're definitely onto something interesting there.
Speaker 2: Yeah. I was when I was young because I wanted
Speaker 2: to learn. But I'm definitely not into that, but go
Speaker 2: for it.
Speaker 1: Yeah, I mean, I know. I think that's an interesting
Speaker 1: point because you know your career, things change for you
Speaker 1: over time, and maybe at some point you are willing
Speaker 1: to make some sacrifices, you know, But I don't know
Speaker 1: if it was the case for me, Like I remember
Speaker 1: my first job out of university, there would be incidents
Speaker 1: in the middle of the night, and I never had
Speaker 1: to deal with that sort of thing in my life
Speaker 1: up until that point. Like I didn't run my own
Speaker 1: data center in my home, and even if I did,
Speaker 1: I don't. It was not at the point where you'd
Speaker 1: be like getting alerts to be woken up to deal
Speaker 1: with one of your virtual machines failing. And the university
Speaker 1: wasn't a thing. You're you're definitely awake while you're causing problems, right,
Speaker 1: things aren't happening while you're sleeping. And so my first
Speaker 1: job like this would happen, and I definitely came away
Speaker 1: from that with it, with the idea this is wrong,
Speaker 1: Like I don't ever want to be woken up in
Speaker 1: the middle of the night, Like you don't have to
Speaker 1: be it's not a requirement. And since then, like I've
Speaker 1: really been on the path of highly reliable systems. And
Speaker 1: I think the part that really stumps a lot of
Speaker 1: people is they focus a lot on the preventative nature
Speaker 1: that they can try to prevent every problem. Oh, get
Speaker 1: one hundred percent test coverage, or you know, have a
Speaker 1: highly reliable solution by duplicating the infrastructure in multiple regions.
Speaker 1: And I mean the thing I think you said at
Speaker 1: the beginning of the episode, which is that it will
Speaker 1: go down, Like you cannot have one hundred percent reliable
Speaker 1: system and so at some point you have to optimize
Speaker 1: for recovery and not just prevention. And this is where
Speaker 1: I think a lot of people get stuck because like
Speaker 1: at our company, we have a five nines reliability SLA,
Speaker 1: and that means that by the time so one gets
Speaker 1: alerted and they get online, we've already violated the SLA,
Speaker 1: let alone identified and fixed the problem.
Speaker 2: That's a great point you bring that I think this
Speaker 2: system well. First, first of all, getting woken up at
Speaker 2: three am is never a pleasant experience, and it takes
Speaker 2: time for your brain to get into it. And you know,
Speaker 2: maybe you were in some deep sleep and you are
Speaker 2: waking up and kind of having like little panic attack
Speaker 2: or you know, something like tough on your body, and
Speaker 2: then you need to you know, get time to ingest
Speaker 2: the data and so on and so for so we
Speaker 2: know how hard it is for your body and your mind.
Speaker 2: I think that's where aiser is. You know, which are
Speaker 2: like self feeling system or tools that can lead to
Speaker 2: that can help. Is like, hey, this tool can ingest
Speaker 2: so much data in such a small amount of time
Speaker 2: and give you something to get started, like an initial
Speaker 2: root cause analysis. Then by the time you get to
Speaker 2: your computer, you already have something ready to look at.
Speaker 2: Hopefully it's ninety five percent confidence and you know you
Speaker 2: just have to push the fig that they suggests. I
Speaker 2: think it's such a great tool. I think that we'll
Speaker 2: have a great positive impact on the on the health
Speaker 2: of our people. The second thing I think that's that's
Speaker 2: interesting with this tool is that you mentioned you know
Speaker 2: five nines, and you know, we know that it's possible
Speaker 2: to get five, nine or six nines. But the companies
Speaker 2: that are achieving that, like the Google of the world,
Speaker 2: are investing huge amount of resources, uh you know, human
Speaker 2: and financial to reach this level, and for the rest
Speaker 2: of us, the rest of the businesses is simply not
Speaker 2: possible until today. I believe that these self healing tools
Speaker 2: will allow companies to reach this type of you know
Speaker 2: SLA without spending the budget that that Google does. And
Speaker 2: I think that's truly I think that's going to already
Speaker 2: find the sory space.
Speaker 1: Yeah, I mean, I will say that one of the
Speaker 1: biggest struggles we have is actually customer perspective alignment. Like
Speaker 1: it's a challenge for us to know what the status
Speaker 1: of our system is like it's subjective. Is it up
Speaker 1: or down? Is not like you can look at some
Speaker 1: chart and have the answer there. And what's even more
Speaker 1: important is that if we believe that our system is up,
Speaker 1: that our customers also believe that our system is up,
Speaker 1: because this mismatch is really what you're trying to solve
Speaker 1: for If customers always like one hundred percent, reliability is
Speaker 1: not what whether you think it's up, it's whether or
Speaker 1: not you know the people that are paying you money
Speaker 1: to you know, run some system believe it is and
Speaker 1: the customer expectational alignment like that's actually a really that's
Speaker 1: a huge challenge and I'm not sure you can fundamentally
Speaker 1: all that problem. But yeah, I do things as a
Speaker 1: huge gap with a lot of companies being able to
Speaker 1: get for from where they're at, which is like their
Speaker 1: software is going down like at least once a week,
Speaker 1: to something much further than that.
Speaker 2: Yeah, yeah, so, I you know, I do think the
Speaker 2: LM can can help with that in some capacity. Maybe
Speaker 2: I can jump also and share a little bit about
Speaker 2: what are the challenging of building these type of tools.
Speaker 1: Yeah, please, I'm dying to now, right.
Speaker 2: So, I think one of the hardest things, which you know,
Speaker 2: I think it's a big week LACE is obviously the
Speaker 2: non deterministic part of the system. And here I think,
Speaker 2: you know, the old adage you cannot improve or fix
Speaker 2: what you cannot measure is you know, works very well,
Speaker 2: right like for l elms, even if you provide the
Speaker 2: same input, the output will be different. And so it's
Speaker 2: very hard for engineering team to ensure that one, you know,
Speaker 2: is my system running well as you say, it's subjective,
Speaker 2: and I think here it's even more subjective because it's
Speaker 2: not a matter of just hey, am I getting a
Speaker 2: two hundred or five hundred or maybe it's a two hundred,
Speaker 2: but which is too much you know latency. We're speaking
Speaker 2: about an output which is natural language. And second is
Speaker 2: my output better or worse? So that's that's like a
Speaker 2: big challenge in in in building this system. And and
Speaker 2: another point is that this system don't have sking in
Speaker 2: the game. And elms are like dream machines. They are
Speaker 2: designed to put together chain of tokens that are, you know,
Speaker 2: using statistics, the more likely to be pleasing, you know,
Speaker 2: and sometimes this what the assembly is not rooted in reality.
Speaker 2: But they still did their job as they should. And
Speaker 2: so if we compare this to a human when you know,
Speaker 2: if let's say were you're my manager and I'm working
Speaker 2: on trouble shooting this incident, and I'm like, hey, I
Speaker 2: think that's the issue. I think this is where we
Speaker 2: should you know, Look, I have skin in the game, right, like,
Speaker 2: like I'm putting my skills on the line, and so
Speaker 2: you know, when I share this with you, I have
Speaker 2: a certain degree of certainty that this is a probable
Speaker 2: cause for ALMS. There is none of that, right, So here,
Speaker 2: what we've done at routely is that we we have
Speaker 2: two types of agents. We have the master agent, which
Speaker 2: is orchestrating sub agents which are in charge of doing
Speaker 2: the work of gathering days trying to understand, like doing
Speaker 2: the grain walk and then coming up with an answer.
Speaker 2: And the master agent will make sure that the overall
Speaker 2: narrative mix sense. And there is anothern agent that's like,
Speaker 2: you know, coming up with something that doesn't make sense,
Speaker 2: like a manager would do. So what's funny with with
Speaker 2: LMS is that it kind of mimics a human narrative,
Speaker 2: a human dynamic. Yeah.
Speaker 1: No, I mean I feel like the most common questions
Speaker 1: I end up asking are how do you know? And
Speaker 1: why now? And Alan's not so good at solving that one,
Speaker 1: especially when like a bunch of changes all stack together
Speaker 1: to then cause the problem. Right, you know, you look
Speaker 1: at individual changes and they all seem fine, and then
Speaker 1: only together do they cause the issue. So I mean,
Speaker 1: I do see this sort of interaction is necessary. I
Speaker 1: do want to ask you about your models, though, so
Speaker 1: are you taking some fundamental like some foundational model out
Speaker 1: there that's available open source and mind too, it are
Speaker 1: you building it up from scratch? Is there like one
Speaker 1: particular companies models that you like more than others? What
Speaker 1: does this look like for you?
Speaker 2: Yeah, So I think the assumption that I think the
Speaker 2: you know, anyone not not like deep in the space
Speaker 2: would assume is that you need you need to train models,
Speaker 2: you need to tune it using in our case, you know,
Speaker 2: like your customer data or like you know, if you
Speaker 2: are building this internally, your specific data. What we found
Speaker 2: is that this is actually not needed for most of
Speaker 2: the incidents, Like out of out of the shell like
Speaker 2: model like work perfectly fine, and we'll find most of
Speaker 2: the issues. Training model is actually really hard, really costly,
Speaker 2: and we haven't found so far. You know, we're still
Speaker 2: early you know in the space. So I think we'll
Speaker 2: get to this eventually. But I think for now we're
Speaker 2: finding the most value by not doing it. I think again, like,
Speaker 2: it's it's difficult, it's expensive. And then there is a
Speaker 2: lot of skepticism and I think issue with privacy and
Speaker 2: security companies on one their data going into l LMS heaven,
Speaker 2: you know, if we would do this only for their ALM.
Speaker 2: So what we found matter of the most is ready
Speaker 2: the context that you provide. And I think what we've
Speaker 2: found is the most valuable is the non technical stuff.
Speaker 2: But what I mean by this is the human generating
Speaker 2: generated context. And when we link this to roughtly, it's
Speaker 2: two things. The first one is the former postmare terms
Speaker 2: like this is a gold mine of information. Most of
Speaker 2: the time your system is unstable and it's gonna you know,
Speaker 2: this area will remain unstable at least for some period
Speaker 2: of time. Generally you have action items that your team
Speaker 2: is supposed to implement. Sometimes it's actually items are done,
Speaker 2: sometimes not. You know, there is always a priority issue
Speaker 2: with we need to release this future, just fix this
Speaker 2: potential bug. And the second thing is all the communications
Speaker 2: that's happening on Slack or teams or Zoom or Google Meter,
Speaker 2: you know and whatnot where that's how incidents are sold, right,
Speaker 2: it's human communicating between each other and sharing so much
Speaker 2: information that's business specific, right Like Ellen's are trained on
Speaker 2: a ton of data that's online, but it's not specific
Speaker 2: to a company obviously, right, And and and so we
Speaker 2: found that this data is that really boosts the results
Speaker 2: that we get out of these tools.
Speaker 1: Yeah, I mean, I think you you said it a
Speaker 1: different way. I like the context of you have to
Speaker 1: pull in the business criteria, understanding and context in order
Speaker 1: to have a valuable output. And I think it's you know,
Speaker 1: it can even be more than that. It's the fundamental
Speaker 1: nature of l ms that we have today, Like it's
Speaker 1: not a it's transformer architecture, which you know, is fundamentally
Speaker 1: lacking the reasoning piece, Like they'll never be able to reason,
Speaker 1: which means they'll never be able to make a decision
Speaker 1: based off of the business context. But they'll be able
Speaker 1: to do a little bit better of pulling that in
Speaker 1: and combining with the output that it would normally get. So,
Speaker 1: you know, my one of my questions is here, Okay,
Speaker 1: so building up a foundational model, and I think we've
Speaker 1: heard this before auntaventures and DevOps, and that's that it's
Speaker 1: incredibly expensive. Also, the industry is moving quicker that the
Speaker 1: new foundational models are are just as good, So spending
Speaker 1: money on building a new one doesn't make sense. Actually,
Speaker 1: I think we heard one time that even fine tuning
Speaker 1: models doesn't make sense because the next generation while like
Speaker 1: say anthropics three point seven cloud versus three point five,
Speaker 1: it's not it's not really that much of an improvement,
Speaker 1: but you are getting up to date data. If anything. Rather,
Speaker 1: you know the time stamp has changed, and if you
Speaker 1: spend time training it, refine tuning it. Rather then by
Speaker 1: the time the next one comes out, all you're fine.
Speaker 1: Tuning well, first of all is a waste, second all
Speaker 1: is expensive, and third of all, like you may be
Speaker 1: able to throw your quarries at your prompts at the
Speaker 1: new model and get the right answer out anyway, So
Speaker 1: it's good to hear that. Does that mean you're using
Speaker 1: some sort you're using like something from Olama or deep
Speaker 1: seek or something like that.
Speaker 2: We found that what entropy provide is generally, you know,
Speaker 2: the best performing. Ultimately, we are integrating a number of
Speaker 2: different model providers and we use different model a different
Speaker 2: step of the process. You know, I cannot explain in
Speaker 2: detail because it would be too long, but you know,
Speaker 2: when we're basically like the the agent will come up
Speaker 2: with an initial probe that you know, we compose and
Speaker 2: I would say, like different model will for instance, coming
Speaker 2: up with the let's say the master thesis of what
Speaker 2: you know we need to look for might be better created
Speaker 2: by some model, and then you know, the actual technical
Speaker 2: part maybe better than by another model. So it's and
Speaker 2: it's a moving target, as you said, like the industry
Speaker 2: is moving fast. There is a constant flow of new
Speaker 2: models and so so you know, I don't think it's
Speaker 2: something that really set in stone.
Speaker 1: Do you do something to validate model changes? So for instance,
Speaker 1: when three point seven came out, are you still using
Speaker 1: three point five before? Or you can, like you have
Speaker 1: some sort of ender templates or system prompts that you
Speaker 1: can throw out and validate that the answers still makes sense,
Speaker 1: that the RCAs and post mortems that you're doing still
Speaker 1: are understandable on match and somehow validating the outputs. Like
Speaker 1: what does this process work for you?
Speaker 2: Yeah, lung Chain as a bunch of open source tools
Speaker 2: that can allow you to do this. So we are
Speaker 2: constantly you know, tracing every like all the different you know,
Speaker 2: it's like kind of a tree right with different nerd
Speaker 2: and paths, and we keep track of everything that's being done,
Speaker 2: the reasoning, the output, and we constantly measure you know,
Speaker 2: the performance. So that's definitely something that we do. That
Speaker 2: being said, I think it's a challenge. It's still a challenge,
Speaker 2: you know to really understand the the quality, how the
Speaker 2: quality is shifting. They were a talk at Aserican in
Speaker 2: Santa Clara a few weeks ago. I think it was
Speaker 2: the AI director at ASIA who were speaking about one
Speaker 2: of the mobile based product that they are using, and
Speaker 2: it was saying that it's very hard for them to
Speaker 2: understand how to measure that, and they are relying on NPS.
Speaker 2: So it's a neutral promo Neutral Promoter Score, which is
Speaker 2: basically an industry standard rating, which is like would you
Speaker 2: recommend this to your friend and family to assess how
Speaker 2: their model are doing? And I think that was like
Speaker 2: really shocking to the audience.
Speaker 1: I mean, we do not We do know from experience
Speaker 1: that like NPS is like totally wrong from the net standpoint,
Speaker 1: because you should never ask from a human psychology standpoint,
Speaker 1: you should never ask one what they would do, but
Speaker 1: metrics on what they have done. And I think that's
Speaker 1: often the problem. But I mean, I think it really
Speaker 1: goes to show that there is no good way of
Speaker 1: adequately measuring these things. You have to do it within
Speaker 1: context of like what your business is doing, you know,
Speaker 1: for instance, really being able to do the incident management.
Speaker 1: And I do think that at least I know I
Speaker 1: have this question, so I'm sure someone else does that.
Speaker 1: There you're getting to the point where you don't want
Speaker 1: to have to make the code changes to like go
Speaker 1: into GitHub or get lab or heaven forbid a bitbucket
Speaker 1: or one of the other ones to actually put up
Speaker 1: a poor request to fix the problem. Wouldn't it be
Speaker 1: great if there was another LM out there that had
Speaker 1: the context of the source code and everything, and you
Speaker 1: could just give it the output from routlely and have
Speaker 1: a different agent do that. And that for me means
Speaker 1: you need to somehow integrate with other agents. And I
Speaker 1: can't believe I'm saying this, but MCP model context protocol, Like,
Speaker 1: how do you feel about that?
Speaker 2: I'm a huge fan of this. I think you are, Like,
Speaker 2: that's exactly the architectures that you need to have in mind.
Speaker 2: Is not an agent is the collection of agents, And
Speaker 2: it can go as deep as like, let's say, like
Speaker 2: you are doing work with GitHub. You can have an
Speaker 2: agent working on commits one, on pull request one, you know,
Speaker 2: like on each type of different resources that you may
Speaker 2: have with gitub you really need to tailor the agent
Speaker 2: too for them to do their best job. Because again,
Speaker 2: like they don't always have the business logic and understanding
Speaker 2: that we do as a human and bringing this context
Speaker 2: in each of the small like sub set of agent
Speaker 2: is critical for MCP. I you know, I'm a big
Speaker 2: fan of MCP. I've been wearing an MCP badge at
Speaker 2: Aserican and Cuicon because I truly think it's it's the
Speaker 2: technology itself is nothing, you know, amazing, Like at the
Speaker 2: end of the day, it's just to get away to
Speaker 2: an API.
Speaker 1: I mean, I really want, I really want to stress
Speaker 1: that enough, Like anyone who's not cut off on this,
Speaker 1: like it's nothing special, Like just imagine you deploy a
Speaker 1: new API or verse proxy CDN in front of your
Speaker 1: existing software and you're mapping from one protocol to another
Speaker 1: one like from TCP to UDP, from h GDP to
Speaker 1: or from rest to g RPC. It's really just another one.
Speaker 1: And I think the joke right now is that the
Speaker 1: AS and MCP stands for security.
Speaker 2: Yeah, you know, I think I mean that. Yeah, I
Speaker 2: think we go back to the you know what we
Speaker 2: discussed at the beginning of the conversation. You have the engineer,
Speaker 2: we will be an athayer and obviously there is a
Speaker 2: lot of things that are wrong with this, with this protocol,
Speaker 2: it's not stable, it's full of bug securities. Absolutely not
Speaker 2: to concern. But I think what's interesting is the concept
Speaker 2: of breaking this world between UH this AIG and and
Speaker 2: all the resources that are out there, whether it's data
Speaker 2: or system and and m cps just think it as USB, right,
Speaker 2: it's facilitating the conversation between between these two entities. It's
Speaker 2: it's open source and and it really unleash a lot
Speaker 2: of power for for AI agent and data sources, which
Speaker 2: as you say, most of the time is going to
Speaker 2: be race API to communicate in a way that's very
Speaker 2: optimized because one there are many issues with agent to
Speaker 2: consume most of the time it's it's race APIs as
Speaker 2: we say, like they may not have the business logic.
Speaker 2: So getting an information from an API main equest you
Speaker 2: to do multiple calls to multiple route and you know,
Speaker 2: like the LM may not know what if it's feasible,
Speaker 2: they may not use the best way to do that
Speaker 2: may get lost. And so MCP is really enabling this,
Speaker 2: removing all of all of this complexity. And for instance,
Speaker 2: I built an m CP server for Rutely and what
Speaker 2: it allows the developer to do is to when they
Speaker 2: get paid and stuff opening, you know, the web up
Speaker 2: going to rut looking the incident and you know it
Speaker 2: taket time context switching, which we know is bad for developers.
Speaker 2: They can just ask into their favorite power, I get
Speaker 2: me the last latest incident is going to put up
Speaker 2: in their chat and assuming it's simple in US and
Speaker 2: there is in US data in the payload of the incident,
Speaker 2: you can ask in this case, I use yourself to
Speaker 2: fix the incident. And so you go from production incident
Speaker 2: to resolution in a matter of a minute. Again, it's
Speaker 2: as you said, it's you know, some people were like, yeah,
Speaker 2: it's a joke, it's it's not truly, it's not revolutionary,
Speaker 2: it's not. But I think what's great is that it
Speaker 2: allows workflows to be done and it reduced a lot
Speaker 2: of friction. And we see a lot of companies in
Speaker 2: customer like Canvas and Bricks. They are huge engineering organization
Speaker 2: and they're like, investigate so much into MCP because they
Speaker 2: want their developer to remain where they produce the most value,
Speaker 2: which is in their idea, and so they are trying
Speaker 2: to bring as many you know, ICP server and then
Speaker 2: it doesn't matter if it's ICP. Actually, IBM really is
Speaker 2: a competitive protocol which is called ACP, which does the
Speaker 2: same thing. But you know, they're trying to bring all
Speaker 2: the contexts and the context that engineers need to do
Speaker 2: their work into the idea, and MCP is allowing just this.
Speaker 1: I think I'll be remissive. I didn't bring up Randall
Speaker 1: Monroe's comic on the we have fourteen competing standards for this,
Speaker 1: you know what, we need one universal standard to do this,
Speaker 1: you know, and then time later we have fifteen competing
Speaker 1: standards for this. I mean, because there really are. There's
Speaker 1: like a Tobos came out with not long ago Smithy
Speaker 1: for u h GDP services design pattern for documenting their APIs.
Speaker 1: We had open API specification. It's on version three point
Speaker 1: one right now, so that's you know, three versions later,
Speaker 1: and there's a whole bunch of these that different companies use,
Speaker 1: and I think the biggest trouble a lot of them
Speaker 1: have is that, like we have open API specification for authors,
Speaker 1: is that even if getting a human to understand what
Speaker 1: was written there is quite challenging, and so like feeding
Speaker 1: that into a model is you know, nonsensical, Like it's
Speaker 1: just not going to get you the word. As you
Speaker 1: pointed out, often the pattern is multiple things. I mean,
Speaker 1: we have things like graph ql, which you know has
Speaker 1: its own problems and whatnot. So I think we're just
Speaker 1: going to keep seeing more of these and I don't
Speaker 1: think we're ever going to really be able to settle
Speaker 1: on one. It would be nice if we could have one.
Speaker 1: The thing about MCP is, even if we pretend for
Speaker 1: one moment, does the worst thing in the world. As
Speaker 1: you pointed out, like I think Azure, GCP, and ABS
Speaker 1: all released MCP servers for they're built in like AI products,
Speaker 1: so you can interact with AWS better rock through an
Speaker 1: MCP server and like, so irrelevant what you think about that.
Speaker 1: It now exists and large companies have put some effort
Speaker 1: behind it, and maybe they're just trying to capture some
Speaker 1: of the market share and later things can evolve. I
Speaker 1: do think that, especially if a lot of companies are
Speaker 1: going to speed over quality that we may not get
Speaker 1: for like that many more iterations of a protocol to work.
Speaker 1: I mean, any I'll take this over the using sound
Speaker 1: high frequencies to communicate between devices that have you know, LMS,
Speaker 1: like I don't need that. It can go over the internet, please, Like,
Speaker 1: that's where I'm comfortable with my security. I'm not comfortable
Speaker 1: with things going through the airwaves because otherwise it's going
Speaker 1: to be the Alexa. Please order me, you know, another
Speaker 1: twenty four roll of toilet paper from an advertisement running
Speaker 1: on my television and actually have it happen and like
Speaker 1: this is recorded, that doesn't happen. Like, so I I
Speaker 1: don't need that to happen. People will have this happening.
Speaker 1: So I think MCP is still a little bit more
Speaker 1: secure than some of these other protocols that are out there.
Speaker 2: It is. Yeah, you know again, I'm not you know,
Speaker 2: I'm not an MCP evangelist. I think I'm not vouching
Speaker 2: for the technology, but not the concept. I think there's
Speaker 2: some serious limitation, a lot of issue with it. I
Speaker 2: think one of the security I think we've already discussed.
Speaker 2: We won't let that. But I think one issue is,
Speaker 2: for instance, you spoke about open API, so you can
Speaker 2: fit actually your open MPI and MCP can use this
Speaker 2: as a reference, which is great because you know, if
Speaker 2: you if your API is constantly updated with the latest
Speaker 2: you know state and translated into an API, then you
Speaker 2: make sure you're m CPCR is always up to date.
Speaker 2: What we found out at Rutely is that because we
Speaker 2: work with large corporations like LinkedIn, Canva and Cisco and
Speaker 2: so on and so for, they have like very specific
Speaker 2: requests in how they want to run their internet management.
Speaker 2: So our API is very verbal. We have a lot
Speaker 2: of routes to please our customer, and if you expose
Speaker 2: all of this to MCP, it's going to get lost
Speaker 2: in it, even though it's supposed you know to do this,
Speaker 2: So you need to restrict the amount of route that
Speaker 2: you expose. And the second thing is even at the
Speaker 2: next level in the MCP server chain is within the
Speaker 2: client in the editor like what people recommend, you can
Speaker 2: have up to five, two ten MCP server. After that
Speaker 2: your local agent is going to get lost because again
Speaker 2: too much context. So you know that this technology is
Speaker 2: I don't know if it's going to mature or something
Speaker 2: is going to replace it. You know, then you need
Speaker 2: to envision maybe something that centralized this MCP server into
Speaker 2: the central hubs so you don't have to configure like
Speaker 2: fifty of them. But I think it's on the right
Speaker 2: track and I think we see adoption and but but yeah,
Speaker 2: we will see where this move open AI recently and
Speaker 2: now that they are supporting MCP, which you know is
Speaker 2: is interesting because they're competing with Entropy. So yeah, I
Speaker 2: think there will be more of this for sure.
Speaker 1: But I actually my pack at the end of the
Speaker 1: episode will actually related to that. So I think it's
Speaker 1: really interesting that you brought that up. Yeah, I mean
Speaker 1: there's a lot. There's a lot there realistically, and I
Speaker 1: don't like, unless you need it, you probably don't need
Speaker 1: to spend any time looking at the MTP uh. You know,
Speaker 1: it's highly specific here for for agents talking communicating with
Speaker 1: each other. I think the hard problem that will get
Speaker 1: to very quick is at scale, being concise and meaningful
Speaker 1: and focused on what the business value is is going
Speaker 1: to be even more important. And arguably it has always
Speaker 1: been important, but it's very easy to add another route
Speaker 1: to your open API specification or your you know, your
Speaker 1: your web service or whatever you're running and having users
Speaker 1: should be like, oh, they'll deal with it, right, they'll
Speaker 1: deal with the problem. And I think realistically, you know,
Speaker 1: you want to be as clear and concise about what
Speaker 1: you're offering and what your business is and what the
Speaker 1: product is offering, but still give your customer's freedom to
Speaker 1: utilize your product how you want. And now you are
Speaker 1: almost required to make it happen because of limited context
Speaker 1: windows for for LMS, for agents. For MCP is going
Speaker 1: to be even more a problem. I mean, you scared
Speaker 1: me by saying two to five. I feel like if
Speaker 1: you have any more than one, I think you really
Speaker 1: have to question, you know, what the thing is that
Speaker 1: you're fundamentally offering. I mean I do see platforms like
Speaker 1: at LASTI ends, where like you have may have one
Speaker 1: for Gyra and one for Confluent, you know, because that's
Speaker 1: like a knowledge base, and there's like the day issues
Speaker 1: and one for maybe the GIT server. Each one of
Speaker 1: those could potentially be a different server. You say you're
Speaker 1: not an evangelist, but you are the first person on
Speaker 1: this episode, on this podcast to come on and say MTP,
Speaker 1: so that I think, by definition makes you the evangelist.
Speaker 1: And I think there may be a good a good
Speaker 1: moment to switch over to PECKS. But before we do that,
Speaker 1: I'll ask you, you know, is there any one last thing
Speaker 1: that you want to share?
Speaker 2: Yeah, if you are curious about MCP, and you know
Speaker 2: I've been to Cube con Asserica and the vast majority
Speaker 2: of people still don't know about it. We are organizing
Speaker 2: an event on April twenty fourth at guitub in San Francisco.
Speaker 2: We love speaker from brother Bays, Entropic, Open the Eye, Guitub,
Speaker 2: Factory I, and a lot of other companies. We have
Speaker 2: demo and a panel. Well, we'll go over what the
Speaker 2: heck is MCP and I think mab broady, you know,
Speaker 2: as we would chart it, like where what does this
Speaker 2: mean for the industry and where is this going? So yeah,
Speaker 2: if you type m C P Rootly even guitub on
Speaker 2: Google or perhaps we can share this in the description
Speaker 2: of the episode.
Speaker 1: Yeah, for sure, they'll be a link. Okay, then I
Speaker 1: think it's a great point to move on to our
Speaker 1: our picks. Uh. So I'll go first. My pick is
Speaker 1: this short article online by ed Zeitron. He has a
Speaker 1: blog and it's called where the Money. Uh. It's he's
Speaker 1: arguing that there's no AI revolution. Uh. You know, if
Speaker 1: you look at companies like Anthropic and Open AI, they're
Speaker 1: funneling tons of money in to it and they're not
Speaker 1: getting the value out and so in a way they're
Speaker 1: doing the nice thing of subsidizing all our great AI usage,
Speaker 1: you know, so get it. While the fountain is going.
Speaker 1: Really's got a great one, it seems you know there
Speaker 1: are ones out there. Uh, it's just it's a really
Speaker 1: great breakdown of you know, how companies are supposed to work,
Speaker 1: how where the money is coming from, you know, where
Speaker 1: it's being spent, and challenging some of those assumptions. So
Speaker 1: if you are only optimistic about everything related to AI,
Speaker 1: I highly recommend reading the article because there's there's a
Speaker 1: bunch of really good points that are made that are
Speaker 1: are hard to argue against.
Speaker 2: Love it. Yeah, that's an interesting question. Yeah, I think
Speaker 2: the you know a gi and and you know the
Speaker 2: goal of great getting to this great intelligence, so you
Speaker 2: see that, you know, that's why the money is just bringing.
Speaker 1: Yeah, I mean there is this theory that basically we
Speaker 1: can spend literally all of humanity's resources to achieve this
Speaker 1: because once we have it, it will produce so much value.
Speaker 1: That's you know, that theory hasn't been proven yet, but
Speaker 1: I'll leave it to people to read the article. Who
Speaker 1: he's articulated this much better than I have. Okay, so
Speaker 1: you've got for us today.
Speaker 2: Well, I'm going it's going to be my pig that
Speaker 2: I wrote, and I know it's going to be controversial,
Speaker 2: which is why I want to share it even better.
Speaker 2: You know, we spoke a lot about this. We didn't
Speaker 2: speak a lot about this episode, but online everybody is
Speaker 2: speaking about vibe coding, and so I think what's coming
Speaker 2: for us SA is is incident vibing, because the amount
Speaker 2: of incidents that is going to come our way is
Speaker 2: going to probably going to increase. And more importantly, I
Speaker 2: think a lot of the fundamentals that makes an engineering
Speaker 2: organizations solid are going away. A few things. For instance,
Speaker 2: I think a team that knows their code base very well,
Speaker 2: it's kind of going away because humans are not doing
Speaker 2: the coding anymore, right, they are merely like reading it
Speaker 2: doing coveryview. Perhaps they will, you know, use another l LM,
Speaker 2: another model to do the could review of another model.
Speaker 2: But anyway, I think in general we know that the
Speaker 2: knowledge of the code base is going to go down.
Speaker 2: The other one is having matter experts in some fields,
Speaker 2: especially as your company grow. You know, let's say maybe
Speaker 2: you want someone with like very sharp on database or
Speaker 2: website or whatever it is. And this again is going
Speaker 2: away because of what I've just mentioned, but also because
Speaker 2: I think it's going to be increasingly harder for young
Speaker 2: professional to gain this experience and this flair that senior
Speaker 2: engineer have. And so what's the solution. I think it's
Speaker 2: incident vibing. And I think it's one of this story
Speaker 2: where if you cannot beat them, you should join them.
Speaker 2: And so in this article I speak about what some
Speaker 2: of the ways that the companies can can get ready
Speaker 2: with incident vibing.
Speaker 1: I love it, well, we'll share that like an OPEC
Speaker 1: section of the episode. I mean, I I both love
Speaker 1: and hate your pick honestly, because, like I am, I'm
Speaker 1: so with you that vibe coding is terrible. And if
Speaker 1: we look at the door Report or the episode we
Speaker 1: did on the Door Report from twenty twenty four, we
Speaker 1: see that the LAM sacrifice speed for quality. We also
Speaker 1: know that there's a huge problem coming and companies are
Speaker 1: still adopting it. So you have to live with the outcome,
Speaker 1: Like even if you are using lams as best as
Speaker 1: you can, you're gonna that means you're gonna get more incidents.
Speaker 1: And so I'm totally with you. I hate that this
Speaker 1: is happening, but Uh, there's no avoiding it. And so
Speaker 1: the next level is also viving the incident resolution. Okay,
Speaker 1: it is.
Speaker 2: And and we've seen companies, you know, hiring people engineer
Speaker 2: and and they cannot cut, they only have, they can
Speaker 2: only prompt And Yeah, whether you like it or it's,
Speaker 2: it's happening, it's coming. It's the future of software engineering
Speaker 2: in some capacity. And so I, you know, I just
Speaker 2: think we need to get ready for it. That's the
Speaker 2: only thing you can do.
Speaker 1: I mean, I love the per respective. You know, it
Speaker 1: doesn't matter if you agree or disagree with with utilizing it.
Speaker 1: It's it's happening. Uh. And that I'll say, thank you
Speaker 1: Silvin so much for coming on this episode and sharing
Speaker 1: your perspective and what really has I've been doing.
Speaker 2: Thank you very intriving me
Speaker 1: Yea, and thanks for all the listeners and viewers of
Speaker 1: this podcast.