WEBVTT 1 00:00:02.279 --> 00:00:05.919 Welcome back everyone to another episodes of ventures and DevOps. 2 00:00:06.000 --> 00:00:09.240 And I am really excited today because we're going to 3 00:00:09.320 --> 00:00:12.199 jump into one of the areas that I find personally 4 00:00:12.199 --> 00:00:15.800 really interesting, but also our guest has worked at a 5 00:00:15.880 --> 00:00:19.120 number of companies in areas that I feel like lots 6 00:00:19.160 --> 00:00:21.120 of companies get wrong. So I just want to welcome 7 00:00:21.120 --> 00:00:24.480 to the show. Sylvan from from Rutley, who is the 8 00:00:24.519 --> 00:00:26.800 head of veloper Relations. Hey, how are you doing? 9 00:00:27.760 --> 00:00:29.640 Thank you for having me. We're in all good. 10 00:00:30.839 --> 00:00:33.840 Good to hear, so, I said, head of velper Relations, 11 00:00:33.880 --> 00:00:35.560 And I got to be honest, I feel like a 12 00:00:35.560 --> 00:00:38.759 lot of companies have started to over utilize this term 13 00:00:38.799 --> 00:00:43.200 to mean a wide variety of different roles and responsibilities. 14 00:00:43.560 --> 00:00:45.200 Can you give me like a breakdown like what it 15 00:00:45.240 --> 00:00:46.479 means for you today? 16 00:00:47.359 --> 00:00:51.759 Yeah? Absolutely, Actually to a great question because I have 17 00:00:52.560 --> 00:00:56.479 just puts it on LinkedIn two days ago when I 18 00:00:56.640 --> 00:01:02.520 was higher and historically the role of developer relations is 19 00:01:02.600 --> 00:01:11.319 to empower developers to use a product developer tool by 20 00:01:11.359 --> 00:01:16.560 providing education resources, by answering any questions they may have, 21 00:01:17.439 --> 00:01:24.120 and then just overall marketing you know, the product in 22 00:01:24.159 --> 00:01:28.079 a way that fits engineer, which is not product marketing, right, 23 00:01:28.120 --> 00:01:31.359 it's all about what you get out of it. Don't 24 00:01:31.439 --> 00:01:36.640 don't tell, show me, So it's tutorials, talks, you know 25 00:01:36.640 --> 00:01:39.560 in the former article, YouTube, video, and so on and 26 00:01:39.560 --> 00:01:44.000 so forth. But as I joinedly, it was clear that 27 00:01:45.200 --> 00:01:50.200 AI and more specifically here llms are the future of 28 00:01:50.719 --> 00:01:55.040 incident management. For those who don't know, rutly Routly is 29 00:01:55.480 --> 00:01:59.719 and on call and incident management platform. So when something 30 00:01:59.760 --> 00:02:03.040 breaks and you have people on call we need to 31 00:02:03.079 --> 00:02:06.920 respond to the incident, that's where they go to manage 32 00:02:06.959 --> 00:02:11.560 the incident up to it being solved. So I played 33 00:02:11.560 --> 00:02:13.280 a big role in that, and we'll speak about it 34 00:02:13.360 --> 00:02:17.360 during this episode and most of my time actually at 35 00:02:17.400 --> 00:02:20.520 truly most of my AERG, I would say like maybe 36 00:02:20.759 --> 00:02:25.639 seventy five percent has been dedicated to agent AI agent 37 00:02:25.719 --> 00:02:34.319 relations because we can see II agent as just another 38 00:02:34.479 --> 00:02:38.599 member of the team, and this agent also need to 39 00:02:38.639 --> 00:02:42.000 be taught and onboarded, just in a way that's different 40 00:02:42.039 --> 00:02:45.759 from humans. So I would say, while I'm the head 41 00:02:45.759 --> 00:02:49.479 of developer relation, can also say I'm the head of 42 00:02:50.080 --> 00:02:51.400 AI agent relations. 43 00:02:52.560 --> 00:02:54.240 Well, that's that's definitely a wide area. 44 00:02:54.280 --> 00:02:54.439 You know. 45 00:02:54.599 --> 00:02:57.840 I'm all interested because I found you mentioned this that 46 00:02:57.879 --> 00:03:00.719 you're not doing product marketing, you're still marked in some 47 00:03:00.800 --> 00:03:06.120 way to engineers. Uh, they're notoriously the biggest challenge to 48 00:03:06.319 --> 00:03:08.840 get engineers on board with whatever you're trying to sell. 49 00:03:08.879 --> 00:03:11.800 I mean I found of all groups of people, uh 50 00:03:12.479 --> 00:03:15.599 even ones in the technology space, I feel like engineers 51 00:03:15.599 --> 00:03:19.120 always want to do things themselves, right it. 52 00:03:19.000 --> 00:03:23.240 Is and and with already we are targeting SARES psyche 53 00:03:23.240 --> 00:03:28.319 truly ability engineers who are even more skeptical and hard 54 00:03:28.360 --> 00:03:30.919 to convince because and for good reason. Right, their job 55 00:03:30.960 --> 00:03:36.120 is to ensure that the infrastructure is rolling smoothly and 56 00:03:36.159 --> 00:03:39.039 in optimized fation. And so you want to be careful 57 00:03:39.439 --> 00:03:42.680 with the tool and the frame. A new framework or 58 00:03:42.800 --> 00:03:47.479 new tool or methodology that might include might bring chaos 59 00:03:47.560 --> 00:03:52.599 or instability. And uh, you know, so I used to 60 00:03:52.639 --> 00:03:58.159 be an SARE myself. Back then SARE was not truly 61 00:03:58.199 --> 00:04:00.680 a thing yet. I was working for flight Chair as 62 00:04:00.719 --> 00:04:04.840 a develops engineer. We were in the top fifteen months 63 00:04:04.919 --> 00:04:10.400 visited website in the world back then, displaying about one 64 00:04:10.439 --> 00:04:14.039 point five billion slide a day, which you know was 65 00:04:14.080 --> 00:04:17.839 definitely large volume. And we got acquired by LinkedIn, where 66 00:04:17.879 --> 00:04:20.439 I work as a senior SR. This time for three 67 00:04:20.519 --> 00:04:23.120 year and you can imagine the volume. So I've been 68 00:04:23.279 --> 00:04:26.040 on the side, on the engineering side, and so I 69 00:04:26.079 --> 00:04:29.920 completely you know, get the personal and I think at 70 00:04:29.959 --> 00:04:32.000 the end of the day, it's just that these people 71 00:04:32.040 --> 00:04:35.240 they don't want to waste time with marketing copies. They 72 00:04:35.240 --> 00:04:39.240 want to understand what's in it for them and their job. Right, 73 00:04:40.079 --> 00:04:43.279 So for me, it comes, you know, very naturally. I 74 00:04:43.279 --> 00:04:46.000 don't think it's a challenge, but I think for someone 75 00:04:46.040 --> 00:04:49.360 who does not come from an engineering background and doesn't 76 00:04:49.399 --> 00:04:53.759 have as good of an understanding as you know, an 77 00:04:53.759 --> 00:04:56.120 engineer may have, it may be hard to communicate to 78 00:04:56.160 --> 00:04:59.560 this audience where you know, I think this thing might 79 00:04:59.560 --> 00:05:01.560 come from. 80 00:05:01.639 --> 00:05:05.560 Yeah, no, I totally got it. I know I'm gonna 81 00:05:05.639 --> 00:05:07.800 I really want to dive into the core topic, which 82 00:05:07.839 --> 00:05:10.639 is self healing systems. And we were talking a little 83 00:05:10.639 --> 00:05:12.680 bit before the episode started about like this has been 84 00:05:12.720 --> 00:05:16.800 your like a lifelong project area. How did you get 85 00:05:16.839 --> 00:05:18.800 into this? I would say it's the first thing, like 86 00:05:18.879 --> 00:05:21.560 was this, Like you always knew this is this was 87 00:05:21.600 --> 00:05:23.959 the thing you're going to go into, and like how 88 00:05:23.959 --> 00:05:25.519 long you've been doing this and what does that really 89 00:05:25.560 --> 00:05:27.600 mean to work in self healing systems? 90 00:05:28.360 --> 00:05:31.399 Yeah, so it goes back to actually I started the 91 00:05:31.439 --> 00:05:34.920 project when I was working for a slight share as 92 00:05:34.959 --> 00:05:40.120 a develops engineer slate slash slate site reabity engineer, and 93 00:05:40.319 --> 00:05:42.759 I was on call and I had to you know, 94 00:05:42.839 --> 00:05:47.199 manage outages back then who are using puppet as a 95 00:05:47.240 --> 00:05:51.800 way to ensure that unfraustructure was as it should And 96 00:05:51.839 --> 00:05:55.879 you know, ultimately there were a lot of repeat incident 97 00:05:56.040 --> 00:06:00.920 or a lot of outage or you know issues that 98 00:06:01.959 --> 00:06:05.720 were coming from the same type of problems. I think. 99 00:06:06.560 --> 00:06:09.639 You know, as you as engineer grow in their careers, 100 00:06:09.639 --> 00:06:13.439 they kind of know what are the main failure types 101 00:06:14.639 --> 00:06:18.360 and so and so, you know, like I think it, 102 00:06:18.360 --> 00:06:21.000 it gets repetitive, and I think a great engineer wants 103 00:06:21.079 --> 00:06:25.040 to automate itself and not do the same thing over 104 00:06:25.079 --> 00:06:30.000 and over again. And so yeah, that's where the idea 105 00:06:29.279 --> 00:06:35.879 of building a self feeling system came about. So maybe just. 106 00:06:35.839 --> 00:06:37.839 Sort off like you said, you would see the same 107 00:06:37.879 --> 00:06:40.399 sort of regressions over and over again. Was there like 108 00:06:40.920 --> 00:06:43.360 one in your mind that was just like the most 109 00:06:43.360 --> 00:06:45.839 common where it was like every single time it happened. It 110 00:06:45.879 --> 00:06:47.680 was like the driver for you to make a real 111 00:06:47.800 --> 00:06:49.040 change in an organization. 112 00:06:50.160 --> 00:06:52.480 Yeah, you know, I think. I mean, I know it's 113 00:06:52.519 --> 00:06:55.959 been like more than a decade, so you know, I 114 00:06:56.000 --> 00:06:59.199 won't have like super sharp example, but I think the 115 00:06:59.519 --> 00:07:03.600 classic ones are, you know, issue with lack of resources, 116 00:07:03.600 --> 00:07:08.160 whether it's you know, storage or CPU or memory, and 117 00:07:08.759 --> 00:07:13.319 you need either to increase decrease the lots somehow or 118 00:07:13.879 --> 00:07:18.360 distribute or scale. Could be like a service that misbehaving 119 00:07:18.439 --> 00:07:22.800 and you need to restart. Could be a lot of 120 00:07:22.839 --> 00:07:27.079 things that I think, So I think the industry took 121 00:07:27.160 --> 00:07:30.480 different throughout. I think now with Kubernetists, what we do 122 00:07:30.519 --> 00:07:35.079 is that if if this is misbehaving, we just shut 123 00:07:35.079 --> 00:07:37.160 it down, right, we get rid of it and we 124 00:07:37.480 --> 00:07:41.839 start a new one. And obviously kubernets is great at scaling, 125 00:07:43.000 --> 00:07:48.040 so I think this tool to like the self feeling system, 126 00:07:48.160 --> 00:07:50.680 it still works in this way, right like by new 127 00:07:50.759 --> 00:07:54.480 king thing or scaling thing, you can heal a system. 128 00:07:55.199 --> 00:07:57.680 I think in my mind I wanted to take a 129 00:07:57.680 --> 00:08:03.040 different approach where I was training. I was envisioning a 130 00:08:03.079 --> 00:08:08.480 system that would actually address the root cause I think 131 00:08:08.519 --> 00:08:10.639 in some of the cases, you know, instead of just 132 00:08:10.720 --> 00:08:14.120 new kings, this thing like really try to mimic what 133 00:08:15.360 --> 00:08:19.240 a human engineer would do. So, Yeah, that's that's kind 134 00:08:19.240 --> 00:08:21.759 of the philosophy that I had back then. 135 00:08:23.360 --> 00:08:26.279 I mean, there's definitely a huge population of engineers who 136 00:08:26.360 --> 00:08:29.360 think that the what they would do in those examples 137 00:08:29.399 --> 00:08:32.360 would be for sure to restart the machine or the 138 00:08:32.399 --> 00:08:34.639 container or the node if it started to run out 139 00:08:34.679 --> 00:08:38.600 of memory or processing power. And I feel like that's 140 00:08:38.639 --> 00:08:40.200 sort of the crux of one of the issues that 141 00:08:40.240 --> 00:08:42.840 I've seen over and over again is that we do 142 00:08:42.919 --> 00:08:47.039 build those systems that I say we as collective humanity 143 00:08:47.159 --> 00:08:52.399 and not at my current company, that automatically restart or 144 00:08:52.600 --> 00:08:55.320 you know, allocate more memory or processing power. And I 145 00:08:55.320 --> 00:08:58.840 feel like the automatic scale scale out or scale up 146 00:09:00.080 --> 00:09:02.799 for resources can make sense if it doesn't create a 147 00:09:02.840 --> 00:09:06.279 negative impact on the feedback loop that you have to 148 00:09:06.320 --> 00:09:07.639 solve the problem. And I feel like this is one 149 00:09:07.679 --> 00:09:10.120 of the problems with automatic restarts is that it doesn't 150 00:09:10.159 --> 00:09:13.559 really solve the problem. It's still is persistent there, It's 151 00:09:13.559 --> 00:09:17.360 going to keep happening, and also you're delaying actually doing 152 00:09:17.399 --> 00:09:20.240 the investigation, and you're also eliminating some of the evidence 153 00:09:20.559 --> 00:09:23.200 that would allow you to identify the problem there. So 154 00:09:23.240 --> 00:09:25.720 it's really great to hear that you know, you thought 155 00:09:25.799 --> 00:09:28.480 that the appropriate process was, you know, go out like 156 00:09:28.519 --> 00:09:31.279 why is there extra memory usage? Why is you know, 157 00:09:31.279 --> 00:09:33.879 the machine getting stuck, et cetera, et cetera. And I 158 00:09:33.879 --> 00:09:36.840 feel like that's sort of the thing that sets apart 159 00:09:36.919 --> 00:09:40.480 the best esses from the ones that are just coming 160 00:09:40.480 --> 00:09:43.320 into quote unquote do the job it is. 161 00:09:43.519 --> 00:09:45.879 I think then you need to strike the right balance 162 00:09:45.919 --> 00:09:50.559 between achieving the end results, which is capability, stability, and 163 00:09:50.679 --> 00:09:53.279 if restarting is the way to go, and you know, 164 00:09:53.320 --> 00:09:56.879 you don't need to spend engineering resources. And obviously, I 165 00:09:56.919 --> 00:10:01.840 mean it's walking, so you know, I don't think it's 166 00:10:01.879 --> 00:10:04.080 it's a valid issue, but yeah, as you said, you 167 00:10:04.120 --> 00:10:08.200 need to find a strike the right balance between just 168 00:10:09.639 --> 00:10:13.639 doing this over and over and if it's repeat like 169 00:10:13.759 --> 00:10:19.559 out edsure issue, you know might need investigation. And so 170 00:10:21.000 --> 00:10:25.879 the idea that I had back in two Southern was 171 00:10:26.039 --> 00:10:31.120 I think I started in twelve thirteen was too really 172 00:10:31.919 --> 00:10:37.879 ingest as many data as you can from a distributed system, 173 00:10:38.120 --> 00:10:45.320 you know, whether it's any lugs, any metrics application, lugs, traces, 174 00:10:46.159 --> 00:10:48.600 and ingest all of this in the dallabas. Back then 175 00:10:48.639 --> 00:10:52.879 we were using fluend, which is an open source message, 176 00:10:52.879 --> 00:10:55.559 but we still exist and actually it's very popular. Back 177 00:10:55.559 --> 00:10:58.120 then I was we were one with lecture of the 178 00:10:58.240 --> 00:11:02.519 first main user big use there actually shout out to 179 00:11:02.519 --> 00:11:04.879 the team if they are listening. And now they went 180 00:11:05.000 --> 00:11:07.840 very far with this technology and still all of this 181 00:11:07.960 --> 00:11:13.240 in unstructured database. Back then we I bet on on 182 00:11:13.360 --> 00:11:15.679 Mongo DIB. It doesn't really matter the technology, but that's 183 00:11:15.799 --> 00:11:18.679 what we use for the prototype. And then based on 184 00:11:18.759 --> 00:11:24.120 that come up with like a state of a system 185 00:11:24.320 --> 00:11:29.919 and and use and then like try to resolve this 186 00:11:30.080 --> 00:11:34.799 issue by throwing at the system a bunch of actions 187 00:11:34.799 --> 00:11:38.279 that would be safe. You know, I'm speaking like any 188 00:11:38.320 --> 00:11:41.360 action like a like AREM action or something like you know, 189 00:11:41.440 --> 00:11:45.200 drop database like you need to be careful. But but 190 00:11:45.200 --> 00:11:50.519 but other set of safe action and then use machine 191 00:11:50.600 --> 00:11:55.720 learning as an engine to learn basically what, depending of 192 00:11:55.799 --> 00:11:58.960 the state of the system, what could solve the issue. 193 00:12:00.679 --> 00:12:05.320 And so we designed this for a way distributed infrastructure. 194 00:12:05.440 --> 00:12:10.200 Actually continued this work at LinkedIn and they asked me 195 00:12:10.279 --> 00:12:16.480 to write a patent, which eventually was accepted. But yeah, 196 00:12:16.480 --> 00:12:19.759 it never got a chance to build it, unfortunately, because 197 00:12:19.759 --> 00:12:23.039 then I left to become an entrepreneur. But that's why, 198 00:12:23.360 --> 00:12:26.559 you know, I was telling you have been swimming in 199 00:12:26.639 --> 00:12:28.360 this topic for a little while. 200 00:12:28.799 --> 00:12:31.000 Yeah, I mean I still want to go further into that. 201 00:12:31.120 --> 00:12:34.759 So just to summarize a little bit, the strategy is, 202 00:12:35.320 --> 00:12:38.519 we're collecting tons of logs, maybe metrics, et cetera. We 203 00:12:38.639 --> 00:12:41.919 maybe have access to the source code, and we train 204 00:12:42.080 --> 00:12:45.240 on that data to identify based off what sort of 205 00:12:45.320 --> 00:12:49.039 errors we're actually seeing, how to pinpoint potentially is it 206 00:12:49.080 --> 00:12:51.799 a part of the source code or the infrastructure which 207 00:12:51.799 --> 00:12:55.559 could be problematic, and then utilizing that on errors that 208 00:12:55.600 --> 00:12:58.679 actually do come out of the system to help dive 209 00:12:58.720 --> 00:13:01.159 in to identify the cause or does it go further 210 00:13:01.200 --> 00:13:01.440 than that? 211 00:13:02.000 --> 00:13:05.039 Yes, I think that's a good point if we speak 212 00:13:05.039 --> 00:13:09.000 about I think I think back then I was I 213 00:13:09.120 --> 00:13:15.519 directly dive into resolution. I was a young engineer, you know, 214 00:13:15.559 --> 00:13:19.200 I was like twenty five, twenty six, maybe even younger 215 00:13:19.240 --> 00:13:23.399 than this, so I was not really mature. But I 216 00:13:23.759 --> 00:13:29.519 think I think starting with the root cause analysis is 217 00:13:29.559 --> 00:13:33.679 the right approach. Obviously, you know now with insight it 218 00:13:33.759 --> 00:13:37.360 makes sense, but but my goal was really resolution, which 219 00:13:37.440 --> 00:13:40.919 ultimately you know where you want to go. So yeah, 220 00:13:40.960 --> 00:13:44.960 I was really really focusing on on direct resolution, and 221 00:13:45.159 --> 00:13:49.320 I would be a mix of kind of run books 222 00:13:49.840 --> 00:13:52.799 that that you know, we could feed and then but 223 00:13:52.960 --> 00:14:02.039 then more interestingly set of safe action safe commands that 224 00:14:02.080 --> 00:14:06.080 the system could run and could see if this solved 225 00:14:06.080 --> 00:14:10.320 the issue, and then kind of do like a learn 226 00:14:10.440 --> 00:14:13.039 from it, you know, maybe try something. It doesn't work, 227 00:14:13.080 --> 00:14:16.559 it's fine, you know, we just ditch this kind of 228 00:14:17.039 --> 00:14:20.200 pass instruction instruction set as an option, but sometimes it 229 00:14:20.240 --> 00:14:23.720 will succeed and it will use this for the next incident. 230 00:14:24.120 --> 00:14:27.200 And all of this would be based on machine learning obviously, 231 00:14:27.240 --> 00:14:30.399 like you know, like the more success you have with 232 00:14:30.440 --> 00:14:32.840 an instruction, the more you are likely to use it 233 00:14:32.919 --> 00:14:35.879 next time. Back then, machine learning was nearly not as 234 00:14:35.960 --> 00:14:41.519 advanced as it is today, you know, so it's it 235 00:14:41.600 --> 00:14:44.039 was hard like to achieve this goal. And I think 236 00:14:45.080 --> 00:14:48.799 the industry, I don't think anyone build this type of 237 00:14:48.879 --> 00:14:55.159 system until like now you add one player that did 238 00:14:55.200 --> 00:14:59.639 something similar, which is Facebook. They build a system that's 239 00:14:59.679 --> 00:15:05.679 called f bar F B A A R that the 240 00:15:05.759 --> 00:15:10.519 definers self feeling it was to manage Facebook that center racks, 241 00:15:10.759 --> 00:15:14.840 so it was not system but tracks where it would 242 00:15:15.519 --> 00:15:20.960 auto automatically perform action to solve some production issue. But 243 00:15:21.039 --> 00:15:25.720 it was deterministic, so it was not the well, no, 244 00:15:25.720 --> 00:15:30.240 none of machine learning used in it. And then Dropbox 245 00:15:30.320 --> 00:15:38.679 in twenty sixteen presented no rout Aserican and this was 246 00:15:38.799 --> 00:15:44.200 a self feeling system but for distribute distributed systems, this 247 00:15:44.279 --> 00:15:48.039 time for web infrastructure, but same year it was deterministic. 248 00:15:48.120 --> 00:15:50.679 So I you know, I think this system I've been 249 00:15:50.720 --> 00:15:53.919 around for like more than a decade, and I've been 250 00:15:53.960 --> 00:15:58.120 produce producing value. I know they've been in production, and 251 00:15:58.480 --> 00:16:03.120 I think as area generally not up with having mechanism 252 00:16:03.759 --> 00:16:06.480 or system working on their behalf and kind of in 253 00:16:06.519 --> 00:16:08.360 the shadow. But the truth is that they've been around 254 00:16:08.399 --> 00:16:11.399 for a while. I think. Now the main difference, the 255 00:16:11.440 --> 00:16:14.320 big difference, which is a huge difference, is that we 256 00:16:14.480 --> 00:16:20.279 are including this machine learning LLM part which is non deterministic. 257 00:16:20.399 --> 00:16:21.600 And that's a big deal. 258 00:16:22.200 --> 00:16:24.159 So let's just dive into that for a second. When 259 00:16:24.159 --> 00:16:27.360 we say deterministic in the history of self healing systems, 260 00:16:27.360 --> 00:16:31.600 we're talking about like auto scaling groups or identifying specifically 261 00:16:31.639 --> 00:16:34.919 based off of rules that some engineer wrote, what they're 262 00:16:34.960 --> 00:16:39.360 seeing and then how to handle the situation very concretely. 263 00:16:39.480 --> 00:16:40.080 Is that accurate? 264 00:16:40.480 --> 00:16:43.559 It's a great and also a one that was called 265 00:16:43.639 --> 00:16:46.720 earths and it's exactly how you describe it, like a 266 00:16:46.759 --> 00:16:49.759 runt book that's threaten by a human and then this 267 00:16:49.919 --> 00:16:53.000 rent book is a trigger based on a specific signal. 268 00:16:53.080 --> 00:16:54.919 But it's absolutely deterministic. 269 00:16:55.320 --> 00:16:58.679 Yeah, And I think the interesting thing with the deterministic 270 00:16:58.759 --> 00:17:01.559 systems is that really require you to do the root 271 00:17:01.559 --> 00:17:04.119 cause analysis so that you could write like if no 272 00:17:04.240 --> 00:17:06.160 run book applied or the run or a run book 273 00:17:06.160 --> 00:17:09.720 applied either by a human or through automation, didn't have 274 00:17:10.519 --> 00:17:13.920 an effect that actually resolved whatever incident you had, you 275 00:17:13.960 --> 00:17:15.839 had to actually still do the root cause analysis. And 276 00:17:15.920 --> 00:17:18.440 now I feel like we're getting into you know, I 277 00:17:18.440 --> 00:17:20.680 think everyone's waiting for us to talk about this how 278 00:17:20.680 --> 00:17:25.480 to apply Uh, I hate to say AI to this concept. 279 00:17:25.599 --> 00:17:28.640 So now that we're now that it's twenty twenty five, 280 00:17:29.839 --> 00:17:33.880 what does it mean to deploy LM to be able 281 00:17:33.920 --> 00:17:36.359 to self heal a system? What does that actually look 282 00:17:36.400 --> 00:17:37.039 like in practice? 283 00:17:38.279 --> 00:17:41.440 Yeah? So, you know, I think you hit the nail 284 00:17:41.440 --> 00:17:44.079 on the head with speaking about the root cause analysis. 285 00:17:44.079 --> 00:17:47.920 That's obviously the first step that this system need to 286 00:17:47.960 --> 00:17:53.119 do is to understand what's up. I think in the 287 00:17:53.160 --> 00:17:55.839 past it was a mix of for this self feeling 288 00:17:55.880 --> 00:18:02.440 system I think if part of this runt book where 289 00:18:03.160 --> 00:18:09.519 automatically based on a very specific maybe anologue or you know, 290 00:18:09.559 --> 00:18:13.519 something like very trivial, and then it would be automatically applied. 291 00:18:13.599 --> 00:18:16.400 In other cases, a human would need to do the 292 00:18:16.480 --> 00:18:19.599 root cause analysis and then you would push this rund book, 293 00:18:19.720 --> 00:18:22.000 which would still save a ton of time, right because 294 00:18:23.000 --> 00:18:25.680 this system would orchestrate all the action that needs to 295 00:18:25.680 --> 00:18:29.720 be done. And when we are speaking about very complicated 296 00:18:29.920 --> 00:18:33.960 infrasecure like at Google, Facebook or LinkedIn, you know, it 297 00:18:33.960 --> 00:18:37.240 can be a lot of work. But here the idea 298 00:18:37.240 --> 00:18:41.640 is that we can throw whatever broken system to this 299 00:18:41.839 --> 00:18:47.400 LLM and it should understand what's up or at least 300 00:18:48.000 --> 00:18:54.160 come up with hypothesis. And I think the hypothesis is 301 00:18:54.960 --> 00:18:58.759 very trivial, very important. Sorry, I don't think we need 302 00:18:58.880 --> 00:19:02.240 We should not consider this LM to be God or 303 00:19:02.279 --> 00:19:06.319 to be the silver bullet. We should consider them just 304 00:19:06.359 --> 00:19:10.480 as another human which can make mistakes. We make a 305 00:19:10.519 --> 00:19:13.799 lot of mistakes, and so I think one key element 306 00:19:13.960 --> 00:19:18.039 is to understand this that this ipothesis also kind of 307 00:19:18.200 --> 00:19:22.480 a degree of certainty, and so I think a great 308 00:19:22.559 --> 00:19:28.880 ais are will provide as part of the diagnosis what 309 00:19:29.079 --> 00:19:33.240 the degree of certainty that it has about the diagnosis, 310 00:19:33.400 --> 00:19:36.960 so human can say, hey, if it's fifty percent, maybe 311 00:19:37.000 --> 00:19:39.000 I should not pay too much attention attention to it. 312 00:19:39.000 --> 00:19:41.759 If it's ninety five percent, okay, maybe I should really 313 00:19:41.839 --> 00:19:42.440 look into it. 314 00:19:43.680 --> 00:19:48.880 So what sorts of source data are you utilizing in 315 00:19:49.000 --> 00:19:51.680 order to feed into the l I'm going to ask 316 00:19:51.720 --> 00:19:55.559 you questions about about that afterwards, but specifically right now, 317 00:19:55.640 --> 00:19:58.240 like is it you know a list of like source 318 00:19:58.319 --> 00:20:01.000 code and some other things like what does the sourt 319 00:20:01.000 --> 00:20:01.640 settle look like? 320 00:20:02.079 --> 00:20:07.240 M So I think with as with many tools in 321 00:20:07.279 --> 00:20:13.519 the LM air space today, context is king, and I'm 322 00:20:13.519 --> 00:20:16.440 going to speak about what we are building at Rutely, 323 00:20:17.480 --> 00:20:21.279 which again is an incident management platform. And why I 324 00:20:21.319 --> 00:20:24.680 repeat this because it really matters in the sense that 325 00:20:26.519 --> 00:20:30.000 engineering teams will all the signals that are associated with 326 00:20:30.039 --> 00:20:34.079 an incident will flow through their incident management platform. It 327 00:20:34.480 --> 00:20:38.200 just makes sense, right, And so these platforms such as 328 00:20:38.319 --> 00:20:43.559 routely have pretty much all the context that is available 329 00:20:43.640 --> 00:20:49.720 generally to solve this incident. So it can be monitoring, 330 00:20:50.720 --> 00:20:55.079 logging traces, it can be more than this. It can 331 00:20:55.160 --> 00:20:59.680 be slight conversation, right, it can be a zoom call 332 00:21:00.079 --> 00:21:01.960 here you know, you can do a transcript as a 333 00:21:02.039 --> 00:21:03.640 zoom call, so you can fit this in the l 334 00:21:03.720 --> 00:21:07.200 l M. But I think there also are very important 335 00:21:07.279 --> 00:21:12.079 data such as the history of post mortem incident resolution 336 00:21:12.319 --> 00:21:17.720 report right where everything is documented from what happened, how 337 00:21:17.759 --> 00:21:21.960 it happened, how did we solve the incident, how this 338 00:21:22.079 --> 00:21:25.400 incident like how it do I solve? Who solved it? 339 00:21:26.440 --> 00:21:31.079 And all this data is like super important for for 340 00:21:31.240 --> 00:21:35.599 the AI agent to to to find a root cause. 341 00:21:37.519 --> 00:21:39.759 And and the last one I I forgot you mentioned 342 00:21:39.960 --> 00:21:44.559 is obviously anything that's linked to changes, which you know 343 00:21:44.599 --> 00:21:47.400 takes the form of cod So the list of commits 344 00:21:48.119 --> 00:21:51.920 is often you know where you'll find the issue. 345 00:21:53.480 --> 00:21:55.319 So you've actually got a system that and justs all 346 00:21:55.319 --> 00:21:58.960 this information and that's out. You know, here's a here's 347 00:21:59.160 --> 00:22:01.440 some action items take And then I imagine some companies 348 00:22:01.440 --> 00:22:05.279 are actually automating based off of that to remediate the 349 00:22:05.319 --> 00:22:07.599 problems in production. Or is this like you need a 350 00:22:07.640 --> 00:22:10.880 human to review this before you do anything else. 351 00:22:13.079 --> 00:22:18.880 Yeah, I think the safe this field is extremely like 352 00:22:19.039 --> 00:22:23.440 the auto healing mechanism around a LAMB is extremely young. 353 00:22:24.440 --> 00:22:27.279 You know, the oldest company in the field maybe two 354 00:22:27.359 --> 00:22:30.720 year old, perhaps even less, there is a lot of 355 00:22:30.720 --> 00:22:36.559 competition in the space, seen at least twenty five products, 356 00:22:37.319 --> 00:22:40.480 and I've spoken to a lot of engineers who are 357 00:22:40.480 --> 00:22:46.160 building this internally at large companies, so you know, I 358 00:22:46.200 --> 00:22:50.000 think everybody's doing differently. The maturity of the product is 359 00:22:50.279 --> 00:22:54.039 also very different. But I think for s ARE is 360 00:22:54.079 --> 00:23:00.799 obviously a really ability is ultimately the goal, and not experimentation, 361 00:23:01.000 --> 00:23:05.960 right its secondary goal definitely Right, starting with just investigation 362 00:23:06.920 --> 00:23:10.680 is the right way to go. And then I think, 363 00:23:10.759 --> 00:23:16.000 as the space meteor and perhaps the model, what's work 364 00:23:16.039 --> 00:23:19.119 great with this technology? You can teach it right, It 365 00:23:19.200 --> 00:23:23.680 becomes better over time, so you can train models on 366 00:23:24.480 --> 00:23:28.160 your data, you can tune it right. So as the 367 00:23:28.200 --> 00:23:31.599 technology mature and the model is learning, and perhaps we 368 00:23:31.640 --> 00:23:38.839 are learning as humans, we can go more towards allowing 369 00:23:38.960 --> 00:23:40.720 this tool to do the resolution. But I think the 370 00:23:40.759 --> 00:23:44.240 first step in is just fun our root cause analysis. 371 00:23:45.079 --> 00:23:48.279 Yeah, I mean, at least from my personal standpoint, I'm 372 00:23:48.359 --> 00:23:53.200 scared to hand over the tools to make changes to 373 00:23:53.240 --> 00:23:59.599 production infrastructure automatically without involving some sort of review process. 374 00:23:59.799 --> 00:24:01.799 And I mean, I guess, I guess it's fine to 375 00:24:01.839 --> 00:24:05.200 have like another LM review. The first llm's work in 376 00:24:05.240 --> 00:24:07.960 some way, but I don't know if the direction matters. 377 00:24:08.000 --> 00:24:10.759 Like I think you need someone to review the context 378 00:24:10.759 --> 00:24:13.359 of what's happening, just like you probably want multiple engineers 379 00:24:13.920 --> 00:24:17.519 on call to actually validate any sort of code changes 380 00:24:17.559 --> 00:24:19.880 that would have to go into production, because I mean, 381 00:24:19.920 --> 00:24:22.240 otherwise you're in a situation where there's a critical event. 382 00:24:22.599 --> 00:24:25.720 It's three, something's going wrong with the database. You log 383 00:24:25.759 --> 00:24:30.039 in and accidentally drop the production dB. I mean, I'll 384 00:24:30.039 --> 00:24:32.119 pause there because that this has actually happened to more 385 00:24:32.119 --> 00:24:35.400 than one company, But I think there's one in our history. 386 00:24:36.000 --> 00:24:38.839 I think it's almost ten years old now, a major 387 00:24:39.440 --> 00:24:44.359 source code get server company had a production incident, very 388 00:24:44.359 --> 00:24:46.440 famous with their I think it was Postcress at the time. 389 00:24:47.799 --> 00:24:51.640 So engineer is definitely not infallible when it comes to remediation. 390 00:24:52.480 --> 00:24:54.759 But I guess my question is going to be do 391 00:24:54.799 --> 00:24:58.039 you find with all the data that you're collecting that 392 00:24:58.119 --> 00:25:02.519 the set of incidents all point back to some like 393 00:25:02.559 --> 00:25:06.000 as far as you're concerned, repeatable or already seen problems 394 00:25:06.039 --> 00:25:10.680 like oh yeah, this sort of software development issue or 395 00:25:10.680 --> 00:25:15.559 some syntax problem like no reference exception or dynamic module 396 00:25:15.640 --> 00:25:18.440 loading or you know, memory exhaustion or something like that, 397 00:25:19.160 --> 00:25:22.359 or are there like minor differences as time goes on, 398 00:25:22.440 --> 00:25:24.559 like oh well it used to be this said, but 399 00:25:24.960 --> 00:25:27.119 the next thing is sort of something that we haven't 400 00:25:27.119 --> 00:25:30.440 discovered yet, and so you're still discovering sort of new 401 00:25:30.440 --> 00:25:31.160