WEBVTT 1 00:00:12.160 --> 00:00:20.760 Hey man, it's not ned. You're not why not? You know, 2 00:00:20.839 --> 00:00:23.280 it's just the way I feel this morning. 3 00:00:23.359 --> 00:00:25.120 Yeah, you've been had a little too much weekend. 4 00:00:25.160 --> 00:00:29.760 I think I'm Carl Franklin. That's Richard Campbell, you know. Okay, 5 00:00:29.879 --> 00:00:33.600 I won't do that again until marijuana is completely legal 6 00:00:33.640 --> 00:00:36.159 in all fifty states of America. Oh, it's going to 7 00:00:36.200 --> 00:00:42.520 be a while and then I'll hey man, how you doing, Richard? 8 00:00:42.840 --> 00:00:43.320 I'm good. 9 00:00:43.799 --> 00:00:45.880 I did have a bit of a crazy weekend. You know, 10 00:00:46.119 --> 00:00:48.759 between my birthday, my wife's birthday, and and many of 11 00:00:48.759 --> 00:00:51.679 our friends are all like this last two weeks of July, 12 00:00:51.880 --> 00:00:54.560 so we bought them all up to the coast and 13 00:00:54.560 --> 00:00:57.399 it's yeah, I've been a week in a debauchery, honestly. 14 00:00:57.600 --> 00:00:59.359 Yeah, your nose does look a little red. 15 00:00:59.520 --> 00:01:02.000 Actually, I'm a little pinked up, a little pinked up 16 00:01:02.039 --> 00:01:05.319 today doing some damage. But you know, yeah, there's a 17 00:01:05.359 --> 00:01:07.239 couple of buddies up here that are literally known for 18 00:01:07.239 --> 00:01:09.079 forty years. Like that's when I'm wow, get old. 19 00:01:09.400 --> 00:01:10.280 That's so cool though. 20 00:01:10.400 --> 00:01:13.400 Yeah, it's something. Yeah, how about you. You're staying out 21 00:01:13.400 --> 00:01:15.799 of trouble. You're playing I see, I see your calendar. 22 00:01:15.840 --> 00:01:17.239 You're playing summer time. 23 00:01:17.359 --> 00:01:20.640 Yeah. Yeah, the band is currently just on a tear 24 00:01:20.799 --> 00:01:21.200 right now. 25 00:01:21.439 --> 00:01:23.879 That's awesome, dude, And uh. 26 00:01:23.439 --> 00:01:27.680 We're getting our original bass player back in September. August 27 00:01:27.760 --> 00:01:31.200 thirty first is our current bass player's last date. Oh, 28 00:01:31.280 --> 00:01:34.120 I'll tell you a little band story. So it was 29 00:01:34.200 --> 00:01:39.280 during COVID that Kevin, our original bass player, basically stopped 30 00:01:39.280 --> 00:01:42.680 coming to rehearsals because of COVID and and he had 31 00:01:42.680 --> 00:01:46.079 a good reason too. He was immune deficient and it 32 00:01:46.120 --> 00:01:50.239 turns out he got leukemia. He had and his immune 33 00:01:50.359 --> 00:01:53.519 system was compromised. Because of all that stuff, he had 34 00:01:53.519 --> 00:01:56.560 to stay home and he was like, guys, I'm I'm out. 35 00:01:56.599 --> 00:01:59.439 I can't do this and you know, my wife and 36 00:01:59.519 --> 00:02:03.920 child all that. Yeah. So then he gradually got over it, 37 00:02:03.959 --> 00:02:06.359 and I kept calling him once in a while and said, hey, man, 38 00:02:06.400 --> 00:02:08.759 how you doing. He's like, yeah, I'm okay, but I'm 39 00:02:08.800 --> 00:02:10.759 still not. You know, I tried playing out a couple 40 00:02:10.719 --> 00:02:14.080 of times with you know, somebody else, and I'm still not. 41 00:02:14.800 --> 00:02:19.800 And then finally he's like, okay, I'm retiring. I'm in oh, 42 00:02:19.840 --> 00:02:23.400 but not till September. And it turns out that I 43 00:02:23.439 --> 00:02:28.000 called him because our current bass player Chris basically met 44 00:02:28.000 --> 00:02:32.639 a girl, sold his house, and moved three hours away 45 00:02:33.000 --> 00:02:33.719 and married her. 46 00:02:33.879 --> 00:02:36.039 Hopefully not necessarily an order, but okay. 47 00:02:35.840 --> 00:02:38.919 Yeah, yeah, yeah. But he said, don't worry, guys, I'm 48 00:02:39.000 --> 00:02:41.560 still in the band. And we're like, oh, yeah, right, 49 00:02:41.759 --> 00:02:47.039 you know so. But props to Chris three hours. Yeah, 50 00:02:47.639 --> 00:02:49.479 he didn't come to rehearsals so that, you know, we 51 00:02:49.520 --> 00:02:52.960 couldn't really learn any new stuff, but we could send 52 00:02:53.039 --> 00:02:54.759 him a song and say, hey, learn this, and he 53 00:02:54.800 --> 00:02:56.599 was pretty good about it. And he never had a 54 00:02:56.599 --> 00:02:59.159 problem on the gig. But he showed up at every 55 00:02:59.240 --> 00:03:03.400 gig three hours away. It's a lot of driving. But 56 00:03:03.520 --> 00:03:05.759 I was going to say that the whole purpose of 57 00:03:05.800 --> 00:03:08.639 this was that it comes September. See, Kevin knows a 58 00:03:08.639 --> 00:03:11.599 lot of our original tunes that Chris doesn't know, and 59 00:03:11.680 --> 00:03:13.879 a lot more than the Steely Dan songs which were 60 00:03:13.879 --> 00:03:16.599 known for. So we're going to be playing a lot 61 00:03:16.639 --> 00:03:20.639 more original tunes, hopefully in bigger venues. Yeah, more Franklin Brothers. 62 00:03:20.759 --> 00:03:23.199 It's cool. Anyway, I've taken up enough time with my 63 00:03:23.400 --> 00:03:27.840 stupid stories of music and bands. Let's play role. Let's 64 00:03:27.919 --> 00:03:37.840 roll the music for better no framework, awesome, man, what 65 00:03:37.919 --> 00:03:40.439 do you got. I can't believe I've never talked about 66 00:03:40.479 --> 00:03:46.240 this particular project before. It's curated by the dot Net Foundation. 67 00:03:47.080 --> 00:03:51.000 It's Fluent validation. Okay, it's a validation library for dot 68 00:03:51.000 --> 00:03:53.680 Net that uses a fluent interface. You know what that is? This? 69 00:03:53.759 --> 00:03:56.879 Do this? Do that? Mm hmm and lambda expressions for 70 00:03:56.960 --> 00:04:02.080 building strongly typed validation rules, right, okay, So you can 71 00:04:02.120 --> 00:04:05.000 create these rules and then you can obviously use them, 72 00:04:05.240 --> 00:04:07.840 and it's very popular. 73 00:04:08.080 --> 00:04:10.599 Well, and it's a way of building abstract validation rules 74 00:04:10.639 --> 00:04:13.840 you might actually reuse or ither than keep recoding it exactly. 75 00:04:14.159 --> 00:04:16.480 Yeah. Yeah, And you know, if you decide to do 76 00:04:16.560 --> 00:04:18.879 it on the fly, where do you start that information? 77 00:04:19.639 --> 00:04:21.480 Where does it go? Does it go in a database? 78 00:04:22.480 --> 00:04:24.399 You know? Does it change? How does it change? Who 79 00:04:24.480 --> 00:04:29.040 changes it? Like, there's so many, so many things. But yeah, 80 00:04:29.560 --> 00:04:34.399 eight point nine thousand stars. Yeah, I would learn it, 81 00:04:34.439 --> 00:04:37.439 love it. Man. That's clearly a great tool. It's cool, 82 00:04:37.480 --> 00:04:39.199 and I'm going to start using it. I can't believe 83 00:04:39.199 --> 00:04:43.040 that I haven't even known about this existence before. So 84 00:04:43.120 --> 00:04:44.480 that's what I got. Who's talking to us? 85 00:04:44.600 --> 00:04:46.800 Richard grabbed a comment off of the show eighteen ninety one, 86 00:04:46.800 --> 00:04:48.759 which you did back in March of twenty four with 87 00:04:48.920 --> 00:04:53.040 her friend Anthony LRB. Who did they he was he'd 88 00:04:53.040 --> 00:04:56.319 written that open source library for API observability. 89 00:04:56.680 --> 00:04:57.480 Yeah right. 90 00:04:57.639 --> 00:04:59.639 We had a great conversation with her. An API tool 91 00:04:59.680 --> 00:05:01.879 kit I was his two line. I know we're going 92 00:05:01.879 --> 00:05:04.399 to talk about APIs a bunch today. And our friend 93 00:05:04.439 --> 00:05:07.600 Matt Lacy actually commented on the show. He said, hey, 94 00:05:07.720 --> 00:05:10.279 ten plus years ago, I built something for API. Because 95 00:05:10.319 --> 00:05:12.480 this is your checking on the client end. I assume 96 00:05:12.519 --> 00:05:14.480 there was a potential business in it, but I couldn't 97 00:05:14.480 --> 00:05:16.759 work out how, you know, because building good software and 98 00:05:16.839 --> 00:05:20.040 selling software two different jobs, right, Like that's a different thing. 99 00:05:21.160 --> 00:05:23.240 Great to see API toolkitd in existence now I'm making 100 00:05:23.279 --> 00:05:26.319 APIs more reliable because that was our whole conversation, right 101 00:05:26.519 --> 00:05:29.759 was you know you changed some you changed an API 102 00:05:29.920 --> 00:05:33.000 and somebody with a dependency on it suddenly goes down. Yeah, 103 00:05:33.279 --> 00:05:35.720 you make people sad in a big hurry, and so 104 00:05:36.079 --> 00:05:38.800 API toolkit was all about doing that validation. Hey, Matt, 105 00:05:38.920 --> 00:05:40.720 thanks so much for being a long time listener. And 106 00:05:41.000 --> 00:05:42.360 I don't know if you have a copy of Music 107 00:05:42.439 --> 00:05:44.879 Code Buy already, but you do now if you'd like 108 00:05:44.879 --> 00:05:46.399 a copy of Music code By, I write a comment 109 00:05:46.439 --> 00:05:48.240 on the website at don at Rocks dot com or 110 00:05:48.279 --> 00:05:50.519 on the facebooks. We publish every show there, and if 111 00:05:50.519 --> 00:05:52.000 you comment there and every reading the show, we'll send 112 00:05:52.040 --> 00:05:54.120 you a copy of music code By and Music to code. 113 00:05:53.959 --> 00:05:56.040 By for those who don't know, is something you can 114 00:05:56.120 --> 00:05:59.920 listen to while you're coding, or to calm a restless dog, 115 00:06:00.519 --> 00:06:02.360 or to put your children to sleep at night. 116 00:06:03.399 --> 00:06:07.399 Here I thought that was the nuclear weapons geek out 117 00:06:07.439 --> 00:06:08.639 for putting children to sleep. 118 00:06:08.839 --> 00:06:11.879 Oh come on, now, that was amazing, and it always 119 00:06:12.040 --> 00:06:14.720 is every year, but I was that particular one. 120 00:06:14.759 --> 00:06:17.439 Somebody told me it's like what I was depressed. So 121 00:06:17.519 --> 00:06:19.360 the key, I'm a little more level in that. 122 00:06:19.480 --> 00:06:19.680 Yeah. 123 00:06:19.680 --> 00:06:21.600 And I did almost all of the talking too, So 124 00:06:21.839 --> 00:06:23.639 apparently it's pretty good knocking kids out. 125 00:06:23.879 --> 00:06:27.319 Well, there you go. Hopefully we'll have something more positive 126 00:06:27.439 --> 00:06:27.759 to say. 127 00:06:27.800 --> 00:06:29.680 This is only positive about nuclear weapons. 128 00:06:29.759 --> 00:06:34.839 Go on, Well you know, okay, Well let's get into it. 129 00:06:34.959 --> 00:06:37.560 We're gonna introduce our guest right now. I'm going to 130 00:06:37.639 --> 00:06:42.240 introduce our guest. Andrea Kamenev is our guest, and from 131 00:06:42.319 --> 00:06:46.120 twenty sixteen to twenty twenty four, Andrea worked at Microsoft 132 00:06:46.240 --> 00:06:49.360 in various architect roles in Europe helping customers to bring 133 00:06:49.439 --> 00:06:53.199 their applications to Azure. Now he works as a product 134 00:06:53.279 --> 00:06:57.319 manager at Azure Api Management. All that do that we 135 00:06:57.439 --> 00:06:59.959 were kind of just talking about, but in the Azure way. 136 00:07:00.759 --> 00:07:04.079 Welcome Andre, thanks for hearing me, Thanks for being here. 137 00:07:04.879 --> 00:07:08.040 We first started like the quick start teams, those folks 138 00:07:08.120 --> 00:07:12.199 that helped onboard people into the cloud. Did that digital transformation. 139 00:07:11.759 --> 00:07:14.279 Thing, Uh, I mean the the service by itself. 140 00:07:14.600 --> 00:07:17.600 Yeah, your earlier role before you joined the product team. 141 00:07:17.839 --> 00:07:19.959 Yeah, yeah, So I was a part of what was 142 00:07:20.040 --> 00:07:23.480 called here at Microsoft Global Black Built Team. Okay, so 143 00:07:23.600 --> 00:07:27.480 it's like a bunch of cloudsision architects who help local 144 00:07:27.600 --> 00:07:32.279 teams like field engineers and customer social architects in the 145 00:07:32.360 --> 00:07:36.199 MEA region to build stuff with Azure. So our team 146 00:07:36.240 --> 00:07:38.560 was mostly focusing on Kubernatis related stuff. So I was 147 00:07:38.600 --> 00:07:41.759 working a lot with customers on bringing workloads to Azurecubneti 148 00:07:41.839 --> 00:07:43.720 service asually had to open shift and so on. 149 00:07:43.959 --> 00:07:46.560 So yeah, deep, So the move over to API management 150 00:07:46.600 --> 00:07:49.319 makes sense because that's a lynchpin problem when you expose 151 00:07:49.360 --> 00:07:52.439 stuff on the cloud that was typically just on prem before. 152 00:07:53.439 --> 00:07:53.600 Yeah. 153 00:07:53.639 --> 00:07:56.519 Absolutely, Yeah, we've seen a lot of customers who interested 154 00:07:56.519 --> 00:07:59.120 in APIM like and even even back then when I 155 00:07:59.240 --> 00:08:01.759 was not a part of a PAM team, like, okay, 156 00:08:01.839 --> 00:08:03.560 I have a bunch of APIs and my cuminator is 157 00:08:03.560 --> 00:08:05.800 how do they expose them securely? Like what what can 158 00:08:05.839 --> 00:08:06.319 you go for me? 159 00:08:06.439 --> 00:08:06.920 Microsoft? 160 00:08:07.360 --> 00:08:08.920 And then I think back back then there was a 161 00:08:09.000 --> 00:08:11.680 self hosted gate when it is it is still out there. 162 00:08:12.560 --> 00:08:13.279 It was a solution. 163 00:08:13.560 --> 00:08:16.759 So yeah, so what are you working on these days? Yeah? 164 00:08:16.800 --> 00:08:21.040 So these days, I guess there's a lot of interest 165 00:08:21.199 --> 00:08:24.399 in gen ai in large language models. Chugupt is all 166 00:08:24.439 --> 00:08:28.800 over the place. So goodness, right now, we in I 167 00:08:28.879 --> 00:08:31.000 believe in May, Yeah, in May we released the gen 168 00:08:31.040 --> 00:08:34.840 Ai Gatewakypical just Nature pay Management to help customers build 169 00:08:35.559 --> 00:08:37.480 like intelligent applications with lll ms. 170 00:08:37.519 --> 00:08:40.679 So yeah, that's I have a great idea. Let's give 171 00:08:40.759 --> 00:08:43.519 our AI all of our API keys and let them 172 00:08:43.559 --> 00:08:44.639 do whatever they want to do with it. 173 00:08:45.600 --> 00:08:48.799 Yeah, that's that's actually a better approach. That's that's what 174 00:08:49.120 --> 00:08:50.840 That's one of the things that were actually trying to 175 00:08:50.919 --> 00:08:52.320 solve for customers. 176 00:08:52.159 --> 00:08:54.679 Right, I know. Yeah, it just sounds crazy, doesn't it, 177 00:08:54.960 --> 00:08:58.480 given all the y the AI hiccups, and things that 178 00:08:58.639 --> 00:09:01.840 people don't really trust them. But I mean, so let's 179 00:09:02.039 --> 00:09:05.440 let's bust that myth. You know, why would we use 180 00:09:05.679 --> 00:09:10.200 AI to make our API calls, manage our APIs do 181 00:09:10.279 --> 00:09:14.279 all those things that normally trusted folks to. Yeah. 182 00:09:14.360 --> 00:09:17.039 So here I think, like from APM side, we have 183 00:09:17.279 --> 00:09:20.960 two different stories. Like, first of all, we use gen 184 00:09:20.960 --> 00:09:25.000 AI ourselves to help customers write the policies that we 185 00:09:25.120 --> 00:09:29.679 have an API M. And the second thing is we 186 00:09:29.799 --> 00:09:34.000 have customers for building intelligent applications using Judge PT models, 187 00:09:34.200 --> 00:09:37.279 using other models that are out there in az REAI studio, 188 00:09:38.200 --> 00:09:40.600 and they have challenges because if you think about as 189 00:09:40.679 --> 00:09:43.519 open EI for example, it is still an API, you 190 00:09:43.639 --> 00:09:46.759 still have the same challenges when it comes to managing 191 00:09:46.840 --> 00:09:51.039 and securing access to APIs. So we built but they 192 00:09:51.120 --> 00:09:54.159 have specific kind of a number of challenges which are 193 00:09:54.279 --> 00:09:57.399 kind of a specific to l ms. So this is 194 00:09:57.480 --> 00:09:59.960 kind of the second part where we help customers to secure, 195 00:10:00.279 --> 00:10:03.320 manage and scale Open the Eye deployments for the applications 196 00:10:03.360 --> 00:10:07.120 with that's what we call JENNYI Gateway and HPA management. 197 00:10:07.200 --> 00:10:09.120 So what are some of the ways in which AI 198 00:10:09.240 --> 00:10:15.480 can be used with APIs. Obviously creating the management stuff 199 00:10:15.519 --> 00:10:22.000 around it. But would you necessarily trust an AI with 200 00:10:22.240 --> 00:10:25.480 your API keys and say, you know, here are the 201 00:10:25.600 --> 00:10:28.879 rules and times under which you would make these API calls. 202 00:10:29.000 --> 00:10:32.159 And I'm trying to put wrap my head around that. 203 00:10:32.679 --> 00:10:38.159 Yeah, I think yeahs as always, it depends, right, So Yeah, 204 00:10:39.639 --> 00:10:43.840 if you have like specific like access controls in place, 205 00:10:44.440 --> 00:10:46.799 why not Like if you, for example, that apimor you 206 00:10:46.840 --> 00:10:51.159 can you can provide the specific keys for the LLM 207 00:10:51.519 --> 00:10:54.440 to like enhance the experience for those who use those 208 00:10:54.559 --> 00:10:57.320 l ms you can and all that. Yeah, then why 209 00:10:57.360 --> 00:11:00.000 not Like you're not given the access to like full API, 210 00:11:00.159 --> 00:11:03.759 You're just giving access to a subset of operations. 211 00:11:03.679 --> 00:11:05.519 Sure, which are for example, wedn't. 212 00:11:05.240 --> 00:11:08.519 Leave, or they just have access to specific data that 213 00:11:08.720 --> 00:11:11.559 you're not really they're not, which is not really trick. 214 00:11:11.679 --> 00:11:14.159 So yeah, that's definitely why I will use it. 215 00:11:14.200 --> 00:11:18.200 I'd be definitely okay with gets, but posts and puts. 216 00:11:18.600 --> 00:11:21.879 Yeah, I don't know, yeahuse. 217 00:11:21.080 --> 00:11:23.039 Otherwise you'd have to make these rules yourself, right, I 218 00:11:23.080 --> 00:11:24.960 mean that that's the point here is that you got 219 00:11:24.960 --> 00:11:26.840 a machine learning model essentially that's figuring out what the 220 00:11:26.879 --> 00:11:28.399 optimal rules are for utilization. 221 00:11:28.679 --> 00:11:30.039 Yeah, so there is a lot. There is one thing 222 00:11:30.120 --> 00:11:32.639 so as I mentioned, like first thing we've built like 223 00:11:32.639 --> 00:11:35.360 an a PM. For example, we're helping customers with with 224 00:11:35.639 --> 00:11:41.000 lllms to configure a PM. So I guess you're kind 225 00:11:41.039 --> 00:11:44.120 of familiar with AM. We have this XML policies which 226 00:11:44.240 --> 00:11:46.360 can be pretty long documents. 227 00:11:46.840 --> 00:11:49.919 Yeah, so typically that you're doing the thing, you're trying 228 00:11:49.960 --> 00:11:52.519 to say no one user can do more than this 229 00:11:52.720 --> 00:11:55.759 many or if it's growing, you know, massively limited so 230 00:11:55.919 --> 00:11:58.679 you don't knock other people off and make sure they're 231 00:11:58.720 --> 00:12:01.399 using the right accounts like it's it's just forwarding. 232 00:12:01.799 --> 00:12:03.000 Yeah, a lot of stuff in there. 233 00:12:03.039 --> 00:12:04.720 You know where stuff is, and you know what failover 234 00:12:04.799 --> 00:12:07.480 modes look like, like yeah, AP. We've done a few 235 00:12:07.519 --> 00:12:10.879 shows on API management now and it's like pretty powerful stuff. 236 00:12:11.159 --> 00:12:14.600 You know, you're gonna pub put an API in a 237 00:12:14.679 --> 00:12:18.759 public like you're paying when everybody somebody calls that, so 238 00:12:19.000 --> 00:12:20.759 you kind of want to put governance around that. But 239 00:12:20.919 --> 00:12:22.960 write and all those rules like when you really dig 240 00:12:23.039 --> 00:12:25.320 into it, it's complicated. So that was my first thought 241 00:12:25.320 --> 00:12:27.120 when I thought about, Yeah, what do I want Generator 242 00:12:27.159 --> 00:12:29.200 I to do. It's like, look at what's actually going 243 00:12:29.279 --> 00:12:30.720 on and write me better rules. 244 00:12:30.600 --> 00:12:32.720 Yeah, exactly. And that's kind of the two kind of 245 00:12:32.759 --> 00:12:35.879 two use cases that we focused on with Copilot. Like, first, 246 00:12:36.000 --> 00:12:38.960 we we decided that, oh, we we know that writing 247 00:12:39.000 --> 00:12:41.720 policies is hard. We have like fifty sixty different policses 248 00:12:41.799 --> 00:12:45.320 nippets to do like Validay, Jotan retripolicies like write limits 249 00:12:45.320 --> 00:12:48.279 and stuff like that. So we decided like, let's have 250 00:12:48.399 --> 00:12:50.519 a let's have a way for customers to express and 251 00:12:50.799 --> 00:12:52.840 plant English, like, for example, I want to have a 252 00:12:52.919 --> 00:12:55.919 policy to write limit this API for you know, five 253 00:12:56.000 --> 00:12:59.360 for quest per second, and then Copilot will just explain 254 00:12:59.440 --> 00:13:01.320 on sorry not explain, but generate a policy for you 255 00:13:01.360 --> 00:13:03.559 and then just copy and paste this into the XM 256 00:13:03.679 --> 00:13:07.080 leditor and that's it. And another one is, as I 257 00:13:07.120 --> 00:13:10.679 mentioned that the second scenario, policies can become pretty long, 258 00:13:10.799 --> 00:13:12.960 like two hundred three hundred, and sometimes you don't even 259 00:13:13.000 --> 00:13:14.200 understand what's going on there. 260 00:13:14.679 --> 00:13:20.759 In XML, in XML exactly. To another theme on the 261 00:13:20.840 --> 00:13:24.559 show lately is we hate XML. Yeah, there's a use 262 00:13:24.639 --> 00:13:27.679 for AI right there. Hey, translate this XML to. 263 00:13:27.720 --> 00:13:30.759 Me in English exactly, And that's actually what we do 264 00:13:30.879 --> 00:13:33.559 with the second scenario. So yeah, you can just select 265 00:13:33.879 --> 00:13:37.360 XML whole thing or just a policy snippet and then 266 00:13:37.480 --> 00:13:39.960 you ask it to explain it to you and the 267 00:13:40.039 --> 00:13:44.399 fund stuff. It's only explaining just like oh, this policy does, 268 00:13:44.519 --> 00:13:47.240 this policy is that, but it also understands the context, 269 00:13:47.320 --> 00:13:49.679 like if you have too different variables, you have context, 270 00:13:49.840 --> 00:13:54.159 you have policy expressions some logic in there. It also 271 00:13:54.200 --> 00:13:57.120 will explain that, for example, and all you are doing 272 00:13:57.200 --> 00:14:00.360 validate job policy and you have this admin claim that 273 00:14:00.440 --> 00:14:03.120 you're checking. If you're checking the saddening claim, if it exists, 274 00:14:03.240 --> 00:14:06.240 then you are allowed to do this operation. So yeah, 275 00:14:06.240 --> 00:14:08.919 it's it's pretty it's pretty good in explaining policies because 276 00:14:09.440 --> 00:14:12.600 we also we are not using like the plane model, 277 00:14:13.000 --> 00:14:16.759 but we're also like using this it's called retrieval augmented 278 00:14:17.120 --> 00:14:20.360 generation pattern where we also have like policy snippets that 279 00:14:20.440 --> 00:14:23.639 are stored in a storage and this model can also 280 00:14:23.759 --> 00:14:27.240 use this policy snippets that we provided to better like 281 00:14:27.399 --> 00:14:30.320 respond with correct policies with better explanations. 282 00:14:30.360 --> 00:14:32.200 And so yeah, so the same way I would actually 283 00:14:32.240 --> 00:14:34.600 write policies is I go cut and paste from well 284 00:14:34.639 --> 00:14:37.759 written policies exactly. Yeah, you've trained a model on well 285 00:14:37.799 --> 00:14:39.840 written policy so that it has a good chance of 286 00:14:40.360 --> 00:14:41.840 expressing better ones. 287 00:14:41.679 --> 00:14:42.720 For a customer. 288 00:14:42.919 --> 00:14:48.159 Yeah, exactly, Okay, I mean I could see a few 289 00:14:48.159 --> 00:14:50.200 different things going on here at once because you and 290 00:14:50.240 --> 00:14:52.480 I'll include a link to this blog post here. You're 291 00:14:52.519 --> 00:15:00.559 also talking about using the the API APIM to manage 292 00:15:00.720 --> 00:15:05.039 utilization of the open Ai service because that stuff gets expensive, 293 00:15:05.279 --> 00:15:07.759 like exactly, Yeah, those tokens run away on you and 294 00:15:07.840 --> 00:15:09.000 like you're having a bad day. 295 00:15:09.360 --> 00:15:12.080 Yeah, yeah, that's that's that's actually an interesting use case 296 00:15:12.120 --> 00:15:14.360 because as I mentioned, we have customers who are trying out, 297 00:15:14.399 --> 00:15:19.360 they're building pocs, they're building small applications, and there's azual 298 00:15:19.360 --> 00:15:21.600 open Eye service and it makes it really easy for 299 00:15:21.720 --> 00:15:24.639 you to start. You just deploy open endpoint, you select 300 00:15:24.720 --> 00:15:26.799 for example, you have you want to have GPT four 301 00:15:26.879 --> 00:15:29.399 model and the end you're good to go, like you can. 302 00:15:29.840 --> 00:15:32.679 That's just an API. You get your ap I key, 303 00:15:32.759 --> 00:15:34.879 you import dais the care of your choice to your application, 304 00:15:35.080 --> 00:15:37.759 and and that's it. You're sending prompts, you're saving completions. 305 00:15:37.840 --> 00:15:43.480 Everything's fine, but then customers realize that okay token comes exactly, Yeah, 306 00:15:44.840 --> 00:15:47.399 there are tokens, and tokens is like something which is 307 00:15:47.679 --> 00:15:50.480 super important in edge open and in general and l 308 00:15:50.720 --> 00:15:53.799 MS you spend tokens for prompts, you spend tokens for completions, 309 00:15:54.519 --> 00:15:58.799 and even when you do play open I instance, there 310 00:15:58.919 --> 00:16:01.600 is a quota associate to your model which is expressed 311 00:16:01.639 --> 00:16:04.840 and TPM which is tokens per minute. Right, and then 312 00:16:04.879 --> 00:16:08.840 after all of these experimentations customers, they started to realize that, Okay, 313 00:16:08.919 --> 00:16:11.360 now we need to wait to manage this because okay, 314 00:16:11.440 --> 00:16:13.840 we've built our first POC. We have one team who 315 00:16:13.919 --> 00:16:16.799 developed this kind of a private preview app which is 316 00:16:16.840 --> 00:16:18.679 not full in production right now. But now we have 317 00:16:18.879 --> 00:16:22.039 ten different departments, ten different teams who also want to 318 00:16:22.080 --> 00:16:24.840 get access to this model, And now, how can I 319 00:16:24.919 --> 00:16:28.559 manage that? How can I limit the consumption per team, 320 00:16:28.639 --> 00:16:31.120 per department, per developer, How can I. 321 00:16:31.200 --> 00:16:34.320 Make sure signed costs out like my sessonment had is 322 00:16:34.440 --> 00:16:37.200 firmly on right now, It's like there's nothing better in 323 00:16:37.240 --> 00:16:39.720 this world. And being able to build out resources to 324 00:16:39.759 --> 00:16:41.120 the individual teams for what they do. 325 00:16:42.679 --> 00:16:44.639 Yeah, and that's and that's a huge issue, Like you 326 00:16:44.720 --> 00:16:46.759 need to figure out how many tokens were consumed by 327 00:16:46.759 --> 00:16:49.519 a specific team, sure what kind of model they used, 328 00:16:50.720 --> 00:16:53.759 And then like okay, at the beginning, you have one endpoint. 329 00:16:53.960 --> 00:16:56.159 But what if you want to have multiple endpoints because 330 00:16:56.519 --> 00:17:00.480 like you're going production, you want to scale. How do 331 00:17:00.559 --> 00:17:03.759 you all balance how do you like create circuit breaker 332 00:17:03.840 --> 00:17:06.200 rules to make sure that for example, okay, wile our 333 00:17:06.279 --> 00:17:10.039 first instance is throat out responses with four twenty nine, 334 00:17:10.319 --> 00:17:13.240 how can I fail over to a different endpoint? 335 00:17:13.480 --> 00:17:13.599 Right? 336 00:17:13.720 --> 00:17:16.920 Yeah, so, yeah, there are a lot of challenges. Now 337 00:17:17.200 --> 00:17:21.319 you mentioned the given access API keys. Distributing API keys 338 00:17:21.359 --> 00:17:23.720 to all of these teams also doesn't sound like a 339 00:17:23.759 --> 00:17:27.960 good idea. So that's why we've built like a lot 340 00:17:28.000 --> 00:17:30.359 of stuff that is in this blog post for Jenny 341 00:17:30.400 --> 00:17:33.920 I announcement. We wanted to solve these challenges for customers 342 00:17:34.359 --> 00:17:36.920 who are kind of scaling and trying to like productize 343 00:17:37.039 --> 00:17:41.559 their their investment into as open THEI specifically, but also 344 00:17:42.400 --> 00:17:45.119 for other models like elms and stuff. 345 00:17:45.279 --> 00:17:45.480 Yeah. 346 00:17:46.359 --> 00:17:49.200 Certainly, one of the experiences I've dealt with with companies 347 00:17:49.240 --> 00:17:53.160 building a software into the cloud, even when they you know, 348 00:17:53.200 --> 00:17:55.799 they've got authentication and they're building back to the customer, 349 00:17:56.480 --> 00:17:59.039 the customer makes a mistake with the API and racks 350 00:17:59.119 --> 00:18:02.640 up a couple a million transactions that were test transactions, 351 00:18:02.759 --> 00:18:04.839 Like they're not making money on the back end. Then 352 00:18:04.920 --> 00:18:07.559 you're sending them this ugly bill and they, you know, 353 00:18:07.759 --> 00:18:11.359 want help. In the meantime, you've also gotten an ugly bill, 354 00:18:12.160 --> 00:18:13.880 you know, because you ran it on the back end. 355 00:18:13.920 --> 00:18:16.400 So this is this whole game of like who's. 356 00:18:16.200 --> 00:18:17.200 Holding the bag here? 357 00:18:17.720 --> 00:18:19.400 You know, you don't want to punish your customer for 358 00:18:19.480 --> 00:18:23.039 making a mistake. If you do, you may lose them 359 00:18:23.039 --> 00:18:25.960 as a customer. You're not necessarily going to get remediated, 360 00:18:26.119 --> 00:18:28.519 you know, back to Azure too. But although I've certainly 361 00:18:28.559 --> 00:18:30.519 had that experience where I've done stupid stuff in Azure 362 00:18:30.519 --> 00:18:32.200 and called them like I'm really sorry I did this, 363 00:18:32.400 --> 00:18:34.039 or like yep, fine, I'll wipe it. 364 00:18:34.319 --> 00:18:38.200 Oh you're the guy. Yeah, we've been waiting for your call. 365 00:18:39.440 --> 00:18:39.920 What was that? 366 00:18:40.519 --> 00:18:47.799 But the business reality of this consumption model is you 367 00:18:47.920 --> 00:18:50.759 don't always get paid for the stuff that you used, 368 00:18:51.279 --> 00:18:55.039 right or and or are willing to like that's this. 369 00:18:55.400 --> 00:18:58.440 All of these mechanisms to me speak to let's catch 370 00:18:58.519 --> 00:19:00.480 why didn't you notice? Why didn't you catch it before 371 00:19:00.519 --> 00:19:03.920 it ran away? You know, after the first million tokens? 372 00:19:04.200 --> 00:19:07.119 Why didn't you stop me? And these are the tools, 373 00:19:07.279 --> 00:19:10.799 right like, this is how this stops from being worse exactly. 374 00:19:11.000 --> 00:19:11.240 Yeah. 375 00:19:11.519 --> 00:19:14.119 Yeah, So we were trying to make sure that customers 376 00:19:14.200 --> 00:19:17.640 have the right tools to have the proper governance in place. 377 00:19:18.880 --> 00:19:20.960 So one of the things that you mentioned like tokens, 378 00:19:21.480 --> 00:19:24.839 So we introduced the So we already had like rate 379 00:19:24.880 --> 00:19:29.079 limiting policy that works for requests like you can say, 380 00:19:29.319 --> 00:19:31.880 as I mentioned previously, like five requests per second for example, 381 00:19:32.400 --> 00:19:34.440 and now we need we had to build something for 382 00:19:34.599 --> 00:19:37.119 tokens which is aware of these tokens, which is kind 383 00:19:37.119 --> 00:19:40.720 of the main currency of open the eye, as I mentioned. So, yeah, 384 00:19:40.720 --> 00:19:43.359 we introduced the stoken limit policy. It works pretty similarly 385 00:19:43.440 --> 00:19:46.480 to rate limit policy. You can say that, okay, we 386 00:19:46.599 --> 00:19:49.680 have this application, we have this department, we have this team. 387 00:19:50.440 --> 00:19:55.759 Now we assign let's say that one thousand tokens permitted 388 00:19:55.799 --> 00:19:58.920 to this application to make sure that do not consume more. 389 00:20:00.519 --> 00:20:02.920 And yeah, and that that prot works pretty well. And 390 00:20:03.039 --> 00:20:07.240 if you want to be extra careful, you also want 391 00:20:07.319 --> 00:20:10.519 to you also can configure the policy to estimate the 392 00:20:11.200 --> 00:20:13.759 uh the tokens which are in the prompt So whenever 393 00:20:13.839 --> 00:20:15.680 there is a request coming with a prompt and you 394 00:20:16.039 --> 00:20:19.039 calculate the number of prompts, the number of tokens which 395 00:20:19.079 --> 00:20:21.240 is used in the prompt and then if we on 396 00:20:21.319 --> 00:20:24.160 APM side understand that it already exceeds the limit, we 397 00:20:24.200 --> 00:20:26.160 will not send this to the to the back end. 398 00:20:26.559 --> 00:20:28.440 Right, so you will consume in the first place, you're 399 00:20:28.440 --> 00:20:31.880 already pressing against the limit. Yes, how do you bubble 400 00:20:32.079 --> 00:20:33.319 up that you've hit a limit? 401 00:20:34.519 --> 00:20:34.559