Speaker 1: Hey man, it's not ned. You're not why not? You know,
Speaker 1: it's just the way I feel this morning.
Speaker 2: Yeah, you've been had a little too much weekend.
Speaker 1: I think I'm Carl Franklin. That's Richard Campbell, you know. Okay,
Speaker 1: I won't do that again until marijuana is completely legal
Speaker 1: in all fifty states of America. Oh, it's going to
Speaker 1: be a while and then I'll hey man, how you doing, Richard?
Speaker 1: I'm good.
Speaker 2: I did have a bit of a crazy weekend. You know,
Speaker 2: between my birthday, my wife's birthday, and and many of
Speaker 2: our friends are all like this last two weeks of July,
Speaker 2: so we bought them all up to the coast and
Speaker 2: it's yeah, I've been a week in a debauchery, honestly.
Speaker 1: Yeah, your nose does look a little red.
Speaker 2: Actually, I'm a little pinked up, a little pinked up
Speaker 2: today doing some damage. But you know, yeah, there's a
Speaker 2: couple of buddies up here that are literally known for
Speaker 2: forty years. Like that's when I'm wow, get old.
Speaker 1: That's so cool though.
Speaker 2: Yeah, it's something. Yeah, how about you. You're staying out
Speaker 2: of trouble. You're playing I see, I see your calendar.
Speaker 2: You're playing summer time.
Speaker 1: Yeah. Yeah, the band is currently just on a tear
Speaker 1: right now.
Speaker 2: That's awesome, dude, And uh.
Speaker 1: We're getting our original bass player back in September. August
Speaker 1: thirty first is our current bass player's last date. Oh,
Speaker 1: I'll tell you a little band story. So it was
Speaker 1: during COVID that Kevin, our original bass player, basically stopped
Speaker 1: coming to rehearsals because of COVID and and he had
Speaker 1: a good reason too. He was immune deficient and it
Speaker 1: turns out he got leukemia. He had and his immune
Speaker 1: system was compromised. Because of all that stuff, he had
Speaker 1: to stay home and he was like, guys, I'm I'm out.
Speaker 1: I can't do this and you know, my wife and
Speaker 1: child all that. Yeah. So then he gradually got over it,
Speaker 1: and I kept calling him once in a while and said, hey, man,
Speaker 1: how you doing. He's like, yeah, I'm okay, but I'm
Speaker 1: still not. You know, I tried playing out a couple
Speaker 1: of times with you know, somebody else, and I'm still not.
Speaker 1: And then finally he's like, okay, I'm retiring. I'm in oh,
Speaker 1: but not till September. And it turns out that I
Speaker 1: called him because our current bass player Chris basically met
Speaker 1: a girl, sold his house, and moved three hours away
Speaker 1: and married her.
Speaker 2: Hopefully not necessarily an order, but okay.
Speaker 1: Yeah, yeah, yeah. But he said, don't worry, guys, I'm
Speaker 1: still in the band. And we're like, oh, yeah, right,
Speaker 1: you know so. But props to Chris three hours. Yeah,
Speaker 1: he didn't come to rehearsals so that, you know, we
Speaker 1: couldn't really learn any new stuff, but we could send
Speaker 1: him a song and say, hey, learn this, and he
Speaker 1: was pretty good about it. And he never had a
Speaker 1: problem on the gig. But he showed up at every
Speaker 1: gig three hours away. It's a lot of driving. But
Speaker 1: I was going to say that the whole purpose of
Speaker 1: this was that it comes September. See, Kevin knows a
Speaker 1: lot of our original tunes that Chris doesn't know, and
Speaker 1: a lot more than the Steely Dan songs which were
Speaker 1: known for. So we're going to be playing a lot
Speaker 1: more original tunes, hopefully in bigger venues. Yeah, more Franklin Brothers.
Speaker 1: It's cool. Anyway, I've taken up enough time with my
Speaker 1: stupid stories of music and bands. Let's play role. Let's
Speaker 1: roll the music for better no framework, awesome, man, what
Speaker 1: do you got. I can't believe I've never talked about
Speaker 1: this particular project before. It's curated by the dot Net Foundation.
Speaker 1: It's Fluent validation. Okay, it's a validation library for dot
Speaker 1: Net that uses a fluent interface. You know what that is? This?
Speaker 1: Do this? Do that? Mm hmm and lambda expressions for
Speaker 1: building strongly typed validation rules, right, okay, So you can
Speaker 1: create these rules and then you can obviously use them,
Speaker 1: and it's very popular.
Speaker 2: Well, and it's a way of building abstract validation rules
Speaker 2: you might actually reuse or ither than keep recoding it exactly.
Speaker 1: Yeah. Yeah, And you know, if you decide to do
Speaker 1: it on the fly, where do you start that information?
Speaker 1: Where does it go? Does it go in a database?
Speaker 1: You know? Does it change? How does it change? Who
Speaker 1: changes it? Like, there's so many, so many things. But yeah,
Speaker 1: eight point nine thousand stars. Yeah, I would learn it,
Speaker 1: love it. Man. That's clearly a great tool. It's cool,
Speaker 1: and I'm going to start using it. I can't believe
Speaker 1: that I haven't even known about this existence before. So
Speaker 1: that's what I got. Who's talking to us?
Speaker 2: Richard grabbed a comment off of the show eighteen ninety one,
Speaker 2: which you did back in March of twenty four with
Speaker 2: her friend Anthony LRB. Who did they he was he'd
Speaker 2: written that open source library for API observability.
Speaker 1: Yeah right.
Speaker 2: We had a great conversation with her. An API tool
Speaker 2: kit I was his two line. I know we're going
Speaker 2: to talk about APIs a bunch today. And our friend
Speaker 2: Matt Lacy actually commented on the show. He said, hey,
Speaker 2: ten plus years ago, I built something for API. Because
Speaker 2: this is your checking on the client end. I assume
Speaker 2: there was a potential business in it, but I couldn't
Speaker 2: work out how, you know, because building good software and
Speaker 2: selling software two different jobs, right, Like that's a different thing.
Speaker 2: Great to see API toolkitd in existence now I'm making
Speaker 2: APIs more reliable because that was our whole conversation, right
Speaker 2: was you know you changed some you changed an API
Speaker 2: and somebody with a dependency on it suddenly goes down. Yeah,
Speaker 2: you make people sad in a big hurry, and so
Speaker 2: API toolkit was all about doing that validation. Hey, Matt,
Speaker 2: thanks so much for being a long time listener. And
Speaker 2: I don't know if you have a copy of Music
Speaker 2: Code Buy already, but you do now if you'd like
Speaker 2: a copy of Music code By, I write a comment
Speaker 2: on the website at don at Rocks dot com or
Speaker 2: on the facebooks. We publish every show there, and if
Speaker 2: you comment there and every reading the show, we'll send
Speaker 2: you a copy of music code By and Music to code.
Speaker 1: By for those who don't know, is something you can
Speaker 1: listen to while you're coding, or to calm a restless dog,
Speaker 1: or to put your children to sleep at night.
Speaker 2: Here I thought that was the nuclear weapons geek out
Speaker 2: for putting children to sleep.
Speaker 1: Oh come on, now, that was amazing, and it always
Speaker 1: is every year, but I was that particular one.
Speaker 2: Somebody told me it's like what I was depressed. So
Speaker 2: the key, I'm a little more level in that.
Speaker 1: Yeah.
Speaker 2: And I did almost all of the talking too, So
Speaker 2: apparently it's pretty good knocking kids out.
Speaker 1: Well, there you go. Hopefully we'll have something more positive
Speaker 1: to say.
Speaker 2: This is only positive about nuclear weapons.
Speaker 1: Go on, Well you know, okay, Well let's get into it.
Speaker 1: We're gonna introduce our guest right now. I'm going to
Speaker 1: introduce our guest. Andrea Kamenev is our guest, and from
Speaker 1: twenty sixteen to twenty twenty four, Andrea worked at Microsoft
Speaker 1: in various architect roles in Europe helping customers to bring
Speaker 1: their applications to Azure. Now he works as a product
Speaker 1: manager at Azure Api Management. All that do that we
Speaker 1: were kind of just talking about, but in the Azure way.
Speaker 1: Welcome Andre, thanks for hearing me, Thanks for being here.
Speaker 1: We first started like the quick start teams, those folks
Speaker 1: that helped onboard people into the cloud. Did that digital transformation.
Speaker 3: Thing, Uh, I mean the the service by itself.
Speaker 1: Yeah, your earlier role before you joined the product team.
Speaker 3: Yeah, yeah, So I was a part of what was
Speaker 3: called here at Microsoft Global Black Built Team. Okay, so
Speaker 3: it's like a bunch of cloudsision architects who help local
Speaker 3: teams like field engineers and customer social architects in the
Speaker 3: MEA region to build stuff with Azure. So our team
Speaker 3: was mostly focusing on Kubernatis related stuff. So I was
Speaker 3: working a lot with customers on bringing workloads to Azurecubneti
Speaker 3: service asually had to open shift and so on.
Speaker 2: So yeah, deep, So the move over to API management
Speaker 2: makes sense because that's a lynchpin problem when you expose
Speaker 2: stuff on the cloud that was typically just on prem before.
Speaker 1: Yeah.
Speaker 3: Absolutely, Yeah, we've seen a lot of customers who interested
Speaker 3: in APIM like and even even back then when I
Speaker 3: was not a part of a PAM team, like, okay,
Speaker 3: I have a bunch of APIs and my cuminator is
Speaker 3: how do they expose them securely? Like what what can
Speaker 3: you go for me?
Speaker 1: Microsoft?
Speaker 3: And then I think back back then there was a
Speaker 3: self hosted gate when it is it is still out there.
Speaker 3: It was a solution.
Speaker 1: So yeah, so what are you working on these days? Yeah?
Speaker 3: So these days, I guess there's a lot of interest
Speaker 3: in gen ai in large language models. Chugupt is all
Speaker 3: over the place. So goodness, right now, we in I
Speaker 3: believe in May, Yeah, in May we released the gen
Speaker 3: Ai Gatewakypical just Nature pay Management to help customers build
Speaker 3: like intelligent applications with lll ms.
Speaker 1: So yeah, that's I have a great idea. Let's give
Speaker 1: our AI all of our API keys and let them
Speaker 1: do whatever they want to do with it.
Speaker 3: Yeah, that's that's actually a better approach. That's that's what
Speaker 3: That's one of the things that were actually trying to
Speaker 3: solve for customers.
Speaker 1: Right, I know. Yeah, it just sounds crazy, doesn't it,
Speaker 1: given all the y the AI hiccups, and things that
Speaker 1: people don't really trust them. But I mean, so let's
Speaker 1: let's bust that myth. You know, why would we use
Speaker 1: AI to make our API calls, manage our APIs do
Speaker 1: all those things that normally trusted folks to. Yeah.
Speaker 3: So here I think, like from APM side, we have
Speaker 3: two different stories. Like, first of all, we use gen
Speaker 3: AI ourselves to help customers write the policies that we
Speaker 3: have an API M. And the second thing is we
Speaker 3: have customers for building intelligent applications using Judge PT models,
Speaker 3: using other models that are out there in az REAI studio,
Speaker 3: and they have challenges because if you think about as
Speaker 3: open EI for example, it is still an API, you
Speaker 3: still have the same challenges when it comes to managing
Speaker 3: and securing access to APIs. So we built but they
Speaker 3: have specific kind of a number of challenges which are
Speaker 3: kind of a specific to l ms. So this is
Speaker 3: kind of the second part where we help customers to secure,
Speaker 3: manage and scale Open the Eye deployments for the applications
Speaker 3: with that's what we call JENNYI Gateway and HPA management.
Speaker 1: So what are some of the ways in which AI
Speaker 1: can be used with APIs. Obviously creating the management stuff
Speaker 1: around it. But would you necessarily trust an AI with
Speaker 1: your API keys and say, you know, here are the
Speaker 1: rules and times under which you would make these API calls.
Speaker 1: And I'm trying to put wrap my head around that.
Speaker 3: Yeah, I think yeahs as always, it depends, right, So Yeah,
Speaker 3: if you have like specific like access controls in place,
Speaker 3: why not Like if you, for example, that apimor you
Speaker 3: can you can provide the specific keys for the LLM
Speaker 3: to like enhance the experience for those who use those
Speaker 3: l ms you can and all that. Yeah, then why
Speaker 3: not Like you're not given the access to like full API,
Speaker 3: You're just giving access to a subset of operations.
Speaker 1: Sure, which are for example, wedn't.
Speaker 3: Leave, or they just have access to specific data that
Speaker 3: you're not really they're not, which is not really trick.
Speaker 3: So yeah, that's definitely why I will use it.
Speaker 1: I'd be definitely okay with gets, but posts and puts.
Speaker 3: Yeah, I don't know, yeahuse.
Speaker 2: Otherwise you'd have to make these rules yourself, right, I
Speaker 2: mean that that's the point here is that you got
Speaker 2: a machine learning model essentially that's figuring out what the
Speaker 2: optimal rules are for utilization.
Speaker 3: Yeah, so there is a lot. There is one thing
Speaker 3: so as I mentioned, like first thing we've built like
Speaker 3: an a PM. For example, we're helping customers with with
Speaker 3: lllms to configure a PM. So I guess you're kind
Speaker 3: of familiar with AM. We have this XML policies which
Speaker 3: can be pretty long documents.
Speaker 2: Yeah, so typically that you're doing the thing, you're trying
Speaker 2: to say no one user can do more than this
Speaker 2: many or if it's growing, you know, massively limited so
Speaker 2: you don't knock other people off and make sure they're
Speaker 2: using the right accounts like it's it's just forwarding.
Speaker 1: Yeah, a lot of stuff in there.
Speaker 2: You know where stuff is, and you know what failover
Speaker 2: modes look like, like yeah, AP. We've done a few
Speaker 2: shows on API management now and it's like pretty powerful stuff.
Speaker 2: You know, you're gonna pub put an API in a
Speaker 2: public like you're paying when everybody somebody calls that, so
Speaker 2: you kind of want to put governance around that. But
Speaker 2: write and all those rules like when you really dig
Speaker 2: into it, it's complicated. So that was my first thought
Speaker 2: when I thought about, Yeah, what do I want Generator
Speaker 2: I to do. It's like, look at what's actually going
Speaker 2: on and write me better rules.
Speaker 3: Yeah, exactly. And that's kind of the two kind of
Speaker 3: two use cases that we focused on with Copilot. Like, first,
Speaker 3: we we decided that, oh, we we know that writing
Speaker 3: policies is hard. We have like fifty sixty different policses
Speaker 3: nippets to do like Validay, Jotan retripolicies like write limits
Speaker 3: and stuff like that. So we decided like, let's have
Speaker 3: a let's have a way for customers to express and
Speaker 3: plant English, like, for example, I want to have a
Speaker 3: policy to write limit this API for you know, five
Speaker 3: for quest per second, and then Copilot will just explain
Speaker 3: on sorry not explain, but generate a policy for you
Speaker 3: and then just copy and paste this into the XM
Speaker 3: leditor and that's it. And another one is, as I
Speaker 3: mentioned that the second scenario, policies can become pretty long,
Speaker 3: like two hundred three hundred, and sometimes you don't even
Speaker 3: understand what's going on there.
Speaker 2: In XML, in XML exactly. To another theme on the
Speaker 2: show lately is we hate XML. Yeah, there's a use
Speaker 2: for AI right there. Hey, translate this XML to.
Speaker 3: Me in English exactly, And that's actually what we do
Speaker 3: with the second scenario. So yeah, you can just select
Speaker 3: XML whole thing or just a policy snippet and then
Speaker 3: you ask it to explain it to you and the
Speaker 3: fund stuff. It's only explaining just like oh, this policy does,
Speaker 3: this policy is that, but it also understands the context,
Speaker 3: like if you have too different variables, you have context,
Speaker 3: you have policy expressions some logic in there. It also
Speaker 3: will explain that, for example, and all you are doing
Speaker 3: validate job policy and you have this admin claim that
Speaker 3: you're checking. If you're checking the saddening claim, if it exists,
Speaker 3: then you are allowed to do this operation. So yeah,
Speaker 3: it's it's pretty it's pretty good in explaining policies because
Speaker 3: we also we are not using like the plane model,
Speaker 3: but we're also like using this it's called retrieval augmented
Speaker 3: generation pattern where we also have like policy snippets that
Speaker 3: are stored in a storage and this model can also
Speaker 3: use this policy snippets that we provided to better like
Speaker 3: respond with correct policies with better explanations.
Speaker 2: And so yeah, so the same way I would actually
Speaker 2: write policies is I go cut and paste from well
Speaker 2: written policies exactly. Yeah, you've trained a model on well
Speaker 2: written policy so that it has a good chance of
Speaker 2: expressing better ones.
Speaker 1: For a customer.
Speaker 2: Yeah, exactly, Okay, I mean I could see a few
Speaker 2: different things going on here at once because you and
Speaker 2: I'll include a link to this blog post here. You're
Speaker 2: also talking about using the the API APIM to manage
Speaker 2: utilization of the open Ai service because that stuff gets expensive,
Speaker 2: like exactly, Yeah, those tokens run away on you and
Speaker 2: like you're having a bad day.
Speaker 3: Yeah, yeah, that's that's that's actually an interesting use case
Speaker 3: because as I mentioned, we have customers who are trying out,
Speaker 3: they're building pocs, they're building small applications, and there's azual
Speaker 3: open Eye service and it makes it really easy for
Speaker 3: you to start. You just deploy open endpoint, you select
Speaker 3: for example, you have you want to have GPT four
Speaker 3: model and the end you're good to go, like you can.
Speaker 3: That's just an API. You get your ap I key,
Speaker 3: you import dais the care of your choice to your application,
Speaker 3: and and that's it. You're sending prompts, you're saving completions.
Speaker 3: Everything's fine, but then customers realize that okay token comes exactly, Yeah,
Speaker 3: there are tokens, and tokens is like something which is
Speaker 3: super important in edge open and in general and l
Speaker 3: MS you spend tokens for prompts, you spend tokens for completions,
Speaker 3: and even when you do play open I instance, there
Speaker 3: is a quota associate to your model which is expressed
Speaker 3: and TPM which is tokens per minute. Right, and then
Speaker 3: after all of these experimentations customers, they started to realize that, Okay,
Speaker 3: now we need to wait to manage this because okay,
Speaker 3: we've built our first POC. We have one team who
Speaker 3: developed this kind of a private preview app which is
Speaker 3: not full in production right now. But now we have
Speaker 3: ten different departments, ten different teams who also want to
Speaker 3: get access to this model, And now, how can I
Speaker 3: manage that? How can I limit the consumption per team,
Speaker 3: per department, per developer, How can I.
Speaker 2: Make sure signed costs out like my sessonment had is
Speaker 2: firmly on right now, It's like there's nothing better in
Speaker 2: this world. And being able to build out resources to
Speaker 2: the individual teams for what they do.
Speaker 3: Yeah, and that's and that's a huge issue, Like you
Speaker 3: need to figure out how many tokens were consumed by
Speaker 3: a specific team, sure what kind of model they used,
Speaker 3: And then like okay, at the beginning, you have one endpoint.
Speaker 3: But what if you want to have multiple endpoints because
Speaker 3: like you're going production, you want to scale. How do
Speaker 3: you all balance how do you like create circuit breaker
Speaker 3: rules to make sure that for example, okay, wile our
Speaker 3: first instance is throat out responses with four twenty nine,
Speaker 3: how can I fail over to a different endpoint?
Speaker 1: Right?
Speaker 3: Yeah, so, yeah, there are a lot of challenges. Now
Speaker 3: you mentioned the given access API keys. Distributing API keys
Speaker 3: to all of these teams also doesn't sound like a
Speaker 3: good idea. So that's why we've built like a lot
Speaker 3: of stuff that is in this blog post for Jenny
Speaker 3: I announcement. We wanted to solve these challenges for customers
Speaker 3: who are kind of scaling and trying to like productize
Speaker 3: their their investment into as open THEI specifically, but also
Speaker 3: for other models like elms and stuff.
Speaker 1: Yeah.
Speaker 2: Certainly, one of the experiences I've dealt with with companies
Speaker 2: building a software into the cloud, even when they you know,
Speaker 2: they've got authentication and they're building back to the customer,
Speaker 2: the customer makes a mistake with the API and racks
Speaker 2: up a couple a million transactions that were test transactions,
Speaker 2: Like they're not making money on the back end. Then
Speaker 2: you're sending them this ugly bill and they, you know,
Speaker 2: want help. In the meantime, you've also gotten an ugly bill,
Speaker 2: you know, because you ran it on the back end.
Speaker 2: So this is this whole game of like who's.
Speaker 1: Holding the bag here?
Speaker 2: You know, you don't want to punish your customer for
Speaker 2: making a mistake. If you do, you may lose them
Speaker 2: as a customer. You're not necessarily going to get remediated,
Speaker 2: you know, back to Azure too. But although I've certainly
Speaker 2: had that experience where I've done stupid stuff in Azure
Speaker 2: and called them like I'm really sorry I did this,
Speaker 2: or like yep, fine, I'll wipe it.
Speaker 1: Oh you're the guy. Yeah, we've been waiting for your call.
Speaker 1: What was that?
Speaker 2: But the business reality of this consumption model is you
Speaker 2: don't always get paid for the stuff that you used,
Speaker 2: right or and or are willing to like that's this.
Speaker 2: All of these mechanisms to me speak to let's catch
Speaker 2: why didn't you notice? Why didn't you catch it before
Speaker 2: it ran away? You know, after the first million tokens?
Speaker 2: Why didn't you stop me? And these are the tools,
Speaker 2: right like, this is how this stops from being worse exactly.
Speaker 1: Yeah.
Speaker 3: Yeah, So we were trying to make sure that customers
Speaker 3: have the right tools to have the proper governance in place.
Speaker 3: So one of the things that you mentioned like tokens,
Speaker 3: So we introduced the So we already had like rate
Speaker 3: limiting policy that works for requests like you can say,
Speaker 3: as I mentioned previously, like five requests per second for example,
Speaker 3: and now we need we had to build something for
Speaker 3: tokens which is aware of these tokens, which is kind
Speaker 3: of the main currency of open the eye, as I mentioned. So, yeah,
Speaker 3: we introduced the stoken limit policy. It works pretty similarly
Speaker 3: to rate limit policy. You can say that, okay, we
Speaker 3: have this application, we have this department, we have this team.
Speaker 3: Now we assign let's say that one thousand tokens permitted
Speaker 3: to this application to make sure that do not consume more.
Speaker 3: And yeah, and that that prot works pretty well. And
Speaker 3: if you want to be extra careful, you also want
Speaker 3: to you also can configure the policy to estimate the
Speaker 3: uh the tokens which are in the prompt So whenever
Speaker 3: there is a request coming with a prompt and you
Speaker 3: calculate the number of prompts, the number of tokens which
Speaker 3: is used in the prompt and then if we on
Speaker 3: APM side understand that it already exceeds the limit, we
Speaker 3: will not send this to the to the back end.
Speaker 2: Right, so you will consume in the first place, you're
Speaker 2: already pressing against the limit. Yes, how do you bubble
Speaker 2: up that you've hit a limit?
Speaker 1: Like?
Speaker 2: What does that look like for the customer? What does
Speaker 2: it look like for the operator?
Speaker 3: Yeah, so there is a pattern with great limited. For example,
Speaker 3: you typically it's four twenty nine returned, retry with retry
Speaker 3: after header with a specific like number of seconds.
Speaker 1: That's a message that says sorry yeah version yeah, Canadian
Speaker 1: version yeah.
Speaker 3: And that that's what we've built forty for this token
Speaker 3: limit policy as well. So whenever the limit has hit
Speaker 3: four twenty nine, retry after a specific number of seconds
Speaker 3: or minutes, depending on how you can figure it.
Speaker 1: Right, if you're being rate limited. Yeah, I use the
Speaker 1: Google YouTube API, and I'm working on a new publisher
Speaker 1: and it's going to be publishing to YouTube, just like
Speaker 1: we talked about earlier. And it's weird. I work for
Speaker 1: a couple hours on this in the morning, and I
Speaker 1: make several requests and then I get the you know,
Speaker 1: quota exceeded, and I'm going to look at my quota
Speaker 1: and it's like ten thousand API calls. I'm like, I
Speaker 1: need to make ten thousand API calls. So it's just
Speaker 1: an anecdote, but yeah, I'm looking at the response for that,
Speaker 1: you know when I try to authenticate myself and it'll
Speaker 1: say nope, quote exceeded. Sorry. Yeah.
Speaker 3: And we also trying to make sure that it is
Speaker 3: fully transparent for developers because there is a huge ecosystem
Speaker 3: of different tools for open the I and other llms
Speaker 3: like as open the I, s decay, lung chain, prompt
Speaker 3: flow like there are a lot of different tools and
Speaker 3: typically typically developers they start with the direct access to
Speaker 3: open THEI because as the case, they expect a specific
Speaker 3: like ur L and the open the eye side, they
Speaker 3: expect the apike and so on. So on our side,
Speaker 3: we wanted to make sure that this experience is the
Speaker 3: same for developer. So which means that if we put
Speaker 3: API m behind or sorry between open EI and the developer,
Speaker 3: they will never notice that something changed. So for us,
Speaker 3: it was super important to make sure that the developer
Speaker 3: experience is still the same. That's why yes, yes, so
Speaker 3: that's why we return for twenty nine because that's what
Speaker 3: open EI does. We are trying to follow the same
Speaker 3: structure to make sure that everything works as it worked before, isn't.
Speaker 1: One of the things that ap I M does is
Speaker 1: you can if you have a process for the developer
Speaker 1: that includes several API calls, maybe two different services or
Speaker 1: different you can make one sort of master API that
Speaker 1: then makes calls and proxies out on your behalf to
Speaker 1: these other ones and comes with a single result. I've
Speaker 1: used that feature of API M. There's just so much stuff,
Speaker 1: and when I got into it, there's just so much
Speaker 1: stuff in there. We could probably spend two hours just
Speaker 1: talking about all the features of API M. But you
Speaker 1: mentioned that you put out a Microsoft put out a
Speaker 1: white paper about, you know, some of these new features.
Speaker 1: I guess can we get a link to that and
Speaker 1: what what are some of the other amazing things that
Speaker 1: we might not know about that are in that now.
Speaker 3: There's a lot of innovation happening in APAM, so Jenny
Speaker 3: I gateway is definitely one of those things that I mentioned.
Speaker 3: We're also currently working on the enhancing the for example,
Speaker 3: the workspaces feature that we have an APIM to make
Speaker 3: sure that each team has its soul in workspace with
Speaker 3: isolation like control plan isolation, data play isolation, and so on. Recently,
Speaker 3: we also released a couple of new SKUs for APIM
Speaker 3: which are way faster to provision, they work better, they
Speaker 3: work in a new architecture under the hood. There is
Speaker 3: a slightly different price in model. But yeah, that's we
Speaker 3: have a lot of stuff going on there. To your
Speaker 3: point for the as you mentioned that it's really hard
Speaker 3: to understand what's going on in a PM, like a
Speaker 3: lot of policies and stuff like that. With Jenny I
Speaker 3: gateway that we were discussing, that's also one of the
Speaker 3: challenges that we wanted to address, like, Okay, we have
Speaker 3: this intelligent application developers. They use JGPT, they know how
Speaker 3: to use that, but they're not familiar with APM, and
Speaker 3: now we're asking them to write a bunch of aximal
Speaker 3: policies to limit to have the token limit, to have
Speaker 3: the authorization in place, load balancing in place, like metrics
Speaker 3: for token consumption in place, and so on. So we
Speaker 3: wanted to address it, and we also kind of we
Speaker 3: thought that it would be nice to have an easy
Speaker 3: experience of for those developers and apim to import exist
Speaker 3: natural open AAPIs. So we now have this kind of
Speaker 3: UI portal experience where you can just say, okay, I
Speaker 3: was using this open the endpoint, let's configure that one.
Speaker 3: And also I want to have token limit off I
Speaker 3: don't know, two thousand GPM, and we can configure everything
Speaker 3: for them, so they don't really need to care about
Speaker 3: the eximal policies. They don't really need to look into those.
Speaker 3: Of course, if you need to change something later on
Speaker 3: or you need like most of his scated policies, of
Speaker 3: course I need to learn something, but at least to
Speaker 3: getting started experiences is like super opimal.
Speaker 2: Well Microsoft, Yeah, I like the copilot ASPD here of Also,
Speaker 2: I know I wrote this a month ago, but I
Speaker 2: don't know what it says anymore. Like PARTSES for me,
Speaker 2: like again with my admin head on, it's like often
Speaker 2: I have a service level agreement I'm making with certain
Speaker 2: customers that's written in legal ease and I'm trying to
Speaker 2: translate it into haven't helped Me XML, But the idea
Speaker 2: that I have an intermediary tool that would then take
Speaker 2: them at legally to try and make the XML for me,
Speaker 2: and then after it's done. I could ask for it
Speaker 2: back and say, like, how close have I gotten here?
Speaker 2: I actually hit the rules that we've agreed to in
Speaker 2: the SLA. That translation that layer has always been a
Speaker 2: challenging part of it? Has this always been about the money?
Speaker 2: Like that's the main thing that's happening here is you
Speaker 2: don't want to run it, you know, I presume you'll
Speaker 2: always sell us more cloud, you know by the transaction.
Speaker 2: If you just keep requesting calls, that's fine. It's just
Speaker 2: then one day you're going to have to pay for
Speaker 2: it and it's not what you intended. So is that
Speaker 2: the important part in API management? Like, I'm not worried
Speaker 2: about tipping over the cloud, am I?
Speaker 3: Well, I guess it depends on your shower, of course.
Speaker 3: But yeah, that's one of the one of the things
Speaker 3: that you can put into APIM, Like whatever control you need,
Speaker 3: you can you can build it with the kind of
Speaker 3: a pretty powerful police engine that we have in APM.
Speaker 1: That's cool.
Speaker 2: I appreciate that, And gentlemen, I needed to take a
Speaker 2: break for one moment for these very important messages, and
Speaker 2: we're back. It's don at Rock's I'mateurd Campbell, that's Carl
Speaker 2: Franklin yoh Yo Yo talking to our friend Andre a
Speaker 2: bit about these improvements to API M which we all
Speaker 2: should be using. If we're gonna expose an API through
Speaker 2: the cloud to the world, don't leave it naked, give
Speaker 2: it some armor, and this tool helps. These gen AI
Speaker 2: tools help us to configure it correctly, operate it well,
Speaker 2: but then also deal with the additional complexities when it
Speaker 2: comes to the as you open AI, APIs with limit
Speaker 2: issuing tokens for software to utilize open ai and put
Speaker 2: limits in place for all of those good things.
Speaker 1: Have I summarized that correctly?
Speaker 2: Andre?
Speaker 1: Yeah? I think so. Yeah, I think I'm starting to
Speaker 1: understand what you doing here. Man. I'm pretty excited. Richard
Speaker 1: is the human AI.
Speaker 2: I don't know that's true. It's yeah, real, definitely created.
Speaker 2: Like you said, a very important phrase is sticking with
Speaker 2: me now, which is tokens or currency?
Speaker 1: Yeah? Absolutely.
Speaker 3: You can think about it as your main currency, your
Speaker 3: main resource you have with all of these models, and
Speaker 3: that's also what you're paying for, and.
Speaker 1: It's what what you pay that's what you pay for exactly.
Speaker 2: And so of course it's a currency because it does
Speaker 2: ultimately translate into FIA currency. Of whatever form you're using,
Speaker 2: you're going to you're going to pay for that stuff,
Speaker 2: and then you get it that pays your models and
Speaker 2: all you have all that choices when you have these
Speaker 2: controls over top of Can we talk a little about
Speaker 2: the semantic casing policies. That sounds like a way to
Speaker 2: save money and potentially improve performance. That's interesting.
Speaker 3: Yes, yeah, that's that's actually very interest simple see and
Speaker 3: every interesting implementation from all side. So yeah, as as
Speaker 3: you mentioned, so first of all, we solve the latency
Speaker 3: problem just with regular cushion that already exists in APAM
Speaker 3: for a while, you can cash request, you can cush
Speaker 3: responses for specific requests, but with with all items is
Speaker 3: a little bit different because your prompts can be different,
Speaker 3: but they're semantically similar, right, That's what we do with
Speaker 3: semantic cash, And so there is a an open opening.
Speaker 3: I provide and embedding models. Embedding model which generates vectors
Speaker 3: which represent the kind of you can think about it
Speaker 3: as a kind of semantic minion of a specific prompt
Speaker 3: war specific like stream, and then we generated for a
Speaker 3: specific prompt and then if we realize that there is
Speaker 3: a semantically similar prompt coming in, we will check the
Speaker 3: cash and we will retrieve the response from the cash
Speaker 3: instead of hitting the open the endpoints. So first of all,
Speaker 3: as I mentioned, were solving the latency problems or the
Speaker 3: response is getting to the client faster, but we all
Speaker 3: sort of saving on the token consumption because this prompt
Speaker 3: will never go to help on the A endpoint while
Speaker 3: we have the response cached. In our case, we're using
Speaker 3: reddis for vector search, so that's where story is responses.
Speaker 3: So yeah, if you're saying hi or saying hello afterwards,
Speaker 3: they're semantically similar where we just returned.
Speaker 2: I immediately go to a scenario like imagine an incident
Speaker 2: that's happened that has caused a lot of flights to
Speaker 2: be canceled.
Speaker 1: That would never happen, Richard, Come on, you a real example.
Speaker 2: Folks are trying to find out if their flights canceled,
Speaker 2: So you're going to get many requests from different sources
Speaker 2: that are essentially the same thing. Is this flight canceled?
Speaker 2: You really only need to want to fetch that once.
Speaker 2: Now it's sitting in the cash, and you very quickly respond, yes,
Speaker 2: all flights are canceled.
Speaker 1: But you know.
Speaker 2: What I like about a cashing model like that is
Speaker 2: that it will evolve over time, you know, you imagine
Speaker 2: other scenarios whence those flights are gone, there's other flights
Speaker 2: like but you're often only going to need to make
Speaker 2: that actual request back to the engine once and use
Speaker 2: it over and over again. So a good caching opportunity
Speaker 2: when you're going to have multiple people more or less
Speaker 2: making the same requests but in many different ways of phrasing.
Speaker 1: And also a way to bust the cash once the
Speaker 1: flights are back to normal.
Speaker 2: Yeah, rather than do code it yourself where you have
Speaker 2: to it's cashing is not hard. Expiring is hard, yeah,
Speaker 2: inspiring's always hard.
Speaker 1: So wait a minute, what why is it? Oh? How
Speaker 1: many times?
Speaker 2: Although maybe and again i'm reading here this is an
Speaker 2: early version. This is your first sort of go with this.
Speaker 3: Yes, yeah, yeah, yeah, well that's that's the nearly preview
Speaker 3: version for now. We're still like so there there are
Speaker 3: a lot of customer use cases for that. So as
Speaker 3: you mentioned, uh that that was a good example. Uh,
Speaker 3: but then we also have so basically like whenever whenever
Speaker 3: the company builds some sort of a chat service for
Speaker 3: answering questions, then you always have frequently asked questions.
Speaker 2: And that's where you're Hey, you're going to build a
Speaker 2: factable inevitably, but rather than you define it, let utilization
Speaker 2: define it with a cash exactly.
Speaker 1: Yeah.
Speaker 3: Yeah, and that's where you you have a lot of
Speaker 3: token saved just with the semantic cash and policy.
Speaker 1: Yeah.
Speaker 3: Also also for internal knowledge base, that's also important. Like
Speaker 3: we have a bunch of for example, support engineers sitting
Speaker 3: in this in the call center and sometimes problems are similar. Yeah,
Speaker 3: it's and you're just doing the search through the Chad
Speaker 3: jubt and yeah, your your responsors are turning from cash
Speaker 3: and you're not hitting the opening endpoint.
Speaker 1: Yeah.
Speaker 2: I was recently reading about folks that aren't securing these
Speaker 2: kinds of services properly, and people discover them and just
Speaker 2: use them as their free version of chat ept, basically
Speaker 2: leaving that that vendor holding the bag for the token costs.
Speaker 1: It's a great idea, Richard. Yeah, nice, glad. I never
Speaker 1: I can't believe in everything, but this is what I'm thinking.
Speaker 2: It's like, I'm not even talking about the you know,
Speaker 2: the proper utilizations and run away API calls and so far,
Speaker 2: but genuine nefarious use that somebody's like, oh, look, you've
Speaker 2: exposed chat to me and I can use it for anything,
Speaker 2: So I'm not even gonna worry about your product. I'm
Speaker 2: just going to exploit your token availability to run the
Speaker 2: queries I want to run, and you know you get
Speaker 2: to eat it. Congratulations.
Speaker 3: Yeah, that's that's why. First of all, it's important to
Speaker 3: have something like APIM where you have API keys which
Speaker 3: are on APM side represents specific color or application. But
Speaker 3: also there are certain tools in a measure OPENINGI itself
Speaker 3: where you can say that there's a specific filter on
Speaker 3: the content that this model is supposed to respond to,
Speaker 3: for example, if you're asking it, if you're training it.
Speaker 3: In our case, we're trained to respond about APIM policies
Speaker 3: if someone asks about the weather right now or something else,
Speaker 3: or summarizing a document which is which doesn't have anything
Speaker 3: to do with APIM, and we will just respond sorry,
Speaker 3: I cannot do that. I'm not trained to do that.
Speaker 1: My job. Go find your own chatbot.
Speaker 2: Yeah, and that documents particularly evil because that needs a
Speaker 2: lot of tokens. When you shove a document up to
Speaker 2: summarizes formul like absolutely as a token intensive and an
Speaker 2: easy mistake to make if you haven't boxed that interface properly.
Speaker 1: Talking about some of the some more the new awesome features.
Speaker 1: Is there anything that we haven't talked about yet that
Speaker 1: customers have asked for that you've implemented in this next version.
Speaker 3: Yeah, there is an interesting I wouldn't say that's specific feature,
Speaker 3: but that's kind of a challenge that we saw in
Speaker 3: a PM. So we supported Service cent Events technology for
Speaker 3: a while in APM, but we had some certain problems
Speaker 3: with that because that's essentially streaming. So when you when
Speaker 3: you send the request to judge a BT, typically what
Speaker 3: you will see and experience that you're used to most
Speaker 3: likely is that it will be it will be responding
Speaker 3: in chunk of text. It's not just it's not sending,
Speaker 3: like you, the full response, it's just responding it in
Speaker 3: streaming fashion. And it turns out the customers want to
Speaker 3: use streaming because that's what users are used to. They
Speaker 3: want to see the same experience in their chat experiences
Speaker 3: as well, like in their propilots and so on whatever
Speaker 3: applications they build. But there is a certain problem with
Speaker 3: that because whenever you introduce some sort of buffering, then
Speaker 3: the streaming experience breaks, which which is the case for
Speaker 3: you PM right now. Because whenever you have a log
Speaker 3: in policy or a monitoring policy, or you have a retripolicy,
Speaker 3: so whenever you do a buffer and a response or request,
Speaker 3: the streaming breaks. So we had certain challenges to make
Speaker 3: sure that talking limit and the talking metric policies they
Speaker 3: work with streaming scenarios as well. So that's kind of challenging,
Speaker 3: I would say, and that's kind of one of the
Speaker 3: things that customers requests to add support for.
Speaker 2: Yeah, for sure, there's more features still to come down
Speaker 2: the pipe, you know, like there's a lot we could
Speaker 2: be doing in here.
Speaker 1: Yeah, over time.
Speaker 2: It's although honestly, when we started this conversation, like I
Speaker 2: think you guys already done too many things Like sting
Speaker 2: is all as out is challenging, and I know there's
Speaker 2: still more that could be done.
Speaker 3: No, there are certainly a lot of scenarios like as
Speaker 3: as you mentioned, one of these scenarios is kind of
Speaker 3: content safety. Just to make sure that we do not
Speaker 3: respond on specific I don't know, if there is a
Speaker 3: specific question and a prompt, we should not respond to
Speaker 3: this prompt.
Speaker 2: Which doesn't sound like an API responsibility. You are at
Speaker 2: the gateway point where doing content filtering. This is a
Speaker 2: logical opportunity to hit that. Yeah, that's definitely a different area. Yeah,
Speaker 2: and actually that's something that you can do today. Like
Speaker 2: we get access to the request, you can look at
Speaker 2: the headers, you can look at the body, and then
Speaker 2: you can write whatever regular expression you want to deny
Speaker 2: the request. But that's to my point that policies are hard,
Speaker 2: especially for those who are not used to if I am.
Speaker 2: We just want to make sure that it's easy, easy
Speaker 2: to use, and easy to configure. So yeah, that's something
Speaker 2: that we're looking at. Adding like content safety concerns are real,
Speaker 2: there might be like PII data, there might be some
Speaker 2: confidential data in the request or response. You want to
Speaker 2: filter this out.
Speaker 3: And Gateway seems like a natural place to do this
Speaker 3: kind of stuff because that's the kind of single point
Speaker 3: where you see all the requests and responses.
Speaker 2: Because see asuary AI studio has a whole mechanism for
Speaker 2: content controls and so forth, you kind of want to
Speaker 2: pick the policies you've built were there and then push
Speaker 2: them in a hook to the API side.
Speaker 1: It's say, here's our saying, I only want to write
Speaker 1: one set.
Speaker 2: Of policies, but I want to be able to catch
Speaker 2: them into different places where it would matter.
Speaker 3: Yeah, there is also a big piece of kind of
Speaker 3: a governance and kind of best practices within an organization.
Speaker 3: For example, you can have multiple model deployments and they
Speaker 3: have different content safety configurations. With APIM, you're just having
Speaker 3: kind of this platform engineering side of GENNAI. Let's say
Speaker 3: where you can say that, oh, these are our rules
Speaker 3: and all of the models that are deployed they should
Speaker 3: be behind APIM. And then in APIM you can figure
Speaker 3: all of the rules that you have in your organization
Speaker 3: to comply with the basically policies whatever you have an organization.
Speaker 3: So in that case, you're basically shifting the control to
Speaker 3: APIM instead of configuring stuff on the models level.
Speaker 2: Yeah no, And you could see that associated with particular
Speaker 2: authentication accounts too. So it's like, hey, I provide a
Speaker 2: service for medical and so some pictures are going to
Speaker 2: be the kind that you wouldn't know normally want to
Speaker 2: show anywhere. But that's the business here, so it needs
Speaker 2: a different rule set.
Speaker 1: Yeah.
Speaker 2: Interesting, interesting array of problems here, Like you guys are
Speaker 2: up against it.
Speaker 1: I appreciate this.
Speaker 2: Uh, you've got an AI gateway samples on GitHub. Should
Speaker 2: I include a link to that? That looks pretty cool
Speaker 2: and super current.
Speaker 3: Yeah, yeah, that's that's an amazing repole that was built
Speaker 3: one of the by one of the gbb's that we
Speaker 3: work with. So that's basically a set of labs that
Speaker 3: you can try with with API M. So typically probably
Speaker 3: know that the typical like space for AI engineer is
Speaker 3: a Python notebook. Yeah, and that's something that we wanted
Speaker 3: to implement in those labs. So there's a bunch of
Speaker 3: there's a bunch of Python notebooks, and then there is
Speaker 3: a code. Usually there is a code that is calling
Speaker 3: open the E through a p I M with the
Speaker 3: Azure opening I is decate, so it's pretty natural for
Speaker 3: you engineers. And then we demonstrate kind of a different
Speaker 3: token limits policy emy token metric policy. Then d a
Speaker 3: lot of additional stuff like low balance in and sending
Speaker 3: the augmenting the response with the RAC pattern and so on.
Speaker 2: Yeah, so it'll seem familiar pretty quickly, dude. Yeah, you
Speaker 2: know there's different people coming in from different angles. Right,
Speaker 2: You've got your service builder on the back end, once
Speaker 2: controls and throttles and logging and that kind of thing.
Speaker 2: You've got your ll M folks who you know, want
Speaker 2: to automate the flow and control of tokens. You've got
Speaker 2: administrators trying to keep things up and make sure buildings
Speaker 2: go into right places. Anybody involved in cost control, which
Speaker 2: is lots of folks. Like my experience talking developers when
Speaker 2: they're starting to experiment in ll MS is they want
Speaker 2: the ladies and greatest of everything. But the price, you know,
Speaker 2: may the technology may or may not be needed, and
Speaker 2: the price tag is huge for the latest versions compared
Speaker 2: to Hey, would this have worked with GPT three point
Speaker 2: five lass like, because it's a tenth the price, Like,
Speaker 2: it makes a difference. Yeah, if you don't concern about
Speaker 2: any of that, you just don't. Nope, give me four zero,
Speaker 2: I want it all.
Speaker 1: Yeah.
Speaker 3: What's interesting with the alms, that's actually the opposite. Usually
Speaker 3: usually like four always cheaper there than four or than
Speaker 3: three five. Oh really Yeah, that's because they're more kind
Speaker 3: of optimized, so they say that they consume more less resources,
Speaker 3: so they're more optimized.
Speaker 2: That's why it's it's cheaper.
Speaker 1: Interesting.
Speaker 3: So yeah, but that's that's actually a good point that
Speaker 3: we we basically distinguish we have internally we think about
Speaker 3: two personas. We have a I engineer who's kind of
Speaker 3: building the application, who's using all of the s DKs
Speaker 3: they want latest and greatest, and then we have a
Speaker 3: I platform engineer who is kind of providing access to
Speaker 3: those models and he here she cares about the token
Speaker 3: consumption like cross charge and low balance and all this
Speaker 3: kind of stuff. And I engineer they also they always
Speaker 3: want something new, and that's also kind of one of
Speaker 3: the challenges for us because the space is evolve when
Speaker 3: like super fuss, like we are just trying to keep
Speaker 3: up with with different models. For example, for all I
Speaker 3: was recently announced. We're just working on adding the support
Speaker 3: for this model right now because it's multimodel. It supports images, audio,
Speaker 3: not on the text like for GBT four or in
Speaker 3: JBT three five. But then we're working on this right now,
Speaker 3: and recently they announced GPT four All Mini. So it's
Speaker 3: it's really like it's really hard to keep up with
Speaker 3: the with the industry and like a lot of open
Speaker 3: source projects building the gateways, building the capabilities to document
Speaker 3: the lllms. So yeah, it's it's a fascinating place.
Speaker 2: Yeah, job security for you, it sounds like just trying
Speaker 2: to keep up, right, But I think that's part of
Speaker 2: the strength for the customers using this is to go, oh,
Speaker 2: new model arrived, Okay, well it's in APIM so we're okay,
Speaker 2: we can add that connection that there. But certainly as
Speaker 2: they switch over to multimodel, like I suspect your inputs
Speaker 2: are different. It's not just a blob of text going
Speaker 2: and work could be almost anything.
Speaker 3: Yeah, that's actually also an interesting problem because like sometimes
Speaker 3: people think that, oh, okay, I have this GPT four
Speaker 3: and what if GPT four is not available, I will
Speaker 3: go to I don't know, some different model for example
Speaker 3: mistroll large. But in reality you have the engineers will
Speaker 3: work a lot on the prompts and make sure that
Speaker 3: these prompts work with a specific model, right, And typically
Speaker 3: if you switch to the underland model, most likely the
Speaker 3: result will be not that not what you expected, right, right? Oh,
Speaker 3: it's important to test it against multiple models.
Speaker 2: Well yeah, I said, this is such a moving space. Heck,
Speaker 2: let's face it, you can fire the same prompt at
Speaker 2: the same model several.
Speaker 1: Times and get the results.
Speaker 2: Yeah, right, Absolutely, We're not living in a land at
Speaker 2: consistency right now.
Speaker 1: Brian McKay and I used to do this show called
Speaker 1: the ai Bot Show, and it was a YouTube thing
Speaker 1: and he would have something to show and he would
Speaker 1: be practicing it the night before we recorded, and we recorded,
Speaker 1: the prompts would be completely different, probably because it learned
Speaker 1: overnight modified. Yeah, something that he could jail break today
Speaker 1: tomorrow is impossible. Crazy.
Speaker 3: Yeah, that's why prompt engineering is like it's super important.
Speaker 3: Like whatever you build, prompt engineering is always going to
Speaker 3: be very important. And the thing is that, like whenever
Speaker 3: you're tested, it's not deterministic, Like you cannot say that, Okay,
Speaker 3: it works as you mentioned, it works today, but it
Speaker 3: might not work work tomorrow.
Speaker 1: And as developers, that really messes with our head because
Speaker 1: we're used to absolute results. Yeah.
Speaker 2: I think we're also used to building on an existing
Speaker 2: data set, where so far they're pretty much tearing down
Speaker 2: his models and rebuilding them over and over and over again.
Speaker 2: So you can't expect that what worked before works again.
Speaker 2: That's just not the thing, because we don't revised models.
Speaker 2: We replace models for better or worse. I was describing
Speaker 2: paredolia on a walk this weekend that paradolia is the
Speaker 2: tendency for humans to see faces in things. Right, You
Speaker 2: look at a bowling ball and it's like, that's got
Speaker 2: a face, or the front of a car, it's got
Speaker 2: a face, right, And how that's an evolved trade of
Speaker 2: humans because if you detected the face first in the trees,
Speaker 2: you were the one running before the other people were running,
Speaker 2: so you probably lived. And the downside of courses, when
Speaker 2: you see faces that aren't there is almost is very low.
Speaker 2: It's not a big deal. Right, So we're talking about
Speaker 2: model bility. I'm like, so, imagine I'd take a shotgun
Speaker 2: and I shoot at a target and then I say,
Speaker 2: do you like this face right now? If you say no,
Speaker 2: I want a better face, I don't take the same target.
Speaker 2: I did a new target and I shoot it again.
Speaker 2: And that's you know, the nature of constantly rebuilding models
Speaker 2: is that you typically don't get the same results again.
Speaker 1: Yeah.
Speaker 2: I'm sorry that was a very long winded way of
Speaker 2: going about that. But I like saying paradolia.
Speaker 1: But I love that you, you know, introduced that word
Speaker 1: that I've already forgotten something. But for me, that's like
Speaker 1: staring up the clouds. You know, our block ink blot
Speaker 1: tests or shack tests.
Speaker 2: Yeah, yeah, humans see things that aren't there because it
Speaker 2: used to be useful at least Now I don't know,
Speaker 2: it's creating its own set of complexities. In a question, well,
Speaker 2: how many versions out are you planned? Andre, Like, I
Speaker 2: could see lots of demand from different folks for various features.
Speaker 2: We talked about the whole content management thing. But you
Speaker 2: know what comes next for you?
Speaker 3: Yeah, So we started with as open AI being kind
Speaker 3: of the one that is easy to use an Azure
Speaker 3: and have the most popular one, but we also want
Speaker 3: to extend to other models because there is definitely demand
Speaker 3: to use other models like Lama, Mistrol here, hug and
Speaker 3: Face and others. So we're looking at how to expand
Speaker 3: our genera k to akpabilities to support more models, to
Speaker 3: make sure that customers can use multiple models and the
Speaker 3: same in the same APM instance, without like the need
Speaker 3: to customize policies right crazy post expressions.
Speaker 1: And so on.
Speaker 3: Then U there is a there's a huge demand like
Speaker 3: on logging in monitoring side, as I mentioned, we we
Speaker 3: started with the talken tracking, but it turns out there
Speaker 3: are certain phases of the intelligent applications development where you
Speaker 3: actually want to collect all of the proms and completions
Speaker 3: to make sure that your model behaves correctly. And in general,
Speaker 3: logan is pretty easy with API M if you are
Speaker 3: not using streaming, if you're not using the SECE events,
Speaker 3: because again I mentioned there's a buffering problem, so that's
Speaker 3: something that we're looking at how we can solve this
Speaker 3: in the future. And also kind of in general, like
Speaker 3: focus on security traffic management like prompt manipulation policies, like
Speaker 3: let's say that this example that you share that I
Speaker 3: just found some copilot that I can use now for
Speaker 3: my personal personal Again, but what if I have some
Speaker 3: policy that says that, oh, for whatever context which is
Speaker 3: presented in the prompt, I will rewrite it so that
Speaker 3: I know that my application works with it perfectly well.
Speaker 3: So in that case, whatever you send us a problem
Speaker 3: that will be rewritten on APIM side, and you will
Speaker 3: not get the response that you wanted from this copile
Speaker 3: that you just found out in the Internet.
Speaker 1: So it just occurred to me at the end of
Speaker 1: the show here that I should ask this question long ago.
Speaker 1: But is it possible to write two policies that contradict
Speaker 1: each other? And what happens if that's possible?
Speaker 3: I believe technically you can do that. But at the
Speaker 3: same time, we have the policies are executed from top
Speaker 3: to bottom, so whatever you have at the bottom will
Speaker 3: be enforced, right, So it's.
Speaker 1: The order of execution, yep, which is pretty common. Yeah, yeah, yeah,
Speaker 1: it is. It can lead to some confusion. Now, it'd
Speaker 1: be nice to have some something when you're creating those
Speaker 1: policies to say, hey, you know, by the way, this
Speaker 1: contradicts this policy, you might want to take a look
Speaker 1: at that. Yeah.
Speaker 3: We definitely validate policies, so we have the policy ENGINET validates.
Speaker 3: If there's something like which doesn't make sense, there will
Speaker 3: be a validation error, so you will not be able
Speaker 3: to save the policy.
Speaker 1: Yeah.
Speaker 3: But yeah, if that's something.
Speaker 1: Allow Carl to access this API, don't allow Carl to
Speaker 1: access to.
Speaker 3: This ABA yeah, yeah exactly, and stuff like that.
Speaker 1: Cool.
Speaker 3: So yeah, and in general, like for the future of JENNI,
Speaker 3: so we are good at like security, traffic management, just
Speaker 3: kind of general ease of operations and APM. So that's
Speaker 3: what we are focusing on to make sure that customers
Speaker 3: have all of these secure access control to those models,
Speaker 3: like all of the policies and governments in place. But
Speaker 3: at the same time, we want to make it easier
Speaker 3: to build intelligent applications. So whatever we build, we are
Speaker 3: trying to give those AI engineers like an easy to
Speaker 3: use interface if they're not familiar, to make sure that
Speaker 3: it's easy for them to set up, configure and basically
Speaker 3: get all the benefits of a APMs GENEI gateway when
Speaker 3: they're building applications.
Speaker 1: Great, well, it sounds like the end of the show,
Speaker 1: Andre Kamanov, thank you for being of this. Is there
Speaker 1: anything that we missed that you wanted to mention or
Speaker 1: a shout out or call it action or anything.
Speaker 3: I would say, just make sure to check the AZRA
Speaker 3: updates when we release new stuff, and we'll get a
Speaker 3: technique community blocks where we publish all of the latest
Speaker 3: and greatest in in a PM. And yeah, all right,
Speaker 3: that's that's it.
Speaker 1: Awesome. Well, it's been great talking to you.
Speaker 3: Thanks for having me.
Speaker 1: It was great talking to good to Thank you very much.
Speaker 1: All right, we'll talk to you next time I'm done.
Speaker 1: Dot net Rocks is brought to you by Franklin's Net
Speaker 1: and produced by Pop Studios, a full service audio, video
Speaker 1: and post production facility located physically in New London, Connecticut.
Speaker 1: And of course in the cloud online at pwop dot com.
Speaker 1: Visit our website at d O T N E t
Speaker 1: R O c k S dot com for RSS feeds, downloads,
Speaker 1: mobile apps, comments, and access to the full archives going
Speaker 1: back to show number one, recorded in September two thousand
Speaker 1: and two. And make sure you check out our sponsors.
Speaker 1: They keep us in business. Now, go write some code.
Speaker 1: See you next time. You got J middle Vans and