Speaker 1: To like already click the button.
Speaker 2: And that's how we're starting the episode.
Speaker 1: Oh for those of you just getting started, actually you
Speaker 1: are just getting started because I just clicked on the
Speaker 1: record button. But hey, welcome, thanks for joining the podcast. Warren.
Speaker 1: How are you?
Speaker 2: I'm I'm great. You know, I feel like, you know,
Speaker 2: we should go back to this is Adventures and DevOps.
Speaker 2: You know, for the first time listeners, if this is
Speaker 2: just the one episode they happen to click on, then
Speaker 2: they're in for a real treat today because I'm really
Speaker 2: interested in today's topic.
Speaker 1: Yeah, and today's topic, we have gorkam Erkan with us
Speaker 1: and we're gonna be talking about how to how to
Speaker 1: integrate AI and mL into teams using tooling that's a
Speaker 1: little bit more aligned with how we do DevOps.
Speaker 2: That sound of out right?
Speaker 3: Yeah? That sounds right?
Speaker 1: All right?
Speaker 3: Cool?
Speaker 1: Before we dig into the details of that, can you
Speaker 1: share with our listeners a little bit about your background?
Speaker 3: Sure. I am the CEA of josu U. The josu
Speaker 3: is a company that is trying to make it easier
Speaker 3: for enterprises to adopt AI and mL in into their applications. Uh.
Speaker 3: And prior to JOSU I was with red Hat for many,
Speaker 3: many years. I was as a distinguished engineer with red
Speaker 3: Hat worked on developer tools, which in red hat it
Speaker 3: covers anything from I D E S to C C
Speaker 3: D pipelines to them op so it's it's all everything
Speaker 3: that you can think of that is open source and
Speaker 3: that will tool My group kind of got involved with
Speaker 3: those as part of that. I did Java tools early
Speaker 3: in my career for something called J two E if
Speaker 3: anyone remembers that. So and then I actually had the
Speaker 3: great idea of integrating Java tooling into vs code. The
Speaker 3: current Java tooling is something that I have done in
Speaker 3: a month or so and that has grown to what
Speaker 3: it is today. My group got involved with things like
Speaker 3: tech Tile, CD Foundation and so and so forth. So
Speaker 3: I've been doing devobs for the last probably five years,
Speaker 3: or tools for DeVos for five years, and tools in general.
Speaker 3: Before that, before joining red Hat, I was working for Nokia,
Speaker 3: the phone company, so I was in the middle of
Speaker 3: the mobile revolution and again mostly doing open source projects
Speaker 3: at Noukia at the time.
Speaker 2: When you say open source and Java and red Hat,
Speaker 2: my mind immediately goes to Jenkins and I don't have
Speaker 2: a particular soft spot in my heart for that product,
Speaker 2: but I can imagine you've had to spend a lot
Speaker 2: of time with it. Is that what they were using.
Speaker 3: So yeah, we did start with Jenkins. We do support,
Speaker 3: We did support or red Hats still supports Jenkins to
Speaker 3: some degree. But at the time when we tried to
Speaker 3: do a project, code open shift that I owe and
Speaker 3: one of the things that we tried to do at
Speaker 3: the time was to run Jenkins in a manner that
Speaker 3: it is not supposed to be run at the time.
Speaker 3: I think Jenkins is doing better now, but at the time,
Speaker 3: Jenkins was created as a server, so it was meant
Speaker 3: to run as a server. But then when you're on
Speaker 3: the cloud trying to run Jenkins, one thing that you
Speaker 3: want to do is you want to try to run
Speaker 3: its servers, like if I need to go and build something,
Speaker 3: especially if you're doing this as a SAS service, you
Speaker 3: want to just on demand start a Jenkins instance, do
Speaker 3: your bills, and then shut it down so that you're
Speaker 3: not spending too much on the cloud resources. That didn't
Speaker 3: work well for us. So one of the things that
Speaker 3: happened at the time was the Canadia project. If you're
Speaker 3: familiar with it, and as part of the Canadia project
Speaker 3: first code drop before even it even got public, it
Speaker 3: was open to some of the redheaders, including myself, and
Speaker 3: one of the things that we noticed was there was
Speaker 3: a built piece in the k native code which was
Speaker 3: actually doing server spiels essentially. So the whole idea of oh,
Speaker 3: can we actually make this and turned this into something
Speaker 3: It turned out to be the techtile project later and
Speaker 3: then it became the tech Time's cd c CD system
Speaker 3: that we know today. So it actually we got involved
Speaker 3: with that because of Jenkins because we couldn't actually make
Speaker 3: it cloud native at the time, so it felt like, oh,
Speaker 3: we should actually, you know, do something cloud native instead.
Speaker 3: So that's that's how involvement with Jane. But on the
Speaker 3: other hand, I would still bet that a big chunk
Speaker 3: of workloads on kimbernites in the world is actually Jenkins.
Speaker 1: Survey to edit.
Speaker 3: I mean, I don't know, I don't have un versa.
Speaker 3: I like that would be my guess. It's it's it's
Speaker 3: in double digits. I'm pretty sure enough.
Speaker 2: I feel like I feel like this is where ignorance
Speaker 2: is bliss. Like I don't I'm going to pretend that's
Speaker 2: not happening and just continue living my life in whatever
Speaker 2: delusion I have currently going on.
Speaker 3: Unfortunately, it is true. It's there, It's it's in many enterprises.
Speaker 3: You can't ignore Jenkins. It's hundreds of instances for sure. Yeah.
Speaker 1: I have a love hate relationship with Jenkins. I hate
Speaker 1: working with it, but I love how it can just
Speaker 1: do absolutely anything you needed to do.
Speaker 3: And the ecosystem, Yeah, like the ecosystem behind Jenkins. You know,
Speaker 3: I think nowadays get up actions maybe at that level,
Speaker 3: but the ecosystem behind Jenkins, it's, it's it's pretty big.
Speaker 3: It's very important. You can find a plug in for
Speaker 3: anything essentially, yeah, or anything that you care about, right
Speaker 3: for sure.
Speaker 1: So let's talk a little bit about aim L since
Speaker 1: that's our topic for today. And there's like, you know,
Speaker 1: I whenever I think about this, the most common scenarios
Speaker 1: that I see coming up are people using chat, GPT
Speaker 1: or tools like that, And it seems like a very
Speaker 1: early stage product. So to get that enterprise ready, like,
Speaker 1: what the what are the challenges that people are facing?
Speaker 3: Yeah, so I think check GPT, Like what we are
Speaker 3: going through right now is that that in every enterprise
Speaker 3: there is one or two let's call it experiments that
Speaker 3: is going on. And usually when you're starting the experiment,
Speaker 3: the first thing that you do is like what is
Speaker 3: the least resistance that you can face, and that is
Speaker 3: use the check GPT API s right, the open API
Speaker 3: or Microsoft has Azure APIs as well, So use those
Speaker 3: and do whatever you need to do and come up
Speaker 3: with your first early experiments. And when you look at
Speaker 3: those experiments in many of the organizations, what you see
Speaker 3: is there are a lot of success that is coming in.
Speaker 3: There's a lot of learning that they are going through,
Speaker 3: but there's also successful stories as well coming out of them. Uh.
Speaker 3: The one thing about AI is in deterministic, right, It's
Speaker 3: like the first time that you do something with AI,
Speaker 3: you're going to fail, and then the second you're going
Speaker 3: to do another iteration, and because you're going to understand
Speaker 3: that behavior, and then you're going to do another iteration.
Speaker 3: I think check GPT and all the other APIs out there,
Speaker 3: of the open API and similar AI API is out there.
Speaker 3: I think it provides them this opportunity to do the experimentation.
Speaker 3: But the way I see it is this is this
Speaker 3: is temporary, right, This is just dipping your toes into
Speaker 3: the water. When you actually want to start doing something
Speaker 3: that will change your business and try to get some
Speaker 3: financial benefits to your company. That means integrating with many
Speaker 3: main systems. That means like it's like we have seen
Speaker 3: it with cloud, we have seen it with mobile. Right,
Speaker 3: It's like it's not a oh, here's one application that
Speaker 3: is mobile, or here's one application that runs on the cloud.
Speaker 3: You're starting to adopt it to your organization or if
Speaker 3: you are even older, there was a time where there
Speaker 3: was we talked about paper this office. Right, So it's
Speaker 3: like it takes a little bit of time, it's a process,
Speaker 3: and you actually need to internalize it into your organization.
Speaker 3: I think the next step for many of these organizations
Speaker 3: is to start to internalize it. And when you're internalizing it,
Speaker 3: you cannot really depend on a third party service. And
Speaker 3: some of the industries cannot even share their data give
Speaker 3: their data to these services where there is concerns for
Speaker 3: privacy and security and so on and so forth. So
Speaker 3: they need to start running these essentially base models internally
Speaker 3: in their infrastructure or in their cloud infrastructure so that
Speaker 3: they can get the benefits. And the one thing that
Speaker 3: I see is when you go to your data scientists
Speaker 3: or data data scientists or mL engineer. They are separate
Speaker 3: from the DevOps engineers in the company. They are not
Speaker 3: well integrated. They have a very different tool set. It's
Speaker 3: just because of the way the AI has been developed
Speaker 3: and coming into the world. And one thing that you
Speaker 3: notice is there is no shortage of open source projects
Speaker 3: and tools in the AI world, but there is a
Speaker 3: shortage of projects and open source tools in the AI
Speaker 3: world that are standards. You can do. Just think of
Speaker 3: any subject and you can find ten alternates to do it.
Speaker 3: None of them talks to each each other. Nothing is standard.
Speaker 3: So this is one of the things that we have noticed.
Speaker 3: We actually were trying to solve this for ourselves as well,
Speaker 3: and one of the things we decided what was okay,
Speaker 3: So I have so many ways of storing a model
Speaker 3: and moving that to production. I have so many ways
Speaker 3: of storing a data set and moving that to inference
Speaker 3: or training and testing and so on and so forth,
Speaker 3: but none of it is standard, like this should be
Speaker 3: easier than what it is right now. So what we
Speaker 3: did is we came up with an open source project
Speaker 3: called ki toops dot mL. And what kittops dot mL
Speaker 3: provides is what we call model kits. A model kit
Speaker 3: is essentially an OCI artifact and an OCI artifact like
Speaker 3: a container image. It's not a container image itself, but
Speaker 3: it's an OCI artifact. The advantage of being an OCI
Speaker 3: artifact is it can be stored in any OCI registry
Speaker 3: like docer hub ECR as you're wherever your image registry is,
Speaker 3: and using the information that is stored in the model kit,
Speaker 3: you can actually move your models as part of your
Speaker 3: c c D pipelines much much efficient because your c
Speaker 3: c D pipelines actually knows how to usually knows how
Speaker 3: to talk to OCI registrict. For instance, when today we
Speaker 3: have a a registry that we are running ourselves in
Speaker 3: jose dot mL and you will see many signed model
Speaker 3: kits in there. The tool that we are using to
Speaker 3: sign model kits is cosign, which is what you would
Speaker 3: use for signing an imagery, a regular container image. So
Speaker 3: the idea behind having the OCI artifacts was to introduce
Speaker 3: a standard way of storing and standard way of sharing
Speaker 3: your AI artifacts between data scientists and mL engineers and
Speaker 3: DevOps engineers.
Speaker 2: You said something really interesting in there, and that's I
Speaker 2: haven't heard of put it this way before that your
Speaker 2: first foray into using any sort of AI or mL
Speaker 2: models in your company will be an experiment that ends
Speaker 2: in failure. And I, you know, I think there's something
Speaker 2: really to that, because I can imagine if you like
Speaker 2: walked around and put like a hammer on every engineer's
Speaker 2: desk and said, you will now start to develop with
Speaker 2: this hammer instead of what you were using before. I
Speaker 2: feel like it's a ridiculous analogy, but I think there's
Speaker 2: a lot of sense there that like that for sure
Speaker 2: will not end in success there, Like, however you thought
Speaker 2: you could utilize it internally is probably not going to
Speaker 2: be super effective. And that obviously extends into the area
Speaker 2: of if you are building AI into your product, not
Speaker 2: just using it as a as a tool for development,
Speaker 2: and you're really talking about this next level which isn't
Speaker 2: just using it but actually creating it to be effective
Speaker 2: as part of the product offering.
Speaker 3: Yeah, And the reason for that is it's really in
Speaker 3: the terministic right with the AI itself. So you may
Speaker 3: actually think that they're like when we are integrating applications together,
Speaker 3: the rule based the applications, the outcome is very deterministic.
Speaker 3: We know that these are the inputs and these will
Speaker 3: be the outputs. In the case of AI, that's not
Speaker 3: the case, right, So when you go through your first
Speaker 3: initial cycle and go through your you know, you receive
Speaker 3: your inputs, and then you start to get your outputs,
Speaker 3: you start to see these outputs that you haven't expected before.
Speaker 3: And the usually, like, for instance, it's very likely that
Speaker 3: you're missing actually some data that you're feeding into the model,
Speaker 3: and you're not getting the outputs that you're expecting. Because
Speaker 3: let's say that you're giving the customers can data, but
Speaker 3: then you when you give the eleventh data, the AI
Speaker 3: is now able to make the code nection or or
Speaker 3: or make the analysis better and give it a better result.
Speaker 3: So you will start to notice things like that, and
Speaker 3: it is not very easy to notice this at the
Speaker 3: design phase. It's not even possible to notice these things
Speaker 3: at the design phase. So that's why I'm saying that
Speaker 3: your first experiment is going to fail, and you're gonna
Speaker 3: do another one with more data or less data or
Speaker 3: changed data, and then you will get to a state
Speaker 3: where you're you're you're happy with the outputs.
Speaker 2: I feel I have a fear here now that, like
Speaker 2: you know, if I'm incorporating it into one of our products.
Speaker 2: You know, we have a couple and we have tried
Speaker 2: going down the route of adding some AI into into
Speaker 2: one of them. Uh, there is this to only say
Speaker 2: non deterministic. We mean like literally the same inputs gives
Speaker 2: us different output. And exactly, I wonder if we're training
Speaker 2: our customers to believe that they should just you know,
Speaker 2: turn it off and turn it on again in a sense, right,
Speaker 2: Like you know, you have to keep repeating the same
Speaker 2: question over and over again to get the answer you want.
Speaker 2: And that just that feels like a dangerous thing to
Speaker 2: push that expectation onto the users of our products.
Speaker 3: Yeah, there is a Yeah, I see what you're you're saying,
Speaker 3: and and and I think that's that's that's the part
Speaker 3: of the AI that that still needs to be enhanced
Speaker 3: a little bit. You know, the the whole subject of
Speaker 3: guardrails on AI is a little bit it feels early. Oh,
Speaker 3: I'm told, I'm totally I'm totally with you.
Speaker 2: No, there was actually a study that was released just recently,
Speaker 2: and I think we'll kind of appreciate this that it's
Speaker 2: not being like prompt engineering isn't being used for malicious purposes.
Speaker 2: It's generally being used by power users to actually get
Speaker 2: value out of the system more than anything, which I
Speaker 2: think is really surprising that these systems aren't being abused
Speaker 2: in that way realistically yet, And so adding in guardrails
Speaker 2: of anything is just preventing those early adopters from being
Speaker 2: able to utilize your product more effectively, rather than preventing
Speaker 2: malicious attackers.
Speaker 3: Right. And we actually like since you said prompt Engini,
Speaker 3: we actually talked with an enterprise recently and one of
Speaker 3: their needs was to share their prompts, Like, uh, think
Speaker 3: of it this way. You have a database of all
Speaker 3: your all your enterprises data, all your customer data, and
Speaker 3: so on and so forth. In the past, the sequel
Speaker 3: statements that you have used for finance was very different
Speaker 3: from the psychoal statesments that you have used for sales,
Speaker 3: so you'd never really actually care to share the psycholos statements.
Speaker 3: But now what they're seeing is, as you said, there
Speaker 3: are these users who are using these prompts and getting
Speaker 3: really good results out of those prompts that are querying
Speaker 3: there or that are connected to their their data. Right,
Speaker 3: So what they want to be able to do is
Speaker 3: to to be able to share these prompts like between
Speaker 3: finance and and and HR and and and sales and
Speaker 3: so on and so forth, so that they don't have
Speaker 3: to reinvent the veil every time. But I think that
Speaker 3: we're starting to see a different picture now in the
Speaker 3: in the AI world where with with these kind of things.
Speaker 3: But when you are opening your data through and an
Speaker 3: AI model directed to your customers, I think there's still
Speaker 3: work that needs to be done on the guard for this. Yes, yeah,
Speaker 3: for sure.
Speaker 1: It reminds me a lot of there's two things here
Speaker 1: that that seem like there's lessons we can learn here
Speaker 1: from other engineering or other industries. Like whenever I work
Speaker 1: with early stage startups, I always say that like the
Speaker 1: goal of your first product as a startup is to launch,
Speaker 1: like the let me rephrase that. To become a successful startup,
Speaker 1: you have to identify the difference between the product that
Speaker 1: you built and the product that your customers thought you
Speaker 1: built and close that gap. And it feels like this
Speaker 1: is very similar to that. You know, because AI will
Speaker 1: answer whatever question you ask it, and then you have
Speaker 1: to figure out what assumptions it made that you didn't
Speaker 1: want it to make. And that's where I think really
Speaker 1: sharing those prompt those prompt models can be helpful. For
Speaker 1: a while there on X there was a trend where
Speaker 1: peopleeople were you know, sharing what they were doing with
Speaker 1: different AI models and then also sharing the prompts that
Speaker 1: they used too, And I found that really interesting to
Speaker 1: see the different caveats that they would give in the
Speaker 1: prompt to keep it from wandering off some other path
Speaker 1: that gave you an answer that looked right, but as
Speaker 1: wasn't actually the answer that you needed.
Speaker 2: No, you say that and it's like, actually, really ridiculous.
Speaker 2: Like I'm going through right now. I'm making a presentation
Speaker 2: for the European as reinvent that I'm speaking at, and
Speaker 2: I want some pictures, and I will ask I'm using
Speaker 2: one of the tools, and I will ask the same
Speaker 2: question multiple times. But when I get a picture that
Speaker 2: I actually like, I will take the prompt that I
Speaker 2: used for it, and I will save it as the
Speaker 2: file name of the picture so that I can get like,
Speaker 2: I mean, it's a ridiculous, but that's the name for me,
Speaker 2: that really is the thing that identifies it, so that
Speaker 2: I can, you know, potentially go back later and be like,
Speaker 2: what did I use to actually generate that image?
Speaker 3: So let me complicate that problem a little bit. So
Speaker 3: you have you're you're using a check GPT for instance, like.
Speaker 2: It's not the one from open Ai, but yeah, for sure,
Speaker 2: it's an l you're using.
Speaker 3: You're using and it's actually a check giv A service.
Speaker 3: So you don't really know the version number. Really, you
Speaker 3: don't really have to know the data sets that has
Speaker 3: been used and retrained and so and so forth. So
Speaker 3: for some of the organizations that's not going to work.
Speaker 3: So some organizations are going to say, hey, you know what,
Speaker 3: I need to know exactly the model, the data set
Speaker 3: that it was trained on, and I need to know
Speaker 3: where my fine tuning is coming from, where my RAGS
Speaker 3: is coming from. So I need all the lineage of
Speaker 3: of of this when this prompt was issued and right.
Speaker 3: So you need systems that are capable of telling precisely
Speaker 3: what was the data set, what is the model, what
Speaker 3: is the fine tuning data set? Right? And what is
Speaker 3: if you have been using RAG, what is what is
Speaker 3: the version of the vector database or the lineage of
Speaker 3: the vectors. All of this stored and available somewhere because
Speaker 3: something may happen at any point saying that, oh, you
Speaker 3: need to go back and figure out what went wrong
Speaker 3: with this? Answer what went wrong with this today? Like
Speaker 3: if you're using an API, you can't really say what
Speaker 3: what went wrong? And so if you think about it
Speaker 3: this way, we actually have a system that was used
Speaker 3: for this purpose. We used s BOMs to be able
Speaker 3: to say exactly what we have used in production for
Speaker 3: our applications. Right, so can we actually utilize the same
Speaker 3: system with AI as well? Like, oh, I'm using the
Speaker 3: application x y Z, it's built out of these dependencies,
Speaker 3: these libraries and so on and so forth. An AI
Speaker 3: application is not two different. It's basically saying that, oh,
Speaker 3: I'm using the same application, the same dependencies. I'm using
Speaker 3: the inference engine x y Z, and then I'm using
Speaker 3: the model weights that are coming from here. I'm using
Speaker 3: a data set that was used in that is coming
Speaker 3: from here. And these are all in an s bomb
Speaker 3: which existing systems know how to pars and get results from.
Speaker 3: And we know how to sign s bomps, we know
Speaker 3: how to store s bomps, you know CI artifacts. So
Speaker 3: I think that's kind of you know, is the missing
Speaker 3: link nowadays where we are doing these things in our enterprises,
Speaker 3: adopting models based models from somewhere, and none of the
Speaker 3: data sets that we are using is enough. So some
Speaker 3: one of our engineers just goes to Google and searches
Speaker 3: for a data set and uses that to you know,
Speaker 3: do some of the training. You don't know what is
Speaker 3: in that data set. So it's it's a little bit
Speaker 3: wild west at the moment with the AI adoption. So
Speaker 3: and when it was wild west for the regular applications,
Speaker 3: you know, we we ended up with things like, uh,
Speaker 3: the we ended up with things like what was the
Speaker 3: look for j right? So how many how many days
Speaker 3: any Joba they will prespend trying to figure out if
Speaker 3: they have used the wrong version of look for Jack
Speaker 3: when that came out right, So, uh so things happened.
Speaker 3: Uh and I think we need to be prepared for it.
Speaker 3: And it's a little bit wild wild West at the
Speaker 3: moment with the with the I amount.
Speaker 1: It's you know, as we talk about this more, it
Speaker 1: kind of feels like there is it's almost like a
Speaker 1: communication problem. One of the things you said was knowing
Speaker 1: the lineage of the data and with when it comes
Speaker 1: to these AI models, we have a very human feeling
Speaker 1: interface to it, and so we I think we naturally
Speaker 1: try to leverage that. But the difference being, if if
Speaker 1: I ask either of you who was the last president,
Speaker 1: you probably are going to assume that I'm talking about
Speaker 1: current time. But if I'm talking to a data set
Speaker 1: and that data set is twenty years old and I
Speaker 1: ask it who the last president is, I'm going to
Speaker 1: get the wrong answer. And I feel like it's those
Speaker 1: hidden assumptions that we have when we communicate that make
Speaker 1: us have to be overly verbose with these tools.
Speaker 2: See, I don't think I don't think AI is special here.
Speaker 2: I think that our own culture and values propagate much
Speaker 2: too far into it, and as well as our neural diversity.
Speaker 2: Like I'm on some spectrum somewhere and you say, who
Speaker 2: is the last president? I you know, what I heard
Speaker 2: was like the last president there will ever be like as.
Speaker 4: If you know, you know, you're talking about side of
Speaker 4: history and like, you know, if I look at it
Speaker 4: and today's you know, I look at today's like political climate,
Speaker 4: you know that is a you know I am.
Speaker 2: I'm evaluating that based off you know, the model that
Speaker 2: I've trained in myself, and I can also imagine like
Speaker 2: for other cultures out there, depending on who you're talking to,
Speaker 2: you know, I feel like in the West or in
Speaker 2: the United States, it may be much more clear. But
Speaker 2: you know, depending on who you're talking to and where
Speaker 2: they've come from, Uh, they're gonna have much different answers
Speaker 2: on that. And I feel like we may be more
Speaker 2: closely able to understand what some of those implicit inputs
Speaker 2: into that human model there are based off of what
Speaker 2: we know about them or the current world. And I
Speaker 2: think we definitely do throw a lot of that out
Speaker 2: the window when we are utilizing a prompt sitting on
Speaker 2: a website somewhere.
Speaker 3: Yeah, for sure.
Speaker 1: So what are the what are the big challenges that
Speaker 1: that we're facing with this from a DevOps perspective, like
Speaker 1: how do we how do we set the framework into
Speaker 1: guardrails to get that tool from the AI developers into
Speaker 1: a production grade capacity.
Speaker 3: So I guess the first one is, you know, have
Speaker 3: you packaged this, have you set this sober in a manner?
Speaker 3: And you know, we have one solution with kit tops
Speaker 3: for it. The second challenge that we have noticed and
Speaker 3: started working on is AI is more nondeterministic and it's
Speaker 3: more complex. So when you think about a workflow that
Speaker 3: you use on DevOps, it's more linear, and there is
Speaker 3: a starting point and there's an endpoint, and there's an
Speaker 3: art fact that you can record and send to production
Speaker 3: and so on and so forth. With AI, it works
Speaker 3: a lot a little bit different because you sent the production,
Speaker 3: but you after that, you keep on watching and because drifts,
Speaker 3: the data changes the model, you know, twenty years passes,
Speaker 3: the president changes and so on and so forth. So
Speaker 3: you start to notice these drifts and you need to
Speaker 3: start to constantly observe the models and then react to
Speaker 3: their changes, and not just the models itself as well.
Speaker 3: Before you know, the training needs to be like even
Speaker 3: if you're doing fine tuning, whether your fine tuned model
Speaker 3: is adequate or not, that's actually a bunch of tests
Speaker 3: that needs to be run and then compared and so
Speaker 3: on and so forth. So the one thing that we
Speaker 3: have noticed is this is almost like there's a lot
Speaker 3: of signals that are coming in from these develops systems
Speaker 3: that we're going to.
Speaker 5: Use for training, inference, production and observability of the AI,
Speaker 5: but there isn't a real solution out there that reacts
Speaker 5: to it, that brings everything together so that you can.
Speaker 3: Use these things together in a more orderly fashion or easier,
Speaker 3: integrate them, easier to be to cite it in a
Speaker 3: different way. So what we I don't know if this
Speaker 3: is the right language to use, but what we said was, oh,
Speaker 3: let's build a control plane for this so that it
Speaker 3: receives all these signals for all your AI m L applications,
Speaker 3: including everything that is coming in from observability, from your
Speaker 3: c I, c D pipelines, your training pipelines, your dataset changes,
Speaker 3: your data extractions and so forth. And in the control
Speaker 3: plane we define what needs to be and then the
Speaker 3: control plane reacts to these signals and runs the data
Speaker 3: plane components like their CICD and some so forth, or
Speaker 3: your deployments in such a way that the system tries
Speaker 3: to get back to the state that you want it
Speaker 3: to be. So we started working on a control plane
Speaker 3: for AI and mL applications so that some of the
Speaker 3: some of the challenges of running and integrating AI and
Speaker 3: mL into an enterprise is boxed into a control plane. Essentially.
Speaker 2: Maybe this is a controversial statement to make. I get
Speaker 2: the sense that our current capabilities for generating new models
Speaker 2: restricts those models in a way which makes them not
Speaker 2: very good at reasoning or analysis and If they're not
Speaker 2: good in that, then what we're left with is pretty
Speaker 2: much data replay construction content creation. But we know that
Speaker 2: the models that we're creating, they're not good at that
Speaker 2: because they suffer from hallucinations from making up stuff by
Speaker 2: combining different pieces. I think I read somewhere that the
Speaker 2: summarization strategy that a lot of models have taken in
Speaker 2: their creation is to look at the headlines of articles
Speaker 2: that have been written and then consume the content of
Speaker 2: the article and index that and then do a reverse
Speaker 2: index look up. So you know, hey, here's my article,
Speaker 2: what should what summarizes for me? It will spit back
Speaker 2: out the title. And if we look at the content
Speaker 2: that humans have created so far, titles are all click baity,
Speaker 2: you know, have nothing to do with the conclusion. And
Speaker 2: so even if they were right, you know, they're it's
Speaker 2: still not very accurate. And so, like I am, I'm
Speaker 2: wondering if we are not still a bit too early
Speaker 2: in trying to automate or improve some of these things
Speaker 2: when the base thing we're getting out still is fundamentally
Speaker 2: limited in capability.
Speaker 3: Uh, I don't think so, Like I think I need
Speaker 3: to I need to be very careful about this because
Speaker 3: I don't want to add to the hype. Yeah, because
Speaker 3: you know, I think it's the AI is over hyped
Speaker 3: at the moment.
Speaker 2: No, No, definitely not.
Speaker 3: What well, yep, I know it's a shock to me too,
Speaker 3: So it is a little bit hyped, let's call it.
Speaker 3: But on the other hand, I don't think with the
Speaker 3: capabilities and we we when we talk about the hype
Speaker 3: of the A I, we we are usually talking about
Speaker 3: the gen a I, right, the l l MS and
Speaker 3: generative AI. But but in reality there is also other
Speaker 3: kinds of AI and m l as well, so that
Speaker 3: we don't usually talk about. But it's actually getting a
Speaker 3: lot of benefits from this hype essentially, So when I
Speaker 3: think about it, that whole space as A as A
Speaker 3: as one one piece of of of technology area, I
Speaker 3: think that this is this is not a this is
Speaker 3: not a like we can start adapting A I and
Speaker 3: m l projects into into our enterprises. Even with generative AI,
Speaker 3: which was what you were referring to, Essentially, you can
Speaker 3: get really good results out of it. Like that was
Speaker 3: one article, I think it was not even an article.
Speaker 3: It was Walmart reporting their quarterly results. They basically said
Speaker 3: you know, we are more efficient with our online business
Speaker 3: because of generative AI. I think they were doing generating
Speaker 3: product the descriptions more efficiently or something, so they actually
Speaker 3: started doing better financially. So there are pockets of even
Speaker 3: direct financial benefit that we are starting to see in
Speaker 3: from generative AI and from AI mL and other kinds
Speaker 3: of AI. There has always been in the five year
Speaker 3: last five years, there has always been projects that were
Speaker 3: bringing benefits to the business. What is changing with the
Speaker 3: hype is now there is more eyeballs look into this
Speaker 3: area so that there is an accelerated opportunity for adoption.
Speaker 3: You know, in the past you had a chance to
Speaker 3: get one base model for something for categorizations, let's say,
Speaker 3: but now you have ten, and you can actually test
Speaker 3: the ten categorization based models and then get a better
Speaker 3: fit for your data. So there are things like that
Speaker 3: that is actually happening that's going to I think help
Speaker 3: the the enterprises and the businesses get more value out
Speaker 3: of AI n mL. So I guess generative AI it's
Speaker 3: a little bit too much hyped up, but that is
Speaker 3: not a bad thing for the rest of DAI world.
Speaker 2: I mean, you say that and now you help me
Speaker 2: thinking and maybe a more philosophical perspective if if we're
Speaker 2: just lacking a number of people paying more attention and
Speaker 2: investing their resources in actually researching or building up the
Speaker 2: technology more.
Speaker 3: Is it is?
Speaker 2: Are the limitations that we're facing something fundamentally within the
Speaker 2: capability of human society to overcome, like or is reaching
Speaker 2: general artificial intelligence actually completely made up dream? Yeah, And
Speaker 2: for the record, this is not a I expect that
Speaker 2: you want to have an accurate answer to.
Speaker 3: This, but I don't have any accurate answer. Of course,
Speaker 3: I don't think anyone has, or they may have an
Speaker 3: interpretation for or expectation for it. But to be honest,
Speaker 3: the amount of compute that we have been using to
Speaker 3: trade models, even even the small ones, is enormous For
Speaker 3: us to get to a place where we have more
Speaker 3: capable models, I think it will take some kind of
Speaker 3: a leap before something needs to happen. This amount of
Speaker 3: compute that we need for that is just just enormous
Speaker 3: at the moment. And the other thing that I'm actually
Speaker 3: worried about is without actually getting adoption and value, how
Speaker 3: much can we continue to spend on training more and
Speaker 3: more capable AI models At some.
Speaker 2: Point I don't know if it's actually a monetary problem.
Speaker 2: I understand that we're actually resource constrained, and I'm wondering
Speaker 2: if those chips are coming from Taiwan, of which has
Speaker 2: its own sort of issues, and we don't have another
Speaker 2: opportunity to actually produce what is necessary in order to
Speaker 2: accelerate further. It's like a fundamental limitation of we're limited
Speaker 2: not by our ability or by the technology's ability, but
Speaker 2: realistically our actual capability of a resource allocation.
Speaker 3: Produce fast enough. Yeah. And also not to forget the data,
Speaker 3: because the way that the AIS be trained today, it's
Speaker 3: troven large amounts of data to it, and we true
Speaker 3: pretty much everything that we can so far.
Speaker 2: It's worth I mean, that's its own sort of problem
Speaker 2: because the Internet is sort of over as far as
Speaker 2: what it used to be a free, free, public sharing
Speaker 2: of knowledge. That's no longer the thing, and so we're
Speaker 2: only going to be accessible to a smaller pool of
Speaker 2: information going forward for future models that that data now
Speaker 2: has finally have a price on it, and realistically, as
Speaker 2: you point out, it's not that much, and it's not
Speaker 2: even that good. If this is the best we can
Speaker 2: come out out with it at this point. So it
Speaker 2: does seem like we're at some sort of obstacle to
Speaker 2: get further, both from a data creation standpoint and a
Speaker 2: resource creation standpoint.
Speaker 3: Right, But to be honest, like the general AI is
Speaker 3: a like it's a very good ambition to have, but
Speaker 3: most enterprises and businesses do not need it. They that
Speaker 3: what we have today with their data is plenty enough
Speaker 3: for them to actually get financial benefits and efficiencies and
Speaker 3: value out of AI. And about like, uh there, I
Speaker 3: think there's there are two facets to this. There is like, oh,
Speaker 3: there's these ambitions which we should actually try to pursue.
Speaker 3: But then there is oh, can we actually start adopting
Speaker 3: the technology that we have generated over these years so
Speaker 3: that we start creating value to our businesses and communities? Right,
Speaker 3: So I don't think the two are connected as connected
Speaker 3: as we think. Where we don't need to wait the
Speaker 3: first to do the second?
Speaker 2: Yeah, yeah, for sure. Where is where is AI adoption
Speaker 2: on the WILL technology adoption scale?
Speaker 1: Not really a not really a priority for me just
Speaker 1: but I think I think just because of the industry
Speaker 1: that I'm in, for me, the most value that it
Speaker 1: has is like a personal value of using it to
Speaker 1: remember the things that I've forgotten, you know, like how
Speaker 1: do I you know, how do I do an a
Speaker 1: single weight in javascriptor or things like that. From a
Speaker 1: business perspective, I don't have person have any use cases
Speaker 1: that I'm actively involved with, though I think it has
Speaker 1: a lot of power and potential. It's not that I've
Speaker 1: ruled it out. It's just that it's not a priority
Speaker 1: for me right at the moment.
Speaker 2: No, I totally get it right, like it always optimizes
Speaker 2: for like the norm, right, and so as long as
Speaker 2: the norm is better than where you are, it's a
Speaker 2: good thing to use it because you'll get to that norm.
Speaker 2: But will you're such a great engineer that what it's
Speaker 2: going to suggest to you is going to be worse
Speaker 2: than what you would create by default.
Speaker 1: Which comes back into I think that comes back into
Speaker 1: the prompt engineering topic that we were having, you know,
Speaker 1: to get useful information out of it, I have to
Speaker 1: give it all of the internal mental models that I
Speaker 1: would use to get a similar or better answer.
Speaker 2: And that's really where all the effort goes into, Like
Speaker 2: how do I even correctly? It's like I have an
Speaker 2: intuition about how to think about this, Now I need
Speaker 2: to convey that intuition to another being using natural language,
Speaker 2: of which I've never thought about how I'm even thinking
Speaker 2: about this problem before. I totally agree that's where all
Speaker 2: of the challenge really comes in to get valuable output
Speaker 2: from these models.
Speaker 3: So it's one of the things that I have the test,
Speaker 3: and I'm sure that there is a scientific explanation to
Speaker 3: it which I haven't really looked up up. But when
Speaker 3: you actually give a working example of what you're trying
Speaker 3: to achieve to the model, you usually get much much
Speaker 3: better results from the model. Like if you teach it,
Speaker 3: you know this is how it should work. Once, then
Speaker 3: the rest it goes much easier for you.
Speaker 2: I mean, and we know that must be true because
Speaker 2: like in the lms are designed by humans, and humans
Speaker 2: operate in this like sixth level of thinking hierarchy, where
Speaker 2: like the bottom level is like wrote, repetition and that
Speaker 2: means you saw the answer in some format before and
Speaker 2: you can repeat it again. And I feel like that's
Speaker 2: the first level if you want an answer that looks
Speaker 2: like something you like, even another human, Right, I need
Speaker 2: to show you what success looks like before I expect
Speaker 2: you to output it, you know, especially in experienced engineers.
Speaker 2: But then there's like five more levels of this where
Speaker 2: I know for sure that it's not going to get
Speaker 2: to without also deep conscious thinking and deliberate input on
Speaker 2: those levels and order like don't just do one I need,
Speaker 2: because I'm going to ask it to create unit tests
Speaker 2: for me. I love this example because I actually think
Speaker 2: this is something that really works well, especially going from
Speaker 2: one language to another.
Speaker 3: One.
Speaker 2: The steps are we were just doing this in an
Speaker 2: authoress as we were converting some of our code from
Speaker 2: JavaScript to RUSS. We say, okay, you know we have
Speaker 2: this code in JavaScript, These tests in JavaScript, right tests
Speaker 2: that look like this in RUSS wright the actual code,
Speaker 2: and then run the RUSS unit tests against the code,
Speaker 2: so we have the sort of validation and the first
Speaker 2: level is really the unit test. But we then need
Speaker 2: to say things like oh, but also make sure to
Speaker 2: handle these weird edge cases and like specifically list out
Speaker 2: some edge cases so someone has to think about those.
Speaker 2: And then there's like, okay, now I need to generate
Speaker 2: like ways in which this could fail, which is like
Speaker 2: another level on top of that, and you know, propose
Speaker 2: different ways in which we could re architect this whole
Speaker 2: aspect and I can't just say that, right, I can't
Speaker 2: just say, oh, yeah, you know, refactor this code for
Speaker 2: me in an optimal way. I need to actually like
Speaker 2: how do I know what to refactor? How do I
Speaker 2: know what to do that? It's just such a challenge
Speaker 2: to articulate, even from one person to another one And
Speaker 2: I almost want to, like draw a diagram how do
Speaker 2: I feed that in? Though like there had a different
Speaker 2: len right exactly. You know, it does create this whole
Speaker 2: other challenge in order to really get the value out
Speaker 2: to a high enough level that where we really wanted
Speaker 2: to have.
Speaker 3: Yeah, I mean it's something that we have recently done
Speaker 3: with a you know, it's languaging language out right. Lms
Speaker 3: are are really good at those. It's like we have
Speaker 3: done a similar thing. We actually had to implement a
Speaker 3: piece of code that was implemented in Rust, but we
Speaker 3: had to run it in a different environment which didn't
Speaker 3: allow us to run Rust at the time. So we said, okay,
Speaker 3: here's some Rust code that implements this algorithm and this thing,
Speaker 3: and then just can you convert that to a typescript
Speaker 3: in that at that time? It did like it did,
Speaker 3: and it runs, it runs as good as it like,
Speaker 3: it converts all the unit tests and as you said,
Speaker 3: it's like and it works perfectly, except that you don't
Speaker 3: get the the performance, Like there is no way that
Speaker 3: you can get the performance, and all of a sudden
Speaker 3: you're faced with the fact that, oh, okay, this code
Speaker 3: is maybe equivalant, but not equal. So there's also that
Speaker 3: fact as well. But the good thing about it was
Speaker 3: we actually spend on the what forty five minutes to
Speaker 3: just get that code converted and learn that, oh, this
Speaker 3: will never perform the same, so that we had to
Speaker 3: go a different way. So I don't know how much
Speaker 3: time we saved by just doing that, but that was
Speaker 3: worth it. So it's not always about actually getting the
Speaker 3: result that you want. It's also sometimes about getting the
Speaker 3: wrongers out quicker as well.
Speaker 1: Oh true story.
Speaker 2: Yeah, yeah, go ahead.
Speaker 1: Well I was just gonna reinforce that because a lot
Speaker 1: of times that's the best learning that you can get.
Speaker 1: You know, if you get the right answer the first time,
Speaker 1: you don't learn as much as you do when you
Speaker 1: think you were going to get the right answer and
Speaker 1: get something completely opposite, you know, Like it's like when
Speaker 1: tackling a new project or a new problem, one of
Speaker 1: the most important things to learn is what questions should
Speaker 1: I be asking that I don't know I should be asking,
Speaker 1: and getting the wrong answer helps you along that path.
Speaker 2: I think there's a huge challenge there when you almost
Speaker 2: know from intuition that the response is wrong. I mean,
Speaker 2: it's wrong, obviously it came from it all alum, but
Speaker 2: I mean it's like there's something special about it that
Speaker 2: you know, you just know. You're like, oh, that that
Speaker 2: can't really be the best way to do this, and
Speaker 2: you're like, do it differently, do it differently, And like
Speaker 2: I'm running out of words to basically say, try again,
Speaker 2: but don't use any of those constructs you use this time.
Speaker 2: And it's just every time it's like just subtly wrong
Speaker 2: in a different way. And so there is certainly a
Speaker 2: learning there of, you know, all these ways in which
Speaker 2: not to do it. I would just, you know, one day,
Speaker 2: I would like it to actually give me a right answer.
Speaker 3: Yeah, it's it's the are sneaky. So they're gonna give
Speaker 3: you an answer that is ninety nine percent correct, but
Speaker 3: then that one percent they are going to hide a
Speaker 3: bug that you're gonna spend the next two days trying
Speaker 3: to find in production.
Speaker 2: Yeah, I mean you're lucky if it happens. You're definitely
Speaker 2: lucky if it's short term turnaround like that right where
Speaker 2: you know it happens quickly and not like you know,
Speaker 2: months or years down the road when you may not
Speaker 2: even be part of the team anymore. And now you've
Speaker 2: got that, which is where a lot of the you know,
Speaker 2: bugs come up there. I'm interested, though, if we flash
Speaker 2: back to something you were talking about before about where
Speaker 2: the some of the real value is. We were talking
Speaker 2: about Walmart and the content automatic creation descriptions for the
Speaker 2: products that they're having. I wonder how many of those
Speaker 2: are out there, because I mean, I think a lot
Speaker 2: of the hype right now is focused on this. I'm
Speaker 2: gonna say lie that it increases the productivity of software engineers.
Speaker 2: A lot of companies are repeating this. I'm waiting for
Speaker 2: some actual real data on it. But I'm really curious
Speaker 2: about the ways in which the enterprises, the larger companies,
Speaker 2: you know, are finding opportunities that aren't just like a
Speaker 2: chatbot on a website.
Speaker 3: Right, I mean, that's I think that's table states at
Speaker 3: this point. The software engineering and get psyche. Yeah, there
Speaker 3: are there benefits to it. There are no benefits to it.
Speaker 3: It's a controversial, but then there are other users like
Speaker 3: furnish this with. One of the usages that I have
Speaker 3: seen is uh they the company, Uh they're running support
Speaker 3: service phone support service, and what they were doing was
Speaker 3: uh doing sentiment analysis of the calls so that they're
Speaker 3: uh they can do better callbacks to customers who are
Speaker 3: whose sentiment was was not as as good as as
Speaker 3: they they wanted to be. So I think that's one
Speaker 3: way of doing uh, you know, customer satisfaction and and
Speaker 3: and keeping your brand uh recognition, uh and so on
Speaker 3: and so forth. So it's like and one of the
Speaker 3: things that they actually had was they first tried it
Speaker 3: the sentiment analysis, but they had to actually feed additional
Speaker 3: data to the model to get the sentiment analysis correct.
Speaker 3: It's one of those examples where you have, oh you
Speaker 3: may not get it right from the first run, but
Speaker 3: then you need to like adjust for it and then
Speaker 3: try again. I wish I could give their name that
Speaker 3: that's a very exciting name, but uh, but what they
Speaker 3: did was really uh you know, it's it's they were
Speaker 3: really happy with it. Uh. They were to the point
Speaker 3: they were always doing callbacks to the customers, but the
Speaker 3: selection process was not something that they were able to
Speaker 3: streamline to the level that you know that they get
Speaker 3: more and more better results out of it. Now they were.
Speaker 3: Now they're able to target the right customers because they
Speaker 3: are able to do the sentiment and now out of
Speaker 3: those goals.
Speaker 2: Center analysis is an interesting one. We actually have a product,
Speaker 2: it's like the number one stand up bought and Slack,
Speaker 2: and we tried to run some AI models on our
Speaker 2: responses for stand ups, and like it was very clear
Speaker 2: whether it was like a one or a five on
Speaker 2: a five point scale, but like you know, you get
Speaker 2: into and those are obvious, but the ones in the
Speaker 2: middle were quite the challenging spot there, and we ended
Speaker 2: up discarding the notion entirely because it is the sort
Speaker 2: of thing that I feel like, you really want to
Speaker 2: get right and not be in the danger area. But
Speaker 2: your example made me think of something that I had
Speaker 2: read recently actually, where a call center realized that the site,
Speaker 2: the psychological safety of their staff actually being super important
Speaker 2: for the success of the organization, was being threatened by
Speaker 2: irate callers that would call up and yell and scream.
Speaker 2: Maybe obscenities at them because you know, obviously supports are
Speaker 2: being made by people who are usually not happy. I mean,
Speaker 2: maybe that's a controversial statement, but I believe that's true.
Speaker 2: You know, generally speaking, you're calling up the support line
Speaker 2: because you're not thrilled with the service you're getting, and
Speaker 2: it was challenging for them. And what I heard that
Speaker 2: they were doing was actually uh using AI to alter
Speaker 2: the vocal expression of the calls they were getting so
Speaker 2: that they seemed more demur in content. Like they weren't screaming,
Speaker 2: they weren't swearing, they weren't like high volume pitches, automatically
Speaker 2: changing what the staff were hearing so that they wouldn't
Speaker 2: be susceptible or subjugated to an environment that would be
Speaker 2: worse for them and the company.
Speaker 3: Yeah, I mean that is that is you know, that
Speaker 3: is a very no way to way of using sexually right.
Speaker 3: It's yeah, I it's it's always hurt when you're on
Speaker 3: the first line. The customer is not happy, so it's
Speaker 3: it helps, I'm pretty sure.
Speaker 1: And then there's the people where the more swear words
Speaker 1: they have is actually an indicator that they are happy.
Speaker 1: We may be able to identify someone who fits in
Speaker 1: that category sort of tangential to this. That's one of
Speaker 1: the things I did in using chat GPT is I
Speaker 1: told it. I don't know if you' allre familiar with
Speaker 1: who David Goggins is, but he's a very vocal and
Speaker 1: blunt person. I think we can just say that. But
Speaker 1: I told chat gpt that it could adopt that person's
Speaker 1: speaking style and personality, and like instantly I was more
Speaker 1: productive with it because I could drop F bombs and
Speaker 1: it would respond with hell, yeah, let's go, And like
Speaker 1: all of a sudden, I was communicating with someone who
Speaker 1: spoke the way that I did.
Speaker 2: So I just assumed, and this is obviously not really
Speaker 2: knowing your history that if you've seen Full Metal Jacket
Speaker 2: and Drel Sargent, like I just assume that this is
Speaker 2: the life you've gone through, Will and your path being
Speaker 2: in the Navy, that you know there is something just
Speaker 2: for you special about being yelled at in this way
Speaker 2: which really gets you into gear.
Speaker 1: Right, Screaming F bombs is my love language.
Speaker 3: Okay, we're doing this podcast compete the wrong right?
Speaker 2: I mean, I guess we could release two verse is
Speaker 2: the stream, you know, the natural one and the one
Speaker 2: that's attenuated using AI that strips out all of the
Speaker 2: obscenities that are coming out of Will's mouth. So if
Speaker 2: you're not hearing him, yeah, I mean, if you're not
Speaker 2: hearing Will swear every other word. That's what's actually happening
Speaker 2: right now.
Speaker 3: The bill special, Right.
Speaker 1: There's so many ways that could go wrong.
Speaker 3: Yeah, but yeah, there's there's always that. I tried to
Speaker 3: do the same with the chick GPT back in the
Speaker 3: day on SWAL that didn't go well for me.
Speaker 1: So one thing I'm curious about, like, are there common
Speaker 1: use cases you're seeing from specific type of types of customers,
Speaker 1: like enough where you could say, like this industry or
Speaker 1: this segment is betting really heavily lead on AI and
Speaker 1: m L and they're actually making progress.
Speaker 2: That's a good question.
Speaker 3: Finance definitely, is there insurance companies both on the customer
Speaker 3: support side and also on the risk and analysis and
Speaker 3: so and so forth. Essentially the risk andalysos they have
Speaker 3: been doing for for many years, but as I said,
Speaker 3: the hype, the result of the hype is they are
Speaker 3: now able to adopt things that are more advanced, easier,
Speaker 3: and in some cases cheaper than they were able to
Speaker 3: do in the past. So those are the two that
Speaker 3: we are seeing. The one thing that I was surprised
Speaker 3: about is the consulting companies. Was so because as we're
Speaker 3: providing a tool, perhaps that's why as well we get
Speaker 3: a lot of interest from consulting companies, which are essentially
Speaker 3: some of them are actually building develops pipelines for other
Speaker 3: companies that are very interested in adopting our tools. So
Speaker 3: that was, you know, that's something that I wasn't expecting
Speaker 3: expected for me. So I do not really know what
Speaker 3: kind of industries these consultant companies are working for, and
Speaker 3: some of them are really large, So I don't know
Speaker 3: if you.
Speaker 2: Know, if you can tell you no, I mean you
Speaker 2: say consulting. And I had two things give to mind,
Speaker 2: like like management consulting, of which we knew all along
Speaker 2: that what the words coming out of their mouth could
Speaker 2: easily be puppeted by an LM and maybe third party
Speaker 2: contractors who are you're hiring to outsource some work. I
Speaker 2: could imagine they're trying to sneakly basically use LLMS instead
Speaker 2: of even cheaper because you know, supposedly it's even more
Speaker 2: cheaper labor than even outsourcing to deliver the work items.
Speaker 3: No, in this case, no, it was no. In this case,
Speaker 3: they are actually hired as experts were building a pipeline
Speaker 3: for their For instance, one of the cases that we're
Speaker 3: helping with is they're building a pipeline for consuming base
Speaker 3: models and retraining them or fine tizing them. So like,
Speaker 3: it's not that kind of work. It's not like adapting
Speaker 3: the LM for doing something. It's more about you know,
Speaker 3: taking base models, making sure the lineagures there, as bounds
Speaker 3: are there, and so and so forth. It's what it
Speaker 3: essentially is is they are building CICD pipeliness, very complex
Speaker 3: c CD pipelines for these companies, and as far as
Speaker 3: I understand, they are working on multiple projects at the
Speaker 3: same time. So there is a lot of consultancy activity
Speaker 3: that we are I'm sure that there is more. It's
Speaker 3: just we're seeing a few of examples for adapting our tools,
Speaker 3: but there's a lot of that going on. So that
Speaker 3: only tells me that it's like there is a bit
Speaker 3: of activity happening in industries that are known to use consultancy.
Speaker 3: So and to be honest, I am impressed on the
Speaker 3: fact that they are actually doing the right thing. It's
Speaker 3: like you don't at this point of the hype, you
Speaker 3: don't usually get the people doing the right thing and
Speaker 3: you know, building provenance, attestations, s bombs and so on
Speaker 3: and so forth. So when you see that, you're like, hmm,
Speaker 3: I'm impressed.
Speaker 2: No, I totally get it. I mean different field. But
Speaker 2: you know, we offer login and access control as a
Speaker 2: SaaS product, and we're talking with our customers and they
Speaker 2: say them things like, well, you know, because of these reasons,
Speaker 2: we decided to go with your product, And it makes
Speaker 2: me feel like really happy that at least there are
Speaker 2: some people out there that are that are thinking through
Speaker 2: this effectively, like whether or not they decided to use
Speaker 2: what we have or not. It's like I can actually
Speaker 2: see how they're thinking about it, and it's so much
Speaker 2: better than like the rest of what I see, which
Speaker 2: is like, oh, no, you know, we decided to hack
Speaker 2: this together, or you know, we're not tracking what we're
Speaker 2: doing or auditing.
Speaker 3: In it any way.
Speaker 2: You know, it's you know, I hate to say yoloing
Speaker 2: it out there, because you know they can it's not
Speaker 2: the most critically important thing for them as they see it.
Speaker 2: Even though we no longer term tracking the models that
Speaker 2: are being used, and how you're fine tuning it, like
Speaker 2: actually being able to trace it back. It's like going
Speaker 2: out there and as well loves to say right to
Speaker 2: production using v I to edit those production files. You know,
Speaker 2: it's great in the moment, but you know you got
Speaker 2: to trace that back to the get commit if you
Speaker 2: care about your production reliability.
Speaker 3: I mean, we actually encourred to a case where the
Speaker 3: model that was deployed was forgotten completely and it drifted,
Speaker 3: but it was still part of the application flow. It
Speaker 3: was still producing data to the application, and the application
Speaker 3: was putting that into into databases essentially, So it was
Speaker 3: at the end it wasn't something critical that they were doing,
Speaker 3: but it took them like they couldn't understand at first,
Speaker 3: Like that's what they told us. They couldn't understand at
Speaker 3: first how that happened, and they weren't even aware that
Speaker 3: there was a model there at that point that was
Speaker 3: that was generating the the data that they are seeing
Speaker 3: was drifting. I mean it's it was doing a simple categorization,
Speaker 3: but then it drifted so much that it was always
Speaker 3: putting things into irrelevant categories that it wasn't supposed to
Speaker 3: put into so, and that's what happens with with AI.
Speaker 3: That's why you need to be really careful with the pipeline.
Speaker 3: Like the v I example that that you have given.
Speaker 3: It's like you can get away with ed editing something
Speaker 3: with v I in production if if something is if
Speaker 3: if that deployment changes in one time in fifteen years,
Speaker 3: yeah right, so, but you know, if you have a
Speaker 3: deployment that changes only once in fifteen years, you may
Speaker 3: get away with that because your input and output never changes, right,
Speaker 3: It's like that thing is if it is working, it
Speaker 3: is working unless you have a bucket and software never
Speaker 3: has bugs. So but then with AI, it is going
Speaker 3: to change, whether that happens in six months, that happens
Speaker 3: in a year. You need to monitor that. You need
Speaker 3: to have a pipeline that will retrain and get rid
Speaker 3: of the drift and so on and so forth and
Speaker 3: all that. So I think that's where the main differences
Speaker 3: start to happen between DeVos and maybe mlops.
Speaker 2: That's a really good point, actually, I think you're worded
Speaker 2: that really well. You are only going to use mL
Speaker 2: in situations that are already so complex or changed so
Speaker 2: frequently to get the value out that you have to
Speaker 2: imagine that you built such a complex like pretend you
Speaker 2: didn't use mL. Think about how complex of a system
Speaker 2: you would have had in its place. There must have
Speaker 2: been so much testing around that, so much reliability, so
Speaker 2: much concern and fine tuning of that to get it right.
Speaker 2: You can't just throw away all those extra pieces because
Speaker 2: you're using some sort of model now. You still need
Speaker 2: all of them in place otherwise, you know, just think
Speaker 2: about that legacy system that's thirty years old and has
Speaker 2: you know, maybe one hundred million lines of code, Like
Speaker 2: that's just absolutely ridiculous to be running as a critical
Speaker 2: piece of software, right, I.
Speaker 3: Mean, there are problems like for instance, again this is
Speaker 3: a real one, let's say that you're producing shirts, yeah,
Speaker 3: and you're distributing these shirts to all over the world.
Speaker 3: How do you know what size is to produce and
Speaker 3: where to distribute them? Someone needs to make that decision.
Speaker 3: You cannot send the same amount of large to every country,
Speaker 3: and you cannot send the same amount of xx large
Speaker 3: to every country.
Speaker 2: There was actually, I think there was a good paper
Speaker 2: that was released. I think it was the US Army
Speaker 2: where they were trying to devise standard kits for sizes,
Speaker 2: and they performed a number of physical measurements of everyone.
Speaker 2: Everyone drafted to decide, okay, you know, we're gonna have
Speaker 2: a small medium in large kit how big should those
Speaker 2: things be? And after measuring and calculating the norms, they
Speaker 2: realized that there was never there was not one person
Speaker 2: in the set of like a thousand people that were
Speaker 2: able to fit into a standard distribution of one of
Speaker 2: those kids. They were like, it was like, it's really
Speaker 2: ridiculous and a good reminder that there is no such
Speaker 2: thing as following the norm and getting it actually right,
Speaker 2: and so there's no reason to believe that it would
Speaker 2: work in the eye world either.
Speaker 3: Yeah. But on the other hand, given enough data, you
Speaker 3: can actually predict me you know, how many how many
Speaker 3: larges that you need in China, and how many largest
Speaker 3: you need in US, and how many large you need
Speaker 3: in Canada because you have the historical data as well
Speaker 3: as other demography fixs and so on and so forth.
Speaker 3: So when you combine all of that, and this is
Speaker 3: actually used somewhere, when you combine all of that, you
Speaker 3: can actually come up with a data say that, oh,
Speaker 3: you know what I need this many excels in Belgium, right, so,
Speaker 3: and that is very important because you know, sending the
Speaker 3: wrong shirt sizes healthway around the world where you won't
Speaker 3: be able to sell is a lot of cost.
Speaker 2: I worked for a long time in manufacturing and manufacturing logistics,
Speaker 2: so I am well aware of the challenges there. So
Speaker 2: I appreciate the example. We're getting pretty up on time here,
Speaker 2: so I'm actually wondering if it may be the moment
Speaker 2: that we switched to doing some picks.
Speaker 3: Let's do some picks. That's exciting, Yeah, do it?
Speaker 1: So, Warren, you want to kick us off?
Speaker 2: Yeah, you know, I always go first. I think that's
Speaker 2: that's the secret format here.
Speaker 1: You know, feel free to call me out and I'll
Speaker 1: go first at any point.
Speaker 2: No, No, I think I go first. I think that's
Speaker 2: the uh, that's the rule here. All our listeners will
Speaker 2: know that by now. So my pick I sort of
Speaker 2: alluded this to this earlier about psychological safety on teams.
Speaker 2: There's actually a paper that was released by Google in
Speaker 2: twenty sixteen about trying to figure out what makes a
Speaker 2: great team, and the actual article is released on The
Speaker 2: New York Times twenty sixteen. What Google learned from its
Speaker 2: quest to build the perfect team, and I feel like
Speaker 2: in twenty two, twenty four, this shouldn't be a shocker anymore.
Speaker 2: But it's psychological safety and it's sort of ridiculous how
Speaker 2: important that is is really outlined in the paper, and
Speaker 2: that I still talk with companies today. My colleagues at
Speaker 2: other companies are ones that I consult for and it's
Speaker 2: not their priority. And it's like, you look at data
Speaker 2: like this and the only conclusion you've come to is
Speaker 2: it's the number one thing that everyone should be doing,
Speaker 2: because it's the guaranteed way of making the most successful
Speaker 2: company with the most revenue, the happiest customers and the
Speaker 2: best employees. And yet people still aren't doing.
Speaker 3: It fair enough.
Speaker 1: All right, GOK, what'd you bring for a pick today?
Speaker 3: Yeah, so it was a little bit last minute. So
Speaker 3: I'm gonna pitch a coffee machine.
Speaker 1: I'm super interested.
Speaker 2: You have my full attention.
Speaker 3: Okay, So a little bit of background. There is something
Speaker 3: called a golden ratio in the coffee world. If you're
Speaker 3: doing drip coffee right there, that there is a something
Speaker 3: called golden ratio, which is the amount of time that
Speaker 3: your beans are gonna spend with the with the boiled
Speaker 3: water and there is an institution I can't remember what
Speaker 3: that uh measures these these uh uh coffee drip coffee
Speaker 3: machines and issues the Golden ratio certificate. So this, yeah,
Speaker 3: the my my first introduction to this machine actually goes
Speaker 3: all the way back to Nokia, where we had these
Speaker 3: kitchens in every floor in Nokia, and there was in
Speaker 3: every kitchen there was one or two coffee machine that
Speaker 3: which you would go and make your own coffee, and
Speaker 3: uh it was so ridiculous that the coffee would run
Speaker 3: out and someone would have to make a new page.
Speaker 3: And one thing that we have done is we actually
Speaker 3: put cameras on top of them so that we would
Speaker 3: know if there is coffee in there or not before
Speaker 3: we live our spot. So all of these machines and okay,
Speaker 3: buildings were MoCCA masters. And the thing about MoCCA masters
Speaker 3: is they are the simplest machines that you can think of.
Speaker 3: Like all their parts, you can just take them up
Speaker 3: apart and then put them in. Probably the most expensive
Speaker 3: piece in it is the copper wire which boils the water.
Speaker 3: And that's actually the secret of it, because it boils
Speaker 3: the water into the correct degree at the correct time
Speaker 3: and let the water go through the beans your coffee
Speaker 3: with the correct ratio, and therefore it's considered one of
Speaker 3: the better coffee drip machines that you can get, and
Speaker 3: they're expensive. Right.
Speaker 1: And on that note, I just learned last week that
Speaker 1: the webcam was invented and I can't remember who it was.
Speaker 1: I want to say it was at Nokia. The webcam
Speaker 1: was invented because the engineers were tired of going to
Speaker 1: the break room finding the coffee machine empty, so they
Speaker 1: figured out how to hook a camera up to their
Speaker 1: network and invented the first webcam.
Speaker 3: Uh well, by the time I was there it was invented,
Speaker 3: it was, but it was a very common practice, Inchia.
Speaker 1: Yeah, rightly, so, rightly so, all right. So my pick
Speaker 1: is a little bit related to what we've been talking
Speaker 1: about to just with the early stages of AIML, you know,
Speaker 1: and how do we make this production ready. And so
Speaker 1: my pick is actually Voyager one and Voyager two, the
Speaker 1: spacecraft that are outside of our solar system now. And
Speaker 1: the reason I pick those is because just go and
Speaker 1: watch some YouTube videos about this or read articles whatever
Speaker 1: you preferred format, is the level of engineering done on
Speaker 1: these things is so amazing. The data storage on these
Speaker 1: things is an eight track tape drive that's still working
Speaker 1: forty seven years later, and you know, it's gone so
Speaker 1: much further. Both of them have gone so much further
Speaker 1: than what they were ever planned to do. But when
Speaker 1: the engineers were designing it, they kind of hoped that
Speaker 1: this would be the scenario, and so they built for
Speaker 1: this and here we are utilizing it today and crazy
Speaker 1: things like the backups that they have on there. The
Speaker 1: primary thrusters worked for a long time and then they
Speaker 1: rightly so they failed, and so after thirty seven years
Speaker 1: of sitting dormant, they fired up the backup thrusters and
Speaker 1: they worked flawlessly. And so just the level of engineering
Speaker 1: in this in the Voyager spacecraft, I think is worth
Speaker 1: spending a little bit of time to just acknowledge and
Speaker 1: admire and then reflect on how that same engineering philosophy
Speaker 1: can be applied to your common everyday tasks. So Voyager
Speaker 1: one and Voyager two are my picks for the week.
Speaker 2: You got to upstage everyone there?
Speaker 1: Oh well, now, I just it just it was on
Speaker 1: my YouTube feed last night. It wasn't planned at all,
Speaker 1: but after watching it, that was like, wow, this relates
Speaker 1: to what I do on so many levels.
Speaker 2: There's an inspirational YouTube video sitting out there that we
Speaker 2: should get in the link section of this episode.
Speaker 1: I'll pull I'll pull on YouTube history and get the
Speaker 1: link and make sure it gets into the show notes.
Speaker 1: Sounds good, Yeah, because it was a cool video, super
Speaker 1: cool and it's only like twenty minutes. It's not one
Speaker 1: of those two hour ones. And on that note, Gorkam,
Speaker 1: thank you so much for joining us today. This has
Speaker 1: been a cool conversation. I'm looking forward to see how
Speaker 1: it comes out. And now be sure to keep in touch,
Speaker 1: let us know what kind of things you're on, and
Speaker 1: when you come across something cool, I'd love to have
Speaker 1: you back on to talk about.
Speaker 2: It and figure out how it works.
Speaker 1: Awesome, And to all the listeners, thank you so much
Speaker 1: for listening. Be sure and reach out to us if
Speaker 1: you have questions, comments, episode ideas, or someone that you
Speaker 1: want us to have on the show. We'd love your input. Jarren,
Speaker 1: thanks for joining me today. Yeah, as always, all right,
Speaker 1: and we'll see y'all next week.