Speaker 1: How'd you like to listen to dot net rocks with
no ads? Easy? Become a patron for just five dollars
a month. You get access to a private RSS feed
where all the shows have no ads. Twenty dollars a month,
we'll get you that and a special dot net Rocks
patron mug. Sign up now at Patreon dot dot NetRocks
dot com. Hey, welcome back to dot net rocks. I'm
Carl Franklin and I'materi Comp. You are here for your
dot net keeeking out, listening, pleasure and all that stuff,
all the things, all the things anything you want to
announce today, buddy.
Speaker 2: No, Well let me I told you earlier. But yeah,
I know we got a puppy. Okay, because you know,
having a grandchild wasn't enough, we also need a puppy. Yeah,
so I've managed to tie this while she's at yoga.
Got to have the puppy asleep.
Speaker 1: So wait, Manute, I thought you were swearing off dogs
after Zach.
Speaker 2: Listen, I already had the perfect dog was seventeen years.
Don't need it down a dog. I didn't need that dog,
to be clear. But sometimes sometimes you're out numbered one
to one, you know, Oh that's great that it's really
got to be her dog this time. So I'm doing
the only logical thing I can do, which is good
to Australia for three weeks. Okay, I'm leaving on Thursday,
so I'm literally I'm gonna go away for three weeks
and hopefully.
Speaker 1: What kind of puppy is it?
Speaker 2: It's a it's a mix of Ausi Shepherd and border
collie and some poodle and some burnies, a little bit
of everything.
Speaker 1: It's Oh my gosh, it's a cute little.
Speaker 2: Dog, no two ways about it.
Speaker 1: But you know, does it have an identity complex?
Speaker 2: It's got springs and its legs is when it's gone.
Speaker 1: Wow.
Speaker 2: Okay, man, we're in the early stage needle teeth and
pee on the floor and bit by bit figuring it out.
I already took a tick off for her because she
was out in the lawn.
Speaker 1: What's her name, Jojo short for Josephine Lily. Oh she
doesn't look like a deep fried potato wedge then.
Speaker 2: No, not like some dogs, you know. Oh, she's a
cute little dog but cool. And she's got that French
twying tour so that hits the crazy name, right, anyway
I got, I got worse problems like, yeah, it makes her.
If it makes she who must be obeyed happy, then
I'm happy to do fantastic.
Speaker 1: All right, Well, let's jump into it with a better
NO framework.
Speaker 2: All right, what do you got?
Speaker 1: All right? Well, in honor of our AI guest today, Vishwaesle,
who's been on the show several times before, many times,
I found a trending repo on GitHub. It's called krillin
ai k r I l l I n AI or
a one, depending on your point of reference. It's an
AI audio and video translation and dubbing tool. So let
me just read the paragraph here now, just does a caveat.
I have not downloaded it. I don't know if it's
any good, but it is trending, so that tells me
that people kind of like it. Crillin ai is an
all in one solution for effortless video localization and enhancement.
This minimalist yet powerful tool handles everything from translation, dubbing
to voice cloning. I see there's no the commas don't
work here. Translation comma dubbing to voice cloning comma. I
would say, everything from translation and dubbing to voice cloning.
Formatting seamlessly converting videos between landscape in portrait modes for
optimal display across all content platforms. YouTube TikTok, Billy Billy, Duian,
we chat channel, Red Note Kawai show. With its end
to end workflow, crillion Ai transforms raw footage into polished,
platform ready content and just a few clicks. So that's interesting.
You know, translation a localization anyway is one thing in
the world of software and web apps that we know
very well. But dubbing and you know, translating video that
has audio in it.
Speaker 2: Well, the voice cloning one is the craziest one where
I take the voice of the person speaking and then
change their language, but it still.
Speaker 1: Sounds like them change the language.
Speaker 2: Yeah, that's nuts, but it's very jen Ai.
Speaker 1: Yeah, it's nuts, but you know that's the stuff that
we get to see. This is the world we live
in now, so world we live in. So that's what
I got. Richard, who's talking to us.
Speaker 2: Today, grabbed a comment off of show nineteen forty four
just a few weeks ago with our friend Jody Bershall,
doctor Jody Bernshall. Yeah's got a few comments on the show,
and this one's from Joshua hillar Up, one of our regulars.
I'm pretty sure already has a copy of Means to
go By, But you know, Joshua, thank you. And he said,
I'm thinking about the non deterministing aspects of LMS, and
what's frustrating is that there's been many decades of computer
science research on software correctness. It just never seems to
get used in the industry because it's too hard to learn.
If you did that in air quotes, too hard to learn.
LLLMS work really well at covering that hard work, and
I don't think it's as hard as people claim, at
least in the beginning. My hope is assuming enough money
put into developing these languages and tools to make the
use of that research. Now there's such an obvious use
case for it. And we probably should have talked about
this more with Jody too, Joshua, because she does come
from that I hate to say old school, but because
that would be like five years ago form of machine
learning where quality and accuracy, you know, validation was a
huge part of the job.
Speaker 1: That's right. I remember talking to Seth Warez and he said,
ninety percent of my time is cleaning the data.
Speaker 2: Yeah, to get to a meaningful quality. So there is
certainly some pressure on that, but we're we're bringing a
lot of people in without a lot of experience the space,
and they are not concerned about these things and they
should be.
Speaker 3: M H.
Speaker 2: So as usual, Josha, thank you so much for your
great comment, and a coffee of music. Cobi is on
its way to you. And if you'd like a copy music,
go bay rte a comment on the website at dot
netrongs dot com or on the facebooks. We publish every
show there and if you comment there and I've read
on the show, we'll send you a copy music O
bay music to.
Speaker 1: Code by dot net. That is how you can find it.
It's still going strong. Track twenty two is available now
and of course you get the whole collection MP three
Flak or Wave. This is episode nineteen forty seven, and
so I can just give a little brief summary here
about what happened that year. It was a year of
significant global transformations. Of course, India and Pakistan gained independence
from British rule, and the Marshall Plan was initiated to
help rebuild war torn Europe. In the United States, Jackie
Robinson broke baseball's color barrier, and Chuck Yeger broke the
sound barrier. Also, the Truman Doctrine, which pledged US support
for nations threatened by communism, was established, which signaled the
beginning of the Cold War. You got anything to add?
Speaker 2: Nineteen forty seven is the year the invention of the transistors.
That's bartying Britain and Shockley with the first point content
transi should have been talked about for years, but semiconductor
making pure material semi conducts were hard. A lot of
that came out of the war. And so what's funny
is reading some of the documentation at the time and
they weren't sure if it was actually going to be
all that useful. Interesting, you know, because vacuum tubes did
such a good job and this thing did not have
the potential the vacuum two did because the materials weren't
quite good enough. Yet they had no idea what was
about to happen. Huh No, but they you know, Shockley
would go on with Fairshire Semiconductor to help make the
originalized sees with Robert Noise and all those like. All
of that comes from there. But there's the beginning and
one other fallout of the war raytheon's radar range, the
first microwave oven.
Speaker 1: Wow, yeah, radar range, that's what they called it.
Speaker 2: They called it. They called it the radar range literally
because some of the early earlier technicians working on radar
noticed that the chocolate bars in their pockets would melt
when they got too close to the magnetrons.
Speaker 1: Okay, so how safe was this thing? It doesn't sound
like it was very safe.
Speaker 2: It's it can be is superheating the water, so it
will you will get water burns if you get too
close to it.
Speaker 1: Yeah, right, And but radiation poisoning. Were they worried about that?
Speaker 2: No? No, still, there's no I you know, ionizing radiation
is more complicated than this. This is just microwaves, just
like a bright light, and it can burn you because
it vibrates the water molecules, and those water moleoscules, once
they get to a certain temperature, are going to damage
the tissues they're in.
Speaker 1: Okay, all right, let's bring Vishwaz back on dot net
rocks for the empteenth time.
Speaker 2: It's at number seventeen, by the way I counted. Hey,
well that's umpteen, isn't it. That's umpteenth. Yeah, we're ext
the umpteeth category.
Speaker 1: Bishwaslele serves as co founder and CEO at p wind
dot AI, A noted the industry speaker and author. Mister
Lealey is the Microsoft Regional Director for Washington, DC. Welcome back, Vishwas.
So it's new in the world of AI in vishwas Land.
Speaker 3: Well, i've been. We talked to you last year, very
focused still on trying to grow this Jennai startup idea
that we started working.
Speaker 2: Yeah, you left your longtime company to do this startup.
Speaker 3: Yes, I left. I was there for thirty years the
previous company. How's it going last sixteen years? Is the CTO? Well,
it's going. It's going well. I mean, as you can
expect as a startup. World is all about good days
and bad days. But it has been very exciting a
lot of learning and we are growing.
Speaker 1: That's cool.
Speaker 2: Now. If I recall when we talked last time, you
were focused on generating responses to requests for proposal for
the RFPs. That's right, right, which in government land can
be absolutely massive, like hundreds of pages.
Speaker 3: That's right, that's right, and that's what we have been
focused on. We call ourselves a thoughtful copilot. And I'll
explain the word thoughtful in a second. We call ourselves
thoughtful copilot for proposal writers. So what that means is
we will take your information. You're you know in the
years past you have responded to RFP documents that you've done.
You have competency documents, you have some fancy documents like
spars and for those who don't know this already, se
parts are documents that government agencies give you to validate
your experience. So, needless to say, you have a collection
of documents and now a new RFP comes along. And
these RFP documents can be, as you said, Richard, can
be really complex, can be fifteen hundred pages long with
different sections in them. So for example, they might be
a section which describes how should you respond to the RFP,
maybe a section which talks about what would be the
evaluation criteria for somebody looking at your RFB. And then
there may be another section which talks about all the
things that you must do.
Speaker 1: So that's all the boring stuff.
Speaker 3: Our job is other boring.
Speaker 1: You Usually the stuff that you want to read last,
you have to read it first.
Speaker 3: Yes, yes, and all this is part of something called
the Federal Acquisition regulation, which something came out many many
years ago, has different sections related to different parts of solicitation.
So what we specialize in is, now that we have
your knowledge repository built out, we deeply understand the solicitations
that comes in, and then we help you write a
draft quickly solely based on your information, so that we
can give you this first draft quickly and then you
can build on it. And there's a very interesting saying
in the proposal world that proposals are often one at
the end Richard that you know, you leave yourself enough
time to do some creative thinking, then you goos to
your competition, you do some innovative thinking. But the sad
reality of proposal writing is you never reach the end.
You spend all your time in the middle, sure, just
trying to get a good proposal out. So our value
proposition is very simple. Can we accelerate the initial stages
so that we give you more polishing and thinking time.
And that's ultimately what improves your probability of winning And
which is the name of per company, peven dot Ai.
Speaker 1: The probability of winning. Mm hmm, that's really good.
Speaker 2: Well, part of this again, we're calling the previous show
was being able to respond to more RFP so you
have more chances at work opportunities. The thing, of course,
has scared me is you're committing to stuff inside this
RFP yep. And so if the tool is already the
text for it, and it creates a commitment for you, you
can't deliver on like you could be in serious trouble.
Speaker 1: I think that's why you said creates a rough draft, right, Yes,
you can't just like say, okay, here it is, send
it off.
Speaker 3: See. Yeah, And that's that's the thing.
Speaker 1: Yeah, you certainly have to know what you're saying.
Speaker 3: Yeah, no, Carl, that's a very good, good catch there.
That's why I said draft. We're trying to improve the
quality of drafts so that you know, you can focus
on other things. But at the end of the day,
make no mistake, you're putting your name down on the proposal.
Speaker 2: Sure, so you should read it.
Speaker 3: You should read it.
Speaker 1: So it's kind of like having a you know, a
manager under you or a secretary or somebody who knows
your business say hey, you can just say look through
this RFP and give me a rough draft and it
might have all the accurate information from your side, but
you know, you're the one that's got to commit to
what you're agreeing to. And so I guess my question
here is can you go back. I can have a
conversation with this bot if you will, and say, you know,
why did you why did you commit me to doing X?
You know, and have it give you an answer from
your data.
Speaker 3: That's a that's a good point. We have an intermediate stage,
and this is the pattern patent should say that we filed.
We have a provisional patent. Yeah, so what's It's not
like the bot is just or the copilot in this
case of our proposal, copilot is just going to look
at your documents and just spit out one hundred page
document that it thinks that you should be doing. That's
a recipe for disaster because you go pick up things
that you don't want it to do. And there's a
very important threshold where you really have to save time
to people. If people say, hey, I have to first
read your silly forty documents that you generated and then
I have to rewrite it. Thank you, I'm not going
to use this tool. So there's a very important threshold
that you have to cross. So in our model. The
patent that we have filed is called object based writing.
Oh wow, okay, and what that means is and what
that means is that once let's say the r FP
aries and we have gone in and tried to analyze it,
and we create something called the flight plan. And why
are we calling it flight plan because we are the
copilot for proposal writers and in the aviation industry, obviously
there's a very important concept called flight plan that is
used for communication. So what we do is we try
to deeply using AI models, try to deeply understand the RFP,
understand the evaluation criteria, understand the instructions, and we manifest
that into a UI pattern called the flight plan. So
you go in and then you intera to the flight plan,
and this is where you go and say that I
want you to use this architecture. And let's just take
a concrete example that our listeners can follow along. Let's
say the RFP is about a records management solution, right,
and you may have done ten different types of records
management solution, maybe use SharePoint for that, maybe used some
purpose built records management solution things for that.
Speaker 1: By the way, I'm sorry, yep, yep.
Speaker 3: So if you had to do that. So once the
flight plan comes in, what we do is we say, hey,
we think that you have used these are three or
four architectural patterns to go after this kind of work.
Here's a suggestion, but please you need to update that
with maybe, like you said written Carl, SharePoint is not
the right solution. You want to do something else, so
replace that, And in that flight plan you tell us
that you know, I'm wanting to undertake this task using
this architectural pattern. And then you also in the flight
plan you get to tell people do you really understand
the pain points? Do you really understand the motivation of
the buyer? Why did they issue the RFP? And do
you why do you why do you think you're differentiated
than other competitors? If all your differentiation is or we
have all the certifications in place, well, every other competitor
is going to say that. So we force you to
think critically about the response and at a higher level.
Speaker 1: So is it kind of like an outline that you
can interact with? It seems like it is. You you
have a model for a conversation with the AI. Yes,
so yes to this, no to this? What's about that?
Speaker 3: Yes? And you know, so you're setting think of this
as that you're setting the vision for the response that
you want to generate. Right, you're talking about your competitors,
you're talking about your win themes, you're talking about your
pain points, you're talking about your solutioning strategy, and on
and and on. There are lots of data structures that
we capture, and then once we have that information, then
we try to in a thoughtful manner, try to infuse
what you gave us as a direction. Right, if your
capture team, if your salespeople have done a good job
and they weren't their salary, they probably on the golf
course realized that the key motivation for this RFP was
the pain point this organization is going through is the
last three upgrades were not there, and that key point
AI does not know about or would never know about.
And this will be your capture team. And then you
take that information, you have all of the data structures,
you have your knowledge repository, and I'll talk more about it.
We do a lot of work, and I very much
agree with what you were saying Card earlier about sets comment.
We do a lot of collation, classification, tagging of content
and cleaning of content. So now have you have your content,
We have your vision for how you want to respond,
and then we start drafting your content in a thoughtful manner.
And that drafting is done using best practices. And this
is where we are really fortunate to have Shipley. I
talked about this last time. Shipley is a fifty year
old company which is which taught people how to write
persuasive proposals. That's what they do. They teach classes, they
generate content. So now we have this information, we use
their best practices. And one example would be when you're
responding to an our RFP, be sure about customer centricity.
We talk about customer's problem, but speak to those problems
maybe in your terms without sort of taking ownership of
the problem. And as if you're teaching them right about
active voice, passive voice, and then very important those are
all very important. And then you start with the problem understanding.
Then you say where you have done this kind of work,
you do you have really good past performance about this.
So they are best practices of writing that come in.
And this is why it takes us minutes and hours
to generate the draft, right because you know, I was
joking with you last year when I was the show
that if you went to Chad Gypt and said, CHADGEPT,
this is a very important question. I'm going to ask you,
please think about it for five minutes before you respond.
Chad GPT does not know what to do with the
four minutes and fifty nine seconds, right, because it just
gave you the answer right away. Right in our case,
that's how it works. In our case, what we are
doing is, Oh, you have asked us to draft this information.
Let us try to Oh you told us this here, okay,
and your knowledge repository saying this and one other thing
that the comment that you read just a moment to
good Richard that if you tell the language model that
please take these ten things that I'm telling you and
infuse them and generate some content. Language model, maybe listen
to the first or second requirement and maybe the last
requirement and then forget everything in the middle.
Speaker 2: Right.
Speaker 3: So this is where the thoughtful copilot comes in. We
go step by step manner to generate the content.
Speaker 2: I love it, but I appreciate that this is I
got to imagine when you've an experienced RFP person, you're
probably comparing the new RFP to past ones you've done.
It's got to be a certain amount of cut and
paste there at least referencing material, which means you tend
to try and solve the same problem over and over again,
rather than this customer centric like do you understand this
particular problem this time, which may or may not be
explicitly stated, and then make sure you're tuning the material
to that, even if it is from past material. This
is where the language model actually is super advantageous, because
you can train it on the past data and it
will generate original texts yep, presumably with the impetus of
the new customer problem attached to it. This is stuff
that's hard for people to do, but I think the
software be pretty good at it.
Speaker 3: That's right. This is a very important point I in
my previous role we talked about. I was there for
a long time and I didn't write proposals, but I
often served as a technical smme for a certain section. Right, Hey,
this is the section we need to write about. How
should we write about and what would happen there is?
I would tend to talk about the projects that I
focused on, you know, even if they were not the
best fit. Right, because organizational memory influences how you respond
versus This is where the superpower of the language model
is that you have one hundred different past performance to
look for, and maybe the the ones I picked because
of my bias because that's what I've worked on and
I'm very proud of those, Maybe those are not the
best ones. Maybe you should put something else that has
better engagement to the problem at hand.
Speaker 2: Right, So.
Speaker 1: I can see the advantage of this because it's very specific.
But then again, you have to feed it your data.
What about the stuff that's built into office You know
that that knows like all of your documents and your
spreadsheets and your your files and your PDFs and all
that stuff. Is it too broad? Is that knowledge too
broad to feed an RFP into and say come out
with something meaningful.
Speaker 3: Yes, that's a that's a great question. We we don't
get that question now much much much more, But previously
we used to get this question, why should I use
your copilot? We have the M through sixty five co pilot, right,
And the answer to that question is, if I can
even take a step back, you have general purpose Shared
chat GPT, claude.
Speaker 1: What have you?
Speaker 3: Right, and think of them as outside your firewall. Great tools.
I love them. I'm sure all of you have two
or three or more subscriptions because they are great at
synthesizing information, summarizing information. Just you're trying to write something
about a topic, it's great to collaborate with generate some content.
Of course, you have to think you ultimately you own
the content, and there can be inaccuracies, of course. So
that's outside the firewall tools. And I say that with
some hesitation, because only two weeks ago or three weeks ago,
chat gpt announced that, hey, in addition to outside stuff,
you can also point to a Google drive or something
like that and we'll start ingesting your content. So let's
just keep that aside for a moment. Let's just talk
about Claude and chat GPT. Great tools, but outside the
firewall tools. And then you have the office productivity tools
like and three sixty five Copilot an equivalent that are
built into, built into your enterprise. And they're great, right.
They know who you talk to a lot on email,
they know who the subject matter experts are. They have
a full access to your one edrive, what have you.
So they're great. But again they are general purpose, and
two important differences right. You can use them to write
an email to a customer whose irate. You can use
it to plan a party, a going away party for
your co worker. They do well, they will get all
these things right. You don't have to worry about copying
and pasting information because they have access to all the data.
There is security built in. They only get to information
that you have access to. All all great things with
those productivity tools. But when we talk about a domain
specific coal pilot like ours, there are more things that
need to be done. So I talked about having very
specific documents. We go to a customer and say, can
you please give us your past RFP response documents, and
maybe they have two thousands of those documents. We look
at those documents and we break them up into logical boundaries,
We classify them, we auto tag them, things like that.
We are very opinionated about what we are trying to
decipher from your proposal library content.
Speaker 1: It's a specific data set rather than everything in your
enterprise exactly. You might not you know, you might not
want that email to your wife about the new dog.
In your response to the you.
Speaker 3: Don't want that and no, Caul, that's absolutely right. And
there's one other sort of technical issue. Right, the general
purpose tools don't know exactly how to chunk these documents,
and often they chunk it at some page boundary or
some fixed word boundary, and we cannot do that. We
chunk them based on the logical sections the structure of
an RFP document, so that when we provide that as
a context to the language model, we provide the most
precise context. And it's all about context, right. Language models
can get confused easily, so we provide So that's one
big difference, right, all of the upfront work. If you
ask me where where we've done our most R and D.
We are fifty people strong now, so you're growing quite
a bit. So fifty to fifty two or fifty three
people and many data scientists awesome, awesome team of engineers, mlops, DevOps, whatnot,
all running on measure and of course data science engineers.
So one part of research is ingestion. How can we
be opinionated optimized ingestion? That's one part. And then also
how do you glean some metadata from these documents? That's
also important, right, because semantic search can break down if
you're trying to find very factual information. So how do
you extract the metadata things like that. That's one part
of R and D. The other part of R and
D is we talked about the flight plan. We talked
about how you get so much data from the user
about win themes and pain points and solutioning and all
of that. Right, how do you effectively generate prompts dynamically
so that you can generate quality content that adheres to
proposal writing best practices. So those two things together is
why there is a domain specific copilot, and people try
to do that. Initially, Hey, let me just upload the
RFP to the to the M three sixty five copilot
and say can you generate a response? And then in
some cases the response is limited. We are generating a
response which can be one hundred and sixty hundred and
fifty pages long. So, first of all, you can't do
that with a general purpose copilot. And secondly, how do
you without having to write all the prompts yourself? In
our system, the users never have to write a prompt
because the prompts are changing. The best practices for prompts
before the reasoning models came along, and the best practices
for prompting after using models are very different and you
can't expect your end users to keep learning new prompting techniques,
so we dynamically generate the prompts based on what is
in your knowledge repository. Right, So those are the two things.
Sorry for the long answer, but those are the two
things that are different.
Speaker 1: No, that's okay. Yeah, that's really great. And this seems
like a good place to take a break, so we'll
be right back after these very important messages. Stay tuned.
Did you know you can lift and shift your dot
net framework apps to virtual machines in the cloud. Use
the elastic beanstock service to easily migrate to Amazon EC
two with little or no changes. Find out more aws
dot Amazon dot com, slash elastic Beanstock, and we're back.
It's dot net Rox. I'm Carl Franklin, that's Richie Campbell hey,
and that's vishwas Lele our friend. And just as a reminder,
if you don't want to hear these ads before, during,
and after the show, you can become a patron a
Patreon for five dollars a month. Go to Patreon dot
dot netroocks dot com. We'll give you a feed that
has no ads. You still have to hear me apologizing
for them. Okay, well, you know the thing, as you
were talking, it just kind of occurs to me that
what you're doing is you're creating a company around a
specific domain, right that where you know how to ingest
the documents, you know how to query, and you fine
tuned it. It's not like somebody is going to come
to you and say maybe they have, but as anybody
come to you and said, hey, you did such a
great job for this domain, can you make me a
company around this domain? And that's essentially what you've done
with just the starting with the idea of their domain.
And I imagine that's what a lot of our listeners
are trying to do with their own businesses and their
own domains, isn't it.
Speaker 3: So that's an excellent question, Carl, That's an excellent question.
Let me my thinking has evolved over this question. I
think you had asked a similar question last year when
we talked, and my thinking has evolved. So, Number one,
I think the patterns that we have figured out in
terms of breaking down these documents in a domain specific manner,
generating these prompts dynamically, those patterns generally are applicable to
any business problem. So that's right. But what I've also
come to realize is in order to get true value
from jen AAI for any business process, any complex and
proposal writing is a complex business process. There are hundreds
and thousands of business processes. You really have to do
a lot of domain specific work in order to provide
value to the customers. It's not like you could take
these patterns. Well, you have to Let's say you went
to the financial sector, you went to the insurance sector,
You'd have to do a lot of domain analysis to
figure out what are the key entities, what should be
the metadata. You'd have to do a lot of work
to figure out the best practices, think about the prompt
generation engine. So it's just not a matter of taking
these patterns. Maybe the patterns supply at a higher level,
but it is the work that has gone into realizing
these patterns is where we have spent the last year in.
Speaker 1: Right, because it seems like there's a lot of people
that are using RAG and it's kind of like what
you're doing here, But I don't think there from what
I understand, look at it, like as Richard likes to say,
it's in a squirt bottle. You know, we just want
to spray some RAG on our documents and be able
to query them and get what we want out of them.
And what you're talking about here is a lot more involved,
Like it's not just a RAG tool that you're using.
It's everything from the original document, how to break them down,
how to chunk them, how to put them in there,
the things to look for, the way to respond like
it's U And so I imagine you're we don't really
look in terms of accuracy because you're not saying you
know how many widgets are in BIN forty five right now,
You're not. You're really genuine generating a text response which
you ultimately have control over.
Speaker 3: But accuracy is very important part for us. By the way, Carl,
responsible they is an important tenet for us, and it
manifests for us in many ways. And I'm really glad
that we tried to adhere or architect to these tenets
from the very beginning. And let me give you a
couple of examples. Right, so, every time we generate a
piece of text for you, the responsible AI tenant says
that you should try to be transparent with the user.
So you have to tell them exactly which document, which
paragraph did you source the content from. That's one. And
then ultimately we said, it's a shared responsibility model. It's
about the complementarity between the AI models and the human beings.
So how do you help the proposal writers get you
an accurate response sooner? So we generate something called the
hallucination report every time we generate a piece of text.
It and what the haalucination report is that every time
we write out some text, we try to extract any
assertions that you've made in that text or the AI
has made on your behalf, based on your knowledge repository.
And we said, you know, you're saying that you've done
these five thousand no cloud migration, but we really can't
find anything like this in our knowledge repository, right, which is.
Speaker 2: A great thing for it to say when it can't
find something, rather than make something up.
Speaker 3: Yeah, that's right. And we know that the sales team
tends to sometimes take a creative license with some things.
Maybe they combine to projects. Right, So it's not saying
this is wrong, it is saying that it will it
will serve you well to take this statement. We can't
find an explicit reference to the statement in or repository.
Can you come back and sort of take a look,
and maybe you can approve that risk and say, no,
we are taking these two projects in and we're saying
that overall we have done these kinds of migrations for
this aggregated agency, and that's how we are making this assertion.
That's fine, you can approve that risk.
Speaker 1: Yeah, it may be an accurate inference or it may
be completely wacky.
Speaker 2: But at least if you have the references, you can
go evacuate it yourself if you want, Yeah.
Speaker 3: You can. You can go back and read. And the
third thing.
Speaker 2: That we also do and argue the interpretation.
Speaker 3: Third thing that we also do is something called the
completion report. Right, So the RFP document was forty to
fifty pages long. It had instructions, it had evaluation criteria,
it had many many requirements, and it's a very important
aspect of responding that you're compliant. You're answering all the
questions that you were asked to do.
Speaker 2: Yeah, you didn't miss any you didn't miss any.
Speaker 3: So we again try to generate a compliance matrix for
you and say, look, these are the things that were
asked in the RFP. But again, this is a very
important part you have to go and validate that yourself,
because language models can be inaccurate at times. So it's
a very much an assistive technology, and we are not
quite getting it to autonomous anytime soon. That press up
button proposal comes out that you're ready to submit.
Speaker 2: But it also occurs to me that again this is
something it's good at that it could lead you to
cerve responses or lack of response to a requirement, the
same way that a generative AI image recognized er may
point a radiologist to a particular spot on a picture
and say this looks anomalous yep, and encourage them to
look in the right corners. Given all the guidance you're
providing here, which I really appreciate, right that the flight
plan approached the prompt generation to make sure it's accurate
and focus on the right material with the references. Does
the language model actually matter? Like you talked about open
AI last year, Now there's been many more models released.
I mean cloud was always around, but deep seeks appeared
like does any of those make a difference given this
strict set of guidance.
Speaker 3: That's a great question, So let me answer that using
two important points. Right turns out that these rf peace
can also have sensitive information, and you need to have
things like CMMC two standards where you are explicitly telling
the government that this information is not just public RFP.
So think about you're going after some RFP that talks
about some kind of things weapon system and even the
RFPs can have what is known as a controlled unclassified information.
So in order to support and show that you're handling
that information correctly, you need to show to the auditors
where your packets have traversed in that system. And then
ultimately where did you send this data and did let's
say you sent it to the external system like language model.
Did that language model itself have a security classification and
it is giving you guarantees about not logging that data
beyond the metadata and things like that.
Speaker 2: Right, yeah, and the sovereignty where did that they actually go?
Speaker 3: Yes?
Speaker 2: Now, I mean this makes a strong case for running
as a local model.
Speaker 3: Well, that's true, we use but you know the same time,
these RFPs are very complex, so we want the best
comprehension that is available.
Speaker 2: You want a trillion parameters, you need want trills.
Speaker 3: I need a trillion parameters, right, So what we do
is platforms like asure opening. I give you that guarantee
and you can actually show to people how your network,
how do your packets from all the way from incoming
RFP to how the response generation traversed a certain network
path that the auditors can say, yep, we certify, we're
okay with that, that these packets are fine. So that's one.
So it's not just a matter of hey, let me
just go send this data packet to somewhere else unless
they have a classification. So that's one one important consideration
for us.
Speaker 2: So just narrow is your choices of models based on
whether or not they're going to clear see see two.
Speaker 3: Yes, that's one. And secondly, Richard, the other thing that
we've realized is that these models have a personality. So
just because they speak English doesn't mean you can just swap.
Speaker 2: Now anthropomorphic of you Vishwa's goodness, Yeah, you've pushed.
Speaker 1: Richard's answer promorphic button there.
Speaker 3: Yes, so these models have a personality against say saying that,
And so I find this interesting when people say, hey,
these are models are all English pays, so you can
just switch one and see which one works. When you
are generating hundreds and hundreds of pages of prompts. As
we do, it is really important for us to understand
the personality of the model, and we can change the model,
and in fact, we change the models all the time.
When we went from four to four turbo and so on,
and now we're using oh one and three. Every time
we change a model, we have to do extensive testing
to make sure our prompting layer works well. But at
least these models are a certain family. Part of these models,
just switching to us different models.
Speaker 2: You've all described you're only talking about open ai models
there so far.
Speaker 3: Yes, we're talking about OpenAI models, so that's the other part.
So those two important considerations have gone in into deciding
which models we go against. We're thankful to open ai
and Azure open Ai that their models have generally kept up.
It's not like they're not they're the ones who came
out with the reasoning models. Of course, reasoning models are
available on other platforms as well now, but they've kept
up with the advancements that are happening. So we're pretty
happy with where we are. But we constantly run tests
to see how other models would do.
Speaker 2: Now and you mentioned already the difference between using a
model that doesn't have a reasoning feature, and I'm doing
air quotes yep, because I read the Anthropic paper where
they talked about the fact that the whole reasoning behavior
is a post back or behavior that the result to
the prompt generated and then analyzed after the fact by
the tool to generate the quote unquote reasoning. Right.
Speaker 3: So the idea here is that I'm trying to generate
a response to a certain section. Sure, and either I
can just in a non reasoning model, I would just say, hey,
generate an answer, and we'll just go through a certain
chain and just start generating a response immediately. But the
reasoning model, it just tries to develop a plan and say, Okay,
you're asking to generate this response. I could use this
example versus this example. Let's try with this example, go
halfway through and realize that there's not enough meat in
this Let's just go back and switch the example. That's
what reasoning is all about, right.
Speaker 2: Yeah, that's what it implies. But what the Anthropic paper
says is that they're actually doing that after the fact.
They already have the statement they want to say, and
then they take their statement apart and fill in the
pieces to show how they quote reasoned it.
Speaker 1: Yes, more like justification than reasoning.
Speaker 2: You showed me the equation. I knew the answer, and
now I write out quote my work even though I've
already got the answer. I'm just trying to fill it
in to make you feel better because you asked me
to show your work.
Speaker 1: Yeah.
Speaker 3: No, that's the anthropic paper, no question about it. But
we are adding on top of that reasoning layer. We say, hey,
which past performance do you want to use? How should
you be writing about it? Things like that.
Speaker 2: Yeah, and I'm not going to say anything bad about
the fact that it goes and poll's references because you
need those. That's all the quality stuff for there.
Speaker 1: Yep, that's fishs. Last time we talked, we've got on
the subject of how you think jen Ai and AI
in general is going to affect the lives of average
developers like us. Has your thoughts on that changed in
the last year.
Speaker 3: My thoughts have not changed. I think developers who will
greatly benefit from tools like that. It is not a panacea.
You can't just say hey, I'm going to give you
these things and go code me something. I think it
puts developers who have been thoughtful, who have worked on
the craft who understand the edge case as well, who
are able to critically look at the code that the
lllms are generating. They're in a very good position because
they can really get that acceleration at the same time
they're critically analyzing what's being generated for them. I think
it's a real challenge for people who are beginning because
it can very superficially tell you that it is working,
but it may not actually be working exactly the way
you want it to, or.
Speaker 1: The AI might give you a solution which is more complex,
and you're building and rolling your own thing when there's
already something available that does it, and it's not going
to tell you that. And I've had this experience and
you go back and you say, hey, can I just
use X? And it says, oh, yes, absolutely.
Speaker 3: Yes, Yeah.
Speaker 1: Well why didn't you tell me that before? Are you idiot? Right?
Speaker 2: Yeah?
Speaker 1: So to your point, somebody who doesn't know about you know,
tool X or tool why isn't going to find out
that it gave you the answer that you wanted, which
is how do I build this? It didn't say you
didn't say how do I solve this problem? And if
you did that, you might have gotten a different answer.
Speaker 3: Well, that's that's very important. It's also important to do
it incrementally and iteratively so that, yeah, you don't just
get two hundred lines of code, you have no idea
and you run a test and you think it's fine.
But if you do it incrementally, all of the basics
of getting this right. Can I refactor this, can I
get can can this piece be optimized? Using this? Yeah,
it's accelerating that immensely, and so.
Speaker 2: You know, it's funny you're describing exactly what you say
p wind does for RFP that experienced RFP builder respond.
Builders can use this tool to accelerate the behavior. But
it is an interaction the tools pulling resources from past
work and suggesting options that then you could iterate on incrementally,
piece by piece, so that you know, when we talked
a year ago this was about making RFPs faster. Now
you're really talking about higher quality and better tuned to
the demands of the new RFP.
Speaker 3: Definitely, definitely, and the quality. We are reaching a point
to where people say I can spot judge it PT
generated text from a mile right, people see that I
don't know if there's.
Speaker 2: Well AI slop is now in the vernacular for a reason.
There's a lot of bad generative AI slop out there.
Speaker 1: I can definitely tell when images are AI created because
they're so slick, you know, and people have made.
Speaker 2: A half he even fingers.
Speaker 1: Yeah, people have made a I don't know, hobby or
maybe a career out of generating. You know, the most peaceful,
sublime picturesque spot with a house by a brook and
you know this and that, and they just post it
on Facebook and say a beautiful place right with no contacts,
with no you know, people are like, wow, where is that? Well,
it's amazing, right, Yeah, it just doesn't exist. And I
could spot those pictures a mile away.
Speaker 2: Let me tell you, as a conference organizer, I know
when you use chat GPT to write your abstract, abstract
at your abstract because they are all the same.
Speaker 3: Yeah.
Speaker 2: I got a thousand abstracts in and four hundred of
them start with in a world. Yeah right, like come on.
Speaker 3: Yeah and b. Because we're so focused on the quality
of writing, we explicitly look for these things where language
models are trying to quote unquote write the perfect text
because you know, you've given them ten things to write
and they find the best way possible to pack that
information and sort of present it. The problem is it
is very onerous on a reader when you're packing it
so much information right, and then overuse of cliches, making
statements without sort of providing background information. These are all
common sense of LLLM generated writing, and we take explicit
steps in our engine to either correct this mistake or
constantly remind the model to slow it down to make
it readable, to improve the quality. These are all things
that are works in progress. A lot more work to
be done there, but those are the kinds of things
that we're thinking about. Goes back to all of these
things are needed to have a real impact on solving
a business problem. When you think about this, it's not
just a matter of taking this and saying, Okay, I'm
going to go start generating financial reports tomorrow. There'll be
a lot of work that needed to go in.
Speaker 1: As a programmer, you kind of think of yourself as
a senior programmer slash manager, somebody who can write code
but chooses to offload the boring stuff to a junior programmer.
And it's all about the prompts, you know, It's all
about what you ask it to do. And I've had
the experience of chatchypt anyway where I said, you know,
I'm trying to do this, and it might be about programming,
it might not, and it will say, oh, well there's
currently no way to do this and blah blah blah,
this is not possible, and or maybe it gives you
some suggestions that are just stupid, like is it turned on?
You know, that kind of stuff, And then you can say, well,
is any can you check the internet to see if
anyone else is having this problem? And that is the
most powerful prompt you know. If you're not satisfied with
the answer it gives you just say, hey, goog google
it or whatever you know, and it will come back
with sometimes with yes, there's a somebody's having this product.
It seems like this is a common problem and this
is what this people, these people did to overcome it, right, absolutely,
So it's it's all about the prompts and it's all
about thinking in that sort of manager senior programmer to
junior programmer mindset.
Speaker 2: Yep. But back to you. You know the p win tool.
You're mostly writing prompts with the tool to then run
them to get results for the operator, that's right, that's right,
And because there is a repetitive piece to this of
setting the restrictions and making sure the context is correct.
Like I got to imagine there's a couple of big
paragraphs at the top of every prompt just to make
sure it stays in the constraint, and then specifics for
the section you're in in the RFP.
Speaker 3: YEP, there's some systems prompts that go in and every
time to sort of keep.
Speaker 2: And it occurred to me, you know, thinking about this
problem space and thinking about in our reactions to AI
SLOP and you know, detechnics and so forth. It's like,
if you think this pattern matching software is awesome, try
a human because humans are really good at finding patterns, especially,
and that's what they see. There's certain patterns to what
a lot of low quality l M tools generate that
you immediately perceived. And it's now starting to shift our
minds where it's now repulsive. Yeah, and we're in a
funny place, but it's there's some good advice in here
visual so I really appreciate it. Like a year later,
you're speaking very differently about the product you're making and
the tool and the way to use these tools like
it shows that it's the evolution that's going on here.
Speaker 3: Yeah, I know, not Richard and call that's that's certainly true.
Learned a lot in this process of awesome technology. There's
no question about it. I think we'll be it's the
next super cycle, as Gartner Le likes to call. They said,
you know, every tech super cyclist twenty years. The last
one started in two thousand, but digital and the Internet
and we are at the We're at the intersection of
the previous one and launching into the gen AI.
Speaker 2: Yeah. Yeah, this has been I felt like this is
a fundamental shift in U acts.
Speaker 3: It's a fundamental shift.
Speaker 1: The way we interact with our equipment is about to change.
But I think this generation isn't going to be twenty years.
It's probably gonna be more like five or six or ten.
Speaker 3: True's that's that's so.
Speaker 1: Much has happened in the last.
Speaker 2: A lot happened in the first few years of the
twenty first century as Internet came out of the dot
com boom, and you know, we shifted to mobile.
Speaker 1: It's not like we've been going slow so far, no,
but it does seem to be accelerating.
Speaker 3: I think, Yeah, I know, it seems to be accelerating.
But also I would say that there is some platauing,
which you may see counterintuitive when I say that plat
doing because GBT five was not released. They were going
to release GPT five. Right, Just a bigger and bigger
model with more data is not leading to better results.
Speaker 2: Yeah, the last few years we've pretty much consumed the
whole Internet into tokenization. There isn't more Internet to grab,
so that that's not an exponential function, Like we're kind
of addicted to the exponential function, but it only existed
because a small group of people worked extremely hard to
increase the density of silicon substrates. That's the only thing
they you know, most everything else doesn't work that way,
and this doesn't because there's only so much data. And
it's very clear that bringing in more data generated by
AI for this is like taking a photocopy of a photocopy.
It's degenerative, it's not useful data.
Speaker 1: Well, I can't wait to talk to you next year
to see what's new and see what else you've learned. Yeah,
it's fantastic. Thank you, Vishwaz. It's always very talking to you.
Speaker 3: Always a pleasure to be on this show. Thank you
for inviting me again.
Speaker 1: Well, thank you and thanks for listening, and we'll talk
to you next time on dot net rocks. Dot net
Rocks is brought to you by Franklin's Net and produced
by Pop Studios, a full service audio, video and post
production facility located physically in New London, Connecticut, and of
course in the cloud online at pwop dot com.
Speaker 4: Visit our website at d O T N E t
R O c k S dot com for RSS feeds, downloads,
mobile apps, comments, and access to the full archives going
back to show number one, recorded in September two thousand
and two.
Speaker 1: And make sure you check out our sponsors. They keep
us in business. Now go write some code. See you
next time you got and Daddy see a summer time
that means his home, then my Texas in line vegetable