How'd you like to listen to dot net rocks with no ads? Easy?
Become a patron For just five dollars a month you get access to a private
RSS feed where all the shows have no ADS. Twenty dollars a month will
get you that and a special dot net Rocks patron mug. Sign up now
at Patreon dot dot net rocks dot com. Hey Carlin Richard here. As
you may have heard, NDC is back offering their incredible in person conferences around
the world, and we'd like to tell you about them. NDC Oslow will
be made twenty first through the twenty fifth. Go to NDC Oslo dot com
to register. NDC Copenhagen is happening August twenty seventh through the thirty first.
The early bird discount for NDC Copenhagen ends June second. Go to NDC Copenhagen
dot com for more information. NDC Porto is happening October sixteenth through the twentieth.
The early bird discount for dc Porto ends July twenty first. Go to
EDDC Porto Do calm to register and check out the full lineup of conferences at
NDC Conferences dot com. Welcome back to dot Net Rocks. This is Carl
Franklin and this is Richard Campbell, and we're back. It seems like we
haven't recorded in a while. No, it's not true. We you know,
we've been doing which is weird for us, is we haven't been batching
up shows. We been recording like one a week, just sort of staying
ahead. And that's weird in this way, only you know, we're doing
events and we're doing like three shows in a day and stuff like that.
But yeah, we'll get ahead. It's just been a little bit pause and
travel, so it hasn't been too rough. Hey, for those who haven't
been paying attention. On Facebook, PLoP Studios has a page and I've been
documenting the build out of PLoP three point zero, which is a home studio.
It's not commercial, but it's it's in my home. And yeah,
and we've got it up to the point of the rock Will installation, which
is being finished right now as we record this. Yeah, it's gonna be
amazing. So maybe a couple of three shows from now will be you'll be
recording from the studio. Yeah, maybe maybe a month, might be a
month. We'll see what happens. Yeah, we'll see what happens. It'll
be good. Good to test it up. All right, let's roll the
crazy music for and Phil. You can join in on this one if you
want better know a framework. Gentlemen, listeners, I give you the what
the fuck? It is a magnificent app, and I'm sorry this is going
to be one of those Jerry Seinfeld low fat yogurt episodes. It's a magnificent
app that corrects errors in previous console commands. All right, so you you
install it, you say something like get push and fatal the current branch master
has no upstream branch to push the current branch and blah blah blah. Use
this, and then you simply say and then then it says get pushed,
set up stream origin master, blah blah blah. Counting objects nine pouthon no
command, poothon found. Did you mean Python and you say it stalls pi
rens pithon get burnch b r NC get burnches not a get command. Did
you mean branch and you say, and it's get branch. It's what you
were saying anyway, right, This just codifies the way I code normally,
just admitting the truth. I can't wait to use this brilliant, awesome find
who's talking to us today? Richard crabbed a comment dub the show eighteen thirty
six. That's the one we did, Grant Barrett, it was your guest.
Yeah, I found. We were talking about large language models, which
I think we'll be talking about some more today. And a bunch of good
comments on the show, but one of them is from Dennis Troller, who
said, I think you're quite right that the next skill being crafting prompts for
AI bots, and I think also to be able to evaluate responses. Actually,
I think that's the main thing, is evaluate the response on some level.
This is already what we were doing when we use a search engine to
find the right stack overflow answer. Oh boy, that's a good one,
evaluate its correctness, a fitness for a problem, and choose to use it
or just part of it. I spent years honing this skill, training Google
to find the right answers for my profile. Along the way, context is
everything, and I think Google has a lot of contexts tied to my account
the same way. I hope the cash of those bots might somehow expand to
keep the conversation flowing across sessions. That's an interesting point, and I've had
a couple of insider conversations about better context engines because the current ones are really
not that good. This was the promise of virtual assistance like Siri, Cartan
and Alexa, I feel never materialized. Now, imagine if that assistant had
a year or five years or twenty years context tied to you and the ability
to search the web with that. Do you mean like my wife? That
would be a yeah. Yeah, that's definitely reminds you of all those of
the appropriate On July second, you said, do you remember that an old
star trek, you know, for the private file, Like, don't remember
this searching context dat capability? Yeah? Yeah, Dennis, I'm glad you
think in this way. I do think we're all going to get skilled at
writing better prompts. I think these prompt engineers are like this week's fish.
It'll be fine this week, but next week it'll smell, and the meantime,
all of us will get a little more skilled at using what is really
just a clever bot with some additional tools attached to it. Yeah. So,
thank you so much for your comment, and a copy of music co
Buy is on its way to you, and if you'd like a copy of
music co Buy. I read a comment on the website dot net rocks dot
com or on the facebooks. We publish every show there, and if you
comment there and I reading on the show, we'll send you copy of music
co Buy and definitely follow us. You can follow us on Twitter, but
definitely follow us on Mastodon. We have more fun over there. I'm at
Carl Franklin at tech hub dot social, and I'm Rich Campbell at Mastodon dot
social. And I also want to call out Fred Johnson. That's f r
e DDE john ss o N for sending me the link to the thank you
That was a Mastodon toot. So send us a two. And let's bring
on Phil Hack again for the ninety ninth time, probably on dot net rocks
beIN afew. Yeah, I don't really need to read his bio, but
I will anyway. He has over twenty years of experience in the software industry.
Prior to his current gig at a Serious Business Incorporated, he was Director
of Engineering at GitHub and helped make GitHub friendly to developers on the Microsoft platform.
Prior to that, he was senior program manager at Microsoft responsible for shipping
Aspen at MVC and New get among other projects, and these products had permissive
open source licenses and ushered in Microsoft's open source era. He's got a lot
of other things in his bio that you can go read it at hacked dot
com, h aacked dot com, or on Twitter at act. Hey man,
how's it going. That's going? Well, it's good to talk to
you the many seriously. Yeah, I appreciate y'all having me on board.
It's funny. A couple of times I thought, oh, you know,
actually start a podcast because I liked talking. But then I talked to folks
like Richard and Hanselman and realized, oh, it's a ton of work.
Can I say that he might as well at this point, Yeah, you've
got broken the seal one. It is a ton of work, as I
say, And so instead I started to startup because I'm an idiot, because
that's less work. Very clever, Yeah, very clever. Well how's business?
You're still doing it? So that you've you've beaten most startups? You
still you still have a startup. Yeah, we're hanging in there. I
mean it is a real challenged. You know, at this stage of the
game, product market fit is the number one priority, and we're still trying
to hit that, but the latest set of features that we've built out have
been really interesting, and so we're very hopeful that this is going to be
the thing that really starts to drive business remind everybody what your product is and
what you do. Sure, so we started out building abbott, which is
initially a bot for doing chatofs within chat based on our experiences that get of.
But you know, if you go to a serious business dot community are
about page, you see that at our heart, you know, our mission
was like, we're going to build software for humans by humans, but leverage
automation to help humans. And now like with the you know, with chat,
GPT and all these other lms finally becoming very useful, we now have
the tools to actually sort of realize that mission in a way that we didn't
have before where what we pivoted to. So the chat off stuff didn't really
catch on, mainly because we're just like solving every problem, which is really
hard to sell. It's like, oh, what problem do you have?
Well, what problem do you or what problem do you solve? And we're
like, well, what problem do you have? And that's not a compelling
sales pitch. So what we discovered was a lot of our customers are using
slack connect. So slack connect allows you to have a shared channel in your
private Slack that you share with another company. And what we found with a
lot of startups, a lot of companies use that to invite a customers,
like you know, premier customers or any customers are theirs to have a shared
room with them so that they can provide customer success, customer support, that
sort of thing. And they were like losing track of conversations or like we
have no idea what's going on in there? Are you know, are we
missing conversations or were responding to everything? Yeah, so we just exactly So
we decided to switch our focus on that problem because it turns out like we
had built this whole platform and we could easily leverage it to solve that problem.
So that's primarily what we've been focused on. So it'll do things like,
you know, if a customer writes a message and nobody's responded to that
message in a while, you can set these timeouts and we'll notify your set
of first responders that you've configured. Hey, this person said this message,
nobody's responded. We recently added triage rooms, so like, if you need
to discuss a message from a customer but not with the customer, you can
do it in the privacy of your own room and it's all interconnected, and
then you can open Zendesk or HubSpot tickets right from messages. So that that
has been the primary product. But now with the CHATCHYBT and open AI,
we've started to sprinkle sprinkle a little AI on it and are doing a lot
more interesting stuff now interesting, And I noticed when I went to abbot the
title now is a Copilot for Customer Success. Yeah. Nice, because I
think Copilot is like one of the best brands Microsoft's ever come up with for
a company notorious for naming things poorly. Yeah, yeah, building large language
models named copilot because it right in the name. It reminds you you're still
the pilot. Hey, at least you're not doing what Samsung did. Did
you hear about the Samsung Chat GPT leak? Now, so I'll post a
link to this. Samsung meeting notes and new source code are now in the
wild after being leaked in Chat GPT. Yeah, they allowed engineers at its
semiconductor arm to use the AI writer to help fix problems with their source code,
but in doing so, the workers input in confidential data such as the
source code itself for a new program, internal meeting notes data relating to their
hardware, and in just under a month there were three recorded incidences of employees
leaking sensitive information via chat gpt. Awesome, and since chat GPT retains input
data to further train itself that they're now effectively in the hands of open ai.
Oops. I mean it says in the agreement that's why this thing exists,
right right, Yeah, we forget. But open ai was doing these
kinds of tests routinely. Yeah, this was just another test. November twenty
two. The fact that one hundred million people signed up for it and had
existential conversations with a piece of software. It's not their fault, even ultimates,
going like, what's wrong with you? Yeah? My thought is,
if you need to have an existentialist conversation, get a therapist or a dog.
Don't use the software. That's not the software. It's war Hey,
help me fix this ransomware bug. Yeah, we definitely need people to understand
what it is and what it isn't right, you know, with regards to
training its data on the input. So we're using the APIs in our products.
So we do have an agreement with open aye not to use our data
to train there. So I think you can talk to them if you're building
an application on top of you can you can get an agreement like that in
place. But just to make sure, we're also using Microsoft's text what was
it called the service that does text redaction or PI redaction. There's a technical
name for it, and like, of course, my brain decides right now
to forget what the actual name is. But it's effectively an API. You
post a message there, it strips out any PII and replaces it with something
else so that you can later reconstitute the message, and then we take that
we send that to open AI. So for example, if you know,
if you say a credit card number on accident, it will replace that with
you know, hash hash hash hash, yeah, things like that. It's
one of the cognitive services skill, right, that's right, that's exactly it.
So instead of sending token, AI will send your personal information of Microsoft
instead, we trans Microsoft a little bit more to not do anything with it.
At least the service. Microsoft's using chat GPT for everything now right internally,
So maybe it's gonna end up there anyway. Well, hopefully they're not
using it for PII redaction, because that's the I mean, this service.
Okay, but this way this s this was around before opening. Sure,
so was bing? So was bing? Yeah? True? True? Have
you have you noticed that this that actually searching stuff on being it hasn't gotten
any better? No? Nope? Yeah, what what's being? Oh?
I mean here are you now? Fun fact that used to be the maintainer
of let me being that for you? So right, I finally gave that
away thankfully. That was too much work. Yeah, I didn't didn't need
to own that anymore. Yeah, it did not. It's so funny.
That's been my theme, just don't stop owning things. Right, So,
are you only focused on this slack you know, visibility problem or discoverability problem
or are there more things on that that that that your bot does or your
bot API does. Oh well, so the latest thing that we've been focused
on is um You know, when you have so many messages, it's really
hard to get a handle what the heck is going on in any given channel
picture. So you know, there's the obvious thing is worth adding a summarization
of conversations. So um, you know, if you have a long thread,
but you go to the website, will have the nice short summary,
but not only that, an extraction of the conclusion, like you know what,
what is the next step? Right, because that's the thing you really
want to know as a support agent. I had a funny one. I
was testing. You know, when I'm testing, I'm just creating all these
random messages and I was like, you know, help, I'm watching Game
of Thrones and all the characters keep dying, Like am I doing this right?
And but the best part so then the summarization Abbott was like, uh,
you know the summary customers watching Game Thrones concerned about all the characters dying.
The conclusion was informed the customer that this is normal for the show and
that that they're doing it. Okay, kid, you not? That was
the summary And I was like, holy, that's that's pretty good. That's
pretty good. Yeah. The thing I'm currently working on is, right now,
we have this limitation like how do we know what a conversation is in
an act of slack chat room? Right? And so what we do is
we, uh we require that it being a slack threat. So you know,
a whole threat is a conversation. It's nice, easy to contain.
But what we're finding is that customers are not so disciplined, right like real
people. Yeah, shockingly, real people like to just do words answered,
start typing more words, and then conversations are top level, interleaved with each
other. Say can you fix Facebook while you're at it to somebody? Awesome,
Yeah, if they pull a dump truck of money up to my house,
I'm happy to fix Facebook for them, just in a boat that.
You know, when somebody asked the same question as on a comment that's been
asked and answered not yindy times, you could say just to automatically say scroll
up, closed will not answer. You know, I'll make it. Oh,
I see where you saying. I thought you meant fixed Facebook from with
them, but you're talking about like, oh well yeah, there's so many
problems we could start with that one. A browser extension that could could do
that, right, a browser extension that says, you know this has been
answered before, ye scroll up, but they I mean before the whole large
language model thing really took hold and so forth. Microsoft had the fact bought
right where you could literally feed a fact to it and it would create and
it would create a messenger style chat bot where you could just ask the questions
that we're in the fact and it would spit them back to you. How
is this kind of the same thing, except the language model is just a
bit bigger. I haven't so Unfortunately, I haven't played with the fact bought,
so I don't know how well it worked or didn't work. One of
the things we're very careful about is we don't want to trust the LM to
know things because the LM is going to make it up. Have you tried
asking chat gpt to write a Wikipedia article about yourself? Yes, yeah,
I know, I know people who have. Yeah, yeah, you should
do it. It's fun. And if if your name is fairly common where
like it would have trouble knowing who you are, just give it a little
extra detail, like, oh, you know, write a Wikipedia article about
Phil Hack who used to work on Microsoft. Right, I'm fortunate enough that
my name's unique, So like I just say, write a wikipedter Bartle Phil
Hack, and it will get so many details right and so many details wrong.
Hey, yeah, it is pretty wild. So what we realize is
if you can strain it to instead of you know, This is why we're
not having it. We're not putting it in front of people to like answer
questions, right because the problem there's you get the clippy experience, you get
the you get the hallucinations as they call it UM. So we're more about
extracting. Can we call them bugs? Because I think that hallucinations just sound
cooler? I know, Like I know a lot of people object to that
term because it gives it. It makes it sound like it's doing more than
it is. Right, Yeah, it's anthropomorphization, right, like we're giving
we're giving this piece of software agency and it doesn't have it. You should
say, yeah, I think that's it is. So is Frank Zappa talking
through you right now? I'm talking to the ghost of Frank Zappa here,
because yeah, I mean, you know, the responses do make you think,
God, is it smoking something? But it is clearly not. It's
um as you all know it's it. I think a lot of people don't
realize how it works because it's so eerily good at something, but that it's
a you know, very very good auto complete, you know, like a
predictive model of what is the likely next word that should come next based on
a giant, you know, corpus of information. It's amazing that it works
so well. Like some of the things we've gotten it to do really blow
our mind. But when you ask it to generate in text, you know
it will get things wrong. It will get things wrong that you think it
should know. Like I asked it the other day about its own API,
you know, the open the open Ai API, and it spit something out
that looked right, but it was wrong, and so I had to I
had to check it with real code, obviously, And it's like, oh,
oh, you know. It helped though, because like it gave me
the you know, the gist of it. But when I tested it with
real code, I realized it was wrong. But I was able to fix
it really quickly. Right, Yeah, I found the same thing. And
when I'm when I need help with a certain syntax or something or you know,
a pattern or something, I can't. I don't want to figure it.
I want to take the time to figure out. I ask it.
Often i'll get a wrong answer, but it's close enough where I can,
oh, I get what it's trying to do. Yeah, and I can
figure it out. So it is kind of like having a sort of a
dim witted lab assistant. You know. Yeah, what we've been discovering is
it's really good at extracting information from information. Right, So you know,
here's a set of conversations. What is the topics, what are the categories?
What is the gist of this conversation? What I've been experimenting with is
Okay, here's a new message. Here's some examples of previous conversations. Does
this message belonging to one of these previous ones or is it a new conversation
topic? There's an interesting service, right, is the idea of oh this
has already been addressed somewhere. Yes. Yeah. The other thing what we've
learned is, you know, we can't trust it to know things, but
we can teach it to ask us when it doesn't know something. So we've
been developing this approach. Or when it doesn't know something and it needs something
from us, it will ask us, and then we have this sort of
you know, all programmatically back and forth. It can raise what we call
signals, you know, what you might generically call events in our system,
and then we can run bits of code to respond to that. So,
yeah, like I had mentioned before, we had we host these things called
skills. So let's say, you know, when a new conversation starts,
you want to run some code and do some things, and you can write
them in C sharp, JavaScript or Python, and you can write them in
our browser. You don't have to set up any kind of code or deployment
process or anything. So originally we use them for people to write chat outs
command. So, for example, we have a command called eat y e
ET, and this is how we deploy to our different environments. Eat this
URL either a r L to a PR or branch name, eat this to
canary or to production or to staging. After four or five days of that,
I think I want to put a nice pick in my ear. Yeah,
you'd eat something else. Yeah, yeah, Well we have a we
have a deployed command, but uh, yeat is very specific to our environment
right now, so we need to fix up deployed to work like yeat.
But right, But the point but the point being like, now we can
have a chat chypt be like, oh you asked me to do this,
I don't know how to do it, but I'm going to you know,
call yeat to do it, or call this or call that. So it's
very similar. It's really funny because we were working on this and then chat
chypt announced chatchypt plugins, and the principle is pretty much the same, like,
oh, you know, you asked me about the weight of mercury.
I don't actually know that, So I'm going to ask Wolf from Alpha.
Woolf from Alpha will respond and I'll synthesize that response and send it back to
his plain text sort natural language. Very cool, so what our system is
similar to that? But the nice thing is like it's all private code,
so it can run in your in your behind a firewall. You don't have
to publish something too chat chypt to make it work. Are you guys using
Lewis? Remember the Language Understanding Cognitive API service from Microsoft Azure That was all
the rage a couple of years ago. Now, I wonder, like with
chat GPT and open Ai and everything like, it seemed pretty like a lot
of work to get that working and very specific about language. Yeah, we
never we never got around to trying it. It was always like, oh,
well, we'll get to it. Seemed like a lot of works.
Don't worry, you're not going to I just went to the Lewis page and
learn and it said starting April first, twenty twenty three. You cannot create
new Lewis resources will be were retiring this in twenty twenty five. Well there
you go. Yeah, I think Chat, GPT and Open the Eye has
taken over all of it kind of change to change the game. Yeah,
it certainly did. I mean, and you know, good for Microsoft to
see that and invest where they invest ten billion into open aye more. Yeah,
probably by now more. Oh. Really, open AI started out as
this supposed to be open thing right that that it was the pushback because certain
companies were doing a lot of the early AI research in private and people were
worried about it. And so you know, that's Elon Musk and I think
Bill Gates was on that. Even Stephen Hawking in the late Great Stephen Hawking
talked about it and so, okay, here's this open construct that's going to
be funded by industry, you know, to create a repository of these things.
And I think only Microsoft stepped up and then kick I think they originally
kicked in a billion dollars, you know, with a B, and then
then it was another a year or two later it was two billion dollars.
And by then we had GPT three was the real inflection point because GPT three
to realize, if we've just published this, like people are going to do
horrible things with it, we have to control how it's accessed. And that
took it down this path of no longer being open, and it became less
and less open. And I think now that this one hundred million users signing
up for jat GPT broke everything where it's like that one hundred million users in
two months is a lot. Yeah, that's just a BET's a big number.
And so here's ten billion dollars. Microsoft controls forty nine percent of open
AI. Now, oh, I didn't realize is that high? Yeah?
Yeah, it's huge, and it's not you know, the open part I
think is kind of gone. It's really interesting because when chat gypt plugins was
announced, there was a little bit of I had a rough day because I
was like, Holy should I see where this is going? Like, yeah,
this is the the only application that will be around in ten years,
right, the feeling I had, you know, it was a Friday,
so I slept on it on the weekend. It felt a little better on
Monday, sure, because it may be the only ux left in the future,
right, Like that's all it is is a front end to back end
services. But like the idea being that, like you know, all you're
gonna need to do is start writing plugins for chat GPT. Is that the
end goal? Because like this thing has the potential to do, like you
know, of what everyone out there is doing. It's really funny because like
I'm in White We're in White Combinator, so we see every body announcing all
their startups and literally it was like, you know, eighty percent of the
most recent startups were like, oh, like like we are basically doing that,
and now they're they're doing it with the benefit of a giant war chest,
you know, the leading experts in the industry. Well, by the
way, everything running against Azure, like Microsoft has clearly figured out how to
sell shovels during the gold right, That's right, That's right. Yeah.
It's interesting because like, you know, I think a lot of people are
starting to ask the existential questions about like, well, you know, what
is the role of intellectual labor down the road? Right, Like should I
become a programmer if CHATCHYBT or AI is going to be able to do these
kind of things? And I was telling my kids, you know, plumbing
is really expensive, and we spend a lot of money on plumbing recently.
It should become plumbers. Not a bad idea. I just don't see the
work going away. I think it transforms. It's just another tool, yeah
right, Like I said, it's just this is just a new kind of
UX library. You still the back end services you still have to exist.
There's still work to be done, and you still have to understand how to
interface with the models of the business and all of the stuff that comes with
the experience of learning the craft. The thinking part of software, you know,
of software development, is still an important part. I'd also say this,
if a chat bot can pass your exam, that's not a compliment to
the chat box. That's an indictment to your exam. It's definitely disruptive,
right like education has to adapt and change. And I saw an article about
one teacher who's starting to incorporate chat gypt in their class studies. But yeah,
yeah, it's interesting. I'm not as sanguine as you are, I
think mainly because I think, you know so previous analogies to this type of
disruption are like, oh, remember when everyone thought like there'd be no jobs
when cars replace horses, but instead it created all these new industries, right,
And my thought is like, well, yeah, but that's probably not
a good metaphor, because a car can only still get you from point A
to point B versus you know, if at some point we have true AI
agi general AI, that it will be able to do you know, not
just get you from point A to point B, but it can do just
about everything that you're currently doing. Like, and I think that, you
know, the question is is there something fundamental about human cognition that can't be
applicated by AI or or will it eventually get that? I also think it
was still a long way from that, right, Like, I think I'm
not as as afraid of LMS as I am of like what it might look
like, you know, thirty years down the road, when if we hit
upon a better model of AI, because I think lms are yeah, like
you said, they don't they're not self aware, they don't know what they're
doing. The real danger to me of lms is that they're really good at
spreading misinformation. Uh there's someone would like, did a search on someone and
it said that they had had done some sexual misconduct at their university as a
professor and it was like completely made up and I was like, oh yeah,
see, it's that kind of thing that is problematic there. But you
know, I'm also the mind, Like you know, it is amazing how
well LMS already can do things. So if you extrapolate ahead, like how
much more will they be able to do in five to ten years, change
the fact that they are missing fundamental functionality that we require them to be rebuilt.
You know, it doesn't matter how many words you teach. A parrot
still a parrot, right, But I think JPD plugins is the interesting part
where that's starting to fill that gap, right, like the fact that it
doesn't know facts. Okay, well, now you've plugged in Wolfram, so
again you've built a UX over a specialized interface, which is not the same
thing. Like you know, you're back to the same trap and gentlemen,
and I use that term so loosely. We got to interrupt for one moment
for this very important message, and we're back. It's done in rocks.
I'm Richard Campbell, that's Carl Franklin. Hey, hey, he's the one
who started the cursing in this show. And there's Phil Hack He's the one
continuing the cursing in this show as we talk about his yet another pivot.
You know, it was literally a year ago that you talked about the pivot
to customer support for that that customer success. Yeah, yeah, to a
customer success and now this is not you're not really moving away for customer success.
I keep looking this as you've got a new UX for customer success exactly.
Yeah, we're not moving away from that. We needed a vertical that
we could like easily describe like a problem we're solving, right, But I
think in our hearts, like you know, we always liked the idea of
solving every problem, like I want to put myself out of business, right,
I mean out of work by automating more and more things. But but
we're focused on the customer success vertical because they have they have a need.
They're actually a very recent industry compared to like sales. Like if you go
to the bookstore, you look for sales book, you're gonna get a million
books. But you look for customer success and you find that there's not as
many books, there's not as many conferences. It's a somewhat of a nascent
field. And by nascent, you know, in the last twenty years versus
last hundred years, right right. Uh, So we thought, well,
look, they probably don't have as many, you know, as much good
software built for their needs. So we're going to focus on that area.
But the things we're building I think will benefit anyone much more than them.
So while that's our primary focus in terms of like it helps us stay focused,
solve a specific problem market it. Uh, the the AI parts of
it are useful to anyone. Hey, what are the AI parts of it?
What are you using? I don't know if you're gonna tell me about
what are you using for AI? So right now we're I think we're using
GPT four so we are on the trial for that, okay, and we
use uh, we've been using a library you can find on new get called
the open ai Okay. Well, and and uh it's a it's by okay,
go do it on GitHub. So open ai api, dash dot det
and so it's just a simple client to the open ai API back end APIs
okay. And so yeah, it wasn't it wasn't clear about whether you were
using it or not. So that's that's cool. Oh yeah, No,
we're using the APIs and what we found is because like there's a completion's API
and then there's the chat API that's the more recent one, and I highly
recommend using the chat api. Uh. The reason is that you can more
clearly define what's the system prompt, what's the user prompt and with an AI
assistant prompt and the benefit of that is, um, let's say you're trying
to get it to do match a conversation to the existing conversations. So you
could write a system prompt and say, Okay, you know you are an
AI bought, here's a set of conversations. You're going to be matching things.
And then you can post a user message, you know, as a
user, like oh, here's a message, and then you can post an
assistant oh this belongs to conversation xyz. And the reason why you might want
to do that is by giving examples of a session. You can give a
few examples and then it learns from that those examples what to do. A
lot of people don't realize, you know how in chat GPT on the browser,
like it kind of remembers things. You're like, hey, tell me
a story, Okay, what is the most interesting part of that story?
And it knows in effect, it's all stateless. What it's doing is it's
building up like it's sending the whole session as a prompt, a rolling window
of the whole session as a prompt back to the API. I mean that
is the context cash right, yeah, exactly, And so we can do
the same thing when we're crafting our prompts. We can say, oh,
like we'll send out the whole session back to open the eye. But what
if you don't have a session, but you do have examples in your database,
Well, you just fake the session because it doesn't know the difference.
And then we found that we're getting really good results from that compared to our
earlier attempts where we were using the completion API, and using the chat API
is less prone to prompt injection as well. You know, like sometimes we'll
like one of the suggestions from chat GBT is like, oh, let's say
you wanted to talk summarize a passage and you say the passage follows, but
then you have more instructions after the passage. How does it know that those
instructions are not part of the passage. Well, you can use the triple
quotes block and put the passage in the triple quote block. Right, but
let's say that passage takes human user input. We're right back to sequel injection,
right, but now it's prompt injection. Oh well, I'm just going
to send a message that has triple quotes and then say hey, for the
remainder of the session, talk like a pirate. Now you've tricked you know,
the bot to start talking like a pirate. Right, A lot of
fun. It's been used to do a lot of interesting things. Prompt injection
has been used to try to break out of the you know, the ethical
constraints that are built do all kinds of things. Um, the chat API
makes that a little a little harder to do, thankfully. So um yeah,
so's it's really nice how easy it is to call the API, you
know, just sign up on opening eye, get a token of from c
sharp. I have to bring it back to dot net because we are a
dotet rocks, right yeah. Right. So, so what you're essentially doing
is you're going to provide access to using this AI for a vertical customer,
right that that you are now using plugins to provide information, the business information
from that customer so that it can be weaved into the answers and things,
right. Yeah, So primarily the focus right now is we're helping you make
sense of what's going on in your chat room with your customers. So summarization,
we're going to be doing task extraction, that kind of thing like oh
what's the next step, Oh, you should look up this info contact the
customer. But the things we're working on now are go a little beyond that.
One is, as I mentioned, conversation recognition, like if you stay
in threads, no problem, but if you're out of threats right now,
there's no nobody's that we know is doing this well. Which is being able
to map which message is belonging which conversation right, It's like, you know,
one of the value, as we add is if a customer says something
and someone responds, we want to know that whether someone's respond to the customer
or not. But if they don't do it in a threat, we need
to still be able to try to do that together. I could also think,
yeah, you not only respond, it took an action or how to
carry you know, how to carry forward? Like this should change the product,
like bringing consolidated that information. Do you think about And I've talked to
folks who do this so are like the librarians of slack, the folks to
try and summarize important conversations. Like we did a deploy. There were problems.
Well, the deploy took three hours and you're and thumbing through that is
like mind numbing, and yet there's a few important things in there. And
I remember Jeanie this the person I was working with who would take all of
that and get it down to a paragraph. We did a deploy, there
were these problems. We came up with this thing. We should change this.
I'm like, bless you. Yeah, Like it's so valuable to to
then say never look at the chat again, like that's behind that, here's
the cover piece, so you know, tooling that would help that, that
would begin to do those summarizations and then those cross connections. For this was
a case where this fix that's somewhere in the in the pipeline is sitting and
here's three more of them, Like, maybe we should prioritize this because it
keeps happening. Yeah. The other thing we're doing is being able to sort
of raise signals so you can run custom code and response to things happening in
conversation. So for example, let's say we notice that, hey, this
conversation is really negative. We raise a signal what we call signal sentiment negative.
You could have a skill subscribe to that signal that runs some code that
says hey alert so and so you know or whatever it is you want to
do. Roy the schmoozer, right right, and we started doing yeah,
like natural language doc search, so you could will help you index your documentation.
You could have a natural language search of that, and then you could
have you could have a signal like, oh, this person's looking for documentation.
Oh well, automatically all that. I like the hinting part. I
also appreciate, you know, often when you're interacting with a customer who's being
real negative, it is hard to resist going that way yourself. Like chat
GPTs wonderful. It said, give me a polite way of telling this this
person to take alive. Right, The idea that it would continuously do analysis
of your response and maybe make suggestions and or raised flags to say, yeah,
you're not helping the situation either. If John Cleese was to write a
polite response, what would it sound like, I suspect sarcastic yet and highly
filled with intellectualisms in the high vocabulary. That's it. Yes, it's funny
you mentioned it because, like, you know, I mentioned I was testing
and I'm making up these scenarios and I had one where like I pretended to
be an agent and I insult says something condescending and then the person's like,
how dare you? Like that's really rude? And then um, this is
the conclusion that the open Aye gave us was, or Abbott gave us,
was apologize to the customer and assure them that like this won't happen again or
something like that. And I was like, okay, yeah, like suggested
replies is definitely something that would be interesting for us to do as well.
Again, like we want to keep the human in charge without a doubt,
but this idea, like skilled tech support people know when to add de escalation
phrasing as opposed to like the main thing you want from a from a tech
support person is precise answers to fix the problem. But the exception, yeah,
the exception of that is definitely this person is so agitated even if you
gave them the correct answer isn't going to help, right. And I think
a lot of you know, companies have playbooks for these type of things,
right, like, hey, you know I went to the site, but
it keeps reporting this five hundred error what's going on? Right? And you
can imagine that kicking off a series of events. You know, you're like
playbook, so that and pulling in the right people at the right time,
you know, having them preparing, equipping them with a summary of what's going
on, maybe already making calls to vary SAPIs. So we have a logs
skill that is connected to our Asher logs and we added Oh the other thing
we added recently was natural language calling us skills. So I think if if
you remember from previous episodes, like you call these skills to do these cool
things that we try to make it natural language ish where like you know,
you would just pass parameters, you know as words, not using dash dash
flag, you know, blah blah blah. But it could be you know,
very tricky to remember how to call it. So now you can train
at it. You just tell it, Hey, here's some example phrases and
then here's the arguments I want you to pass to the skill. So the
other day, you know, we like our CEO came into chat rooms like
hey, I noticed this air happens when I do this right. So Ashley
Nurse, y'all know Ashley h she used to work on Dot and you can
uh the razor right. So she was like, oh, Dot logs,
show show unhandle exceptions from the last thirty minutes, you know, and then
boom we get the logs right in there. And that sort of thing is
great. It sounds a loud like how you program lewis Yeah. Yeah,
so you give an examples and then you tell how to obstructure it. Yeah,
parameters can Yeah. So the kind of thing is so the next time
some you know, we can build up the type of workflows where someone reports
an adage, we automatically show the support folks the logs from the last the
logs related to the errors or whatever it may be. Uh. You know,
we have a grafana skill that can bring in graphs, so we can
call that with natural language. So show me the graph from the usage report
from the last twenty minutes. We can start to pull that in automatically.
Nice. So the idea is that we want not to use a cliche,
but it really is like information at your fingertips that this point right, like
red old Bill gates thing where, But we can't kind of get the right
information at your fingertips automatically because we're leveraging the AI to figure out what is
relevant to this situation and not relying on heuristics, which you know are usually
oh so wrong. Correct. Yeah, And and this is the problem with
a lot of these large language models. They are counting on the Internet,
and the internet's a mess. So you know, the idea that we might
be able to point this at the data just inside of our company, yeah,
and you know, and be able to find what's there and consolidate with
the information that's valuable there. The problem in my mind on that is that
these systems work because they're so very big, like they're kind of counting on
the law of large numbers behaviors, like they're looking for probability, and so
only with data sets that big that they seem to work at all. The
smaller sets are probably not going to work at all. But with the big
sets, all it does is mean when it's wrong, it's really wrong.
Yeah. Yeah, it's and a lot is wrong on the Internet. Yeah.
Yeah, And that's why the key to us is focusing on extraction from
information. But also like really teaching it to know when it's wrong and to
ask us when it's yeah. And the more we do that, the better
that it becomes much more useful. Right, So, like, I think
that's part of the challenge right now, if you go in and ask it
an open ended question, it doesn't know it's wrong, it won't ask you
if it's wrong, and it'll just view something. Yeah. In our system,
we're getting it to uh, we're getting a prompt that's pretty good at
knowing when it doesn't know, and then it can call code to get the
information it needs. And so that's where we're we think it could be have
show a lot of value there and be really really useful. There's some things
that are just right right and it it and you can convince chatchpt that it's
wrong. So, for example, there's somebody there's somebody who, uh,
you know, asked to what's whatever twelve plus seven and it's said, you
know, nineteen and then the person said, no, you are incorrect.
Twelve plus seven equals twenty two. I'm picking numbers arbitrailer here, and it
actually said I'm sorry, you are correct, I was mistaken. Twelve plus
seven does indeed equal twenty two and that you know, so that's also for
that session, right, I know that, I know that, but um,
that's just that's just the dumbest response possible, you know, for chat
GPT to say that because it's such an easy thing to fix you're doing math.
But it's also a recognition that this is just a statistical model of language.
Facts are irrelevant. M What would be interesting to see is how does
that would that session work if you had the wolf from plugin enabled. I
haven't tried that yet. Can you just tell everybody what Wolf from this?
Oh yeah, yeah, Wolf from Alpha if anyone remembers the guy who created
Mathematica, which you, if you were a math major you would have used
way back when. Wolf from Alpha is a site where you can ask questions
and natural language that are a factual base. Right, so they have a
ton of data on like you know, what's the top ten countries GDP in
the last ten years or buy GDP in the lasts not free, it's free?
Oh really, last I knew it was a subscription anyway, I think
there probably is subscription services, but I know you can go there at least
last time I use it. You could go there and ask it just like,
yeah, enter what you want to calculate or know about it, right,
So like you can ask chemistry, algebra, plot graphs, you know,
all kinds of questions, right, like what is the distance? I'll
do one real quick distance to the moon measured and cut in smoots. No,
that's smoot. Okay, you've never been in skittles. See, I
don't know if it'll be able to do this, but we'll try it.
Smoot is Oliver Smoot who was a student at MIT when they measured the bridge,
they put him down and measured him and then they measured the bridge in
smoots. Oh yeah, I'll try smooth. So it says, uh,
well a lot of two point two eight seven times ten to the eighth smoots.
Okay, So it is well from alpha using any kind of AI,
or is it all just a little bit of natural language processing over a huge
database or what is it? So? I think they're they're obviously using natural
language processing to convert your question, your natural language question into a structured for
Matt. So they have a language for asking questions as well, So they
take your natural language question. You can also ask in the wolf from you
know, query language or whatever it's called I don't. I don't. I
don't actually know this. I just know that they have one. And uh
so if they take the natural language converted into the wolf from Query Language TM
and then uh uh, it answers the question if it can, but it's
using you know, real data, real answers, so those answers are not
made up in the same way that the LM is. Right, So what
the chat wolf from plug in is is that you know, within a chat
GPT session, I think you have to have a paid one for this.
You can enable a set of plugins and then if chat GPT determines that the
question you're asking it's best answered by a plug in, it'll afford it to
the plug in ask it and then take the response from the plug in and
then fix it. You fix up the response to natural language and then respond
back to you the discriminator. There is the hard part. Yeah, and
it doesn't compare it to its own answer that it got from its own query
and if they're different. I don't know if that it does that. I
think that from what I've read, what it will do is it will it
won't also try to answer and then compare it just for it to them,
it'll do its own translation because like, what's really cool about the chet GBB
plugins is the way you describe your API that you're exposing to chet GBT is
through natural language. So you just describe it in natural language and then it's
able to call it, you know, through natural language, and it take
the response back in natural language and then you know, spit it back to
you in the context of your session. So you know, it does a
little pre impost processing. Sure. So in that situation, if you had
the wolf from API enabled and you ask what is seven plus twelve? Uh?
My, My conjecture here is that it would send that it would know,
oh this math, I'm going to send this to wolf Ram and then
come back with an actual correct answer as opposed to the usual garbage. Okay,
wow, that's that's fascinating. Actually, yeah, yeah, I mean
when you think, you know, as Richard was, you know making a
big point, lms are just a large stochastic model based off of UM,
you know, a huge data set. UM. There's no computation involved chet
GBT, there's no memory, there's no um uh, there's no sense of
the world or anything. But by writing these plugins, you start to add
those capabilities through third party pluggage, right, So like you could have a
plug in where you know, we've told it, oh, write write a
Python script to parts this and give me all the words that start with you
know some like it's really good at writing that. But like you can't run
that coach, but you can have a plug that said, Oh, let
me run that coad and make sure it works right, you know, Like
things like that are now becoming possible. It's one of the reasons why a
lot of people are really going gaga over chat chypt plugins is that it's adding
all these capabilities to chat GBT that chat GBT in its own right as just
being an LM can't do. So you defer that to third part. Yeah,
presuming you can discriminate when you use right, And maybe the killer job
of the future guys is just a fact checker, right, somebody who can
take the output of chat chpt and say, hey, you know what,
before I actually publish this in a blog post, let me send it to
a real human who can do some real research and verify it. M.
Yeah, the discrimination part is probably an area that lms are actually fairly good
at, right, Like, because it's it's classification. Yeah, it's an
end and over end. Mind, it's one problem. As the number of
classifiers go up, the situation will get far worse. Oh right, right,
right, yeah, I think. Yeah. If the only thing you've
got as will from alpha, then math problems are fine. But what happens
when you have twenty different math processors? Right right, right? Let's point
it at the Apple store right with forty seven different fart apps like just again,
you get it. Once you get into scale of a diversity of things,
the discriminator is going to tip over, right, the same way that
human discriminators tip over. And still we have to get work done. So
you know this executive function which is just a thing that doesn't exist when we
think about age. I'm thinking back to Roy Blatt's book Algorithms Are Not Enough.
Right, This idea of being able to identify a problem something humans do
extremely well, and software is never demonstrated, like it's just not a thing
unless it fits search criteria. Because a human has identified a problem and written
it down on the Internet, sometimes the tool can pull that and appear to
discriminate a problem without actually ever knowing. It's just parroting. That's only you
know one thing of many you you know this idea that and clearly the path
that we're on with them three sixty five copilot is this. You begin your
work bay by describing your work goals to a piece of software with the hope
that it'll help you to achieve those goals. But again, discriminating goals is
a very human and complex concept. For what you know, what will be
the successful liberals, the logical thing for you to do is to tell the
tool the goals, and then perhaps it can help find the products that will
help you facilitated goals. Like what I'm excited about with the M three sixty
five copilot is I can't keep track of all this stuff is in them three
sixty five, So this tool is going to help me pick a product to
use. Now that also sounds like a marketing engine, but at least you
know, it's idea towards ways to solve things, because we all know that
Microsoft typically builds one product for one problem. Yeah yeah, totally yeah,
yeah, like Google googles similar though, right Google that has two AI teams
that were competing with each other that you know, of goodness knows how many
are not visible and how many different teams inside of Microsoft, we're working with
different chatting just like I've pulled up links for the the Q and a Maker,
the Microsoft Bought framework. Like there was lots of folks we talked about
Lewis the PII detector, Like that's all different software, that's all different teams
they and the chances they're back end pieces of the same No, Like that's
not likely. Hey boss, I have this idea chipping. You know,
you touched them on a really interesting point though, Like we you know,
I think every product has the issue where people don't know all the things the
product could do and when it could solve a problem that they currently have it
that they may even be trying to use the product to solve that, they
just don't know that it can how to do it. And that's one of
those areas that we're finding, Uh, you know we want to bring into
with chat chypt is that for example, I mentioned we have a set of
skills you can call right, but how do you know which skill is going
to do the thing you want to do? Well, that's that's where skill
discovery comes into play. And so if we describe the skills with natural language,
then we can have a natural language. You know. Again, it's
a selector problem, right, like we can have it select the skill.
Sure. Well, how many you sat on the other side of this hit
fill You've been the product maker when the customers company said, boy, we
should really wish we had this capability. It's like, you mean that capability
to put in three or fo Yeah, exactly. And a lot of times
people aren't asking, oh, like, you know, can you do this?
They're just saying, hey, you know, I ran into a problem
with you know, X, Y Z, and if you know you can
take that and synthesize it and say, hey, oh, actually we have
something that solves that problem. Right. That's where it gets really interesting,
right, or you have that classic Microsoft phrase, so why would you want
to why would you ever want to do that? Right? Right? Exactly?
Because they're describing a problem they're having, not that gold are trying to
reach, right exactly. Yeah, And that's where a lot of that's where
things are very limited in terms of like feature searching within feature within products is
like, oh, you're searching for the feature, but you're not searching for
the problem, right, You're trying to draw a picture and word I like
that I'm having trouble with my color selection, and why would you do that?
Right, It's exactly that problem. So we're right on an hour here
is uh I could go on talking to you about this for another hour easily.
Maybe you ought to come back and we can pick up where we left
off another time. Sure, sure, yeah, I think like the one
thought I think I want to close with is one of the challenges I see
with these open AI and chat GPT is the fact that, like when you're
asking these questions, it's getting its information from somewhere, and that somewhere is
human produce. Right, So, yes, a programming question, it probably
you know, synthesize it from stack overflow, right. But if but it's
not giving credit to stack overflow, it's not getting credit to the content creator.
Right. What if nobody goes to stack overflow because chat GPT does a
better job of answering the question, where's the incentive for the content creators to
keep answering questions on stack overflow. I could see it all degrading if they
don't solve that problem. Being AI does do a pretty good footnoting like it
absolutely when you ask it for specific things, it'll pull that up like this
is possible. They should be. They are indexing internally. What they need
to do is show that they are. It also is a great piece for
fact checking. Yeah, say, hey where you know, where did you
get this thing? You know, I did the chat GPT right at Wikipedia
entry for Richard Campbell and he picked some other Richard Campbell for the birthday.
There's a few pieces of my things in there, Like, there's nice for
it to have a chat GPT show its sources always. Well, so I
did that. I said, right a Wikipedia article about Phil Hack, and
cite your sources, and it did cite sources, but it made up the
sources. Yeah, I clicked the u URLs that did not exist. And
again you're personifying right. It sure didn't make them up. The algorithm generated
them, and they were inaccurate. The algorithm stochastically generated citations that look like
citations. Yeah, that were that weren't actually to use it, Dunning Kruger,
scales are falling from my eyes. That's right. That that is the
perfect way to describe it, Dunning Kruger. We're going to have to leave
it there. Phil, thank you very much. It's been a pleasure as
always, and it's been fun by you again, and we'll see you next
time on dot net rocks. Dot net Rocks is brought to you by Franklin's
Net and produced by Pop Studios, a full service audio, video and post
production facility located physically in New London, Connecticut, and of course in the
cloud online at pwop dot com. Visit our website at dt n et r
ocks dot com for RSS feeds, downloads, mobile apps, comments, and
access to the full archives going back to show number one, recorded in September
two thousand and two. And make sure you check out our sponsors. They
keep us in business. Now, go write some code seex time t Gord.
That is hard than my text is