Speaker 1: Welcome to the Deep Dive, the show where we try
Speaker 1: to cut through the noise and get you truly well informed.
Speaker 2: Fast glad to be here.
Speaker 1: So today we're plunging into a topic we've all encountered,
Speaker 1: and let's be honest, sometimes really really disliked the chatbot.
Speaker 1: Oh yeah, you know the ones they just don't understand
Speaker 1: a single word you say, or they send you in
Speaker 1: these endless circles, or you know, make you desperately mash
Speaker 1: that speak to a human button.
Speaker 2: It's such a universal pain point, isn't it. And it
Speaker 2: really spotlights a critical challenge. Yeah, how do we build
Speaker 2: AI that actually understands us? Yeah, and you know helps that's.
Speaker 1: The core question exactly. So for this Deep Dive, we're
Speaker 1: unpacking the secrets behind creating genuinely well delightful AI interactions.
Speaker 2: Hopefully we're pulling insights from some really interesting new research,
Speaker 2: Effective conversational AI Chatbots that work by Ennikarrosa, Andrew Freed,
Speaker 2: and Corey Jacobs. It just came out in twenty twenty five.
Speaker 1: Yeah, and our mission today is basically to reveal why
Speaker 1: some bots succeed where others just spectacularly fit, and also
Speaker 1: how the newest advancements in AI are truly changing the
Speaker 1: game for the better.
Speaker 2: Think of this as your shortcut maybe to understanding how
Speaker 2: to build or even just identify a truly effective conversational AI.
Speaker 1: Okay, let's get into it then, So let's.
Speaker 2: Maybe start with a clear definition for you. Conversational AI.
Speaker 2: It's essentially a set of technologies designed to mimic human
Speaker 2: interaction or sometimes even replace it using natural language.
Speaker 1: Right.
Speaker 2: It goes by lots of names chatbots, virtual agents, AI assistants,
Speaker 2: sometimes even digital.
Speaker 1: Employees, digital employees, huh okay.
Speaker 2: And you mostly see it use for automating customer service,
Speaker 2: powering voice assistants like Alexa or Siri, and even sometimes
Speaker 2: pre screening interactions before they actually go to a human.
Speaker 1: So it's way more than just that little chat window
Speaker 1: that pops up on a website. It's kind of everywhere,
Speaker 1: it really is, and the book breaks these down into
Speaker 1: I think three main functional categories.
Speaker 2: Is that right precisely? Yeah? First, you have your question
Speaker 2: answering bot. People often call them faqbots. They're designed to
Speaker 2: give direct responses to pretty simple factual questions like when
Speaker 2: are you open or where you located? No follow up needed. Really,
Speaker 2: they just spit out the information.
Speaker 1: So these are the quick hit ones. Get in, get
Speaker 1: the answer, get out. Is there like a common mistake
Speaker 1: people make when they're designing just these simple bots.
Speaker 2: Well, I think the main pitfall is underestimating the sheer
Speaker 2: variety of ways users might ask the same simple question.
Speaker 2: You know that mismatch leads straight to misunderstanding.
Speaker 1: Ah, right, makes sense.
Speaker 2: Then you have the process oriented or transactional solutions. Now,
Speaker 2: these are designed to guide users through a series of
Speaker 2: steps to actually achieve a specific goal.
Speaker 1: Like booking an appointment or checking inn account balance maybe.
Speaker 2: Exactly checking in account balance, booking something. They might collect
Speaker 2: info for someone else to handle later, or sometimes they
Speaker 2: can even execute the transaction right then and there.
Speaker 1: Okay, And the last category, what's that?
Speaker 2: That's the routing agent. It's holds you basically is to
Speaker 2: figure out where to send you next, so like a
Speaker 2: dispatcher kinda yeah, either to another more specialized bot, or
Speaker 2: you know, when it's necessary, hand you off to a
Speaker 2: human agent.
Speaker 1: Okay.
Speaker 2: And what's fascinating here, and the book points is out,
Speaker 2: is that many real world AI solutions are actually a
Speaker 2: clever mix of all three. Oh interesting, Like how We'll
Speaker 2: think of a retail banking chatbot. It might answer FAQs
Speaker 2: about bank hours, right, but it could also guide you
Speaker 2: through opening a new account that's transactional, and then route
Speaker 2: you to a human specialist for something complex like fraud reporting.
Speaker 1: Right right. It blends the functions that really paints a
Speaker 1: clear picture of how versatile these things can be.
Speaker 2: Yeah, when they work well.
Speaker 1: So given this sort of intricate blend of categories, how
Speaker 1: does this sophisticated dance actually happen? Behind the scenes? The
Speaker 1: book describes this fascinating three step process.
Speaker 2: It is quite elegant when you break it down. But
Speaker 2: here's the real insight. I think the brilliance of a
Speaker 2: truly effective bot it lies in the seamless execution of
Speaker 2: these three fundamental steps. If any one of them falters,
Speaker 2: the whole experience just kind of collapses.
Speaker 1: Okay, so what's step one?
Speaker 2: Step one, the bond has to figure out what the
Speaker 2: user actually wants. This is done using natural language understanding
Speaker 2: or NLU.
Speaker 1: NLU got it.
Speaker 2: Often this uses a machine learning text classifier. Think of
Speaker 2: it like an AI that learns to categorize text, maybe
Speaker 2: like sorting your emails into urgent or promotion. It uses
Speaker 2: that to figure out the user's intent.
Speaker 1: Okay, so intent when I type something? The first challenge
Speaker 1: is the bot figuring out what I'm actually trying to do,
Speaker 1: like distinguishing between me wanting to reset my password versus say,
Speaker 1: find a store.
Speaker 2: Exactly that you nailed it, that's the intent y Step two.
Speaker 2: Once it thinks it knows the intent, the bot needs
Speaker 2: to gather any extra information it needs to actually fulfill
Speaker 2: that want. Okay, So a dialogue engine will ask clarifying questions,
Speaker 2: and it might use something called orchestration layers to interact
Speaker 2: with other systems through APIs.
Speaker 1: APIs right like ways for computer systems to talk to
Speaker 1: each other.
Speaker 2: Precisely, it's the bot's way of securely talking to other
Speaker 2: databases or services to pull the specific details it needs,
Speaker 2: like your account info or whatever.
Speaker 1: Okay, intent figured out, info gathered? What's step three?
Speaker 2: Step three? Give the user what they want, simple as
Speaker 2: that ideally, whether that's fulfilling their request directly providing the information,
Speaker 2: or connecting them to a human agent.
Speaker 1: And throughout this whole thing.
Speaker 2: The critical takeaway, and the book really emphasizes this is
Speaker 2: it must be quick, easy, and crucially follow ethical guidelines.
Speaker 1: Ethical guidelines like what specifically like.
Speaker 2: Handling sensitive information securely, and a big one never ever
Speaker 2: pretending the AI is actually a human. Transparency is key.
Speaker 1: Okay, it sounds so logical laid out like that, Yet,
Speaker 1: as we said, for so many of us, the actual
Speaker 1: experience with conversational AI causes so much pain. Yeah, the
Speaker 1: book points out those classic frustrations, Right, the bot didn't
Speaker 1: understand the thing I said, or you get that robot
Speaker 1: voice initiating some totally confusing dialogue, or you just immediately
Speaker 1: hit the button to talk to a person.
Speaker 2: We've all been there.
Speaker 1: It really begs the question, what exactly causes this weak understanding?
Speaker 1: Why are they so often bad?
Speaker 2: Well, weak understanding shows up in several frustrating ways. Right
Speaker 2: the chatbot gives you the wrong answers, or it uses
Speaker 2: that fallback intent way too much, you know, the sorry
Speaker 2: I'm not sure what you're asking message yes, Or you
Speaker 2: see frequent escalations to human agents, declining user engagement over time,
Speaker 2: people just giving up and leaving, increasing abandonment rates.
Speaker 1: So if users are constantly being asked to rephrase or
Speaker 1: the bot just gives totally irrelevant responses, that's a dead giveaway.
Speaker 2: Absolutely clear sign the understanding just isn't.
Speaker 1: There, So what's behind it? Is it just like bad
Speaker 1: luck or is there something fundamentally limited?
Speaker 2: No, not usually bad luck. The book identifies a few
Speaker 2: really common culprits, and the insight here is that these
Speaker 2: are often design failures or sometimes maintenance fail things that
Speaker 2: could have been prevented. Okay, like what well. One is
Speaker 2: manufactured training data, so examples that don't truly reflect how
Speaker 2: real users actually speak or type.
Speaker 1: Right, if you train it on perfect grammar, but people
Speaker 1: use slang or type fragments.
Speaker 2: Exactly, the loot's going to fail. Another big one is
Speaker 2: insufficient scope or gaps in topic coverage. Basically, the bot
Speaker 2: just doesn't know enough about the things users are asking about, like.
Speaker 1: That Meti World Pharma bought example, during the vaccine rollout.
Speaker 2: Perfect example yeaheah. Initially it could handle general COVID nineteen
Speaker 2: questions fine, but when people started asking about you know,
Speaker 2: vaccine eligibility or booking appointments, the bot was totally stumped.
Speaker 1: Because it hadn't been updated. The world changed faster than the.
Speaker 2: Bot exactly, which highlights another cause new information that the
Speaker 2: bot hasn't been taught. And the fourth one, which can
Speaker 2: often be the trickiest to sort out, is a lack
Speaker 2: of vetting or proper gatekeeping round changes.
Speaker 1: What do you mean by that? Like too many cooks
Speaker 1: in the kitchen?
Speaker 2: Kind of untested changes or updates made by team who
Speaker 2: aren't familiar with the whole system can accidentally introduce duplication
Speaker 2: or create conflicts between different intents or mess up the
Speaker 2: balance of the training data.
Speaker 1: Wow.
Speaker 2: Yeah. The book mentions a client where they saw their
Speaker 2: classifiers accuracy just plummet from around eighty percent down to
Speaker 2: like fifty five percent over time.
Speaker 1: Fifty five percent that's barely better than guessing for some things, right, And.
Speaker 2: It was because of all these unvetted changes piling up.
Speaker 2: The insight here is that building an effective chatbot isn't
Speaker 2: a one and done thing. It needs really diligent processes
Speaker 2: to stop that kind of entropy from creeping in.
Speaker 1: That's a huge drop. So how do we actually measure
Speaker 1: this understanding for traditional AI to stop that kind of
Speaker 1: decline from happening.
Speaker 2: Well, for traditional classification based AI, we rely on a
Speaker 2: few core metrics accuracy, precision, and recall.
Speaker 1: Okay, break those down for us accuracy seems straightforward.
Speaker 2: Accuracy is yeah, basically the overall percentage of correct predictions
Speaker 2: the bot makes simple enough. Recall is the bot's ability
Speaker 2: to identify the correct intent. Think of it as catching
Speaker 2: all the relevant questions for a specific topic.
Speaker 1: So if recall is low, it means.
Speaker 2: The bot is missing a lot of relevant questions. Like
Speaker 2: the example in the book, if a hashtag login issue
Speaker 2: intent had a really low recall maybe zero point four
Speaker 2: to four, it means the bot missed more than half
Speaker 2: the questions that were actually about login problems.
Speaker 1: Ouch, okay, and precision.
Speaker 2: Precision, on the other hand, is the bot's ability to
Speaker 2: avoid giving a wrong intent. So if precision is low,
Speaker 2: your bot might be confidently misunderstanding.
Speaker 1: Users, which might be even worse.
Speaker 2: It can be yeah, more frustrated than the bot just
Speaker 2: saying I don't know. So the real insight here is
Speaker 2: how critical it is to balance both precision and recall.
Speaker 2: Sometimes improving one can actually hurt the other, so you
Speaker 2: need to watch both.
Speaker 1: That makes sense. It's a balancing act. So we're talking
Speaker 1: rigorous measurement, But how do we actually test this in
Speaker 1: a way that reflects the real world. You mentioned kfold
Speaker 1: cross validation or blind testing.
Speaker 2: Yeah, those are standard method and AI generated data can
Speaker 2: be useful for blind testing, especially when you're just starting out.
Speaker 2: But what's really fascinating in the book highlights this is
Speaker 2: that the most reliable, least biased testing data it comes
Speaker 2: from representative.
Speaker 1: Production logs, meaning the actual conversations people have had with
Speaker 1: the bot exactly.
Speaker 2: These logs show what users actually ask and precisely how
Speaker 2: they phrase it. It gives you the truest measure of
Speaker 2: how the bot performs in the wild.
Speaker 1: But that sounds like it requires a lot of work
Speaker 1: to go through and label correctly.
Speaker 2: It often does. It frequently requires careful, sometimes even manual
Speaker 2: annotation by humans to identify what the golden or correct
Speaker 2: intent should have been for each user message. But the
Speaker 2: insight is clear. Real user data is gold standard for testing.
Speaker 1: It sounds like incredibly diligent work making sure the bot's
Speaker 1: brain is truly learning the right lessons from real interactions.
Speaker 2: It is.
Speaker 1: Yeah, just as we're learning how to really fine tune
Speaker 1: these traditional systems, there's been this monumental shift in AI
Speaker 1: that's just complete lately, changing the rules of the game
Speaker 1: for chatbots, oh, absolutely, which brings us to the real
Speaker 1: game changer. Generative AI. How is this revolutionizing the very
Speaker 1: nature of conversational interaction?
Speaker 2: Right? Generative AI. It's kind of a blanket term really
Speaker 2: for AI that's powered by these foundation models. So specifically,
Speaker 2: we're usually talking about large language models or lms.
Speaker 1: LMS. We hear that term everywhere.
Speaker 2: Now, Yeah, think of them as these incredibly vast machine
Speaker 2: learning models. They've been trained on well basically all the
Speaker 2: Internet's text, or huge chunks of it anyway.
Speaker 1: Okay, and how do they work? Fundamentally?
Speaker 2: Their core function essentially is to predict the next word
Speaker 2: in a sequence, and because they're trained on so much text,
Speaker 2: they get incredibly good at it, good enough to generate
Speaker 2: everything from coherent paragraphs to you know, entire pages of
Speaker 2: texts that sound remarkably human.
Speaker 1: Wow. Okay, So how do these incredibly powerful lms help
Speaker 1: solve those common chat butt pain points we just talked about,
Speaker 1: the ones that make us want to, you know, throw
Speaker 1: our phones.
Speaker 2: They offer potential solutions across the board. Really for that
Speaker 2: weak understanding problem. Lllms can help train much stronger traditional
Speaker 2: intents or and this is a big one, they can
Speaker 2: even entirely replace traditional intent recognition using something called retrieval,
Speaker 2: augmented generation or air gray greg.
Speaker 1: Okay, we'll definitely need to dive.
Speaker 2: Into that, Yeah we will. But the point is lllms
Speaker 2: are just far more adaptive to nuance in all the
Speaker 2: varied ways people phrase things.
Speaker 1: And what about the complexity issue bots getting too confusing?
Speaker 2: They can help there too. Lms can assist in writing simpler,
Speaker 2: clearer dialogue for the bot, or they can even be
Speaker 2: used to test dialogue flows for unexpected complexity before you
Speaker 2: deploy them.
Speaker 1: Okay, and the immediate opt outs people just giving up
Speaker 1: right away.
Speaker 2: Generative AI can help write much more engaging, maybe even
Speaker 2: more empathetic pros for the bot's messages. Setting a better
Speaker 2: tone right from the start can make a huge difference
Speaker 2: in making the user feel heard and willing to continue.
Speaker 1: So they're not just for the end user experience, but
Speaker 1: they're also tools for the people actually building the bots.
Speaker 1: I saw a table in the source about key applications exactly.
Speaker 2: Lllms have both consumer facing applications like generating answers using
Speaker 2: R which we mentioned, or maybe summarizing long conversation transcripts
Speaker 2: or human agents who take over a call. It's useful, yeah,
Speaker 2: hugely and then they have powerful build assistant tasks. They
Speaker 2: can help copy it or even write dialogue flows from scratch,
Speaker 2: or they can augment training data for the human builders,
Speaker 2: which is just a massive time saver.
Speaker 1: But with all that power, especially if they're trained on
Speaker 1: all the Internet's text, there must be some pretty significant danger,
Speaker 1: some pitfalls we need to watch out for. Oh.
Speaker 2: Absolutely, that's a critical point. The Internet, as we all
Speaker 2: know is it's full of bias, hateful speech, misinformation, you
Speaker 2: name it, and lms can unfortunately learn from all of that.
Speaker 2: So guardrails are absolutely crucial, non negotiable.
Speaker 1: Really, what kind of guardrails are we talking about?
Speaker 2: Things like content filters, but also process guardrails, like a
Speaker 2: beforehand review process that means the LLM might assist a
Speaker 2: human maybe drafting your response, but the human always has
Speaker 2: the final say and is ultimately responsible for the output.
Speaker 1: Human in the loop exactly.
Speaker 2: Yeah, and perhaps most importantly, grounding the LM's output in
Speaker 2: your own company's verified accurate documents through Araghi. That stops
Speaker 2: it from just pulling answers from.
Speaker 1: The wild web, like that unforgettable Canadian Airline chatbot example
Speaker 1: that went viral.
Speaker 2: Precisely that case is legendary. Now, their chatbot offered a
Speaker 2: bereavement discount that didn't actually exist based on some information
Speaker 2: it hallucinated or pulled in correctly.
Speaker 1: And the airline tried to argue the bot was separate.
Speaker 2: They tried to argue as a separate legal entity, if
Speaker 2: you can believe it. The court strongly disagreed and made
Speaker 2: them honor the discount. Wow, it really underscores a critical insight.
Speaker 2: Companies are responsible for what their bots say, which highlights
Speaker 2: the absolute necessity of these guardrails, especially like RAG, to
Speaker 2: ensure accuracy and frankly avoid legal nightmares.
Speaker 1: H that's a very expensive lesson and it really drives
Speaker 1: home the need for proper implementation. So let's talk more
Speaker 1: about this argo retrieval augmented generation. You said it's a
Speaker 1: big part of solving the weak understanding problem, especially for
Speaker 1: those less common, more specific queries.
Speaker 2: It absolutely is traditional intent based systems. They really struggle
Speaker 2: with what's called the long tail problem.
Speaker 1: Long tail.
Speaker 2: Yeah, think about it. Imagine trying to write a specific
Speaker 2: rule or train an intent for every single possible question
Speaker 2: someone could ask about your products or services. It's an
Speaker 2: impossible task, right, there's a long tail of very specific,
Speaker 2: infrequent questions.
Speaker 1: Right, you can cover the common stuff, but not everything exactly.
Speaker 2: So when users ask questions that deviate from those pre
Speaker 2: defined intents you did train, or questions that are simply
Speaker 2: too uncommon to have specific training data for, the traditional
Speaker 2: bot just breaks down. It throws up its hands because
Speaker 2: it has no rule to follow.
Speaker 1: Okay, so Argie is the answer to that long tail problem.
Speaker 1: How does it handle it differently, say, compared to just
Speaker 1: adding a search function to the chatbot.
Speaker 2: That's a great comparison. Let's think about traditional search within
Speaker 2: a chat bot first. It would work kind of like
Speaker 2: the pharmabout example before Eric. It finds relevant documents or passages,
Speaker 2: maybe on ibuprofen and blood pressure. Okay, the benefits are clear.
Speaker 2: You get a breadth of information, it's relatively easy to maintain,
Speaker 2: just add or edit your documents, and search technology is
Speaker 2: well established. But the downsides, the drawbacks are significant for
Speaker 2: the user experience. It often just returns links or maybe
Speaker 2: short snippets of text. It forces the user to click through, read,
Speaker 2: and piece together the answer themselves, which is really frustrating.
Speaker 1: You ask the bot for an answer, not homework.
Speaker 2: Exactly, and it's particularly bad for voice interactions. You can't
Speaker 2: exactly click a link when you're talking to a voice assistant.
Speaker 1: Good point. So how does argon improve on that?
Speaker 2: This is where retrieval augmented generation really shines. It offers
Speaker 2: a truly powerful leap forward. Argin combines that search based
Speaker 2: retrieval step with the power of generitive models. The LMS.
Speaker 1: Okay, so it searches and generate precisely.
Speaker 2: The insight here is that it first retrieves the most
Speaker 2: relevant passages from your own verified knowledge base, your documents,
Speaker 2: your website content, whatever you feed it, and then the
Speaker 2: LM takes those retrieved passages and synthesizes them into a cohesive,
Speaker 2: contextually aware answer in natural language.
Speaker 1: Ah So, instead of just giving me links about ibuprofen
Speaker 1: and blood pressure, the pharma bought with our gig would
Speaker 1: actually read those relevant bits and then write me a clear,
Speaker 1: single summary answer.
Speaker 2: Exactly, and crucially, that answer is grounded in that verified
Speaker 2: source information you provided.
Speaker 1: Grounded That seems like the key word it really is.
Speaker 2: It means the answer is based on your accurate, up
Speaker 2: to day data, not just the llm's general knowledge is
Speaker 2: great from the internet years ago. This dramatically expands the
Speaker 2: bot's versatility. It can answer way more questions far more accurately,
Speaker 2: and it's significantly reduces those bot doesn't understand and too
Speaker 2: much complexity pain points.
Speaker 1: Okay, that sounds amazing. How is it actually implemented behind
Speaker 1: the scenes? It sounds potentially quite complex.
Speaker 2: It involves a few pretty fascinating steps. First, your large
Speaker 2: documents think manuals, website pages, knowledge based articles are broken
Speaker 2: down or chunked into smaller manageable pieces, maybe paragraphs or
Speaker 2: logical section chunking, got it. Then an AI model called
Speaker 2: an embedding model converts these chunks into numerical representations. We
Speaker 2: call these embeddings.
Speaker 1: Numerical representations like coordinates on a map.
Speaker 2: Kind of Yeah, it's like creating a unique numeric fingerprint
Speaker 2: for the meaning of each piece of text. Texts with
Speaker 2: similar meanings end up with similar fingerprints or closer coordinates
Speaker 2: in this high dimensional space. Whoa these embeddings. These fingerprints
Speaker 2: are then stored in a special kind of database called
Speaker 2: a vector database. Think of it as a super fast,
Speaker 2: intelligent library index that doesn't just look for keywords, but
Speaker 2: for semantic similarity for similar.
Speaker 1: Okay, so you've indexed all your chunked documents by their meaning.
Speaker 1: What happens when I ask a question?
Speaker 2: Right at runtime? When you ask something, your question is
Speaker 2: also turned into an embedding using the same model. The
Speaker 2: system then searches the vector database to find the chunks
Speaker 2: whose embeddings are closest, meaning most semantically similar to your
Speaker 2: questions embedding, So it.
Speaker 1: Finds the most relevant paragraphs based.
Speaker 2: On meaning exactly, and then those retrieved passages are fed
Speaker 2: to the LLM along with your original question with instructions
Speaker 2: like answer the user's question based only on this provided information.
Speaker 2: The LLM then synthesizes the final grounded answer.
Speaker 1: Wow, that's a lot of intricate steps chunking, embeddings, vector databases, retrieval, synthesis.
Speaker 1: So for you, the listener, who might be thinking, Okay,
Speaker 1: if the lms are so powerful, why not just ask
Speaker 1: the LLM directly, why bother with all this a rag stuff?
Speaker 2: That's a really crucial question, And the reason is simple
Speaker 2: control and reliability. Llms used on their own.
Speaker 1: Can hallucinate, hallucinate, make things up.
Speaker 2: Yeah, literally makeup facts, or provide outdated information because their
Speaker 2: training data isn't perfectly current. Remember they're trained on a
Speaker 2: massive general data set, but they don't inherently know the specific,
Speaker 2: up to the minute details of your company's policies or
Speaker 2: product features, and you have much less direct control over
Speaker 2: the answers they generate.
Speaker 1: Okay, So RAG fixes that.
Speaker 2: Eric directly addresses this. By grounding the LM's answers in
Speaker 2: your specific, verified, up to day documents, you ensure accuracy
Speaker 2: and reliability. You maintain control over the knowledge source. It's
Speaker 2: about combining that amazing generative power of the LLM with controlled, accurate,
Speaker 2: trustworthy knowledge.
Speaker 1: That makes perfect sense. It's like giving the LLM guardrails
Speaker 1: made of your own information.
Speaker 2: That's a great way to put it.
Speaker 1: Okay, So you've built this amazing bot. Maybe it uses
Speaker 1: our gay, maybe it has finally tuned intents, it's got guardrails.
Speaker 1: You're done right, set it and forget it.
Speaker 2: Oh, if only, wouldn't that be not? No? Conversational AI
Speaker 2: is definitely not static. It can't be. Why not, which
Speaker 2: leads us straight to the critical insight of continuous improvement.
Speaker 2: Think about it. User needs change constantly, business rules and
Speaker 2: policies evolve, new technologies like generative AI itself emerge, and
Speaker 2: better AI models become available. All the time.
Speaker 1: So if you don't keep up, the bot just gets
Speaker 1: worse over time.
Speaker 2: Absolutely, performance inevitably degrades if you don't actively maintain and
Speaker 2: improve it. The book calls it fighting entropy in a
Speaker 2: constantly changing environment. You have to keep investing just to
Speaker 2: stay level, let alone get better.
Speaker 1: Okay, So what does this essential continuous improvement cycle look
Speaker 1: like in practice? Is there a process?
Speaker 2: Yes, it's an iterative process, a constant loop of refinement. Really,
Speaker 2: you first measure the baseline performance of your system. Get
Speaker 2: your starting.
Speaker 1: Point, okay, establish the baseline.
Speaker 2: Then you identify a problem, what's not working well? And
Speaker 2: ideally you connect that problem directly to a business metric.
Speaker 2: Not just the bot is confused, but maybe too many
Speaker 2: calls about exer transferring to an agent or customer satisfaction
Speaker 2: scores are declining. For why reason?
Speaker 1: Make it concrete and business relevant?
Speaker 2: Got it exactly? Next, you devise a solution. What's your
Speaker 2: plan to fix it? Then you develop and deliver those changes.
Speaker 2: And crucially, the book advice is making small, iterative changes
Speaker 2: rather than huge, big bang updates. Right, small changes because
Speaker 2: they have a smaller blast zone. If something goes wrong,
Speaker 2: it's easier to roll back a small change that causes
Speaker 2: problems than a massive overhaul makes sense safer. And the
Speaker 2: last step, finally, you monitor and evaluate did the changes
Speaker 2: actually deliver the improvements you expected? Did that metric you
Speaker 2: were tracking actually move in the right direction? Then you
Speaker 2: start the loop again measure, identify, solve, deliver, monitor.
Speaker 1: And it's all about connecting this technical work back to
Speaker 1: actual business value, isn't it You mentioned metrics. It can't
Speaker 1: just be tech jargon for its own.
Speaker 2: Sake, absolutely not. You have to speak the stakeholder's language.
Speaker 2: That means focusing on things like cost reduction.
Speaker 1: How does a better bought reduce costs several.
Speaker 2: Ways through containment that's completing calls or interactions without any
Speaker 2: human involvement, by reducing average handle time or AHT for
Speaker 2: the human agents who do get involved because maybe the
Speaker 2: bot gathered info better, or by reducing human touches which
Speaker 2: means fewer calls getting routed to the wrong place and
Speaker 2: needing another transfer.
Speaker 1: Okay, cost reduction? What else?
Speaker 2: The other big one is customer satisfaction or sees that
Speaker 2: you measure that with things like net Promoter score or
Speaker 2: MPs surveys, or by looking at metrics like timed resolution,
Speaker 2: How quickly did the customer get their issue solved or
Speaker 2: even reduce customer churn? Are fewer customers.
Speaker 1: Leaving you that example in the book about the medical
Speaker 1: ensurer though, that was fascinating. They improved the accuracy of
Speaker 1: their claim to night reason intent right.
Speaker 2: And it worked. It increased containment, more people got the
Speaker 2: answer from the bot, but their NPS scores actually dropped
Speaker 2: because the unhappy callers who previously would have just escalated
Speaker 2: to complain to a human were now self serving with
Speaker 2: the bot and then taking the post call survey and
Speaker 2: expressing their displeasure.
Speaker 1: Wow, so fixing one metric hurt another. What a nuanced problem.
Speaker 2: It's a perfect illustration that business goals can sometimes actually
Speaker 2: contradict each other. The real insight here is that you
Speaker 2: need to deeply understand those nuances and potential trade offs.
Speaker 2: Solving one problem might just uncover or even create another.
Speaker 1: One, which brings us beautifully to the user journey itself.
Speaker 1: Streamlining it reduced some complexity and trying to stop those
Speaker 1: dreaded opt outs. The book talks quite a bit about
Speaker 1: the pain of complexity.
Speaker 2: Yeah, complex conversations are just intimidating and confusing for users.
Speaker 2: They lead directly to friction, people having to retry things
Speaker 2: and ultimately abandonment giving up.
Speaker 1: Like that insurance company's voice system for checking claim Stata.
Speaker 2: Oh, that was a rough one. It had only a
Speaker 2: forty percent success rate forty percent, way so low because
Speaker 2: it required users to verbally input five separate pieces of
Speaker 2: numeric information before it would even start searching for the claim.
Speaker 2: Think about that provider I member ID, claim date, claim number.
Speaker 1: That sounds exhausting just listening to it. Who has all
Speaker 1: that ready?
Speaker 2: Exactly? It was grueling? And the insight here is that
Speaker 2: every single extra step, every additional piece of information you
Speaker 2: require from the user, creates another potential point where they
Speaker 2: just drop off.
Speaker 1: So how did they fix it? What was the impact?
Speaker 2: They simplified it by using the caller ID to potentially
Speaker 2: identify the member automatically and then only asking for I
Speaker 2: think four fields instead of five. They grastically improved the
Speaker 2: success rate. That relatively small simplification made a huge difference
Speaker 2: to the user experience and therefore the business outcome.
Speaker 1: So how can builders spot these overly complex dialogue flows
Speaker 1: in their own bots? Are there specific warning signs to
Speaker 1: look for? Anti patterns the book called them.
Speaker 2: Yeah, the book lists several good ones. Asking for information
Speaker 2: users are unlikely to have handy right then, having really
Speaker 2: rigid input requirements like it only accepts dates in one
Speaker 2: specific format, no flexibility.
Speaker 1: Yeah, that's annoying.
Speaker 2: Asking ambiguous questions where the user isn't sure what's being asked,
Speaker 2: Treating all users exactly the same, regardless of their context
Speaker 2: or history, Presenting too many options at once, choice overload,
Speaker 2: asking for information in a weird, disjointed order that doesn't
Speaker 2: feel natural, and a big one for voice delivering channel
Speaker 2: unsuited information, like trying to read out a really long
Speaker 2: complex URL over the phone. Clear signs of unnecessary complexity.
Speaker 1: Okay, so once you spot these complexity traps, how do
Speaker 1: you actively simplify the journey for the user?
Speaker 2: Well, A key strategy and a really powerful insight here
Speaker 2: is using contextual information personalize the experience. How So, based
Speaker 2: on things you might already know about the user, their location,
Speaker 2: maybe their time zone, the type of device they're using,
Speaker 2: their stated preferences, maybe even their past behavior or previous interactions.
Speaker 1: With you, like that banking chatbot example, MAX giving generic advice.
Speaker 2: Exactly MAX giving generic credit card advice to Emma, who's
Speaker 2: already a customer. That's just frustrating and unhelpful. But if
Speaker 2: Max new Emma's transaction history may be her current credit cards,
Speaker 2: it could give much more tailored relevant recommendations that saves
Speaker 2: her time, saves her effort, and makes the interaction feel
Speaker 2: actually valuable.
Speaker 1: That's a great application of using what you know what
Speaker 1: else helped simplify things.
Speaker 2: Slot filling is another really powerful technique.
Speaker 1: Slot filling like filling in blanks.
Speaker 2: Kind of, it allows the bot to capture multiple pieces
Speaker 2: of information from a single user utterance, letting it skip
Speaker 2: unnecessary follow up questions.
Speaker 1: Oh, I see like if I say.
Speaker 2: Like if you say I'd like to make a reservation
Speaker 2: for two people this Saturday at eight pm, A good
Speaker 2: bot can pull out the party size to the day
Speaker 2: Saturday and the time APM all from that one sentence.
Speaker 2: It doesn't need to ask how many people than what day?
Speaker 2: Than what time?
Speaker 1: Right, much smoother. The insight is really about efficiency. Then
Speaker 1: move the conversation forward.
Speaker 2: Faster, exactly get to the point without tedious back and forth.
Speaker 1: And allowing for flexibility too. I imagine users don't always
Speaker 1: say exactly what the bot expects them to say.
Speaker 2: Oh never, So you have design for multiple correct responses,
Speaker 2: especially with things like yes, no, or choice confusion.
Speaker 1: Like the example where the bot asks text message or phone.
Speaker 2: Call right and the user just says yes, or rigid
Speaker 2: bot might just say sorry, I didn't understand, but a
Speaker 2: flexible bot could perhaps infer that yes means text message
Speaker 2: if that's the first option or the more common preference,
Speaker 2: or at least ask a better clarifying question. The bot
Speaker 2: needs to try hard to meet the user where they are,
Speaker 2: not force them into a rigid script.
Speaker 1: Okay, so we've talked complexity, but why else do users
Speaker 1: just bail on chatbots, give up and leave? Besides getting confused?
Speaker 2: Well, there are several reasons, and the insight here often
Speaker 2: comes down to trust and managing expectations. Sometimes it's just
Speaker 2: prior poor experiences with other bots. They come in already skeptical.
Speaker 1: Fair enough scar tissue.
Speaker 2: Yeah. Other big reasons include the bot clearly not understanding
Speaker 2: them or asking them to rephrase too many times, feeling
Speaker 2: like they're stuck in a loop, or just not making
Speaker 2: progress towards their actual goal, or sometimes they simply don't
Speaker 2: like the answer the bot gives them. Or they had
Speaker 2: different expectations for what the bot could even do.
Speaker 1: Okay, so knowing why they leave, what are the best
Speaker 1: strategies to actually reduce those frustrating opt outs? How do
Speaker 1: we keep them engaged?
Speaker 2: The book suggests several key things. First off, great.
Speaker 1: Greetings the very beginning of the chat.
Speaker 2: Yeah, SE's the tone. A good greeting should affirm the
Speaker 2: bot's purpose, maybe briefly preview the journey ahead, set realistic
Speaker 2: expectations for completion time, and sometimes even incentivize the user.
Speaker 1: Like that utility company example where the bot started by
Speaker 1: saying something like this process should only take a few
Speaker 1: minutes exactly.
Speaker 2: That simple sentence manages expectations and gives the user a
Speaker 2: reason to stick with it. It's a small thing, but powerful.
Speaker 1: Okay, great greetings? What else?
Speaker 2: Second, try hard to understand, which goes back to what
Speaker 2: we discussed. Continuous investment in good relevant training data. Using
Speaker 2: conversational search or RV for those broader domains so you
Speaker 2: can handle more queries makes sense.
Speaker 1: Be better at understanding.
Speaker 2: Third, try hard to be understood, Use really clear, simple wording,
Speaker 2: allow for those flexible responses we talked about, and employ
Speaker 2: graceful error handling Instead of just saying retry, maybe try
Speaker 2: to disambiguate, offer a couple of options based on what
Speaker 2: you thought they meant.
Speaker 1: Okay, handle error is better.
Speaker 2: And fourth, fourth, implement smart opt out retention flows. If
Speaker 2: a user does ask for a human, don't just immediately transfer.
Speaker 2: Try to discover their true underlying goal. First, assess if
Speaker 2: the bot actually could help with that goal. Maybe try
Speaker 2: to convince the use of the bot can handle it effectively,
Speaker 2: or if not, at least route them intelligently to the
Speaker 2: next best action, maybe another specialized virtual agent or then finally,
Speaker 2: the right human team.
Speaker 1: So try to salvage the interaction if possible, or at
Speaker 1: least make the handoff smooth.
Speaker 2: Exactly, don't just dump them.
Speaker 1: And generative AI can actually help with crafting those messages themselves, right,
Speaker 1: making the bot sound less robotic and more helpful.
Speaker 2: Yes, absolutely, that's another powerful application. Llms can be great
Speaker 2: at rewriting potentially rude or unhelpful system error messages, transforming
Speaker 2: something blunt like you didn't provide a thirteen digit number
Speaker 2: into something much kinder and more instructive.
Speaker 1: Yes, off in the edges exactly.
Speaker 2: Yeah, and they can also help craft those greeting messages
Speaker 2: to be more helpful, more welcoming, and less like talking
Speaker 2: to a machine, making that initial interaction much less likely
Speaker 2: to cause an immediate opt out.
Speaker 1: Okay, this has been fantastic. Finally, let's touch on the
Speaker 1: human AI partnership. It seems like it's not just about
Speaker 1: building bots for humans, but increasingly about how AI, especially llms,
Speaker 1: can augment the human builders themselves. How do they become collaborators.
Speaker 2: That's a really exciting area. Lms are becoming incredible partners
Speaker 2: in the actual development process. They can help solve the
Speaker 2: cold start problem, you know, when you're building a brand
Speaker 2: new bot and have literally no real user data to
Speaker 2: start training with.
Speaker 1: Right where do you begin?
Speaker 2: Lms can generate realistic sounding initial training data, or they
Speaker 2: can help expand your existing data by filling gaps for
Speaker 2: maybe rare but really important intents like fraud reporting, where
Speaker 2: you might not have enough real world examples to train effectively.
Speaker 2: The insight here is that llms don't replace human creativity
Speaker 2: or oversight, but they can seriously supercharge the.
Speaker 1: Process so they can essentially generate data for both training
Speaker 1: and testing. That sounds like it could save a huge
Speaker 1: amount of manual effort.
Speaker 2: Oh. Absolutely, they can generate lists of synonyms for specific terms,
Speaker 2: different nouns for credentials maybe, or various verbs for lost
Speaker 2: or misremembered. But they can also generate entire utterances with
Speaker 2: lots of varied grammatical structures, statements, questions, even fragments or commands,
Speaker 2: just like real users.
Speaker 1: So instead of just forgot password, it might generate I
Speaker 1: can't remember my account information, or my account is locked,
Speaker 1: or help logging in.
Speaker 2: Precisely, this provides much more robust training data and also
Speaker 2: more realistic testing data. It helps reduce the inherent bias
Speaker 2: that might come from just a few humans righting examples,
Speaker 2: and it can significantly improve the classifiers accuracy beyond what
Speaker 2: manual efforts alone could likely achieve. It's about getting a
Speaker 2: truly comprehensive and diverse data set.
Speaker 1: And what about AI assisted process flows. That sounds almost
Speaker 1: like the AI is designing the conversation itself in a way.
Speaker 2: Yeah, it's about rapid prototyping and also hardening the system.
Speaker 2: Llams can actually suggest or even design entire process flows
Speaker 2: from scratch, Like you could ask it to design a
Speaker 2: flow for handling a medical insurance claim status check, and
Speaker 2: it might propose the steps, the questions to ask and
Speaker 2: even justify its design choices. Wow. They can also help
Speaker 2: execute dialogue flows at runtime and some newer architectures, acting
Speaker 2: as the chatbot's brain, dynamically figuring out what questions to
Speaker 2: ask next, to collect the information needed for say in
Speaker 2: an API call. Okay, and testing and critically, for testing,
Speaker 2: they can simulate user inputs. They can generate all sorts
Speaker 2: of weird, una expected or edge case inputs that human
Speaker 2: testers might never even think of. This really helps to
Speaker 2: harden the chatbot against unexpected interactions, making it far more
Speaker 2: robust when it meets real users.
Speaker 1: That's a massive timesaver and quality booster for developers. Okay,
Speaker 1: one last piece, those inevitable handoffs to human agents. What
Speaker 1: about summarization. It must be incredibly frustrating for human agent
Speaker 1: to have to read through a long, rambling chat log
Speaker 1: when they take over it really is.
Speaker 2: It wastes time and forces the customer to wait or
Speaker 2: repeat themselves. And the insight here is all about efficiency
Speaker 2: and improving the customer experience during that transfer. Agents don't
Speaker 2: need the entire transcript. They need a brief, targeted summary
Speaker 2: of what happened and what the.
Speaker 1: User needs, so llms can create that summary.
Speaker 2: Yes, yeah, llms are excellent at summarization. They can take
Speaker 2: a full chat transcript and condense it into concise pros
Speaker 2: or even better, sometimes they can extract key structure details
Speaker 2: like the user's ID, maybe a claim number, the specific issue,
Speaker 2: even a sentiment analysis score, and present that clearly to
Speaker 2: the agent.
Speaker 1: That sounds incredibly useful.
Speaker 2: It ensures a much smoother handoff, It saves the human
Speaker 2: agent valuable time letting them get straight to the issue,
Speaker 2: and it dramatically improves customer satisfaction because the customer doesn't
Speaker 2: have to frustratingly repeat everything they just told the bot.
Speaker 1: What a deep dive this has been. We've really explored
Speaker 1: how conversational AI works, haven't we, From those basic intent
Speaker 1: based bots all the way to the truly transformative power
Speaker 1: of generative AI and things like our rage.
Speaker 2: Yeah, and we've seen how thoughtful design, that constant cycle
Speaker 2: of continuous improvement, and this emerging strategic partnership between human
Speaker 2: builders and the lms themselves, how all that can overcome
Speaker 2: those common frustrations. We started with.
Speaker 1: Right streamlining complex processes and ultimately creating virtual assistance that
Speaker 1: feel genuinely understanding and actually helpful.
Speaker 2: And the real aha moment here for me at least,
Speaker 2: is that the ultimate goal isn't just about building a
Speaker 2: smarter machine, although they're getting incredibly intelligent, incredibly fast.
Speaker 1: It's more than that, I think.
Speaker 2: So it's really about designing an intuitive, maybe even empathetic,
Speaker 2: and definitely valuable experience for you, the user. It's about
Speaker 2: leveraging all this amazing technology to meet people where they are,
Speaker 2: truly understand their unique needs in that moment, and deliver
Speaker 2: real value.
Speaker 1: Whether that value comes from a quick answer, or helping
Speaker 1: with a complex transaction, or even just making that handoff
Speaker 1: to a human completely seamless.
Speaker 2: Exactly making the interaction feel effective and respectful of the
Speaker 2: user's time.
Speaker 1: So here's a final thought to leave you with. As
Speaker 1: conversational AI continues to evolve at this well breakneck speed,
Speaker 1: what new ethical considerations, what new design challenges will become
Speaker 1: paramount especially as that line between human and AI interaction
Speaker 1: gets increasingly blurred. How will our own expectations of what
Speaker 1: understanding even means continue to shift. That's a big question,
Speaker 1: definitely something for all of us to maul over as
Speaker 1: these systems become more and more deeply ingrained in our
Speaker 1: daily lives.