Speaker 1: You know, it's wild how second nature it's become to
Speaker 1: just talk to our devices. Hey, Google, set a timer, Siri,
Speaker 1: what's the weather? We barely think about it.
Speaker 2: Yeah, it really feels like something we just take for
Speaker 2: granted now, But.
Speaker 1: Pull back for a second. How does that actually happen?
Speaker 1: How does your phone hear you, understand what you want,
Speaker 1: and then you know, do something.
Speaker 2: It does feel a bit like a magic trick, doesn't it.
Speaker 2: But behind that simple interaction is this whole layered world
Speaker 2: of technology.
Speaker 1: It's quite complex, actually, and that's exactly the world we're
Speaker 1: diving into today. We're taking a deep dive into how
Speaker 1: you build these voice based applications, specifically thinking about Android devices.
Speaker 2: Okay, and our guide for this exploration is fascinating. It
Speaker 2: is a detailed technical guide published back in twenty thirteen.
Speaker 1: Twenty thirteen, so a bit of a snapshot from that
Speaker 1: era exactly.
Speaker 2: It gives us a really interesting look at the tools
Speaker 2: and approaches developers we're using them, leveraging Google's own capabilities
Speaker 2: and also some open source software.
Speaker 1: Right, So, our mission in this deep dive is to
Speaker 1: kind of cut through the complexity. We want to unpack
Speaker 1: the core concepts the essential building blocks.
Speaker 2: Fundamental, Yeah, the.
Speaker 1: Fundamentals, and show you the journey developers took to create
Speaker 1: apps you could talk to all without you needing to
Speaker 1: pour over the original technical manual yourself.
Speaker 2: Sounds good.
Speaker 1: We'll start with the absolute basics, speaking and listening and
Speaker 1: build up from there, maybe getting into complex conversations and
Speaker 1: even early virtual assistance.
Speaker 2: Okay, let's unpack this starting at the very beginning, seems right.
Speaker 2: Before a device can respond or have any kind of conversation,
Speaker 2: it first has to be able to speak. And here right.
Speaker 1: The fundamental capabilities Android provides for this are text to
Speaker 1: speech TTS and automated speech recognition or ASR.
Speaker 2: And thinking about why these are important, Well, it opens
Speaker 2: up so many possibilities. Imagine your hands are full like
Speaker 2: you're driving and need directions.
Speaker 1: Oh yeah, it's simple then or and.
Speaker 2: This is crucial for accessibility. Think about someone with a
Speaker 2: visual impairment using a green reader that's TTS or ASR
Speaker 2: helping someone communicate if they have difficulty speaking, much like
Speaker 2: Stephen Hawking used speech synthesis technology.
Speaker 1: They're really foundational tools. Okay, let's start with TTS text
Speaker 1: to speech. Simply put, it turns written text into spoken audio.
Speaker 2: Right, and the technology works in stages. First, it needs
Speaker 2: to understand the text itself, things like you know how
Speaker 2: to pronounce words that look similar but sound different based
Speaker 2: on context.
Speaker 1: Like reads versus read just exactly.
Speaker 2: Or converting numbers and abbreviations into full words. The cool
Speaker 2: part for developers often is the system handles a lot
Speaker 2: of this linguistic complexity. You don't always have to get
Speaker 2: into the weeds.
Speaker 1: Okay, So it understands the text, then it has to
Speaker 1: generate the actual sound exactly.
Speaker 2: A common approach, especially around the time this book was written,
Speaker 2: was something called concatenative synthesis.
Speaker 1: Concatenative synthesis okay, okay.
Speaker 2: Think of it like building a sentence by stitching together
Speaker 2: tiny prerecorded pieces of speech. These could be sounds, syllables, words,
Speaker 2: even short phrases.
Speaker 1: Ah like digital lego bricks for voice.
Speaker 2: Kind of algorithms select the right pieces and join them smoothly,
Speaker 2: trying to mimic natural rhythm and intonation. When it's done well,
Speaker 2: it can sound remarkably natural.
Speaker 1: It's kind of amazing how they make those pieces fit together.
Speaker 1: But wait, if they're stitching together pre recorded bits, why
Speaker 1: not just record a human voice actor saying everything the
Speaker 1: app might need to say.
Speaker 2: That's a great point, and sometimes apps do use professional
Speaker 2: voice actors, you know, for specific tromps where quality and
Speaker 2: consistency are paramount.
Speaker 1: Like a standard greeting or instruction exactly.
Speaker 2: But TTS becomes absolutely essential when the text is dynamic,
Speaker 2: when you can't possibly pre record everything it might need
Speaker 2: to say.
Speaker 1: Ah, right, like reading out a text message.
Speaker 2: You just got, or a news headline that just updated, or.
Speaker 1: Someone's name from your contact.
Speaker 2: List Precisely, you just can't anticipate every single phraser name.
Speaker 2: So while the quality might sometimes be a trade off
Speaker 2: compared to say, a perfectly recorded voice, linets offers that
Speaker 2: vital flexibility for dynamic content.
Speaker 1: And Android has had this built in for ages. Right, how
Speaker 1: do developers actually hook into it?
Speaker 2: Yeah, the capability has been there since Android one point six.
Speaker 2: Believe or not, developers use the framework provided. A key
Speaker 2: step is making sure the necessary language data is actually
Speaker 2: on the user's.
Speaker 1: Device the voice itself.
Speaker 2: Yeah, the voice files the rules for that language. The
Speaker 2: system lets developers check for this using something called an intent,
Speaker 2: and even prompt the user to install it if it's missing.
Speaker 1: Okay.
Speaker 2: The book suggests using a common software design pattern, a singleton,
Speaker 2: basically ensuring only one instance of the TTS engine is created.
Speaker 2: This helps manage resources efficiently smart and the examples in
Speaker 2: the book show how you might use this to say,
Speaker 2: read back text the user piped in, or maybe read
Speaker 2: text loaded from a file. You can specify the language too,
Speaker 2: like English or even regional variation.
Speaker 1: Okay, so that's how the device speaks. Now the other
Speaker 1: side hearing us automated Speech recognition ASR. This is turning
Speaker 1: our spoken words into text.
Speaker 2: Right, and like TTS, it involves steps. First, the device
Speaker 2: needs to capture the sound from the microphone and process it.
Speaker 2: Think of it as cleaning up.
Speaker 1: The audio, getting rid of background noise.
Speaker 2: Yeah, removing noise maybe echo, and just preparing it digitally
Speaker 2: for analysis.
Speaker 1: Then comes the recognition part itself. Yeah, breaking down the
Speaker 1: audio into.
Speaker 2: What sounds basically yeah, into tiny segments phones the basic
Speaker 2: sounds of the language, and then it tries to match them.
Speaker 1: Wow.
Speaker 2: This is where powerful statistical models come into play. These
Speaker 2: models are trained on massive amounts of recorded speech, learning
Speaker 2: how different sounds are typically pronounced in different context by
Speaker 2: different people.
Speaker 1: Wow.
Speaker 2: Okay, they build what's called an acoustic model. It's essentially
Speaker 2: a statistical map of how sounds relate to words.
Speaker 1: But words can sound exactly alike you mentioned read and
Speaker 1: read or two and two. How does it know the difference?
Speaker 2: Ah, good question. That's where another statistical model helps, the
Speaker 2: language model.
Speaker 1: Language model.
Speaker 2: This one understands the probability of words appearing together in sequence.
Speaker 2: It knows that after I went, the word two is
Speaker 2: far far more likely than two.
Speaker 1: Right context exactly.
Speaker 2: The language model provides that crucial context to help resolve
Speaker 2: those ambiguities.
Speaker 1: And the result isn't always just one single interpretation, is
Speaker 1: it like? It's not always certain?
Speaker 2: No, definitely not. Typically, the ASR system gives you back
Speaker 2: a list of possible results, ranked by how confident it
Speaker 2: is in each one. A list, yeah, it's often called
Speaker 2: an end best list. Each possibility comes with a confidence score,
Speaker 2: usually from zero to one. A score near one means
Speaker 2: the system is pretty.
Speaker 1: Sure it got it right, and that's useful for the developer.
Speaker 2: Incredibly valuable. They can just pick the top result. If
Speaker 2: the confidence is high, or if it's lower, or if
Speaker 2: the top one doesn't make sense in context, they can
Speaker 2: look at the others in the list. Or maybe even
Speaker 2: use the confidence score to decide HM I better ask
Speaker 2: the user to confirm this.
Speaker 1: This capability has also been around on end for a
Speaker 1: while since version two point one. Often when you tap
Speaker 1: the little microphone icon on the keyboard.
Speaker 2: Yes exactly, and developers have flexibility here too. You can
Speaker 2: use a simple built in tool and intent that handles
Speaker 2: the speak now, prompt and feedback automatically, super easy, quick
Speaker 2: and dirty, pretty much. Or if you want more control
Speaker 2: over the look and feel the user interface, you could
Speaker 2: use a more advanced component a speech recognizer instance. This
Speaker 2: lets you manage the UI yourself and react to specific
Speaker 2: recognition events like when the user starts or stops speaking.
Speaker 1: More control, more work typically yeah.
Speaker 2: The book again suggests using a library approach here, like
Speaker 2: an ASRLB, just to keep the code organized and reusable.
Speaker 1: And you mentioned language models. Can you tell the system
Speaker 1: what kind of speech to expect, like am I dictating
Speaker 1: an email or just barking a search query?
Speaker 2: Exactly? You can specify different language models. There's one design
Speaker 2: for free form dictation like long sentences, and another optimized
Speaker 2: for shorter phrases like web search queries.
Speaker 1: Ah.
Speaker 2: The book does note though, that even with these models,
Speaker 2: the input can still be quite open ended, so the
Speaker 2: developer might need to do more processing afterwards to figure
Speaker 2: out the specific command or meaning.
Speaker 1: Oh and because these systems often connect to cloud services
Speaker 1: for the heavy lifting.
Speaker 2: Right, the recognition part. Yeah, the app usually needs permission
Speaker 2: to access the Internet, and you need to handle potential
Speaker 2: errors like no speech detected or no match found or
Speaker 2: maybe a network problem.
Speaker 1: Got it. So we've got the building blocks. The device
Speaker 1: can speak TTS, and it can listen and turn speech
Speaker 1: into text ASR even giving us a list of possibilities
Speaker 1: with confidence scores. How do we actually put those together
Speaker 1: to build simple interactions?
Speaker 2: That's the next logical step, right, moving from just hearing
Speaker 2: or speaking to creating a basic back and forth. Think
Speaker 2: about those early voice actions.
Speaker 1: Like on Google Now back in the.
Speaker 2: Day, exactly telling your phone call mom or go to
Speaker 2: Wikipedia dot org. These are structured commands, simple cause and.
Speaker 1: Effect, and they're built just by combining those core TTS
Speaker 1: and ASR capabilities we just talked about.
Speaker 2: Pretty much the book provides them straightforward examples. One is
Speaker 2: an app called voice Search.
Speaker 1: It just takes whatever you say, listens using ASR right.
Speaker 2: Grabs the top result from that end best list the
Speaker 2: one with the highest confidence it seems it's right, and
Speaker 2: immediately plugs it into a standard Android web search intent.
Speaker 2: Boom search results appear very.
Speaker 1: Simple, okay, but that immediately brings up a potential problem
Speaker 1: which you hinted at. What if the ASR got it
Speaker 1: wrong exactly? This seems particularly tricky in another example app.
Speaker 1: The book mentions voice Launch, which tries to launch an
Speaker 1: installed application based on what the user says. Right, what
Speaker 1: if you don't say the exact app name, like maybe
Speaker 1: you say music player, but the app is actually called
Speaker 1: play Music.
Speaker 2: This is where the idea of similarity measures comes in.
Speaker 2: It's a crucial concept. The app needs a way to
Speaker 2: compare what the user said to the actual names of
Speaker 2: the apps installed on the device to find the best
Speaker 2: to match, even if it's not identical.
Speaker 1: How does it do that? Just check if the letters
Speaker 1: are similar.
Speaker 2: That's part of an orthographic similarity looking at the spelling.
Speaker 2: But crucially, it can also look at phonetic similarity.
Speaker 1: How words sound alike?
Speaker 2: Yes, so it could figure out that, I don't know,
Speaker 2: photos and photos probably refer to the same thing, even
Speaker 2: if the spelling's different.
Speaker 1: Okay, that's clever.
Speaker 2: The book mentions using algorithms like soundex for this phonetic comparison,
Speaker 2: although it notes the specific implementation they included was primarily
Speaker 2: tuned for English. The key thing is normalizing the input first,
Speaker 2: like her moving spaces, making everything lower case before you do.
Speaker 2: The comparison makes sense.
Speaker 1: Okay, So even with similarity measures, ASR isn't perfect. That
Speaker 1: potential for error means you often need to double check
Speaker 1: with the user right confirm things absolutely.
Speaker 2: Confirmation is vital for robust interaction. The book includes a
Speaker 2: simple example building on that Voicer chap. After recognizing something
Speaker 2: like pizza places right, yeah, might use TTS to ask
Speaker 2: did you say pizza places? And then it uses ASR again,
Speaker 2: but this time listening specifically for a simple.
Speaker 1: Yes or no uh, constraining the expected input exactly.
Speaker 2: It's a basic but really important step, especially when you're
Speaker 2: dealing with single critical pieces of data before taking an action.
Speaker 1: So we can make the device speak, listen, perform. These
Speaker 1: simple command action pairs handle some ambiguity with similarity, and
Speaker 1: even ask for basic yes no confirmation. But these interactions
Speaker 1: still feel quite rigid. You know, you have to say
Speaker 1: things in a very specific way or is just one
Speaker 1: command at a time. How do you make the conversation
Speaker 1: more flexible, like guide the user through collecting multiple pieces
Speaker 1: of information?
Speaker 2: Okay, yeah, that takes us into the realm of more
Speaker 2: structured conversations, often called form filling.
Speaker 1: Dialogue form filling like on a website.
Speaker 2: Exactly the same idea. The goal is to gather several
Speaker 2: distinct pieces of information from the user, one by one,
Speaker 2: but doing it through voice instead of textboxes and dropdowns.
Speaker 1: Okay, So, like booking a flow, it might ask what
Speaker 1: city are you flying from? Then once you answer, what
Speaker 1: is your destination exactly?
Speaker 2: And then maybe what date do you want to travel.
Speaker 2: To manage this, you need a system. You need a
Speaker 2: way to define the pieces of information you need. Think
Speaker 2: of these as slots to be.
Speaker 1: Filled like fields on a form.
Speaker 2: Precisely, and you need an algorithm, some logic that knows
Speaker 2: how to navigate the conversation to collect the info for
Speaker 2: EID slot in some sensible order.
Speaker 1: The book points to something called VoiceXML as a kind
Speaker 1: of model for this.
Speaker 2: Yeah. VoiceXML is or was, a W three C standard
Speaker 2: for defining these kinds of voice dialogues often used in
Speaker 2: call center systems. It uses concepts like forms, which contain
Speaker 2: fields or slots. Each field has a prompt, which is
Speaker 2: what the system asks the user.
Speaker 1: What is your destination right?
Speaker 2: And optionally, fields can have grammars associated with them, which
Speaker 2: constrain or help interpret what the user can say in response.
Speaker 1: So for a destination field, the grammar might only accept
Speaker 1: city names.
Speaker 2: Potentially yes and VoiceXML uses a concept called the form
Speaker 2: interpretation algorithm or FIA. It's basically the logic engine that
Speaker 2: steps through the form, asking for one piece of required
Speaker 2: information at a time until all the necessary slots are filled.
Speaker 2: The book uses a simplified subset of these ideas specifically
Speaker 2: for Android development, and.
Speaker 1: There's a specific library in the book to help build this.
Speaker 2: Yes a library called form filip containing classes to represent
Speaker 2: these forms and fields. It works by parsing XML files
Speaker 2: that the developer writes. These XML files define the structure
Speaker 2: of the conversation, what questions to ask, in what order,
Speaker 2: which fields are needed, so.
Speaker 1: The conversation logic is separate from the main app code exactly.
Speaker 2: It uses standard Android tools like XML pull parser handled
Speaker 2: via another helper library, xml lib to read these definitions.
Speaker 2: Then a key piece called the dialogue interpreter class steps
Speaker 2: through this structure, triggering the right TTS prompt and listening
Speaker 2: for ASR responses to fill each field.
Speaker 1: Does it handle background tasks like parsing might take time?
Speaker 2: Good point. It's designed to do the potentially slow work
Speaker 2: like parsing the XML or waiting for ASR in the
Speaker 2: background using Android's acing task, so the main app remains responsive.
Speaker 2: That separation of concerns is really nice.
Speaker 1: A great example used in the book is the music
Speaker 1: Brain app. What does that do?
Speaker 2: Right? The music Brain demo app uses this form filling library.
Speaker 2: It guides the user through a voice dialogue asking for
Speaker 2: details like maybe a word that appears in an album
Speaker 2: title or a start an end.
Speaker 1: Date range, using that form structure.
Speaker 2: Exactly once, it collects all the pieces of information needs
Speaker 2: by filling the slots in its form, and use that
Speaker 2: collected information to query the music Brain's web service, which
Speaker 2: is a big online music database.
Speaker 1: Ah. So it's combining the voice interface with external data a.
Speaker 2: Mashup, precisely. It shows how you can take destruction data
Speaker 2: gathered via voice and uses to interact with online services
Speaker 2: retrieve information. Process it may be filter or sort the
Speaker 2: results like sorting albums by release date using helper classes,
Speaker 2: and then present that back to the user, perhaps speaking
Speaker 2: the results or showing them on screen.
Speaker 1: Okay, so form filling lets us manage these multi step
Speaker 1: conversations to get structured data like album name and date range.
Speaker 1: But you mentioned that the ASR input within each step
Speaker 1: was still somewhat open ended in these basic examples. How
Speaker 1: do we make the app understand more than just the
Speaker 1: words the user says? How do we get it to
Speaker 1: understand the meaning behind the words?
Speaker 2: Right? That's a critical step towards more intelligent interaction, and
Speaker 2: that's where grammars come in, leading us into the field
Speaker 2: of natural language understanding or NLU.
Speaker 1: Grammars and NLU Okay.
Speaker 2: Grammars are tools designed specifically to help the application interpret
Speaker 2: more complex user inputs. They help extract not just the
Speaker 2: sequence of words, but the underlying meaning and specific structured
Speaker 2: pieces of information.
Speaker 1: So going beyond just recognizing show me flights to London
Speaker 1: as a sequence of words.
Speaker 2: So understanding that the user's intent is to see flights
Speaker 2: and the destination parameter is London.
Speaker 1: Got it? How do you create these grammars?
Speaker 2: The book discusses two main approaches. First, there are handcrafted.
Speaker 1: Grammars written manually by developers.
Speaker 2: Exactly, you write them yourself, often in an XML format
Speaker 2: like SRGS Speech Recognition Grammar Specification, though the book uses
Speaker 2: its own simplified XML format. You define the structure of
Speaker 2: acceptable phrases using rules, items within rules, alternatives, optional parts,
Speaker 2: and links between different rules.
Speaker 1: Can you give an example for that flight query?
Speaker 2: Sure, you might have a top level rule like rule
Speaker 2: ID fine flight. Inside that you might have an item
Speaker 2: for the phrase show flights or fine flights. Then maybe
Speaker 2: an optional item repeat zero one for the word two,
Speaker 2: and then crucially a reference to another rule like ruler
Speaker 2: ref u r F hashtag city which defines all the
Speaker 2: valid city names.
Speaker 1: And the hashtag city rule would list London, Paris, New
Speaker 1: York right.
Speaker 2: And within those rules you can use special semantic tags.
Speaker 2: So next to the item London in the city rule,
Speaker 2: you might have a tag like tag out lhr tag.
Speaker 2: This tells the system if the user says London, don't
Speaker 2: just return the word London, return the airport code lhr AH.
Speaker 1: Extracting structured data directly based on the grammar match. That's powerful,
Speaker 1: very powerful.
Speaker 2: But as you can imagine writing these grammars to cover
Speaker 2: all the different ways a user might phrase.
Speaker 1: Something flights to London, show me London flights. I want
Speaker 1: to flight to London exactly.
Speaker 2: Designing handcrafted grammars for spontaneous, unpredictable speech is incredibly hard
Speaker 2: and very time consuming. That's the big challenge.
Speaker 1: So what's the alternative.
Speaker 2: That's where the second type comes in. Statistical grammars, or
Speaker 2: more broadly, statistical NLU.
Speaker 1: Models learn from data.
Speaker 2: Yes, these aren't written by hand. They're trained on vast
Speaker 2: amounts of real world language data using machine learning techniques, and.
Speaker 1: The advantage is they can be much more flexible handle
Speaker 1: variations you didn't explicitly code for.
Speaker 2: That's the key benefit. Because they work based on probabilities
Speaker 2: and patterns learn from how people actually speak, they can
Speaker 2: often handle more or irregular wording, synonyms, even slightly ungrammatical
Speaker 2: inputs that would break a strict handcrafted grammar.
Speaker 1: What's the downside?
Speaker 2: The main one is they require huge data sets to
Speaker 2: train effectively, and access to these trained models often comes
Speaker 2: via cloud services. The book mentions a service from a
Speaker 2: company called Maluba as an example of a real world
Speaker 2: statistical NLU system available around that time.
Speaker 1: And that kind of system tries to identify the core
Speaker 1: intention and the relevant details the entities.
Speaker 2: Precisely, you give it a phrase like what's the weather
Speaker 2: in Belfast for tomorrow, and a statistical NLU system could
Speaker 2: analyze it and return something structured like categories whether action is,
Speaker 2: check status, and the entities are location, billfast, and date tomorrow,
Speaker 2: maybe even resolving tomorrow to the actual calendar date. It's
Speaker 2: focused on extracting that core meaning, often regardless of the
Speaker 2: exact sentence structure used.
Speaker 1: Does the book include a library to help developers work
Speaker 1: with these different grammar types.
Speaker 2: It does. An NLU lib. It contains classes for handelling
Speaker 2: those handcrafted grammars, parsing the XML definitions into Java objects,
Speaker 2: inverting the rules into patterns, often using regular expressions behind
Speaker 2: the scenes, and then using Java's matching tools to check
Speaker 2: user input against these patterns. It also extracts the semantic
Speaker 2: information based on those tags we talked.
Speaker 1: About, and for the statistical ones.
Speaker 2: The library also incling code demonstrating how to connect to
Speaker 2: external statistical NLU services like that mlluda API, sending the
Speaker 2: user's text and parsing the structured semantic result that comes back, assuming,
Speaker 2: of course, the developer has API access.
Speaker 1: And is there a demo app to play with this?
Speaker 2: Yes, a grammar test app. It's quite useful. It lets
Speaker 2: you input text, either typing it or using the results
Speaker 2: from ASR, and then test that input against either a
Speaker 2: handcrafted grammar file you provide or by sending it off
Speaker 2: to the statistical.
Speaker 1: Service so you can see the difference exactly.
Speaker 2: It shows you whether the input is considered valid according
Speaker 2: to the grammar and more importantly, what semantic information or
Speaker 2: structured representation some extracts. It's a clear way to see
Speaker 2: the different capabilities and outputs of the two approaches to
Speaker 2: understanding language.
Speaker 1: Okay, this is really building up. We've gone from basic
Speaker 1: speaking and listening to simple commands, structured form filling, and
Speaker 1: now understanding meaning with grammars and NLU. How do we
Speaker 1: make these voice apps even more robust and user friendly,
Speaker 1: maybe for a wider audience or in different situations.
Speaker 2: Well, two key aspects the book covers next are multi
Speaker 2: linguality and multimodality.
Speaker 1: Multi linguality supporting different languages seems obvious, but important.
Speaker 2: Absolutely essential if you want your app to reach a
Speaker 2: global audience. It means being able to use TTS and
Speaker 2: ASR and languages other than.
Speaker 1: Just the default How do developers handle that.
Speaker 2: They specify the languages using standard codes like ISO six
Speaker 2: three nine tosh one codes N for English as for
Speaker 2: Spanish and so on. The Android system provides ways again
Speaker 2: using intents to check which languages are actually supported or
Speaker 2: installed on the user's specific device.
Speaker 1: Because not all devices might have all languages pre installed, right, and.
Speaker 2: This ties directly into the broader concept of localization in
Speaker 2: Android development, you know, providing different text strings, images layouts,
Speaker 2: and resource folders like resvalues for Spanish users versus Ree's
Speaker 2: values and for English users. It's about adapting the whole
Speaker 2: app experience. The book has a very simple silly parrot
Speaker 2: app just to show switching TTSASR language.
Speaker 1: Okay, so that's multiple languages. What about multimodality sounds complex?
Speaker 2: It just means combining voice interaction with the traditional graphical
Speaker 2: user interface, the GUI, the buttons and screens we're used
Speaker 2: to tapping.
Speaker 1: On ah, voice and touch working together. Why do that?
Speaker 2: Because sometimes tapping is just easier or faster than speaking
Speaker 2: for certain inputs, or users might want visual feedback confirming
Speaker 2: what the system understood, or maybe they start by voice
Speaker 2: and when finished by touch or vice versa. The idea
Speaker 2: is to create a seamless link, a link between what
Speaker 2: between the fields of information we were talking about in
Speaker 2: form filling dialogues and the visual elements on the screen
Speaker 2: like drop down lists, spinners and android lists. You can
Speaker 2: scroll list views, radio buttons, checkboxes, text entry fields, edit text.
Speaker 1: Okay, so how does that work in practice?
Speaker 2: Imagine you have a field in your voice dialogue for say, urgency,
Speaker 2: with options low, medium, high. You might also have radio
Speaker 2: buttons on the screen for the same options. Right, If
Speaker 2: the user says medium urgency, the app not only fills
Speaker 2: that voice field internally, but it also automatically checks the
Speaker 2: meeting radio button on.
Speaker 1: The screen ah synchronization exactly.
Speaker 2: And conversely, if the user taps the high radio button
Speaker 2: on the screen, the app knows that the urgency information
Speaker 2: has been provided, so the voice dialogue system shouldn't ask
Speaker 2: for it orally anymore. The state is shared.
Speaker 1: That sounds really useful, but potentially complex to manage.
Speaker 2: It requires careful design. Grammars become important here too, to
Speaker 2: ensure that the voice input for a specific field actually
Speaker 2: matches one of the valid options available in the corresponding
Speaker 2: GUI element, Like the items.
Speaker 1: And a drop down list, and the book provides hell
Speaker 1: for this too.
Speaker 2: Another library, Yes, building on the form filling one, a
Speaker 2: multimodal form FILLIP. It extends the basic form library by
Speaker 2: adding grammar checking within the dialogue flow, so it only
Speaker 2: accepts voice input that actually matches the grammar defined for
Speaker 2: the current field. And crucially, it includes methods like oral
Speaker 2: to gee and GI toral specifically designed to synchronize the
Speaker 2: state between the internal voice feels and the visual GUI elements.
Speaker 1: Is there an example app for this?
Speaker 2: They use a mock send message app as a demonstration.
Speaker 2: It lets the user provide details for sending a message
Speaker 2: recipient urgency, maybe the message body itself, either by speaking the.
Speaker 1: Information following the form filling prompts.
Speaker 2: Right, or by interacting directly with the GeOI elements on
Speaker 2: the screen, like picking a contact from a list or
Speaker 2: tapping a radio button for urgency. The app keeps track
Speaker 2: of the information consistently, regardless of whether it came via
Speaker 2: voice or touch. It really highlights how voice and touch
Speaker 2: don't have to be separate, isolated interaction modes. They can
Speaker 2: compliment each other within the same task.
Speaker 1: Okay, this is quite a journey We've layered on all
Speaker 1: these capabilities speaking, listening, simple commands, handling, ambiguity, multistep forms,
Speaker 1: really understanding language with Grammars in NLU, adding flexibility with
Speaker 1: multiple languages, and combining voice with the screen through multimodality.
Speaker 1: What happens when you take all these pieces and integrate
Speaker 1: them into one coherent system, Well, that's when.
Speaker 2: You get into the realm of virtual Personal assistance.
Speaker 1: Vpas AH the serious, the Google Assistance, the Alexis of
Speaker 1: the world exactly.
Speaker 2: These are the conversational agents we're much more familiar with today.
Speaker 2: They really represent the culmination of all these underlying technologies
Speaker 2: we've been discussing, designed to understand potentially complex requests and
Speaker 2: perform a whole range of tasks.
Speaker 1: And the fundamental challenge for a VPA trying to bring
Speaker 1: everything together must be accurately figuring out the user's intention
Speaker 1: from whatever they happen to say, however they say it.
Speaker 2: That's the core of it. And as we saw when
Speaker 2: discussing Grammars in NLU, you can approach this understanding in
Speaker 2: different ways. You might try to classify the user's input
Speaker 2: using statistical methods trained on lots of.
Speaker 1: Data, which is good for more open ended questions or requests.
Speaker 2: Right, or you might use more structured grammars for specific commands,
Speaker 2: extracting the core intent in any relevant details or parameters.
Speaker 2: Often modern systems use a hybrid approach.
Speaker 1: And once the intention is hopefully understood, the VPA needs
Speaker 1: more logic. Right, it needs a system to decide what
Speaker 1: to do next.
Speaker 2: Definitely, it needs dialogue management. Based on the understood intent
Speaker 2: and the current state of the conversation, the dialogue manager
Speaker 2: decides the next action. Does it need to ask a
Speaker 2: clarifying question, Does it have enough information to perform the task?
Speaker 2: Does it need to access some data?
Speaker 1: And then it needs to generate a response.
Speaker 2: Yes, response generation figuring out what to say back to
Speaker 2: the user via PTS or what information to display on
Speaker 2: the screen, or what action to actually perform on the
Speaker 2: device or via web service.
Speaker 1: This whole idea of conversational AI has a long history,
Speaker 1: doesn't it even before smartphone?
Speaker 2: Oh yeah, it goes way back. The book even mentions
Speaker 2: early chatbots like Eliza from the nineteen sixties, but for
Speaker 2: building more sophisticated vpas around the time the book was written.
Speaker 2: It focuses on using a specific platform called Pandora Bots.
Speaker 1: Pandora Bots, what's that?
Speaker 2: It's a platform still reunt today actually for creating and
Speaker 2: hosting conversational agents or chatbots uses a language called AML.
Speaker 2: AML Artificial Intelligence Markup language is based on XML, and
Speaker 2: it lets you define the bot's conversational behavior using categories.
Speaker 2: Each category has a pattern which is basically.
Speaker 1: What the user might say a potential input phrase.
Speaker 2: Right, and a template which defines how the bots should respond.
Speaker 2: You can use wild cards like in the patterns to
Speaker 2: make them more flexible, matching multiple variations of user input,
Speaker 2: and there are special tags like srey that let you
Speaker 2: redirect from one pattern to another, helping to handle synonyms
Speaker 2: or rephrase requests without duplicating logic.
Speaker 1: What happens if the user says something that doesn't match
Speaker 1: any pattern?
Speaker 2: Good question? AML has the concept of an ultimate default category,
Speaker 2: a fallback pattern that matches anything else, usually triggering a
Speaker 2: response like sorry, I didn't understand that or could you rephrase?
Speaker 2: It's crucial for making the bot seem less brittle.
Speaker 1: Okay, So the core conversation logic matching input patterns to
Speaker 1: output templates runs on the Pandora Bots platform. How does
Speaker 1: that connect to doing things on the Android device.
Speaker 2: This is where a really interesting feature mentioned in the
Speaker 2: book comes in, specifically designed for mobile vpas. It's the
Speaker 2: UB tag oh ob out a band. It's a special
Speaker 2: tag that the AML developer can embed within the box
Speaker 2: response template alongside the text the bot is supposed to say,
Speaker 2: how does that work? So the box template might have
Speaker 2: the text okay, I'll search the web for that for you,
Speaker 2: but hitting within the same AML template, there could also
Speaker 2: be an UB tag containing a command like search query
Speaker 2: the user asked for search okay, the Android app communicates
Speaker 2: with the Pandora bot online. It sends the user's transcribed
Speaker 2: speech from ASR. The Pandora bot finds the matching pattern
Speaker 2: and sends back the AML template response. The Android app
Speaker 2: then parses this response. It takes the regular tax part okay,
Speaker 2: I'll search and sends it to the TTS engine to
Speaker 2: be spoken. But it also looks for any ubtags. If
Speaker 2: it find one, like search, it intercepts that command and
Speaker 2: executes the corresponding action on the device, in this case
Speaker 2: launching a web search with a specified query wow.
Speaker 1: So the conversation, logic and knowledge stays centralized on the
Speaker 1: Pandora bot server define an AML, but the actual device
Speaker 1: actions searching, launching apps, making calls, opening URLs are triggered
Speaker 1: locally on the Android device by these hidden commands embedded
Speaker 1: in the bot's response exactly.
Speaker 2: It's a clever way to decouple the conversational intelligence from
Speaker 2: the specific device functionalities. The book provides a vp lib
Speaker 2: library to handle this communication, connecting to a specific Pandora
Speaker 2: bot online using its bodhead sending the ASR input, parsing
Speaker 2: the XML response from Pandora bots, looking for the spoken
Speaker 2: part often marked by a FAT tag, and checking for ubtags,
Speaker 2: and then running the course wonding functions on the device
Speaker 2: based on those tags. The book did note a limitation
Speaker 2: at the time that the gender of the TTS voice
Speaker 2: couldn't be controlled programmatically, which might affect the perceived persona
Speaker 2: of the bot.
Speaker 1: Does the book include sample vpas built using this Pandora
Speaker 1: bots approach?
Speaker 2: It does. It described three Jack, Derek and Stacy. Jack
Speaker 2: is based on a well known general purpose amail bot
Speaker 2: called Alice, designed for broad open ended conversation. Derek, on
Speaker 2: the other hand, is presented as a specialized bot. It
Speaker 2: trains specifically on a particular knowledge domain. The example uses
Speaker 2: FAQs about type two diabetes. This highlights how you can
Speaker 2: encode expert or domain specific knowledge using these AMMEL patterns
Speaker 2: and templates.
Speaker 1: So one generalist, one specialist, and Stacy.
Speaker 2: Stacy is basically Jack the general conversational bot, but enhanced
Speaker 2: with that oub tag functionality, so Stacey can not only chat,
Speaker 2: but can also actually control device functions like searching online
Speaker 2: or launching apps based on understanding those embedded commands within
Speaker 2: the conversation flow.
Speaker 1: That really brings it all together. If you look back
Speaker 1: at how the book illustrates the overall VPA structure, you know,
Speaker 1: starting with ASR capturing the audio, then spoken language understanding
Speaker 1: whether that's NLU, grammars or AML pattern matching to figure.
Speaker 2: Out the intent right the understanding part.
Speaker 1: Then dialogue management deciding the next step, response generation formulating
Speaker 1: the output, and finally TTS speaking the response.
Speaker 2: All while potentially connecting to external data sources, knowledge bases,
Speaker 2: and triggering those device actions via things like the ouptech.
Speaker 1: You can really see how all those individual building blocks
Speaker 1: we discuss, TTS, ASR, forms, grammars, multimodality fit together into
Speaker 1: that complete VPA system.
Speaker 2: It is the full picture. Now show how you assemble
Speaker 2: these components to create those conversational agency.
Speaker 1: And that really brings our deep dove to a close,
Speaker 1: doesn't it. We've journeyed all the way from the absolute fundamentals,
Speaker 1: the device's ability to speak using TTS and listen using ASR.
Speaker 2: The core building blocks, through.
Speaker 1: Building those first simple command interactions, bringing in ideas like
Speaker 1: similarity measures and confirmation.
Speaker 2: Making them a bit more robust, and.
Speaker 1: Structuring more complex multi turn conversations using form filling dialogues,
Speaker 1: managing the flow to gather.
Speaker 2: Information, getting that structured data, then.
Speaker 1: Diving into how to understand user input more deeply, moving
Speaker 1: beyond words to meaning using both handcrafted and statistical grammars via.
Speaker 2: NLU, extracting intent and entities, and.
Speaker 1: Then adding layers of flexibility through multilinguality supporting different languages
Speaker 1: and multimodality, seamlessly combining voice with graphical interfaces.
Speaker 2: Making the interaction richer and more adaptable.
Speaker 1: And finally we saw how all these technologies converge in
Speaker 1: the architecture of virtual personal assistance, using platforms like Pandora
Speaker 1: bots with AML and leveraging clever techniques like that OOB
Speaker 1: tag to bridge the gap between conversation and actually doing
Speaker 1: things on the device.
Speaker 2: Absolutely, This deep dive, even though it's rooted in a
Speaker 2: technical guide from over a decade ago, now really gives
Speaker 2: you a solid conceptual shortcut. It helps you understand the
Speaker 2: core challenges and the fundamental building blogs involved in bringing
Speaker 2: voice interaction to life on a device.
Speaker 1: Yeah, it lays out the pieces really clearly and considering
Speaker 1: all these elements together, from breaking down sounds and words
Speaker 1: to understanding complex intent, managing dialogue flow over multiple turns,
Speaker 1: generating responses, and having the ability to trigger pretty much
Speaker 1: any device function or connect to any web service. It
Speaker 1: really opens up your imagination, doesn't it, to the kind
Speaker 1: of truly personalized voice applications you could potentially create.
Speaker 2: Right, It makes you think beyond the general purpose assistance
Speaker 2: we mostly use today. What if a voice assistant wasn't
Speaker 2: just generic? What if it was deeply, deeply specialized.
Speaker 1: Yeah, Like, imagine a VPA that understands the incredibly specific vocabulary,
Speaker 1: the jargon, the unique needs of your particular hobby or
Speaker 1: your specific job, maybe using a custom built grammar or
Speaker 1: a finely tuned n LU model, and it could instantly
Speaker 1: connect you to niche online resources or internal databases that
Speaker 1: a general assistant wouldn't even know existed, Or.
Speaker 2: Think about something more personal, maybe an assistant that reads
Speaker 2: you recipes, but it uses curated audio samples to sound
Speaker 2: like a comforting, familiar voice, maybe even your grandmother's voice.
Speaker 2: If you had the recordings and it lets you navigate
Speaker 2: the cooking steps completely hands free, using just simple voice
Speaker 2: commands tailored to that specific.
Speaker 1: Task, that would be amazing. It makes you wonder what
Speaker 1: kinds of unique, genuinely helpful or even just wonderfully quirky,
Speaker 1: and specific voice interfaces are still waiting to be built,
Speaker 1: especially when you start combining these foundational technologies tts ASR
Speaker 1: NLU dialogue management, device control in new and unexpected ways.
Speaker 2: Exactly what's the potential beyond the mainstream assistance we interact
Speaker 2: with now? It definitely leaves you with something to think about.