Based on our experience, we would also describe such a simple transformation as magic!

Through this blogpost, I will illustrate why with a few counterexamples.

Generating the output

Presenting complex information

Within a chatbot, text information can be enriched with images, hyperlinks, slideshows, etc. Some use cases such as navigation assistance or purchase recommendations would seem impossible to implement without those tools.

In other cases, several voice interactions would be required to reach the same result as a single visual output. For example, here is my best shot at transforming the output of a appointment scheduling chatbot for a voicebot: 

Trail of previous interactions

What does a chatbot do if the user is not paying attention, has poor memory, or has forgotten to put on their glasses? Nothing! The output remains there for the user to re-read as they see fit, which makes certain cases that are absolutely necessary in a verbal interaction become completely useless in a written conversation:

Persona and voice features

The persona (demographics, language level, personality) of the virtual agent, as well as its consistency, are important in both modes. While in text mode you have to think about the visual output of the chatbot, in voice mode, you have to look for a voice that represents the desired characteristics, while being natural, and this can limit our options. For example, trying to create an informal voice agent can be near impossible, especially when using TTS instead of a recorded voice (which also has its limitations).

Support of multiple channels

Finally, even if our use cases are channel-agnostic, our personna very simple and our agent very talkative, it is clear that we must at least be able to play different messages depending on the channel so that SSML is included in audio messages. Unfortunately, some dialog engines hardly support multiple channels and this can greatly increase the challenges of implementing a common agent for both voice and text.

Input interpretation

“What about the other way around? The user won’t send images or carousels of images to the chatbot. For sure, interpreting the input can’t be that different.” I will answer with a dramatic example. Let’s look at Bob who is trying to express what he needs to a vocal agent:

Of course, Bob and his legendary bad luck are not real, but the cases I have presented are taken from real-life examples. Even though some STT models can now ignore “mhms”, noises and secondary voices, the transcription will still have its share of errors.

Uncertainty

There are ways to reduce these errors or their impacts, whether it’s through the configuration of the engine, systematic changes of the transcription, or the adaptation of the NLP model to the sentences received. There remains, however, an additional uncertainty related to the STT which must be taken into account in the development of a voice application.

Strategies for dealing with uncertainty

To increase our confidence in the interpretation of the input, we will use more strategies for dealing with uncertainty in the dialogue of a vocal agent than in the dialogue of a textual agent. 

For example, we can think of:

• Add a step to explicitly or implicity confirm an intent or an entity
• Add a step to disambiguate the input when intentions are too similar
• Enable changes or fixes

Choosing use cases

Addresses, emails or people’s names are difficult pieces of information to transcribe correctly for many reasons, but they present lesser challenges in writing. If some of these pieces of information are critical for a use case, it could be very complex, risky, or inappropriate for the user experience to implement it though a vocal agent..

Real-time management

The last big difference between voice and text conversations is time management. A text conversation is asynchronous: the input is received in one block, and the response that follows is sent in one block. The audio, on the other hand, is transmitted continuously, so the time must be managed accordingly.

Short response time and user experience

In a vocal conversation, we are expecting a response within a few tenths of a second, while in text mode, it is completely normal to wait for much longer. Long silences on the phone are uncomfortable, and even if it is possible to play sounds or music-on-hold, between two interactions, the “…” hint cannot be replaced. It is therefore much more critical to ensure that the system is fast and to warn the user in case of a longer operation in voice mode.

Interruptions

Because voice output has a duration, the user can try to interrupt a voice agent. Supporting interruption correctly involves additional technical complexity, but also has additional impact on the dialogue. For example, we want to make the assumption that if the user says “yes” when presenting several options, this means that he chooses the first one, and we will support this case.

User Silence

Although a virtual agent isn’t discomforted by silences, the treatment of what is commonly called a no-input differs greatly depending on the mode of communication. In a voice conversation, a few seconds of silence usually means the user is hesitating or their voice is too low; an appropriate help message will therefore be played.

In text mode, it is useless to harass the user with error messages because the absence of input is treated like any inaction on a website: after a determined time, the user will be disconnected if necessary, and the conversation is ended.

So, finally…

How then does one answer the question: “What can be reused from a voice agent to create a chatbot or vice versa?” The answer is very nuanced and a little disappointing. Switching from a voice agent to a chatbot will generally allow more reuse because the former is generally more restrictive: perhaps it will be enough to adapt the messages a little, to add or remove a few dialogue paths.

However, in both cases, it is important to take a step back and re-evaluate our use cases and our persona: are they appropriate, feasible and realistic on this new channel? For what comes out of this questioning, business rules and high-level flows of the dialogue can probably be reused. The NLU model (textual data, organization of intentions and entities) and the messages of one may serve as a basis for the other, but will be subject to change. Indeed, the approach will have to be adapted to the results of user tests and data collection, so that the user experience does not suffer in favor of the simplicity of development.

The post Voice agents VS. Chatbots: Where does the difference lie? first appeared on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

The post Voice agents VS. Chatbots: Where does the difference lie? appeared first on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

Error handling in vanilla Dialogflow: follow-up intents and fallbacks

Dialogflow uses fallback intents when it fails to associate a user input with an intent. This, coupled with the follow-up intent mechanism, allows for contextual error handling. For example, suppose a very simple agent containing these intents :

DoTheThing is a normal intent that will be triggered by specific user inputs (in this case, a training phrase for this intent could be, for example, “I want to do the thing”). Intents can also be triggered by events, but it’s not really relevant to the current discussion.

The DefaultFallbackIntent is created automatically when you generate a new Dialogflow agent, and is triggered when Dialogflow can’t match the user input with anything else. You can think of the DefaultFallbackIntent as a safety net that will catch everything, but only as a last resort. Custom fallback intents can also be created, as we’ll see.

In Dialogflow, it’s possible to declare intents as follow-up to other intents. To continue with our same example, let’s suppose that the agent ask the user if he’s sure that he wants to do the thing when DoTheThing is matched:

DoTheThink-yes and DoTheThing-no are follow-up intents: they can only be matched if DoTheThing was previously matched. The way Dialogflow determines that is by keeping track of contexts. Specifically, when DoTheThing is matched, a new context, DoTheThing-followup, is generated. This context is then considered as one of the conditions to trigger DoTheThing-yes or DoTheThing-no, as we can see in this screenshot:

Finally, there is the follow-up fallback, DoTheThing-fallback. This kind of fallback works the same way as the DefaultFallbackIntent, but will only be operational when a specific context is active (DoTheThing-followup), like with any follow-up intent.

This allows a Dialogflow agent to have contextual error messages for virtually any situation: simply create follow-up fallbacks for every point in your dialogue where the user could say something that is not supported by your agent (pro-tip: most of the time, it’s all the time). You can even implement progressive error handling with this, since follow-up fallback intents can be daisy-chained to ensure proper escalation of error prompts:

Here, if a user triggers DoTheThing and then says something that is not recognized, DoTheThing-fallback will handle the situation. If the user then says something that is contextually correct (something that would match DoTheThing-yes or DoTheThing-no), the dialogue will continue normally. If they say another unsupported input, the agent will use the next follow-up fallback, DoTheThing-fallback-fallback.

The problem with slot filling error handling

So all is good, right? After all, we just demonstrated that you can do anything you want with follow-ups and fallbacks. Well, almost anything. Most notably, this approach will not work with slot filling. But before we tell you why and what can be done about it, we need to take a closer look at how slot filling is handled in Dialogflow.

First of all, what is slot filling? It is a simple conversational pattern where a bot is given a list of information to obtain (or slots to fill). Usually, each of these slots will correspond to an entity that the chatbot will try to extract from the user’s input. When a slot is considered filled, the bot will simply move to the next slot, until they’re all filled.

In Dialogflow, slot filling is tied to a given intent. In other words, a user will first need to trigger a specific intent for the agent to start the slot filling. One important detail to note is that the agent will stay on the same intent for the duration of the slot filling.

Let’s see how it works with this image taken from Dialogflow’s documentation:

The agent is given a list of three slots to fill : number-int, color and size, each of a different entity type. When the user triggers the intent that contains the slot filling, the agent will ask the user for each slot, in order. By default, the question is in the form “What is SLOT_NAME?”, but customized prompts can be defined for each slot.

What constitutes an error in slot filling is when the agent is not able to extract the expected entity from a user input. In this situation, by default, the Dialogflow agent will simply repeat the question until it receives a valid answer. Forever.

Error handling in slot filling is problematic because we can’t use the strategies that work in other circumstances (follow-up intents and fallbacks), since slot filling always happens within a single intent, and these strategies are built on top of the intent detection mechanism. It results in a very basic error handling for slot filling, and while it at least provides the possibility of continuing the dialogue, there is simply no error message guiding the user, which is not really interesting in terms of conversational UX.

Let’s try to fix that.

Detecting slot filling error context

(Please note that for the purpose of this article, we’ll suppose that the fulfillment is deployed in a serverless execution environment such as Google Cloud Function, which must remain stateless. Other approaches are of course possible.)

The first step to implement error handling in slot filling is to properly configure your agent to use your fulfillment webhook, and to activate calls to your fulfillment for the slot filling in your intent:

That was easy enough. The next step is to make your fulfillment code detect occurrences of errors in the slot-filling. The problem is that Dialogflow doesn’t communicate that directly. The only information that is clearly exposed in the fulfillment is the detected intent (which doesn’t change for the duration of the slot filling). But you can also extract from the contexts the current slot that the dialogue is trying to fill (more on that later). Even if there is no mention of any failed entity extraction, we already have everything we need to implement slot filling error detection.

The idea is to use contexts to leave a trace of every time slot filling is attempted for any given slot. Here are the main steps for slot filling error detection:

1. Detect if you are in slot filling
2. If so, construct an error ID from the intent’s name and the current slot’s name
3. Check if a context with that name already exists:
   1. If it doesn’t, create it and continue as normal
   2. If it does, you just detected an error

How does this work? Well, by making sure to create a custom context every time your agent tries to fill a slot, you leave evidence of that particular attempt. If the slot is properly filled, the agent will move to the next slot, and your fulfillment will generate another error ID (different from the one from before, since the slot is different). But if a call to your fulfillment is made with your custom context already created, it means that the same slot is trying to be filled again, which means that the user didn’t respond correctly to the slot query. In other words, you just detected an error in slot filling!

Let’s see how each step can be implemented:

One easy way to validate that you are indeed in slot filling is to use the WebhookClient and look in client.contexts for a context (automatically generated by Dialogflow) whose name starts with `${INTENT}_dialog_params_`, where INTENT is the name of the current intent, in lower cases. If the context doesn’t exist, you are not in slot filling. If it does, you are, and you can extract the name of the current slot from the end of that context’s name (immediately after “dialog_params_”).

And now, to create the error ID: since we can find the name of the intent and the name of the current slot, we can use this information to forge a unique error ID for each slot in the dialogue. Doesn’t have to be fancy, something like `error_${INTENT}_${SLOT}` will do the trick.

To check if our custom context already exists, we can use the following function from the WebhookClient:

client.getContext(errorID);

That’s pretty much self explanatory. As for context creation:

client.setContext({
name: `error_${INTENT}_${SLOT}`,
lifespan: 1
});

Here, we set the lifespan of the custom error context to 1, to make it disappear as soon as possible when it’s not needed anymore.

And that’s it for slot filling context error detection. From there, you already have most of the things you need to implement actual error handling: you just have to override the original bot response with a contextually appropriate message, which can be done with this simple command:

client.add(message);

Of course, you now need a way to retrieve or generate these contextually correct messages, from within the fulfillment. In other words, you need to do that programmatically. A good way to implement that while keeping your code clean and language agnostic is to leverage the error ID you created as a key and use a localization library to consolidate all your error messages. From there, it’s a simple matter of calling the localization library with the error ID to have it resolved to a proper error message.

Going progressive

Now that we have implemented contextual error handling for the slot filling, let’s improve it by making it progressive. An easy way to do that is to add a count parameter to your custom context:

client.setContext({
name: `error_${INTENT}_${SLOT}`,
lifespan: 1
parameters: { count: 1 }
});

From there, if an error occurs, you can append the count to the error ID (`${errorID}_${COUNT}`) and use that new ID template to identify your error messages in your localization library (or whatever similar strategy you use) and thus have any number of distinct error messages for any given context. Simply don’t forget to increment the count value in the context each time you detect an error by calling setContext again. You may also want to implement a cap to that number, to avoid having a user reaching an error message that was not implemented.

Next steps

This concludes our short discussion on slot filling error handling in Dialogflow. Of course, the proposed implementation is really bare bone and doesn’t support really useful functionalities like constraints, disambiguation (“Did you mean A or B?”) or certain particular situations (things could be a little more complex if you decided to activate the Knowledge Base feature, for example). These issues will most probably be covered in one of our future blog posts about Dialogflow. Thanks for reading!

The post Dialogflow distilled: On error handling first appeared on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

The post Dialogflow distilled: On error handling appeared first on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

Let’s say that one of your chatbot requirements is to perform a banking payment to a specific merchant. This is a fairly standard transaction involving a single intent capturing a few slots such as amount, account, merchant and a date.

> Read full version on Medium

The post Dialogflow Distilled: On Preemptive Slot-filling versus Branching first appeared on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

The post Dialogflow Distilled: On Preemptive Slot-filling versus Branching appeared first on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

What makes it great!

Browsing around Dialogflow’s website, you can easily find the benefits. Some are accurate, others are less evident as true benefits or are lacking as of yet. Let’s start by considering what some of the true benefits are that we have been able to experience and take advantage of:

Quick and easy to start building: Yes! You can truly create your first bot within a matter of minutes. I think my mother did one in, like, 16 minutes, while cooking her famous apple jelly. You login to the console, create an agent project, create one or two intents, and then try it out within the console. Instant reward. A few more clicks to activate the Phone Gateway integration and then you can, yes, call in (assuming your bot is en-US for now). Small dopamine rush.

Built on Google infrastructure: From within a project, you can tap into the rich Google Cloud ecosystem. Conversation logs get pushed over Stackdriver, fulfillment is only a few Google Cloud Functions or Firebase Functions away, security and containment are directly handled as part of IAM, while speech-to-text and text-to-speech relies on Google Cloud respective APIs. And frankly, although we are using some beta features, the platform is quite stable.

Easy to scale: Seriously! This is an additional benefit to the point above. No need to worry about any sort of viral effect your bot might experience. There’s not even a dial or knob to tweak or turn on. It’s essentially all done behind the scenes. Coupled with Google Cloud Functions for serverless fulfillment stage and you’re golden.

Strong natural language understanding (NLU) capabilities: So far so good on this front. We definitely have yet to push the limits but its context-based approach appears to be robust. More on that to come in future posts.

Where it might fall a little short…

> CLICK HERE to read the full blog post

The post Dialogflow and Beyond first appeared on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

The post Dialogflow and Beyond appeared first on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.