The scientific literature presents several ways to approach the problem, but we were more specifically interested in the answer selection task. This task aims to predict the correct answer among a set of candidate answers.

It assumes that there is always a correct answer for each question. However, in a real Q&A system, sometimes we do not want to provide an answer, for example if a user asks an out-of-domain question. The answer triggering task offers this possibility.

System Definition

To accomplish the answer triggering task for a given question, the chosen implementation performs two subtasks:

1. A machine learning model (a classifier) ​​accepts as input a vector representation of the question and returns the probabilities by class. Each class is associated with a question-answer pair.
2. The highest probability is compared to a threshold to determine whether the answer will be returned or not.

Experiments were then carried out to find the best combination of vectorization model and classifier to perform the first subtask.

Experiments

For the experiments, the banking dataset BANKING77, created by the conversational solutions company PolyAI, was used. The vectorization model and classifier combinations were evaluated on the test set with the accuracy metric, which calculates the percentage of correct predictions.

Vectorization Models

Among the different vectorization models, the one that performed the best is Google’s Universal Sentence Encoder (USE). It is a neural network pretrained simultaneously on several semantic tasks which accepts a text as input and outputs a sentence embedding (a vector representation of the sentence). The pretraining is done on large text corpora like Wikipedia, which allows it to capture the semantic similarity of sentences never seen before, as shown in the example below.

Semantic similarity of sentences taken from BANKING77 with USE. Reference

Classifiers

The evaluated classifiers include a k-nearest neighbor classifier (KNN) and a neural network. The main advantage of the KNN over the neural network is that it does not need to be trained, it just memorizes the training data. Adding new questions to the model therefore does not require retraining. Another advantage is that its predictions are interpretable. To predict the class of a test example, the KNN finds its k-nearest neighbors and returns the majority class. The number of neighbors as well as the distance function are configurable hyperparameters.

To illustrate how a KNN works, a simplified example is provided below for a binary classification problem with 2-dimensional data.

Example for a KNN with k=3 and Euclidean distance. For the test example in gray, we will
predict the class in blue (majority class among the 3 nearest neighbors).

First, experiments were performed with a KNN with cosine distance. Afterwards, other experiments were carried out with a learned distance function (metric learning) in order to improve performance. The purpose of the algorithm was to bring together the examples of the same class and to distance the examples belonging to different classes. In both cases, the accuracy obtained with the neural network was superior, which led to the latter being chosen as the classifier.

System Evaluation

The experiments described above have established that the best model was the one combining the Universal Sentence Encoder with an MLP. To evaluate this model, a comparison was performed on the answer triggering task with the NLU intention classification models of Dialogflow ES and Rasa. To do this, a new “out_of_scope” intent containing questions about COVID-19 has been added to the BANKING77 test set. For all models, rejecting out-of-domain examples proved more difficult than properly classifying in-domain examples. However, overall, it was the USE model combined with a neural network that stood out. This evaluation has therefore demonstrated that this model can be used to develop an effective and efficient Q&A system.

For more details on the models used, the methodology followed and the results of the experiments, we invite you to consult this article.

The post Exploring Approaches for a Question Answering System first appeared on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

The post Exploring Approaches for a Question Answering System appeared first on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

Our banking IVR is a self-service application in which a caller can execute tasks like paying a bill or getting an account balance. The dialogue includes a loop that allows the caller to carry out tasks as many times as he or she wants:

The application accommodates expert users by allowing them to quickly complete their tasks:

It also adapts to less experienced users by providing them with a more directed dialogue when needed:

The IVR supports mixed initiative strategies like digressions:

As well as change requests or corrections:

The dialogue also handles error recovery, including errors that are specific to the voice channel:

This is only an overview of the patterns handled by our dialogue model; several others are already implemented. We are also planning on adding other dialogue strategies, such as cancelling a task or handling more complex corrections and change requests.

Implementation: why we opted for a deterministic approach

Early on, we decided to rely on a deterministic approach for our use cases. The main reasons why machine learning was less adapted to our current use cases are as follows:

• Task related dialogues are predictable and relatively directed
• Tasks can be executed repeatedly and the application behavior should be identical every time
• Tasks should be interrupted and resumed reliably
• Tokens of information that are collected for a given task are not relevant for another
• Tasks are independent and must not interact with each other

In addition, our requirements for error recovery and change management were rather strict.

Deterministic approach

Our deterministic approach consists in managing actions using a stack. A stack is a data structure of type last in first out; in other words, the last item added to the stack is the first one to be removed. The action at the top of the stack is the one in focus. When we add an action to a stack, it becomes in focus. When the action in focus is completed, it is removed from the stack and the dialogue goes back to the previous action. This allows to interrupt and resume actions in a predictable and robust manner.

This can be illustrated as follows:

Here is an example of the state of a stack for a dialogue in which the user interrupts the ongoing action (digresses) to ask for a list of their accounts. Once the digression to hear the list of accounts is completed, this action is removed from the stack and the focus comes back to the previous action, that is, the bill payment action.

In addition to allowing us to manage digressions elegantly, our approach allows us to define isolated contexts for each action. This ensures that they remain independent.

Next are some additional details on how this was implemented using Rasa.

Custom policy

In Rasa, a policy is what allows to predict/specify the next action to be executed in the dialogue depending on context (tracker). Out-of-the-box, Rasa offers a combination of deterministic and machine learning based policies. It is also possible to create our own policy. This is what we did by implementing a deterministic policy that alternates between waiting for the next user input and triggering an action used as an action manager (responsible for managing the action stack). Since we do not make predictions as to what action must be executed, our policy does not use stories and does not require a training phase. This is the only policy that we use.

Action manager

We have created an abstraction to manage the action stack that was described above. The stack is a complex object that we store in an unfeaturized slot. The action to be executed depends on the user’s input as well as the state of the stack.

Custom dialogue patterns

One frequent dialogue pattern is information token collection, or slot-filling. This pattern is used, for instance, by the bill payment action. Rasa provides an action for this pattern: the FormAction. However, we needed to support other, more complex patterns than what is offered by the FormAction, for example: slot confirmation when the speech recognition confidence score is low, final confirmation at the end of an action, etc. We have therefore created a custom class “Task” that handles these more complex patterns. Some of our actions inherit this class. We appreciate that Rasa offers the flexibility that we need to implement our own dialogue management strategies.

Unit test framework

Since we have implemented our dialogues using a deterministic approach, we were able to build our own unit test framework to test our dialogues. This allowed us to increase our application’s reliability.

Next steps

Although we have been relying on a deterministic approach to develop our banking use cases, we are also currently experimenting with machine learning to develop dialogues for different use cases.

• Here are some of the next items that we will explore:
• Use the Recurrent Embedding Dialogue Policy to support uncooperative user behavior
• Use Rasa X to learn from real conversations
• Create more natural dialogues by using recorded prompts instead of TTS
• Try to integrate machine learning in our deterministic model

We will share our experience in future posts as we move forward. Stay tuned!

The post Developing Conversational IVR Using Rasa Part 3: Dialogue Management first appeared on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.

The post Developing Conversational IVR Using Rasa Part 3: Dialogue Management appeared first on AI Virtual Voice Experts with Google Dialogflow CX - CCAI - Nu Echo.