How to work with multiple intents on Microsoft's LUIS - nlp

Question

I'm working with LUIS and want to manage and deal not only with the top scoring intent but also with all others. In this specific situation occurs when someone inquires about two things in the same phrase.
For example: "I want to buy apples" ("Buy" intent) and "I want to sell bananas" ("Sell" intent) versus "I want to buy bananas and sell apples" ("buy" and "sell" intents on the same utterance).
The idea is to define a threshold that will accept as "valid" any intentions score above this confidence number.
During some tests I found out this can work if we have very few intents on the same utterance.
However if we increase the number of intents on the same utterance the results degrades very fast.
I included some examples to clarify what I mean: The output examples below were generated on a LUIS with 4 intents ("buy", "sell", "none" and "prank") and 1 entity ("fruit")
I want to buy apples ==>
{
"query": "i want to buy apples",
"topScoringIntent": {
"intent": "Buy",
"score": 0.999846
},
"intents": [
{
"intent": "Buy",
"score": 0.999846
},
{
"intent": "None",
"score": 0.2572831
},
{
"intent": "sell",
"score": 2.32163586e-7
},
{
"intent": "prank",
"score": 2.32163146e-7
}
],
"entities": [
{
"entity": "apples",
"type": "Fruit",
"startIndex": 14,
"endIndex": 19,
"resolution": {
"values": [
"apple"
]
}
}
]
}
I want to sell bananas ==>
{
"query": "i want to sell bananas",
"topScoringIntent": {
"intent": "sell",
"score": 0.999886036
},
"intents": [
{
"intent": "sell",
"score": 0.999886036
},
{
"intent": "None",
"score": 0.253938943
},
{
"intent": "Buy",
"score": 2.71893583e-7
},
{
"intent": "prank",
"score": 1.97906232e-7
}
],
"entities": [
{
"entity": "bananas",
"type": "Fruit",
"startIndex": 15,
"endIndex": 21,
"resolution": {
"values": [
"banana"
]
}
}
]
}
I want to eat a pizza ==>
{
"query": "i want to eat a pizza",
"topScoringIntent": {
"intent": "prank",
"score": 0.997353
},
"intents": [
{
"intent": "prank",
"score": 0.997353
},
{
"intent": "None",
"score": 0.378299
},
{
"intent": "sell",
"score": 2.72957237e-7
},
{
"intent": "Buy",
"score": 1.54754474e-7
}
],
"entities": []
}
Now with two intents... The score of each one starts to reduce aggressively
I want to buy apples and sell bananas ==>
{
"query": "i want to buy apples and sell bananas",
"topScoringIntent": {
"intent": "sell",
"score": 0.4442593
},
"intents": [
{
"intent": "sell",
"score": 0.4442593
},
{
"intent": "Buy",
"score": 0.263670564
},
{
"intent": "None",
"score": 0.161728472
},
{
"intent": "prank",
"score": 5.190861e-9
}
],
"entities": [
{
"entity": "apples",
"type": "Fruit",
"startIndex": 14,
"endIndex": 19,
"resolution": {
"values": [
"apple"
]
}
},
{
"entity": "bananas",
"type": "Fruit",
"startIndex": 30,
"endIndex": 36,
"resolution": {
"values": [
"banana"
]
}
}
]
}
and if we include the third intent, LUIS seems to collapse:
I want to buy apples, sell bananas and eat a pizza ==>
{
"query": "i want to buy apples, sell bananas and eat a pizza",
"topScoringIntent": {
"intent": "None",
"score": 0.139652014
},
"intents": [
{
"intent": "None",
"score": 0.139652014
},
{
"intent": "Buy",
"score": 0.008631414
},
{
"intent": "sell",
"score": 0.005520768
},
{
"intent": "prank",
"score": 0.0000210663875
}
],
"entities": [
{
"entity": "apples",
"type": "Fruit",
"startIndex": 14,
"endIndex": 19,
"resolution": {
"values": [
"apple"
]
}
},
{
"entity": "bananas",
"type": "Fruit",
"startIndex": 27,
"endIndex": 33,
"resolution": {
"values": [
"banana"
]
}
}
]
}
Do you know/recommend any approach that I should use to train LUIS in order to mitigate this issue? Dealing with multiple intents in the same utterance is key to my case.
Thanks a lot for any help.

Answer 1

You will likely need to do some pre-processing of the input using NLP to chunk the sentences and then train/submit the chunks one at a time. I doubt that LUIS is sophisticated enough to handle multiple intents in compound sentences.
Here's a sample code for preprocessing using Spacy in Python - have not tested this for more complicated sentences but this should work for your example sentence. You can use the segments below to feed to LUIS.
Multiple intents are not an easy problem to address and there may be other ways to handle them
import spacy
model = 'en'
nlp = spacy.load(model)
print("Loaded model '%s'" % model)
doc = nlp("i want to buy apples, sell bananas and eat a pizza ")
for word in doc:
if word.dep_ in ('dobj'):
subtree_span = doc[word.left_edge.i : word.right_edge.i + 1]
print(subtree_span.root.head.text + ' ' + subtree_span.text)
print(subtree_span.text, '|', subtree_span.root.head.text)
print()

Answer 2

If you know the permutations you are expecting you might be able to get the information you need.
I defined a single "buy and sell" intent, in addition to the individual buy and sell intents. I created two entities "Buy Fruit" and "Sell Fruit", each of which contained the "Fruit" entity from your example. Then in the "buy and sell" intent I used sample utterances like "I want to by apples and sell bananas", as well as switching the buy/sell around. I marked the fruit as a "fruit" entity, and the phrases as "buy fruit" and "sell fruit" as respectively.
This is the kind of output I get from "I want to buy a banana and sell an apple":
{
"query": "I want to buy a banana and sell an apple",
"prediction": {
"topIntent": "buy and sell",
"intents": {
"buy and sell": {
"score": 0.899272561
},
"Buy": {
"score": 0.06608531
},
"Sell": {
"score": 0.03477564
},
"None": {
"score": 0.009155964
}
},
"entities": {
"Buy Fruit": [
{}
],
"Sell Fruit": [
{}
],
"Fruit": [
"banana",
"apple"
],
"keyPhrase": [
"banana",
"apple"
],
"$instance": {
"Buy Fruit": [
{
"type": "Buy Fruit",
"text": "buy a banana",
"startIndex": 10,
"length": 12,
"score": 0.95040834,
"modelTypeId": 1,
"modelType": "Entity Extractor",
"recognitionSources": [
"model"
]
}
],
"Sell Fruit": [
{
"type": "Sell Fruit",
"text": "sell an apple",
"startIndex": 27,
"length": 13,
"score": 0.7225706,
"modelTypeId": 1,
"modelType": "Entity Extractor",
"recognitionSources": [
"model"
]
}
],
"Fruit": [
{
"type": "Fruit",
"text": "banana",
"startIndex": 16,
"length": 6,
"score": 0.9982499,
"modelTypeId": 1,
"modelType": "Entity Extractor",
"recognitionSources": [
"model"
]
},
{
"type": "Fruit",
"text": "apple",
"startIndex": 35,
"length": 5,
"score": 0.98748064,
"modelTypeId": 1,
"modelType": "Entity Extractor",
"recognitionSources": [
"model"
]
}
],
"keyPhrase": [
{
"type": "builtin.keyPhrase",
"text": "banana",
"startIndex": 16,
"length": 6,
"modelTypeId": 2,
"modelType": "Prebuilt Entity Extractor",
"recognitionSources": [
"model"
]
},
{
"type": "builtin.keyPhrase",
"text": "apple",
"startIndex": 35,
"length": 5,
"modelTypeId": 2,
"modelType": "Prebuilt Entity Extractor",
"recognitionSources": [
"model"
]
}
]
}
}
}
}
To make this work you would have to cater for all the possible permutations, so this isn't strictly a solution to discerning multiple intents. It's more about defining a composite intent for each permutation of individual intents that you wanted to cater for. In many applications that would not be practical, but in your example it could get you a satisfactory result.