I'm facing this issue with geo places. For example.
USA, United states, United states of America, US.
Luis is able to detect them as built in entity of geographyV2
but I want NLP to pass all the similar terms of the user input. Ex user said US we send back geographyV2 entity and the similar synonyms of it , can luis do that by any chance? I'm desperate for this
It does not appear this is possible using the built-in geographyv2 entity. The entity just returns the value that was entered and the category is belongs to (country/region, city, point of interest, etc.). If you use the verbose response it will also give you additional information such as the starting index and length. But I can't find any option to have it return all related values/alternate names.
GeographyV2 prebuilt entity for LUIS app
Even if you create a custom list entity in LUIS with a normalized value and synonyms, I don't think you can get the full list of synonyms for that entity, only the normalized value. So if you need to see a list of alternate spellings, you're going to need some other function or API. I did a brief search and there's not much out there. The best I found was REST Countries, which you can pass a partial or alternate name in, and it will give you a bunch of information in the response. You may have to do some legwork, though, because it will give multiple matches (e.g. a search for United States returns results for United States of America and United States Minor Outlaying Islands). This is as close as I can figure out on how to solve your issue.
Related
I am building chatbot for universities. In the training phrase, how can I add the generic parameter without specify it. For example, if I need to ask the specific course offered this semester "Is CSCI1000 offered this semester?". And my database had more than 100 courses. So I need to type more than 100 training phrases for each courses with that question. Can I put it in generic form like this "Is course_name offered this semester?". course_name will match with the specific course name that user type in. I know DialogFlow is deprecated template mode. Do we have any other way to make it simpler? Please help.
In DialogFlow (DF) you can do more than that. Based on your provided information, let's assume you have 100 courses ID and names in your database. Here's the steps you should take to make DF recognize them:
Create an entity for your courses name (course_name)
Import course id (CSCI1000) as entity name (you can also import the name of the course as synonym, for situations where the user asks "is Computer Science offered this semester?"). You have to implement all this in your back-end
At this point you have an Entity with name all your courses from your database (so whenever the used types the name of that course DF will recognize it). Note that you can use other tricks for creating entity using regular expressions.
The final step, you have to create one intent 'Open courses' and and add Parameter, and for the entity use your created entity above (#course_name). Then type few training phrases. DF will detect courses id.
This is what came to my mind since I have done something similar before. For your reference Link to DOC
Hope all is well during these challenging times!
I would like to put it out there with regards to the best practice on Extending Dialogflow Entities and tackling non US/UK english locale input.
My challenge was to get accurate location values for Suburbs and Cities (parameters) in Australia. My team has tested many different ways of doing so with just using the standard system entities but non was able to capture the above accurately. It seems most of the location based entity types are geared towards US and UK but not AU (and perhaps other countries non US/UK).
The team had to implement a custom entity that has the list of suburbs and cities for australia. A roadblock that we came across shows that you can only grab 10,000 max entities (else Dialogflow and AoG won't sync/update). Our procured list has more than that for suburbs and hence we have to do a fair bit of work trimming it down which we did.
Question is, are we doing it the right way -> custom entity with csv list imported? For instance should we use the training path instead and stick to Sys.Geo-City and Sys.location?
Yes, if your set of suburbs and cities can be organized in this way, uploading a CSV would provide better accuracy for your use case. If you are over 10K values for a given entity type, you can split it into several entities and allow an intent to match either entity.
I am using "list" entity. However, I do not achieve my expected result.
Here is what I have for LUIS intent:
getAnimal
I want to get a cat**[animal]**.
Here is what I have with LUIS entities:
List Entities [animal]
cat: russian blue, persian cat, british shorthair
dog: bulldog, german shepard, beagle
rabbit: holland lop, american fuzzy lop, florida white
Here is what I have with LUIS Phrase lists:
Phrase lists [animal_phrase]
cat, russian blue, persian cat, british shorthair, dog, bulldog, german shepard, beagle, etc
Desired:
When user enters "I want to get a beagle." It will be match with "getAnimal" intent.
Actual:
When user enters "I want to get a beagle." It will be match with "None" intent.
Please help. Your help will be appreciated.
So using a phrase list is a good way to start, however you need to make sure you provide enough data for LUIS to be able to learn the intents as well as the entities separate from the phrase list. Most likely you need to add more utterances.
Additionally, if your end goal is to have LUIS recognize the getAnimal intent, I would do away with the list entity, and instead use a simple entity to take advantage of LUIS's machine learning, and do so in combination with a phrase list to boost the signal to what an animal may look like.
As the documentation on phrase lists states,
Features help LUIS recognize both intents and entities, but features
are not intents or entities themselves. Instead, features might
provide examples of related terms.
--Features, in machine learning, being a distinguishing trait or attribute of data that your system observes, and what you add to a group/class when using a phrase list
Start by
1. Creating a simple entity called Animal
2. Add more utterances to your getAnimal intent.
Following best practices outlined here, you should include at least 15 utterances per intent. Make sure to include plenty of examples of the Animal entity.
3. Be mindful to include variation in your utterances that are valuable to LUIS's learning (different word order, tense, grammatical correctness, length of utterance and entities themselves). Highly recommend reading this StackOverflow answer I wrote on how to build your app properly get accurate entity detection if you want more elaboration.
above blue highlighted words are tokens labeled to the simple Animal entity
3. Use a phrase list.
Be sure to include values that are not just 1 word long, but 2, 3, and 4 words long in length, as different animal names may possibly be that long in length (e.g. cavalier king charles spaniel, irish setter, english springer spaniel, etc.) I also included 40 animal breed names. Don't be shy about adding Related Values suggested to you into your phrase list.
After training your app to update it with your changes, prosper!
Below "I want a beagle" reaches the proper intent. LUIS will even be able to detect animals that were not entered in the app in entity extraction.
Not able to identify simple phrases like "my name is not Harry, it's Sam".
It is giving me name as harry and company name as Sam, Since name and company name was required in the same sentence.
It should have taken name as Sam and prompted the user again for company name OR should have given complete fallback.
Hi and welcome to Stackoverflow.
Dude. This is not a simple phrase.
Negative questions are always very difficult to catch by Dialogflow.
Suppose I have a question like,
I want to check *google* revenue for the year *2017*
As you can see, google and 2017 are the entities.
But now in the same way if you say,
I don't want to check *google* revenue for the year *2017*
The chances of hitting that old intent is very high as dialogflow matches almost 90% of this sentence with your old sentence. So it might fail.
Hope you are trying to ask something similar to this.
Anyhow coming to your point, If company name and name are different entities, then
Two things you can avoid:
As everyone mentioned,check your entities. The values should not be present in both the entities. This will fail because dialogflow will not know whether it should treat 'Sam' as your name or company name.
If you are not using the values from an entity, instead using '$ANY', then It has a very high chance of failing. And If you are using Dialogflow's system entity like, $given-name, then that is also not preferred as it does not catch all the names. So avoid these entities.
Things you can try:
Train Train And Train. As you would be aware, the training section in dialogflow is pretty good. Train it a few times and it will automatically learn and master it.
But , please note: Wrong training will result in wrong results. It should be 100% accurate. Always check before you approve a training.
And try using webHooks, actions, and/or events to figuring your way out from an external API.
We’ve been having a great deal of difficulty with chatbot entities over-generalising on Api.ai, i.e. returning values that have not been specified for that entity when using the “Define Synonyms” feature on custom entities, even when the “Allow automated expansion” flag is turned off.
Our key example is an entity we use for confirming a user choice called confirm_accept. We had an entry: “that’s it”, with synonyms: “thats it”, “that is it”, “that’s it thanks”, “thats it thanks”, “that is it thanks”. This entity value was being returned unexpectedly in expressions where just a stray “it” was appearing.
In general, we have seen a lot of inappropriate entity generalisation which seems to indicate there is some form of stop word removal and stemming/lemmatization going on during entity identification... and which can’t be turned off.
This returns poor entity classifications, making it difficult to create entities for which very precise values are important, e.g. where a single word or character can make a big difference in meaning. Our key use case involves a lot of address processing, so it is important we get back only values we have specified.
Types of over-generalisations we’ve seen include:
inappropriate identification of determiners (a, an, the, this, that, etc.) as part of entities: as in “it” returning “that’s it”
stemmed words: as in stray mentions of “driving”, returning “drive” (a valid street type entity)
inappropriate plural stems: a stray mention of “children” returning “child”, or a stray “will” returning “wills” (which in our case “child” and “wills” are street name entities, so we don’t want “children” or “will” to be returned)
This is currently making it difficult to create a production quality chatbot using the Api.ai service.
Anyone had more luck at either getting a response from Api.ai or solving the over-generalisation problem?
Entities are meant to extract information from conversation:
API.AI's entities are meant to be used to extract data from conversational input not parse different phrases and parts of speech. For your examples (that’s it, thats it, that is it, that’s it thanks, thats it thanks, that is it thanks) all seem to indicate that the user's intent is to indicate that the last message from the API.AI agent was correct. For instances like these, it would be best to use these phrases as examples for an intent or an existing intent with other responses indicating that the user wants to indicate that the last response was correct.
API.AI captures entity tenses and plurals automatically: To address your other concern (driving entity, returning drive value, children returning child, or wills returning wills): API.AI intentionally captures different tenses and plurals of entities to provide a better experience for users who many not know the exact entities you've entered in your database. This allows users of your conversational app to have a natural conversation with your users and not require precise wording or