I've written up a quick proof of concept console app to test out the functionality of the AzureSpell Cognitive Services product, however it doesn't seem to often detect obvious spelling mistakes.
Having experimented with recommendations through other SO answers, I've had limited success.
Even using the demo located at https://azure.microsoft.com/en-us/services/cognitive-services/spell-check/ produces no results.
For example, consider the following piece of text: "Currently growing my compny which is a UK based Online compny with clients across the world. Working since 2001 to help indivduals."
This produces no results. I've looked at regional settings, PROOF vs SPELL, character counts to no avail.
Has anyone had any success with this service, or, even better, does the above text snippet produce results for you?
Spell mode is working for me with your sample, see below:
The JSON result is:
{
"_type": "SpellCheck",
"flaggedTokens": [
{
"offset": 21,
"token": "compny",
"type": "UnknownToken",
"suggestions": [
{
"suggestion": "company",
"score": 0.9264452620075305
}
]
},
{
"offset": 55,
"token": "compny",
"type": "UnknownToken",
"suggestions": [
{
"suggestion": "company",
"score": 0.8740149238635179
}
]
},
{
"offset": 120,
"token": "indivduals",
"type": "UnknownToken",
"suggestions": [
{
"suggestion": "individuals",
"score": 0.753968656686115
}
]
}
]
}
Ok, so after a fair amount of trial I've had some success, which has solved some issues and created others. I've not been able to get a reliable result from Spell mode, but I have with Proof, however after adding a fairly short piece of text, it would again not report any results. Inspecting the API shows the text is encoded in the POST, removing both "%0D" and "%0A", line feed chars allows me to Proof long texts with success, which would be fine, however being UK based, lots of correct spellings are now flagged as incorrect as the PROOF mode is only available in the US. So, I've still been unable to solve getting a functioning SPELL result (which works for very short pieces of text). I understand the documentation states upto 130 chars for GET, but 10,000 chars for POST and my typical example POSTS are around 1,000 chars. Possibly a ticket with MS unless anyone has any ideas?
Related
I am leveraging the latest iteration of Azure's Custom question answering module in language studio in an external app that I've created, and I cannot figure out how to receive the actual source when the question is answered. I don't know if that's because you just can't right now or what, but in the actual API docs for the request/answer sample, the answer sample includes the source field - no matter what I've tried, I can't get it to show up.
Page where API doc is found - https://learn.microsoft.com/en-us/rest/api/cognitiveservices/questionanswering/question-answering/get-answers#knowledgebaseanswer
Quick example snippet of how I've adapted the API:
{
"question": "<question>",
"top": 3,
"userId": "<user>",
"confidenceScoreThreshold": 0.2,
"rankerType": "Default",
"filters": {
"metadataFilter": {
"metadata": [
],
},
},
"answerSpanRequest": {
"enable": true,
"confidenceScoreThreshold": 0.2,
"topAnswersWithSpan": 1
},
},
"includeUnstructuredSources": true
}
I understand the metadata bit has nothing there, I may add something later but as of now I'm not messing with the metadata aspect in language studio sources themselves.
At any rate, the bottom line is I don't see an option to display a source, and I don't get it back in the body of the request - yet I see it in the sample response in the API doc, so what gives, am I missing something?
We are currently running into an issue when expanding our current azure search features.
When we have the following string indexed in azure search:
AEDE190EACWWG4VGLDE02000UHKPT
And we search for that complete string, we are not able to find it.
However, when we only use 20 chars, we are able to find it.
So the with the string below, we are able to find it
AEDE190EACWWG4VGLDE
However, when adding just 1 more char it disappears again. And this is not only within our implementation. This is also in azure itself when entering this within the query string
The field is set up as
Retrievable
Filterable
Searchable
Anyone know how to solve this issue?
I tested your scenario now, and it works fine. I cannot reproduce the problem you have. You don't specify which analyzer you use, so I'm going to assume you use the standard analyzer.
Here is how I tested.
I create a new index with two fields Id and Ordcode.
I upload two records via Postman
"value": [
{
"#search.action": "mergeOrUpload",
"Id": "1",
"Ordcode" : "AEDE190EACWWG4VGLDE02000UHKPT"
},
{
"#search.action": "mergeOrUpload",
"Id": "2",
"Ordcode": "ABC123"
}]
I search for your the string AEDE190EACWWG4VGLDE02000UHKPT using searchMode=all, queryType=full. The response is as expected.
{
"#odata.context": "https://<search-service>.search.windows.net/indexes('dg-test-65143696')/$metadata#docs(*)",
"#odata.count": 1,
"value": [
{
"#search.score": 0.2876821,
"Id": "1",
"Ordcode": "AEDE190EACWWG4VGLDE02000UHKPT"
}
]
}
I also tried to reproduce via the Search Explorer in the Azure Portal, even with simple mode and any (the default).
search=AEDE190EACWWG4VGLDE02000UHKPT&$count=true&$select=Id,Ordcode
There is a limit on the tokens produced (depending on the analyzer you use), but it's not 20 unless you have defined a shorter max token length.
I am using https://learn.microsoft.com/en-us/azure/cognitive-services/form-recognizer/quickstarts/curl-train-extract to build a training model without using Labels.
The problem I am running into is when I run a file through the model (the file was used to train the model), it is not picking up the "table" part. What I mean is, there is no "tables" node.
From what I have seen, it should be able to build this as part of the JSON, but its breaking it down into super granular OCR, such as
{
"key": {
"text": "__Tokens__34",
"boundingBox": null,
"elements": null
},
"value": {
"text": "2 X 3/4",
"boundingBox": [
3.1181,
3.7292,
3.5278,
3.7292,
3.5278,
3.8583,
3.1181,
3.8583
],
"elements": null
},
"confidence": 1.0
}
Am I missing a flag or something?
Thank you in advance.
Seems like the table is not detected automatically with Train without labels, can you please share an image of the table, please remove any PII information. You can also try the Train with labels or the Layout API to see if it recognizes the table automatically.
I had the same problem, but I've noticed it working when I enable Full Text
we are building a product on LUIS / Microsoft Bot framework and one of the doubt we have is Person Name understanding. The product is set to use by anyone by just signing up to our website. Which means any company who is signing up can have any number of employees with any name obviously.
What we understood is the user entity is not able to recognize all names. We have created a phrase list but as per we know there is a limit to phrase list (10K or even if its 100K) and names in the world can never have a limit. The other way we are thinking is to not train the entity with utterances. However if we have 100s of customers with 1000s of users each, the utterances will not be a good idea in that case.
I do not see any other way of handling this situation. Probably I am missing something here? Anyone faced similar problem and how it is handled?
The worst case would be to create a separate LUIS instance for each customer but that's really a big task to do only because we cant handle names.
As you might already know, a person's name could literally be anything: e.g. an animal, car, month, or color. So, there isn't any definitive way to identify something as a name. The closest you can come is via text analysis parts of speech and either taking a guess or comparing to an existing list. LUIS or any other NLP tool is unlikely to help with this. Here's one approach that might work out better. Try something like Microsoft's Text Analytics cognitive service, with a POST to the Key Phrases endpoint, like this:
https://westus.api.cognitive.microsoft.com/text/analytics/v2.0/keyPhrases
and the body:
{
"documents": [
{
"language": "en-us",
"id": "myid",
"text": "Please book a flight for John Smith at 2:30pm on Wednesday."
}
]
}
That returns:
{
"languageDetection": {
"documents": [
{
"id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
"detectedLanguages": [
{
"name": "English",
"iso6391Name": "en",
"score": 1.0
}
]
}
],
"errors": []
},
"keyPhrases": {
"documents": [
{
"id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
"keyPhrases": [
"John Smith",
"flight"
]
}
],
"errors": []
},
"sentiment": {
"documents": [
{
"id": "e4263091-2d54-4ab7-b660-d2b393c4a889",
"score": 0.5
}
],
"errors": []
}
}
Notice that you get "John Smith" and "flight" back as key phrases. "flight" is definitely not a name, but "John Smith" might be, giving you a better idea of what the name is. Additionally, if you have a database of customer names, you can compare the value to a customer name, either exact or soundex, to increase your confidence in the name.
Sometimes, the services don't give you an 100% answer and you have to be creative with work-arounds. Please see the Text Analytics API docs for more info.
Have asked this question to few MS guys in my local region however it seems there is no way LUIS at moment can identify names.
Its not good as being NLP, it is not able to handle such things :(
I found wit.ai (best so far) in identifying names and IBM Watson is also good upto some level. Lets see how they turn out in future but for now I switched to https://wit.ai
I'm trying to use Freebase to search for multiple items at a time (using one API call). For example, if I have two items:
Robert Downey, Jr.
The Avengers
I want to query Freebase once and get back results for both items. Basically all I need is the mid for the top 3 or 4 results for both items. I would like to rely on Freebase's search API to provide disambiguation for topics. For example, I'd like to be able to search for "Robert Downey, Jr." with the abbreviation: "RDJ".
This is easy to do when searching one item at a time:
https://www.googleapis.com/freebase/v1/search?query=rdj
Making two calls like this would give me exactly what I'm looking for, but I would like to stay away from making these calls individually.
Reconciliation
I did run across the json-rpc call for reconciliation, and I have tried the following:
Endpoint: https://www.googleapis.com/rpc
POST body:
[
{
"method": "freebase.reconcile",
"apiVersion": "v1",
"params": {
"name": ["RDJ"],
"key": "api_key",
"limit":10
}
},
{
"method": "freebase.reconcile",
"apiVersion": "v1",
"params": {
"name": ["the avengers"],
"key": "api_key",
"limit":10
}
}
]
This works fairly well for Robert Downey, Jr in that I get a result of type /film/actor as I did using the search api. However, for The Avengers, I get a set of results with type /book/book rather than the 2012 film. These results don't seem to be prioritized the same way as the search results.
I tried something similar using json-rpc for a Freebase search method:
{
"method": "freebase.search",
"apiVersion": "v1",
"params": {
"name": ["RDJ"],
"key": "api_key",
"limit":10
}
}
But the "freebase.search" method didn't seem to exist.
One thing to note is that I will not know the expected type of the items I am looking for before hand.
Long story short: I want the exact results the search API provides, but with multiple queries wrapped up into one call.
Am I missing something terribly simple like an OR operator for the search API?? I've been searching for days, but can't seem to find a good solution. I would appreciate any help at all!
Why not just make two calls asynchronously? That would give you the results you need with almost no penalty in latency.
A few relevant facts:
The Reconcile API is still experimental. It's intended for use in reconciling against a type at a minimum and usually scoring using additional property values.
The Search API isn't included in the RPC mechanism because its freeform output doesn't work with the assumptions of the RPC framework. Ditto for the Topic API, although that's not really relevant here.
The Search API has a fairly expressive S-expression language. You don't say if you want the queries scored independently or together, but if you want them ranked jointly, you can use a filter expression like [(any name:rdj name:"The Avengers")]
https://www.googleapis.com/freebase/v1/search?query=&limit=10&filter=%28any%20name:rdj%20name:%22the%20avengers%22%29