Error when creating Custom Analyzer in API version 2016-09-01 - azure

I am currently working with Custom Analyzers in Azure Search. I have previously had a lot of success with the preview version of the Azure Search API "2015-02-28-Preview" which introduced the feature. I'm currently trying to migrate my custom analyzers to API version "2016-09-01" which according to this article (https://learn.microsoft.com/en-us/azure/search/search-api-migration) includes Custom Anlayzer support. My analyzers are configured as follows:
"analyzers": [
{
"name": "phonetic_area_analyzer",
"#odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
"tokenizer": "area_standard",
"tokenFilters": [ "lowercase", "asciifolding", "areas_phonetc" ]
},
{
"name": "partial_area_analyzer",
"#odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
"tokenizer": "area_standard",
"tokenFilters": [ "lowercase", "area_token_edge" ]
},
{
"name": "startsWith_area_analyzer",
"#odata.type": "#Microsoft.Azure.Search.CustomAnalyzer",
"tokenizer": "area_keyword",
"tokenFilters": [ "lowercase", "asciifolding", "area_edge" ]
}
],
"charFilters": [],
"tokenizers": [
{
"name":"area_standard",
"#odata.type":"#Microsoft.Azure.Search.StandardTokenizer"
},
{
"name":"area_keyword",
"#odata.type":"#Microsoft.Azure.Search.KeywordTokenizer"
}
],
"tokenFilters": [
{
"name": "area_edge",
"#odata.type": "#Microsoft.Azure.Search.EdgeNGramTokenFilter",
"minGram": 2,
"maxGram": 50
},
{
"name": "area_token_edge",
"#odata.type": "#Microsoft.Azure.Search.EdgeNGramTokenFilter",
"minGram": 2,
"maxGram": 20
},
{
"name": "areas_phonetc",
"#odata.type": "#Microsoft.Azure.Search.PhoneticTokenFilter",
"encoder": "doubleMetaphone"
}
]
This configuration works when using version "2015-02-28-Preview" but when I try version "2016-09-01" I get the following error as a response:
{
"error": {
"code": "",
"message": "The request is invalid. Details: index : The tokenizer of type 'standard' is not supported in the API version '2016-09-01'.\r\n"
}
}
Is there a problem with my configuration, or does version "2016-09-01" only allow for a limited subset of custom analyzer features? If this is the case, could someone please point me in the direction of some documentation detailing which features are supported?

Sorry, there was a delay in the process that updates the documentation. Here is my pull request that has the changes we introduced in 2016-09-01: https://github.com/Azure/azure-docs-rest-apis/pull/218 (request access here https://azure.github.io/)
In your example, change KeywordTokenizer to KeywordTokenizerV2, same for the StandardTokenizer and the EdgeNGramTokenFilter.
Update:
The new version of the documentation is online: https://learn.microsoft.com/en-us/rest/api/searchservice/custom-analyzers-in-azure-search

Related

Available Fields for Sharepoint Search Query on Microsoft Graph API

I'm using the Graph API to make a global search in my sharepoint website, and I need to retrieve some specific fields. I didn't find any documentation that specifies the available fields that I can use for the fields property on my payload, only a documentation for specific document library search.
I have to use the global search because my search needs to access all the document libraries on my sharepoint web site.
The field that I wanted to get from the request is the version of the document in the list. I could add this field in sharepoint, and my view is displaying the version values, but the request does not take this value. I'm using this request below:
Endpoint: https://graph.microsoft.com/v1.0/search/query
Payload:
"requests": [
{
"entityTypes": [
"listItem"
],
"query": {
"queryString": ""
},
"fields": [
"title",
"_UIVersionString"
]
}
]
}
Response:
{
"value": [
{
"searchTerms": [],
"hitsContainers": [
{
"hits": [
{
"hitId": "83C63693-C621-4CFE-B4F7-A36B68AEB421",
"rank": 1,
"summary": "...",
"resource": {
"#odata.type": "#microsoft.graph.listItem",
"fields": {
"title": "Calc.22090615231879"
}
}
},
],
"total": 1,
"moreResultsAvailable": false
}
]
}
],
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#Collection(microsoft.graph.searchResponse)"
}
The name of the field I'm using in the payload that corresponds to the version is _UIVersionString, where I got it from the specific list search request, using the https://graph.microsoft.com/v1.0/sites/{site-id}/lists/{list-id}/items?$expand=fields endpoint. But sadly, the version is not appearing on my search result.
Is there some documentation I could use to see a list of available fields for this request? I'm trying to find it in MS GraphAPI documentation, but it looks be a big real encyclopedia.
Do you know the name of the field that corresponds to the version?
Thanks a lot!
Other information:
Sharepoint Version: Sharepoint Web (Online)
Type of the lists: Document Library
Lists version configuration:
- Require content approval for submitted items?: No
- Create a version each time you edit a file in this document library?: Create Major Versions
- Require documents to be checked out before they can be edited?: No
Since Graph is using there managed properties from sharepoint you can try "UIVersionStringOWSTEXT".
{
"requests": [
{
"entityTypes": [
"listItem"
],
"query": {
"queryString": "test"
},
"fields": [
"title",
"UIVersionStringOWSTEXT"
]
}
]
}
The results look like
{
"value": [
{
"searchTerms": [
"test"
],
"hitsContainers": [
{
"hits": [
{
"hitId": "GUID",
"rank": 1,
"summary": "test",
"resource": {
"#odata.type": "#microsoft.graph.listItem",
"fields": {
"title": "test",
"uiVersionStringOWSTEXT": "1.0"
}
}
}
],
"total": 1,
"moreResultsAvailable": false
}
]
}
],
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#Collection(microsoft.graph.searchResponse)"
}
In my tenant this property was already created. I guess it is by default. If your search schema does not have this managed property you can create it and name it like you wish. Map it to the crawled property "ows_q_TEXT__UIVersionString" and make sure your managed property is set to "retrievable".

Azure Rule engine File extension with json file

I am trying to create a rule engine in a cdn endpoint. like this:
But using a Json file (The result in the image has been achieved manually but now i want to automate this).
So far I got this:
"deliveryPolicy": {
"description": "Rewrite and Redirect",
"rules": [
{
"name" : "UrlFileExtension",
"order": 2,
"conditions": [
{
"name": "UrlFileExtension",
"parameters": {
"#odata.type": "#Microsoft.Azure.Cdn.Models.UrlFileExtensionMatchConditionParameters",
"Extension": 0,
"operator": "LessThanOrEqual",
"matchValues": [0]
}
}
],
"actions": [
{
"name": "UrlRewrite",
"parameters": {
"#odata.type": "#Microsoft.Azure.Cdn.Models.DeliveryRuleUrlRewriteActionParameters",
"sourcePattern": "/",
"destination": "/index.html",
"preserveUnmatchedPath": false
}
}
]
},
The action works just fine, but the urlfileextentionI cant get it to work, it does not recognize the odata.type either.
Please any hint ot suggestion how to fix the condition?
You might want to try with this odata.type for the condition
"#odata.type": "#Microsoft.Azure.Cdn.Models.DeliveryRuleUrlFileExtensionMatchConditionParameters",
instead of
"#odata.type": "#Microsoft.Azure.Cdn.Models.UrlFileExtensionMatchConditionParameters",
https://learn.microsoft.com/en-us/python/api/azure-mgmt-cdn/azure.mgmt.cdn.models.urlfileextensionmatchconditionparameters?view=azure-python
(I'm aware the documentation says Python, I could not find better, but it could be your solution)

Apollo GraphQL Deprecation Warnings

I am getting lots of deprecation warnings. I am not sure why I am getting it. Could anyone please guide me out.
I have created a boilerplate for apollo graph ql with koa node js frameworks. I am planning to use a graph ql subscription in my next project. However, these warnings or errors making me confused. this is my git https://github.com/sumitbhavra/graphql-koa.git
{
"name": "include",
"description": "Directs the executor to include this field or fragment only when the `if` argument is true.",
"locations": [
"FIELD",
"FRAGMENT_SPREAD",
"INLINE_FRAGMENT"
],
"args": [
{
"name": "if",
"description": "Included when true.",
"type": {
"kind": "NON_NULL",
"name": null,
"ofType": {
"kind": "SCALAR",
"name": "Boolean",
"ofType": null
}
},
"defaultValue": null
}
]
},
{
"name": "deprecated",
"description": "Marks an element of a GraphQL schema as no longer supported.",
"locations": [
"FIELD_DEFINITION",
"ENUM_VALUE"
],
"args": [
{
"name": "reason",
"description": "Explains why this element was deprecated, usually also including a suggestion for how to access supported similar data. Formatted using the Markdown syntax, as specified by [CommonMark](https://commonmark.org/).",
"type": {
"kind": "SCALAR",
"name": "String",
"ofType": null
},
"defaultValue": "\"No longer supported\""
}
]
}
That is not a deprecation warning. That is an introspection result that includes information about the #deprecated directive, which can be used inside your type definitions to mark individual fields or enum values as deprecated.

how to map distinct custom skillset to index

trying to add custom skill in the skillset and map it in the index
here is in detail
I'm using the azure Named Entity Recognition in my skillset as
{
"#odata.type": "#Microsoft.Skills.Text.MergeSkill",
"description": "Merge text content with image tags",
"insertPreTag": " ",
"context": "/document",
"inputs": [
{
"name": "text",
"source": "/document/fullTextAndCaptions"
},
{
"name": "itemsToInsert",
"source": "/document/normalized_images/*/Tags/*/name"
}
],
"outputs": [
{
"name": "mergedText",
"targetName": "finalText"
}
]
}
and in the indexer as
{
"sourceFieldName": "/document/finalText/pages/*/entities/*/value",
"targetFieldName": "entities"
},
{
"sourceFieldName": "/document/finalText/pages/*/locations/*",
"targetFieldName": "locations"
},
and it works 100% now I want to add the Distinct custom skill from https://github.com/Azure-Samples/azure-search-power-skills/tree/master/Text/Distinct
I did publish the function and when I go to test it manually it works as expected.
however overall its not working in skillset. I want it to take the location and filter it and output the distinct only in it's own field in the search index.
I'm having a really hard time to configure the skillset and indexer to get it to work.
any help please?
You'll need to add the distinct custom skill like this, assuming you want to dedup over the whole document
{
"skills": [
...
{
"#odata.type": "#Microsoft.Skills.Custom.WebApiSkill",
"description": "Distinct skill",
"uri": "<https://distinct-skill>",
"context": "/document",
"inputs": [
{
"name": "locations",
"source": /document/finalText/pages/*/locations/*"
}
],
"outputs": [
{
"name": "distinct",
"targetName": "distinctLocations"
}
]
}
...
]
}
and an output field mapping to put it into the index.
{
"sourceFieldName": "/document/distinctLocations",
"targetFieldName": "distinctLocations"
}
See https://learn.microsoft.com/en-us/azure/search/cognitive-search-custom-skill-interface#consuming-custom-skills-from-skillset for adding a custom skill.
The skill inputs for the custom skill must be configured to point to the data you want to disambiguate. In this case, you didn't really need to modify the code, all you had to do was have an input with name 'words' and source '/document/finalText/pages//locations/'.

Microsoft.Azure.Search (sdk v3.0.3) don't return all the facets correctly

When I use Microsoft.Azure.Search (v3.0.3)'s "SearchAsync" and "Search" methods to return the indexed items, the sdk doesn't return all the facets.
However; when I try the same thing using Postman, it returns all the facets correctly.
Could this be a bug of the sdk (I believe it is as a direct call to an sdk method doesn't return the all the facets correctly - but couldn't find any records about this possible bug)? If yes, is there a fix for this for the sdk? Any help is appreciated.
UPDATE:
After spending some more time, I have found out that the bug is not .NET SDK Specific.
Both .NET SDK and REST API appear to have this problem and none of them returns all the facets. Can you please tell me is there a known bug for this and what is the fix for it?
Please see the following example;
There must be 2 Coaching facets but only 1 is returning from the Azure Search Service.
The new search query(facet specialisms added)
https://MYPROJECT-search.search.windows.net/indexes/myproject-directory-qa/docs?api-version=2016-09-01&$count=true&facet=specialisms&$filter=listingType eq 'Therapist'
"Coaching:Development coaching", --> This doesn't return as a facet.
"Coaching:Executive coaching", -->This returns fine.
"#search.facets": {
"specialisms#odata.type": "#Collection(Microsoft.Azure.Search.V2016_09_01.QueryResultFacet)",
"specialisms": [
{
"count": 5,
"value": "Anxiety, depression and trauma:Depression"
},
{
"count": 4,
"value": "Addiction, self-harm and eating disorders:Obsessions"
},
{
"count": 4,
"value": "Anxiety, depression and trauma:Post-traumatic stress"
},
{
"count": 4,
"value": "Coaching:Executive coaching"
},
{
"count": 4,
"value": "Identity, culture and spirituality:Self esteem"
},
{
"count": 4,
"value": "Relationships, family and children:Pregnancy related issues"
},
{
"count": 4,
"value": "Stress and work:Redundancy"
},
{
"count": 3,
"value": "Addiction, self-harm and eating disorders:Eating disorders"
},
{
"count": 3,
"value": "Anxiety, depression and trauma:Bereavement"
},
{
"count": 3,
"value": "Anxiety, depression and trauma:Loss"
}
]
},
{
"#search.score": 1,
"contactId": "df394997-6e94-e711-80ed-3863bb34db00",
"location": {
"type": "Point",
"coordinates": [
-2.58586,
51.47873
],
"crs": {
"type": "name",
"properties": {
"name": "EPSG:4326"
}
}
},
"profileImageUrl": "https://myprojectwebqa.blob.core.windows.net/profileimage/3e31457c-5113-4062-b960-30f038ce7bfc.jpg",
"locationText": "Bristol",
"listingType": "Therapist",
"disabledAccess": true,
"flexibleHours": true,
"offersConcessionaryRates": false,
"homeVisits": true,
"howIWillWork": "<p>Some test data</p>",
"specialisms": [
"Health related issues:Asperger syndrome",
"Health related issues:Chronic fatigue syndrome/ME",
"Addiction, self-harm and eating disorders:Addictions",
"Addiction, self-harm and eating disorders:Eating disorders",
"Addiction, self-harm and eating disorders:Obsessions",
"Anxiety, depression and trauma:Bereavement",
"Anxiety, depression and trauma:Depression",
"Anxiety, depression and trauma:Loss",
"Coaching:Development coaching",
"Coaching:Executive coaching",
"Identity, culture and spirituality:Self esteem",
"Identity, culture and spirituality:Sexuality",
"Relationships, family and children:Infertility",
"Relationships, family and children:Relationships",
"Stress and work:Redundancy"
],
"clientele": [
"Adults",
"Children",
"Groups"
],
"approaches": [
"CBT",
"Cognitive",
"Psychoanalytic",
"Psychosynthesis"
],
"sessionTypes": [
"Home visits",
"Long-term face to face work"
],
"hourlyRate": 50,
"fullName": "Test Name",
"id": "ZWUwNGIyNjYtYjQ5Ny1lNzExLTgwZTktMzg2M2JiMzY0MGI4"
}
Please see details below;
For my case I have found out that by default the Azure Search Service returns 10 of the facets. That is why I couldn't see all my facets.
After updating my search query as follows, I have fixed my problem and now I can see all my facets in the search results - please see the facet bit updated to this; facet=specialisms, count:9999.
https://MYPROJECTNAME-search.search.windows.net/indexes/MYPROJECTNAME-directory-qa/docs?api-version=2016-09-01&$count=true&facet=specialisms, count:9999&facet=clientele, count:9999&$filter=listingType eq 'Therapist'
For the Microsoft documentation, please see the following link.
"max # of facet terms; default is 10"
https://learn.microsoft.com/en-us/rest/api/searchservice/search-documents

Resources