Microsoft Azure Documents BadRequestException An invalid query has been specified with filters against path(s) that are not range-indexed. Consider adding allow scan header in the request.
My query is:
SELECT c.id FROM users c WHERE (c.lat < 29.89)
OVER ?? number of documents (as there are no way to get the number of document in collection with DocumentDB)
If you look at the blogpost here:
http://azure.microsoft.com/blog/2015/01/27/performance-tips-for-azure-documentdb-part-2/
Indexing Policy Tip #3: Specify range index path type for all paths
used in range queries
DocumentDB currently supports two index path types: Hash and Range.
Choosing an index path type of Hash enables efficient equality
queries. Choosing an index type of Range enables range queries (using
>, <, >=, <=).
It gives an example in C# to add a Range Index to make the path comparable, but there is similar functionality in the node.js library.
When you create a collection, you can pass the IndexingPolicy through the body parameter. The IndexingPolicy has a couple of members. One of which is the IncludedPaths, where you can define indices.
var policy = {
Automatic: true,
IndexingMode: 'Lazy',
IncludedPaths: [
{
IndexType: "Range",
Path: "path to be indexed (c.lat)",
NempericPrecission: "1",
StringPrecission: "1"
}
],
ExcludedPaths: []
}
client.createCollection(
'#yourdblink',
{
id: 10001,
indexingPolicy: policy
});
The Policy can be changed in the new Azure Portal (https://portal.azure.com), under (your DocumentDB resource) -> Settings -> Indexing Policy.
Related
Working on Azure Cognitive Search with backend as MS SQL table, have some scenarios where need help to define a query.
Sample table structure and data :
Scenarios 1 : Need to define a query which will return data based on category.
I have tied query using search.ismatch but its uses prefix search and matches other categories as well with similar kind of values i.e. "Embedded" and "Embedded Vision"
$filter=Region eq 'AA' and search.ismatch('Embedded*','Category')
https://{AZ_RESOURCE_NAME}.search.windows.net/indexes/{INDEX_NAME}/docs?api-version=2020-06-30-Preview&$count=true&$filter=Region eq 'AA' and search.ismatch('Embedded*','Category')
And it will response with below result, where it include "Embedded" and "Embedded Vision" both categories.
But my expectation is to fetch data only if it match "Embedded" category, as highlighted below
Scenario 2: For the above Scenario 1, Need little enhancement to find records with multiple category
For example if I pass multiple categories (i.e. "Embedded" , "Automation") need below highlighted output
you'll need to use a different analyzer which will break the tokens on every ';' just for the category field rather than 'whitespaces'.
You should first ensure your Category data is populated as a Collection(Edm.String) in the index. See Supported Data Types in the official documentation. Each of your semicolon-separated values should be separate values in the collection, in a property called Category (or similar).
You can then filter by string values in the collection. See rules for filtering string collections. Assuming that your index contains a string collection field called Category, you can filter by categories containing Embedded like this:
Category/any(c: c eq 'Embedded')
You can filter by multiple values like this:
Category/any(c: search.in(c, 'Embedded, Automation'))
Start with clean data in your index using proper types for the data you have. This allows you to implement proper facets and you can utilize the syntax made specifically for this. Trying to work around this with wildcards is a hack that should be avoided.
To solve above mention problem used a below SQL function which will convert category to a json string array supported by Collection(Edm.String) data type in Azure Search.
Sql Function
CREATE FUNCTION dbo.GetCategoryAsArray
(
#ID VARCHAR(20)
)
RETURNS NVARCHAR(MAX)
AS
BEGIN
DECLARE #result NVARCHAR(MAX) = ''
SET #result = REPLACE(
STUFF(
(SELECT
','''+ TRIM(Value) + ''''
FROM dbo.TABLEA p
CROSS APPLY STRING_SPLIT (Category, ';')
WHERE p.ID = #ID
FOR XML PATH('')
),1,1,''),'&','&')
RETURN '[' + #result + ']'
END
GO
View to use function and return desired data
CREATE View dbo.TABLEA_VIEW AS
select
id
,dbo. GetCategoryAsArray(id) as CategoryArr
,type
,region
,Category
from dbo.TABLEA
Defined a new Azure Search Index using above SQL View as data source and during Index column mapping defined CategoryArr column as Collection(Edm.String) data type
Query to use to achieve expected output from Azure Search
$filter=Region eq 'AA' and CategoryArr/any(c: search.in(c, 'Embedded, Automation'))
This question already has answers here:
Cloud Firestore Case Insensitive Sorting Using Query
(3 answers)
Are Cloud Firestore queries still case sensitive?
(1 answer)
Closed 1 year ago.
To support case-insensitive or any other canonicalization do we need to write a separate field that contains the canonicalized version and query against that??.
For example:
db.collection("users").where("name", "==", "Dan")
db.collection("users").where("name_lowercase", "==", "dan")
What I would do:
Before querying (maybe client-side): convert the query term in two or more variations (10 variations is maximum). For example, the search term "dan" (String) becomes an array of ["dan", "DAN", "Dan"]
Then I would do a "in" query, where I would search all of those variations in the same name field.
The "in" query type supports up to 10 equality (==) clauses with a logical "OR" operator. (documentation here)
This way, you can keep only one field "name" and query with possible variations on it.
It would look like this:
let query_variations = ["dan", "DAN", "Dan"]; // TODO: write a function that converts the query string into this kind of Array
let search = await db.collection("users").where("name", "in", query_variations).get();
In short, yes.
This is because Cloud Firestore (and the Firebase Realtime Database, when enabled) are indexed databases based on the values of each property in a document.
Rather than search through hundreds (if not thousands and thousands) of documents for matches, the index of the relevant property is queried for matching document IDs.
Consider the following "database" and it's index based on the name in the documents:
const documents = {
"docId1": {
name: "dan"
},
"docId2": {
name: "dan"
},
"docId3": {
name: "Dan"
},
"docId4": {
name: "Dan"
}
}
const nameIndex = {
"dan": ["docId1, docId2"],
"Dan": ["docId3, docId4"]
}
Instead of calling Object.entries(documents).filter(([id, data]) => data.name === "dan") on the entire list of documents, you can just ask the index instead using nameIndex["dan"] yielding the final results ["docId1, docId2"] near-instantly ready to be retrieved.
Continuing that same example, calling nameIndex["daniel"] gives undefined (no documents with that name) which can quickly be used to say that the data doesn't exist in the database).
Firestore introduced composite indexes, which allows you to index across multiple properties such as "name" and "age" so you can also quickly and efficiently search documents where the name is "Dan" but they are also 42 years of age.
Further reading: The Firebase documentation covers one solution for text-based search here.
I am a new bee with the Datastore gCloud. And I want to filter in an entity called Score all the scores that have relation with a list of companies. My entity is formed as follows:
{
"company_id": 1,
"score": 100,
}
I have several entities with different company IDs. I tried to filter using the query.add_filter command but got the error ValueError:
('Invalid expression: "IN"', 'Please use one of: =, <, <=,>,> =.')
The reason for the error is very clear to me, but I have not found anything in the documentation on how to run filters with IN or CONTAINS, in addition to types other than those listed in the above error message.
The filters operators you seek are not supported by the datastore. The only supported ones are listed in Filters:
The comparison operator can be any of the following:
Operator Meaning
EQUAL Equal to
LESS_THAN Less than
LESS_THAN_OR_EQUAL Less than or equal to
GREATER_THAN Greater than
GREATER_THAN_OR_EQUAL Greater than or equal to
Some of the client libraries may offer some equivalents, but with limitations. For example the standard environment GAE-specific ndb library (not usable in your context) offers such support. The example from The != and IN Operations can be useful as you can implement it in a similar manner in your code:
Similarly, the IN operation
property IN [value1, value2, ...]
which tests for membership in a list of possible values, is
implemented as
(property == value1) OR (property == value2) OR ...
With Microsoft Graph I can access rows from a table like this:
/v1.0/drives/..../workbook/worksheets/Sheet4/tables/2/rows
The documentation states:
This method supports the OData Query Parameters to help customize the response.
I am able to use the $select query parameter:
/v1.0/drives/..../workbook/worksheets/Sheet4/tables/2/rows?$select=values.
But how can I use $search or $filter query parameters? For example, I want to search the rows where column 'employeeName' contains the string "John".
In order to filter data from Excel you should get a workbook session id first:
POST https://graph.microsoft.com/v1.0/drives/.../workbook/createSession
BODY => {persistChanges:false}
You may change the value of persistChanges to true if you want to keep any changes you make to the worksheet. This will return an id which you will be using it as part of the headers when applying the filter:
POST https://graph.microsoft.com/v1.0/drives/.../workbook/worksheets('Sheet4')/tables(id='4')/columns('employeeName')/filter/apply
HEADER => workbook-session-id: session_Id
BODY => { criteria: { filterOn: "Custom", criterion1: "=John", operator: "Or", criterion2: null }
Finally you can retrieve the rows by using:
GET https://graph.microsoft.com/v1.0/drives/.../workbook/worksheets('Sheet4')/tables('4')/range/visibleView/rows?$select=values
HEADER => workbook-session-id: session_Id
Here is some reference on how to setup the criteria
And a general reference about Excel and the Graph API
Microsoft Graph has some documentation about the optional query parameters here. There's also more documentation about the OData Query standards here.
Microsoft Graph only allows the $search query parameter to be used with message and person collections. Here is an example to find all messages that contain "pizza":
GET https://graph.microsoft.com/v1.0/me/messages?$search="pizza"
The $filter query parameter doesn't have this limitation. Here is an example to find all the users with names that start with "A":
GET https://graph.microsoft.com/v1.0/users?$filter=startswith(displayName,'A')
So, say I have created some records/documents under a bucket and the user updates only one column out of 10 in the RDBMS, so I am trying to send only that one columns data and update it in couchbase. But the problem is that couchbase is overwriting the entire record and putting NULL`s for the rest of the columns.
One approach is to copy all the data from the exisiting record after fetching it from Cbase, and then overwriting the new column while copying the data from the old one. But that doesn`t look like a optimal approach
Any suggestions?
You can use N1QL update Statments google for Couchbase N1QL
UPDATE replaces a document that already exists with updated values.
update:
UPDATE keyspace-ref [use-keys-clause] [set-clause] [unset-clause] [where-clause] [limit-clause] [returning-clause]
set-clause:
SET path = expression [update-for] [ , path = expression [update-for] ]*
update-for:
FOR variable (IN | WITHIN) path (, variable (IN | WITHIN) path)* [WHEN condition ] END
unset-clause:
UNSET path [update-for] (, path [ update-for ])*
keyspace-ref: Specifies the keyspace for which to update the document.
You can add an optional namespace-name to the keyspace-name in this way:
namespace-name:keyspace-name.
use-keys-clause:Specifies the keys of the data items to be updated. Optional. Keys can be any expression.
set-clause:Specifies the value for an attribute to be changed.
unset-clause: Removes the specified attribute from the document.
update-for: The update for clause uses the FOR statement to iterate over a nested array and SET or UNSET the given attribute for every matching element in the array.
where-clause:Specifies the condition that needs to be met for data to be updated. Optional.
limit-clause:Specifies the greatest number of objects that can be updated. This clause must have a non-negative integer as its upper bound. Optional.
returning-clause:Returns the data you updated as specified in the result_expression.
RBAC Privileges
User executing the UPDATE statement must have the Query Update privilege on the target keyspace. If the statement has any clauses that needs data read, such as SELECT clause, or RETURNING clause, then Query Select privilege is also required on the keyspaces referred in the respective clauses. For more details about user roles, see Authorization.
For example,
To execute the following statement, user must have the Query Update privilege on travel-sample.
UPDATE `travel-sample` SET foo = 5
To execute the following statement, user must have the Query Update privilege on the travel-sample and Query Select privilege on beer-sample.
UPDATE `travel-sample`
SET foo = 9
WHERE city = (SELECT raw city FROM `beer-sample` WHERE type = "brewery"
To execute the following statement, user must have the Query Update privilege on `travel-sample` and Query Select privilege on `travel-sample`.
UPDATE `travel-sample`
SET city = “San Francisco”
WHERE lower(city) = "sanfrancisco"
RETURNING *
Example
The following statement changes the "type" of the product, "odwalla-juice1" to "product-juice".
UPDATE product USE KEYS "odwalla-juice1" SET type = "product-juice" RETURNING product.type
"results": [
{
"type": "product-juice"
}
]
This statement removes the "type" attribute from the "product" keyspace for the document with the "odwalla-juice1" key.
UPDATE product USE KEYS "odwalla-juice1" UNSET type RETURNING product.*
"results": [
{
"productId": "odwalla-juice1",
"unitPrice": 5.4
}
]
This statement unsets the "gender" attribute in the "children" array for the document with the key, "dave" in the tutorial keyspace.
UPDATE tutorial t USE KEYS "dave" UNSET c.gender FOR c IN children END RETURNING t
"results": [
{
"t": {
"age": 46,
"children": [
{
"age": 17,
"fname": "Aiden"
},
{
"age": 2,
"fname": "Bill"
}
],
"email": "dave#gmail.com",
"fname": "Dave",
"hobbies": [
"golf",
"surfing"
],
"lname": "Smith",
"relation": "friend",
"title": "Mr.",
"type": "contact"
}
}
]
Starting version 4.5.1, the UPDATE statement has been improved to SET nested array elements. The FOR clause is enhanced to evaluate functions and expressions, and the new syntax supports multiple nested FOR expressions to access and update fields in nested arrays. Additional array levels are supported by chaining the FOR clauses.
Example
UPDATE default
SET i.subitems = ( ARRAY OBJECT_ADD(s, 'new', 'new_value' )
FOR s IN i.subitems END )
FOR s IN ARRAY_FLATTEN(ARRAY i.subitems
FOR i IN items END, 1) END;
If you're using structured (json) data, you need to read the existing record then update the field you want in your program's data structure and then send the record up again. You can't update individual fields in the json structure without sending it all up again. There isn't a way around this that I'm aware of.
It is indeed true, to update individual items in a JSON doc, you need to fetch the entire document and overwrite it.
We are working on adding individual item updates in the near future.