I want to add indexes in the containers.
So the first thing that I did was to put this code in the bicep and I run:
resource AzureCosmosDatabasesContainers 'Microsoft.DocumentDB/databaseAccounts/sqlDatabases/containers#2022-05-15' = [for (azureCosmos, index) in XXXXXXXXXXXXXXXX[environmentCode]: {
name: '${azureCosmos.account}/${azureCosmos.database}/${azureCosmos.name}'
properties: {
resource: {
id: azureCosmos.name
indexingPolicy: {
indexingMode: 'consistent'
automatic: true
includedPaths: [
{
path: '/*'
indexes: [
{
kind: 'Range'
dataType: 'Number'
precision: -1
}
{
kind: 'Range'
dataType: 'String'
precision: -1
}
]
}
]
}
partitionKey: {
paths: [
'/_partitionKey'
]
kind: 'Hash'
version: 2
}
conflictResolutionPolicy: {
mode: 'LastWriterWins'
conflictResolutionPath: '/_ts'
}
}
}
dependsOn: [
AzureCosmosDatabases
]
}]
But what happens was that the indexes don't appear.
The second thing that I did was to go to the Azure portal - Opened a cosmodb - Opened the Data Explorer pane - select the container that I want to add indexes - Added this code :
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Range",
"dataType": "String",
"precision": -1
}
]
And finally, saved.
What happens is that I save and it's ok but when I open it again the configuration disappears.
Can anyone help me?
If you are wondering about not able to see kind, dataType and precision attributes in your indexing policy after it is saved, then it is expected behavior.
From this link:
This indexing policy is equivalent to the one below which manually
sets kind, dataType, and precision to their default values. These
properties are no longer necessary to explicitly set and you should
omit them from your indexing policy entirely (as shown in above
example). If you try to set these properties, they'll be automatically
removed from your indexing policy.
Related
I am storing the below data in azure cosmos graph db.
"properties": {
"A": {
"value": "prop1 new value"
},
"settings": {
"DigitalInput": {
"Input1": {
"nTransIn1": {
"tagName": {
"value": ""
}
}
},
"Input2": {
"nTransIn2": {
"tagName": {
"value": ""
}
}
}
When I am trying to query by single property
g.V().has('s_objectId',within('9d8cf5c6-7b5f-4d0b-af70-bf516f219d73')).
valueMap("p_A")
is giving expected output but how to retrieve with property 'settings' which has nested properties. When I try
g.V().has('s_objectId',within('9d8cf5c6-7b5f-4d0b-af70-bf516f219d73')).
valueMap("p_settings")
it is not giving the correct output as the setting property is stored like below in graph database
"p_settings.DigitalInput.Input1.nTransIn1.tagName": [
{
"id": "6057e448-a2e8-48e4-820f-5396003bdcae",
"value": ""
}
],
Your queries and sample data seem to use different field names. It would be helpful if you could add to the question an addV step that creates the structure you are using in a way that can be tested with TinkerGraph.
In general with Gremlin, the way to access map structures is to select your way into it. Something like
valueMap("p_A").select("p_settings")
I have following documents in my Azure Cosmos DB:
{
"id": "token",
"User": {
"UserToken": "token",
"Email": "test#email.com"
},
"_ts": 1541493290
}
When I run the following query:
SELECT * FROM root
WHERE ((root["User"]["UserToken"] = "token")
OR CONTAINS(root["User"]["Email"], "token"))
ORDER BY root["_ts"] DESC
Nothing is returned. But when I change it a bit. For example byconverting Email to email:
SELECT * FROM root
WHERE ((root["User"]["UserToken"] = "token")
OR CONTAINS(root["User"]["email"], "token"))
ORDER BY root["_ts"] DESC
The result is found. Moreover when I remove ORDER BY clause, also query returns me a result. So the query is like following
SELECT * FROM root
WHERE ((root["User"]["UserToken"] = "token")
OR CONTAINS(root["User"]["Email"], "token"))
Moreover, when I edit the document (like open it, add an empty line and save), some magic happens in the background and the document is found. For quite "new" documents (less than 1-3 months), I can search them without my "magic" trick.
Indexes definition is:
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Hash",
"dataType": "String",
"precision": 3
}
]
}
],
"excludedPaths": []
}
What I did wrong?
UPDATE the answer is not a full explanation but it helps a lot. Full explanation is in my blog (https://stapp.space/ridiculous-bug-in-azure-cosmos-db/)
CONTAINS(root["User"]["Email"], "token") won't work if you have strings indexed as Hash. They need to be Range with -1 precision. Hash only works for equality checks.
That's why the lowercase one is working. Because it cannot find the property and it just ignores it, falling back to the equality check. The first one finds it, sees that it's not indexed as Range and it just fails to return.
Changing indexing to this, will work:
{
"indexingMode": "consistent",
"automatic": true,
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"kind": "Range",
"dataType": "Number",
"precision": -1
},
{
"kind": "Range",
"dataType": "String",
"precision": -1
}
]
}
],
"excludedPaths": []
}
On a side note, the _ts field is not the best way to do ordering based on creation. It is a unix timestamp in seconds, so any documents created in the same second won't be properly ordered.
I'm having issues setting up an ARM Template for Azure Web Apps in that I can't add ConnectionString parameters where it sets the values if the parameters are set, but leave blank (default) if the parameters aren't set.
Here is how it looks in the template.json file:
"connectionStrings": [
{
"name": "[parameters('connString').connName)]",
"connectionString": "[parameters('connString').string]",
"type": "[parameters('connString').connType]"
}
],
And in the parameters.json file:
"connString": {
"value": {
"connName": "",
"string": "",
"connType": ""
}
},
When running the deployment with the above it fails on "Parameter name cannot be empty"
I attempted to use an equals function to set the value as empty if the parameter is empty, but set the parameter if the parameter is filled out, however it doesn't like the empty value.
"name": "[if(equals(parameters('connString').connName,''),'',parameters('connString').connName)]"
Also attempted an empty function:
"name": "[not(empty(parameters('connString').connName))]"
However this returns "False" if empty and "True" if the parameter is set (as designed)
The deployment works fine if I set dummy values as the parameters, is it possible to set a function or something similar so if the parameter is empty it uses whatever value is sent as if the connectionStrings section wasn't present in the template? These parameters are optional but it looks like because they're in the actual template.json file its expecting a value.
Cheers
EDIT
Going to post what my end templates looked like in case someone else needs assistance.
Template File
Variables
"variables": {
"empty": []
},
Resources
"connectionStrings": "[if(empty(parameters('connString')), variables('empty'), array(parameters('connString')))]",
Parameter File
If setting a connection string
"connString": {
"value": [{
"name": "test",
"connectionString": "ufgndjkngsdgjkn",
"type": "Custom"
}]
},
If not wanting to set a connection string
"connString": {
"value": [
]
},
you should just do this:
parameters:
"connString": {
"value": {}
},
variables:
"empty": [],
template:
"connectionStrings": "[if(empty(parameters('connString')), variables('empty'), array(parameters('connString')))]"
you could try:
parameters:
"connString": [{
"value": {
"name": null,
"connectionString": null,
"type": "Custom"
}
}],
variables:
template:
"connectionStrings": "[if(empty(parameters('connString')[0].value.name), json('null'), parameters('connString'))]"
I'm struggling with something that should be easy but it's making no sense to me, I have these 2 documents in a database:
{ "name": "foo", "type": "typeA" },
{ "name": "bar", "type": "typeB" }
And I'm posting this to _find:
{
"selector": {
"type": "typeA"
},
"sort": ["name"]
}
Which works as expected but I get a warning that there's no matching index, so I've tried posting various combinations of the following to _index which makes no difference:
{
"index": {
"fields": ["type"]
}
}
{
"index": {
"fields": ["name"]
}
}
{
"index": {
"fields": ["name", "type"]
}
}
If I remove the sort by name and only index the type it works fine except it's not sorted, is this a limitation with couchdbs' mango implementation or am I missing something?
Using a view and map function works fine but I'm curious what mango is/isn't doing here.
With just the type index, I think it will normally be almost as efficient unless you have many documents of each type (as it has to do the sorting stage in memory.)
But since fields are ordered, it would be necessary to do:
{
"index": {
"fields": ["type", "name"]
}
}
to have a contiguous slice of this index for each type that is already ordered by name. But the query planner may not determine that this index applies.
As an example, the current pouchdb-find (which should be similar) needs the more complicated but equivalent query:
{
selector: {type: 'typeA', name: {$gte: null} },
sort: ['type','name']
}
to choose this index and build a plan that doesn't resort to building in memory for any step.
In Cosmos DB graph when I am defining Indexing policy as Automatic, I am able to run queries but when I am updating indexing policy to Manual and defining Indexing path (/label/?) and Indexing mode set as 'Consistent', the query is not fetching any data.
Let's say my first query (when Indexing policy set as Manual) is :
g.addV('Azure').property('name','Cerulean Software'))
Result is :
[
{
"id": "0c14a00a-edf6-46b1-9e40-45cc37f750ea",
"label": "Azure",
"type": "vertex",
"properties": {
"name": [
{
"id": "f89ee2ee-74df-4256-a5d4-2b47eb526976",
"value": "Cerulean Software"
}
]
}
}
]
Now, my second query (when Indexing policy set as Manual (see Edit #1 below)) is:
g.V().hasLabel('Azure')
This second query is not fetching any result even though there is vertex present in graph named as 'Azure'.
What could be the possible reason behind this?
Edit #1: Manual Indexing Policy Before Change
"indexingPolicy": {
"automatic": false,
"excludedPaths": [],
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"dataType": "Number",
"kind": "Range",
"precision": -1
},
{
"dataType": "String",
"kind": "Hash",
"precision": 3
}
]
},
{
"path": "/label/?",
"indexes": [
{
"dataType": "String",
"kind": "Hash",
"precision": 3
},
{
"dataType": "Number",
"kind": "Range",
"precision": -1
}
]
}
],
"indexingMode": "consistent"
},
Edit #2: Manual Indexing Policy After Change
"indexingPolicy": {
"automatic": false,
"excludedPaths": [],
"includedPaths": [
{
"path": "/*",
"indexes": [
{
"dataType": "Number",
"kind": "Range",
"precision": -1
},
{
"dataType": "String",
"kind": "Hash",
"precision": 3
}
]
},
{
"path": "/_isEdge/?",
"indexes": [
{
"dataType": "String",
"kind": "Hash",
"precision": 3
},
{
"dataType": "Number",
"kind": "Range",
"precision": -1
}
]
}
],
"indexingMode": "consistent"
},
With Cosmos, graph statements are not executed as traversals on the Azure side. The graph client actually translates gremlin statements into Document SQL calls and then aggregates the results back to you on the client side. In the case of your statement g.V().hasLabel('Azure') the call is actually translated to {"query":"SELECT N_2 FROM Node N_2 WHERE (IS_DEFINED(N_2._isEdge) = false AND (N_2.label = 'Azure'))"}
This can be verified through the use of a proxy such as Fiddler which will allow you to inspect the outbound calls from your machine.
The top level _isEdge property seems to be used across almost all Gremlin translated queries so I suspect that if you add that property to your indexing policy you should start to see the expected results.
EDIT:
I originally missed the part of your indexing policy that sets automatic: false. According to the Cosmos docs (under the heading Opting in and opting out of indexing), By default, all documents are automatically indexed, but you can choose to turn it off. When indexing is turned off, documents can be accessed only through their self-links or by queries using ID.
If you choose to run with indexing turned off, then the rest of your indexing policy is effectively meaningless and queries that aren't directly by document Id will no longer work. Can you elaborate as to what you're actually trying to accomplish here? There seems to be a bit of confusion. The indexing settings you've placed on label and isEdge aren't even necessary because they are the same as the value you've put for * which is the default rule matching all paths.
Post what your goals are for your indexing strategy and I can try to make an appropriate recommendation but you're definitely going to want to put automatic: true back into your policy.