I have the following view in CouchDB that is reduced via _count:
function (doc) {
if (doc.type === "signature") {
emit([doc.worksite_id, doc.uid, doc.timestamp], doc._id);
}
}
There are cases where rather than using group_level=2 in my query to get my count values sorted by doc.worksite_id and doc.uid pairs (as shown below)...
{
"rows": [
{
"key": [
"worksite-1",
"id-1"
],
"value": 2
},
{
"key": [
"worksite-2",
"id-1"
],
"value": 1
},
{
"key": [
"worksite-2",
"id-2"
],
"value": 26
}
]
}
...I, instead, need to get count values sorted strictly by doc.uid, with an example of something similar to the following:
{
"rows": [
{
"key": [
"id-1"
],
"value": 3
},
{
"key": [
"id-2"
],
"value": 26
}
]
}
Is there an efficient way to do this based on the current view I'm querying from? And if not, what is the most efficient way to do this?
Related
I am using CouchDB 3.1.1 to perform Mango queries against a database containing a large number of documents. A very common requirement in my application is to perform queries on a very specific and dynamic set of documents. From what I understand at this moment, these are the only choices I have on how to confront my problem:
Make multiple requests to /db/_find each with a distinct "_id"
Make a single call to /db/_find
Of the ways I can accomplish the second choice:
Use an "$or" array on all the "_id": value pairs
Use an "$or" array on all the values of the "_id" key
The second choice is what I would prefer to use since making multiple POST requests would incur overhead. Unfortunately using "$or" seems to get in the way of the query engine making use of the "_id" index.
Thus, choice #1 returns with a speedy 2 ms per transaction but the results are not sorted (requiring my application to do the sorting). Choice #2, given an array of 2 _ids, regardless of the $or syntax, takes over 3 seconds to render.
What is the most efficient way to use a CouchDB Mango query index against a specific set of documents?
Fast Example: Results using a single _id
{
"selector": {
"_id": "184094"
},
"fields": [
"_id"
]
}
documents examined: 26,312
results returned: 1
execution time: 2 ms
Slow Example: Results using $or of key / value pairs
{
"selector": {
"$or": [
{
"_id": "184094"
},
{
"_id": "157533"
}
]
},
"fields": [
"_id"
]
}
documents examined: 26,312
results returned: 2
execution time: 2,454 ms
Slow Example: Results using $or array of values
{
"selector": {
"_id": {
"$or": [
"184094",
"157533"
]
}
},
"fields": [
"_id"
]
}
documents examined: 26,312
results returned: 2
execution time: 2,522 ms
Slow Example: Results using $in (which is illegal but still returns results)
{
"selector": {
"_id": {
"$in": [
"184094",
"157533"
]
}
},
"fields": [
"_id"
]
}
documents examined: 26,312
results returned: 2
execution time: 2,618 ms
Index: The registered index for _id
{
"_id": "_design/508b5b51e6085c2f96444b82aced1e5dfec986b2",
"_rev": "1-f951eb482f9a521752adfdb6718a6a59",
"language": "query",
"views": {
"foo-index": {
"map": {
"fields": {
"_id": "asc"
},
"partial_filter_selector": {}
},
"reduce": "_count",
"options": {
"def": {
"fields": [
"_id"
]
}}}}}
Explain: An 'explain' summary done to one of the slow queries. Note that the registered index was used.
{
"dbname": "dnp_person_comment",
"index": {
"ddoc": "_design/508b5b51e6085c2f96444b82aced1e5dfec986b2",
"name": "foo-index",
"type": "json",
"partitioned": false,
"def": {
"fields": [
{
"_id": "asc"
}
]
}
},
"partitioned": false,
"selector": {
"$or": [
{
"_id": {
"$eq": "184094"
}
},
{
"_id": {
"$eq": "157533"
}
}
]
},
"opts": {
"use_index": [],
"bookmark": "nil",
"limit": 25,
"skip": 0,
"sort": {},
"fields": [
"_id"
],
"partition": "",
"r": [
49
],
"conflicts": false,
"stale": false,
"update": true,
"stable": false,
"execution_stats": false
},
"limit": 25,
"skip": 0,
"fields": [
"_id"
],
"mrargs": {
"include_docs": true,
"view_type": "map",
"reduce": false,
"partition": null,
"start_key": [],
"end_key": [
"<MAX>"
],
"direction": "fwd",
"stable": false,
"update": true,
"conflicts": "undefined"
}
}
Please don't tell me to "googleit"!
I have been poring over the Apache pages and the IBM pages for days trying to find the full allowed syntax for a Design Doc.
From the above readings:
the 'map' property is always a Javascript function
the 'options' property may be one/both of local_seq or include_design.
When I use Fauxton to edit a Mango Query, however, I see that the reality is much broader.
I defined a query ...
{
"selector": {
"data.type": {
"$eq": "invoice"
},
"data.idib": {
"$gt": 0,
"$lt": 99999
}
},
"sort": [
{
"data.type": "desc"
},
{
"data.idib": "desc"
}
]
}
... with an accompanying index ...
{
"index": {
"fields": [
"foo"
]
},
"name": "foo-json-index",
"type": "json"
}
... and then looked at the design doc produced ...
{
"_id": "_design/5b1cf1be5a6b7013019ba4afac2b712fc06ea82f",
"_rev": "1-1e6c5b7bc622d9b3c9b5f14cb0fcb672",
"language": "query",
"views": {
"invoice_code": {
"map": {
"fields": {
"data.type": "desc",
"data.idib": "desc"
},
"partial_filter_selector": {}
},
"reduce": "_count",
"options": {
"def": {
"fields": [
{
"data.type": "desc"
},
{
"data.idib": "desc"
}
]
}
}
}
}
}
Both of the published syntax rules are broken!
map is not a function
options defines the fields of the index
Where can I find a full description of all the allowed properties of a Design Document?
We have a table with this type of structure:
{_id:15_0, createdAt: 1/1/1, task_id:[16_0, 17_0, 18_0], table:”details”, a:b, c: d, more}
We created indexes using
{
"index": {},
"name": "paginationQueryIndex",
"type": "text"
}
It auto created
{
"ddoc": "_design/28e8db44a5a0862xxx",
"name": "paginationQueryIndex",
"type": "text",
"def": {
"default_analyzer": "keyword",
"default_field": {
},
"selector": {
},
"fields": [
],
"index_array_lengths": true
}
}
We are using the following query
{
"selector": {
"createdAt": { "$gt": 0 },
"task_id": { "$in": [ "18_0" ] },
"table": "details"
},
"sort": [ { "createdAt": "desc" } ],
"limit”: 20
}
It takes 700-800 ms for first time, after that it decreases to 500-600 ms
Why does it take longer the first time?
Any way to speed up the query?
Any way to add indexes to specific fields if type is “text”? (instead of indexing all the fields in these records)
You could try creating the index more explicitly, defining the type of each field you wish to index e.g.:
{
"index": {
"fields": [
{
"name": "createdAt",
"type": "string"
},
{
"name": "task_id",
"type": "string"
},
{
"name": "table",
"type": "string"
}
]
},
"name": "myindex",
"type": "text"
}
Then your query becomes:
{
"selector": {
"createdAt": { "$gt": "1970/01/01" },
"task_id": { "$in": [ "18_0" ] },
"table": "details"
},
"sort": [ { "createdAt": "desc" } ],
"limit": 20
}
Notice that I used strings where the data type is a string.
If you're interested in performance, try removing clauses from your query one at-a-time to see if one is causing the performance problem. You can also look at the explanation of your query to see if it using your index correctly.
Documentation on creating an explicit text query index is here
This question already has answers here:
How to Update Multiple Array Elements in mongodb
(16 answers)
Updating a Nested Array with MongoDB
(2 answers)
Closed 5 years ago.
Building a Nodejs app, I'm trying to pull all doubly nested records from a Mongo Database. Attempts that I've made only removed one doubly nested record or all nested records. As in the example data below I've been trying to remove all tickets that has the same keyId. I've reduced the example but tickets as an array there might be other elements with the same structure with different "keyIds" that shouldn't be removed. I've looked this question but it only refrains to removing one record of a doubly nested array, not all of them at once.
[
{
"_id": "59fe54098448d822f89a7e62",
"ownerId": "59b23449b20b7c1838eee1a3",
"name": "Home",
"keys": [
{
"id": "6d7435625564594f4a563554796c6a77",
"name": "Front Door"
}
],
"grants": [
{
"id": "307658775634774a556b677650347072",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": [
{
"keyId": "6d7435625564594f4a563554796c6a77",
"iv": "b7090268bdaf9ab55270e133b5629e28"
}
]
},
{
"id": "37703369365765485763484a4358774d",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": [
{
"keyId": "6d7435625564594f4a563554796c6a77",
"iv": "d2e2de0f9387c5d9b16424e8ac66a3c1"
}
]
},
{
"id": "3451483977564d755278397939593633",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": [
{
"keyId": "6d7435625564594f4a563554796c6a77",
"iv": "582ff50ac3d337c62eb53094470e3161"
}
]
},
{
"id": "7059684f4e42754d55456e726b35664e",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": [
{
"keyId": "6d7435625564594f4a563554796c6a77",
"iv": "b110ee5cb5da8941cc8ad6e1c3fe501c"
}
]
}
]
}
]
After removing all tickets with keyId=6d7435625564594f4a563554796c6a77 the intended data should look like this:
[
{
"_id": "59fe54098448d822f89a7e62",
"ownerId": "59b23449b20b7c1838eee1a3",
"name": "Home",
"keys": [
{
"id": "6d7435625564594f4a563554796c6a77",
"name": "Front Door"
}
],
"grants": [
{
"id": "307658775634774a556b677650347072",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": []
},
{
"id": "37703369365765485763484a4358774d",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": []
},
{
"id": "3451483977564d755278397939593633",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": []
},
{
"id": "7059684f4e42754d55456e726b35664e",
"userId": "59b23449b20b7c1838eee1a3",
"tickets": []
}
]
}
]
This code remove all grants at once:
db.places.update({}, {
$pull: {
"grants": {
"tickets": {
$elemMatch: { "keyId": keyID }
}
}
}
}, { multi: true });
This pull out just the first ticket and with "$pullAll" doesn't do anything:
db.places.findAndModify(
{
ownerId: ownerID, "grants.tickets.keyId": keyID
},
[ ],
{ $pull: { "grants.$.tickets": { keyId: keyID } } },
{ multi: true },
next
);
And this throws me an error saying: cannot use the part (grants of grants.tickets.$*.keyId) to traverse the element
db.places.update({ "grants.tickets.keyId": keyID }, {
$pull: {
"grants.tickets.$*.keyId": keyID
}
}, { multi: true });
Cloudant is returning error message:
{"error":"invalid_key","reason":"Invalid key use-index for this request."}
whenever I try to query against an index with the combination operator, "$or".
A sample of what my documents look like is:
{
"_id": "28f240f1bcc2fbd9e1e5174af6905349",
"_rev": "1-fb9a9150acbecd105f1616aff88c26a8",
"type": "Feature",
"properties": {
"PageName": "A8",
"PageNumber": 1,
"Lat": 43.051523,
"Long": -71.498852
},
"geometry": {
"type": "Polygon",
"coordinates": [
[
[
-71.49978935969642,
43.0508382914137
],
[
-71.49978564033566,
43.052210148524
],
[
-71.49791499857444,
43.05220740550381
],
[
-71.49791875962663,
43.05083554852429
],
[
-71.49978935969642,
43.0508382914137
]
]
]
}
}
The index that I created is for field "properties.PageName", which works fine when I'm just querying for one document, but as soon as I try for multiple ones, I would receive the error response as quoted in the beginning.
If it helps any, here is the call:
POST https://xyz.cloudant.com/db/_find
request body:
{
"selector": {
"$or": [
{ "properties.PageName": "A8" },
{ "properties.PageName": "M30" },
{ "properties.PageName": "AH30" }
]
},
"use-index": "pagename-index"
}
In order to perform an $or query you need to create a text (full text) index, rather than a json index. For example, I just created the following index:
{
"index": {
"fields": [
{"name": "properties.PageName", "type": "string"}
]
},
"type": "text"
}
I was then be able to perform the following query:
{
"selector": {
"$or": [
{ "properties.PageName": "A8" },
{ "properties.PageName": "M30" },
{ "properties.PageName": "AH30" }
]
}
}