Compare two date fields in MongoDB - node.js

in my collection each document has 2 dates, modified and sync. I would like to find those which modified > sync, or sync does not exist.
I tried
{'modified': { $gt : 'sync' }}
but it's not showing what I expected. Any ideas?
Thanks

You can not compare a field with the value of another field with the normal query matching. However, you can do this with the aggregation framework:
db.so.aggregate( [
{ $match: …your normal other query… },
{ $match: { $eq: [ '$modified', '$sync' ] } }
] );
I put …your normal other query… in there as you can make that bit use the index. So if you want to do this for only documents where the name field is charles you can do:
db.so.ensureIndex( { name: 1 } );
db.so.aggregate( [
{ $match: { name: 'charles' } },
{ $project: {
modified: 1,
sync: 1,
name: 1,
eq: { $cond: [ { $gt: [ '$modified', '$sync' ] }, 1, 0 ] }
} },
{ $match: { eq: 1 } }
] );
With the input:
{ "_id" : ObjectId("520276459bf0f0f3a6e4589c"), "modified" : 73845345, "sync" : 73234 }
{ "_id" : ObjectId("5202764f9bf0f0f3a6e4589d"), "modified" : 4, "sync" : 4 }
{ "_id" : ObjectId("5202765b9bf0f0f3a6e4589e"), "modified" : 4, "sync" : 4, "name" : "charles" }
{ "_id" : ObjectId("5202765e9bf0f0f3a6e4589f"), "modified" : 4, "sync" : 45, "name" : "charles" }
{ "_id" : ObjectId("520276949bf0f0f3a6e458a1"), "modified" : 46, "sync" : 45, "name" : "charles" }
This returns:
{
"result" : [
{
"_id" : ObjectId("520276949bf0f0f3a6e458a1"),
"modified" : 46,
"sync" : 45,
"name" : "charles",
"eq" : 1
}
],
"ok" : 1
}
If you want any more fields, you need to add them in the $project.

For MongoDB 3.6 and newer:
The $expr operator allows the use of aggregation expressions within the query language, thus you can do the following:
db.test.find({ "$expr": { "$gt": ["$modified", "$sync"] } })
or using aggregation framework with $match pipeline
db.test.aggregate([
{ "$match": { "$expr": { "$gt": ["$modified", "$sync"] } } }
])
For MongoDB 3.0+:
You can also use the aggregation framework with the $redact pipeline operator that allows you to process the logical condition with the $cond operator and uses the special operations $$KEEP to "keep" the document where the logical condition is true or $$PRUNE to "remove" the document where the condition was false.
Consider running the following aggregate operation which demonstrates the above concept:
db.test.aggregate([
{ "$redact": {
"$cond": [
{ "$gt": ["$modified", "$sync"] },
"$$KEEP",
"$$PRUNE"
]
} }
])
This operation is similar to having a $project pipeline that selects the fields in the collection and creates a new field that holds the result from the logical condition query and then a subsequent $match, except that $redact uses a single pipeline stage which is more efficient:

Simply
db.collection.find({$where:"this.modified>this.sync"})
Example
Kobkrits-MacBook-Pro-2:~ kobkrit$ mongo
MongoDB shell version: 3.2.3
connecting to: test
> db.time.insert({d1:new Date(), d2: new Date(new Date().getTime()+10000)})
WriteResult({ "nInserted" : 1 })
> db.time.find()
{ "_id" : ObjectId("577a619493653ac93093883f"), "d1" : ISODate("2016-07-04T13:16:04.167Z"), "d2" : ISODate("2016-07-04T13:16:14.167Z") }
> db.time.find({$where:"this.d1<this.d2"})
{ "_id" : ObjectId("577a619493653ac93093883f"), "d1" : ISODate("2016-07-04T13:16:04.167Z"), "d2" : ISODate("2016-07-04T13:16:14.167Z") }
> db.time.find({$where:"this.d1>this.d2"})
> db.time.find({$where:"this.d1==this.d2"})
>

Use Javascript, use foreach And convert Date To toDateString()
db.ledgers.find({}).forEach(function(item){
if(item.fromdate.toDateString() == item.todate.toDateString())
{
printjson(item)
}
})

Right now your query is trying to return all results such that the modified field is greater than the word 'sync'. Try getting rid of the quotes around sync and see if that fixes anything. Otherwise, I did a little research and found this question. What you're trying to do just might not be possible in a single query, but you should be able to manipulate your data once you pull everything from the database.

To fix this issue without aggregation change your query to this:
{'modified': { $gt : ISODate(this.sync) }}

Related

MongoDB: How to find documents by value?

Two documents containing ObjectId("6148a371c13a6a0be492ebf4")
Document 1
{
"_id" : ObjectId("6144f66fb9543917f96fc"),
"refId" : "ford",
"template" : "6144f61cb96d772317f96f9",
"fieldValues" : {
"PDV" : [
"6126938cd24a8aa3d37b4992",
ObjectId("6148a371c13a6a0be492ebf4")
]
},
"group" : ObjectId("6144f66fb96d7731917f96fd"),
"createdAt" : ISODate("2021-09-17T20:11:27.440Z"),
"updatedAt" : ISODate("2021-09-20T15:06:26.146Z"),
"__v" : 0
}
Document 2
{
"_id" : ObjectId("6144f66fb96d77rr3217f96fc"),
"refId" : "CCM",
"template" : "6144f613296d7731917f96f9",
"fieldValues" : {
"DDB" : [
"6126938cd2448aa3d37b4992",
"5443938cd2448aa3d37b4992",
ObjectId("6148a371c13a6a0be492ebf4"),
]
},
"group" : ObjectId("6144f66fb96de431917f96fd"),
"createdAt" : ISODate("2021-09-17T20:11:27.440Z"),
"updatedAt" : ISODate("2021-09-20T15:06:26.146Z"),
"__v" : 0
}
ObjectId that we looking for is always inside fieldValues but instead of PDV or DDB we will always have the different naming.
So we can't use this type of query:
db.getCollection('products').find({"fieldValues.PDV":ObjectId('6148a371c13a6a0be492ebf4')})
PS. This query should work only on DB, we can't afford to query all products and do calculation on backend there might to be a millions of products.
You can use this one:
db.collection.aggregate([
{
$set: {
kv: { $first: { $objectToArray: "$fieldValues" } }
}
},
{ $match: { "kv.v": ObjectId("6148a371c13a6a0be492ebf3") } },
{ $unset: "kv" }
])
Mongo Playground
db.products.find({'_id': ObjectId("6148a371c13a6a0be492ebf4")})
The mistake in your code is that you used key instead of _id.
This way of writing it is much easier on the fingers though.
You'd think a solution like this would work but one reason why this may not is because you're trying to use === on an object. If you refer to this thread, it might help if you use .equals() instead of ===.

MongoDB Shema to support concurrent update on a document

We were working on a project with a 300 documents with currentValue field in a main collection, in order to track the history of each document of first collection. we created another collection named history with approximately 6.5 millions of documents.
For each input of system we have to add around 30 history item and update currentValue field of main collection, so, We tried computational field design pattern for currentValue, which lead us to have writeConfilict in concurrent situations (at concurrency of around 1000 requests).
Then we tried to compute currentValue field with sum (amount field) and groupBy(mainId field) on history collection which takes too long (> 3s).
Main collection docs:
{
"_id" : ObjectId(...),
"stock" : [
{
"currentAmount" : -313430.0,
"lastPrice" : -10.0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
{
"currentAmount" : 30,
"lastPrice" : 0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
.
.
.
],
"name" : "name",
}
History collection docs:
{
"_id" : ObjectId("..."),
"mainId" : ObjectId("..."),
"amount" : 5,
}
If you have any other idea to handle this situation(application or db level), I would be thankful.
UPDATE 1
The update query if I use computed pattern would be:
mainCollection.findOneAndUpdate(
{
$and: [
{ _id: id },
{ "stock.storage": fromId },
{ "stock.deletedAt": null }
],
},
{
$inc: {
"stock.$.currentAmount": -1 * amount,
}
},
{
session
}
)
And Aggregation pipeline if I want to calculate currentAmount everytime:
mainCollection.aggregate([
{
$match: {
branch: new ObjectId("...")
}
},
{
$group: {
_id: "$ingredient",
currentAmount: {
$sum: "$amount"
}
}
}])
in order to have computed field, mongo design patterns, suggested computed field,
The Computed Pattern is utilized when we have data that needs to be computed repeatedly in our application. link
like below:
// your main collection will look like this
{
"_id" : ObjectId(...),
"stock" : [
{
"currentAmount" : -313430.0,
"lastPrice" : -10.0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
{
"currentAmount" : 30,
"lastPrice" : 0,
"storage" : ObjectId("..."),
"alarmCapacity" : 12
},
"totalAmount": 20000 // for example
}
but for having concurrent there is a better way to solve this problem with cumulative summation, in this algorithm, we sum last documents inputs, with current input:
{
"_id" : ObjectId("..."),
"mainId" : ObjectId("..."),
"amount" : 5,
"cumulative": 15 // sum of last documents input
}

MongoDB merge two collections with unmatched documents

I am trying to compare and find different documents from two collections
below are the samples, Mongodb version:4.0, ORM:mongoose
**col1: Has one new document**
{ "id" : 200001, "mobileNo" : #######001 }
{ "id" : 200002, "mobileNo" : #######002 } //mobileNo may not be unique.
{ "id" : 200003, "mobileNo" : #######002 }
{ "id" : 200004, "mobileNo" : #######004 }
**col2:**
{ "id" : 200001, "mobileNo" : #######001 }
{ "id" : 200002, "mobileNo" : #######002 }
{ "id" : 200003, "mobileNo" : #######003 }
Now I want to insert the document { "id" : 200004, "mobileNo" : #######004 } from col1 to col2
i.e; the documents which doesn't match.
This is what I've tried so far :
const col1= await Col1.find({}, { mobileNo: 1,id: 1, _id: 0 })
col1.forEach(async function (col1docs) {
let col2doc = await Col2.find({ mobileNo: { $ne: col1docs.mobileNo},
id:{$ne:col1docs.id} }, { mobileNo: 1, _id: 0, id: 1 })
if (!(col2doc)) {
Col2.insertMany(col1docs);
}
});
I have also tried with $eq instead of $ne but neither i am getting the unmatched documents nor they are getting inserted. Any suggestions??? Combination of id+phoneNo is unique
I would say instead of doing two .find() calls plus iteration & then third call to write data, try this query :
db.col1.aggregate([
{
$lookup: {
from: "col2",
let: { id: "$id", mobileNo: "$mobileNo" },
pipeline: [
{
$match: { $expr: { $and: [ { $eq: [ "$id", "$$id" ] }, { $gte: [ "$mobileNo", "$$mobileNo" ] } ] } }
},
{ $project: { _id: 1 } } // limiting to `_id` as we don't need entire doc of `col2` - just need to see whether a ref exists or not
],
as: "data"
}
},
{ $match: { data: [] } // Filtering leaves docs in `col1` which has no match in `col2`
},
{ $project: { data: 0, _id: 0 } }
])
Test : mongoplayground
Details : From the above query you're taking advantage of specifying conditions in $lookup to get docs from col1 which have reference in col2. Let's say $lookup will run on each document of col1 - So with the unique combination of id & mobileNo from current document in col1 has a matching in col2 then col2 doc's _id will be pushed in data array, at the end what we get out of col1 is data: [] to say no matching docs were found for these col1 doc's. Now you can just write all the returned docs to col2 using .insertMany(). Actually you can do this entire thing using $merge on MongoDB version > 4.2 without any need of 2nd write call(.insertMany()).
For your scenario on MongoDB version > 4.2 something like this will merge docs to second collection :
{ $merge: 'col2' } // Has to be final stage in aggregation
Note : If this has to be done periodically - no matter how you do this, try to minimize data that you're operating on, maybe maintain a time field & you can use that field to filter docs first & do this job, or you can also take advantage of _id to say we've done for all these docs in last run & we need to start from this docs - which helps you a lot to reduce data to be worked on. Additionally don't forget to maintain indexes.

How to do mongoose aggregation with nested array documents

I have a Mongodb collection, Polls with following schema
{
"options" : [
{
"_id" : Object Id,
"option" : String,
"votes" : [ Object Id ] // object ids of users who voted
},.....
]
}
Assume i have userId of the user in node js to whom I want to send this info.
My task is to
(1) include an extra field in the above json object (which i get using mongoose).
as
"myVote" : option._id
I need to find option._id for which
options[someIndex].votes contains userId
(2) change the existing "votes" field in each option to represent number of votes on a particular option as can be seen in example
Example:
{
"options" : [
{
"_id" : 1,
"option" : "A",
"votes" : [ 1,2,3 ]
},
{
"_id" : 2,
"option" : "B",
"votes" : [ 5 ]
},
{
"_id" : 3,
"option" : "C",
"votes" : [ ]
}
]
}
So if i user with user id = 5 wants to see the poll, then i need to send following info:
Expected Result :
{
"my_vote" : 2, // user with id 5 voted on option with id 2
"options" : [
{
"_id" : 1,
"option" : "A",
"votes" : 3 //num of votes on option "A"
},
{
"_id" : 2,
"option" : "B",
"votes" : 1 //num of votes on option "B"
},
{
"_id" : 3,
"option" : "C",
"votes" : 0 //num of votes on option "C"
}
]
}
Since it was the question that you actually asked that was neither really provided in the current acceptance answer, and also that it does some unnecessary things, there is another approach:
var userId = 5; // A variable to work into the submitted pipeline
db.sample.aggregate([
{ "$unwind": "$options" },
{ "$group": {
"_id": "$_id",
"my_vote": { "$min": {
"$cond": [
{ "$setIsSubset": [ [userId], "$options.votes" ] },
"$options._id",
false
]
}},
"options": { "$push": {
"_id": "$options._id",
"option": "$options.option",
"votes": { "$size": "$options.votes" }
}}
}}
])
Which of course will give you output per document like this:
{
"_id" : ObjectId("5573a0a8b67e246aba2b4b6e"),
"my_vote" : 2,
"options" : [
{
"_id" : 1,
"option" : "A",
"votes" : 3
},
{
"_id" : 2,
"option" : "B",
"votes" : 1
},
{
"_id" : 3,
"option" : "C",
"votes" : 0
}
]
}
So what you are doing here is using $unwind in order to break down the array for inspection first. The following $group stage ( and the only other stage you need ) makes use of the $min and $push operators for re-construction.
Inside each of those operations, the $cond operation tests the array content via $setIsSubset and either returns the matched _id value or false. When reconstructing the inner array element, specify all elements rather than just the top level document in arguments to $push and make use of the $size operator to count the elements in the array.
You also make mention with a link to another question about dealing with an empty array with $unwind. The $size operator here will do the right thing, so it is not required to $unwind and project a "dummy" value where the array is empty in this case.
Grand note, unless you are actually "aggregating" across documents it generally would be advised to do this operation in client code rather than the aggregation framework. Using $unwind effectively creates a new document in the aggregation pipeline for each element of the array contained in each document, which produces significant overhead.
For such an operation acting on distinct documents only, client code is more efficient to process each document individually.
If you really must persist that server processing is the way to do this, then this is probably most efficient using $map instead:
db.sample.aggregate([
{ "$project": {
"my_vote": {
"$setDifference": [
{ "$map": {
"input": "$options",
"as": "o",
"in": { "$cond": [
{ "$setIsSubset": [ [userId], "$$o.votes" ] },
"$$o._id",
false
]}
}},
[false]
]
},
"options": { "$map": {
"input": "$options",
"as": "o",
"in": {
"_id": "$$o._id",
"option": "$$o.option",
"votes": { "$size": "$$o.votes" }
}
}}
}}
])
So this just "projects" the re-worked results for each document. The my_vote is not the same though, since it is a single element array ( or possible multiple matches ) that the aggregation framework lacks the operators to reduce to a non array element without further overhead:
{
"_id" : ObjectId("5573a0a8b67e246aba2b4b6e"),
"options" : [
{
"_id" : 1,
"option" : "A",
"votes" : 3
},
{
"_id" : 2,
"option" : "B",
"votes" : 1
},
{
"_id" : 3,
"option" : "C",
"votes" : 0
}
],
"my_vote" : [
2
]
}
Check out this question.
It's not asking the same thing, but there's no way to do what you're asking without multiple queries anyway. I would modify the JSON you get back directly, as you're just displaying extra info that is already contained in the result of the query.
Save the userID you're querying for.
Take the results of your query (options array in an object), search through the votes of each element in the array.
When you've found the right vote, attach the _id (perhaps add 'n/a' if you don't find a vote).
Write a function that does 2 and 3, and you can just pass it a userID, and get back a new object with myVote attached.
I don't think doing it like this will be slower than doing another query in Mongoose.

MongoDB-Query Optimization

I have a collection with a sub-document consisting of more than 40K records.
My aggregate query takes about 300 secs. I have tried optimizing the same using compound as well as multi-key indexing, which completes in 180 secs.
I still require a reduced query time execution.
here is my collection:
{
"_id" : ObjectId("545b32cc7e9b99112e7ddd97"),
"grp_id" : 654,
"user_id" : 2,
"mod_on" : ISODate("2014-11-06T08:35:40.857Z"),
"crtd_on" : ISODate("2014-11-06T08:35:24.791Z"),
"uploadTp" : 0,
"tp" : 1,
"status" : 3,
"id_url" : [
{"mid":"xyz12793"},
{"mid":"xyz12794"},
{"mid":"xyz12795"},
{"mid":"xyz12796"}
],
"incl" : 1,
"total_cnt" : 25,
"succ_cnt" : 25,
"fail_cnt" : 0
}
and following is my query
db.member_id_transactions.aggregate([ { '$match':
{ id_url: { '$elemMatch': { mid: 'xyz12794' } } } },
{ '$unwind': '$id_url' },
{ '$match': { grp_id: 654, 'id_url.mid': 'xyz12794' } } ])
has anyone faced the same issue?
here's the o/p for aggregate query with explain option
{
"result" : [
{
"_id" : ObjectId("546342467e6d1f4951b56285"),
"grp_id" : 685,
"user_id" : 2,
"mod_on" : ISODate("2014-11-12T11:24:01.336Z"),
"crtd_on" : ISODate("2014-11-12T11:19:34.682Z"),
"uploadTp" : 1,
"tp" : 1,
"status" : 3,
"id_url" : [
{"mid":"xyz12793"},
{"mid":"xyz12794"},
{"mid":"xyz12795"},
{"mid":"xyz12796"}
],
"incl" : 1,
"__v" : 0,
"total_cnt" : 21406,
"succ_cnt" : 21402,
"fail_cnt" : 4
}
],
"ok" : 1,
"$gleStats" : {
"lastOpTime" : Timestamp(0, 0),
"electionId" : ObjectId("545c8d37ab9cc679383a1b1b")
}
}
One way to reduce the number of records being filtered further is to include the field grp_id, in the first $match operator.
db.member_id_transactions.aggregate([
{$match:{ "id_url.mid": 'xyz12794',"grp_id": 654 } },
{$unwind: "$id_url" },
{$match: { "id_url.mid": "xyz12794" } }
])
See how the performance is now. Add grp_id to the index to get better response time.
The above aggregation query though it works, is unnecessary. since you are not altering the structure of the document, and you expect only one element in the array to match the filter condition, you could just use a simple find and project.
db.member_id_transactions.find(
{ "id_url.mid": "xyz12794","grp_id": 654 },
{"_id":0,"grp_id":1,"id_url":{$elemMatch:{"mid":"xyz12794"}},
"user_id":1,"mod_on":1,"crtd_on":1,"uploadTp":1,
"tp":1,"status":1,"incl":1,"total_cnt":1,
"succ_cnt":1,"fail_cnt":1
}
)

Resources