I need to calculate this ((86400 + (7200 - 28800)) % 86400) in MongoDB's aggregate. Is it possible? Or need to do this on JavaScript.
Is mongo support remainder of division?
Yes, quite possible. The arithmetic operators provide mathematic operations on numbers. The remainder is supported with the $mod operator. The desired calculation can be done using the expression, for example:
pipeline = [
{
"$project": {
"result": {
"$mod": [
{
"$add": [
86400,
{ "$subtract": [7200, 28800] }
]
},
86400
]
}
}
}
]
Executing this aggregate pipeline on a collection will yield:
db.collection.aggregate(pipeline)
Sample Output
{
"_id" : ObjectId("58aacd498caf670a837e7093"),
"result" : 64800
}
Related
I have a document-scheme like this:
{
"pair":"BTCUSDT",
"ask":{
"amount":33107101.800000004,
"total":507,
"high":72000,
"low":65132
},
"bid":{
"amount":32368164.399999995,
"total":498,
"high":65131.99,
"low":60200.2
},
"updateStamp":1636632371639
}
now my DB there are documents with different values in pair and also documents with the same value. Some of them have a updateStamp that is, lets say a few seconds old, and some have a updateStamp that is a few minutes old or older.
(I wrote simpler values in updateStamp for simplicity)
{
"pair":"BTCUSDT",
"ask": ...,
"updateStamp": 100
},
{
"pair":"BTCUSDT",
"ask": ...,
"updateStamp": 200
},
{
"pair":"ETHUDST",
"ask": ...,
"updateStamp": 500
},
{
"pair":"ETHUDST",
"ask": ...,
"updateStamp": 200
},
{
"pair":"DOGEUSDT",
"ask": ...,
"updateStamp": 600
},
Now I want to compare every latest document of a pair and find the 10 documents, for the pairs with the largest ask.total-value. Simple saif, like a Top-10 from the latest of every pair.
But I don't get it how do manage this? I have been fiddeling around with aggregation and multiple finds for a while now. Maybe someone knows how to solve this?
You can first $group by pair. Then use the result to perform sub-pipeline $lookup to fetch the "last 10" documents.
In the sub-pipeline:
$match with pair; let is used to assign the value in grouped pair into variable p; which is later refered to as $$p. The $match means the variable $$p is equals to pair in the $lookup, which is equal to we only getting record related to that specific pair
$sort by updateStamp
$limit by 10
db.collection.aggregate([
{
$group: {
_id: "$pair"
}
},
{
"$lookup": {
"from": "collection",
let: {
p: "$_id"
},
pipeline: [
{
$match: {
$expr: {
$eq: [
"$pair",
"$$p"
]
}
}
},
{
$sort: {
updateStamp: -1
}
},
{
$limit: 10
}
],
"as": "output array field"
}
}
])
Here is the Mongo playground for your reference.
I am new to using mongodb and mongoose for my backend stack and Im having a hard time getting from SQL to NoSQL when it comes to query building.
I have an array of object that looks like this:
{
timestamp: "12313113",
symbol: "XY",
amount: 121212
value: 24324234
}
I want to query the collection to get the following output grouped by symbol:
{
symbol: xy,
occurences: 1231
summedAmount: 2131231
summedValue: 23131313
}
Could anyone tell me how to do it using aggregate on the Model? My timestamp filtering works already, but the grouping throws errors
let result = await TransactionEvent.aggregate([
{
$match : {
timestamp : { $gte: new Date(Date.now() - INTERVALS[timeframe]) }
}
},
{
$group : {
what to do in here
}
]);
Lets say I have another field in my object with a key of "direction" that can either be "IN" our "OUT". How could I also group the occurences of these values?
Expected output
{
symbol: xy,
occurences: 1231
summedAmount: 2131231
summedValue: 23131313
in: occurrences where direction property is "IN"
out: occurences where direction property is "OUT"
}
In MongoDB's $group stage, the _id key is mandatory and
it should be the keys which you want to be merged (It's symbol in your case).
Make sure that you pre-fix it with a `$ sign since you are referencing a key in your document.
Following the _id key, you can add all the additional operations to be performed for the required keys. In your specific use case, use $sum to add values to the user-defined key.
Note: Use "$sum": 1 to add 1 for each occurences ans "$sum": "$<Key-Name>" to add existing key's value.
Below code should be your $group stage
{
"$group": {
"_id": "$symbol", // Group by key (Use Sub-Object to group by multiple keys
"occurences": {"$sum": 1}, // Add `1` for each occurences
"summedAmount": {"$sum": "$amount"}, // Add `amount` values of grouped data
"summedValue": {"$sum": "$value"}, // Add `value` values of grouped data
}
}
Comment if you have any additional doubts.
You use $group and $sum
db.collection.aggregate([
{
"$group": {
"_id": "$symbol",
"summbedAmount": {
"$sum": "$amount"
},
"summbedValue": {
"$sum": "$value"
},
"occurences": {
$sum: 1
}
}
}
])
Working Mongo playground
Update 1
you can use $cond to check condition.
First parameter what is the condition
Second parameter - what we need to do if the condition is true (We need to increase by 1 if condition true)
Third parameter - what we need to do if the condition is false (No need to increase anything)
Here is the code
db.collection.aggregate([
{
"$group": {
"_id": "$symbol",
"summbedAmount": { "$sum": "$amount" },
"summbedValue": { "$sum": "$value" },
"occurences": { $sum: 1 },
in: {
$sum: {
$cond: [ { $eq: [ "$direction", "in" ] }, 1, 0 ]
}
},
out: {
$sum: {
$cond: [ { $eq: [ "$direction", "out" ] }, 1, 0 ] }
}
}
}
])
Working Mongo playground
I'm trying to create a mongodb query using the filtered value in the filter. For example:
var myIdVariable = '1jig23h34r34r30h';
var myVisibleVariable = false;
var myDistanceVariable = 100;
db.getCollection.find({
'_id': myIdVariable,
'isVisible': myVisibleVariable,
'distanceRange': {$lte: {myDistanceVariable - distanceRange}}
})
So, I want filter the distanceRange from database based on the calculation of (myDistanceVariable - distanceRange), with the distanceRange given in the same query.
I don't know if I give you a clear explanation of my problem. It's possible?
Thanks you.
Use the $expr operator to build a query expression that allows you to compare fields from the same document as well as compare the distanceRange field with the calculation of the field itself and your variables.
You would need to use the logical $and query operator to include the other query expressions thus your final query would look like the following:
db.getCollection('collectionName').find({
'$expr': {
'$and': [
{ 'isVisible': myVisibleVariable },
{ '$lte': [
'$distanceRange', {
'$subtract': [
myDistanceVariable, '$distanceRange'
]
}
] }
]
}
})
If your MongoDB server doesn't support the $expr operator then go for the aggregation framework route with $redact
db.getCollection('collectionName').aggregate([
{ "$redact": {
"$cond": [
{
'$and': [
{ 'isVisible': myVisibleVariable },
{ '$lte': [
'$distanceRange', {
'$subtract': [
x, '$distanceRange'
]
}
] }
]
},
"$$KEEP",
"$$PRUNE"
]
} }
])
Note
Including the _id in the query expressions means you are narrowing down your selection to just a single document and the query may not return any results since it's looking for a specific document with that _id AND the same document should satisfy the other query expressions.
mongodb native for node.js (driver version is 2.2.4 and MongoDB shell version: 3.2.9)
My collection has objects like this:
{x:[{v:0.002},{v:0.00002}],t:0.00202} //<this one has the full total in its values
{x:[{v:0.002},{v:0.002}],t:0.00202}
{x:[{v:0.002},{v:0.002}],t:0.3}
(shown here without their object ids)
I am unsure how to add up all the x.v to return only objects where the total of x.v is greater or equal to the objects t
aggregate({"t":{"$gte":{"$add":["x.v"]}}})
returns every object, I don't have any other idea on the order of syntax from reading the docs.
Can mongodb even do this in a query?
With MongoDB 3.2, a couple of approaches you can take here. You can query with the $where operator:
db.collection.find({
"$where": function() {
return (this.x.reduce(function (a, b) {
return a + b.v;
}, 0) > this.t);
}
})
Sample Output
/* 1 */
{
"_id" : ObjectId("587107b3cbe62793a0f14e74"),
"x" : [
{
"v" : 0.002
},
{
"v" : 0.002
}
],
"t" : 0.00202
}
But note this is bound to be a not very efficient solution since a query operation with the $where operator calls the JavaScript engine to evaluate JavaScript code on every document and checks the condition for each.
This is very slow as MongoDB evaluates non-$where query operations before $where expressions and non-$where query statements may use an index.
It is advisable to combine with indexed queries if you can so that the query may be faster. However, it's strongly recommended to use JavaScript expressions and the $where operator as a last resort when you can't structure the data in any other way, or when you are dealing with a small subset of data.
A better approach would be to use the aggregation framework where you can use the $unwind operator to flatten the array x, calculate the sums for x.v within a $group pipeline and subsequently filtering the documents using the $redact pipeline stage. This allows you to proccess the logical condition with the $cond operator and uses the special operations $$KEEP to "keep" the document where the logical condition is true or $$PRUNE to "remove" the document where the condition is false.
This operation is similar to having a $project pipeline that selects the fields in the collection and creates a new field that holds the result from the logical condition query and then a subsequent $match, except that $redact uses a single pipeline stage which is more efficient.
db.collection.aggregate([
{ "$unwind": "$x" },
{
"$group": {
"_id": "$_id",
"x": { "$push": "$x" },
"t": { "$first": "$t" },
"y": { "$sum": "$x.v" }
}
},
{
"$redact": {
"$cond": [
{ "$gt": [ "$y", "$t" ] },
"$$KEEP",
"$$PRUNE"
]
}
}
])
Sample Output
/* 1 */
{
"_id" : ObjectId("587107b3cbe62793a0f14e74"),
"x" : [
{
"v" : 0.002
},
{
"v" : 0.002
}
],
"t" : 0.00202,
"y" : 0.004
}
However, as much as this solution is better than the previous solution that uses $where, bear in mind that the use of $unwind operator can also limit performance with larger datasets since it produces a cartesian product of the documents i.e. a copy of each document per array entry, which uses more memory (possible memory cap on aggregation pipelines of 10% total memory) and therefore takes time to produce as well processing the documents during the flattening process.
Also, this solution requires knowledge of the document fields since this is needed in the $group pipeline where you retain the fields in the grouping process by using the accumulators like $first or $last. That can be a huge limitation if your query needs to be dynamic.
For the most efficient solution, I would suggest bumping your MongoDB server to 3.4, and use the combination of the $redact pipeline stage and the new $reduce array operator to filter the documents in a seamless manner.
The $reduce is for calculating the sum of the x.v fields in the array by applying an expression to each element in an array and combining them into a single value.
You can then use this an an expression with the $redact pipeline's evaluation to get the desired result:
db.collection.aggregate([
{
"$redact": {
"$cond": [
{
"$gt": [
{
"$reduce": {
"input": "$x",
"initialValue": 0,
"in": { "$add": ["$$value", "$$this.v"] }
}
},
"$t"
]
},
"$$KEEP",
"$$PRUNE"
]
}
}
])
Sample Output
/* 1 */
{
"_id" : ObjectId("587107b3cbe62793a0f14e74"),
"x" : [
{
"v" : 0.002
},
{
"v" : 0.002
}
],
"t" : 0.00202
}
Now that I've had a weekend of banging my head on $project, aggregate(), and $group, it's time for another round of throwing myself on your mercy. I'm trying to do a call where I get back the totals for users, grouped by sex (this was the easier part) and grouped by age range (this is defeating me).
I got it to work with one group:
Person.aggregate([
{
$match: {
user_id: id
}
},
{
$group: {
_id: '$gender',
total: { $sum: 1 }
}
}
])
.exec(function(err, result) {
etc...
From that, it'll give me how many men, how many women in a nice json output. But if I add a second group, it seems to skip the first and throw hissy fits about the second:
Person.aggregate([
{
$match: {
user_id: id
}
},
{
$group: {
_id: '$gender',
total: { $sum: 1 }
},
$group: {
_id: '$age',
age: { $gte: 21 },
age: { $lte: 30 },
total: { $sum: 1 }
}
}
])
.exec(function(err, result) {
etc...
It doesn't like the $gte or $lte. If I switch it to $project, then it'll do the gte/lte but throws fits about $sum or $count. On top of that, I can't find any examples anywhere of how to construct a multi-request return. It's all just "here's this one thing," but I don't want to make 12+ calls just to get all the Person age-groups. I was hoping for output that looks something like this:
[
{"_id":"male","total":49},
{"_id":"woman","total":42},
{"_id":"age0_10", "total": 1},
{"_id":"age11_20", "total": 5},
{"_id":"age21_30", "total": 15}
]
(I have no idea how to make the _id for age be something other than the actual age, which doesn't make sense, b/c I don't want an id of 1517191919 or whatever, I want a reliable name so I know where to output it in my template. So I do know that _id: "$age" won't give me what I want, but I don't know how to get what I want, either.)
The only time I've seen more than one thing, it was a $match, a $group, and a $project. But if $project means I can't use $sum or $count, can I do multiple $groups, and if I can, what's the trick to it?
As for the case of producing the results in different age groupings, the $cond operator of the aggregation framework can help here. As a ternary operator, it takes a logical result ( if condition ) and can return a value where true ( then ) or otherwise where false ( else ). In the case of varying age groups you would "nest" the calls in the else condition to meet each range until logically exhausted.
The overall case is not really practical to do in a single pass with both results for "gender" and "age" in groupings. Whilst it "could" be done, the only method is basically accumulating all data in arrays and working that out again for subsuquent groupings. Not a great idea, as it almost always would break the practical BSON limit of 16MB when attempting to keep the data. So a better approach is generally required.
As such, where the API supports ( you are under nodejs, so it does ), then it is usually best to run each query separately and combine the results. The node async library has just such features:
async.concat(
[
// Gender aggregator
[
{ "$group": {
"_id": "$gender",
"total": { "$sum": 1 }
}}
],
// Age aggregator
[
{ "$group": {
"_id": {
"$cond": {
"if": { "$lte": [ "$age", 10 ] },
"then": "age_0_10",
"else": {
"$cond": {
"if": { "$lte": [ "$age", 20 ] },
"then": "age_11_20",
"else": {
"$cond": {
"if": { "$lte": [ "$age", 30 ] },
"then": "age_21_30",
"else": "age_over_30"
}
}
}
}
}
},
"total": { "$sum": 1 }
}}
]
],
function(pipeline,callback) {
Person.aggregate(pipeline,callback);
},
function(err,results) {
if (err) throw err;
console.log(results);
}
);
The default execution of async.concat here will kick off the tasks to run in parallel, so both can be running on the server at the same time. Each pipeline in the input array will be passed to the aggregate method, which is going to then return the results and combine the output arrays in the final result.
The end result is not only do you have the results nicely keyed to age groups, but the two result sets appear to be in the same combined response, with no other work required to merge the content.
This is not only convenient, but the parallel execution makes this much more time efficient and far less taxing ( if not beating the impossible ) on the aggregation method being used to return the results.