I have a big mongodb query that has some dynamic properties based on filter options, and including filtering between dates. My query is currently causing scanned Objects / returned results ratio to go above 1000. I am sure my query can be improved as well as adding suitable indexes but I am not sure of the correct index's / improvements to my query.
const userFilter = user ? { assignee: new Types.ObjectId(user) } : null;
const clientFilter = client ? { client: new Types.ObjectId(client) } : null;
Collection.aggregate([
{
$match: {
...userFilter,
...clientFilter,
status: 1,
customer: new Types.ObjectId(customer),
},
},
{
$match: {
$or: [
{
end: {
$gte: new Date(end),
},
start: {
$lte: new Date(start),
},
},
{
end: {
$gte: new Date(end),
},
start: {
$lte: new Date(end),
$gte: new Date(start),
},
},
{
start: {
$lte: new Date(start),
},
end: {
$lte: new Date(end),
$gte: new Date(start),
},
},
{
start: {
$gte: new Date(start),
},
end: {
$lte: new Date(end),
},
},
{
start: {
$lte: new Date(end),
},
// #ts-ignore
start: {
$gte: new Date(start),
},
},
{
end: {
$lte: new Date(end),
},
// #ts-ignore
end: {
$gte: new Date(start),
},
},
],
},
},
{
$sort: { createdAt: 1 },
},
}
Depending on filter options we could be filtering by assignee and/or client.
We also only want returned results where the start or end date of the document falls within the start and end date filters.
I have tried a few variations of the query itself, as well as adding some compound indexes but had no real success improving the query or index's.
This is not an answer, but a bit too long for a comment:
Do you really like conditions like
{
$match: {
null,
null,
status: 1,
customer: new Types.ObjectId(customer),
},
}
Or should you better use:
let match = {
status: 1,
customer: new Types.ObjectId(customer)
};
if (user)
match.assignee = new Types.ObjectId(user);
if (client)
match.client = new Types.ObjectId(client);
Collection.aggregate([
{ $match: match },
...
I never tried, perhaps {$match: null} prevents any index use.
Related
Hi I am trying the below query in my nodejs code
const totalCount = await model.countDocuments({
'createdAt': { $gte: new Date(startDate), $lte: new Date(endDate) },
}).exec();
const activeCount = await model.countDocuments({
'createdAt': { $gte: new Date(startDate), $lte: new Date(endDate) },
'enabled': true,
}).exec();
const inactiveCount = (totalCount - activeCount);
return { totalCount, activeCount, inactiveCount };
Is there any way i can combine the above in a single query using aggregate in mongoose? Kindly guide me to the best solution .
Yes, quite simple using some basic operators, like so:
model.aggregate([
{
$match: {
createdAt: {
$gte: new Date(startDate),
$lte: new Date(endDate)
}
}
},
{
$group: {
_id: null,
totalCount: {
$sum: 1
},
activeCount: {
$sum: {
$cond: [
{
$eq: [
"$enabled",
true
]
},
1,
0
]
}
}
}
},
{
$project: {
_id: 0,
totalCount: 1,
activeCount: 1,
inactiveCount: {
$subtract: [
"$totalCount",
"$activeCount"
]
}
}
}
])
Mongo Playground
I am trying to show results where the sum of records is greater or equal to 4 and the status matches a string. If I leave off the status field it works fine but adding it in always gives me an empty array even when there should be data.
const bookings = await Booking.aggregate([
{
$group: {
_id: {
$dateToString: {
format: "%Y/%m/%d",
date: "$bookingDate",
},
},
totalBookings: {
$sum: 1,
},
},
},
{
$match: {
totalBookings: {
$gte: 4,
},
status: "Accepted",
},
},
]);
Each booking will have it's own status. So you need to add that as part of $group
{
$group: {
_id: {
status: "$status",
"d": {
$dateToString: {
format: "%Y/%m/%d",
date: "$bookingDate",
}
},
},
totalBookings: {
$sum: 1,
},
}
}
Then you need to change your match as below
$match: {
totalBookings: {
$gte: 4,
},
"_id.status": "Accepted",
}
It will give you all the Accepted Booking on the given BookinDate which is >= 4.
I am trying to do a query between dates. In compass I can do the query without any problem using the native function ISODate(). But when trying in my code I can't import that function, and new Date() is not warking.
Documents as example:
let trxs = [{
_id:612e112f7a7eaa7a5c1fd0d3
created:2021-09-31T11:23:25.184+00:00
amount:19.98
user:"612e112f7a7eaa7a5c1fd0d1"
type:"deposit"
},
{
_id:612e112f7a7eaa7a5c1fd0d6
created:2021-09-31T11:23:25.184+00:00
amount:10
user:"612e112f7a7eaa7a5c1fd0d4"
type:"deposit"
}
]
Query
let trxs = await Transaction.aggregate([
{
$match: {
type: req.query.type,
$and: [
{
created:
{
$gt: new Date(new Date().setHours(0, 0, 0))
},
},
{
created:
{
$lt:new Date(new Date().setHours(23, 59, 59))
}
}
]
}
}, {
$group: {
_id: null,
amount: {
$sum: '$amount'
}
}
}
]);
//More info:
console.log(new Date(new Date().setHours(0, 0, 0))) // 2021-08-31T22:00:00.953Z
console.log(new Date(new Date().setHours(23, 59, 59))) // 2021-09-01T21:59:59.952Z
//Error
ReferenceError: amount is not defined
I tried to import ISODate function but I don't find the way to do it.
Try the moment.js library. There is no need for $and: []
{
$match: {
type: req.query.type,
created: {
$gt: moment().startOf('day').toDate(),
$lt: moment().endOf('day').toDate(),
}
}
}
I've been pulling my hair out for weeks over this one.
I have a collection (this is a cut down version):
const SubscriberSchema = new Schema({
publication: { type: Schema.Types.ObjectId, ref: "publicationcollection" },
buyer: { type: Schema.Types.ObjectId, ref: "buyercollection" },
postCode: { type: String },
modifiedBy: { type: String },
modified: { type: Date }
});
I also have a collection containing the 1.75 million UK Postcodes
const PostcodeSchema = new Schema({
postcode: { type: String }
});
What I want to do is to return any record in the Subscriber collection which doesn't exist within the Postcode collection.
When I try a very simple aggregation using Mongoose on anything >100 records in the Subscriber collection, I'm getting either a timeout or a >16MB return error.
Here's what I've tried so far:
router.get(
"/badpostcodes/:id",
passport.authenticate("jwt", { session: false }),
(req, res) => {
const errors = {};
Subscriber.aggregate([
{
$match: {
publication: mongoose.Types.ObjectId(req.params.id),
postCode: { "$ne": null, $exists: true }
}
},
{
$lookup: {
'from': 'postcodescollections',
'localField': 'postCode',
'foreignField': 'postcode',
'as': 'founditem'
}
},
// {
// $unwind: '$founditem'
// },
{
$match: {
'founditem': { $eq: [] }
}
}
], function (err, result) {
if (err) {
console.log(err);
} else {
if (result.length > 0) {
res.json(result);
} else {
res.json("0");
}
}
})
}
);
The unwind didn't seem to do anything but it's commented out to show I tried to use it.
I've also tried using a pipeline on the lookup instead but that didn't work, similar to the following (sorry, I don't have my original code attempt so this is from memory only):
$lookup: {
'from': 'postcodescollections',
'let': { 'postcode': "$postCode" },
'pipeline': [
{
'$match': {
'postcode': { $exists: false }
}
},
{
'$unwind': "$postCode"
}
],
'as': 'founditem'
}
Thanks in advance so I can hopefully retain some hair!
You are doing a match on all postcodes that don't match and then unwinding those - that will be a 1.75m documents for each subscriber! The syntax in $lookup is also incorrect I think.
I think you can try something like the following - adjust accordingly for your data:
Do a $lookup to find a matching postcode in postcodes, then do a match to filter those subscribers that that don't have any founditem elements: "founditem.0": {$exists: false}
See an example:
db.getCollection("subscribers").aggregate(
[
// Stage 1
{
$match: {
postCode: { "$ne": null, $exists: true }
}
},
// Stage 2
{
$project: {
_id: 1,
postCode: 1
}
},
// Stage 3
{
$lookup: {
from: "postcodescollections",
let: { p: "$postCode" },
pipeline: [
{
$match: {
$expr:
{
$eq: ["$$p","$postcode"] }
}
},
{ $project: { _id: 1 } }
],
as: "founditem"
}
},
// Stage 4
{
$match: {
"founditem.0": {$exists: false}
}
},
]
);
I'm trying to make a query in my javascript code, when I try to execute the query it in robo3t it works, but when I try it in my angular code, it doesn't can you please help me?
Here is the code in robo3t.
db.getCollection('interviews').aggregate({
$match: {
status: {
$ne: 'Callback'
},
dateInserted: {
$gte: ISODate("2019-02-07 00:00:00"),
$lte: ISODate("2019-02-08 00:00:00")
},
'insertedBy.adminId': '5c353f840fe0fd000440df01'
}
},
{
$group: {
_id: {
insertedBy: '$insertedBy.email'
},
timeExported: {$first: '$dateInserted'},
total: {
$sum: 1
}
},
},
{
$limit: 100
}
)
and the result shows:
result image
Now here is my code in angular
query = [{
$match: {
status: {
$ne: 'Callback'
},
dateInserted: {
$gte: new Date("2019-02-07 00:00:00").toISOString(),
$lte: new Date("2019-02-08 00:00:00").toISOString()
},
'insertedBy.adminId': localStorage.getItem('_lgu_')
}
},
{
$group: {
_id: {
insertedBy: '$insertedBy.email'
},
timeExported: {$last: '$dateInserted'},
total: {
$sum: 1
}
},
},
{
$limit: 100
},
{
$sort: {
total: 1
}
}
]
Now when I try the query in angular, it doesn't give any result and when I remove the date condition:
dateInserted: {
$gte: new Date("2019-02-07 00:00:00").toISOString(),
$lte: new Date("2019-02-08 00:00:00").toISOString()
},
It will give a result but not what I am expecting.