Mongoose: can't query array of dates - node.js

I have a test document like that:
{
"_id" : ObjectId("5fb6b0ed9cad6e97cfc24c2d"),
"dates" : [
{
"date" : ISODate("2020-01-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-02-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-03-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-04-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-05-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-06-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-07-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-08-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-09-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-10-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-11-01T00:00:00.000Z")
},
{
"date" : ISODate("2020-12-01T00:00:00.000Z")
}
]
}
Now I want to retrieve only dates $gt: 2020-14-01T00:00:00.000Z. I tried a lot of combinations but none of them worked for me.
This is one of the queries I tried (taken from Mongodb docs):
db.getCollection('things').find({_id: ObjectId("5fb6b0ed9cad6e97cfc24c2d"), "dates.date": { $gt: new Date("2020-04-01T00:00:00.000Z")} } )
But it return all the document, not the gt... i tried using new Date and new ISODate too, but same effect.
Thank you.

First of all, according to mongo documentation dates are in the format "<YYYY-mm-dd>".
Also more formats are allowed, but If you try to use 2020-14-01 as date it will fail (unless you convert string to date with an specific format) because month is 14.
But, answering the question, you need a query like this:
EDITED
db.collection.aggregate({
"$match": {
"_id": ObjectId("5fb6b0ed9cad6e97cfc24c2d"),
}
},
{
"$project": {
"dates": {
"$filter": {
"input": "$dates",
"as": "item",
"cond": {
"$gt": [
"$$item.date",
ISODate("2020-01-14T00:00:00.000Z")
]
}
}
}
}
})
First $match by _id to get only the document you want. And then using $project to create the fields you want to get. You can filter in the array those values whose field date is greater than your date using $filter and $gt
Note that I've used 2020-01-14 to avoid errors.
Example here.
Also another example using $dateFromString in this query.
Edit: You can use $dateFromString and specify a format. Check this example

Related

Sort From Nested Object Array MongoDB - NodeJS

hope you are doing well and Happy New Year to everybody. Hope it will end up better than 2020 X). I am facing an issue here that I can not find a solution so far. Here is a data MongoDB model example :
`{ "_id" : ObjectId("5fe715bf43c3ca0009503f1d"),
"firstname" : "Nicolas ",
"lastname" : "Mazzoleni",
"email" : "nicolas.mazzoleni#orange.fr",
"items" : [
{
"item" : ObjectId("5f6c5422eeaa1364b0d7b267"),
"createdAt" : "2020-12-26T10:51:43.685Z"
},
{
"item" : ObjectId("5f6c5422eeaa1364b0d7b270"),
"createdAt" : "2021-01-04T09:21:46.260Z"
}
],
"createdAt" : ISODate("2020-12-26T10:51:43.686Z"),
"updatedAt" : ISODate("2021-01-04T09:21:46.260Z"),
"__v" : 0
}`
Let's assume that I have a lot of rows like this one above, with different 'items.createdAt' dates.
At the end, the goal is to do a query that would find an item where the ObjectId is equal to what I am looking for, and sort by the matching 'items.createdAt' date. Here is the query I use at the moment that is not sorting by the matching nested object :
`const items = await repo
.find({ 'items.item': ObjectId("5f6c5422eeaa1364b0d7b270") })
.sort({ 'items.createdAt': -1 })
.limit(5)`
PS : I also tried to use aggregations that ended up with this query
`const items = await repo.aggregate([
{ $unwind: '$items' },
{ $match: { 'items.item': ObjectId("5f6c5422eeaa1364b0d7b270") } },
{ $sort: { 'items.createdAt': -1 } },
{ $limit: 5 },
])`
This one is working like a charm, but it does not use indexes, and I am searching into millions of records, which makes this query way to long to execute.
Thank you for all your help ! Nicolas
Unfortunately, MongoDB can't sort nested array by simple Query. You need to use aggregate for that.
db.collection.aggregate([
{
"$match": {
"items.item": ObjectId("5f6c5422eeaa1364b0d7b270")
}
},
{
"$unwind": "$items"
},
{
"$sort": {
"items.createdAt": -1
}
},
{ "$limit" : 5 },
{
"$group": {
"items": {
"$push": "$items"
},
"_id": 1
}
])

The api is returning the date that matches and also dates that does not matches the specified date

I'am developing an api in nodejs.
I have the document in follwing stucture:
{
"_id" : ObjectId("5ecd26504df3372a38afffd9"),
"balance" : 104000,
"bankID" : "Bank-1",
"userEmail" : "kumarshreyas073#gmail.com",
"bankName" : "Corporation Bank",
"accountNumber" : "03214569874563",
"ifsCode" : "CORP0001236",
"branch" : "Udupi",
"address" : "Udupi",
"city" : "Udupi",
"state" : "Karnataka",
"openingBalance" : 100000,
"transactions" : [
{
"credit" : 2000,
"debit" : 0,
"_id" : ObjectId("5ecd26614df3372a38afffea"),
"transactionID" : "CashTransaction-5ecd26614df3372a38afffe8",
"date" : "30-05-2026",
"particulars" : "By Cash-1",
"voucherType" : "Cash"
},
{
"credit" : 0,
"debit" : 2000,
"_id" : ObjectId("5ecd272d4df3372a38b00012"),
"transactionID" : "Receipt-5ecd272d4df3372a38b00009",
"date" : "29-07-2020",
"particulars" : "To Suresh kumar",
"voucherType" : "Receipt"
},
{
"credit" : 0,
"debit" : 2000,
"_id" : ObjectId("5ecd272d4df3372a38b00014"),
"transactionID" : "Receipt-5ecd272d4df3372a38b00003",
"date" : "30-05-2024",
"particulars" : "To Karthik",
"voucherType" : "Receipt"
}
],
"idCounter" : 1,
"__v" : 0
}
I need to extract only those transactions between from date = "20/07/2020" and to date = "31/07/2020".
The code I written is as follows:
exports.trail_balance = async (req, res, next) => {
var trailBalance = {
userEmail: req.body.userEmail,
fromDate: req.body.fromDate,
toDate: req.body.toDate,
};
var bankAccount = await Bank.aggregate([
{ $match: { userEmail: req.body.userEmail } },
{
$addFields: {
transactions: {
$filter: {
input: "$transactions",
as: "transactions",
cond: {
$and: [
{
$gte: ["$$transactions.date", trailBalance.fromDate],
},
{
$lte: ["$$transactions.date", trailBalance.toDate],
},
],
},
},
},
},
},
]);
res.status(200).json({
result: 1,
bankAccount: bankAccount.length > 0 ? bankAccount : [],
});
};
Actual result I expect is:
{
"result": 1,
"bankAccount": [
{
"_id": "5ecd26504df3372a38afffd9",
"balance": 104000,
"bankID": "Bank-1",
"userEmail": "kumarshreyas073#gmail.com",
"bankName": "Corporation Bank",
"accountNumber": "03214569874563",
"ifsCode": "CORP0001236",
"branch": "Udupi",
"address": "Udupi",
"city": "Udupi",
"state": "Karnataka",
"openingBalance": 100000,
"transactions": [
{
"credit" : 0,
"debit" : 2000,
"_id" : ObjectId("5ecd272d4df3372a38b00012"),
"transactionID" : "Receipt-5ecd272d4df3372a38b00009",
"date" : "29-07-2020",
"particulars" : "To Suresh kumar",
"voucherType" : "Receipt"
}
],
"idCounter": 1,
"__v": 0
}
But, I'am getting all transactions.
I even tried passing date in, from date = "20-07-2020" and to date = "31-07-2020". This too returns all transaction.
All the date stored in DB are of type String.
The problem is your date format. As you have started your date with day in your saved data in database and type of date is string, so in comparison to your query, it always start with day and it's incorrect. because in date comparison, at first years must be compared, then months and lastly, day. But you are doing it in wrong way.
In this scenario, mongodb is doing write! because in your from date, 2 is less or equal to 2 and 3 and in your to date, 3 is greater or equal to 2 and 3. So its doing well.
I changed your saved data date format to yyyy-mm-dd and your query was correct.
If changing data is not possible for you, you can also change data in a pipeline stage of your aggregate query. Use the link below:
https://docs.mongodb.com/manual/reference/operator/aggregation/dateFromString/
You are almost there. The problem is with your data. It has invalid date format.
mongo playground
I changed to proper date format for one of the transactions as below
"date": ISODate("2026-05-30"),
mongo date format
So if you have proper format, then the query will work.
new Date("<YYYY-mm-dd>") returns the ISODate with the specified date.
new Date("<YYYY-mm-ddTHH:MM:ss>") specifies the datetime in the client’s local timezone and returns the ISODate with the specified datetime in UTC.
new Date("<YYYY-mm-ddTHH:MM:ssZ>") specifies the datetime in UTC and returns the ISODate with the specified datetime in UTC.
new Date(<integer>) specifies the datetime as milliseconds since the Unix epoch (Jan 1, 1970), and returns the resulting ISODate instance.
Suggesting few fixes,
convert your string date to ISO date using new Date("2020-07-31"),
var trailBalance = {
userEmail: req.body.userEmail,
fromDate: new Date(req.body.fromDate),
toDate: new Date(req.body.toDate),
};
convert collection's field transactions.date string to ISO date using $dateFromString
format: %d-%m-%Y should match exact format of date in transactions.date
{
$dateFromString: {
dateString: "$$transactions.date",
format: "%d-%m-%Y"
}
}
Look at the Working Playground: https://mongoplayground.net/p/-KWgRCSwD8h
Your final query,
var bankAccount = await Bank.aggregate([
{
$match: {
userEmail: trailBalance.userEmail
}
},
{
$addFields: {
transactions: {
$filter: {
input: "$transactions",
as: "transactions",
cond: {
$and: [
{
$gte: [
{
$dateFromString: {
dateString: "$$transactions.date",
format: "%d-%m-%Y"
}
},
trailBalance.fromDate
]
},
{
$lte: [
{
$dateFromString: {
dateString: "$$transactions.date",
format: "%d-%m-%Y"
}
},
trailBalance.toDate
]
}
]
}
}
}
}
}
]);

mongodb - Find first subdocument that matches query

I've created a database schema very similar to the documentation on time series data here.
I ended up with the following documents:
{
"_id" : "unique_name_1",
"data" : {
"2017" : {
"5" : {
"21" : {
"61" : [
{
"timestamp" : 1498162890460.0,
"value" : true
}
],
"84" : [
{
"timestamp" : 1498183202126.0,
"value" : false
}
]
},
"22" : {
"24" : [
{
"timestamp" : 1498215602957.0,
"value" : true
}
],
"61" : [
{
"timestamp" : 1498249322863.0,
"value" : false
}
]
}
},
"9" : {
"16" : {
"36" : [
{
"timestamp" : 1508249031987.0,
"value" : true
},
{
"timestamp" : 1508249429052.0,
"value" : false
}
]
}
}
}
}
}
The first subdocument under 'data' represents the year. The second subdocument represents the month. The third subdocument represents the day. And the fourth subdocument represents the 15 minute interval of the day in which the event happened.
I want to query the database and get the first true (and maybe the immediate false that follows if possible) every day while ignoring all subsequent entries.
The first entry every day may or may not be true. The data does not necessarily always go from true to false back to true. For example, I may have several trues in a row, in which case I would want to ignore all subsequent true values.
This structure is really great for querying specific times, if you know them, but I'm at a loss for querying specific values. Should I just query the entire document and parse it on the front end? That becomes more difficult if I want to run the same query on hundreds of documents.
Thanks for the help.
To analyze your data in context of single day you have to use a long series of $objectToArray combined with $unwind. That's because the only way to figure out what represents day, month, year is to analyze the level of nesting. However once you do that, the rest might be quite simple. Try:
db.col.aggregate([
{
$project: {
data: { $objectToArray: "$data" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: "$data.k",
data: { $objectToArray: "$data.v" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: 1,
month: "$data.k",
data: { $objectToArray: "$data.v" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: 1,
month: 1,
day: "$data.k",
data: { $objectToArray: "$data.v" }
}
},
{
$unwind: "$data"
},
{
$project: {
year: 1,
month: 1,
day: 1,
interval: "$data.k",
data: "$data.v"
}
},
{
$unwind: "$data"
},
{
$match: { "data.value": true }
},
{
$sort: {
"data.timestamp": 1
}
},
{
$group: {
_id: {
year: "$year",
month: "$month",
day: "$day"
},
firstPositive: {
$first: "$data"
}
}
}
])
Note that because you're using key/value pairs rather than an array, you would need to use wildcard keys in order to perform such a query. This simply isn't possible in MongoDB.
That being said, there's a new aggregation operator as of version 3.6 called $objectToArray which may prove useful for resolving this problem. I would recommend taking your research there and attempting to develop the appropriate aggregation pipeline with that operator in mind.
If you get stuck on the pipeline, then that would be a good time to return with a new question showing what you've managed to write up so far. Until then, there's unfortunately not much more that can be done to help you work around your existing document structure.
The alternatives, of course, are either to completely restructure your documents (the nuclear option), which may not be a possibility if you're working on a production system which would require updating all relevant pieces of code, or to process this data directly on the backend instead of through a MongoDB query.

NodeJS MongoDB: Querying ISO Date

I have a collection of ISO dates stored in MongoDB as strings, like so:
{ "date": "2014-12-12T03:33:33.333Z" },
{ "date": "2014-12-13T03:33:33.333Z" }
In the console, I can query these perfectly using
{ "date": ISODate("2014-12-12T03:44:00.000Z") }
However, I'm using the NodeJS driver, and I cannot use ISODate. I have found several questions pertaining to this problem on here, but none of the proposed solutions seem to work. For instance:
// These does not find any matches
db.find({ "date": new Date("2014-12-12T03:44:00.000Z") })
db.find({ "date": { '$eq': '2014-12-12T03:44:00.000Z' } })
db.find({ "date": { '$eq': new Date('2014-12-12T03:44:00.000Z') } })
//This throws an error stating $date is not an operator
db.find({ "date": { '$date': '2014-12-12T03:44:00.000Z' } })
Why are these queries failing?
Edit: Here's another sample, straight from the database:
{
"_id": "5a7e88f34b5916723589183f",
"date": "2014-12-12T03:42:00.000Z",
"granularity": 180
}
EDIT 2: This query produces the following error MongoError: $dateFromString requires that 'dateString' be a string, found: date with value 2014-12-12T03:44:00.000Z
async loadCandle(date, granularity) {
date = date + ''; //Aded to ensure date is a string, but still get the error.
var candle = await this.data.collection('dates').findOne(
{ $expr :
{$eq :
[
{$dateFromString : {dateString : "$date"}},
new Date("2014-12-12T03:33:33.333Z") //Normally would pass in the `date` variable here
]
} });
because $date is not an operator
you need to use $dateFromString to convert string date to ISODate for comparison
db.datez.find(
{$expr :
{$eq :
[
{$dateFromString : {dateString : "$date"}},
new Date("2014-12-12T03:33:33.333Z")
]
}
}
)
using aggregation
db.datez.aggregate([
{$match :
{$expr :
{$eq :
[
{$dateFromString : {dateString : "$date"}},
new Date("2014-12-12T03:33:33.333Z")
]
}
}
}
])
collection
> db.datez.find()
{ "_id" : ObjectId("5a7e795e80aae386f73cf0fe"), "date" : "2014-12-12T03:33:33.333Z" }
{ "_id" : ObjectId("5a7e795e80aae386f73cf0ff"), "date" : "2014-12-13T03:33:33.333Z" }
>
result
> db.datez.find({$expr : {$eq : [{$dateFromString : {dateString : "$date"}}, new Date("2014-12-12T03:33:33.333Z")]}})
{ "_id" : ObjectId("5a7e795e80aae386f73cf0fe"), "date" : "2014-12-12T03:33:33.333Z" }
You can use $dateToString operator which generate the string date of any specified format, which can be compared later.
For string comparison of date, input format should be YYYY-MM-DD, any other format would get fail for ranges date queries
Let me explain through example:
Here is my collection in mongoDb :
{
"_id" : ObjectId("5f2d0a0c632ec022e08c3191"),
"date" : ISODate("2020-07-12T00:00:00Z")
}
{
"_id" : ObjectId("5f2d0a12632ec022e08c3192"),
"date" : ISODate("2020-07-13T00:00:00Z")
}
Now the query to be fired from Node for comparison of such stored ISODates is as follow
db.collection.aggregate(
[
{
$addFields: {
formattedDate: { // An extra field "formattedDate" is added in each document which can be compared later through pipeline using $match
$dateToString: {
format: "%Y-%m-%d",
date: "$date" // in "$date" date is variable from db
}
}
}
},
{
$match: {
formattedDate: {
$eq: "2020-07-12" // here you can provide your input date yyyy-mm-dd
}
}
}
]
)
So for above query you will get output as
{
"_id" : ObjectId("5f2d0a0c632ec022e08c3191"),
"date" : ISODate("2020-07-12T00:00:00Z"),
"formattedDate" : "2020-07-12"
}
Hope this will help you or somebody else!

How to get count and max date with single query in MongoDB?

I have Post collection as like as following:
{ "_id" : ObjectId(..), "date" : ISODate("2014-03-01T08:00:00Z") }
{ "_id" : ObjectId(..), "date" : ISODate("2014-03-01T09:00:00Z") }
{ "_id" : ObjectId(..), "date" : ISODate("2014-03-15T09:00:00Z") }
{ "_id" : ObjectId(..), "date" : ISODate("2014-04-04T11:21:39.736Z") }
{ "_id" : ObjectId(..), "date" : ISODate("2014-04-04T21:23:13.331Z") }
I need to get total count and max date of post. So desired result for coeumtns above is the following:
{count: 5, date: ISODate("2014-04-04T21:23:13.331Z")}
How to get desired result with single query to MongoDB without handling and counting in application code?
EDIT: #chridam thanks for the response. I've accepted your answer as best one! Could help me with one more thing?
Let's say that posts are not exists yet, so I need to fetch result with zero count and current date as timestamp like the following:
{count: 0, [Date.now()]}
Is it possible with MongoDB ?
Use the $max and $sum operators as
Model.aggregate([
{
"$group": {
"_id": null,
"count": { "$sum": 1 },
"date": { "$max": "$date" }
}
}
]).exec(function (err, result) {
console.log(result);
})
EDIT: Addressing your further question with regards to an empty collection, the aggregate function will return an empty cursor since there wont be any documents to aggregate in the collection. So you would need to address this logic on the client i.e. check the results from the above aggregation, if the result is an empty array then create the placeholder doc as required:
Model.aggregate([
{
"$group": {
"_id": null,
"count": { "$sum": 1 },
"date": { "$max": "$date" }
}
}
]).exec(function (err, result) {
console.log(result);
if (!result.length) {
result = [{ count:0, date: new Date() }];
}
});

Resources