Mongoose get array of Date with Values - node.js

i wanna enter a startDate and a endDate to my Mongoose Aggregation to get the registered Users foreach day between the range. I know i could user the normal count function but i thinkt its better to use the aggregations for it what do you think?
var startDate = 2016-01-05;
var endDate = 2016-01-07;
history : [
['2016-01-05', 23], // Day 1 = 23 Users
['2016-01-06', 34], // Day 2 = 34 Users
['2016-01-07', 43] // Day 3 = 43 Users
];
so i'm searching for something like
User.aggregate(
[
{
"$group": {
"created": {
$gte: startDate,
$lt: endDate
}
}
}
], function (err, users) {
history.push(users);
}
);

Aggregation indeed would be the ideal solution but how it generates the result is different from your expected output, depending on your application needs (I presume most probably you want to use this data
to display in a graph).
You could approach this in a couple of ways. Not the best solution since it depends on how your data is structured, i.e. it assumes the created date field in your users collection follows a time series (a users is created per day) and is a proper mongo date, but will at least get you going.
Let's consider the aggregation pipeline first. Since you want to produce a list of user counts for each day within a specified data range, you could construct
your pipeline in such a way that after the $match pipeline step, use the $group operator to create as the group key a string representation of the created date field using the $dateToString operator, which can be used to group by a day in the "YYYY-MM-DD" format. Then get the count of each accumulated documents using the $sum
operator:
var startDate = new Date("2016-01-05");
var endDate = new Date("2016-01-08");
var pipeline = [
{
"$match": {
"created": {
"$gte": startDate,
"$lt": endDate
}
}
},
{
"$group": {
"_id": {
"yearMonthDay": {
"$dateToString": {
"format": "%Y-%m-%d",
"date": "$created"
}
}
},
"count": { "$sum": 1 }
}
}
];
You can then run the aggregation pipeline, manipulate the result to get the final array in the desired format:
User.aggregate(pipeline, function (err, users){
if (err) {/* handle error */ }
console.log(JSON.stringify(users, null, 4));
var data = users.map(function (u){ return [u._id.yearMonthDay, u.count]; });
console.log(JSON.stringify(data, null, 4));
});

Related

nodejs: $lte for date is not working as expected in mongo db

As the question explains, I am trying to query on dates and the result is not as expected. Here is the how the objects, that I am trying to query, look like:
{
"_id":{"$oid":"5f660cfcde436c3035b59648"},
"orderid":"2020-09-19-8939",
"orderdate":"2021-08-09T11:02:31.202+00:00",
"paycondition_ref":{"$oid":"5f211e9690e310990ea6aa5d"},
"receivedate":{"$date":"2020-09-22T05:59:38.211Z"},
"duedate":"2020-09-19T13:51:56.219Z",
"currency":{"$oid":"5f660cfcde436c3035b59647"},
"supplier_ref":{"$oid":"5f2e12286b15925440f03b56"},
"history":false,
"remark":"Clarance - Tuesday"
}
And here is how I am querying it:
"$match": {
"$and": [
{ 'purchaseorder.orderdate': { $gte: new Date(startDate), $lte: new Date(endDate) } },
{ 'purchaseorder.history': false },
]
}
And the startDate and endDate objects look like this:
startDate 2021-08-08T19:00:00.000Z
endDate 2021-08-08T19:00:00.000Z
This should return the object above, shouldn't it? But it doesn't! What am I doing wrong here? How can I resolve this?
use ISODate instead of new Date and remove "purchaseorder"
"$match": {
"$and": [
{ 'orderdate': { $gte: ISODate(startDate), $lte: ISODate(endDate) } },
{ 'history': false },
]
}
Looking at your data object and query I noticed two things,
Why are you fetching data using purchaseorder.orderdate and purchaseorder.history'. You can directly access field
You are storing the date fields as string instead of date type, that is the reason comparison won't work when you do new Date . Change that schema to hold date field as type of date
Also you are storing dates in local Time Zone, so when comparing you should not use UTC to match your existing date. To make it simple it is always recommended to store dates in UTC TZ.
The following comparisons work for me:
db.getCollection("test-code").aggregate({
"$match": {
'orderdate':
{ $gte: new Date("2021-08-08T16:32:31.202+05:30"), $lte: new Date("2021-08-10T16:32:31.202+05:30") },
'history': false,
}
})
Note: I have used local TZ and my fields were of type date in DB 2021-08-10T16:32:31.202+05:30

Mongoose: Running Scheduled Job Query by Date

I want to create a scheduled job for patients in a hospital. The patients will be informed every month by their reg_date.
I'm using new Date().getDate() inside my scheduled Jobs to run at 8.00 AM in the morning to send SMS to my patients. Meanwhile, I had been using string format date to save reg_date in my mongoDB. Here is snippets of my mongoDB docs :
{
customer: "John",
reg_date: "2017-02-17T16:39:26.969Z"
}
I've ben surfing for solutions but it turns out nothing, so I decided to post myself. Here is what i am trying to do :
customer.find({"reg_date.getDate()" : new Date(2017, 03, 17).getDate()})
.then(function(data) {
for (var key in data.length) {
sendTheSMS(key[data]);
};
});
E.g: What I am doing is "I want to get every patient who register at 17th day of the month and send them a SMS".
Any help will be appreciated. :D
For this type of bit complex query you need to use aggregation method instead regular find method.
$project this will help you to project your fields, here we are creating a new temporary field day with only date of the reg_date. Then we query using the new field day and we get the result.
This temp field day will never added to your schema or model, it is just like temp view we are creating like in SQL.
Here i projected only customer and day but Please project all the fields necessary in the result.
function getCustomerList(day, callback){
customer.aggregate([
{
$project:{
"customer": "$customer", //repeat the same for all field you want in result
"reg_date": "$reg_date",
"day":{$dayOfMonth:"$reg_date"} //put day of month in 'day'
}
},
{
$match:{
"day": day //now match the day with the incoming day value
}
},
], function(err, result){
callback(err, result);
})
}
getCustomerList(17, function(err, result){ // call the function like this with date you want
// Process the err & result here
});
Result will be like this
[{
"_id" : ObjectId("571f2da8ca97eb10163e6e17"),
"customer" : "John",
"reg_date" : ISODate("2016-04-17T08:58:16.414Z"),
"day" : 17
},
{
"_id" : ObjectId("571f2da8ca97eb10163e6e17"),
"customer" : "Prasanth",
"reg_date" : ISODate("2016-04-17T08:58:16.414Z"),
"day" : 17
}]
Ignore the day field projected during your process...
With reg_date in string you can't query for day of month as it only works with ISODate. I suggest first you convert the string in reg_date in all your documents with a script.
Then the following query should work
customer.aggregate([
{
$project:{
"document": "$$ROOT", //to get the whole document
"day":{$dayOfMonth:"$date"} //put day of month in 'day'
}
},
{
$match:{
"day": 17 //match 17
}
},
], function(data) {
for (var key in data.length) {
sendTheSMS(key[data]);
};
})
Use greater than and less than
var previousDate =new Date(2017, 1, 16); //month starts with 0
var nextDate=new Date(2017, 1, 18);
customer.find({reg_date : { $gt:previousDate,$lt:nextDate}})
.then(function(data) {
for (var key in data.length) {
sendTheSMS(key[data]);
};
});
Since reg_date is stored as a string, and not a Date/ISODate, you're limited as to what kind of query you can run (so I concur with the comment in one of the other answers that you should consider converting them to proper ISODate).
Considering that you want to query a date string for entries with a particular day-of-month, you can use a regular expression query:
customer.find({ reg_date : /-17T/ })
Or, dynamically:
let today = new Date();
let dom = ('00' + today.getDate()).slice(-2); // zero-pad day of month
let re = new RegExp('-' + dom + 'T');
customer.find({ reg_date : re })
You should also read this regarding speed optimizations, but still, regex queries aren't very fast.

mapReduce not calling map nor reduce

I just started working with mongodb and I am having troubles using mapReduce function.
For some reason it seems to not be calling the map and reduce functions.
Here is my code:
#getMonthlyReports: (req, res) ->
app_id = req.app.id
start = moment().subtract('years', 1).startOf('month').unix()
end = moment().endOf('day').unix()
console.log(start)
console.log(end)
map = ->
geotriggers = 0
pushes = 0
console.log("ok")
date = moment(#timestamp).startOf('month').unix()
for campaign in #campaigns
if campaign.geotriggers?
geotriggers += campaign.geotriggers
else if campaign.pushes?
pushes += campaign.pushes
emit date,
geotriggers: geotriggers
pushes: pushes
reduce = (key, values) ->
console.log("ok")
geotriggers = 0
pushes = 0
for value in values
geotriggers += value.geotriggers
pushes += value.pushes
geotriggers: geotriggers
pushes: pushes
common.db.collection(req.app.id + "_daily_campaign_reports").mapReduce map, reduce,
query:
timestamp:
$gte: start
$lt: end
out:
inline: 1
, (err, results) ->
console.log(results)
ResponseHelper.returnMessage req, res, 200, results
I put some console.logs and it seems the map and reduce functions are not being called.
Also my results is undefined.
Is there something I am missing?
Apart from how I have already commented on the reason your mapReduce is failing is due to calling a library function that does not exist on your server (moment.js), this is not really a good usage of mapReduce.
While mapReduce has it's uses, a simple aggregation case like this is better suited to the aggregation framework as it is a native C++ implementation as opposed to mapReduce which runs inside a JavaScript interpreter. As a result the processing is much faster.
All you need are your existing unix timestamp values for start and end as well as the current day of the month ( dayOfMonth ) in order to do the date math:
db.collection.aggregate([
// Match documents using your existing start and end values
{ "$match": {
"timestamp": { "$gte": start, "$lt": end }
}},
// Unwind campaigns array
{ "$unwind": "$campaigns" },
// Group on the start of month value
{ "$group": {
"_id": {
"$subtract": [
"$timestamp",
{ "$mod": [ "$timestamp", 1000 * 60 * 60 * 24 * dayOfMonth ] }
]
},
"geotriggers": {
"$sum": {
"$cond": [
"$campaigns.geotriggers",
1,
0
]
}
},
"pushes": {
"$sum": {
"$cond": [
"$campaigns.pushes",
1,
0
]
}
},
}}
])
If I am reading your code correctly you have have each document containing an array for "campaigns", so to deal with this in the aggregation framework you use the $unwind pipeline stage to expose each array member as it's own document.
The date math is done in the $group stage for the _id key by changing the "timestamp" value to be equal to the starting date of the month which is the same thing that your code is trying to do. It's debatable that you could just use null here as your range selection is only going to result in a singular date value, but this is just to show that the date math is possible.
With the "unwound" array elements, we process every element just like the "for loop" does and conditionally adds the values for "geotriggers" and "pushes" using the $cond operator. Again this presumes as by your code these fields evaluate to boolean true/false which is the evaluation part of $cond
Your query condition is of course just met with the $match stage at the start of the pipeline, using the same range query.
That basically does the same thing without relying on additional libraries in server side processing and does it much faster as well.
See the other Aggregation Framework operators for reference.

Node, MongoDB (mongoose) distinct count

I have a collection with multiple documents and every one of them has and 'eID' field that is not unique. I want to get the count for all the distinct 'eID'.
Example: if there are 5 documents with the 'eID' = ObjectID(123) and 2 documents with 'eID' = ObjectID(321) I want to output something like:
{
ObjectID(123): 5,
ObjectID(321): 2
}
I don't know if that can be done in the same query but after knowing what are the most ocurring eID's I want to fetch the referenced documents using the ObjectID
Mongoose version 3.8.8
$status is the specific field of collection that i need to count distinct number of element.
var agg = [
{$group: {
_id: "$status",
total: {$sum: 1}
}}
];
model.Site.aggregate(agg, function(err, logs){
if (err) { return res.json(err); }
return res.json(logs);
});
//output
[
{
"_id": "plan",
"total": 3
},
{
"_id": "complete",
"total": 4
},
{
"_id": "hault",
"total": 2
},
{
"_id": "incomplete",
"total": 4
}
]
This answer is not in terms of how this query can be written via mongoose, but I am familiar with the nodejs MongoClient class if you have further questions regarding implementation.
The best (most optimal) way I can think of doing this is to use mapReduce or aggregation on your database. The closest thing to a single command would be the distinct command, which can be invoked on collections, but this will only give you an array of distinct values for the eID key.
See here: http://docs.mongodb.org/manual/core/map-reduce/
For your specific problem, you will want your map and reduce functions roughly as follows:
var map = function() {
var value = 1;
emit(this.eID, value);
};
var reduce = function(key, values) {
var result = 0;
for(var i=-1;++i<values.length;){
var value = values[i];
result += value;
};
return result;
};
There might be an easier way to do this using the aggregation pipeline (I would post the link but I don't have enough reputation).
I also found the mapReduce command for mongoose: http://mongoosejs.com/docs/api.html#model_Model.mapReduce

MongoDB/Mongoose querying at a specific date?

Is it possible to query for a specific date ?
I found in the mongo Cookbook that we can do it for a range Querying for a Date Range
Like that :
db.posts.find({"created_on": {"$gte": start, "$lt": end}})
But is it possible for a specific date ?
This doesn't work :
db.posts.find({"created_on": new Date(2012, 7, 14) })
That should work if the dates you saved in the DB are without time (just year, month, day).
Chances are that the dates you saved were new Date(), which includes the time components. To query those times you need to create a date range that includes all moments in a day.
db.posts.find({ //query today up to tonight
created_on: {
$gte: new Date(2012, 7, 14),
$lt: new Date(2012, 7, 15)
}
})
...5+ years later, I strongly suggest using date-fns instead
import endOfDayfrom 'date-fns/endOfDay'
import startOfDay from 'date-fns/startOfDay'
MyModel.find({
createdAt: {
$gte: startOfDay(new Date()),
$lte: endOfDay(new Date())
}
})
For those of us using Moment.js
const moment = require('moment')
const today = moment().startOf('day')
MyModel.find({
createdAt: {
$gte: today.toDate(),
$lte: moment(today).endOf('day').toDate()
}
})
Important: all moments are mutable!
tomorrow = today.add(1, 'days') does not work since it also mutates today. Calling moment(today) solves that problem by implicitly cloning today.
Yeah, Date object complects date and time, so comparing it with just date value does not work.
You can simply use the $where operator to express more complex condition with Javascript boolean expression :)
db.posts.find({ '$where': 'this.created_on.toJSON().slice(0, 10) == "2012-07-14"' })
created_on is the datetime field and 2012-07-14 is the specified date.
Date should be exactly in YYYY-MM-DD format.
Note: Use $where sparingly, it has performance implications.
Have you tried:
db.posts.find({"created_on": {"$gte": new Date(2012, 7, 14), "$lt": new Date(2012, 7, 15)}})
The problem you're going to run into is that dates are stored as timestamps in Mongo. So, to match a date you're asking it to match a timestamp. In your case I think you're trying to match a day (ie. from 00:00 to 23:59 on a specific date). If your dates are stored without times then you should be okay. Otherwise, try specifying your date as a range of time on the same day (ie. start=00:00, end=23:59) if gte doesn't work.
similar question
You can use following approach for API method to get results from specific day:
# [HTTP GET]
getMeals: (req, res) ->
options = {}
# eg. api/v1/meals?date=Tue+Jan+13+2015+00%3A00%3A00+GMT%2B0100+(CET)
if req.query.date?
date = new Date req.query.date
date.setHours 0, 0, 0, 0
endDate = new Date date
endDate.setHours 23, 59, 59, 59
options.date =
$lt: endDate
$gte: date
Meal.find options, (err, meals) ->
if err or not meals
handleError err, meals, res
else
res.json createJSON meals, null, 'meals'
i do it in this method and works fine
public async getDatabaseorderbyDate(req: Request, res: Response) {
const { dateQuery }: any = req.query
const date = new Date(dateQuery)
console.log(date)
const today = date.toLocaleDateString(`fr-CA`).split('/').join('-')
console.log(today)
const creationDate = {
"creationDate": {
'$gte': `${today}T00:00:00.000Z`,
'$lt': `${today}T23:59:59.999Z`
}
};
`
``
Problem I came into was filtering date in backend, when setting date to 0 hour, 0 minute, 0 second, 0 milisecond in node server it does in ISO time so current date 0 hour, 0 minute, 0 second, 0 milisecond of client may vary i.e. as a result which may gives a day after or before due to conversion of ISO time to local timezone
I fixed those by sending local time from client to server
// If client is from Asia/Kathmandu timezone it will zero time in that zone.
// Note ISODate time with zero time is not equal to above mention
const timeFromClient = new Date(new Date().setHours(0,0,0,0)).getTime()
And used this time to filter the documents by using this query
const getDateQuery = (filterBy, time) => {
const today = new Date(time);
const tomorrow = new Date(today.getDate() + 1);
switch(filterBy) {
case 'past':
return {
$exists: true,
$lt: today,
};
case 'present':
return {
$exists: true,
$gte: today,
$lt: tomorrow
};
case 'future':
return {
$exists: true,
$gte: tomorrow
};
default:
return {
$exists: true
};
};
};
const users = await UserModel.find({
expiryDate: getDateQuery('past', timeFromClient)
})
This can be done in another approach using aggregate if we have timezoneId like Asia/Kathmandu
const getDateQuery = (filterBy) => {
const today = new Date();
const tomorrow = new Date(today.getDate() + 1);
switch(filterBy) {
case 'past':
return {
$exists: true,
$lt: today,
};
case 'present':
return {
$exists: true,
$gte: today,
$lt: tomorrow
};
case 'future':
return {
$exists: true,
$gte: tomorrow
};
default:
return {
$exists: true
};
};
};
await UserModel.aggregate([
{
$addFields: {
expiryDateClientDate: {
$dateToParts: {
date: '$expiryDate',
timezone: 'Asia/Kathmandu'
}
}
},
},
{
$addFields: {
expiryDateClientDate: {
$dateFromParts: {
year: '$expiryDateClientDate.year',
month: '$expiryDateClientDate.month',
day: '$expiryDateClientDate.day'
}
}
},
},
{
$match: {
expiryDateClientDate: getDateQuery('past')
}
}
])
We had an issue relating to duplicated data in our database, with a date field having multiple values where we were meant to have 1. I thought I'd add the way we resolved the issue for reference.
We have a collection called "data" with a numeric "value" field and a date "date" field. We had a process which we thought was idempotent, but ended up adding 2 x values per day on second run:
{ "_id" : "1", "type":"x", "value":1.23, date : ISODate("2013-05-21T08:00:00Z")}
{ "_id" : "2", "type":"x", "value":1.23, date : ISODate("2013-05-21T17:00:00Z")}
We only need 1 of the 2 records, so had to resort the javascript to clean up the db. Our initial approach was going to be to iterate through the results and remove any field with a time of between 6am and 11am (all duplicates were in the morning), but during implementation, made a change. Here's the script used to fix it:
var data = db.data.find({"type" : "x"})
var found = [];
while (data.hasNext()){
var datum = data.next();
var rdate = datum.date;
// instead of the next set of conditions, we could have just used rdate.getHour() and checked if it was in the morning, but this approach was slightly better...
if (typeof found[rdate.getDate()+"-"+rdate.getMonth() + "-" + rdate.getFullYear()] !== "undefined") {
if (datum.value != found[rdate.getDate()+"-"+rdate.getMonth() + "-" + rdate.getFullYear()]) {
print("DISCREPENCY!!!: " + datum._id + " for date " + datum.date);
}
else {
print("Removing " + datum._id);
db.data.remove({ "_id": datum._id});
}
}
else {
found[rdate.getDate()+"-"+rdate.getMonth() + "-" + rdate.getFullYear()] = datum.value;
}
}
and then ran it with mongo thedatabase fixer_script.js
Well a very simple solution to this is given below
const start = new Date(2020-04-01);
start.setHours(0, 0, 0, 0);
const end = new Date(2021-04-01);
end.setHours(23, 59, 59, 999);
queryFilter.created_at={
$gte:start,
$lte:end
}
YourModel.find(queryFilter)
So, the above code simply finds the records from the given start date to the given end date.
Seemed like none of the answers worked for me. Although someone mentioned a little hint, I managed to make it work with this code below.
let endDate = startingDate
endDate = endDate + 'T23:59:59';
Model.find({dateCreated: {$gte: startingDate, $lte: endDate}})
startingDate will be the specific date you want to query with.
I preferred this solution to avoid installing moment and just to pass the startingDate like "2021-04-01" in postman.

Resources