Purging Data Logic in MongoDb based on Dates - node.js

I am unable to think of logic for the following Issue.
Consider I want to Store Data in mongoose in collection called packet where I can store data of different users. I want to purge the data once a certain threshold has been reached (Say for example 10 days). We know that the Mongoose by default gives us CreatedAt and UpdatedAt fields.
Suppose my data is created at 22nd February 2015 and Current Date is 24th February 2015.I will have a PurgeData number(column used for purging of data) as 2 (difference between the two dates). Every day I want to change the value of the PurgeData number by comparing the difference between the current date and the CreatedAt date. I want to schedule this operation every day and delete the data that has reached the threshold so I save memory space. Can Somebody help me with the logic for it and scheduling of the event?
Thanks in Advance

Haven't actually tried this but I would suggest some logic like schedule a job using the mongo-scheduler package where you first do an update and then remove the documents that breach the PurgeData condition. Something like this (untested):
/*
Scheduler Arguments
connection - mongodb connections string (i.e.: "mongodb://localhost:27017/scheduler-db") or a mongoose connection object
options - Options object
*/
var mongoose = require('mongoose');
var Scheduler = require('mongo-scheduer');
var scheduler = new Scheduler(connection, options);
var packet = new mongoose.Schema({
CreatedAt : {type : Date, default: Date.now},
PurgeData : {type : Number, index : true}
});
var Packet = mongoose.model('Packet', packet);
var moment = require('moment');
//Schedule the event.
var event = {
name: 'purge',
collection: 'packet',
after: new Date(),
cron: "0 15 10 * * ?" //cron string representing a frequency - fire at 10:15am every day
};
scheduler.schedule(event)
scheduler.on('purge', function(packet, event) {
console.log(packet.PurgeData)
Packet.find({"PurgeData": {"$lt": 10} }, function (err, packets) {
packets.forEach(function (p){
var days_diff = moment().diff(moment(p.CreatedAt), 'days') // using momentjs library
Packet.update(
{
"_id": p._id
},
{
"$set": {
PurgeData: days_diff
}
}, function (err, doc){
console.log(doc);
});
});
});
Packet.find({"PurgeData": {"$gte": 10} }, function (err, packets) {
packets.forEach(function (p){
Packet.remove({ "_id": p._id }, function (err){
handleErr(err);
});
});
});
})

Related

mongoDB query to pass from string to integer and then sum

I have these fields:
_id: {
contents: [{
duration: "4",
startDate : "2020-10-14",
}, {
duration: "15",
startDate : "2020-11-17",
}],
}
And I need to find the contents that will expire in the next 10 days.
This is the logic I have to follow:
db.collection.find()
If duration > 10 =>
If ((duration - 10) + startDate) ==== today
But I can't transform the duration or startDate into number, or even how to do the conditional properly
So for this ive made a collection in my mongo local that has a field that has numbers in it like the below image
Now to find sum of durations that are greater that 2....
Now as u see above the trick is to treat the comparator as a string as well.
Lemme attach a node snippet with mongoose as well
var mongoose = require('mongoose');
var mongoDB = 'mongodb://127.0.0.1/my_database';
mongoose.connect(mongoDB, {useNewUrlParser: true, useUnifiedTopology: true});
var db = mongoose.connection;
var Schema = mongoose.Schema;
var SomeModelSchema = new Schema({
duration: String
});
var SomeModel = mongoose.model('SomeModel', SomeModelSchema );
SomeModel.aggregate([{ $match: {duration: { $gt : "2" }}},{ $group: { _id : null, sum: { $sum: {$toDouble: "$duration" }} } }], function(err, data){
console.log(data)
});
db.on('error', console.error.bind(console, 'MongoDB connection error:'));
Strings can not take advantages of numbers operations with indexes. If the table is too big that could end up with a very slow query.
Question 1: why are that fields inserted as string (just curious)
Question 2: why not update all to add a new field 'expiration' with exactly the moment (date or Unix time). That way is easy for the database find those records.

How can I set correct timezone for a date field in Mongoose?

I'm using moment for generating time and date:
const moment = require('moment-timezone');
const emailModel = require('./api/models/emails');
sentTime=moment().tz('America/Los_Angeles').format();
console.log(sentTime); //console log shows correct time
emailModel.findOneAndUpdate({ _id: emailInfo._id }, {sentTime: sentTime }, { upsert: true },function (err, doc) {
if (err)
console.log(err);
});
And this is Schema that I'm using mongoose :
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
const EmailSchema = new Schema({
.
.
.
sentTime: {
type: Date,
trim: true
}
.
.
.
});
Problem is:
Console log shows correct time 2020-01-07T12:23:00-08:00 BUT mongoose saved incorrect timezone in DB : 2020-01-07T20:23:01.000+00:00
Currently the default behavior of Mongodb is to: (From the docs)
MongoDB stores times in UTC by default, and will convert any local
time representations into this form.
As a solution (and rightly so) what they recommend is:
Applications that must operate or report on some unmodified local time
value may store the time zone alongside the UTC timestamp, and compute
the original local time in their application logic.
Update:
Since you are already using moment-timezone a simple way I would go about this is:
Change the EmailSchema to have a timezone field and create a Mongoose virtual field on that schema to get adjusted time.
const schemaOpts = { toJSON: { virtuals: true } };
const EmailSchema = new Schema(
{
sentTime: {
type: Date,
trim: true
},
timeZone: {
type: String
}
},
schemaOpts
);
EmailSchema.virtual("adjustedTime").get(function() {
return moment.tz(this.sentTime, this.timeZone).format();
});
//fetching data
const result = await EmailSchema.findOne({}).exec();
console.info("result::", result.toJSON());
//note that if using .lean() for performance which has a caveat on using .toJSON()
trick for this case is before save, you need to add time with date. Ex: 2021/01/02 ==> 2021/01/02 15:00:00, ofcouse hour is always equal or greater than 04:00:00. Becase without time, date will be 00:00:00 and mongo will convert it to default timezone and substract hour with 4.

how to change date timezone in mongoose?

In model schema,
Using
updated: {
type: Date,
default: Date.now
In server.js
put(function(req, res) {
var query = {name: req.params.name};
// use our bear model to find the bear we want
Domain.find(query, function(err, domains) {
if (err)
res.send(err);
var domain = domains[0];
domain.password = req.body.password; // update the bears info
domain.updated = new Date();
// save the bear
domain.save(function(err, data) {
if (err)
res.send(err);
res.json({ status: 'success', message: 'domain updated!' }, data);
});
});
});
However,
In db side it shows,
"updated": "2016-02-27T16:20:42.941Z"
But, my timezone is UTC+02.00
So it should be like 18:20:42
What I'm doing wrong?
I'm using moment-timezone
npm install moment-timezone
const moment = require('moment-timezone');
const dateThailand = moment.tz(Date.now(), "Asia/Bangkok");
console.log(dateThailand); // "2018-08-20T16:35:14.033+07:00"
*** Asia/Bangkok +07:00
Schema in the mongoose.
const categorySchema = new Schema(
{
_id: {type: mongoose.Schema.Types.ObjectId, auto: true},
c_name: String,
created_by: String,
created_date: {type: Date, default: dateThailand},
updated_by: String,
updated_date: {type: Date, default: dateThailand}
}, {_id: false}
);
See up that created_date, updated_date: {type: Date, default: dateThailand }
Read more: http://momentjs.com/timezone/docs/
*If you using Robo 3T tool.
You can set "Display Dates In..."
Options > Display Dates In... > Local Timezone
:) Work for me.
The timestamps are timezone agnostic, stored as a unix timestamp. This timestamp will work across timezones, and node interprets it using current timezone of the server. The date you've shown is correctly stored. As soon as you'll retrieve it, if your server's timezone is UTC+2, it will show you correct time.
There is nothing wrong in your code. MongoDb saves date in UTC format no matter in whichever timezone you try to insert your date.
If you log domain.updated before saving in DB, result will be UTC+2 (your local time)
If you see updated column in DB, result will be in UTC
If you fetch updated column value from DB, then again result will be in UTC+2 (your local time)
I changed this,
var utc = new Date();
utc.setHours( utc.getHours() + 2);
domain.updated = utc;
Now it works.
You can create a Date Object from a specific UTC time:
new Date(Date.UTC(year, month, day, hour, minute, second))
Remember that no matter what you use to set time in mongoose schema, mongoose will always use UTC time, hence you need to dynamically allocate the UTC timestamp inside the Schema. Here it goes :-
var current = new Date();
const timeStamp = new Date(Date.UTC(current.getFullYear(),
current.getMonth(),current.getDate(),current.getHours(),
current.getMinutes(),current.getSeconds(), current.getMilliseconds()));
//Here goes your schema
const auditSchema = mongoose.Schema({
dateTime : { type: Date, default : timeStamp }
})
Using moment.js it is as easy as:
var moment = require('moment');
var utcDate = moment.utc().toDate();
Enjoy!

Time to live in mongodb, mongoose dont work. Documents doesnt get deleted

Im using this scheme for a session in my node.js app
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
// define the schema for our user session model
var UserSessionSchema = new Schema({
sessionActivity: { type: Date, expires: '15s' }, // Expire after 15 s
user_token: { type: String, required: true }
});
module.exports = mongoose.model('UserSession', UserSessionSchema);
And I create a "session" in my app with:
...
var session = new Session();
session.user_token = profile.token;
session.save(function(save_err) {
if (save_err) {
....
} else {
// store session id in profile
profile.session_key = session._id;
profile.save(function(save_err, profile) {
if (save_err) {
...
} else {
res.json({ status: 'OK', session_id: profile.session_id });
}
});
...
The problem is that the document lives permanetly, its never expires. It should only live for 15 seconds (up to a minute). Whats wrong with my code? I have tried to set the expries: string to a number i.e 15, to a string '15s' and so on.
var UserSessionSchema = new Schema({
sessionActivity: { type: Date, expires: '15s', default: Date.now }, // Expire after 15 s
user_token: { type: String, required: true }
});
A TTL index deletes a document 'x' seconds after its value (which should be a Date or an array of Dates) has passed. The TTL is checked every minute, so it may live a little longer than your given 15 seconds.
To give the date a default value, you can use the default option in Mongoose. It accepts a function. In this case, Date() returns the current timestamp. This will set the date to the current time once.
You could also go this route:
UserSessionSchema.pre("save", function(next) {
this.sessionActivity = new Date();
next();
});
This will update the value every time you call .save() (but not .update()).
To double check the indexes that have been created in the DB you can run this command in your mongo shell db.yourdb.getIndexes(). When changing the indexes you have to manually delete it in the collection before the new one will take effect. Check here for more information Mongoose expires property not working properly

how to get number of documents which is created today with mongoose?

how to get number of documents which is created today with mongoose? I'm using MEAN stack. I read Mongoose Api Docs but doesn't understand anything :/
By default there is no way.
Documents don't have creation date associated with them unless you explicitly store it in MongoDB.
In other words your Mongoose schema should have something like created field:
const blogSchema = new Schema({
created: {type: Date, default: Date.now}
});
Then search for matching docs:
const now = new Date();
const today = new Date(now.getFullYear(), now.getMonth(), now.getDate());
doc.find({created: {$gte: today}}).exec(callback);
Supposing you have the date field createdAt in your schema and you would like to get the number of users registered today, you should create a two date object instances to use in your date range query, start and end dates.
Your start date object should hold the current date time hours at 00:00:00.000 (milliseconds precision) and set the hours for today's date to 23:59:59.999 to the end date variable:
var start = new Date();
start.setHours(0,0,0,0);
var end = new Date();
end.setHours(23,59,59,999);
Then pass the modified date objects as usual in your MongoDB aggregation pipeline in the $match operator:
var pipeline = [
{
"$match": {
"createdAt": { "$gte": start, "$lt": end }
}
},
{
"$group": {
"_id": null,
"count": { "$sum": 1 }
}
}
];
Model.aggregate(pipeline, function (err, result){
if (err) throw new Error();
console.log(JSON.stringify(result));
});
If you are using the momentjs library, this can be done by using the startOf() and endOf() methods on the moment's current date object, passing the string 'day' as arguments:
var start = moment().startOf('day'); // set to 12:00 am today
var end = moment().endOf('day'); // set to 23:59 pm today

Resources