Let's say I am creating a web application that posts event markers to a map to help flash mobs find out where to meet up. In this hypothetical app and due to the nature of flash mobs, these markers would have to become inactive after a certain time interval so as to not pollute the map with events that are no longer occurring. My question is what is the appropriate way to allow this to happen in mongoDB using node.js without deleting the document?
If I have a mongoose model (Marker) like this:
const mongoose = require('mongoose');
const markerSchema = new mongoose.Schema({
name: {
type: String,
},
description: {
type: String,
trim: true,
},
createdAt: {
type: Date,
default: Date.now(),
select: false,
},
expiresAt: {
type: Date
},
active: Boolean,
location: {
type: {
type: String,
enum: ['Point'],
required: true,
},
coordinates: {
type: [Number],
required: true,
},
},
});
markerSchema.pre('save', function (next) {
this.expiresAt = this.createdAt + 60 * 60 * 1000 * 6;
next();
});
const Marker = mongoose.model('Marker', tourSchema);
module.exports = Marker;
What would be the best way to update the markers active field to false after an arbitrary amount of time, say 6 hours?
I have researched Mongodb's TTL and I have also considered using the createdAt field to query for markers that are within the time frame of active using Marker.find({ expiresAt: {$gt: Date.now() }) but I do not know if this would be the best. Is there a way to run a function or middleware that periodically checks each document and sets the active field to false if the time frame is up? Just curious how you would approach this problem so querying Marker.find({ active: true }) would provide the data I am looking for.
Setting TTL in MongoDB runs a scheduled task every 60 seconds and will remove the document once expired. If you don't want to remove the document and mark it instead.
Run a CRON/Scheduled task every minute
Update the document matching your custom TTL criteria
Use the criteria in all select clause to ignore expired documents.
You can use node-cron to achieve your goal.
For example, rub a job every minute :
cron.schedule('* * * * *', function() {
console.log('running a task every minute');
});
More details and full usage example tutorials :
https://www.digitalocean.com/community/tutorials/nodejs-cron-jobs-by-examples
https://blog.logrocket.com/task-scheduling-or-cron-jobs-in-node-using-node-cron/
Related
so i know there is something called TTL in mongo but i dont think it will work for what i want to do, i want to set a document from my schema where the default value is basic but when the customer pays his membership i want to set it to "plus" i did that query in my controller, but i want to know if there is a way to make this document when the value is "plus" to have an expiration time, like 1 week or 1 month, and when the time is out set it again to basic, there is the schema
const UserSchema = new Schema({
username: {type: String, required: true},
nombre_empresa: {type: String},
email: {type: String, required: true, unique: true},
password: {type: String, required: true},
tipo_cuenta: {type: String, required: true},
isNewUser: {type: String, default: 'basic'}
},
{
timestamps: true
},
{
typeKey: '$type'
});
and here the process when i change his status
userCtrl.renderMembershipSucess = async (req, res) => {
const status = 'plus'
const status_basic = 'basico'
const status_user = await User.findByIdAndUpdate(
req.user.id, {
$set: { isNewUser: status }
}
)
console.log(status_user)
res.render('users/sucess')
}
Rather than relying on a trigger that must fire and correctly update the user records, store the expiration and check it when determining if the user is plus or not.
For example, you might add a plusExpires field to the user schema with a default value of null. When you upgrade the user to plus, set plusExpires to the Date that it should no longer be valid.
Add an instance method to the schema to perform that check:
UserSchema.methods.isPlus = function() {
return this.isNewUser == "plus" && (( this.plusExpires == null ) || (this.plusExpires > new Date())
}
Then any time you need to test if a user is plus or not, just call User.isPlus() on the user object.
i dont think there mongodb provide a built-in way to do what you want, however you can run a cronjob every midnight to fetch your "plus" customers, check their membership and update it if necessary.
another way maybe to design your mongodb collections to leverage the TTL, you can separate the User and PlusMembership collections, when a user make a payment, insert a PlusMembership data with TTL
I created a User schema in my React App as follows:
const userSchema = new Schema(
{
profileId: String,
expirationDate: { type: Date, default: new Date() },
credits: { type: Number, default: 0 },
},
{ timestamps: { createdAt: "created_at" } }
);
When the user pays me, I want to reset/update two fields: the expirationDate and credits via a post method. Here’s the code I use on my expressjs backend server to update the database entry on MongoDB Atlas:
req.user.expirationDate = new Date(
req.user.expirationDate.setDate(req.user.expirationDate.getDate() + 30)
);
req.user.credits += 1;
const user = await req.user.save();
res.send(user);
Once the operation succeeded, I can see the field of “credits” gets updated (increased by 1). However, the “expirationDate” field remains unchanged. What’s more curious is that when I send the updated user object to my frontend server with “res.send(user)”, I can see the updated expirationDate in the console.
Successfully updated user model as designed/intended: seen from frontend console
But below is what I saw in my mongoDB:
Updated user entry in MongoDB: the Date field "expirationDate" is not updated; but, the "credits" field is.
What is going on here? How to fix it?
I was having a similar issue recently and haven't figured out the actual reason behind this, but as a workaround try telling mongoose explicitly that the expirationDate-field was changed:
req.user.expirationDate = new Date(
req.user.expirationDate.setDate(req.user.expirationDate.getDate() + 30)
);
req.user.markModified('expirationDate');
await req.user.save();
EDIT:
Just debugged it again and I think the reason behind this behaviour is your default value for expirationDate. Try passing it the Date.now function instead of immediately setting it to a new date:
expirationDate: {type: Date, default: Date.now},
This fixed it for me without having to use markModified().
Although we still have some unsolved issues on why the same set of codes works differently. I have decided not to deal with it for the moment. Here's my solution to the original problem: change the datatype from Date to String. Here's the new set of codes:
Creating User schema at the frontend:
const userSchema = new Schema(
{
profileId: String,
expirationDate: { type: String, default: new Date().toDateString() },
credits: { type: Number, default: 0 },
},
{ timestamps: { createdAt: "created_at" } }
);
Updating user at the backend to MongoDB Atlas::
d = new Date(req.user.expirationDate);
req.user.expirationDate = new Date(
d.setDate(d.getDate() + 30)
).toDateString();
req.user.credits += 1;
const user = await req.user.save();
console.log(typeof req.user.expirationDate);//Checking the datatype of "expirationDate"
Let's take the following example:
I have two mongoDb collections named user and products. A user can add unlimited products, but each product expires in 5 days and it has to be dynamically removed from the db when it expires.
user.js
const UserSchema = mongoose.Schema(
{
username: { type: String },
email: { type: String },
timezone: { type: String },
products: [{ type: Schema.Types.ObjectId, ref: 'Products' }]
}
)
export default mongoose.model('User', UserSchema);
products.js
var RunSchema = mongoose.Schema(
{
name: { type: String },
completedAt: { type: Date }
}
)
export default mongoose.model('Run', RunSchema);
Is there another way of doing this in nodejs rather than having one cron job which runs every day and checks all documents from the products collection?
I'm thinking about a solution where a each user can have a cron job that starts once they add a product...
It is too late answer the question even though , I would like to give alternative operation with minimal operation and without Cron Job
If you want remove product document after creation of 5 day's interval then you can achieve with 2 line of code in model itself.
You have add one property i.e(expireAt) in model and pass the interval to that property with help of moment npm library
var RunSchema = mongoose.Schema(
{
name: { type: String },
completedAt: { type: Date },
expireAt: {
type: Date,
default: moment(new Date()).add("5", "days"),
expires: 432000000
}
}
)
export default mongoose.model('Run', RunSchema);
after creation of product document date and time . it will automatically remove document from collection of every 5 days of interval.
First method:
You can add a cron in NodeJs and it will execute your task every x
Example:
Add the package cron in your project.
npm i cron -S
Create a file cron.js
'use strict';
let express = require('express'),
CronJob = require('cron').CronJob,
moment = require('moment'),
mongoose = require('mongoose');
Collection = mongoose.model('Collection');
var collection = {
clean : function() {
// Each hour
new CronJob('0 * * * *', function() {
console.log("Hello I'm Cron!");
// Delete all the product expired of your db
// Your code ...
}, null, true, 'Europe/Paris');
}
}
module.exports = {
init : function() {
return (
collection.clean()
)
}
}
In your server.js, add the following code to initialize the cron
/**
* Init cron tasks
*/
var cron = require('./cron');
cron.init()
Second method:
Setting expiry time in your model, check that answer for more informations.
I have this error in my ExpressJS app:
Error updating stream: MongoError: Resulting document after update is larger than 16777216
_http_server.js:192
throw new RangeError(`Invalid status code: ${statusCode}`);
So I assume that my document has exceeded the limit.
How can I increase the limit?
I got this link from this answer. But how do I use it in my app? How do I start?
I am using mongoose to store and retrieve my data.
Any ideas/ hints?
This is my document schema in mongoose:
var mongoose = require("mongoose");
var mongoosePaginate = require('mongoose-paginate');
// Declare schema
var streadatamSchema = new mongoose.Schema({
user_id: {
type: String,
required: true
},
title: {
type: String,
required: true
},
description: {
type: String,
required: true
},
data: {
type: Object
},
entries_number: {
type: Number,
default: 0
},
last_entry_at: {
type: Date
},
created_at: {
type: Date,
default: Date.now,
index: 1 // Note 1
},
});
streamSchema.plugin(mongoosePaginate);
// Export schema
// Model.paginate()
mongoose.model("Stream", streamSchema);
I reckon it is the data field has too much data in it now.
So I assume that my document has exceeded the limit.
How can I increase the limit?
The size limit in Mongo is hardcoded in the source code:
/* Note the limit here is rather arbitrary and is simply a standard. generally the code works
with any object that fits in ram.
Also note that the server has some basic checks to enforce this limit but those checks are not exhaustive
for example need to check for size too big after
update $push (append) operation
various db.eval() type operations
*/
const int BSONObjMaxUserSize = 16 * 1024 * 1024;
/*
Sometimes we need objects slightly larger - an object in the replication local.oplog
is slightly larger than a user object for example.
*/
const int BSONObjMaxInternalSize = BSONObjMaxUserSize + ( 16 * 1024 );
const int BufferMaxSize = 64 * 1024 * 1024;
The only way to change it is by changing the source code and building your own version of Mongo from the source.
Im using this scheme for a session in my node.js app
var mongoose = require('mongoose');
var Schema = mongoose.Schema;
// define the schema for our user session model
var UserSessionSchema = new Schema({
sessionActivity: { type: Date, expires: '15s' }, // Expire after 15 s
user_token: { type: String, required: true }
});
module.exports = mongoose.model('UserSession', UserSessionSchema);
And I create a "session" in my app with:
...
var session = new Session();
session.user_token = profile.token;
session.save(function(save_err) {
if (save_err) {
....
} else {
// store session id in profile
profile.session_key = session._id;
profile.save(function(save_err, profile) {
if (save_err) {
...
} else {
res.json({ status: 'OK', session_id: profile.session_id });
}
});
...
The problem is that the document lives permanetly, its never expires. It should only live for 15 seconds (up to a minute). Whats wrong with my code? I have tried to set the expries: string to a number i.e 15, to a string '15s' and so on.
var UserSessionSchema = new Schema({
sessionActivity: { type: Date, expires: '15s', default: Date.now }, // Expire after 15 s
user_token: { type: String, required: true }
});
A TTL index deletes a document 'x' seconds after its value (which should be a Date or an array of Dates) has passed. The TTL is checked every minute, so it may live a little longer than your given 15 seconds.
To give the date a default value, you can use the default option in Mongoose. It accepts a function. In this case, Date() returns the current timestamp. This will set the date to the current time once.
You could also go this route:
UserSessionSchema.pre("save", function(next) {
this.sessionActivity = new Date();
next();
});
This will update the value every time you call .save() (but not .update()).
To double check the indexes that have been created in the DB you can run this command in your mongo shell db.yourdb.getIndexes(). When changing the indexes you have to manually delete it in the collection before the new one will take effect. Check here for more information Mongoose expires property not working properly