I am trying to set a TTL via mongoose when a document is created in MongoDB, but I'm not having any luck with any of my attempts. Latest version of mongoose is being used in my project and from what I can tell I've tried the most common answers here on SO and elsewhere online.
My Schema
const mongoose = require('mongoose');
const jobSchema = new mongoose.Schema(
{
positionTitle: {
type: String,
},
description: {
type: String,
}
});
const Jobs = mongoose.model('job', jobSchema);
module.exports = Jobs;
I have tried adding a createdAt with expires based on this question answer:
const jobSchema = new mongoose.Schema({
positionTitle: {
type: String,
},
description: {
type: String,
},
createdAt: { type: Date, expires: 3600 },
});
Along with this option that's also in the same question to have createdAt be created automatically via timestamps:
const jobSchema = new mongoose.Schema(
{
positionTitle: {
type: String,
},
description: {
type: String,
},
},
{ timestamps: true }
);
Trying variations of the following to set an index with timestamps defined:
jobSchema.index({ createdAt: 1 }, { expires: 86400 });
jobSchema.index({ createdAt: 1 }, { expires: '1 day' });
jobSchema.index({ createdAt: 1 }, { expireAfterSeconds: 3600 });
Regardless of which option I try, the document is removed after MongoDB's 60-second cycle when a createdAt field is set on the document. Would really love to know what I'm doing wrong.
After trying all the solutions in the thread you mentioned, none of them worked. In the end this code did the trick. It involves setting the expireAt field to the actual time that you want it deleted, which makes sense really.
const Schema = mongoose.Schema;
const YourSchema = new Schema({
expireAt: {
type: Date,
default: Date.now() + 10 * 60 * 1000 // expires in 10 minutes
},
});
This is the only thing that worked, all the other solutions I tried always deleted after 1min, no matter the amount of time I added.
I've been having issues with this as well. I found this thread here https://github.com/Automattic/mongoose/issues/2459 and it worked for me. Translated into your code would look like this.
const mongoose = require('mongoose');
const jobSchema = new mongoose.Schema(
{
positionTitle: {
type: String,
},
description: {
type: String,
},
expireAt: {
type: Date,
default: Date.now,
index: { expires: '5s' }
}
});
const Jobs = mongoose.model('job', jobSchema);
module.exports = Jobs;
On the link I added, it is the very last solution. I'm not exactly sure what this is doing but here is the mongo link of what it should be doing for anyone else with this issue. https://docs.mongodb.com/manual/tutorial/expire-data/. To change the amount of time that you need the document just change the expires. It accepts '#s' and '#d' for sure. Also if you want your document to be deleted at a specific time then you can do this.
expireAt: {
type: Date,
default: new Date('July 22, 2013 14:00:00'),
index: { expires: '0s' }
}
This will delete the document 0 seconds after the specified date.
Problem in TTL, Reason behind Document does not delete after some / few seconds, how to expire document in MongoDB / Mongoose using schema. Solution expireAfterSeconds / expires / index.
NOTE: - MongoDB's data expiration task runs once a minute, so an expired doc might persist up to a minute past its expiration. This feature requires MongoDB 2.2 or later. It's up to you to set createdAt to the current time when creating docs or add a default to do it for you as suggested here.
NOTE :- Below code is working fine and the document will delete after 5 minutes.
const verficationSchema = new mongoose.Schema({
email: {
type: String,
required: true,
unique: true,
lowercase: true,
trim: true,
validate(email) {
if (!validator.isEmail(email)) {
throw new Error("Email is not valid!");
}
},
},
otp: {
type: Number,
required : true
},
expireAt : {
type: Date,
default: Date,
expires : 300 // means 300 seconds = 5 minutes
}
});
NOTE :- Upper code is working fine, But document will delete after 1 minutes, because MongoDB check expiration procedure after every 1 minutes.
const verficationSchema = new mongoose.Schema({
email: {
type: String,
required: true,
unique: true,
lowercase: true,
trim: true,
validate(email) {
if (!validator.isEmail(email)) {
throw new Error("Email is not valid!");
}
},
},
otp: {
type: Number,
required : true
},
expireAt : {
type: Date,
default: Date,
expires : 8 // means 8 seconds
}
});
I'm creating an application that serves transportation order, while I'm stress testing the application by injecting about 100 orders simultaneously, the mongodb hits WriteConflict due to the numeric sequence id (nid it is required) that I created using the hook. I have tried to add ?retryWrites=true&w=majority into the connection URI and set those in the transaction level, but none of those work. May I know any parts that I'm missing and should give a try (Knowing that is critical section, and using lock/queue should fix, but want to know any other way)? Thank you very much.
The schema for sequence and order (will keep _id that default created, and mongoose did the thing so haven't mention in the schema)
const SequenceSchema = new Schema<ISequenceDoc>(
{
collectionName: {
type: String,
trim: true,
required: true
},
nid: {
type: Number,
required: true
}
},
{
collection: 'sequences',
timestamps: true
}
);
const OrderSchema = new Schema<IOrderDoc>(
{
name: {
type: String,
trim: true,
required: true
},
nid: {
type: Number
}
},
{
collection: 'orders',
timestamps: true
}
);
Pre saving hook to assign an numeric id index of hex id to the schema
OrderSchema.pre('save', async function(next) {
if (this.isNew) {
const data = await this.db.model<ISequenceDoc>('Sequence').findOneAndUpdate(
{
collectionName: this.collection.collectionName
},
{
$inc: { nid: 1 }
},
{
upsert: true,
new: true,
session: this.$session()
}
);
this.set('nid', data.nid);
}
next();
});
The code that create the order model and save
async createOrder(orderInfo) => {
const session = await this.connection.startSession();
session.startTransaction({
readConcern: {
level: 'majority'
},
writeConcern: {
w: 'majority',
j: true
}
});
......
await order.save({session});
await session.commitTransaction();
return order;
}
The problem is that all of your transactions modify the same document in the findOneAndUpdate.
Try acquiring your sequence number before starting the transaction.
I am attempting to add a form result to an existing client in a collection and all form data variables being passed are added successfully, however, a default date variable is not being created and saved despite being in the schema.
Here is the schema:
const mongoose = require('mongoose');
const Schema = mongoose.Schema;
// Create Schema
const FormSchema = new Schema({
formID: {
type: String
},
formName: {
type: String
},
date_completed: {
type: Date,
default: Date.now
},
formData: {
type: JSON
}
});
const ClientSchema = new Schema({
clientID: {
type: String,
required: true,
unique: true
},
dob: {
type: Date,
required: true
},
formResults: {
tags: [
{
type: FormSchema
}
]
}
});
module.exports = Client = mongoose.model('client', ClientSchema);
And here is the method posting the form results:
router.post('/:id', auth, (req, res) => {
Client.update(
{ clientID: req.params.id },
{
$push: {
formResults: {
$each: [
{
formID: req.body.formID,
formName: req.body.formName,
formData: req.body.formData
}
]
}
}
}
)
.then(() => res.json({ success: true }))
.catch(
err => res.status(404).json({ success: false }) && console.log(err)
);
});
I have tried forcing the date by passing date_completed: Date.now with the other form variables but this makes no difference. The results are still saved with no date variable listed. I have also tried dropping the collection and recreating it, this gave no changes. And I have checked the indexes for the collection, for which there is only _id and clientID.
Here is the data in saved in the database when executed and showing there is no date_completed: value.
Stored Data
At first glance your code is correct and should have no problem as it complies with the documentation and tutorials of mongoose, you can test this code:
// Create Schema
const FormSchema = new Schema({
formID: {
type: String
},
formName: {
type: String
},
date_completed: {
type: Date,
default: function() {
if (!this.date_completed) {
return Date.now();
}
return null;
}
},
formData: {
type: JSON
}
});
or:
var minuteFromNow = function(){
var timeObject = new Date();
return timeObject;
};
// Create Schema
const FormSchema = new Schema({
formID: {
type: String
},
formName: {
type: String
},
date_completed: {
type: Date,
default: minuteFromNow
},
formData: {
type: JSON
}
});
Let us also say this null is a valid value for a Date property, unless you specify required. Defaults only get set if the value is undefined, not if its falsy.
Honestly, I wasn't able to find any reason why it was not working. I waited a few days and the issue fixed itself. I don't believe there was anything wrong with the code, I think the issue was to do with MongoDB Atlas not updating results and displaying the dates that were being created but again I have no idea why this would be the case.
Is there a way to add created_at and updated_at fields to a mongoose schema, without having to pass them in everytime new MyModel() is called?
The created_at field would be a date and only added when a document is created.
The updated_at field would be updated with new date whenever save() is called on a document.
I have tried this in my schema, but the field does not show up unless I explicitly add it:
var ItemSchema = new Schema({
name : { type: String, required: true, trim: true },
created_at : { type: Date, required: true, default: Date.now }
});
UPDATE: (5 years later)
Note: If you decide to use Kappa Architecture (Event Sourcing + CQRS), then you do not need updated date at all. Since your data is an immutable, append-only event log, you only ever need event created date. Similar to the Lambda Architecture, described below. Then your application state is a projection of the event log (derived data). If you receive a subsequent event about existing entity, then you'll use that event's created date as updated date for your entity. This is a commonly used (and commonly misunderstood) practice in miceroservice systems.
UPDATE: (4 years later)
If you use ObjectId as your _id field (which is usually the case), then all you need to do is:
let document = {
updatedAt: new Date(),
}
Check my original answer below on how to get the created timestamp from the _id field.
If you need to use IDs from external system, then check Roman Rhrn Nesterov's answer.
UPDATE: (2.5 years later)
You can now use the #timestamps option with mongoose version >= 4.0.
let ItemSchema = new Schema({
name: { type: String, required: true, trim: true }
},
{
timestamps: true
});
If set timestamps, mongoose assigns createdAt and updatedAt fields to your schema, the type assigned is Date.
You can also specify the timestamp fileds' names:
timestamps: { createdAt: 'created_at', updatedAt: 'updated_at' }
Note: If you are working on a big application with critical data you should reconsider updating your documents. I would advise you to work with immutable, append-only data (lambda architecture). What this means is
that you only ever allow inserts. Updates and deletes should not be
allowed! If you would like to "delete" a record, you could easily
insert a new version of the document with some timestamp/version
filed and then set a deleted field to true. Similarly if you want
to update a document – you create a new one with the appropriate
fields updated and the rest of the fields copied over.Then in order to
query this document you would get the one with the newest timestamp or
the highest version which is not "deleted" (the deleted field is undefined or false`).
Data immutability ensures that your data is debuggable – you can trace
the history of every document. You can also rollback to previous
version of a document if something goes wrong. If you go with such an
architecture ObjectId.getTimestamp() is all you need, and it is not
Mongoose dependent.
ORIGINAL ANSWER:
If you are using ObjectId as your identity field you don't need created_at field. ObjectIds have a method called getTimestamp().
ObjectId("507c7f79bcf86cd7994f6c0e").getTimestamp()
This will return the following output:
ISODate("2012-10-15T21:26:17Z")
More info here How do I extract the created date out of a Mongo ObjectID
In order to add updated_at filed you need to use this:
var ArticleSchema = new Schema({
updated_at: { type: Date }
// rest of the fields go here
});
ArticleSchema.pre('save', function(next) {
this.updated_at = Date.now();
next();
});
As of Mongoose 4.0 you can now set a timestamps option on the Schema to have Mongoose handle this for you:
var thingSchema = new Schema({..}, { timestamps: true });
You can change the name of the fields used like so:
var thingSchema = new Schema({..}, { timestamps: { createdAt: 'created_at' } });
http://mongoosejs.com/docs/guide.html#timestamps
This is what I ended up doing:
var ItemSchema = new Schema({
name : { type: String, required: true, trim: true }
, created_at : { type: Date }
, updated_at : { type: Date }
});
ItemSchema.pre('save', function(next){
now = new Date();
this.updated_at = now;
if ( !this.created_at ) {
this.created_at = now;
}
next();
});
Use the built-in timestamps option for your Schema.
var ItemSchema = new Schema({
name: { type: String, required: true, trim: true }
},
{
timestamps: true
});
This will automatically add createdAt and updatedAt fields to your schema.
http://mongoosejs.com/docs/guide.html#timestamps
Add timestamps to your Schema like this then createdAt and updatedAt will automatic generate for you
var UserSchema = new Schema({
email: String,
views: { type: Number, default: 0 },
status: Boolean
}, { timestamps: {} });
Also you can change createdAt -> created_at by
timestamps: { createdAt: 'created_at', updatedAt: 'updated_at' }
If use update() or findOneAndUpdate()
with {upsert: true} option
you can use $setOnInsert
var update = {
updatedAt: new Date(),
$setOnInsert: {
createdAt: new Date()
}
};
For NestJs with Mongoose, use this
#Schema({timestamps: true})
In your model :
const User = Schema(
{
firstName: { type: String, required: true },
lastName: { type: String, required: true },
password: { type: String, required: true }
},
{
timestamps: true
}
);
And after that your model in collection would be like this :
{
"_id" : ObjectId("5fca632621100c230ce1fb4b"),
"firstName" : "first",
"lastName" : "last",
"password" : "$2a$15$Btns/B28lYIlSIcgEKl9eOjxOnRjJdTaU6U2vP8jrn3DOAyvT.6xm",
"createdAt" : ISODate("2020-12-04T16:26:14.585Z"),
"updatedAt" : ISODate("2020-12-04T16:26:14.585Z"),
}
This is how I achieved having created and updated.
Inside my schema I added the created and updated like so:
/**
* Article Schema
*/
var ArticleSchema = new Schema({
created: {
type: Date,
default: Date.now
},
updated: {
type: Date,
default: Date.now
},
title: {
type: String,
default: '',
trim: true,
required: 'Title cannot be blank'
},
content: {
type: String,
default: '',
trim: true
},
user: {
type: Schema.ObjectId,
ref: 'User'
}
});
Then in my article update method inside the article controller I added:
/**
* Update a article
*/
exports.update = function(req, res) {
var article = req.article;
article = _.extend(article, req.body);
article.set("updated", Date.now());
article.save(function(err) {
if (err) {
return res.status(400).send({
message: errorHandler.getErrorMessage(err)
});
} else {
res.json(article);
}
});
};
The bold sections are the parts of interest.
In your model schema, just add an attribute timestamps and assign value true to it as shown:-
var ItemSchema = new Schema({
name : { type: String, required: true, trim: true },
},{timestamps : true}
);
var ItemSchema = new Schema({
name : { type: String, required: true, trim: true }
});
ItemSchema.set('timestamps', true); // this will add createdAt and updatedAt timestamps
Docs: https://mongoosejs.com/docs/guide.html#timestamps
You can use the timestamp plugin of mongoose-troop to add this behavior to any schema.
You can use this plugin very easily.
From the docs:
var timestamps = require('mongoose-timestamp');
var UserSchema = new Schema({
username: String
});
UserSchema.plugin(timestamps);
mongoose.model('User', UserSchema);
var User = mongoose.model('User', UserSchema)
And also set the name of the fields if you wish:
mongoose.plugin(timestamps, {
createdAt: 'created_at',
updatedAt: 'updated_at'
});
we may can achieve this by using schema plugin also.
In helpers/schemaPlugin.js file
module.exports = function(schema) {
var updateDate = function(next){
var self = this;
self.updated_at = new Date();
if ( !self.created_at ) {
self.created_at = now;
}
next()
};
// update date for bellow 4 methods
schema.pre('save', updateDate)
.pre('update', updateDate)
.pre('findOneAndUpdate', updateDate)
.pre('findByIdAndUpdate', updateDate);
};
and in models/ItemSchema.js file:
var mongoose = require('mongoose'),
Schema = mongoose.Schema,
SchemaPlugin = require('../helpers/schemaPlugin');
var ItemSchema = new Schema({
name : { type: String, required: true, trim: true },
created_at : { type: Date },
updated_at : { type: Date }
});
ItemSchema.plugin(SchemaPlugin);
module.exports = mongoose.model('Item', ItemSchema);
if you'r using nestjs and #Schema decorator you can achieve this like:
#Schema({
timestamps: true,
})
The timestamps option tells mongoose to assign createdAt and updatedAt fields to your schema. The type assigned is Date.
By default, the names of the fields are createdAt and updatedAt.
Customize the field names by setting timestamps.createdAt and timestamps.updatedAt.
My mongoose version is 4.10.2
Seems only the hook findOneAndUpdate is work
ModelSchema.pre('findOneAndUpdate', function(next) {
// console.log('pre findOneAndUpdate ....')
this.update({},{ $set: { updatedAt: new Date() } });
next()
})
const mongoose = require('mongoose');
const config = require('config');
const util = require('util');
const Schema = mongoose.Schema;
const BaseSchema = function(obj, options) {
if (typeof(options) == 'undefined') {
options = {};
}
if (typeof(options['timestamps']) == 'undefined') {
options['timestamps'] = true;
}
Schema.apply(this, [obj, options]);
};
util.inherits(BaseSchema, Schema);
var testSchema = new BaseSchema({
jsonObject: { type: Object }
, stringVar : { type: String }
});
Now you can use this, so that there is no need to include this option in every table
Since mongo 3.6 you can use 'change stream':
https://emptysqua.re/blog/driver-features-for-mongodb-3-6/#change-streams
To use it you need to create a change stream object by the 'watch' query, and for each change, you can do whatever you want...
python solution:
def update_at_by(change):
update_fields = change["updateDescription"]["updatedFields"].keys()
print("update_fields: {}".format(update_fields))
collection = change["ns"]["coll"]
db = change["ns"]["db"]
key = change["documentKey"]
if len(update_fields) == 1 and "update_at" in update_fields:
pass # to avoid recursion updates...
else:
client[db][collection].update(key, {"$set": {"update_at": datetime.now()}})
client = MongoClient("172.17.0.2")
db = client["Data"]
change_stream = db.watch()
for change in change_stream:
print(change)
update_ts_by(change)
Note, to use the change_stream object, your mongodb instance should run as 'replica set'.
It can be done also as a 1-node replica set (almost no change then the standalone use):
Run mongo as a replica set:
https://docs.mongodb.com/manual/tutorial/convert-standalone-to-replica-set/
Replica set configuration vs Standalone:
Mongo DB - difference between standalone & 1-node replica set
I actually do this in the back
If all goes well with the updating:
// All ifs passed successfully. Moving on the Model.save
Model.lastUpdated = Date.now(); // <------ Now!
Model.save(function (err, result) {
if (err) {
return res.status(500).json({
title: 'An error occured',
error: err
});
}
res.status(200).json({
message: 'Model Updated',
obj: result
});
});
Use a function to return the computed default value:
var ItemSchema = new Schema({
name: {
type: String,
required: true,
trim: true
},
created_at: {
type: Date,
default: function(){
return Date.now();
}
},
updated_at: {
type: Date,
default: function(){
return Date.now();
}
}
});
ItemSchema.pre('save', function(done) {
this.updated_at = Date.now();
done();
});
Use machinepack-datetime to format the datetime.
tutorialSchema.virtual('createdOn').get(function () {
const DateTime = require('machinepack-datetime');
let timeAgoString = "";
try {
timeAgoString = DateTime.timeFrom({
toWhen: DateTime.parse({
datetime: this.createdAt
}).execSync(),
fromWhen: new Date().getTime()
}).execSync();
} catch(err) {
console.log('error getting createdon', err);
}
return timeAgoString; // a second ago
});
Machine pack is great with clear API unlike express or general Javascript world.
You can use middleware and virtuals. Here is an example for your updated_at field:
ItemSchema.virtual('name').set(function (name) {
this.updated_at = Date.now;
return name;
});
I have the following message schema in mongoose:
var messageSchema = mongoose.Schema({
userID: { type: ObjectId, required: true, ref: 'User' },
text: { type: String, required: true }
},
{
timestamps: true
});
Is there anyway to ignore the updatedAt timestamp? Messages won't be updated so updatedAt will be wasted space
Maybe even better with Mongoose v5 is to do the following;
const schema = new Schema({
// Your schema...
}, {
timestamps: { createdAt: true, updatedAt: false }
})
Edit I've amended the answer to reflect the better option to use the default as per #JohnnyHK
You can handle this yourself by declaring the createdAt (or whatever you want to call it) in your schema:
mongoose.Schema({
created: { type: Date, default: Date.now }
...
Alternatively we can also update values on new document in a pre save hook:
messageSchema.pre('save', function (next) {
if (!this.created) this.created = new Date;
next();
})
Along those lines is also the flag isNew which you can use to check if a document is new.
messageSchema.pre('save', function (next) {
if (this.isNew) this.created = new Date;
next();
})
Older topic but there may be a better option depending on your schema...
If you're sticking with the default of having mongodb/mongoose auto-gen _id, there's already a timestamp built in. If all you need is "created" and not "updated" just use...
document._id.getTimestamp();
From MongoDB docs here...
ObjectId.getTimestamp()
And Here... stackoverflow
Mongoose timestamp interface has these optional fields.
interface SchemaTimestampsConfig {
createdAt?: boolean | string;
updatedAt?: boolean | string;
currentTime?: () => (Date | number);
}
We can pass the boolean for the field we want(createdAt: true and updatedAt: true will add both fields).
We can use the currentTime function to overwrite the date format.
example:
import mongoose from 'mongoose';
const { Schema } = mongoose;
const annotationType = ['NOTES', 'COMMENTS'];
const referenceType = ['TASKS', 'NOTES'];
const AnnotationSchema = new Schema(
{
sellerOrgId: {
type: String,
required: true,
},
createdById: {
type: String,
required: true,
},
annotationType: {
type: String,
enum: annotationType,
},
reference: {
id: { type: String, index: true },
type: {
type: String,
enum: referenceType,
},
},
data: Schema.Types.Mixed,
},
{ timestamps: { createdAt: true },
);
const AnnotationModel = mongoose.models.annotation || mongoose.model('annotation', AnnotationSchema);
export { AnnotationModel, AnnotationSchema };