MongoDB - MongoError: Resulting document after update is larger than 16777216 - node.js

I have this error in my ExpressJS app:
Error updating stream: MongoError: Resulting document after update is larger than 16777216
_http_server.js:192
throw new RangeError(`Invalid status code: ${statusCode}`);
So I assume that my document has exceeded the limit.
How can I increase the limit?
I got this link from this answer. But how do I use it in my app? How do I start?
I am using mongoose to store and retrieve my data.
Any ideas/ hints?
This is my document schema in mongoose:
var mongoose = require("mongoose");
var mongoosePaginate = require('mongoose-paginate');
// Declare schema
var streadatamSchema = new mongoose.Schema({
user_id: {
type: String,
required: true
},
title: {
type: String,
required: true
},
description: {
type: String,
required: true
},
data: {
type: Object
},
entries_number: {
type: Number,
default: 0
},
last_entry_at: {
type: Date
},
created_at: {
type: Date,
default: Date.now,
index: 1 // Note 1
},
});
streamSchema.plugin(mongoosePaginate);
// Export schema
// Model.paginate()
mongoose.model("Stream", streamSchema);
I reckon it is the data field has too much data in it now.

So I assume that my document has exceeded the limit.
How can I increase the limit?
The size limit in Mongo is hardcoded in the source code:
/* Note the limit here is rather arbitrary and is simply a standard. generally the code works
with any object that fits in ram.
Also note that the server has some basic checks to enforce this limit but those checks are not exhaustive
for example need to check for size too big after
update $push (append) operation
various db.eval() type operations
*/
const int BSONObjMaxUserSize = 16 * 1024 * 1024;
/*
Sometimes we need objects slightly larger - an object in the replication local.oplog
is slightly larger than a user object for example.
*/
const int BSONObjMaxInternalSize = BSONObjMaxUserSize + ( 16 * 1024 );
const int BufferMaxSize = 64 * 1024 * 1024;
The only way to change it is by changing the source code and building your own version of Mongo from the source.

Related

Making Mongodb documents inactive after certain time

Let's say I am creating a web application that posts event markers to a map to help flash mobs find out where to meet up. In this hypothetical app and due to the nature of flash mobs, these markers would have to become inactive after a certain time interval so as to not pollute the map with events that are no longer occurring. My question is what is the appropriate way to allow this to happen in mongoDB using node.js without deleting the document?
If I have a mongoose model (Marker) like this:
const mongoose = require('mongoose');
const markerSchema = new mongoose.Schema({
name: {
type: String,
},
description: {
type: String,
trim: true,
},
createdAt: {
type: Date,
default: Date.now(),
select: false,
},
expiresAt: {
type: Date
},
active: Boolean,
location: {
type: {
type: String,
enum: ['Point'],
required: true,
},
coordinates: {
type: [Number],
required: true,
},
},
});
markerSchema.pre('save', function (next) {
this.expiresAt = this.createdAt + 60 * 60 * 1000 * 6;
next();
});
const Marker = mongoose.model('Marker', tourSchema);
module.exports = Marker;
What would be the best way to update the markers active field to false after an arbitrary amount of time, say 6 hours?
I have researched Mongodb's TTL and I have also considered using the createdAt field to query for markers that are within the time frame of active using Marker.find({ expiresAt: {$gt: Date.now() }) but I do not know if this would be the best. Is there a way to run a function or middleware that periodically checks each document and sets the active field to false if the time frame is up? Just curious how you would approach this problem so querying Marker.find({ active: true }) would provide the data I am looking for.
Setting TTL in MongoDB runs a scheduled task every 60 seconds and will remove the document once expired. If you don't want to remove the document and mark it instead.
Run a CRON/Scheduled task every minute
Update the document matching your custom TTL criteria
Use the criteria in all select clause to ignore expired documents.
You can use node-cron to achieve your goal.
For example, rub a job every minute :
cron.schedule('* * * * *', function() {
console.log('running a task every minute');
});
More details and full usage example tutorials :
https://www.digitalocean.com/community/tutorials/nodejs-cron-jobs-by-examples
https://blog.logrocket.com/task-scheduling-or-cron-jobs-in-node-using-node-cron/

Why wouldn’t mongoDB update the “Date” field of my user entry?

I created a User schema in my React App as follows:
const userSchema = new Schema(
{
profileId: String,
expirationDate: { type: Date, default: new Date() },
credits: { type: Number, default: 0 },
},
{ timestamps: { createdAt: "created_at" } }
);
When the user pays me, I want to reset/update two fields: the expirationDate and credits via a post method. Here’s the code I use on my expressjs backend server to update the database entry on MongoDB Atlas:
req.user.expirationDate = new Date(
req.user.expirationDate.setDate(req.user.expirationDate.getDate() + 30)
);
req.user.credits += 1;
const user = await req.user.save();
res.send(user);
Once the operation succeeded, I can see the field of “credits” gets updated (increased by 1). However, the “expirationDate” field remains unchanged. What’s more curious is that when I send the updated user object to my frontend server with “res.send(user)”, I can see the updated expirationDate in the console.
Successfully updated user model as designed/intended: seen from frontend console
But below is what I saw in my mongoDB:
Updated user entry in MongoDB: the Date field "expirationDate" is not updated; but, the "credits" field is.
What is going on here? How to fix it?
I was having a similar issue recently and haven't figured out the actual reason behind this, but as a workaround try telling mongoose explicitly that the expirationDate-field was changed:
req.user.expirationDate = new Date(
req.user.expirationDate.setDate(req.user.expirationDate.getDate() + 30)
);
req.user.markModified('expirationDate');
await req.user.save();
EDIT:
Just debugged it again and I think the reason behind this behaviour is your default value for expirationDate. Try passing it the Date.now function instead of immediately setting it to a new date:
expirationDate: {type: Date, default: Date.now},
This fixed it for me without having to use markModified().
Although we still have some unsolved issues on why the same set of codes works differently. I have decided not to deal with it for the moment. Here's my solution to the original problem: change the datatype from Date to String. Here's the new set of codes:
Creating User schema at the frontend:
const userSchema = new Schema(
{
profileId: String,
expirationDate: { type: String, default: new Date().toDateString() },
credits: { type: Number, default: 0 },
},
{ timestamps: { createdAt: "created_at" } }
);
Updating user at the backend to MongoDB Atlas::
d = new Date(req.user.expirationDate);
req.user.expirationDate = new Date(
d.setDate(d.getDate() + 30)
).toDateString();
req.user.credits += 1;
const user = await req.user.save();
console.log(typeof req.user.expirationDate);//Checking the datatype of "expirationDate"

Access other fields in getter

I'm trying to make it that every time I get subtotal, it adds 2 other fields, productTotal, and tax.
Here's my code:
const cost = new mongoose.Schema({
payment: {
productTotal: Number,
tax: Number,
subtotal: Number, // (productTotal + tax)
}
});
const Cost = mongoose.model('Cost', cost);
How can I add 2 feilds from the same schema when getting a different field?
You can achieve that in mongoose by creating a virtual field for subtotal. The mongoose reference documents this pretty well here: https://mongoosejs.com/docs/tutorials/virtuals.html.
EDIT:
The code snippet below shows how you can define a cost schema that has a subtotal virtual on the payment subdocument:
const PaymentSchema = new Schema({
productTotal: { type: Number, default: 0 },
tax: { type: Number, default: 0 },
});
PaymentSchema.virtual('subtotal').get(function () { return this.productTotal + this.tax; });
const CostSchema = new Schema({
payment: PaymentSchema,
});
From the above snippet you can get the subtotal from a Cost document instance via cost.payment.subtotal.

How to make Mongoose Schema id generate a 16 digits random number

I'm new to mongoose. I'm creating a model (creditnote) that consists of a 16 digits unique reference (serves as primary key) and an integer that represents the number of credits.
I want the reference to be the model's id and to be an auto-generated 16 digits number. How can I achieve this?
Right now I have the following code:
var mongoose = require('mongoose');
// Setup schema
var creditnoteSchema = mongoose.Schema({
reference: {
type: mongoose.ObjectId,
required: true,
default: function(){
// Generate 16 digits random number
number = (Math.random()+' ').substring(2,10)+(Math.random()+' ').substring(2,10);
}
},
amount: {
type: Number,
required: true,
validate: {
validator: Number.isInteger,
message: '{VALUE} is not an integer value'
}
}
});
All objects created in MongoDB have a property _id which serves as a unique ID. You can also use ObjectId to generate a unique key, like this:
const {ObjectId} = require('mongodb');
console.log(ObjectId());

Fetch linked documents with Mongoose

I have a mongoose Schema that roughly looks like this:
var messageSchema = new Schema({
answerTo: {
type: Schema.ObjectId,
ref: 'Message'
},
content: {
type: String,
required: true
}
})
Note: Sender and receiver are left out for the sake of simplicity.
I would now like to fetch let's say the 10 most recent messages from a conversation. The previous message is linked to in the property answerTo — is there any better way than fetching the documents one by one, looping over it until eventually either the limit is reached or answerTo === null?

Resources