I have a MERN app which uses mongoose to connect to MongoDB Atlas. While the average response time from Atlas is <300ms, every once a while this becomes >30 seconds and everything becomes unusable. I am wondering if something might be wrong with the way I handle connections to the database?
Currently in my app.js file:
mongoose.connect(`mongodb+srv://<db name>:${process.env.DB_PASSWORD}#<...>.kea2z.mongodb.net/<...>?retryWrites=true&w=majority`, {useNewUrlParser: true, useUnifiedTopology: true})
In my routers.js file, I handle routes like the following:
import { Post } from './models.js'
...
const postRouter = express.Router()
postRouter.get('/', async (req, res) => {
try {
const posts = await Post.find()
return res.json(posts)
} catch(e) {
console.log(`Error while indexing posts: ${e}`)
return res.status(404).json('An error has occurred!')
}
})
...
For instance, the above Post collection has 50 documents and 2MB total size, but the simple Post.find() query takes longer than 30 seconds to complete. I have four other collections similar to this; including a collection of images which has a size of 65MB. Is there a problem in the way I am querying the database?
UPDATE 1:
I have moved all the images to the cloud so now my database only stores their URLs. However it still takes ~15s for the Post collection to be queried, which has a size of 1.3MB and contains 50 documents. In case a faulty schema definition may be causing it, here is its model:
const Post = mongoose.model('Post', new mongoose.Schema({
slug: String,
title: String,
authors: [String],
date: Date,
categories: [String],
content: String
}))
It's not a good practice to store images in a mongoDB database.
A better approach is to store the images in some storage (such as AWS S3) and save the image URLs in the database as a string.
This Query may be faster
await Post.find().lean();
NOTICE if you use lean(), it is faster because
you get pure json of document
but you cannot modify document like
posts[0].name = "jack";
await posts[0].save()
Related
I am trying to delete a document in my mongoDB collection using TTL feature of it, and it works as expected, but not fully. It deletes the document later than the specified time.
I specified delete time 10seconds, but sometimes it takes 20seconds to delete it, sometimes 50seconds. Maybe I am making some mistake. I have used UTC date format, and tried with my local area date format too, but still the same. How do I resolve this problem?
One more thing that I want to ask, lets say somehow it works, and I have thousands of documents in my collection, will it decrease the performance of my database for requests and response handling??
because I am not deleting the whole collection, but individual documents, so keeping track of them might decrease performance, am I right?
Here is what I tried.
this is my schema
const mongoose = require('mongoose');
const tokens = mongoose.Schema({
token: String,
createdAt:{
type: Date,
expires: 10,
}
});
module.exports = {
authTokens: mongoose.model('Authtoken', tokens)
}
THIS HOW I AM CREATING DOCUMENT IN COLLECTION
app.post('/createToken', async (req, res) => {
try{
//create document in authTokens Collection
await authTokens.create({
token: req.body.token,
createdAt: new Date().toUTCString() //getting utc value of date
});
}
catch (err) {
res.send("Could not send token!");
console.error(err);
return;
}
res.send("Document Created Successfully");
return;
});
Can anyone help please
Thanks
My collection has 1000-ish records. I have a flutter app, which fetches data from a Heroku API, where the backend is based in NodeJs. While fetching, it sorts the entire collection based on the number of a certain "Vacant" field in descending order. It takes nearly 15 seconds to fetch that data. I really don't think 1000 documents is a lot of data, so what can be the optimal method of approaching this problem?
UPDATE 1: This is the code where I'm fetching the data, where I'm sorting based on the 'Vacant' field.
UPDATE 2: My region on Heroku was set in the US, that's why there was a huge delay in the response time. Shifted to AWS with a server close to me, the response is in milliseconds now.
const dataSchema = new mongoose.Schema({
District: String,
Name: String,
Vacant: Number,
Address: String,
PhoneNumber: String
})
app.get('/:state', (req, res) => {
const State = mongoose.model(req.params.state, dataSchema)
State.find().sort([['Vacant', -1]]).exec((err, foundData) => {
if (err) {
console.log(err)
} else {
res.send(foundData)
}
})
})
I am still kind of new to Mongoose in general. I am building my blogging app which has backend based on Node and MongoDB, I am using Angular for frontend.
I am creating my Restful API which is supposed to allow user click on a post and update it. However, I don't know for sure whether I am doing it the right way here.
This is the schema for my post:
const mongoose = require('mongoose');
// Schema Is ONly bluePrint
var postSchema = mongoose.Schema({
title: {type: String, required: true },
content: {type: String, required: true},
}, {timestamps: true});
module.exports = mongoose.model("Post", postSchema);
In my angular service, I have this function to help me to send the http request to my backend server, the id for this function comes from backend mongoDB, title and content is from the form on the page
updatePost(id: string, title: string, content: string) {
console.log('start posts.service->updatePost()');
const post: Post = {
id: id,
title: title,
content: content
};
this._http.put(`http://localhost:3000/api/posts/${id}`, post)
.subscribe(res => console.log(res));
}
It appears to me that there are at least couple of ways of approaching this for creating my API
Method 1 ( works but highly doubt if this is good practice):
here I am passing the id retrieved from mongoDB back to server via my service.ts file to avoid the 'modifying immutable field _id' error
app.put("/api/posts/:id", (req,res)=>{
console.log('update api called:', req.params.id);
const post = new Post({
id: req.body.id,
title: req.body.title,
content: req.body.content
});
Post.updateOne({_id: req.params.id}, post).then( result=> {
console.log(result);
res.json({message:"Update successful!"});
});
});
Method 2 I consider this is more robust than method 1 but still I don't think its good practice:
app.put("/api/posts/:id", (req, res)=> {
Post.findOne(
{_id:req.params.id},(err,post)=>{
if(err){
console.log('Post Not found!');
res.json({message:"Error",error:err});
}else{
console.log('Found post:',post);
post.title=req.body.title;
post.content=req.body.content;
post.save((err,p)=>{
if(err){
console.log('Save from update failed!');
res.json({message:"Error",error:err});
}else{
res.json({message:"update success",data:p});
}
})
}
}
);
});
I am open to all opinions in the hope that I can learn something from guru of Mongoose and Restful : )
Justification to Choose findOneAndUpdate() in this scenario in simple words are as follow:
You can use findOneAndUpdate() as it updates document based on the
filter and sort criteria.
While working with mongoose mostly we prefer to use this function as compare to update() as it has a an option {new:
true} and with the help of that we can get updated data.
As your purpose here is to updating a single document so you can use findOneAndUpdate(). On the other hand update() should be
used in case of bulk modification.
As update() Always returns on of document modified it won't return updated documents and while working with such a scenario like
your we always returns updated document data in response so we
should use findOneAndUpdate() here
I am developing an application where I am using MongoDB as database with Nodejs + Express in application layer, I have two collections, namely
users
transactions
Here i have to update wallet of thousands of users with some amount and if successful create a new document with related info for each transaction, This is My code :
userModel.update({_id : ObjectId(userId)}, {$inc : {wallet : 500}}, function (err, creditInfo) {
if(err){
console.log(err);
}
if(creditInfo.nModified > 0) {
newTransModel = new transModel({
usersId: ObjectId(userId),
amount: winAmt,
type: 'credit',
});
newTransModel.save(function (err, doc) {
if(err){
Cb(err);
}
});
}
});
but this solution is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss.
I have heard that recently MongoDB has added Transactions support in its 4.0 version, I have read the MongoDB docs but couldn't get it to successfully implement it with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature of MongoDB which have these functions
Session.startTransaction()
Session.abortTransaction()
Session.commitTransaction()
MongoDB Docs : Click Here
with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature
To use MongoDB multi-documents transactions support in mongoose you need version greater than v5.2. For example:
npm install mongoose#5.2
Mongoose transactional methods returns a promise rather than a session which would require to use await. See:
Transactions in Mongoose
Blog: A Node.JS Perspective on MongoDB 4.0: Transactions
For example, altering the example on the resource above and your example, you can try:
const User = mongoose.model('Users', new mongoose.Schema({
userId: String, wallet: Number
}));
const Transaction = mongoose.model('Transactions', new mongoose.Schema({
userId: ObjectId, amount: Number, type: String
}));
await updateWallet(userId, 500);
async function updateWallet(userId, amount) {
const session = await User.startSession();
session.startTransaction();
try {
const opts = { session };
const A = await User.findOneAndUpdate(
{ _id: userId }, { $inc: { wallet: amount } }, opts);
const B = await Transaction(
{ usersId: userId, amount: amount, type: "credit" })
.save(opts);
await session.commitTransaction();
session.endSession();
return true;
} catch (error) {
// If an error occurred, abort the whole transaction and
// undo any changes that might have happened
await session.abortTransaction();
session.endSession();
throw error;
}
}
is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss
You should also consider changing your MongoDB data models. Especially if the two collections are naturally linked. See also Model data for Atomic Operations for more information.
An example model that you could try is Event Sourcing model. Create a transaction entry first as an event, then recalculate the user's wallet balance using aggregation.
For example:
{tranId: 1001, fromUser:800, toUser:99, amount:300, time: Date(..)}
{tranId: 1002, fromUser:77, toUser:99, amount:100, time: Date(..)}
Then introduce a process to calculate the amount for each users per period as a cache depending on requirements (i.e. per 6 hours). You can display the current user's wallet balance by adding:
The last cached amount for the user
Any transactions for the user occur since the last cached amount. i.e. 0-6 hours ago.
If I add a document to the DB (either via my app, or via the DB admin tool RockMongo), I immediately see the documents being added to the DB (as shown by RockMongo), no problem. However, when I call a simple model.find() on the corresponding Mongoose model, the latest additions to the DB are not returned. They eventually show up after a couple of minutes.
It looks like the documents are being read from some kind of cache/buffer which isn't kept updated. Is there anything like that in Mongoose which I'm overlooking?
My backend is something like this:
var mongoose = require('mongoose');
require('./models/Locations');
mongoose.Promise = global.Promise;
mongoose.connect(mongoUrl,{ config: { autoIndex: false } });
var Locations = mongoose.model('Locations');
[...]
app.get('/locations', function(req, res, next)
{
Locations.find(function(err, results)
{
if(err){return res.status(500).json(err);}
res.json(results);
});
});
The model:
var mongoose = require('mongoose');
var LocationsSchema = new mongoose.Schema({
name: String
});
mongoose.model('Locations', LocationsSchema);
If I add an item to Locations manually in the DB, and point my browser to /locations, I don't see the new item until a few minutes later...
Change this:
Locations.find(function(err, results)
{
if(err){return res.status(500).json(err);}
res.json(results);
});
To this:
Locations.find({},function(err, results)
{
if(err){return res.status(500).json(err);}
res.json(results);
});
It's been a long time, but I noticed I never answered this question, so I thought I might in case someone else runs into this. It is very specific to my setup though. The issue was that Gandi uses a caching engine (Varnish) with relatively aggressive settings by default, leading to the slow update I was seeing in my webapp.
To solve this, one can explicitly set a header to deactivate the caching, for instance:
res.setHeader('Cache-Control', 'max-age=0');