I am trying to delete a document in my mongoDB collection using TTL feature of it, and it works as expected, but not fully. It deletes the document later than the specified time.
I specified delete time 10seconds, but sometimes it takes 20seconds to delete it, sometimes 50seconds. Maybe I am making some mistake. I have used UTC date format, and tried with my local area date format too, but still the same. How do I resolve this problem?
One more thing that I want to ask, lets say somehow it works, and I have thousands of documents in my collection, will it decrease the performance of my database for requests and response handling??
because I am not deleting the whole collection, but individual documents, so keeping track of them might decrease performance, am I right?
Here is what I tried.
this is my schema
const mongoose = require('mongoose');
const tokens = mongoose.Schema({
token: String,
createdAt:{
type: Date,
expires: 10,
}
});
module.exports = {
authTokens: mongoose.model('Authtoken', tokens)
}
THIS HOW I AM CREATING DOCUMENT IN COLLECTION
app.post('/createToken', async (req, res) => {
try{
//create document in authTokens Collection
await authTokens.create({
token: req.body.token,
createdAt: new Date().toUTCString() //getting utc value of date
});
}
catch (err) {
res.send("Could not send token!");
console.error(err);
return;
}
res.send("Document Created Successfully");
return;
});
Can anyone help please
Thanks
Related
I have a MERN app which uses mongoose to connect to MongoDB Atlas. While the average response time from Atlas is <300ms, every once a while this becomes >30 seconds and everything becomes unusable. I am wondering if something might be wrong with the way I handle connections to the database?
Currently in my app.js file:
mongoose.connect(`mongodb+srv://<db name>:${process.env.DB_PASSWORD}#<...>.kea2z.mongodb.net/<...>?retryWrites=true&w=majority`, {useNewUrlParser: true, useUnifiedTopology: true})
In my routers.js file, I handle routes like the following:
import { Post } from './models.js'
...
const postRouter = express.Router()
postRouter.get('/', async (req, res) => {
try {
const posts = await Post.find()
return res.json(posts)
} catch(e) {
console.log(`Error while indexing posts: ${e}`)
return res.status(404).json('An error has occurred!')
}
})
...
For instance, the above Post collection has 50 documents and 2MB total size, but the simple Post.find() query takes longer than 30 seconds to complete. I have four other collections similar to this; including a collection of images which has a size of 65MB. Is there a problem in the way I am querying the database?
UPDATE 1:
I have moved all the images to the cloud so now my database only stores their URLs. However it still takes ~15s for the Post collection to be queried, which has a size of 1.3MB and contains 50 documents. In case a faulty schema definition may be causing it, here is its model:
const Post = mongoose.model('Post', new mongoose.Schema({
slug: String,
title: String,
authors: [String],
date: Date,
categories: [String],
content: String
}))
It's not a good practice to store images in a mongoDB database.
A better approach is to store the images in some storage (such as AWS S3) and save the image URLs in the database as a string.
This Query may be faster
await Post.find().lean();
NOTICE if you use lean(), it is faster because
you get pure json of document
but you cannot modify document like
posts[0].name = "jack";
await posts[0].save()
My collection has 1000-ish records. I have a flutter app, which fetches data from a Heroku API, where the backend is based in NodeJs. While fetching, it sorts the entire collection based on the number of a certain "Vacant" field in descending order. It takes nearly 15 seconds to fetch that data. I really don't think 1000 documents is a lot of data, so what can be the optimal method of approaching this problem?
UPDATE 1: This is the code where I'm fetching the data, where I'm sorting based on the 'Vacant' field.
UPDATE 2: My region on Heroku was set in the US, that's why there was a huge delay in the response time. Shifted to AWS with a server close to me, the response is in milliseconds now.
const dataSchema = new mongoose.Schema({
District: String,
Name: String,
Vacant: Number,
Address: String,
PhoneNumber: String
})
app.get('/:state', (req, res) => {
const State = mongoose.model(req.params.state, dataSchema)
State.find().sort([['Vacant', -1]]).exec((err, foundData) => {
if (err) {
console.log(err)
} else {
res.send(foundData)
}
})
})
I have a sever connected to a mongodb database. When I add a first level data and then save that, it works.
For example :
// this works fine
router.post('/user/addsomedata', async (req,res)=>{
try {
const user = await User.findOne({email : req.body.email})
user.username = req.body.username
await user.save()
res.send()
} catch(e) {
res.status(404).send(e)
}
})
BUT if I try to save the object with deeper level data, it's not getting saved. I guess the update is not detected and hence the user didn't get replaced.
Example :
router.post('/user/addtask', auth ,async (req,res)=>{
const task = new Task({
name : req.body.name,
timing : new Date(),
state : false,
})
try {
const day = await req.user.days.find((day)=> day.day == req.body.day)
// day is found with no problem
req.user.days[req.user.days.indexOf(day)].tasks.push(task)
// console.log(req.user) returns exactly the expected results
await req.user.save(function(error,res){
console.log(res)
// console.log(res) returns exactly the expected results with the data filled
// and the tasks array is populated
// but on the database there is nothing
})
res.status(201).send(req.user)
} catch(e) {
res.status(400).send(e)
}
})
So I get the tasks array populated on the console even after the save callback but nothing on the db image showing empty tasks array
You're working on the user from the request, while you should first find the user from the DB like in your first example (User.findOne) and then update and save that model.
Use .lean() with your find queries whenever you are about to update the results returned by mongoose. Mongoose by default return instance objects which are immutable by nature. lean() method with find returns normal js objects which can be modified/updated.
eg. of using lean()
const user = await User.findOne({email : req.body.email}).lean();
You can read more about lean here
Hope this helps :)
I am developing an application where I am using MongoDB as database with Nodejs + Express in application layer, I have two collections, namely
users
transactions
Here i have to update wallet of thousands of users with some amount and if successful create a new document with related info for each transaction, This is My code :
userModel.update({_id : ObjectId(userId)}, {$inc : {wallet : 500}}, function (err, creditInfo) {
if(err){
console.log(err);
}
if(creditInfo.nModified > 0) {
newTransModel = new transModel({
usersId: ObjectId(userId),
amount: winAmt,
type: 'credit',
});
newTransModel.save(function (err, doc) {
if(err){
Cb(err);
}
});
}
});
but this solution is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss.
I have heard that recently MongoDB has added Transactions support in its 4.0 version, I have read the MongoDB docs but couldn't get it to successfully implement it with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature of MongoDB which have these functions
Session.startTransaction()
Session.abortTransaction()
Session.commitTransaction()
MongoDB Docs : Click Here
with mongoose in Node.js, can anyone tell me how this above code be reimplemented using the latest Transactions feature
To use MongoDB multi-documents transactions support in mongoose you need version greater than v5.2. For example:
npm install mongoose#5.2
Mongoose transactional methods returns a promise rather than a session which would require to use await. See:
Transactions in Mongoose
Blog: A Node.JS Perspective on MongoDB 4.0: Transactions
For example, altering the example on the resource above and your example, you can try:
const User = mongoose.model('Users', new mongoose.Schema({
userId: String, wallet: Number
}));
const Transaction = mongoose.model('Transactions', new mongoose.Schema({
userId: ObjectId, amount: Number, type: String
}));
await updateWallet(userId, 500);
async function updateWallet(userId, amount) {
const session = await User.startSession();
session.startTransaction();
try {
const opts = { session };
const A = await User.findOneAndUpdate(
{ _id: userId }, { $inc: { wallet: amount } }, opts);
const B = await Transaction(
{ usersId: userId, amount: amount, type: "credit" })
.save(opts);
await session.commitTransaction();
session.endSession();
return true;
} catch (error) {
// If an error occurred, abort the whole transaction and
// undo any changes that might have happened
await session.abortTransaction();
session.endSession();
throw error;
}
}
is not atomic there is always a possibility of user wallet updated with amount but related transaction not created in transactions collection resulting in financial loss
You should also consider changing your MongoDB data models. Especially if the two collections are naturally linked. See also Model data for Atomic Operations for more information.
An example model that you could try is Event Sourcing model. Create a transaction entry first as an event, then recalculate the user's wallet balance using aggregation.
For example:
{tranId: 1001, fromUser:800, toUser:99, amount:300, time: Date(..)}
{tranId: 1002, fromUser:77, toUser:99, amount:100, time: Date(..)}
Then introduce a process to calculate the amount for each users per period as a cache depending on requirements (i.e. per 6 hours). You can display the current user's wallet balance by adding:
The last cached amount for the user
Any transactions for the user occur since the last cached amount. i.e. 0-6 hours ago.
Due to a management decision, we are using userId for the users collection, postId for the posts collection, and topicId for the topics collection, instead of '_id' for each collection as the unique identifier.
This causes a few problems getting started - one of the problems I have encountered is with upserts -
Using Mongoose, we have a schema that restricts userId to be a unique value - but when doing an update on a user model, with upsert set to true, MongoDB appears to only look at the ObjectIds of a collection to see if the same one exists - it doesn't check to see if a model already exists with the same userId - therefore Mongo does an insert instead of an update.
let me illustrate this with some data:
let's say the user's collection has one document:
{
_id:'561b0fad638e99481ab6d84a'
userId:3,
name:'foo'
}
we then run:
User.update({userId:3},{"$set":{name:'bar'},{upsert:true},function(err,resp){
if(err){
// "errMessage": "insertDocument :: caused by :: 11000 E11000 duplicate key error index: app42153482.users.$userId_1 dup key: { : 3 }",
}
});
one would think that MongoDB would find the existing document with userId:3 and udpate it, so there must be something I am doing wrong since it's giving me the duplicate key error?
Typically the default value ObjectId is more ideal for the _id. Here, in this situation you can either override the default _id or you can have your own field for id(like userId in your case).
Use a separate counters collection to track the last number sequence used. The _id field contains the sequence name and the seq field contains the last value of the sequence.
Insert into the counters collection, the initial value for the userid:
db.counters.insert( {
_id: "userid",
seq: 0 } )
Create a getNextSequence function that accepts a name of the sequence. The function uses the findAndModify() method to atomically increment the seq value and return this new value:
function getNextSequence(name) {
var ret = db.counters.findAndModify(
{
query: { _id: name },
update: { $inc: { seq: 1 } },
new: true
}
);
return ret.seq;
}
Use this getNextSequence() function during insert().
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Sarah C."
}
)
db.users.insert(
{
_id: getNextSequence("userid"),
name: "Bob D."
}
)
This way you can maintain as many sequences as you want in the same counter collection. For the upsert issue, check out the Optimistic Loop block in this link Create an auto-increment sequence field.
The second approach is to use a mongoose middleware like mongodb-autoincrement.
Hope it helps.
I don't know which versions of MongoDB and Mongoose you are using, but I couldn't reproduce your problem with MongoDB 3.0 and Mongoose 4.1.10.
I made a sample for you which will create and save a new user, update (using upsert) it, and create another one through an upsert. Try running this code:
"use strict";
var mongoose=require("mongoose");
var Schema = require('mongoose').Schema;
var ObjectId = mongoose.Schema.Types.ObjectId;
// Connect to test
mongoose.connect("mongodb://localhost:27017/test");
// Lets create your schema
var userSchema = new Schema({
_id: ObjectId,
userId: {type: Number, unique: true },
name: String
});
var User = mongoose.model("User", userSchema, "Users");
User.remove() // Let's prune our collection to start clean
.then( function() {
// Create our sample record
var myUser = new User({
_id:'561b0fad638e99481ab6d84a',
userId:3,
name:'foo'
});
return myUser.save();
})
.then( function() {
// Now its time to update (upsert userId 3)
return User.update({userId:3},{"$set":{name:'bar'}},{upsert:true});
})
.then( function() {
// Now its time to insert (upsert userId 4)
return User.update({userId:4},{"$set":{name:'bee'}},{upsert:true});
})
.then( function() {
// Lets show what we have inserted
return User.find().then(function(data) {console.log(data)});
})
.catch( function(err) {
// Show errors if anything goes wrong
console.error("ERROR", err);
})
.then( function() {
mongoose.disconnect();
});
Following the documentation (of MongoDB 3.0) upsert:true will only not insert a non-existing document if your query conditions match on the _id field.
See: https://docs.mongodb.org/manual/reference/method/db.collection.update/#mongodb30-upsert-id
Why are you not using the user_name for a user as unique id?
Because auto-incrementing fields as ids are a bad practice to use in a mongodb environment, especially if you want to use sharding
=> all your inserts will occur on the latest shard
=> the mongodb cluster will have to rebalance often / redistribute the data around.
(Currently this will not occur on your system as you still use the generated _id field)
You can off course also create a unique index on the user_id field:
https://docs.mongodb.org/manual/core/index-unique/#index-type-unique