I am using mongodb on typescript and my problem is that the connections don't close after querying.
I had the following approach in doing it:
import { Db, MongoClient } from 'mongodb';
let cachedConnection: { client: MongoClient; db: Db } | null = null;
export async function connectToDatabase(mongoUri?: string, database?: string) {
if (!mongoUri) {
throw new Error(
'Please define the MONGO_URI environment variable inside .env.local'
);
}
if (!database) {
throw new Error(
'Please define the DATABASE environment variable inside .env.local'
);
}
if (cachedConnection) return cachedConnection;
cachedConnection = await MongoClient.connect(mongoUri, {
useNewUrlParser: true,
useUnifiedTopology: true,
}).then((client) => ({
client,
db: client.db(database),
}));
return cachedConnection!;
}
and I use it in my nodejs server by doing:
const { db } = await connectToDatabase(config.URI, DB_NAME);
const result = await db.collection(COLLECTION_NAME).aggregate([MyPipelines])
I have the database on Atlas and if I go to the dashboard I see a lot of connections not closed, and when it reaches 500 then the server need to be stopped because it goes in timeout and closes it. They send me one email saying "You're nearing the maximum connections treshold"
In fact now I have all these connections active.
And on console I get these errors:
What do you think I did wrong? Is there a better way to handle mongoDB connection with typescript?
You should close all connections when your app closes completely.
process.on('SIGINT', async () => {
await cachedConnection.db.close();
process.exit(0);
});
Related
Please I am using the MongoDB database for my next application that is my final project for me Bootcamp and there is this error that has prevented me from making queries to my database as I always get the mongoclient not connected error.
I am using mongoose and this error started after I upgraded to the latest mui(material UI) because that is what I am using for this application. I have been trying since yesterday to fix this error as I thought it is something I could handle but till this moment it persist. It has been going from this mongoose MongoNotConnectedError: MongoClient must be connected to perform this operation and this one MongoExpiredSessionError: Cannot use a session that has ended` and it happens on every button that clicks that makes a request to the database.
Below is the code I am using to connect to MongoDB with mongoose:
import mongoose from 'mongoose';
const connection = {};
async function connect() {
if (connection.isConnected) {
console.log('already connected');
return;
}
if (mongoose.connections.length > 0) {
connection.isConnected = mongoose.connections[0].readyState;
if (connection.isConnected === 1) {
console.log('use previous connection');
return;
}
await mongoose.disconnect();
}
const db = await mongoose.connect(process.env.MONGODB_URI, {
useNewUrlParser: true,
useUnifiedTopology: true,
});
console.log('new connection');
connection.isConnected = db.connections[0].readyState;
}
async function disconnect() {
if (connection.isConnected) {
if (process.env.NODE_ENV === 'production') {
await mongoose.disconnect();
connection.isConnected = false;
} else {
console.log('not disconnect');
}
}
}
function convertDocToObj(doc) {
doc._id = doc._id.toString();
doc.createdAt = doc.createdAt.toString();
doc.updatedAt = doc.updatedAt.toString();
return doc;
}
const db = { connect, disconnect, convertDocToObj };
export default db;
I will really appreciate it so much if anybody can help me out with this problem, please. I don't know what is causing it or where it is coming from as I have tried to as much as i can to all to no0 avail
Thanks
I see that you are using the mongo URL as an environment variable with MONGODB_URI.
Do you have the dotenv module installed? Did you require('dotenv').config() in your applications? is the `.env file in the root directory of your project?
If so, if you are using MongoDB Atlas, make sure the URL in your .env file is the correct one. When you generate the connection string from MongoDB Atlas it gives a default DB in the URL string named myFirstDatabase or something like that. Change it to the DB you want to connect to.
Hey everyone i'm developer a simple app in last days using nodejs and create this function to return client instance from mongodb
const mongodb = require("mongodb");
const { db } = require("../config/env");
const conection = async () => {
try {
const client = await mongodb.MongoClient.connect(db.uri, {
useNewUrlParser: true,
useUnifiedTopology: true,
});
return client;
} catch (error) {
throw error;
}
};
module.exports = conection;
and i make this simple function for acess data layer and return records instered
const index = async ({ limit = 10, offset = 0, filter = {} }) => {
const client = await conection();
if (filter._id) {
filter._id = mongodb.ObjectID(filter._id);
}
try {
const collection = client.db("api").collection("user");
const data = await collection
.find({ ...filter })
.skip(offset)
.limit(limit)
.toArray();
return data;
} catch (error) {
throw new Error(error);
} finally {
await client.close();
}
};
I would like to know if I really need to make the connection and close it with each query or should I keep the connection open
NOTE: in this case I am using a simple Atlas cluster (free) but I would like to know if I should do this also when working with sql banks like postgres
Don't close your connection unless you are exiting your app, and then make sure you do. Ensure that you are using the same connection when you come back to do more I/O. Database connections are resource-intensive and should be established as few times as possible and shared as much as possible. You can also get middleware problems with connections like ODBC if you don't pick up existing connections and get weird errors like connection pools running out. Get to know your connector and how to use it most effectively, it will be a rewarding activity :-)
You can use the mongoose module for managing MongoDB.
Installation
npm install mongoose
Usage
mongoose.connect('mongodb://localhost/test', { useNewUrlParser: true });
I am sure that Mongoose can help you solve your issues.
It's a good practice to do so. so that after every operation(insert etc) you close the connection and before every operation(insert etc) you reopen the connection.
I am writing a web application that uses asynchronous database requests as a part of the api. Currently, I have an async express route that awaits function returns from async functions. Both of these functions return booleans and both query the database. One works correctly, however the second one does not.
Here is the MongoClient setup:
const MongoClient = require('mongodb').MongoClient;
const uri = config.uri; // Contains custom url for accessing database
const client = new MongoClient(uri, { useUnifiedTopology: true}, { useNewUrlParser: true }, { connectTimeoutMS: 30000 }, { keepAlive: 1});
where config is from a file imported as.
const config = require("./config.js");
and functions properly.
Here is the express setup:
app.post("/signup", async function(request, response) {
log("POST request at /signup");
log("BEFORE UNIQUE USER");
const isUniqueUser = await validateUniqueUser(request.body.email, request.body.password);
log(isUniqueUser);
const status = {
status: null
};
if (isUniqueUser) {
log("AFTER UNIQUE USER");
let userCreated = await createPracticeProfile(request.body.email, request.body.password);
log("user created: " + userCreated);
if (userCreated) {
status.status = "user_created";
}
response.json(status);
} else {
response.json(status);
}
console.log("********************************end");
});
The console outputs:
BEFORE UNIQUE USER
true (which it should be)
AFTER UNIQUE USER
MongoError: Topology is closed.
user created: undefined
***...***end
Here is the function for validating that a user is unique:
/* VALIDATE_UNIQUE_USER
USE: ensure user does not have existing profile
PARAMS: email (string), password (string)
RETURN: isUniqueUser (bool)
*/
async function validateUniqueUser(email, password) {
// connect to database
const database = await client.connect().catch(err => {
log("ERROR while connecting to database at: validateUniqueUser");
console.log(err);
client.close();
});
// database connection failed
if (!database) {
return false;
}
// connection successful => find user
let user;
try {
user = await database.db("guitar-practice-suite").collection("users").findOne({email: email});
} catch(err) {
log("ERROR while finding user in database at: validateUniqueUser");
console.log(err);
client.close();
return false;
} finally {
client.close();
// user not found (unique)
if (user === null || user === undefined) {
return true;
}
return false;
}
}
Here is the function for inserting the user into the collections:
/* CREATE_PRACTICE_PROFILE
USE: insert a practice profile into the database
PARAMS: email (string), password (string)
RETURN: userCreated (bool)
*/
async function createPracticeProfile(email, password) {
// hash password
let hashedPassword;
try {
hashedPassword = await new Promise((resolve, reject) => {
bcrypt.hash(password, null, null, function(err, hash) {
if (err) {
reject(err);
}
resolve(hash)
});
});
} catch(err) {
log("ERROR while hashing password at: createPracticeProfile");
console.log(err);
return false;
}
// connect to database
const database = await client.connect().catch(err => {
log("ERROR while connecting to database at: validateUniqueUser");
console.log(err);
client.close();
});
// database connection failed
if (!database) {
return false;
}
// database connection successful => insert user into database
let insertUserToUsers;
let insertUserToExercises;
let insertUserToCustomExercises;
try {
insertUserToUsers = await database.db("guitar-practice-suite").collection("users").insertOne({email: email, password: hashedPassword});
insertUserToExercises = await database.db("guitar-practice-suite").collection("exercises").insertOne({email: email});
insertUserToCustomExercises = await database.db("guitar-practice-suite").collection("custom-exercises").insertOne({email: email, exercises: []});
} catch(err) {
log("ERROR while inserting user into database at: createPracticeProfile");
console.log(err);
client.close();
return false;
} finally {
client.close();
return insertUserToUsers && insertUserToExercises && insertUserToCustomExercises;
}
}
I've found the solution to the problem, but I'm not sure I understand the reasoning.
The client.close() in the finally block of the validateUniqueUser function. It was closing the connection before the connection in the createPracticeProfile function was finished inserting the user.
When that line is taken out, the function works.
The issue is client variable needs to be reinstantiated again,
const client = new MongoClient(uri, { useUnifiedTopology: true}, { useNewUrlParser: true }, { connectTimeoutMS: 30000 }, { keepAlive: 1});
Try putting this in start of createPracticeProfile, validateUniqueUser and other functions
I was getting the error
MongoError: Topology is closed
because of the authentication problem
MongoEror: Authentication failed
In my case, the problem was with the password of my database. My password only contained numerical digits.
I changed the password to all characters and both the errors were solved.
Configure your client connection like below example
var MongoClient = require('mongodb').MongoClient;
var Server = require('mongodb').Server;
var mongoClient = new MongoClient(new Server('localhost', 27017));
mongoClient.open(function(err, mongoClient) {
var db1 = mongoClient.db("mydb");
mongoClient.close();
});
In my case - connecting to AtlasDB using the MongoClient - I had to whitelist the IP i was accessing the cluster from
I think your mongodb service is stopped, to start it
Task Manager -> Services -> Mongodb -> RightClick -> Start
My code has been working fine for a long time and hasn't thrown this error before: MongoError: Topology is closed.
But due to the fact that my laptop was turned on for a long time and I was simultaneously developing other projects on it, while the main one was running in the terminal, mongo most likely did not close one of the connections to the database and opened another in parallel, creating some kind of collision.
In general, in my case, the usual restart of the computer helped and a similar error did not occur again.
I am using mongoose to connect to my Mongodb Atlas cluster through my nodejs server.
There is a certain operation which is done as a transaction. Mongoose needs mongoose.startSession() to be called to start a transaction. Very infrequently, this mongoose.startSession() call hangs indefinitely. There is no certain way to reproduce this.
log.info('starting lock session');
const mongoSession = await mongoose.startSession();
log.info('lock session started');
In above code, starting lock session. gets logged, but lock session started doesn't get logged when issue occurs.
I connect to the db like below:
const dburl = 'mongodb+srv://myuser:mypassword#myapp.mongodb.net/mydb?retryWrites=true&w=majority';
mongoose.connect(dburl, {useNewUrlParser: true}, err => {
if (err) {
log.warn('Error occurred when connecting to database. ' + err);
}
});
What could be the reason for this? Could this be due to something wrong with the database? Is there any way I can further troubleshoot this?
This looks like a bug in mongoose, and I reported it to mongoose but still didn't get a response.
https://github.com/Automattic/mongoose/issues/8325
I wrote following function which I can use to wait until the mongoose connection is ready before calling startSession(), and it fixes my problem.
function waitForMongooseConnection(mongoose) {
return new Promise((resolve) => {
const connection = mongoose.connection;
if (connection.readyState === 1) {
resolve();
return;
}
console.log('Mongoose connection is not ready. Waiting for open or reconnect event.');
let resolved = false;
const setResolved = () => {
console.log('Mongoose connection became ready. promise already resolved: ' + resolved);
if (!resolved) {
console.log('Resolving waitForMongooseConnection');
resolved = true;
resolve();
}
};
connection.once('open', setResolved);
connection.once('reconnect', setResolved);
});
}
With above function, I can start session like below:
log.info('starting session');
await waitForMongooseConnection(mongoose);
const mongoSession = await mongoose.startSession();
log.info('session started');
Note that I had to turn off useUnifiedTopology. Otherwise, 'reconnect' didn't get called.
mongoose.connect(config.db, {useNewUrlParser: true, useUnifiedTopology: false}, err => {
if (err) {
log.warn('Error occurred when connecting to database. ' + err);
}
});
I resolved this issue by using connection object coming from
mongoose.createConnection(uri, options)
const connection = mongoose.createConnection(uri, options);
const session = await connection.startSession();
session.startTransaction();
await MySchema.create({
value: "Created?",
session: session, // giving session here
});
await session.commitTransaction();
session.endSession();
mongoose.connection is different than this connection object.
Hi is there a way to switch database with mongoose?
I thought I could do it like that:
mongoose.disconnect();
mongoose.connect('localhost',db);
but it does not work I receive this error:
Error: Trying to open unclosed connection.
I do not know if it is because is asynchronous
As already stated you could do it using useDb function :
Example code :
async function myDbConnection() {
const url = 'mongodb+srv://username:password#cluster0-pauvx.mongodb.net/test?retryWrites=true&w=majority';
try {
await mongoose.connect(url, { useNewUrlParser: true });
console.log('Connected Successfully')
// Here from above url you've already got connected to test DB,
So let's make a switch as needed.
mongoose.connection.useDb('myDB'); // Switching happens here..
/**
* Do some DB transaction with mongoose models as by now models has already been registered to created DB connection
*/
} catch (error) {
console.log('Error connecting to DB ::', error);
}
}
Or if you wanted to create a complete new connection then you've to try mongoose.createConnection(). Just for reference in case of mongoDB driver you would use ::
mongodb.MongoClient.connect(mongourl, function(err, primaryDB) {
// open another database over the same connection
const secondaryDB = primaryDB.db(SECONDARY_DATABASE_NAME);
// now you can use both `database` and `database2`
...
});
Ref : mongoose multiple different connections, mongoose useDb(), mongoDB driver switch connections
It is asynchronous. If you pass a callback function to disconnect and try to connect to the next database in that callback, it will work.
Ex.
var mongoose = require('mongoose')
mongoose.connect('mongodb://localhost/test1', function() {
console.log('Connected to test 1')
mongoose.disconnect(connectToTest2)
})
function connectToTest2() {
mongoose.connect('mongodb://localhost/test2', function() {
console.log('Connected to test 2')
process.exit()
})
}
The top-voted answer had thrown me in a loop for a few hours. To help the OP and others who may have the same issue, here's my take on the solution:
Suppose you have two databases with the same schema and you want to switch between them on the fly. Let's call them DB1 and DB2. Here's how you can go about doing that:
(async () => {
const connection = await mongoose.createConnection(url, options);
const getModel = (database) => {
const dbConnection = connection.useDb(database);
return dbConnection.model('Model', modelSchema);
};
const Db1Model = getModel('DB1');
const Db2Model = getModel('DB2');
})();
Tested on Node.js v12 and Mongoose v5.
One way you can achieve this is by appending the database name with the database URL. For eg: if you are working with localhost
mongoose.connect('mongodb://localhost:portNumber/xyz_db');
When you connect like this, all your models will be saved in the xyz_db under your model as a collection.
You should use the useDb function like so:
mongoose.connect(process.env.MONGO_URI, {
useNewUrlParser: true,
useUnifiedTopology: true
});
mongoose.connection.useDb('Users'); # Change the string to the name of the database you want to use