I am using MongoClient in NodeJs to carry out Db operations, I am running into connection and db caching issues, where the application shows the db connections are getting disconnected while doing db operations
Following are the errors -
MongoError: MongoClient must be connected before calling MongoClient.prototype.db
at MongoClient.db (/var/task/node_modules/mongodb/lib/mongo_client.js:313:11)
Error: Client network socket disconnected before secure TLS connection was established
at connResetException (internal/errors.js:609:14)
at TLSSocket.onConnectEnd (_tls_wrap.js:1557:19)
at Object.onceWrapper (events.js:420:28)
at TLSSocket.emit (events.js:326:22)
at TLSSocket.EventEmitter.emit (domain.js:483:12)
at endReadableNT (_stream_readable.js:1241:12)
at processTicksAndRejections (internal/process/task_queues.js:84:21) {
name: 'MongoNetworkError'
}
Unable to connect with the mongoDb : MongoError: Topology closed
at NativeTopology.close (/var/task/node_modules/mongodb/lib/core/sdam/topology.js:325:38)
at /var/task/node_modules/mongodb/lib/mongo_client.js:267:21
at maybePromise (/var/task/node_modules/mongodb/lib/utils.js:719:3)
at MongoClient.close (/var/task/node_modules/mongodb/lib/mongo_client.js:248:10)
Following is my DB connection and closing code-
I call the client method while going any db operation like findOne, insertOne, etc And closeDBConnection right before returning the API response.
import { MongoClient, Db } from "mongodb";
let cachedDb: Db = null;
let mongoClient: MongoClient = null;
export const client = async (): Promise<Db> => {
if (cachedDb && await mongoClient.isConnected()) {
console.info("cached db connection established");
return cachedDb;
}
mongoClient = new MongoClient('db URL here', {
useUnifiedTopology: true,
useNewUrlParser: true,
connectTimeoutMS: 90000,
poolSize: 10
});
try {
await mongoClient.connect();
if (await !mongoClient.isConnected()) {
await mongoClient.connect(err => {
if (err) {
console.error("Error : ", err);
throw new Error(`Unable to connect MongoDb Client, Reason: ${err}`
);
}
});
}
cachedDb = await mongoClient.db(await "DB name here");
console.info("db connection established");
return cachedDb;
} catch (error) {
console.error("Error in connecting to DB => ", error);
throw error;
}
};
export const closeDBConnection = async () => {
try {
if( mongoClient != null ){
await mongoClient.close();
mongoClient = null;
cachedDb = null;
console.log("db connection closed");
return true;
}
} catch (error) {
console.log("Error in closeDBConnection => ", error);
return false;
}
};
I've added connectTimeoutMS: 90000,poolSize: 10 to increase the default settings of 30000 and 5 respectively. Will it resolve the above errors?
What is the ideal way to cache a db connection in NodeJS on AWS Lambda?
Ref -
https://mongodb.github.io/node-mongodb-native/3.2/api/MongoClient.html
Related
I'm trying to connect to my db for each time I make a CRUD action, and then disconnect. When I connecting at the first time it's working fine, but when I try to connect again it's not working, and making the disconnect function insted.
i tryed to remove the if cheking but it's still happaning.
what am i doing wrong?
database:
import Mongoose from 'mongoose';
let database: Mongoose.Connection;
export async function connect() {
const uri = 'mongodb://localhost:27017/storage';
if (database) { //problem here
return;
}
Mongoose.connect(uri);
database = Mongoose.connection;
database.once('open', async () => {
console.log('Connected to database successfully');
});
database.on('error', () => {
console.log(`Error connecting to database. Check Whether mongoDB
installed or you can try to give opensource Mongo Atlas database`);
});
return database;
};
export async function disconnect() {
if (!database) {
return;
}
Mongoose.disconnect();
console.log('Disconnected Successfuly')
};
controller:
async function create(req: Request, res: Response) {
const data = req.query;
const notValid = validateQueryParams(Object(data))
if (notValid) {
res.send(notValid)
} else {
try {
await connect();
await addTodb(data)
res.send('create')
} catch (err) {
res.send('Error creating')
} finally {
await disconnect();
}
}
}
I am trying to write a code to connect with snowflake inside the azure function app using nodejs stack. Here is the code I have writtern.
async function af2snf()
{
console.log("starting connection");
var snowflake = require('snowflake-sdk');
// Create a Connection object that we can use later to connect.
console.log("prep conn");
var connection = snowflake.createConnection( {
account: "myaccountname",
username: "myusername",
password: "mypass",
database: "dbname",
schema: "schemaname"
}
);
console.log("making connection");
var conn_id = connection.connect(
function(err, conn) {
if (err) {
console.error('Unable to connect: ' + err.message);
}
else {
console.log('Successfully connected to Snowflake.');
// Optional: store the connection ID.
connection_ID = conn.getId();
}
}
)
return conn_id.getId();
}
console.log("outside fucntion");
console.log(af2snf());
console.log("executed fucntion");
The output I am getting is :
outside fucntion
starting connection
prep conn
making conn
Promise { <pending> }
executed fucntion
Successfully connected to Snowflake.
But What I need is :
outside fucntion
starting connection
prep conn
making conn
Successfully connected to Snowflake.
Promise { <pending> }
executed fucntion
I am very new to nodejs. Please help me out on this
af2nsf needs to return a promise :
async function af2snf() {
return new Promise((resolve,reject)=>{
console.log("starting connection");
var snowflake = require('snowflake-sdk');
// Create a Connection object that we can use later to connect.
console.log("prep conn");
var connection = snowflake.createConnection( {
account: "myaccountname",
username: "myusername",
password: "mypass",
database: "dbname",
schema: "schemaname"
});
console.log("making connection");
connection.connect( async function(err, conn) {
if (err) {
console.error('Unable to connect: ' + err.message);
return reject(err);
}
else {
console.log('Successfully connected to Snowflake.');
// Optional: store the connection ID.
var conn_id = await conn.getId();
return resolve(conn_id);
}
})
});
}
console.log("outside fucntion");
af2snf().then(conn_id=>{
console.log(conn_id);
console.log("executed fucntion");
}).catch(err=>{
console.error(err);
});
I'm trying to connect to mongodb(mongodb package) using the tunnel-ssh package. It gets connected and I can log the db but it immediately throws an error and disconnects.
buffer.js:705
throw new ERR_INVALID_ARG_TYPE(
^
TypeError [ERR_INVALID_ARG_TYPE]: The "string" argument must be one of type string, Buffer, or ArrayBuffer. Received type undefined
at Function.byteLength (buffer.js:705:11)
at SSH2Stream.directTcpip (D:\WORK\node_modules\ssh2-streams\lib\ssh.js:1128:23)
at openChannel (D:\WORK\node_modules\ssh2\lib\client.js:1142:21)
at Client.forwardOut (D:\WORK\node_modules\ssh2\lib\client.js:994:10)
at Client.<anonymous> (D:\WORK\node_modules\tunnel-ssh\index.js:16:23)
at Client.emit (events.js:223:5)
at SSH2Stream.<anonymous> (D:\WORK\node_modules\ssh2\lib\client.js:601:10)
at Object.onceWrapper (events.js:312:28)
at SSH2Stream.emit (events.js:223:5)
at parsePacket (D:\WORK\node_modules\ssh2-streams\lib\ssh.js:3911:10) {
code: 'ERR_INVALID_ARG_TYPE'
}
This is my code.
const tunnel = require("tunnel-ssh");
const config = require("config");
const MongoClient = require("mongodb").MongoClient;
const connection = new Promise((resolve, _) => {
// eslint-disable-next-line
tunnel(config.get("server"), async (err, server) => {
server.on("connection", console.log.bind(console, "server error"));
const client = await MongoClient.connect(config.get("mongodb").url, {
useUnifiedTopology: true,
useNewUrlParser: true
});
client.on("error", console.error.bind(console, "mongodb error"));
resolve({ client });
});
});
async function runQuery() {
const { client} = await connection;
console.log(client);
}
runQuery();
There is no problem with config. In fact, the logging in runQuery function works but throws that error immediately.
I have not used the tunnel-ssh package you have mentioned, but I went through the docs and I see that you are using it wrong. I simply copied the configuration given in the docs of tunnel-ssh and it started working for me. pasting the entire code below
const tunnel = require("tunnel-ssh");
const MongoClient = require("mongodb").MongoClient;
const connection = new Promise((resolve, _) => {
// eslint-disable-next-line
tunnel(
{
username: "root",
Password: "secret",
host: "127.0.0.1",
port: 22,
dstHost: "127.0.0.1",
dstPort: 27017,
localHost: "127.0.0.1",
localPort: 27000
},
async (err, server) => {
server.on("connection", console.log.bind(console, "server error"));
const client = await MongoClient.connect(
"mongodb://localhost:27017/user",
{
useUnifiedTopology: true,
useNewUrlParser: true
}
);
client.on("error", console.error.bind(console, "mongodb error"));
resolve({ client });
server.close();
}
);
});
async function runQuery() {
const { client } = await connection;
console.log("Connection Successful");
}
runQuery();
The part where you went wrong is passing string to tunnel package. it expects configuration object not string.
I am JS newbie so this may be some silly trouble. I have a lambda written in NodeJS 10.x and I am trying to add MongoDB Atlas insertion. I have started with this tutorial: https://docs.atlas.mongodb.com/best-practices-connecting-to-aws-lambda/
This is my code:
const MongoClient = require('mongodb').MongoClient;
let cachedDb = null;
function connectToDatabase (uri) {
console.log('Connect to mongo database');
if (cachedDb) {
console.log('Using cached database instance');
return Promise.resolve(cachedDb);
}
return MongoClient.connect(uri)
.then(db => {
console.log('Successful connect');
cachedDb = db;
return cachedDb;
}).catch(err => {
console.log('Connection error occurred: ', err);
callback(err);
});
}
function insertUser(db, email) {
console.log('=> modify database');
return db.collection('users').insertOne({"email" : email})
.then(() => { callback(null, result); })
.catch(err => {
console.log('Insert error occurred: ', err);
callback(err);
});
}
exports.handler = (payload, context, callback) => {
const { email, password } = JSON.parse(payload.body);
context.callbackWaitsForEmptyEventLoop = false;
connectToDatabase(MONGODB_URI)
.then(db => {
console.log('Mongo connected')
insertUser(db, email);
})
.then(result => {
console.log('Mongo insert succeeded', result);
})
.catch(err => {
console.log('Mongo insert failed', err);
return responses.INTERNAL_SERVER_ERROR_500(err, callback, response);
});
console.log('finished mongo stuff');
I can see the following logs in CloudWatch:
START RequestId: 0338d336-7d33-40d5-abc7-1511f1c9ea4c Version: $LATEST
2020-01-11T12:18:00.808Z 0338d336-7d33-40d5-abc7-1511f1c9ea4c INFO Connect to mongo database
2020-01-11T12:18:00.855Z 0338d336-7d33-40d5-abc7-1511f1c9ea4c INFO finished mongo stuff
2020-01-11T12:18:01.416Z 0338d336-7d33-40d5-abc7-1511f1c9ea4c ERROR (node:8) DeprecationWarning: current Server Discovery and Monitoring engine is deprecated, and will be removed in a future version. To use the new Server Discover and Monitoring engine, pass option { useUnifiedTopology: true } to the MongoClient constructor.
END RequestId: 0338d336-7d33-40d5-abc7-1511f1c9ea4c
The item is not inserted in Atlas. I have added more verbose logging but it is not shown. If the mongo connect failed there shall be an error. But it seems that the error handlers are ignored. Where is the problem?
You are very close. Few things are missing
callback function needs to be called from your handler function as the insert function doesn't know about callback.
When you do a .then on a promise and you use curly braces you need to return whatever is in there to send it to the next then (unless you do it in a single line).
db in insert method needs to be db.db()
I would recommend to to use async/await instead of callback style. Here is the equivalent code
exports.handler = async (payload) => {
try {
const { email, password } = JSON.parse(payload.body);
const db = await connectToDatabase(MONGODB_URI);
console.log("Mongo connected");
const result = await insertUser(db, email);
console.log("Mongo insert succeeded", result);
return result;
} catch(err) {
console.error(err);
}
};
I moved away from using the default connection to a explicitly defined connection.
My code is working except that none of the event handlers which track the state of my mongo connection are firing.
the events used to fire when I was using default mongoose connection. (mongoose.connect).
var mongoose = require('mongoose'),
logger = require('../logger'),
config = require('../config');
mongoose.connection.on('connecting', function(){
logger.info("trying to establish a connection to mongo");
});
mongoose.connection.on('connected', function() {
logger.info("connection established successfully");
});
mongoose.connection.on('error', function(err) {
logger.error('connection to mongo failed ' + err);
});
mongoose.connection.on('disconnected', function() {
logger.log('mongo db connection closed');
})
var gracefulExit = function() {
db.close(function(){
logger.log("mongoose connection with db " + server + 'is closing');
process.exit(0);
});
};
process.on('SIGNT', gracefulExit).on('SIGTERM', gracefulExit);
var db = mongoose.createConnection(config.get('DB_URL'));
module.exports = db;
the problem with the code is
None of the event handlers are firing.
Even the process events on SIGNT and SIGTERM are not firing.
I also tried
var db = mongoose.createConnection(config.get('DB_URL'));
db.on('connecting', function(){
logger.info("trying to establish a connection to mongo");
});
but this also doesn't fire.
i also faced with this problem,
and finally i solved it, after understood that the connect() / createConnection() execute before i declare the event handlers.
so the my open or connected events handlers will never called again.
so by declare them first and just after trying to connect - will solve your problem!
for example:
try {
mongoose.connection
.on('error', err => {
console.error(err);
})
.on('open', err => {
console.log(`DB connected`);
})
await mongoose.connect('mongodb://localhost:27017/test', { useNewUrlParser: true });
} catch (error) {
handleError(error);
}
This snippet blow work fine for me
const connection = mongoose.createConnection(`mongodb://${process.env.MONGO_URI}`);
connection.on('connected', () => {
console.log('connected to mongodb');
});
connection.on('disconnected', () => {
console.log('connection disconnected');
});
I just stumbled over this issue myself and found that I needed to set up the EventListeners before the actual connection to the database like so (with async/await):
export class Database {
public static connect( databaseCredentials: string ) {
return new Promise( async resolve => {
// ! The `EventListeners` is setup here.
mongoose.connection.on( "connected", () => {
logger.info( "Database has connected successfully." );
});
mongoose.connection.on( "error", (error) => {
logger.error( " Obs! There was an unexpected error connecting to the database.", error );
});
// ! The acutal connection to the database happens here.
await mongoose.connect( databaseCredentials, {
useNewUrlParser: true,
useUnifiedTopology: true,
keepAlive: true,
reconnectInterval: 500,
connectTimeoutMS: 10000,
});
resolve();
});
}
}
This should also work the same way with mongoose.createConnections.
This will yield the result (if successful):
{ message: 'Database has connected successfully.',
level: 'info',
timestamp: '2019-10-11T15:17:16.616Z' }
I just wanted to chip in and give my solution to the problem at hand and give a full example implementation of the code, as the above answers didn't explain in which order you needed to listen for the events. If you were to put the mongoose.connection.on() below the await.mongoose.connect(), this would not yield any response from the EventListeners.
Edit: Typos
try:
mongoose.connection.on('open', function() {
logger.info("database is ready now");
});
also temporary remove .close()