Why Google Cloud Run gettings massive container restart / new instance creation? - node.js

I've been using Google Cloud Run for a year now and the issue with cloud run containers restarts / new container start is from the beginning.
I've hosted Node + MongoDB app in Cloud Run, but cloud run container is restarting frequently. It's getting around 10 - 12 requests / second, couldn't find any performance bottleneck, requests are serving smoothly, sometimes requests are served more than normal time, might be new container instance cold start delay.
The issue I am facing is the HIGH Number of connections to the MONGODB Server. After some research I could find that I've to close mongodb connection on node process exit so I've added a graceful shutdown function.
// Function to terminate the app gracefully:
const gracefulShutdown = async () => {
console.log(`MONGODB CONNECTION CLOSED!`);
await mongoose.connection.close();
};
// This will handle process.exit():
process.on('exit', gracefulShutdown);
// This will handle kill commands, such as CTRL+C:
process.on('SIGINT', gracefulShutdown);
process.on('SIGTERM', gracefulShutdown);
// This will prevent dirty exit on code-fault crashes:
process.on('uncaughtException', gracefulShutdown);
But even after adding this, I couldn't find this graceful shutdown function is invoked while checking logs.
Does google cloud run really signals when the nodejs process in the container crashed?
Is there any way to identity a container restart or new instance creation in cloud run?
Here is the MongoDB connection code
exports.connect = () => {
try {
mongoose
.connect(MONGO.URI, {
useCreateIndex: true,
keepAlive: 1,
useNewUrlParser: true,
useUnifiedTopology: true,
useFindAndModify: false,
})
.then((docs) => {
console.log(`DB Connected`);
})
.catch((err) => {
console.log(`err`, err);
});
return mongoose.connection;
} catch (err) {
console.log(`#### Error Connecting DB`, err);
console.log(`Mongo URI: `, MONGO.URI);
}
};
Sometimes cloud run issues a high number of connections to MONGODB, and hits the connection limit of 1500 connections.
Any suggestions are appreciated! I've been facing this issue for a year now.

You should not start the node process using npm or yarn but directly as CMD ["node", "index.js"] (when you are inside a Docker container using Dockerfile)
Explanation here https://maximorlov.com/process-signals-inside-docker-containers/

Related

Is it normal for app to still receiving request while MongoDB is closing it's connection?

The app trying to use MongoDB connection that has already closed so I got error :
So I use Kubernetes GKE for my app and I only have one way to disconnect to MongoDB that is triggered by SIGTERM or SIGINT.
Stack that I used
NodeJS 16.19.0, Mongoose 5.8.1, GKE as infra
this is how I close the connection :
const gracefulExit = async (args) => {
console.log('RECEIVED GRACEFUL EXIT', args);
await agenda.stop();
await mongoose.connection.close();
console.info('Mongoose disconnected on app termination');
process.exit(0);
};
process
.on('SIGINT', gracefulExit)
.on('SIGTERM', gracefulExit);
the await process takes seconds to finish (4-7s) so I think that is why it still receive some requests.
So is it normal for app to still receiving request while MongoDB is closing it's connection? or should I just leave the connection and not close it? if not then what can I do to fix this? Thanks!
WHAT I TRIED?
Reconnect to MongoDB if it's disconnected, I know it's strange because the app will go down anyway, but the error still persists
try {
if (!filter) {
throw 'GET ONE: No filter supplied';
}
if(mongoose.connection.readyState !== 1) {
console.log("reconnecting...", mongoose.connection.readyState)
await connectDbAsync()
console.log("reconnected", mongoose.connection.readyState)
}
const result = await this.model.findOne(filter, fields, options);
return result;
} catch (e) {
console.error('GET ONE ERROR:', e);
throw e
}
Trying to change parameters in the connection such as smaller heartbeatMS
const options = {
useUnifiedTopology: !isLocal,
useNewUrlParser: true,
useFindAndModify: false,
useCreateIndex: true,
poolSize: 10,
serverSelectionTimeoutMS: 5000, // set the server selection timeout
heartbeatFrequencyMS: 5000, // set the heartbeat frequency
connectTimeoutMS: 10000, // set the connection timeout
};
What I Expect
No request to the app when the closing the MongoDB connection, or to avoid "Cannot use a session that has ended" error.

How to fix nodejs connect to mongo "MongoError: topology was destroyed" [duplicate]

I have a REST service built in node.js with Restify and Mongoose and a mongoDB with a collection with about 30.000 regular sized documents.
I have my node service running through pmx and pm2.
Yesterday, suddenly, node started crapping out errors with the message "MongoError: Topology was destroyed", nothing more.
I have no idea what is meant by this and what could have possibly triggered this. there is also not much to be found when google-searching this. So I thought I'd ask here.
After restarting the node service today, the errors stopped coming in.
I also have one of these running in production and it scares me that this could happen at any given time to a pretty crucial part of the setup running there...
I'm using the following versions of the mentioned packages:
mongoose: 4.0.3
restify: 3.0.3
node: 0.10.25
It seems to mean your node server's connection to your MongoDB instance was interrupted while it was trying to write to it.
Take a look at the Mongo source code that generates that error
Mongos.prototype.insert = function(ns, ops, options, callback) {
if(typeof options == 'function') callback = options, options = {};
if(this.s.state == DESTROYED) return callback(new MongoError(f('topology was destroyed')));
// Topology is not connected, save the call in the provided store to be
// Executed at some point when the handler deems it's reconnected
if(!this.isConnected() && this.s.disconnectHandler != null) {
callback = bindToCurrentDomain(callback);
return this.s.disconnectHandler.add('insert', ns, ops, options, callback);
}
executeWriteOperation(this.s, 'insert', ns, ops, options, callback);
}
This does not appear to be related to the Sails issue cited in the comments, as no upgrades were installed to precipitate the crash or the "fix"
I know that Jason's answer was accepted, but I had the same problem with Mongoose and found that the service hosting my database recommended to apply the following settings in order to keep Mongodb's connection alive in production:
var options = {
server: { socketOptions: { keepAlive: 1, connectTimeoutMS: 30000 } },
replset: { socketOptions: { keepAlive: 1, connectTimeoutMS: 30000 } }
};
mongoose.connect(secrets.db, options);
I hope that this reply may help other people having "Topology was destroyed" errors.
This error is due to mongo driver dropping the connection for any reason (server was down for example).
By default mongoose will try to reconnect for 30 seconds then stop retrying and throw errors forever until restarted.
You can change this by editing these 2 fields in the connection options
mongoose.connect(uri,
{ server: {
// sets how many times to try reconnecting
reconnectTries: Number.MAX_VALUE,
// sets the delay between every retry (milliseconds)
reconnectInterval: 1000
}
}
);
connection options documentation
In my case, this error was caused by a db.close(); out of a 'await' section inside of 'async'
MongoClient.connect(url, {poolSize: 10, reconnectTries: Number.MAX_VALUE, reconnectInterval: 1000}, function(err, db) {
// Validate the connection to Mongo
assert.equal(null, err);
// Query the SQL table
querySQL()
.then(function (result) {
console.log('Print results SQL');
console.log(result);
if(result.length > 0){
processArray(db, result)
.then(function (result) {
console.log('Res');
console.log(result);
})
.catch(function (err) {
console.log('Err');
console.log(err);
})
} else {
console.log('Nothing to show in MySQL');
}
})
.catch(function (err) {
console.log(err);
});
db.close(); // <--------------------------------THIS LINE
});
Just a minor addition to Gaafar's answer, it gave me a deprecation warning. Instead of on the server object, like this:
MongoClient.connect(MONGO_URL, {
server: {
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 1000
}
});
It can go on the top level object. Basically, just take it out of the server object and put it in the options object like this:
MongoClient.connect(MONGO_URL, {
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 1000
});
"Topology was destroyed" might be caused by mongoose disconnecting before mongo document indexes are created, per this comment
In order to make sure all models have their indexes built before disconnecting, you can:
await Promise.all(mongoose.modelNames().map(model => mongoose.model(model).ensureIndexes()));
await mongoose.disconnect();
I met this in kubernetes/minikube + nodejs + mongoose environment.
The problem was that DNS service was up with a kind of latency. Checking DNS is ready solved my problem.
const dns = require('dns');
var dnsTimer = setInterval(() => {
dns.lookup('mongo-0.mongo', (err, address, family) => {
if (err) {
console.log('DNS LOOKUP ERR', err.code ? err.code : err);
} else {
console.log('DNS LOOKUP: %j family: IPv%s', address, family);
clearTimeout(dnsTimer);
mongoose.connect(mongoURL, db_options);
}
});
}, 3000);
var db = mongoose.connection;
var db_options = {
autoReconnect:true,
poolSize: 20,
socketTimeoutMS: 480000,
keepAlive: 300000,
keepAliveInitialDelay : 300000,
connectTimeoutMS: 30000,
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 1000,
useNewUrlParser: true
};
(the numbers in db_options are arbitrary found on stackoverflow and similar like sites)
I alse had the same error. Finally, I found that I have some error on my code. I use load balance for two nodejs server, but I just update the code of one server.
I change my mongod server from standalone to replication, but I forget to do the corresponding update for the connection string, so I met this error.
standalone connection string:
mongodb://server-1:27017/mydb
replication connection string:
mongodb://server-1:27017,server-2:27017,server-3:27017/mydb?replicaSet=myReplSet
details here:[mongo doc for connection string]
Sebastian comment on Adrien's answer needs more attention it helped me but it being a comment might be ignore sometime so here's a solution:
var options = { useMongoClient: true, keepAlive: 1, connectTimeoutMS: 30000, reconnectTries: 30, reconnectInterval: 5000 }
mongoose.connect(config.mongoConnectionString, options, (err) => {
if(err) {
console.error("Error while connecting", err);
}
});
Here what I did, It works fine. Issue was gone after adding below options.
const dbUrl = "mongodb://localhost:27017/sampledb";
const options = { useMongoClient: true, keepAlive: 1, connectTimeoutMS: 30000, reconnectTries: 30, reconnectInterval: 5000, useNewUrlParser: true }
mongoose.connect(dbUrl,options, function(
error
) {
if (error) {
console.log("mongoerror", error);
} else {
console.log("connected");
}
});
You need to restart mongo to solve the topology error, then just change some options of mongoose or mongoclient to overcome this problem:
var mongoOptions = {
useMongoClient: true,
keepAlive: 1,
connectTimeoutMS: 30000,
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 5000,
useNewUrlParser: true
}
mongoose.connect(mongoDevString,mongoOptions);
I got this error, while I was creating a new database on my MongoDb Compass Community. The issue was with my Mongod, it was not running. So as a fix, I had to run the Mongod command as preceding.
C:\Program Files\MongoDB\Server\3.6\bin>mongod
I was able to create a database after running that command.
Hope it helps.
I was struggling with this for some time - As you can see from other answers, the issue can be very different.
The easiest way to find out whats causing is this is to turn on loggerLevel: 'info' in the options
In my case, this error was caused by an identical server instance already running background.
The weird thing is when I started my server without notice there's one running already, the console didn't show anything like 'something is using port xxx'. I could even upload something to the server. So, it took me quite long to locate this problem.
What's more, after closing all the applications I can imagine, I still could not find the process which is using this port in my Mac's activity monitor. I have to use lsof to trace. The culprit was not surprising - it's a node process. However, with the PID shown in the terminal, I found the port number in the monitor is different from the one used by my server.
All in all, kill all the node processes may solve this problem directly.
Using mongoose here, but you could do a similar check without it
export async function clearDatabase() {
if (mongoose.connection.readyState === mongoose.connection.states.disconnected) {
return Promise.resolve()
}
return mongoose.connection.db.dropDatabase()
}
My use case was just tests throwing errors, so if we've disconnected, I don't run operations.
I got this problem recently. Here what I do:
Restart MongoDb: sudo service mongod restart
Restart My NodeJS APP. I use pm2 to handle this pm2 restart [your-app-id]. To get ID use pm2 list
var mongoOptions = {
useNewUrlParser: true,
useUnifiedTopology: true,
}
mongoose.connect(mongoDevString,mongoOptions);
I solved this issue by:
ensuring mongo is running
restarting my server

Mongodb gets locked when working with crontab

I'm creating an application that pulls data from APIs and save it to a MongoDB using crontab.
This is an example of what my crontab file look like:
*/2 * * * * /home/`whoami`/TLC/dataProvider1.js
*/1 * * * * /home/`whoami`/TLC/dataProvider2.js
*/3 * * * * /home/`whoami`/TLC/dataProvider3.js
The scripts are JS files that I've made executable and they all have a require statement for a separate file that has a database connection, so each of them connects to the DB so it can save the pulled data.
DB connection file:
var mongoose = require('mongoose');
var connect = function () {
mongoose.connect('mongodb://localhost:27017/vario', {
socketTimeoutMS: 0,
keepAlive: true,
reconnectTries: 30,
useNewUrlParser: true,
autoReconnect: true,
keepAliveInitialDelay: 300000
});
};
connect();
mongoose.connection.on('connected', function () {
console.log('Connection established to the Database...');
});
mongoose.connection.on('error', function (err) {
console.log('Could not connect to Database.\nError: ' + err);
});
process.on('SIGINT', function () {
mongoose.connection.close(function () {
console.log('Force to close the MongoDB conection');
process.exit(0);
});
});
module.exports = connect;
Problem: When these jobs run for about an hour, MongoDB somehow gets locked and no more data can be saved. Firing the mongo shell produces an error that the connection has been refused. Using mongod to recover also gives an error that another instance on the data file (/data/db/) is already running so it cannot lock the mongod.lock file. The only way I'm dealing with this right now is by shutting down the deamon and restarting it. How can I handle this?
Note: I used crontab because setInterval was interfering with browsing the app in case it was saving to the DB.
After a careful inspection, I realized that my crontab jobs were creating new connections to the DB that were never closed. All I had to do is ensure that a connection is closed when a script finishes running.

mongoError: Topology was destroyed

I have a REST service built in node.js with Restify and Mongoose and a mongoDB with a collection with about 30.000 regular sized documents.
I have my node service running through pmx and pm2.
Yesterday, suddenly, node started crapping out errors with the message "MongoError: Topology was destroyed", nothing more.
I have no idea what is meant by this and what could have possibly triggered this. there is also not much to be found when google-searching this. So I thought I'd ask here.
After restarting the node service today, the errors stopped coming in.
I also have one of these running in production and it scares me that this could happen at any given time to a pretty crucial part of the setup running there...
I'm using the following versions of the mentioned packages:
mongoose: 4.0.3
restify: 3.0.3
node: 0.10.25
It seems to mean your node server's connection to your MongoDB instance was interrupted while it was trying to write to it.
Take a look at the Mongo source code that generates that error
Mongos.prototype.insert = function(ns, ops, options, callback) {
if(typeof options == 'function') callback = options, options = {};
if(this.s.state == DESTROYED) return callback(new MongoError(f('topology was destroyed')));
// Topology is not connected, save the call in the provided store to be
// Executed at some point when the handler deems it's reconnected
if(!this.isConnected() && this.s.disconnectHandler != null) {
callback = bindToCurrentDomain(callback);
return this.s.disconnectHandler.add('insert', ns, ops, options, callback);
}
executeWriteOperation(this.s, 'insert', ns, ops, options, callback);
}
This does not appear to be related to the Sails issue cited in the comments, as no upgrades were installed to precipitate the crash or the "fix"
I know that Jason's answer was accepted, but I had the same problem with Mongoose and found that the service hosting my database recommended to apply the following settings in order to keep Mongodb's connection alive in production:
var options = {
server: { socketOptions: { keepAlive: 1, connectTimeoutMS: 30000 } },
replset: { socketOptions: { keepAlive: 1, connectTimeoutMS: 30000 } }
};
mongoose.connect(secrets.db, options);
I hope that this reply may help other people having "Topology was destroyed" errors.
This error is due to mongo driver dropping the connection for any reason (server was down for example).
By default mongoose will try to reconnect for 30 seconds then stop retrying and throw errors forever until restarted.
You can change this by editing these 2 fields in the connection options
mongoose.connect(uri,
{ server: {
// sets how many times to try reconnecting
reconnectTries: Number.MAX_VALUE,
// sets the delay between every retry (milliseconds)
reconnectInterval: 1000
}
}
);
connection options documentation
In my case, this error was caused by a db.close(); out of a 'await' section inside of 'async'
MongoClient.connect(url, {poolSize: 10, reconnectTries: Number.MAX_VALUE, reconnectInterval: 1000}, function(err, db) {
// Validate the connection to Mongo
assert.equal(null, err);
// Query the SQL table
querySQL()
.then(function (result) {
console.log('Print results SQL');
console.log(result);
if(result.length > 0){
processArray(db, result)
.then(function (result) {
console.log('Res');
console.log(result);
})
.catch(function (err) {
console.log('Err');
console.log(err);
})
} else {
console.log('Nothing to show in MySQL');
}
})
.catch(function (err) {
console.log(err);
});
db.close(); // <--------------------------------THIS LINE
});
Just a minor addition to Gaafar's answer, it gave me a deprecation warning. Instead of on the server object, like this:
MongoClient.connect(MONGO_URL, {
server: {
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 1000
}
});
It can go on the top level object. Basically, just take it out of the server object and put it in the options object like this:
MongoClient.connect(MONGO_URL, {
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 1000
});
"Topology was destroyed" might be caused by mongoose disconnecting before mongo document indexes are created, per this comment
In order to make sure all models have their indexes built before disconnecting, you can:
await Promise.all(mongoose.modelNames().map(model => mongoose.model(model).ensureIndexes()));
await mongoose.disconnect();
I met this in kubernetes/minikube + nodejs + mongoose environment.
The problem was that DNS service was up with a kind of latency. Checking DNS is ready solved my problem.
const dns = require('dns');
var dnsTimer = setInterval(() => {
dns.lookup('mongo-0.mongo', (err, address, family) => {
if (err) {
console.log('DNS LOOKUP ERR', err.code ? err.code : err);
} else {
console.log('DNS LOOKUP: %j family: IPv%s', address, family);
clearTimeout(dnsTimer);
mongoose.connect(mongoURL, db_options);
}
});
}, 3000);
var db = mongoose.connection;
var db_options = {
autoReconnect:true,
poolSize: 20,
socketTimeoutMS: 480000,
keepAlive: 300000,
keepAliveInitialDelay : 300000,
connectTimeoutMS: 30000,
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 1000,
useNewUrlParser: true
};
(the numbers in db_options are arbitrary found on stackoverflow and similar like sites)
I alse had the same error. Finally, I found that I have some error on my code. I use load balance for two nodejs server, but I just update the code of one server.
I change my mongod server from standalone to replication, but I forget to do the corresponding update for the connection string, so I met this error.
standalone connection string:
mongodb://server-1:27017/mydb
replication connection string:
mongodb://server-1:27017,server-2:27017,server-3:27017/mydb?replicaSet=myReplSet
details here:[mongo doc for connection string]
Sebastian comment on Adrien's answer needs more attention it helped me but it being a comment might be ignore sometime so here's a solution:
var options = { useMongoClient: true, keepAlive: 1, connectTimeoutMS: 30000, reconnectTries: 30, reconnectInterval: 5000 }
mongoose.connect(config.mongoConnectionString, options, (err) => {
if(err) {
console.error("Error while connecting", err);
}
});
Here what I did, It works fine. Issue was gone after adding below options.
const dbUrl = "mongodb://localhost:27017/sampledb";
const options = { useMongoClient: true, keepAlive: 1, connectTimeoutMS: 30000, reconnectTries: 30, reconnectInterval: 5000, useNewUrlParser: true }
mongoose.connect(dbUrl,options, function(
error
) {
if (error) {
console.log("mongoerror", error);
} else {
console.log("connected");
}
});
You need to restart mongo to solve the topology error, then just change some options of mongoose or mongoclient to overcome this problem:
var mongoOptions = {
useMongoClient: true,
keepAlive: 1,
connectTimeoutMS: 30000,
reconnectTries: Number.MAX_VALUE,
reconnectInterval: 5000,
useNewUrlParser: true
}
mongoose.connect(mongoDevString,mongoOptions);
I got this error, while I was creating a new database on my MongoDb Compass Community. The issue was with my Mongod, it was not running. So as a fix, I had to run the Mongod command as preceding.
C:\Program Files\MongoDB\Server\3.6\bin>mongod
I was able to create a database after running that command.
Hope it helps.
I was struggling with this for some time - As you can see from other answers, the issue can be very different.
The easiest way to find out whats causing is this is to turn on loggerLevel: 'info' in the options
In my case, this error was caused by an identical server instance already running background.
The weird thing is when I started my server without notice there's one running already, the console didn't show anything like 'something is using port xxx'. I could even upload something to the server. So, it took me quite long to locate this problem.
What's more, after closing all the applications I can imagine, I still could not find the process which is using this port in my Mac's activity monitor. I have to use lsof to trace. The culprit was not surprising - it's a node process. However, with the PID shown in the terminal, I found the port number in the monitor is different from the one used by my server.
All in all, kill all the node processes may solve this problem directly.
Using mongoose here, but you could do a similar check without it
export async function clearDatabase() {
if (mongoose.connection.readyState === mongoose.connection.states.disconnected) {
return Promise.resolve()
}
return mongoose.connection.db.dropDatabase()
}
My use case was just tests throwing errors, so if we've disconnected, I don't run operations.
I got this problem recently. Here what I do:
Restart MongoDb: sudo service mongod restart
Restart My NodeJS APP. I use pm2 to handle this pm2 restart [your-app-id]. To get ID use pm2 list
var mongoOptions = {
useNewUrlParser: true,
useUnifiedTopology: true,
}
mongoose.connect(mongoDevString,mongoOptions);
I solved this issue by:
ensuring mongo is running
restarting my server

MongoDB auto_reconnect doesn't work

I'm using mongoskin as a wrapper to the native mongodb driver, auto_reconnect doesn't seem to work.
I'm creating the database:
var db = mongo.db(serverUrl, {
database: database,
auto_reconnect: true,
safe: true
});
I added a query on connection close to test it:
db.on('close', function(err) {
setTimeout(function() {
db.collection('users').findOne({short_id: '123'}, console.log);
}, 5000)
});
immediately after running the node process i'm killing and starting mongod, I'm getting '[Error: no open connections]' as the query result.
doesn't it suppose to auto_reconnect on connection lose ? or its trying to auto_reconnect immediately on connection close but mongod isn't up yet ?
can I reconnect manually or do I need to create a new server ?
Make sure you are using the latest node-mongodb-native (1.2.11) driver and it should work as you expect it to.

Resources