The basic idea of the following code is I read messages off an ActiveMQ Artemis installation and insert them into a MongoDB instance.
It works well for up to a hundred or so messages per second but crashes if I throw a few thousand at it. My first guess would be the constant opening and closing of database connections. Should I also think about using an in-memory store and doing bulk database inserts?
The code is all running in node using the mqtt and mongodb npm packages. The code below, the database and the queue are all running in docker containers if it makes any difference.
var mqtt = require('mqtt'),
client = mqtt.connect('mqtt://mq:1883', {
username: "*************",
password: "*************"
}),
MongoClient = require('mongodb').MongoClient,
ObjectId = require('mongodb').ObjectID,
assert = require('assert'),
url = 'mongodb://db:27017/uo-readings';
client.on('connect', function () {
client.subscribe('readings');
});
client.on('error', function(error){
console.log(error)
});
client.on('message', function (topic, message) {
console.log(message.toString());
MongoClient.connect(url, function(err, db) {
assert.equal(null, err);
console.log("Connected correctly to server.");
db.collection('readings').insertOne(JSON.parse(message.toString()), function(err, result) {
assert.equal(err, null);
console.log("Inserted a document into the readings collection.");
});
client.end(function(){
console.log("Closing Connection.");
db.close();
});
});
});
See #Jonathan Muller's comment above
Related
What is the recommended way to establish database connection and close it properly in Node.js using tedious?
For each and every request we are creating new connection and processing the request then closing them in callback.
app.get('/getData/:id', function(req, res){
var id = req.params.id;
var sqlGet = "exec MyStoreProcedure #Id='" + id + "'";
var connection = new Connection(config);
var request = new Request(sqlGet, function(err, result){
connection.close();
if(err)
console.log(err);
else
res.send(result);
});
connection.on('connect', function(err) {
if (err)
{
console.log(err)
}else{
console.log("Connected");
connection.execSql(request);
}
});
});
Is there any other recommended approach to handle this scenario?
UPDATE (Oct 19, 2020):
It appears that tedious-connection-pool is no longer supported/outdated. I've migrated my code to mssql: https://www.npmjs.com/package/mssql
Previous Answer:
You should check out tedious-connection-pool: https://github.com/tediousjs/tedious-connection-pool.
This makes it easy to manage and reuse connections rather than open/close connections continuously.
As part of using connnection pooling, you should extract it out into a separate file so it can be reused across your application.
I have this database connection. Inside the function where the comment is located, there is a data update cycle for rest api. The data is updated, but when the data in the Oracle database is updated, the connection may fail and after that all subsequent updated data will get undefined. How can you properly connect to the database so that there are no failures?
oracledb.getConnection(
{
user: db.user,
password: db.password,
connectString: db.connectString
},
connExecute
);
function connExecute(err, connection) {
if (err) {
console.error(err.message);
return;
}
sql = `SELECT * FROM db.test`;
connection.execute(sql, {}, { outFormat: oracledb.OBJECT },
function (err, db) {
if (err) {
console.error(err.message);
connRelease(connection);
return;
}
// data update loop
connRelease(connection);
});
}
function connRelease(connection) {
connection.close(
function (err) {
if (err) {
console.error(err.message);
}
});
}
You should be using a connection pool. Connection pools have built-in logic to detect connections with issues and create new connections transparently. See this series on creating a REST API for more details: https://jsao.io/2018/03/creating-a-rest-api-with-node-js-and-oracle-database/
Keep in mind that issues can still happen, so you have to handle errors as needed for your application.
Mostly you add listener on connection object and on dissociation or failure again create connection. With minor changes you can adopt this approach and use listeners to check if connection is available if not connect again. There could be several reason that results in connection closing better handle exceptions, check if still connected and reconnect in case of error.
Or you can try this NPM this will do reconnection for you
https://www.npmjs.com/package/oracledb-autoreconnect
Ping me if you need calcification.
var dbConfig = {
host: '----',
user: '----',
password: '----',
database: '----',
port: ----
};
var connection;
function handleDisconnect() {
connection = <obj>.getConnection(dbConfig);
// Recreate the connection, since the old one cannot be reused.
connection.connect( function onConnect(err) {
// The server is either down
if (err) {
// or restarting (takes a while sometimes).
console.log('error when connecting to db:', err);
setTimeout(handleDisconnect, 10000);
// We introduce a delay before attempting to reconnect,
}
// to avoid a hot loop, and to allow our node script to
});
// process asynchronous requests in the meantime.
// If you're also serving http, display a 503 error.
connection.on('error', function onError(err) {
console.log('db error', err);
if (err.code == 'PROTOCOL_CONNECTION_LOST') {
handleDisconnect();
// lost due to either server restart, or a
} else {
// connnection idle timeout (the wait_timeout
throw err;
// server variable configures this)
}
});
}
handleDisconnect();
So i am very new to mongodb and i wish to use it in my application. Now i HATE redundant code but reading up on how to use mongodb with node.js it seems that there is a pattern where you always have to connect before making any CRUD operation.
Example from the offical documentation:
MongoClient.connect(url, function(err, db) {
assert.equal(null, err);
insertDocument(db, function() {
db.close();
});
});
My question is. is it possible to make a middleware that keeps the connection open so you only has to call insertDocument (in the above example) ?
Yea of course, just keep the db variable around until you don't need it any longer - then call close()
var mdb;
MongoClient.connect(url, function(err, db) {
assert.equal(null, err);
mdb = db;
insertDocument(mdb, function() {
// ...
});
});
// do other stuff with mdb
You can also look into using Mongoose as you mentioned middleware.
The connection is only opened once (on the global scope) and then you can use it throughout the app.
I'm trying to implement a call back on NodeJS EC2 server that's interacting with AWS RDS Postgresql. I'm not quite sure how it's done. There seems to be a EventEmitter method within AWS-SDK's RDS module. It's designed for all RDS instance types like MySQL, Aurora, etc. Not specifically for postgres. All I'm trying to do is to get some kind of callback after an INSERT or DELETE query.
It is not specific if your postgres is RDS or standalone on EC2.
You will need
var pg = require('pg');
var dbe={"result":null};
function Q(sqlQuery,callback) {
/* async, vulnerable, simple */
var conString = "postgres://"+dbUser+":"+dbPass+"#"+dbHost+":"+dbPort+"/"+dbName+"?ssl=true";
pg.connect(conString, function(err, client, done) {
if(err) {
return console.error('error fetching client from pool', err);
}
client.query(sqlQuery, function(err, result) {
done();//call `done()` to release the client back to the pool
if(err) {
return console.error('error running query', err);
}
dbe.result = result;
//console.log(JSON.parse(result.setEncoding('utf8');));
callback();
});
});
}
And calling
var res = Q('select now()', function(a) {console.log(dbe.result)});
or similar - I don't have a playground to test atm
I've been looking for a way to do various operations on a mongo database, depending on which route a user connects to on my website. So doing a html-post to www.mysite.com/data would add info to a mongo DB, and doing a html-get at the same url would get data from the same database. I managed to solve that, but everytime I turn on my server with the website I get 5 connections registred at the mongo database. Why is this, and is it bad?
My code:
I'm runing this code in mongo.js:
var mongodb = require('mongodb');
module.exports.init = function (callback) {
var server = new mongodb.Server("127.0.0.1", 27017, {});
new mongodb.Db('test', server, {w: 1}).open(function (error, client) {
//export the client and maybe some collections as a shortcut
module.exports.client = client;
module.exports.myCollection = new mongodb.Collection(client, 'myCollection');
callback(error);
});
};
I initialize everything running (app.js):
/express set-up/
var mongo = require('./mongo.js');
/.../
mongo.init(function (error) {
if (error)
throw error;
app.listen(3000, function(){
console.log("Express server listening on port %d in %s mode",3000,
app.settings.env);
});
});
And, for example, the post looks like (still app.js):
app.post('/App', function(req, res) {
users = users.concat(req.body);
res.redirect('/App');
//Add user to MongoDB
mongo.myCollection.insert({name: req.body.name, car: req.body.car, website: req.body.website, phone: req.body.phone}, {safe:true}, function(err, objects) {
if (err)
console.warn(err.message);
});
Pretty sure my redirect isn't working as I want it too here, but that's another issue.
Anny suggestions on why I get five connects every time I start the server?
The five connections are because that is the default poolSize (5). You can adjust it, as outlined in the server options docs - it also mentions the default.