I am facing the performance issue during load testing of the Rest API for Oracle Database developed by using Express and OracleDB nodejs modules.
I see the decrease in performance during load testing during increasing number of the requests per seconds to the developed API.This issue is reproduced on stored procedure and standard select requests to the database.
From database side I see the standard and stable response time for each of the requests. Requests are recieved by the database in the same time as they were initiated.
From application(reponse) side it looks like responses being puted in some kind queue and being readed by nodejs(OracleDB module) in packages.
With a standard response time in 0,5 seconds per requests, I might recieve 3-5 seconds response time on 10 request per second.
This issues is reproduced on different connection pool sizes(grater than number of requests per seconds).
At the moment I am stuck with possible options that might lead to the issue. What might be the reason for such behaviour? Or what options for diagnostic or turning is available for oracleDB module for the nodejs?
some code bellow:
creating the connection:
const init = async () => {
try {
Logger.info(`oracle instant client address ${process.env.LD_LIBRARY_PATH}`);
const pool = await oracledb.createPool(dbCredentials);
Logger.info('db connections pool created');
return pool;
} catch (err) {
Logger.error(`init() error: ${err.message}`);
}
};
let pool = await init();
route:
router.get('/test', async (req, res, next) => {
try {
const result = await testController(pool);
sendResponse(res, 200, result);
} catch (e) {
sendErrResponse(res, 500, e.message);
}
});
controller:
const testController = async (pool) => {
let connection;
try {
connection = await pool.getConnection();
} catch (error) {
return error;
}
try {
let items = { items: [{ itemSKU: 'xxxxx', itemQTY: 1234 }, { itemSKU: 'yyyyy', itemQTY: 123}] };
items = JSON.stringify(items);
const { rows: data } = await connection.execute(
'select shop_id, prod_id, qnt from TABLE(pkg_ecom.check(:items))',
{ items },
);
return data;
} catch (error) {
return error;
} finally {
if (connection) {
try {
await connection.close();
} catch (err) {
console.error(err);
}
}
}
};
The common problem with scaling node-oracledb connection load is Node.js thread starvation. Increase the value of the environment variable UV_THREADPOOL_SIZE before your application starts. See the documentation Connections, Threads, and Parallelism. On Linux, your package.json file might have:
"scripts": {
"start": "export UV_THREADPOOL_SIZE=10 && node index.js"
},
. . .
You can monitor pool usage by setting the enableStatistics attribute during pool creation and then calling pool.getStatistics() or pool.logStatistics(), see Connection Pool Monitoring. Look out for too many getConnection() requests being queued.
You should also read the node-oracledb case study Always Use Connection Pools — and How.
Related
First, i make an API using nodejs and oracledb.
I have 2 routes with different response time, let say route A with 10s response time and route B 1s response time. When i execute the route A followed by route B , i got the error NJS-003: invalid connection because route B finish and close the connection followed by route A.
Any ideas how to solve this problem?
I'm using oracle pool , getConnection and close connection every API request.
async function DBGetData(req, res, query, params = {}) {
try {
connection = await oracledb.getConnection();
connection.callTimeout = 10 * 1000;
result = await connection.execute(
query,
params,
{
outFormat: oracledb.OUT_FORMAT_OBJECT,
}
);
// send query result
res.json({
status: res.statusCode,
length: result.rows.length,
results: result.rows,
});
} catch (err) {
return res.status(400).json({ error: err.toString() });
} finally {
if (connection) {
// Always close connections
await connection.close();
}
}
}
Add a let connection; before the try so that each DBGetData() invocation is definitely using its own connection. Currently it seems that you are referencing a global variable.
I have this database connection. Inside the function where the comment is located, there is a data update cycle for rest api. The data is updated, but when the data in the Oracle database is updated, the connection may fail and after that all subsequent updated data will get undefined. How can you properly connect to the database so that there are no failures?
oracledb.getConnection(
{
user: db.user,
password: db.password,
connectString: db.connectString
},
connExecute
);
function connExecute(err, connection) {
if (err) {
console.error(err.message);
return;
}
sql = `SELECT * FROM db.test`;
connection.execute(sql, {}, { outFormat: oracledb.OBJECT },
function (err, db) {
if (err) {
console.error(err.message);
connRelease(connection);
return;
}
// data update loop
connRelease(connection);
});
}
function connRelease(connection) {
connection.close(
function (err) {
if (err) {
console.error(err.message);
}
});
}
You should be using a connection pool. Connection pools have built-in logic to detect connections with issues and create new connections transparently. See this series on creating a REST API for more details: https://jsao.io/2018/03/creating-a-rest-api-with-node-js-and-oracle-database/
Keep in mind that issues can still happen, so you have to handle errors as needed for your application.
Mostly you add listener on connection object and on dissociation or failure again create connection. With minor changes you can adopt this approach and use listeners to check if connection is available if not connect again. There could be several reason that results in connection closing better handle exceptions, check if still connected and reconnect in case of error.
Or you can try this NPM this will do reconnection for you
https://www.npmjs.com/package/oracledb-autoreconnect
Ping me if you need calcification.
var dbConfig = {
host: '----',
user: '----',
password: '----',
database: '----',
port: ----
};
var connection;
function handleDisconnect() {
connection = <obj>.getConnection(dbConfig);
// Recreate the connection, since the old one cannot be reused.
connection.connect( function onConnect(err) {
// The server is either down
if (err) {
// or restarting (takes a while sometimes).
console.log('error when connecting to db:', err);
setTimeout(handleDisconnect, 10000);
// We introduce a delay before attempting to reconnect,
}
// to avoid a hot loop, and to allow our node script to
});
// process asynchronous requests in the meantime.
// If you're also serving http, display a 503 error.
connection.on('error', function onError(err) {
console.log('db error', err);
if (err.code == 'PROTOCOL_CONNECTION_LOST') {
handleDisconnect();
// lost due to either server restart, or a
} else {
// connnection idle timeout (the wait_timeout
throw err;
// server variable configures this)
}
});
}
handleDisconnect();
The project in question is using 11 PouchDBs. To ensure syncing, all 11 DBs were instantiated (with sync) when the Angular 5 application was loaded/bootstrapped. Since sync did not function (due to the limitations set by browsers) we moved towards socket-pouch as a solution. We disabled sync on all DBs and incorporated socket-pouch sync to only one DB. The socketPouchServer runs on localhost:5000 & the CouchDB is hosted on DigitalOcean.
On running the system,
the following logs are observed in the browser. As you can see, an "aborting" error is being logged.
https://user-images.githubusercontent.com/26055473/38247973-8eef489a-3764-11e8-8411-6b5b20436d19.png
the code for the same is
import PouchDB from 'pouchdb';
import PouchDBFind from 'pouchdb-find';
import SocketPouchClient from 'socket-pouch/client';
PouchDB.plugin(PouchDBFind);
PouchDB.adapter('socket', SocketPouchClient);
PouchDB.debug.enable('pouchdb:socket:*');
this.dailyMovementDB = new
PouchDB(`${username}_${environment.REQUIRED_DB_VERSION_NUMBER}_daily-movement`, { auto_compaction: true });
this.dailyMovementDBRemote = new PouchDB(
{
adapter: 'socket',
name: `${username}_${environment.REQUIRED_DB_VERSION_NUMBER}_daily-movement`,
url: `${environment.REMOTE_COUCH_DB_BASE_URL}`
});
var syncHandler = this.dailyMovementDB.replicate.to(this.dailyMovementDBRemote
).on('change', function (change) {
// yo, something changed!
console.log('yo, something changed', change);
instance.autosaveMessageService.syncingProcessEndedSubject.next(false);
}).on('paused', function (info) {
console.log('replication was paused, usually because of a lost connection', info);
// replication was paused, usually because of a lost connection
instance.autosaveMessageService.syncingProcessEndedSubject.next(true);
}).on('active', function (info) {
// replication was resumed
console.log('replication was resumed', info);
instance.autosaveMessageService.syncingProcessStartedSubject.next();
}).on('denied', function (info) {
// handle complete
console.log('denied', info);
instance.autosaveMessageService.syncingProcessEndedSubject.next(false);
}).on('complete', function (info) {
// handle complete
console.log('handle complete', info);
instance.autosaveMessageService.syncingProcessEndedSubject.next(false);
}).on('error', function (err) {
console.log(err);
// instance.createDailyMovementPouchDBs(username);
// totally unhandled error (shouldn't happen)
});
And the following logs appear on localhost:5000 (socketPouchServer)
https://user-images.githubusercontent.com/26055473/38248011-ac49b9ca-3764-11e8-8501-74576c8c1e1f.png
the following is the code for the socketPouchServer
var socketPouchServer = require('socket-pouch/server');
const PouchDB = require('pouchdb');
socketPouchServer.listen(5000, {
remoteUrl: 'http://remoteurl:5984',
}, () => {
console.log('Hi');
});
Please guide how to resolve this issue.
First of all, this is one of my first projects in Node.js so I'm very new to it.
I have a project I want to make that is a SOAP (I know, SOAP... backwards compatibility, huh?) interface that connects to an Oracle database.
So I have a WSDL describing what these functions look like (validation for addresses and stuff) and I have a connection to the database.
Now when using the SOAP npm module, you need to create a server and listen using a service that allows you to respond to requests. I have a separate file that contains my SOAP service but this service should do queries on the database to get its results.
How would I go about sort of 'injecting' my database service into my SOAP service so that whenever a SOAP call is done, it orchestrates this to the correct method in my database service?
This is what my code looks like:
databaseconnection.js
var oracledb = require('oracledb');
var dbConfig = require('../../config/development');
var setup = exports.setup = (callback) => {
oracledb.createPool (
{
user : dbConfig.user,
password : dbConfig.password,
connectString : dbConfig.connectString
},
function(err, pool)
{
if (err) { console.error(err.message); return; }
pool.getConnection (
function(err, connection)
{
if (err) {
console.error(err.message);
return callback(null);
}
return callback(connection);
}
);
}
);
};
databaseservice.js
var DatabaseService = function (connection) {
this.database = connection;
};
function doSomething(callback) {
if (!this.database) { console.log('Database not available.'); return; }
this.database.execute('SELECT * FROM HELP', function(err, result) {
callback(result);
});
};
module.exports = {
DatabaseService: DatabaseService,
doSomething: doSomething
};
soapservice.js
var myService = {
CVService: {
CVServicePort: {
countryvalidation: function (args, cb, soapHeader) {
console.log('Validating Country');
cb({
name: args
});
}
}
}
};
server.js
app.use(bodyParser.raw({type: function(){return true;}, limit: '5mb'}));
app.listen(8001, function(){
databaseconnection.setup((callback) => {
var temp = databaseservice.DatabaseService(callback);
soapservice.Init(temp);
var server = soap.listen(app, '/soapapi/*', soapservice.myService, xml);
databaseservice.doSomething((result) => {
console.log(result.rows.length, ' results.');
});
});
console.log('Server started');
});
How would I go about adding the databaseservice.doSomething() to the countryvalidation soap method instead of 'name: args'?
Also: I feel like the structure of my code is very, very messy. I tried finding some good examples on how to structure the code online but as for services and database connections + combining them, I didn't find much. Any comments on this structure are very welcome. I'm here to learn, after all.
Thank you
Dieter
The first thing I see that looks a little off is the databaseconnection.js. It should be creating the pool, but that's it. Generally speaking, a connection should be obtained from the pool when a request comes in and release when you're done using it to service that request.
Have a look at this post: https://jsao.io/2015/02/real-time-data-with-node-js-socket-io-and-oracle-database/ There are some sample apps you could have a look at that might help. Between the two demos, the "employees-cqn-demo" app is better organized.
Keep in mind that the post is a little dated now, we've made enhancements to the driver that make it easier to use now. It's on my list to do a post on how to build a RESTful API with Node.js and Oracle Database but I haven't had a chance to do it yet.
I want my application (lets say a simple node file for now) to work as it is even if redis is not available. I'm not able to do it the correct way. This is what I've tried.
var redis = require('redis');
var redisClient = null;
var getRedisClient = function(){
if(redisClient){
return redisClient;
}
try {
redisClient = redis.createClient({connect_timeout : 5000, max_attempts : 1});
redisClient.on("error", function(err) {
console.error("Error connecting to redis", err);
redisClient = null;
});
return redisClient;
} catch(ex){
console.log("error initialising redis client " + ex);
return null;
}
};
try {
var client = getRedisClient();
console.log("done!");
} catch (ex){
console.log("Exception");
}
However, with this code my application exits if redis is not available (it shouldn't because i've not given a process.exit() command).
How can I solve this?
Checking for Successful Connection on Start
Using a promise, you could guarantee that at least initially, you were able to connect to redis without error within a specified time period:
const redis = require('redis');
const Promise = require('bluebird');
function getRedisClient(timeoutMs){
return new Promise((resolve, reject) => {
const redisClient = redis.createClient();
const timer = setTimeout(() => reject('timeout'), timeoutMs);
redisClient.on("ready", () => {
clearTimeout(timer);
resolve(redisClient);
});
redisClient.on("error", (err) => {
clearTimeout(timer);
reject(err);
});
});
};
const redisReadyTimeoutMs = 10000;
getRedisClient(redisReadyTimeoutMs)
.then(redisClient => {
// the client has connected to redis sucessfully
return doSomethingUseful();
}, error => {
console.log("Unable to connect to redis", error);
});
You Need Proper Error Handling
The redis client being non-null does NOT guarantee using it won't throw an error.
you could experience infrastructure misfortune e.g. crashed redis process, out of memory or network being down.
a bug in your code could cause an error e.g. invalid or missing arguments to a redis command.
You should be handling redis client errors as a matter of course.
DON'T null the Redis Client on Error
It won't give you much but it will force you to check for null every time you try and use it.
The redis client also has inbuilt reconnect and retry mechanisms that you'll miss out on if you null it after the first error. See the redis package docs, look for retry_strategy.
DO Wrap your redis client code with try .. catch ... or use .catch in your promise chain.
DO Make use of a retry_strategy.