In the Nodejs app, I am using 'ioredis' npm module to execute all redis operations. Redis is used heavily in the whole service. But I have seen for 1 of 1000 call, redis latency is increased to 400ms.
I am trying to log all redis operations exceeding a certain timestamp in the code itself. Suggest some ways to log such commands, without slowing any other operation.
Thanks.
You can try this :
var Moment = require('moment');
var start_time = Moment();
//->>> In case of Promises
redisOperation(arguments)
.then(function() {
var time_taken = Moment() - start_time;
console.log(time_taken); //However you want to log, use the logger function here
});
//->>> In case of Callback Function
redisOperation(arguments, function() {
var time_taken = Moment() - start_time;
console.log(time_taken); //However you want to log, use the logger function here
cb();
});
For time being, I wasn't able to find any inbuilt solution. But I handled this by creating a wrapper before the execution of Redis operations.
const traceTime = function (fn) {
return async function (...args) {
const startTime = Date.now();
const executionResp = await fn.apply(this, args);
const endTime = Date.now();
};
}
/* wrap the redis operations with traceTime */
set = traceTime(set); // redis set operation
get = traceTime(get); // redis get operation
/* use redis set operation, wherever you want to use,
it will log startTime and endTime and execute redis operation from there */
set(key, value, (err, response) => {
// err and response handler
});
Related
This is probably pretty easy but I've been unable to piece it together properly.
I'm trying to use the Sequelize NPM to call a stored procedure that I built and then I want to trigger it with a GET request from from an express api and return the output of the procedure to the api.
Here is what my code looks like for the Sequelize portion....
// Testing stored procedure //
const Retrieve = (testName) => connection.testdata_connection.query("EXEC [SPROC] [INPUTS]")
module.exports = {
tests: Tests(),
retrieve: Retrieve()
};
This part "connection.testdata_connection" is just establishing my connection to the database and I have tested this and I know this part is set.
I would like to be able to hit that with something like...
const query = require('./database/queries'); ///Imports sequelize queries
const app = express();
app.get('/decrypt', function(req,res){
query.retrieve()
})
})
This doesn't work at all.
Now if I do something like this in the queries file...
const Retrieve = async function() {
const decrypt = await connection.testdata_connection.query("EXEC [SPROC] [INPUT]")
console.log(decrypt)
}
module.exports = {
tests: Tests(),
retrieve: Retrieve()
};
This will log to my console with correct data when I start the server. I want it to do that when I hit it with my endpoint.
First, your function should be exported but not executed:
// creating an async function (all i/o operations should be async).
const Retrieve = async(testName) => connection.testdata_connection.query("EXEC [SPROC] [INPUTS]")
module.exports = {
retrieve: Retrieve,
// retrieve: Retrieve() if you call the function with (), the function will be executed and we don't want that yet
};
Now we can call it in the route:
const query = require('./database/queries'); ///Imports sequelize queries
const app = express();
// the route is async because it is executing async code
app.get('/decrypt', async (req,res) => {
// waiting for the answer with await
const response = await query.retrieve();
// Doing something with the response
})
You still need to check for errors, but that is the basics.
I'm trying to call getMarchandiseList() method that call a mongoose method " find({}) " in my marchandise router.
the problem is when I send the responce I find that I get a result=0 before the find({}) method send it's result
//the result should be
{id_marchandise : 01 , libelle_marchandise : 'fer'}.
[MarchandiseDAO.js][1]
*
var express = require("express");
var router = express.Router();
var marchandiseDAO = require ('../DAO/marchandises');
router.get('/listmarchandise', function (req,res) {
var marchandise = new marchandiseDAO();
marchandise.getMarchandiseList();
res.send("result" +marchandise.result);
});
module.exports = router;
*
[marchandiserouter.js][2]
var model = require("../Models/model");
var marchandiseDAO = function () {
console.log("get instance ");
this.result = 0 ;
}
marchandiseDAO.prototype.getMarchandiseList = function () {
console.log("begin");
model.MarchandiseModel.find({},function(err,marchandiselist){
console.log("traitement");
if(err) result = err;
else result = marchandiselist;
});
console.log("end");
}
You cannot run mongoose methods synchronously. But if you use a modern version of Node then you can make it appear as if it was run asynchronously.
Unfortunately you posted your code examples as screenshots which would require me to use Photoshop to fix your code, which is obviously not worth the hassle. So instead I will show you a general solution.
Using normal promises:
Model.find({...}).then(data => {
// you can only use data here
}).catch(err => {
// always handle errors
});
Using await:
try {
let data = await Model.find({...});
// you can only use data here
} catch (err) {
// always handle errors
}
The second example can only be used in an async function. See this for more info:
try/catch blocks with async/await
Use await outside async
Using acyns/await in Node 6 with Babel
When do async methods throw and how do you catch them?
using promises in node.js to create and compare two arrays
Keeping Promise Chains Readable
function will return null from javascript post/get
I am testing out aws lambda, using nodejs with the 4.3 version. I'm able to successfully complete all the statements in my handler function within the console test, which includes connecting to a mongodb host within our vpc. But, the function always times out. I've found several posts and resources that discuss using the callback, and setting properties on the context, and IAM role permissions, but no matter what I do, it always ends up timing out. Current code:
'use strict';
var Mongoose = require('mongoose');
var Device = require('./device_model');
var Alarm = require('./alarm_model');
var Event = require('./event_model');
var mongoConnection = process.env.MONGO_URL;
var connection = Mongoose.connect(mongoConnection);
Mongoose.connection.once('open', function() {
console.log("Connecting to mongo at: " + mongoConnection);
console.log("Mongoose connection in lambda opened");
});
Mongoose.connection.on('error', function(){
console.error("Error creating mongoose connection in lambda, exiting!");
process.exit(1);
});
exports.check_alarms = function(event, context, callback) {
context.callbackWaitsForEmtpyEventLoop = false;
console.log("The incoming event: " + JSON.stringify(event));
var device = null;
Device.findByUUID(event.uuid, function(error, result){
if(!error){
device = result;
console.log("the device: " + JSON.stringify(device));
if(event.Ale && event.Ale.length > 0) {
console.log("We have an alarm, checking if already set");
callback(null, {"status":"alarms"});
} else {
console.log("Event contains no alarm; checking for historic active");
callback(null, {"status":"no alarms"});
}
} else {
console.log("there's a problem on mongo");
callback("problem", "status not so good");
}
});
callback(null, {"status":"outside of device find block"});
}
You have a typo:
context.callbackWaitsForEmtpyEventLoop = false;
should be:
context.callbackWaitsForEmptyEventLoop = false;
Here's what the documentation says about the behavior of callbackWaitsForEmptyEventLoop:
callbackWaitsForEmptyEventLoop
The default value is true. This property is useful only to modify the default behavior of the callback. By default, the callback will wait until the Node.js runtime event loop is empty before freezing the process and returning the results to the caller. You can set this property to false to request AWS Lambda to freeze the process soon after the callback is called, even if there are events in the event loop. AWS Lambda will freeze the process, any state data and the events in the Node.js event loop (any remaining events in the event loop processed when the Lambda function is called next and if AWS Lambda chooses to use the frozen process). For more information about callback, see Using the Callback Parameter.
Minimal example:
// Times out due to typo
exports.function1 = (event, context, callback) => {
setInterval(() => console.log('Long wait'), 100000);
context.callbackWaitsForEmtpyEventLoop = false;
callback(null, 'Hello from Lambda');
};
// Returns successfully
exports.function2 = (event, context, callback) => {
setInterval(() => console.log('Long wait'), 100000);
context.callbackWaitsForEmptyEventLoop = false;
callback(null, 'Hello from Lambda');
};
If anybody was confused like I was with how to add callbackWaitsForEmptyEventLoop to new Alexa projects that look like this:
const skillBuilder = Alexa.SkillBuilders.custom();
exports.handler = skillBuilder
.addRequestHandlers(
GetNewFactHandler,
HelpHandler,
ExitHandler,
FallbackHandler,
SessionEndedRequestHandler,
)
.addRequestInterceptors(LocalizationInterceptor)
.addErrorHandlers(ErrorHandler)
.lambda();
Example project found here: https://github.com/alexa/skill-sample-nodejs-fact/blob/master/lambda/custom/index.js
This solution worked for me:
// create a custom skill builder
const skillBuilder = Alexa.SkillBuilders.custom();
exports.handler = (event, context, callback) => {
// we need this so that async stuff will work better
context.callbackWaitsForEmptyEventLoop = false
// set up the skill with the new context
return skillBuilder
.addRequestHandlers(
GetNewFactHandler,
HelpHandler,
ExitHandler,
FallbackHandler,
SessionEndedRequestHandler,
)
.addRequestInterceptors(LocalizationInterceptor)
.addErrorHandlers(ErrorHandler)
.lambda()(event, context, callback);
}
I was triggering a lambda from android code. The lambda was performing a heavy data lifting operation from DynamoDB and generating a JS object as a response. I could see the generated object in my lambda logs, but my android side code always timed out waiting for a response.
The reason for timeout was the Invocation payload (request and response) limit of 6 MB for synchronous and 256 KB for asynchronous lambda implementation. In my case I had a asynchronous implementation and the resultant object was over 256 KB.
There was no appropriate log message for overshooting this limit which derailed me. Reference: https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-limits.html
The workaround I implemented was writing the response to a text file in S3 bucket and passing its reference as a response to the android side instead of passing the actual object itself.
I need to build a function for processing large CSV files for use in a bluebird.map() call. Given the potential sizes of the file, I'd like to use streaming.
This function should accept a stream (a CSV file) and a function (that processes the chunks from the stream) and return a promise when the file is read to end (resolved) or errors (rejected).
So, I start with:
'use strict';
var _ = require('lodash');
var promise = require('bluebird');
var csv = require('csv');
var stream = require('stream');
var pgp = require('pg-promise')({promiseLib: promise});
api.parsers.processCsvStream = function(passedStream, processor) {
var parser = csv.parse(passedStream, {trim: true});
passedStream.pipe(parser);
// use readable or data event?
parser.on('readable', function() {
// call processor, which may be async
// how do I throttle the amount of promises generated
});
var db = pgp(api.config.mailroom.fileMakerDbConfig);
return new Promise(function(resolve, reject) {
parser.on('end', resolve);
parser.on('error', reject);
});
}
Now, I have two inter-related issues:
I need to throttle the actual amount of data being processed, so as to not create memory pressures.
The function passed as the processor param is going to often be async, such as saving the contents of the file to the db via a library that is promise-based (right now: pg-promise). As such, it will create a promise in memory and move on, repeatedly.
The pg-promise library has functions to manage this, like page(), but I'm not able to wrap my ahead around how to mix stream event handlers with these promise methods. Right now, I return a promise in the handler for readable section after each read(), which means I create a huge amount of promised database operations and eventually fault out because I hit a process memory limit.
Does anyone have a working example of this that I can use as a jumping point?
UPDATE: Probably more than one way to skin the cat, but this works:
'use strict';
var _ = require('lodash');
var promise = require('bluebird');
var csv = require('csv');
var stream = require('stream');
var pgp = require('pg-promise')({promiseLib: promise});
api.parsers.processCsvStream = function(passedStream, processor) {
// some checks trimmed out for example
var db = pgp(api.config.mailroom.fileMakerDbConfig);
var parser = csv.parse(passedStream, {trim: true});
passedStream.pipe(parser);
var readDataFromStream = function(index, data, delay) {
var records = [];
var record;
do {
record = parser.read();
if(record != null)
records.push(record);
} while(record != null && (records.length < api.config.mailroom.fileParserConcurrency))
parser.pause();
if(records.length)
return records;
};
var processData = function(index, data, delay) {
console.log('processData(' + index + ') > data: ', data);
parser.resume();
};
parser.on('readable', function() {
db.task(function(tsk) {
this.page(readDataFromStream, processData);
});
});
return new Promise(function(resolve, reject) {
parser.on('end', resolve);
parser.on('error', reject);
});
}
Anyone sees a potential problem with this approach?
You might want to look at promise-streams
var ps = require('promise-streams');
passedStream
.pipe(csv.parse({trim: true}))
.pipe(ps.map({concurrent: 4}, row => processRowDataWhichMightBeAsyncAndReturnPromise(row)))
.wait().then(_ => {
console.log("All done!");
});
Works with backpressure and everything.
Find below a complete application that correctly executes the same kind of task as you want: It reads a file as a stream, parses it as a CSV and inserts each row into the database.
const fs = require('fs');
const promise = require('bluebird');
const csv = require('csv-parse');
const pgp = require('pg-promise')({promiseLib: promise});
const cn = "postgres://postgres:password#localhost:5432/test_db";
const rs = fs.createReadStream('primes.csv');
const db = pgp(cn);
function receiver(_, data) {
function source(index) {
if (index < data.length) {
// here we insert just the first column value that contains a prime number;
return this.none('insert into primes values($1)', data[index][0]);
}
}
return this.sequence(source);
}
db.task(t => {
return pgp.spex.stream.read.call(t, rs.pipe(csv()), receiver);
})
.then(data => {
console.log('DATA:', data);
}
.catch(error => {
console.log('ERROR:', error);
});
Note that the only thing I changed: using library csv-parse instead of csv, as a better alternative.
Added use of method stream.read from the spex library, which properly serves a Readable stream for use with promises.
I found a slightly better way of doing the same thing; with more control. This is a minimal skeleton with precise parallelism control. With parallel value as one all records are processed in sequence without having the entire file in memory, we can increase parallel value for faster processing.
const csv = require('csv');
const csvParser = require('csv-parser')
const fs = require('fs');
const readStream = fs.createReadStream('IN');
const writeStream = fs.createWriteStream('OUT');
const transform = csv.transform({ parallel: 1 }, (record, done) => {
asyncTask(...) // return Promise
.then(result => {
// ... do something when success
return done(null, record);
}, (err) => {
// ... do something when error
return done(null, record);
})
}
);
readStream
.pipe(csvParser())
.pipe(transform)
.pipe(csv.stringify())
.pipe(writeStream);
This allows doing an async task for each record.
To return a promise instead we can return with an empty promise, and complete it when stream finishes.
.on('end',function() {
//do something wiht csvData
console.log(csvData);
});
So to say you don't want streaming but some kind of data chunks? ;-)
Do you know https://github.com/substack/stream-handbook?
I think the simplest approach without changing your architecture would be some kind of promise pool. e.g. https://github.com/timdp/es6-promise-pool
Im using express with node.js, and testing certain routes. I'm doing this tute at http://coenraets.org/blog/2012/10/creating-a-rest-api-using-node-js-express-and-mongodb/
Im calling the http://localhost:3000/wines via ajax (the content doesn't matter). But I want to test latency. Can I do something like make express respond after 2 seconds? (I want to simulate the ajax loader and I'm running on localhost so my latency is pretty much nil)
Use as middleware, for all your requests
app.use(function(req,res,next){setTimeout(next,1000)});
Just call res.send inside of a setTimeout:
setTimeout((() => {
res.send(items)
}), 2000)
To apply it globaly on all requests you can use the following code:
app.use( ( req, res, next ) => {
setTimeout(next, Math.floor( ( Math.random() * 2000 ) + 100 ) );
});
Time values are:
Max = 2000 (sort of.... min value is added so in reality its 2100)
Min = 100
Try connect-pause module. It adds delay to all or only some routes in your app.
app.get('/fakeDelay', function(req,res){
let ms = req.query.t;
ms = (ms>5000 || isNaN(ms)) ? 1000 : parseInt(ms);
setTimeout((()=> res.status(200).send({delay:ms})), ms);
})
Then request the URL as: http://localhost/fakeDelay/?t=2000
(max 5000ms and default of 1000ms on this example)
Update:
Using a Promise. The function 'sleep' can be used for delaying any Express response or other async function.
const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));
app.get('/fakeDelay', async (req, res) => {
await sleep(500);
res.send([])
})
just add a comment on top of the solution of #maggocnx : put this middleware early (before your route handler)
app.use(function(req,res,next){setTimeout(next,1000)});
You could also just write your own generic delay handler using a Promise or callback (using a q promise in this case):
pause.js:
var q = require('q');
function pause(time) {
var deferred = q.defer();
// if the supplied time value is not a number,
// set it to 0,
// else use supplied value
time = isNaN(time) ? 0 : time;
// Logging that this function has been called,
// just in case you forgot about a pause() you added somewhere,
// and you think your code is just running super-slow :)
console.log('pause()-ing for ' + time + ' milliseconds');
setTimeout(function () {
deferred.resolve();
}, time);
return deferred.promise;
}
module.exports = pause;
then use it however you'd like:
server.js:
var pause = require('./pause');
router.get('/items', function (req, res) {
var items = [];
pause(2000)
.then(function () {
res.send(items)
});
});
You can use the express-delay package.
The code below will delay all incoming requests by one second.
const app = require('express')();
const delay = require('express-delay');
app.use(delay(1000));
The package also offers the possibility to introduce a random delay within specified boundaries, e.g. by calling delay(1000, 2000) for a delay between one and two seconds.
In my case I wanted a way to have the same processing time for all of my endpoints.
The solution I found is to override one of the Response methods :
/* MINIMUM DELAY */
const minDelay = 200; /* ms */
app.use((req, res, next) => {
const tmp = res.json;
const start = new Date().getTime();
(res.json as any) = async (body: any) => {
await new Promise((re) => setTimeout(re, minDelay - new Date().getTime() + start));
tmp.apply(res, [body]);
};
next();
});
This way an attacker would not be able to differentiate failed login requests from OK requests just by looking at the response time :)