I'm combining the async and request modules to make api requests asynchronously and with rate limiting.
Here is my code
var requestApi = function(data){
request(data.url, function (error, response, body) {
console.log(body);
});
};
async.forEachLimit(data, 5, requestApi, function(err){
// do some error handling.
});
Data contains all the urls I make request to. Am limiting the number of concurrent request to 5 using forEachLimit method. This code makes the first 5 request then stops.
In the async docs it says "The iterator is passed a callback which must be called once it has completed". But I don't understand this, what should I be doing to signal that the request has completed?
First, you shall add callback to your iterator function:
var requestApi = function(data, next){
request(data.url, function (error, response, body) {
console.log(body);
next(error);
});
};
next(); or next(null); tells Async that all processing is done. next(error); indicates an error (if error not null).
After processing all requests Async calls its callback function with err == null:
async.forEachLimit(data, 5, requestApi, function(err){
// err contains the first error or null
if (err) throw err;
console.log('All requests processed!');
});
Async calls its callback immediately after receiving the first error or after all requests completed succesfully.
Related
I have an API endpoint, which executes a function, which in turn listens to some events via a web socket and then throws the response of it back via a callback to the api endpoint, so the endpoint can respond with this object to a http request.
Sometimes I get only one event, sometimes I get two events from the listener. When only the first event appears, I have no problems. Because the second event has priority, I am using setTimeout to wait for it, in case it still appears.
The problem ist, when the second event appears, my API endpoint responds with its object, but afterwards it crashes. I suppose it has something to do with the second callback and I don't know how to handle it. This is how the code is structured:
app.get('/api/myendpoint', function(req, res) {
myVar_1 = req.query.myvar1;
myVar_2 = req.query.myvar2;
myFunction(myVar_1, myVar_2, function(data, error) {
if (!error) res.json(data);
else res.json({"error": String(error)});
});
});
function myFunction(myVar_1, myVar_2, callback){
listenOnSocket.someEvent(myVar_1,
function(error, event) {
if (error) callback(null, error);
else setTimeout(function() {callback(event, null);}, 6000);
}
);
listenOnSocket.someEvent(myVar_2,
function(error, event) {
if (!error) callback(event, null);
else callback(null, error);
}
);
};
This is what my errors look like:
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent
to the client
Why its Happening?
1st event occurs at which you set a timeout for 6000 ms to call the callback function.
2nd event occurs before 6000 and the callback executes due to event 2. Response is sent here.
timeout completes and the pending function passed to setTimeout executes the callback which would send the response again and its not possible to send headers to a response after its sent.
Solution
A basic solution would be to check a flag variable in the function passed to set timeout that whether the 2nd event has occured or not.
function myFunction(myVar_1, myVar_2, callback){
let executed = false;
listenOnSocket.someEvent(myVar_1,
function(error, event) {
if(!executed){
executed = true
if (error) callback(null, error);
else setTimeout(function() {
callback(event, null);
}, 6000);
}
}
);
listenOnSocket.someEvent(myVar_2,
function(error, event) {
if(!executed){
executed = true;
if (!error) callback(event, null);
else callback(null, error);
}
}
);
};
I have an API POST route where I receive data from a client and upload the data to another service. This upload is done inside of the post request (async) and takes awhile. The client wants to know their post req was received prior to the async (create project function) is finished. How can I send without ending the POST? (res.send stops, res.write doesn't send it out)
I thought about making an http request back to their server as soon as this POST route is hit. . .
app.post('/v0/projects', function postProjects(req, res, next) {
console.log('POST notice to me');
// *** HERE, I want to send client message
// This is the async function
createProject(req.body, function (projectResponse) {
projectResponse.on('data', function (data) {
parseString(data.toString('ascii'), function (err, result) {
res.message = result;
});
});
projectResponse.on('end', function () {
if (res.message.error) {
console.log('MY ERROR: ' + JSON.stringify(res.message.error));
next(new Error(res));
} else {
// *** HERE is where they finally receive a message
res.status(200).send(res.message);
}
});
projectResponse.on('error', function (err) {
res.status(500).send(err.message);
});
});
});
The internal system requires that this createProject function is called in the POST request (needs to exist and have something uploaded or else it doesn't exist) -- otherwise I'd call it later.
Thank you!
I think you can't send first response that post request received and send another when internal job i.e. createProject has finished no matter success or fail.
But possibly, you can try:
createProject(payload, callback); // i am async will let you know when done! & it will push payload.jobId in doneJobs
Possibility 1, If actual job response is not required:
app.post('/v0/projects', function (req, res, next) {
// call any async job(s) here
createProject(req.body);
res.send('Hey Client! I have received post request, stay tuned!');
next();
});
});
Possibility 2, If actual job response is required, try maintaining queue:
var q = []; // try option 3 if this is not making sense
var jobsDone = []; // this will be updated by `createProject` callback
app.post('/v0/projects', function (req, res, next) {
// call async job and push it to queue
let randomId = randomId(); // generates random but unique id depending on requests received
q.push({jobId: randomId });
req.body.jobId = randomId;
createProject(req.body);
res.send('Hey Client! I have received post request, stay tuned!');
next();
});
});
// hit this api after sometime to know whether job is done or not
app.get('/v0/status/:jobId', function (req, res, next) {
// check if job is done
// based on checks if done then remove from **q** or retry or whatever is needed
let result = jobsDone.indexOf(req.params.jobId) > -1 ? 'Done' : 'Still Processing';
res.send(result);
next();
});
});
Possibility 3, redis can be used instead of in-memory queue in possibility 2.
P.S. There are other options available as well to achieve the desired results but above mentioned are possible ones.
The following question is asked in order to better understand from your answers how to think "Async" when developing on Node.js
I have the following code:
router.get('/', function(req, res, next) {
... //Definition of rules and paramsObj
//Validation that returns a promise
Indicative.validate(rules,paramsObj)
.then(function(success){
//we passed the validation. start processing the request
//ProcessRequest has async calls but when all async functions are over, it sets paramObj.someVal with a calculated value.
processRequest(paramsObj,next);
//My problem is here. at this point paramsObj.someVal is not set yet. and therefore the response to the user will be incorrect.
res.send(paramsObj.someVal);
}).catch(function(err){
console.log(err);
next(err);
}).done();
}
I wish to understand how to better think "async" while i need to wait with the response to the user until all async functions are over.
My question is how to execute res.send(paramObj.someVal) only after the paramObj.someVal is set by some async methods in processRequest(paramsObj,next);
If you need to wait on the result of processRequest for paramsObj.someVal to be set then ultimately you need to handle that callback
router.get('/', function(req, res, next) {
... //Definition of rules and paramsObj
//Validation that returns a promise
Indicative.validate(rules,paramsObj)
.then(function(success){
//we passed the validation. start processing the request
//ProcessRequest has async calls but when all async functions are over, it sets paramObj.someVal with a calculated value.
processRequest(paramsObj, function(err) {
if (!err && !paramsObj.someVal) {
// raise a custom error if the value is not set
err = new Error('Value not set');
}
if (err) {
next(err);
} else {
res.send(paramsObj.someVal);
}
});
}).catch(function(err){
console.log(err);
next(err);
}).done();
}
Assuming the second argument to processRequest() is a completion callback, you can pass your own function for that callback and do your res.send() in that custom callback like this:
router.get('/', function(req, res, next) {
... //Definition of rules and paramsObj
//Validation that returns a promise
Indicative.validate(rules,paramsObj)
.then(function(success){
//we passed the validation. start processing the request
//ProcessRequest has async calls but when all async functions are over, it sets paramObj.someVal with a calculated value.
processRequest(paramsObj,function() {
res.send(paramsObj.someVal);
});
}).catch(function(err){
console.log(err);
next(err);
}).done();
}
Since you do res.send(...), I assume you don't want to actually call next() in that code path.
I'm trying to get data from an existing mongo database, and for each document I get, I need to make 2 requests to two different services, github and twitter.
The code works if I put a limit in my cursor, but it stops working when I increase, or remove the limit. I think it is because I'm making too many concurrent requests to either github or twitter or both. I get the message
{ [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
I'm not sure how to fix this.
col.find({}).limit(100).forEach(function (doc) {
var options = {};
var params = {};
request(options, function (err, response, body) {
request (options, function (err, response, body){
// Stuff
});
});
}, function (err) {
if (err) {
console.log(err);
return;
}
db.close();
})
You are basically starting 100 (or the # of documents that your query results in) HTTP requests in parallel.
The very useful async module has a function eachLimit to limit the number of concurrent (running in parallel) asynchronous tasks. In your case, you could leverage it like so:
var async = require('async');
col.find({}).limit(100).toArray(function(err, docs) {
if (err) return console.log(err);
// Limit the # of concurrent HTTP requests to 2(-ish).
async.eachLimit(docs, 2, function(doc, asyncdone) {
request(options, function (err, response, body) {
if (err) return asyncdone(err);
request (options, function (err, response, body){
if (err) return asyncdone(err);
// Stuff
return asyncdone();
});
});
}, function(err) {
// If we get here, we're done.
if (err) {
console.log(err);
return;
}
db.close();
});
});
Be aware that .toArray() reads all the query results into memory first (but your .forEach() does that as well I think).
Should be caused by too many request calls, you can use the async library to do request limit.
mapLimit(arr, limit, iterator, [callback])
it's very easy and trivial to use.
I have a branch in a function that isn't currently being tested. It's an error handler coming from a request operation (using the node module of the same name). Here is the particular line:
request(url, function (error, response, body) {
if (error) return cb(error);
Here is the test:
describe("handles errors", function() {
it("from the request", function (done) {
var api = nock('http://football-api.com')
.get('/api/?Action=today&APIKey=' + secrets.APIKey + '&comp_id=1204')
.reply(500);
fixture.getFixture(FixtureMock, function (err, fixture) {
expect(err).to.exist
done();
});
});
Spec fails:
Uncaught AssertionError: expected null to exist
So either sending a 500 status code as a response with no body does not cause an error in the request callback, or I'm testing the wrong thing.
Use replyWithError from doc:
nock('http://www.google.com')
.get('/cat-poems')
.replyWithError('something awful happened');
This variant of #PiotrFryga's answer worked for me, as my request callback(err, resp, body) was actually checking for the "ETIMEDOUT" error code in err:
nock('http://www.google.com')
.get('/cat-poems')
.replyWithError({code: "ETIMEDOUT"});
The only found workaround is to simulate timeout
nock('http://service.com').get('/down').delayConnection(500).reply(200)
unirest.get('http://service.com/down').timeout(100).end(function(err) {
// err = ETIMEDOUT
});
source