I'm trying to get data from an existing mongo database, and for each document I get, I need to make 2 requests to two different services, github and twitter.
The code works if I put a limit in my cursor, but it stops working when I increase, or remove the limit. I think it is because I'm making too many concurrent requests to either github or twitter or both. I get the message
{ [Error: read ECONNRESET] code: 'ECONNRESET', errno: 'ECONNRESET', syscall: 'read' }
I'm not sure how to fix this.
col.find({}).limit(100).forEach(function (doc) {
var options = {};
var params = {};
request(options, function (err, response, body) {
request (options, function (err, response, body){
// Stuff
});
});
}, function (err) {
if (err) {
console.log(err);
return;
}
db.close();
})
You are basically starting 100 (or the # of documents that your query results in) HTTP requests in parallel.
The very useful async module has a function eachLimit to limit the number of concurrent (running in parallel) asynchronous tasks. In your case, you could leverage it like so:
var async = require('async');
col.find({}).limit(100).toArray(function(err, docs) {
if (err) return console.log(err);
// Limit the # of concurrent HTTP requests to 2(-ish).
async.eachLimit(docs, 2, function(doc, asyncdone) {
request(options, function (err, response, body) {
if (err) return asyncdone(err);
request (options, function (err, response, body){
if (err) return asyncdone(err);
// Stuff
return asyncdone();
});
});
}, function(err) {
// If we get here, we're done.
if (err) {
console.log(err);
return;
}
db.close();
});
});
Be aware that .toArray() reads all the query results into memory first (but your .forEach() does that as well I think).
Should be caused by too many request calls, you can use the async library to do request limit.
mapLimit(arr, limit, iterator, [callback])
it's very easy and trivial to use.
Related
Hello i have a request which fetch some json data from third party API:
request({
url: 'https://api.steampowered.com/IEconService/GetTradeOffers/v1/?key=MYAPIKEY&get_sent_offers=1&active_only=1&format=json',
json: true
}, (err, responser, body, undefined) => {
tradeItems = JSON.stringify(body.response['trade_offers_sent'][0].items_to_give);
});
How can i send tradeItems fetched data to offer.addTheirItems value?
client.on('webSession', function(sessionID, cookies) {
manager.setCookies(cookies, function(err) {
if (err) {
console.log(err);
process.exit(1);
return;
}
let offer = manager.createOffer("https://steamcommunity.com/tradeoffer/new/?partner=123456789&token=1234");
offer.addTheirItems();
offer.setMessage("");
offer.send(function(err, status) {
if (err) {
console.log(err);
return;
}
First, that's are javascript's async issue.
The solution is in many ways.
change the request function to async function. and make tradeItems variable to outside from request function.
I recommend request-promise module
move below codes to in upper code's callback function.
This is a simple answer because your sample code is separated into two parts.
So I have created a Hapi.js restful application. The backend is connected to a SQlite3 database. When the user throws a GET request to any arbitrary endpoint, in this case '/employees,' I am having trouble returning the information obtained from the SQL query to the user with my current callback situation.
Here is my current code:
server.route({
method: 'GET',
path: '/employees',
handler: function(request, h) {
var employees = [];
db.all('SELECT * from Employee;',[], function (err, rows) {
if (err) {
console.log(err);
}
if (rows) {
rows.forEach(elt => {
employees.push(elt);
});
}
}, () => {
return employees.toString();
});
}
// hapi requires me to return a value or promise here
});
Right now the above code is failing because I am not returning anything at the end of the handler function. I am getting rows from my sql query but am having trouble figuring out how to return that information in the response due to my callback structure. Any help would be greatly appreciated.
It seems to me that you are not using db.all() the right way…
According to the documentation here, it seems that db.all take a callback and not two like you do…
Pretty sure your code is failing because you return in a callback that is never used...
You should try something like that:
server.route({
method: 'GET',
path: '/employees',
handler: function(request, h) {
db.all('SELECT * from Employee;',[], function (err, rows) {
if (err) {
console.log(err);
return err;
}
return rows;
});
}
});
This is a wild guess, I have never used Sqlite3 nor Hapi17 but I am pretty confident. Can you please confirm that it is working and keep my mind at peace ?
I have a simple node Express app that has a service that makesa call to a node server. The node server makes a call to an AWS web service. The AWS simply lists any S3 buckets it's found and is an asynchronous call. The problem is I don't seem to be able to get the server code to "wait" for the AWS call to return with the JSON data and the function returns undefined.
I've read many, many articles on the web about this including promises, wait-for's etc. but I think I'm not understanding the way these work fully!
This is my first exposer to node and I would be grateful if somebody could point me in the right direction?
Here's some snippets of my code...apologies if it's a bit rough but I've chopped and changed things many times over!
Node Express;
var Httpreq = new XMLHttpRequest(); // a new request
Httpreq.open("GET","http://localhost:3000/listbuckets",false);
Httpreq.send(null);
console.log(Httpreq.responseText);
return Httpreq.responseText;
Node Server
app.get('/listbuckets', function (req, res) {
var bucketData = MyFunction(res,req);
console.log("bucketData: " + bucketData);
});
function MyFunction(res, req) {
var mydata;
var params = {};
res.send('Here are some more buckets!');
var request = s3.listBuckets();
// register a callback event handler
request.on('success', function(response) {
// log the successful data response
console.log(response.data);
mydata = response.data;
});
// send the request
request.
on('success', function(response) {
console.log("Success!");
}).
on('error', function(response) {
console.log("Error!");
}).
on('complete', function() {
console.log("Always!");
}).
send();
return mydata;
}
Use the latest Fetch API (https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API) to make HTTP calls. It has built-in support with Promise.
fetch('http://localhost:3000/listbuckets').then(response => {
// do something with the response here
}).catch(error => {
// Error :(
})
I eventually got this working with;
const request = require('request');
request(url, function (error, response, body) {
if (!error && response.statusCode == 200) {
parseString(body, function (err, result) {
console.log(JSON.stringify(result));
});
// from within the callback, write data to response, essentially returning it.
res.send(body);
}
else {
// console.log(JSON.stringify(response));
}
})
So im completely stumped and hope someone can help with the combination of Node JS Async and Request modules. I'm attempting to build a list of file to download which I pass to Async, as an array of object contain all the information I need to download and store said file. After tons of debugging I discovered that Request are not even making there way out and I cant figure out why.
async.each(missingFiles,
function (obj, cb) {
console.log(obj.url);
//var file = nfs.createWriteStream(obj.fullPath);
request(obj.url, function (error, response, body) {
if (!error && response.statusCode == 200) {
console.log(response)
}
cb();
})
},
function (err) {
if (err) {
console.log("Async failed");
}
}
);
I came across similar issues before. If you send response outside the async block, http request/response cycle ends before your async tasks complete. The fix is to have send response in the done() callback.
app.post("/download", function(req, res) {
async.eachSeries(missingFiles, function (obj, cb) {
...
//do your work here
cb()
}, function done() {
res.json({success: true});
});
}
I'm combining the async and request modules to make api requests asynchronously and with rate limiting.
Here is my code
var requestApi = function(data){
request(data.url, function (error, response, body) {
console.log(body);
});
};
async.forEachLimit(data, 5, requestApi, function(err){
// do some error handling.
});
Data contains all the urls I make request to. Am limiting the number of concurrent request to 5 using forEachLimit method. This code makes the first 5 request then stops.
In the async docs it says "The iterator is passed a callback which must be called once it has completed". But I don't understand this, what should I be doing to signal that the request has completed?
First, you shall add callback to your iterator function:
var requestApi = function(data, next){
request(data.url, function (error, response, body) {
console.log(body);
next(error);
});
};
next(); or next(null); tells Async that all processing is done. next(error); indicates an error (if error not null).
After processing all requests Async calls its callback function with err == null:
async.forEachLimit(data, 5, requestApi, function(err){
// err contains the first error or null
if (err) throw err;
console.log('All requests processed!');
});
Async calls its callback immediately after receiving the first error or after all requests completed succesfully.