I am developing a script on Node.js that sends a lot of requests to an API. After several requests (more than 380 requests), we receive the following error message : Error: socket hang up (code:ECONNRESET). This is a big issue for our script since we would like to send around 10000 requests.
This is not an issue with the rate limit of the API because we are already handling this.
Our script is running on OVH server, and we send our requests using the package request-promise. Our version of Node.js is v 9.9.0.
Here is the function where the error is thrown :
const pollSession = async (sessionUrl) => {
let session;
try {
session = await rp.get({ url: sessionUrl, json: true }, (err, res, body) => {
if (err) {
console.log('Err: ', err);
} else {
DEBUG && console.log("Status code: ",res && res.statusCode);
DEBUG && console.log("Status: ",res && res.body && res.body.Status);
statusCode = res && res.statusCode;
status = res && res.body && res.body.Status;
}
});
} catch (e) {
console.log ("----- pollSession : in catch with return value :"+e);
return e;
}
return session;
}
When the request is working, we are calling this function few times in order to get the full response (because the response is huge).
When the error "Err: { Error: socket hang up" is thrown, we are calling the function again and it returns this error again. We can't afford to give up on those requests so we would like to know how to work around this error. Maybe it is possible to increase the max number of sockets (I saw it was possible with http agent, but we are using request-promise package) ?
Let me know if you need further information
After a lot of tests, I find out that this is related to the API I am sending requests to, Skyscanner for the record. Some flights I am searching are too long to be retrieved and lead to this error. Fixed this issue by catching the error.
Related
I am trying to use different Axios calls to get some data from a remote server. One by one the calls are working but as soons as I call them directly after each other its throwing the error message about the headers. I did some research already and I guess it has sth to do that there the headers of the first call gets in the way of the second call. That is probably a very simplematic description of the problem but I am new to node js and the way those axios calls are working.
This is an example of one of my Api calls:
app.get('/api/ssh/feedback', function(req, res){
conn.on('ready', function(){
try {
let allData = {}
var command = 'docker ps --filter status=running --format "{{.Names}}"'
conn.exec(command, function(err, stream){
if (err) throw console.log(err)
stream.on('data', function(data){
allData = data.toString('utf8').split('\n').filter(e=>e)
return res.json({status: true, info: allData})
})
stream.on('close', function(code){
console.log('Process closed with: ' + code)
conn.end()
})
stream.on('error', function(err){
console.log('Error: ' + err)
conn.end()
})
})
} catch (err) {
console.error('failed with: ' + err)
}
}).connect(connSet)
})
I am using express js as a middleware and the shh2 package to get the connection with the remote server. How I mentioned before the call is working but crashes if it is not the first call. I am able to use the api again after I restart the express server.
This is how I am calling the api through axios in my node js frontend:
getNetworkStatus(e){
e.preventDefault()
axios.get('/api/ssh/network').then(res =>{
if(res.data.status){
this.setState({network_info: 'Running'})
this.setState({network: res.data.info})
} else {
this.setState({network_info: 'No Network Running'})
this.setState({network: 'No Network detected'})
}
}).catch(err => {
alert(err)
})
}
I would be really grateful for any help or advice how to solve this problem. Thanks to everyone who spends some time to help me out.
There are two issues in the code you've provided:
You are making assumptions about 'data' events. In general, never assume the size of the chunks you receive in 'data' events. You might get one byte or you might get 1000 bytes. The event can be called multiple times as chunks are received and this is most likely what is causing the error. Side note: if the command is only outputting text, then you are better off using stream.setEncoding('utf8') (instead of manually calling data.toString('utf8')) as it will take care of multi-byte characters that may be split across chunks.
You are reusing the same connection object. This is a problem because you will continue to add more and more event handlers every time that HTTP endpoint is reached. Move your const conn = ... inside the endpoint handler instead. This could also be causing the error you're getting.
Here I am making some http requests on links which I filtered in advance. The links are inside an array in string format.
for (let link of linkArray) {
const linkOptions = {
uri: link,
resolveWithFullResponse: true,
};
linkRequest = await rp(linkOptions);
//some other actions with linkRequest response
}
But of course the request statuses of the links can be some bad ones (for example 404 forbidden errors, status codes basically grater than 300).
Here I don't want the program to be crushed, instead I want to skip the response that is forbidden and continue checking remaining links of linkArray.
What I have tried so far:
below the comment I checked the statusCodes, if it's some bad one than continue loop iteration.
if (assetsRequest.statusCode === 204 || assetsRequest.statusCode >= 300) continue
But the request is failing when it's is bad request, in any case without waiting for my checking. Any ideas how to handle this?
p.s. I am using request-promise module
The program crashes because you do not catch the error.
You should always catch errors, especially if you do error prone process such as http request.
Here i've added try catch
for (let link of linkArray) {
const linkOptions = {
uri: link,
resolveWithFullResponse: true,
};
try {
linkRequest = await rp(linkOptions);
} catch(err) {
// do something with error or at least log it
console.log(err);
}
//some other actions with linkRequest response
}
So when rp() fails, it will go straight to the catch block but does not quit the loop, hence the process continue for other link in the array.
I'm trying to pipe an image stored on Amazon S3 using node-request. However, sometimes an image doesn't exist, which is exposed by S3 as a status code 403. I'm struggling to see how I can pipe in case of success (200) but take an alternative action in case of a non-200.
Using the abort() method seemed like the way to go but getting an r.abort is not a function, even though it should be available on the request.
// context: inside a request handler, where `res` is the response to write to
const r = request(url)
.on('response', function (response) {
if (response.statusCode !== 200) {
r.abort(); //failing
// I want to stop the piping here and do whatever instead.
}
})
.pipe(res);
Thoughts?
To answer my own question: don't start piping until sure it's correct:
request(url)
.on('response', function (r) {
if (r.statusCode !== 200) {
//take other action
}
r.pipe(res);
})
I'm getting random Socket Hang Up errors on my node app. My app posts over 1,000 images to a server for processing and it happens at random points so it's very difficult to reproduce. Annoyingly it usually happens just when you think it's not going to happen!
I'm using RXJS and I have created a wrapper library for the node http functions here:
https://github.com/Roaders/rx-http-ts
The code that does the requests is:
export function makeHttpRequest(options: http.RequestOptions, data?: any): Rx.Observable<string>{
return Rx.Observable.defer(() => {
const request = options.protocol === HTTPS ? https.request(options) : http.request(options) ;
const errorObservable = Rx.Observable.fromEvent(<any>request, "error")
.flatMap(error => {
return Rx.Observable.throw(error);
});
const responseObservable = Rx.Observable.fromEvent(<any>request, "response");
if(data){
request.write(data);
}
request.end();
return responseObservable.merge(errorObservable)
.take(1);
})
.flatMap( response => RxNode.fromReadableStream(<any>response))
.toArray()
.map(function(allData){
return allData.join("");
});
}
I don't understand how I can be getting that error and that it's not being caught. I'm listening for the error events from the request object and errors on the stream should be handled by RxNode.fromReadableStream
Any suggestions?
Thanks
I am running node.js with express. I wrote a node module with methods in it so when you go to
http://bla.com/module_name/method_name
it will run the method.
The method follows the typical style of
exports.method_name(req, res, next);
my main app does something like this:
app.all("*", resSetup, controller, render);
and controller is the thing that will call the method based on the path.
it seems that if there is an undefined variable error in the method, express will just hang there and not throw any error. Nothing will appear in the console log either. I can put a console message right before and after where the error occurs and the before will appear in the log, and after will not.
I can wrap it in a try/catch and get this:
[ReferenceError: blabla is not defined]
but no line numbers or anything.
My guess is that express is somehow preventing the errors from coming up. When I put the error in the function called "controller" that is directly in the route, it shows that error correctly.
It might not matter too much, but here is the code I am working on:
https://github.com/RobKohr/quick-site/blob/master/index.js
Line 189 is where the method call happens.
Building on Ruairi's comment above, I had this same issue with when using 'q' (https://github.com/kriskowal/q) and promises with express - node would hang and no error was generated.
By adding a catch to the end of the promise 'callback' chain I was able to see the error and print it to console etc.
The code ends up looking like:
export function index(req, res) {
//Create the 'promise'
var request = req.body;
var queryJobID = req.query.jobId;
console.log('queryJobID: ' + queryJobID);
var jobStatusPromsie = jobManager.job.getStatus(queryJobID);
Q.all([jobStatusPromsie])
.then(
function (result) {
var responseData = {};
console.log('Job Status Response received');
if (result != null) {
//Without the .catch below an error here will be 'silent'
console.log('jobStatus result: ' + util.inspect(result, false, null));
responseData['status'] = 'OK';
responseData['progress'] = result.progress;
res.json(responseData);
} else {
console.log('jobStatus Error');
responseData['status'] = 'Error';
}
res.json(responseData);
console.log('jobStatus Response data sent');
},
function (error) {
console.log('Error while getting job status:', error);
res.json("Error");
})
.catch(function(err) {
//handle errors
console.log('Promise error while getting job status:', err);
});
}
Express heavily relies on Nodes asynchronous nature. Seeing errors thrown like on line 30 would give me the creeps if I was maintaining this. Try refactoring your code to only use the next(err) pattern.
The reason that you app is hanging is that Express hasn't finished the HTTP response (eg: res.send()). This means you have broken plumbing where an Error has bubbled up the call stack but not redirected into the Express middleware pipeline. Try registering some error middleware to see if it gets called with your error.