I am getting strange behavior using the request library in node when hitting certain sites.
for a quick context (that has nothing to do with the technical issue): my goal on this project was to read the NASDAQ dividend history for a number of stocks. The histories can be quite long and I'm certainly not going to sit here and read 100 of them manually. An example of one of these pages: https://www.nasdaq.com/symbol/msft/dividend-history
On to the technical troubles... I am using node v11.10.0 with the request module to call the page - the simplest possible thing. I have used this library to do other similar tasks and have never had an issue. Here was my very simple script:
var request = require("request");
var options = { method: 'GET',
url: 'https://www.nasdaq.com/symbol/msft/dividend-history',
headers:
{ 'Postman-Token': 'e4e9749e-d73e-44b9-994f-9d0522654fb0',
'cache-control': 'no-cache' } };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
Save file as index.js, and run
node index.js
and that should get the page... except I'm getting an error. The error is:
Error: Error: read ECONNRESET
at Request._callback (<localpath>\index.js:8:24)
at self.callback (<localpath>\node_modules\request\request.js:185:22)
at Request.emit (events.js:197:13)
at Request.onRequestError (<localpath>\node_modules\request\request.js:881:8)
at ClientRequest.emit (events.js:197:13)
at TLSSocket.socketErrorListener (_http_client.js:397:9)
at TLSSocket.emit (events.js:197:13)
at emitErrorNT (internal/streams/destroy.js:82:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
at processTicksAndRejections (internal/process/next_tick.js:76:17)
So, thinking that I maybe had something wrong, I tried the same request in postman to validate. But Postman had no trouble doing a get request against the same URL above. I thought perhaps there was a certificate issue with the https stuff, but I toggled certificate validation on and off in postman, and it worked regardless of the setting.
Additional note here - my "very simple script" is actually what I pulled from the Postman "code" tab for a Node/Request code block, so I am really baffled that Postman works but Postman's own node codeblock does not.
At this point I don't even care about what I was trying to do originally... I'm just trying to figure out why I am getting different behavior on two seemingly equivalent ways to make a GET request.
Does anyone have insight on what postman has that node is missing in this case?
Related
I have a firebase function which is running under Node.js 8 on a Pay-as-you-go Blaze plan. It is quite simple. It just uses node-fetch in order to execute an HTTP request:
fetch(`https://api.thinger.io/v2/users/${accountId}/devices/${deviceId}/${resourceId}`, {
method: 'post',
body: JSON.stringify({ "in" : true }),
headers: {
'Content-Type': 'application/json;charset=UTF-8',
'Authorization': `Bearer ${functions.config().thinger.devices_access_token}`
},
})
.catch(err => console.error(err));
What I basically try to do here is to call Thinger.io's devices API. After the call gets executed I am getting the following error (which doesn't reproduce if I am using Postman, for example, in order to create the HTTP request)
{ FetchError: request to https://api.thinger.io/v2/users/*****/devices/*****/frontDoorRelay failed, reason: certificate has expired
at ClientRequest.<anonymous> (/srv/node_modules/node-fetch/lib/index.js:1455:11)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at emitOne (events.js:116:13)
at TLSSocket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:66:8)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
message: 'request to https://api.thinger.io/v2/users/*****/devices/*****/frontDoorRelay failed, reason: certificate has expired',
type: 'system',
errno: 'CERT_HAS_EXPIRED',
code: 'CERT_HAS_EXPIRED' }
So, it is a bit hard for me now to diagnose if this is a firebase issue or node-fetch issue or something else. For sure, it is not Thinger.io issue because as I told you, if I create the request from another HTTP client (e.g. Postman) it works as expected.
NOTE: There's something that I want to mention here. There hasn't been any code base changes. It started to fail at a certain point just like that.
NOTE: If I try to create another HTTP request using node-fetch it works as expected:
fetch('https://api.github.com/users/github')
.then(res => res.json())
.then(json => console.log(json));
The solution for me was to update Node.js version from 8 to 10. Then, everything started to work as expected.
I followed this migration manual by Doug Stevenson. But, anyways, I think this issue needs to be further addressed by the Firebase team.
The problem is that one of the certificates in the chain, the root certificate, expired on 30 May 2020, as analyzed by https://www.ssllabs.com/:
It seems that some clients fail on this condition, even when the root certificate provided in the chain is not really used, as stated in the RFC5246:
"Because certificate validation requires that root keys be distributed
independently, the self-signed certificate that specifies the root
certificate authority MAY be omitted from the chain, under the
assumption that the remote end must already possess it in order to
validate it in any case".
I have fixed the certificate chain now, so, you should not have more problems related with SSL.
My customer is all of a sudden experiencing problems with a HTML scraper job made with Node.js. I have circled in on the cause, and found that it's located in the Request module. That made me write a small test application, which solely gets the HTML of the given URL via the Request module. Like this:
var request = require('request');
request('https://www.politi.dk/da/ompolitiet/jobipolitiet/ledige_stillinger/ledigestillinger', function(err, res, body){
if(err){
console.log(err);
} else {
console.log('statusCode:', res.statusCode);
console.log('statusMessage:', res.statusMessage);
}
});
The above example does not work though, as I am getting the following error when running the application:
{ Error: socket hang up
at TLSSocket.onHangUp (_tls_wrap.js:1137:19)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:111:20)
at TLSSocket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
code: 'ECONNRESET',
path: null,
host: 'www.politi.dk',
port: 443,
localAddress: undefined }
However if I change the URL to any other URL it works and I get the following:
statusCode: 200
statusMessage: OK
I have tried passing other URL's on the politi.dk domain, which doesn't work either. Therefore I can conclude that there's a problem with this domain, when requesting pages via the Request module. The strange thing is just, that it worked up until recently. What can cause this problem? Can some changes in settings be made to the server of politi.dk, that is causing this now? I find it hard to find anything helpful on Google. I found the nodejs-what-does-socket-hang-up-actually-mean thread here on SO, which is the exact same problem. But the answers doesn't help me much.
Anyone?
I am scraping a website in a continuous manner (once a day multiple requests) and using the async and request node modules to do so every time. I run the function getPage in parallel using the async eachLimit (not shown in code here). However once in a few thousands queries I get the following error:
Error: getaddrinfo ENOTFOUND 247sports.com 247sports.com:80
at errnoException (dns.js:26:10)
at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)
Even though I know the url I passed it was valid. I found there were solutions for people using the http module and getting the same error but no one seems to be getting this using the request module.
I know my ip was not blocked as I was able to access the site right after the error. I also know my user-agent is not the problem, as I rotate through a list of user agents, all of which are valid.
My guess is that the problem lies where the request library interacts with the node http module. Unfortunately I have not been able to reproduce the problem accurately as it seems to just get triggered when I push a lot of requests at the same time or in a row.
The function code below is what my function looks like:
function getPage(){
var options = {
url: "http://stackoverflow.com/",
headers: { 'User-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:2.0.1) Gecko/20110506 Firefox/4.0.1'
};
request(options, function(err, resp, body) {
if (err){
throw err;
return;
}
PagesScraped++;
console.log(PagesScraped);
return;
});
};
for (var i = 0; i < 600; i++){
getPage();
};
Note: I realize this code gets the same page 600 times asynchronously, but the error is still present if ran enough times... My local code visits multiple thousands of pages on the same website.
I am attempting to nest two get requests inside of each other
Alamofire.request(.GET, "serverAddress")
.responseJSON { response in
if let JSON = response.result.value {
//populate parameters with the expected args
var parameters:[String:[String]] = [String:[String]]()
//Request2
self.queryFriendsList(parameters)
}
Here is the nested request within a different method...
func queryFriendsList(parameters:[String:[String]]) {
Alamofire.request(.GET, "serverAddress", parameters : parameters, encoding: .JSON)
.responseJSON {
response in
if let res = response.result.value {
print(res)
}
}
}
When I check my logs on Heroku I see:
heroku[router]: sock=client at=warning code=H27 desc="Client Request Interrupted" method=GET path="/query_friends_list/"
The log associated with the app reports:
Error: request aborted
at IncomingMessage.onAborted (/app/node_modules/raw- body/index.js:269:10)
at emitNone (events.js:67:13)
at IncomingMessage.emit (events.js:166:7)
at abortIncoming (_http_server.js:281:11)
at emitOne (events.js:82:20)
at Socket.serverSocketCloseListener (_http_server.js:294:5)
at Socket.emit (events.js:169:7)
The second request is dependent upon the first succeeding.
I am at a loss as to how to debug this one. I am using the free version of Heroku, so I wasn't able to determine from the docs if multiple requests would be an issue ?
The issue with this bug had to do with me not understanding where to look for the error...
sock=client
Denoted that the issue was associated with the client side. Initially I had specified a get request for my http request to heroku. I specified parameters to be encoded with JSON but I realized I cannot send a payload encoded with JSON on a get request.
This was really the root issue at hand. After switching my route to a post request and updating the calls in my swift project, I was able to make the proper query against my DB!
I've read all the related questions to this one on here, I've searched through Github issues and I've read as much as possible about Express to try and figure this one out. As far is I've read most people say it is something to do with the headers being set after they've been sent usually with res.set() or something similar but I'm not doing that anywhere in my code.
This error only occurs shortly after I've deployed my SailsJS app to Modulus:
Error occurred in session middleware :: 'Error: Error getting collection: sessions\n at /mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:160:19\n at Db.collection (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:524:27)\n at MongoStore._get_collection (/mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:158:17)\n at MongoStore.set (/mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:274:12)\n at Session.save (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/node_modules/express-session/session/session.js:63:25)\n at ServerResponse.end (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/node_modules/express-session/index.js:263:21)\n at ServerResponse.res.end (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/node_modules/compression/index.js:116:15)\n at ServerResponse.res.send (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:175:8)\n at ServerResponse.res.json (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:223:15)\n at storedUrlCheckServiceDone (/mnt/data/2/api/controllers/ProcessController.js:44:18)\n at tryCatcher (/mnt/data/2/node_modules/bluebird/js/main/util.js:24:31)\n at Promise._settlePromiseFromHandler (/mnt/data/2/node_modules/bluebird/js/main/promise.js:454:31)\n at Promise._settlePromiseAt (/mnt/data/2/node_modules/bluebird/js/main/promise.js:530:18)\n at Promise._settlePromises (/mnt/data/2/node_modules/bluebird/js/main/promise.js:646:14)\n at Async._drainQueue (/mnt/data/2/node_modules/bluebird/js/main/async.js:175:16)\n at Async._drainQueues (/mnt/data/2/node_modules/bluebird/js/main/async.js:185:10)'
http.js:690
throw new Error('Can\'t set headers after they are sent.');
^
Error: Can't set headers after they are sent.
at ServerResponse.OutgoingMessage.setHeader (http.js:690:11)
at ServerResponse.res.setHeader (/mnt/data/2/node_modules/sails/node_modules/express/node_modules/connect/lib/patch.js:133:22)
at ServerResponse.res.set.res.header (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:577:10)
at ServerResponse.res.send (/mnt/data/2/node_modules/sails/node_modules/express/lib/response.js:142:12)
at Object.<anonymous> (/mnt/data/2/node_modules/sails/lib/hooks/http/middleware/defaults.js:56:24)
at Object.immediate._onImmediate (timers.js:372:16)
at processImmediate [as _immediateCallback] (timers.js:354:15)
[2015-04-13T03:23:03.264Z] Application CRASH detected. Exit code 8.
It's something to do with the sessions and connect-mongo but I have no idea how to figure it out. Usually it happens once or twice straight after deployment but sometimes it'll happen up to 5 or 6 times in a row straight after deployment as well.
I've also seen this error occasionally as well around the same time so it might be related:
/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/base.js:246
throw message;
^
Error: Error setting TTL index on collection : sessions
at /mnt/data/2/node_modules/connect-mongo/lib/connect-mongo.js:169:23
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:1497:46
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:1630:20
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:170:22
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:714:39
at Cursor.close (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:1009:5)
at commandHandler (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/cursor.js:714:21)
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/db.js:1903:9
at Server.Base._callHandler (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/base.js:453:41)
at /mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/server.js:488:18
at MongoReply.parseBody (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/responses/mongo_reply.js:68:5)
at null.<anonymous> (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/server.js:446:20)
at emit (events.js:95:17)
at null.<anonymous> (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/connection_pool.js:207:13)
at emit (events.js:98:17)
at Socket.<anonymous> (/mnt/data/2/node_modules/connect-mongo/node_modules/mongodb/lib/mongodb/connection/connection.js:440:22)
[2015-04-13T03:24:15.011Z] Application CRASH detected. Exit code 8.
[2015-04-13T03:24:22.048Z] Application restarted with pid 14613
I'm using Node v0.10.37, Sails v0.10.5 and Passport for authentication and just the mongo adapter for session storage.
I'd really appreciate any help or insights you can give as I've spent nearly a week trying to figure this out before posting here and even had Newrelic's support team looking at and they couldn't figure it out either.
Thanks
Update
I'm not seeing this error locally, only on Modulus.
This is a shortened down version of my controller.
I basically have a Auth Service that checks if the users is authenticated, if they are I pass the results off to another promise which checks the db for some of their data, if it find a record then it returns it if not then it calls a 3rd party API and does some processing, once thats done it returns and creates a new record, then returns the results back to the browser. If they aren't authenticated then I send back an error.
There are 4 calls to different Sails services in total in the controller, which all return a promise.
text: function (req, res) {
// Check that the host is authenticated
AuthService.hostname(req).done( authServiceDone, authServiceFail );
function authServiceDone (site) {
// Check the DB for the origin
return urlService.check(req, site.owner).done(UrlCheckServiceDone, UrlCheckServiceFail );
}
function authServiceFail (error) {
return res.json({
error : error
});
}
}