My customer is all of a sudden experiencing problems with a HTML scraper job made with Node.js. I have circled in on the cause, and found that it's located in the Request module. That made me write a small test application, which solely gets the HTML of the given URL via the Request module. Like this:
var request = require('request');
request('https://www.politi.dk/da/ompolitiet/jobipolitiet/ledige_stillinger/ledigestillinger', function(err, res, body){
if(err){
console.log(err);
} else {
console.log('statusCode:', res.statusCode);
console.log('statusMessage:', res.statusMessage);
}
});
The above example does not work though, as I am getting the following error when running the application:
{ Error: socket hang up
at TLSSocket.onHangUp (_tls_wrap.js:1137:19)
at Object.onceWrapper (events.js:313:30)
at emitNone (events.js:111:20)
at TLSSocket.emit (events.js:208:7)
at endReadableNT (_stream_readable.js:1064:12)
at _combinedTickCallback (internal/process/next_tick.js:138:11)
at process._tickCallback (internal/process/next_tick.js:180:9)
code: 'ECONNRESET',
path: null,
host: 'www.politi.dk',
port: 443,
localAddress: undefined }
However if I change the URL to any other URL it works and I get the following:
statusCode: 200
statusMessage: OK
I have tried passing other URL's on the politi.dk domain, which doesn't work either. Therefore I can conclude that there's a problem with this domain, when requesting pages via the Request module. The strange thing is just, that it worked up until recently. What can cause this problem? Can some changes in settings be made to the server of politi.dk, that is causing this now? I find it hard to find anything helpful on Google. I found the nodejs-what-does-socket-hang-up-actually-mean thread here on SO, which is the exact same problem. But the answers doesn't help me much.
Anyone?
Related
So I'm trying go through a 8 hours video tutorial on Node.js and Express but I seem to hit a point where I am not understanding well.
There's a quick example of an app running but my app keeps crashing unlike in the videos that seems to be working fine. The app code is below and my dilemma is after as well as the error message on console.
const http = require('http')
const server = http.createServer((req,res)=> {
if(req.url === '/'){
res.end('Welcome to our home page')
}
if(req.url === '/about'){
res.end('Here is our short history')
}
res.end(`
<h1>Oops!</h1>
<p>We can't seem to find the pages you are looking for</p>
Back Home
`)
})
server.listen(5000)
whenever I start the app in my terminal it starts fine. My problem is that when I go to the browser at localhost:5000/, localhost:5000/about my app crashes. If I go to localhost:5000/error* it working fine until I click the link that send me to localhost:5000/ and then crashes again. With the following messages.
node:events:504
throw er; // Unhandled 'error' event
^
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at new NodeError (node:internal/errors:371:5)
at ServerResponse.end (node:_http_outgoing:846:15)
at Server.<anonymous> (/home/chachoz007/NodeJS/tutorial/app.js:12:6)
at Server.emit (node:events:526:28)
at parserOnIncoming (node:_http_server:951:12)
at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
Emitted 'error' event on ServerResponse instance at:
at emitErrorNt (node:_http_outgoing:726:9)
at processTicksAndRejections (node:internal/process/task_queues:84:21) {
code: 'ERR_STREAM_WRITE_AFTER_END'
}
Now I get that my app crashes because res.end and the end is trying to write after it already ended in one of the if statements from before.
My problem is why it is crashing compared to the tutorial that i am following that is not crashing. My version of node.js is 16.**** compared to the tutorials version of 14.****
i was wondering if someone could give me some intell on this issue.
You can only call .end() once for each request.
Your code executes the .end() method twice whenever the request is for / or /about; once in the if-else and then a second time after that. One way to solve it is to use "else if" and then else for the final .end() call.
I have a firebase function which is running under Node.js 8 on a Pay-as-you-go Blaze plan. It is quite simple. It just uses node-fetch in order to execute an HTTP request:
fetch(`https://api.thinger.io/v2/users/${accountId}/devices/${deviceId}/${resourceId}`, {
method: 'post',
body: JSON.stringify({ "in" : true }),
headers: {
'Content-Type': 'application/json;charset=UTF-8',
'Authorization': `Bearer ${functions.config().thinger.devices_access_token}`
},
})
.catch(err => console.error(err));
What I basically try to do here is to call Thinger.io's devices API. After the call gets executed I am getting the following error (which doesn't reproduce if I am using Postman, for example, in order to create the HTTP request)
{ FetchError: request to https://api.thinger.io/v2/users/*****/devices/*****/frontDoorRelay failed, reason: certificate has expired
at ClientRequest.<anonymous> (/srv/node_modules/node-fetch/lib/index.js:1455:11)
at emitOne (events.js:116:13)
at ClientRequest.emit (events.js:211:7)
at TLSSocket.socketErrorListener (_http_client.js:401:9)
at emitOne (events.js:116:13)
at TLSSocket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:66:8)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
message: 'request to https://api.thinger.io/v2/users/*****/devices/*****/frontDoorRelay failed, reason: certificate has expired',
type: 'system',
errno: 'CERT_HAS_EXPIRED',
code: 'CERT_HAS_EXPIRED' }
So, it is a bit hard for me now to diagnose if this is a firebase issue or node-fetch issue or something else. For sure, it is not Thinger.io issue because as I told you, if I create the request from another HTTP client (e.g. Postman) it works as expected.
NOTE: There's something that I want to mention here. There hasn't been any code base changes. It started to fail at a certain point just like that.
NOTE: If I try to create another HTTP request using node-fetch it works as expected:
fetch('https://api.github.com/users/github')
.then(res => res.json())
.then(json => console.log(json));
The solution for me was to update Node.js version from 8 to 10. Then, everything started to work as expected.
I followed this migration manual by Doug Stevenson. But, anyways, I think this issue needs to be further addressed by the Firebase team.
The problem is that one of the certificates in the chain, the root certificate, expired on 30 May 2020, as analyzed by https://www.ssllabs.com/:
It seems that some clients fail on this condition, even when the root certificate provided in the chain is not really used, as stated in the RFC5246:
"Because certificate validation requires that root keys be distributed
independently, the self-signed certificate that specifies the root
certificate authority MAY be omitted from the chain, under the
assumption that the remote end must already possess it in order to
validate it in any case".
I have fixed the certificate chain now, so, you should not have more problems related with SSL.
At some random times, on our production server, we face such type of timeouts and the server crashes.
Error: ESOCKETTIMEDOUT
at ClientRequest.<anonymous> (/usr/src/app/node_modules/request/request.js:816:19)
at Object.onceWrapper (events.js:272:13)
at ClientRequest.emit (events.js:180:13)
at ClientRequest.emit (domain.js:422:20)
at TLSSocket.emitTimeout (_http_client.js:703:34)
at Object.onceWrapper (events.js:272:13)
at TLSSocket.emit (events.js:180:13)
at TLSSocket.emit (domain.js:422:20)
at TLSSocket.Socket._onTimeout (net.js:396:8)
at ontimeout (timers.js:466:11)
at tryOnTimeout (timers.js:304:5)
at Timer.listOnTimeout (timers.js:267:5)
I have gone though a lot of blogs and QAs. To summarize, it's read timeout when the caller doesn't get next bytes of response within some specific time limit. From the offical docs (request-npm), we can see -
Read timeout: Time to wait for a server to send response headers (and start the response body) before aborting the request.
Connection timeout: Sets the socket to timeout after timeout milliseconds of inactivity. Note that increasing the timeout beyond the OS-wide TCP connection timeout will not have any effect (the default in Linux can be anywhere from 20-120 seconds)
However I am trying to reproduce it in local linux machine with a HTTP request to my own Node JS server. I am able to get ETIMEDOUT which is connection timeout but I can't get ESOCKETTIMEDOUT error reproduced. In my destination server I had used this simple script with Express JS as an attempt to get this error.
app.use((req, res, next) => {
res.write('sample string');
// Do nothing, let the req wait
});
Therefore does anybody knows a simple approach to get this reproduced? Also I would like to know how to handle this error instead of having it as uncaught exception.
Thanks!
I am getting strange behavior using the request library in node when hitting certain sites.
for a quick context (that has nothing to do with the technical issue): my goal on this project was to read the NASDAQ dividend history for a number of stocks. The histories can be quite long and I'm certainly not going to sit here and read 100 of them manually. An example of one of these pages: https://www.nasdaq.com/symbol/msft/dividend-history
On to the technical troubles... I am using node v11.10.0 with the request module to call the page - the simplest possible thing. I have used this library to do other similar tasks and have never had an issue. Here was my very simple script:
var request = require("request");
var options = { method: 'GET',
url: 'https://www.nasdaq.com/symbol/msft/dividend-history',
headers:
{ 'Postman-Token': 'e4e9749e-d73e-44b9-994f-9d0522654fb0',
'cache-control': 'no-cache' } };
request(options, function (error, response, body) {
if (error) throw new Error(error);
console.log(body);
});
Save file as index.js, and run
node index.js
and that should get the page... except I'm getting an error. The error is:
Error: Error: read ECONNRESET
at Request._callback (<localpath>\index.js:8:24)
at self.callback (<localpath>\node_modules\request\request.js:185:22)
at Request.emit (events.js:197:13)
at Request.onRequestError (<localpath>\node_modules\request\request.js:881:8)
at ClientRequest.emit (events.js:197:13)
at TLSSocket.socketErrorListener (_http_client.js:397:9)
at TLSSocket.emit (events.js:197:13)
at emitErrorNT (internal/streams/destroy.js:82:8)
at emitErrorAndCloseNT (internal/streams/destroy.js:50:3)
at processTicksAndRejections (internal/process/next_tick.js:76:17)
So, thinking that I maybe had something wrong, I tried the same request in postman to validate. But Postman had no trouble doing a get request against the same URL above. I thought perhaps there was a certificate issue with the https stuff, but I toggled certificate validation on and off in postman, and it worked regardless of the setting.
Additional note here - my "very simple script" is actually what I pulled from the Postman "code" tab for a Node/Request code block, so I am really baffled that Postman works but Postman's own node codeblock does not.
At this point I don't even care about what I was trying to do originally... I'm just trying to figure out why I am getting different behavior on two seemingly equivalent ways to make a GET request.
Does anyone have insight on what postman has that node is missing in this case?
Is there any way to get the data from twitter api for any location?
var sanFrancisco = ['79.86','12.62','80.28','13.21']
var stream = T.stream('statuses/filter', { locations: sanFrancisco })
stream.on('tweet', function (tweet) {
console.log(tweet)
It gives an error:
Error: Bad Twitter streaming request: 401
at Object.exports.makeTwitError (/home/file_upload/node_modules/twit/lib/helpers.js:74:13)
at IncomingMessage. (/home/file_upload/node_modules/twit/lib/streaming-api-connection.js:95:29)
at IncomingMessage.emit (events.js:117:20)
at _stream_readable.js:943:16
at process._tickCallback (node.js:419:13)
I assume you followed the usage guide here https://github.com/ttezel/twit#usage and created the twit object with suitable OAuth keys?
Another possibility for this error is if your clock is out of sync, the twitter endpoint will reject requests if the time is out by a few minutes. Check your computers time is correct.