https.get not working to access a public API with Nodejs - node.js

I'm currently trying to access the JokeAPI using Node's module HTTPS, where below is my code:
const https = require('https');
let url = 'https://v2.jokeapi.dev/joke/Any';
https.get(url, function (res) {
console.log(res);
})
However, somehow I keep getting this error:
node:events:498
throw er; // Unhandled 'error' event
^
Error: self signed certificate in certificate chain
at TLSSocket.onConnectSecure (node:_tls_wrap:1530:34)
at TLSSocket.emit (node:events:520:28)
at TLSSocket._finishInit (node:_tls_wrap:944:8)
at TLSWrap.ssl.onhandshakedone (node:_tls_wrap:725:12)
Emitted 'error' event on ClientRequest instance at:
at TLSSocket.socketErrorListener (node:_http_client:442:9)
at TLSSocket.emit (node:events:520:28)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
code: 'SELF_SIGNED_CERT_IN_CHAIN'
Is it because I don't have a server set up? I'm a beginner with JS and Node, so any help would be very appreciated. Thank you!
Edit: I actually added 'process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';' to the top of the code and it worked, but now I'm getting an 'undefined' for a response body. And I'm not sure why that is happening :(

My system does not show the self signed certificate in certificate chain error, but it sounds like you have a work-around for that.
However, now I'm getting an 'undefined' for a response body. I'm not too sure what's causing this.
https.get() just sends the request and reads the headers. It does not, by itself, read the whole response. To do that, you need to listen for the data event on the res object. Installing the listener for the data event will cause the stream to start reading data from the incoming request. This data will not necessarily come all at once. It can come broken into multiple data events so you would really need to accumulate all the data and then in the end event, you know you have all the data. You can see a full example for how to implement that here in the http.get() doc.
Here's a simpler version that will just show you what's in each data event.
const https = require('https');
let url = 'https://v2.jokeapi.dev/joke/Any';
https.get(url, function(res) {
res.on('data', data => {
console.log(data.toString());
});
});
FYI, I no longer use plain https.get() because all the higher level libraries listed here are more convenient to use and all support promises which is generally an easier way to program with asynchronous requests.
My favorite library from that list is got(), but you can choose whichever one you like the API and/or features for. Here's an implementation using got():
const got = require('got');
const url = 'https://v2.jokeapi.dev/joke/Any';
got(url).json().then(result => {
console.log(result);
}).catch(err => {
console.log(err);
});

Related

How to pipe from standard input to an http request in NodeJS?

I have an http server listening on port 9090 - piping the request to stdout like so:
let server = http.createServer((req, res) => {req.pipe(process.stdout)})
server.listen(9090)
When I send it something with curl like so:
curl -XGET -T - 'http://localhost:9090' < /tmp/text-input
it works, and I see the output on the server's terminal
but when I try the following in node:
const http = require('http')
const nurl = new URL("http://localhost:9090")
let request = http.request(nurl)
request.on('response', (res) => {
process.stdin.pipe(request)
})
request.end() // If I emit this, nothing happens. If I keep this, I get the below error
and try to run it like so: node request.js < /tmp/text-input, I'm getting the following error:
node:events:368
throw er; // Unhandled 'error' event
^
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at new NodeError (node:internal/errors:371:5)
at write_ (node:_http_outgoing:748:11)
at ClientRequest.write (node:_http_outgoing:707:15)
at ClientRequest.<anonymous> (/home/tomk/workspace/js-playground/http.js:17:7)
at ClientRequest.emit (node:events:390:28)
at HTTPParser.parserOnIncomingClient (node:_http_client:623:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
at Socket.socketOnData (node:_http_client:487:22)
at Socket.emit (node:events:390:28)
at addChunk (node:internal/streams/readable:324:12)
Emitted 'error' event on ClientRequest instance at:
at emitErrorNt (node:_http_outgoing:726:9)
at processTicksAndRejections (node:internal/process/task_queues:84:21) {
code: 'ERR_STREAM_WRITE_AFTER_END'
}
I want to pipe my stdin to an http server the same way I can with curl -T -. What is wrong with my request code?
Short answer
To send chunked encoding messages in node, use the POST method:
let request = http.request(url, { method: 'POST' })
process.stdin.pipe(request)
Edit: A more straigt forward approach
Or, to send any request method with chunked encoding:
let request = http.request(url)
request.setHeader("transfer-encoding", "chunked")
request.flushHeaders()
process.stdin.pipe(request)
Slightly longer (yet partial) answer
I opened a listening netcat (listen on plain tcp) like so nc -l 9090 to view how the request from curl differs from my code and found a few key differences in the headers.
In curl, the header Transfer-Encoding: chunked appeared, but was missing from the request my code sent out. Also, my code had a header Connection: closed
I logged the request object and found that useChunkedEncodingByDefault is set to false, which was confusing given the quote from the nodejs http docs:
Sending a 'Content-Length' header will disable the default chunked encoding.
Implying that it should be the default.
But then I found this in the source of node
if (method === 'GET' ||
method === 'HEAD' ||
method === 'DELETE' ||
method === 'OPTIONS' ||
method === 'TRACE' ||
method === 'CONNECT') {
this.useChunkedEncodingByDefault = false;
} else {
this.useChunkedEncodingByDefault = true;
}
Edit
To send chunked encoding anyway, I (eventually) found that I need to explicitly add the Transfer-Encoding: Chunked header explicitly:
request.setHeader("transfer-encoding", "chunked")
# and then
request.flushHeaders()
So, in conclusion, node doesn't allow sending send by default GET requests with chunked encoding, but curl does. Odd, and unfortunately not documented (as far as I could find), but the important thing I got it working

ERROR event: Not Found; when triggering event from another event : probot app built in nodejs

I am building a GitHub app in probot and nodejs and trying to update the PR whenever a pull_request.synchronize event occurs. I know that doesn't make much sense, but the point is I want to be able to update the PR upon the occurrence of a certain event.
app.on('pull_request.synchronize', async context => {
console.log('---------------------- on pull_request.synchronize, body of PR : ', context.payload.pull_request)
console.log('State of PR after pull_request.synchronize event :---------------', context.payload.pull_request.state)
await context.github.pulls.update({
owner:context.payload.repository.owner,
repo :context.payload.repository.name,
pull_number :context.payload.pull_request.number,
title:'updated Pull Request Title: Success',
body:'updated Pull Request Body: Success',
})
})
I get below error every time :
ERROR event: Not Found (id=ssjsk-dd-sdfsdfs-fsfs-fsfsfsffsd)
HttpError: Not Found
at response.text.then.message (C:\GitWorkspace\user\GitHubApps\github-app-name\node_modules\#octokit\request\dist-node\index.js:66:23)
at process._tickCallback (internal/process/next_tick.js:68:7)
--
event: {
"event": "pull_request.synchronize",
"id": "2332-3131-131-31231313-1313232144142",
"installation": 3352243,
"repository": "user/Demo-Repo"
}
06:24:58.460Z ERROR probot: Not Found
HttpError: Not Found
at response.text.then.message (C:\GitWorkspace\user\GitHubApps\github-app-name\node_modules\#octokit\request\dist-node\index.js:66:23)
at process._tickCallback (internal/process/next_tick.js:68:7)
06:24:58.472Z INFO http: POST / 500 - 519.93 ms (id=3424234-43242-42423478a-4242-42342342)
06:24:58.476Z ERROR probot: Internal Server Error
Error: Internal Server Error
at Request.callback (C:\GitWorkspace\user\GitHubApps\github-app-name\node_modules\superagent\lib\node\index.js:706:15)
at IncomingMessage.parser (C:\GitWorkspace\user\GitHubApps\github-app-name\node_modules\superagent\lib\node\index.js:916:18)
at IncomingMessage.emit (events.js:203:15)
at IncomingMessage.EventEmitter.emit (domain.js:448:20)
at endReadableNT (_stream_readable.js:1129:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
Why is it unable to get the detail out of context, when in fact the context does get printed out by my console statement and contains the pull_request details.
I am just a beginner in both probot and nodejs so I am not sure if this has anything to to do with the syntax.
The problem is that context.payload.repository.owner is an object, compare the example payload at https://developer.github.com/v3/activity/events/types/#webhook-payload-example-28
Try to replace it with context.payload.repository.owner.login.
Probot also provides a convenience method which returns and { owner, repo } object based on the context, and you can pass extra parameters to it
Try the following
await context.github.pulls.update(context.repo({
pull_number: context.payload.pull_request.number,
title:'updated Pull Request Title: Success',
body:'updated Pull Request Body: Success',
}))

Node server is getting down with error buffer.js:488 throw new Error('toString failed');

We have an app build on nodejs (Server A), which is fetching data from another nodejs app(Server B) using REST API's.
Similarly for downloading files Server A make request on Server B and Server B fetches files from AWS S3 and pipes the data back to server A.
This system works perfectly and we are able to download files upto 1GB.
Issue is sometimes Server A crashes with below error:
buffer.js:488
throw new Error('toString failed');
^
Error: toString failed
at Buffer.toString (buffer.js:488:11)
at Request.<anonymous> (/var/www/sky/portal/node_modules/request/request.js:1145:39)
at emitOne (events.js:95:20)
at Request.emit (events.js:182:7)
at IncomingMessage.<anonymous> (/var/www/sky/portal/node_modules/request/request.js:1091:12)
at IncomingMessage.g (events.js:273:16)
at emitNone (events.js:85:20)
at IncomingMessage.emit (events.js:179:7)
at endReadableNT (_stream_readable.js:913:12)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)
Server A is calling server B using below code:
exports.downloadAsset = function(req, res, assetUrl, cb) {
res.cookie('fileDownload', 'true');
request(
config.api.baseUrl + assetUrl
,
function(error, response, body) {
}
).pipe(res);
}
We are not sure from where app is getting crashed.
Please suggest.
The Solution I used to sort above issue:
I had to modify our flow. We were streaming data from Server B to Server A, but now Server B is sending signed URL to Server A and file is being downloaded directly from S3 using that signed url.

Unexpected meteorjs error Can't render headers after they are sent to the client

http.js:732
throw new Error('Can\'t render headers after they are sent to the client.'
^
Error: Can't render headers after they are sent to the client.
at ServerResponse.OutgoingMessage._renderHeaders (http.js:732:11)
at ServerResponse.writeHead (http.js:1153:20)
at ProxyServer.<anonymous> (/home/ec2-user/.meteor/packages/meteor-tool/.1.1.4.1tjewoi++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/tools/run-proxy.js:96:21)
at ProxyServer.emit (/home/ec2-user/.meteor/packages/meteor-tool/.1.1.4.1tjewoi++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/http-proxy/node_modules/eventemitter3/index.js:100:27)
at ClientRequest.proxyError (/home/ec2-user/.meteor/packages/meteor-tool/.1.1.4.1tjewoi++os.linux.x86_64+web.browser+web.cordova/mt-os.linux.x86_64/dev_bundle/lib/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:140:16)
at ClientRequest.emit (events.js:117:20)
at Socket.socketOnData (http.js:1593:9)
at TCP.onread (net.js:528:27)
Meteor server throws this error or Production instance only. I'm running the same repository in Staging and it runs smoothly. Meteor throws this error even when nobody is interacting with the client (Crons run on server during regular intervals). I'm not able to figure out the reason. People have faced this problem for different issues, but I didn't find it familiar with my case.
I suspect this code is throwing this error. Not sure though
updateFunction = function(event) {
var res = Meteor.http.call("GET", "http_url"+ event);
var contents = EJSON.parse(res.content).tracks["0"];
if(!contents) return;
var events = [];
contents.map(function(ele){
if(ele.type == "snap") {
ele._id = ele.id;
delete ele.id;
events.push(ele);
}
});
CollectionName.upsert(event,{"$set":{"data": events}});
}

Reject invalid/unexpected request on sockets

How do I make sure that any invalid or unwanted request that isn't following sockets protocol gets rejected and my socket aren't closed.
Consider a simple example :
var net = require('net');
net.createServer(function(socket) {
socket.on('data', function(data) {
var parsedData = JSON.parse(data.toString());
});
}).listen(5555);
server.listen(1337, '127.0.0.1');
and try visiting that port from browser , and again look at terminal. It produces following error :
undefined:1
GET / HTTP/1.1
^
SyntaxError: Unexpected token G
at Object.parse (native)
at Socket.<anonymous> (/home/agauniyal/projects/watch/watch.js:115:35)
at Socket.emit (events.js:107:17)
at readableAddChunk (_stream_readable.js:163:16)
at Socket.Readable.push (_stream_readable.js:126:10)
at TCP.onread (net.js:529:20)
Now I fully agree that visiting that port is a mistake of mine , but closing a socket on different format rather than rejecting that request isn't the ideal way to do it. And what if someone tries to access a webserver at that port only realizing later that there wasn't any at 8080.
So how should I make sure socket isn't closed and rather rejects that request?
EDIT : This is possibily happening because I'm calling JSON.Parse(data.toString()) method on data being received from socket and http headers aren't being parsed by that method.
Entire problem rose because I was getting socket response , converting it to string and then parsing it as JSON. Now in my case , I've either to gurantee that incoming data is valid JSON string or find that a browser is sending request to that socket.
For a real-quick-fix , I am extracting first word from incoming data and checking it to be 'GET' ( which a browser sends ) , and if it evaluates to true , just doing nothing else proceeding as I was earlier. Catching JSON.parse error is better solution too.

Resources