How to pipe from standard input to an http request in NodeJS? - node.js

Question

I have an http server listening on port 9090 - piping the request to stdout like so:
let server = http.createServer((req, res) => {req.pipe(process.stdout)})
server.listen(9090)
When I send it something with curl like so:
curl -XGET -T - 'http://localhost:9090' < /tmp/text-input
it works, and I see the output on the server's terminal
but when I try the following in node:
const http = require('http')
const nurl = new URL("http://localhost:9090")
let request = http.request(nurl)
request.on('response', (res) => {
process.stdin.pipe(request)
})
request.end() // If I emit this, nothing happens. If I keep this, I get the below error
and try to run it like so: node request.js < /tmp/text-input, I'm getting the following error:
node:events:368
throw er; // Unhandled 'error' event
^
Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at new NodeError (node:internal/errors:371:5)
at write_ (node:_http_outgoing:748:11)
at ClientRequest.write (node:_http_outgoing:707:15)
at ClientRequest.<anonymous> (/home/tomk/workspace/js-playground/http.js:17:7)
at ClientRequest.emit (node:events:390:28)
at HTTPParser.parserOnIncomingClient (node:_http_client:623:27)
at HTTPParser.parserOnHeadersComplete (node:_http_common:128:17)
at Socket.socketOnData (node:_http_client:487:22)
at Socket.emit (node:events:390:28)
at addChunk (node:internal/streams/readable:324:12)
Emitted 'error' event on ClientRequest instance at:
at emitErrorNt (node:_http_outgoing:726:9)
at processTicksAndRejections (node:internal/process/task_queues:84:21) {
code: 'ERR_STREAM_WRITE_AFTER_END'
}
I want to pipe my stdin to an http server the same way I can with curl -T -. What is wrong with my request code?

Answer 1

Short answer
To send chunked encoding messages in node, use the POST method:
let request = http.request(url, { method: 'POST' })
process.stdin.pipe(request)
Edit: A more straigt forward approach
Or, to send any request method with chunked encoding:
let request = http.request(url)
request.setHeader("transfer-encoding", "chunked")
request.flushHeaders()
process.stdin.pipe(request)
Slightly longer (yet partial) answer
I opened a listening netcat (listen on plain tcp) like so nc -l 9090 to view how the request from curl differs from my code and found a few key differences in the headers.
In curl, the header Transfer-Encoding: chunked appeared, but was missing from the request my code sent out. Also, my code had a header Connection: closed
I logged the request object and found that useChunkedEncodingByDefault is set to false, which was confusing given the quote from the nodejs http docs:
Sending a 'Content-Length' header will disable the default chunked encoding.
Implying that it should be the default.
But then I found this in the source of node
if (method === 'GET' ||
method === 'HEAD' ||
method === 'DELETE' ||
method === 'OPTIONS' ||
method === 'TRACE' ||
method === 'CONNECT') {
this.useChunkedEncodingByDefault = false;
} else {
this.useChunkedEncodingByDefault = true;
}
Edit
To send chunked encoding anyway, I (eventually) found that I need to explicitly add the Transfer-Encoding: Chunked header explicitly:
request.setHeader("transfer-encoding", "chunked")
# and then
request.flushHeaders()
So, in conclusion, node doesn't allow sending send by default GET requests with chunked encoding, but curl does. Odd, and unfortunately not documented (as far as I could find), but the important thing I got it working