How to best extract content body from a nodejs net response? - node.js

Question

Using the following code:
var net = require("net");
var client = new net.Socket();
client.connect(8080,"localhost",function() {
client.write("GET /edsa-jox/testjson.json HTTP/1.1\r\n");
client.write("Accept-Encoding: gzip\r\n");
client.write("Host: localhost:8080\r\n\r\n");
}
);
client.on("data", function(data) {
console.log(data.toString("utf-8", 0, data.length));
});
I get the following response:
HTTP/1.1 200 OK
Date: Thu, 20 May 2021 22:45:26 GMT
Server: Apache/2.4.25 (Win32) PHP/5.6.30
Last-Modified: Thu, 20 May 2021 20:14:17 GMT
ETag: "1f-5c2c89677c5c7"
Accept-Ranges: bytes
Content-Length: 31
Content-Type: application/json
{"message":"message from json"}
And this response is shown in the console immediately. But since it is coming from the "data" event I guess it would have been coming in chunks if the response was bigger.
So I also tested with the following (all else equal):
var data="";
client.on("data", function(d) {
console.log("1");
data += d.toString("utf-8", 0, d.length);
});
client.on("end", function(d) {
console.log(data);
});
Thinking that I could use the event "end" to be sure that I had the full set of data before doing something else. Which I guess worked, but the unexpected thing was that the "1" was shown immediately but then it took a couple of seconds before the "end" event was triggered.
Question 1) Why is there such a delay with "end" event compared to the last executed "data" event? Is there a better way to do it?
Question 2) Having the above response, which contains both a bunch of headers aswell as a content body. What approach is the best approach to extract the body part?
Note, I want to do this with the net library not the fetch nor the http libraries (or any other abstractions). I want it to be as fast as possible.

Answer 1

i can only see two valid reasons to do it all by hand :
extreme speed needs => then you should consider using "go" or another compiled language
learning (always interesting)
I would recomend you to use express, or any other npm package to deal with everything without reinventing the wheel.
however, i'll help you with what i know :
First thing is to properly decode ut8 strings. You need to use string_decoder because if the data chunk is incomplete, and you call data.toString('utf8'), you will have a mangled character appended. doesn't happen often but hard to debug.
here is a valid way to do it :
const { StringDecoder } = require('string_decoder');
var decoder = new StringDecoder('utf8');
var stdout = '';
stream.on('data', (data) => {
stdout += decoder.write(data);
});
https://blog.raphaelpiccolo.com/post/827
then to answer your questions :
i dont know, may be related to gzip. The server can be slow to stop the connection, or it's the client's fault. Or the network itself. I would try with other clients / servers to be sure, and start profiling.
you need to read http specifications to handle all edge cases (http1/websockets/http2). But i think you are lucky, headers are always separated from the body by a double newline. then if you loop through the data coming from the stream, after it's been decoded, character by character, you can search for this pattern \n\n. Anything coming after will be the body.
one special case i think about is keep alive : if the client and server are in keep alive mode the connection wont be closed between calls. you may need to parse the "Content-Length" header to know how many characters to wait for.