NodeJS consuming Transfer-Encoding chunked using https.request - node.js

Question

I'm making a GET request to an API that responds with Transfer-Encoding: chunked.
It's an https request, I've attached on('data') listener to it which then adds the response to an array.
There are around 5 chunks incoming each of them being a Buffer depending on the response. When I concat the array and try converting it to a String I get the weirdly decoded reply which looks to be encoded. I think it might be due to the request being HTTPS but I'm not sure, below's my code for decoding it
response.on('data', (data) => {
// Data is sent in binary chunks, let's add them up to an array
binaryDataArray.push(data)
}).on('end', () => {
const buffer = Buffer.concat(binaryDataArray);
try{
const formattedData = JSON.parse(buffer.toString('utf8'));
resolveRequest(formattedData);
}catch(error){
setTimeout(makeRequest, 5000);
}
});
Any help would be appreciated.

Answer 1

Check your res.headers['content-encoding'], the response may be encoded by gzip.
So you need to unzip the response buffer like this:
response.on('data', (data) => {
// Data is sent in binary chunks, let's add them up to an array
binaryDataArray.push(data)
}).on('end', () => {
const buffer = Buffer.concat(binaryDataArray);
zlib.gunzip(Buffer.concat(respData), function (err, decoded) {
if (err) throw err;
const formattedData = decoded.toString()
resolveRequest(formattedData)
});
});