Incorrect Header Check when using zlib in node.js - node.js

Question

I am trying to send a simple HTTP POST request, retrieve the response body.Following is my code. I am getting
Error: Incorrect header check
inside the "zlib.gunzip" method. I am new to node.js and I appreciate any help.
;
fireRequest: function() {
var rBody = '';
var resBody = '';
var contentLength;
var options = {
'encoding' : 'utf-8'
};
rBody = fSystem.readFileSync('resources/im.json', options);
console.log('Loaded data from im.json ' + rBody);
contentLength = Buffer.byteLength(rBody, 'utf-8');
console.log('Byte length of the request body ' + contentLength);
var httpOptions = {
hostname : 'abc.com',
path : '/path',
method : 'POST',
headers : {
'Authorization' : 'Basic VHJhZasfasNWEWFScsdfsNCdXllcjE6dHJhZGVjYXJk',
'Content-Type' : 'application/json; charset=UTF=8',
// 'Accept' : '*/*',
'Accept-Encoding' : 'gzip,deflate,sdch',
'Content-Length' : contentLength
}
};
var postRequest = http.request(httpOptions, function(response) {
var chunks = '';
console.log('Response received');
console.log('STATUS: ' + response.statusCode);
console.log('HEADERS: ' + JSON.stringify(response.headers));
// response.setEncoding('utf8');
response.setEncoding(null);
response.on('data', function(res) {
chunks += res;
});
response.on('end', function() {
var encoding = response.headers['content-encoding'];
if (encoding == 'gzip') {
zlib.gunzip(chunks, function(err, decoded) {
if (err)
throw err;
console.log('Decoded data: ' + decoded);
});
}
});
});
postRequest.on('error', function(e) {
console.log('Error occured' + e);
});
postRequest.write(rBody);
postRequest.end();
}

Answer 1

response.on('data', ...) can accept a Buffer, not just plain strings. When concatenating you are converting to string incorrectly, and then later can't gunzip. You have 2 options:
1) Collect all the buffers in an array, and in the end event concatentate them using Buffer.concat(). Then call gunzip on the result.
2) Use .pipe() and pipe the response to a gunzip object, piping the output of that to either a file stream or a string/buffer string if you want the result in memory.
Both options (1) and (2) are discussed here: http://nickfishman.com/post/49533681471/nodejs-http-requests-with-gzip-deflate-compression

Answer 2

In our case we added 'accept-encoding': 'gzip,deflate' to the headers and code started working (solution credited to Arul Mani):
var httpOptions = {
hostname : 'abc.com',
path : '/path',
method : 'POST',
headers : {
...
'accept-encoding': 'gzip,deflate'
}
};

Answer 3

I had this error trying to loop with fs.readdirSync but there is a .Dstore file so the unzip function was applied to it.
Be careful to pass only .zip/gz
import gunzip from 'gunzip-file';
const unzipAll = async () => {
try {
const compFiles = fs.readdirSync('tmp')
await Promise.all(compFiles.map( async file => {
if(file.endsWith(".gz")){
gunzip(`tmp/${file}`, `tmp/${file.slice(0, -3)}`)
}
}));
}
catch(err) {
console.log(err)
}
}

Answer 4

In addition to #Nitzan Shaked's answer, this might be related to the Node.js version.
What I experienced is that https.request (OP uses http.request, but it may behave the same) already decompresses the data under the hood, so that once you accumulate the chunks into a buffer, all you're left with is to call buffer.toString() (assuming utf8 as an example). I myself experienced it in this other answer and it seems to be related to Node.js version.
I'll end up this answer with a live demo of a similar working code which may come handy for future readers (it queries StackExchange API, gets a gzip compressed chunks, and then decompress it):
It includes a code that works on 14.16.0 (current StackBlitz version) - which, as I described, already decompresses the data under the hood - but not on Node.js 15.13.0,
It includes a commented-out code that works for Node.js 15.13.0 the latter but not for 14.16.0.