Gzip decompression of http Response - node.js

Using request module, I am trying to fetch response from a web service which has following header in the API request:
accept-encoding : gzip
and correspondingly, following header in the response :
content-encoding : gzip
When I am trying to decompress the response(get the correct readable response) using zlib(referred here), I am unable to do so.
Code Snippet :
var options = {
url: url,
qs: params.qparams,
method: params.method,
json: params.body,
headers: {
'api_key': configkey,
'Content-Type': 'application/json',
'Accept-Encoding': 'gzip'
},
timeout: constants.request_timeout
};
request(options, function(err, response, body) {
var encoding = response.headers['content-encoding']
if (encoding && encoding.indexOf('gzip') >= 0) {
zlib.gunzip(body, function(err, dezipped) {
//ERROR : { [Error: incorrect header check] errno: -3, code: 'Z_DATA_ERROR' }
var json_string = dezipped.toString('utf-8');
var json = JSON.parse(json_string);
console.log('\nJSON ::\n',json);
});
} else {
console.log('\n\nRESPONSE IS NOT GZIPPED!');
}
}
I am getting an error here(as commented in the code), using zlib.
I could not figure out as where is it going wrong, tried with multiple npm modules like unzipResponse and compress-buffer and tried different approaches as well as suggested at various places for handling gzip.
If someone can help out in resolving this, I'll be really thankful.

I have got a solution as need to add one more key to the options object as :
var options = {
url: url,
qs: params.qparams,
method: params.method,
json: params.body,
headers: {
'api_key': configkey,
'Content-Type': 'application/json',
'Accept-Encoding': 'gzip'
},
timeout: constants.request_timeout,
encoding: null
};
If someone has a better approach to perform the decompression, please add-on.

Related

Axios returning binary data instead of XML

Been spending the past few hours figuring out why Axios is doing this.
I tried to do this in the request library in nodeJS and it works fine, but Axios isn't.
Essentially what i'm doing is sending a request of XML data:
var options = {
'method': 'POST',
'url': 'https://rvices.svc',
'headers': {
'Content-Type': 'text/xml; charset=utf-8',
'SOAPAction': 'http://etProject'
},
data: xmlData};
with XMLData looking simliar to :
let xmlData = `<?xml version="1.0" encoding="UTF-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:com="http://p" xmlns:pws="http://"
etc etc (it has mostly private data in it so I stripped most of it out).
when I try to use axios to get this
const testData = await axios(options)
i get returned a whole lot of code that looks like this:
'���_\x1B�\f�10�\x17�Ч�{O�\x00��5h�S�������\x0F���s�(+Ғ�\x0F�����m�\x15\x01\x13��6b��\x06%\x00��\x15p8<;��W�4����\x0B���\x01���e�\x7FvZ�{���Ï������\x06��-�z��\x01}�!�r�A�\x13��\x11O�w6ũ���{�\x03����;{����\x01\x7Fy��KoՎ���\x1Bߚe��W��mЇ�qD�a��[�7Ӄ���#��F<\x1C/mF�{\x03�h��#�\x16�\x11\x1F\x1F�L9\x0FM\x8A\x0E�\x
17�h���\x03�4�7�f=bj*8�p�\x13_�\x17�5���_�Ӑ�|M>����\r��F�8q�iE�#��\x0E?�v�������O�xq3���x�Q�튱\x1F?G&HG9��6���V\x1B⫯Ev\x01rc\x13\x10�\'�7��`�Ii��x�~LM6�#˒74##�����f�*\x7F\x16(5|\x1CWl��\x07\t\x1F��z�\x15\x00\x1B��4�\x13���LCTG�\x1FI�����\fec�h\x02�~��i`�:Ғ�\x0F���y\b#�]V��g��Ӈ�\x14|���6~\x19~c`�/�O���M\x01��k\x
10�\'+���\x07S\r?|��T�A�\x0FӒ�\x0F��ܷ\'.s�!>�tbX\x05�\fs\x18�\r�"`���\x10lV٠\x05#ܲ�\x02\x0E\x07h���\n' +
'���[�7}�>54 r�����ʦ\x15�\x17��\x0E:
that is the right amount of characters (100k +) but jumbled
compared to doing this with request which returns the xml back I expect ala:
</b:ProjectTaskTypeDetail></b:PwsProjectTaskTypeElement><b:PwsProjectTaskTypeElement><b:ProjectTaskTypeDetail><b:ExternalSystemIdentifier i:nil="true"/><b:ProjectTaskTypeId i:nil="true"/><b:ProjectTaskTypeUid>5776</b:ProjectTaskTypeUid><b:ProjectTaskTypeName>Faon</b:Proj
ectTaskTypeName>
one thing I noticed is axios is breaking my request up into multiple lines like this:
'<com:PwsProjectRef><com:ProjectCode>201268</com:ProjectCode></com:PwsProjectRef>\n' +
'\n' +
'<com:PwsProjectRef><com:ProjectCode>210115-01</com:ProjectCode></com:PwsProjectRef>\n' +
'\n' +
even though there's no \n's in my request or breaks like that.
So i'm wondering if anyone has ran into this before and knows how to solve it?
Request is working but request (from what I can tell?) doesn't work with asynch code (i'm probably wrong about this)
Sorry for the vagueness!
You should be using the responseType config option to set the expected response which reflects the Accept HTTP header and not the Content-Type one:
const options = {
method: 'POST',
url: 'https://rvices.svc',
headers: {
'Content-Type': 'text/xml; charset=utf-8',
'SOAPAction': 'http://etProject'
},
data: xmlData,
responseType: 'document',
responseEncoding: 'utf8'
};
const testData = await axios(options);
The response is being returned in a compressed binary format, but
Axios does not understand the compression format that is being returned from the server. Try forcing the response to a specific compression algorithm like 'deflate' which axios understands. 'gzip' may also work.
The axios 'decompress' option tells axios to automatically decompress the binary data.
var options = {
'method': 'POST',
'url': 'https://rvices.svc',
'headers': {
'Accept-Encoding': 'deflate'
'Content-Type': 'text/xml; charset=utf-8',
'SOAPAction': 'http://etProject'
},
data: xmlData,
decompress: true
};
<!-- end snippet -->
In v1.2.1, it is fixed this error.
try with this code
Save as get-data.js file
const axios = require("axios");
const getData = async () => {
try {
const resp = await axios.get('your xml URL',
{
headers: {
'Accept-Encoding': 'application/xml',
}
}
);
console.log(resp.data);
} catch (err) {
// Handle Error Here
console.error(err);
}
};
getData()
npm install axios
node get-data.js

undefined:1 syntax error: unexpected token < in json at position 0

let options = {
'method': 'GET',
'url': `https://cdn-api.co-vin.in/api/v2/appointment/sessions/public/calendarByPin?pincode=${row.pincode}&date=${day}`,
'headers': {
}
};
request(options, function (error, re) {
if (error) {
console.log('error in request',error)
}
else if(re) {
let data = JSON.parse(re.body);
console.log(data)
}
})
It's working fine in localhost, but when I'm hosting it on digital ocean server, sometime the code executes sometimes it doesn't, it happens with the same code. Please help
You are getting html code instead of json in your api response. Add this to headers
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json',
}
also try checking the html that you got, it can possibly provide you with a clue why you are getting it

How to make a POST with XML body/content-type header using request promise native in Node?

I need to make an XML POST (I know don't ask it's government...) and I can't get it to work with node request promise native.
I have tried turning my string of XML into a buffer, String(), .toString etc. The POST works if I turn json:true so I don't think it's a networking issue ( when passed a xml string with json true it sends a json like { 'variablename': 'stringed XML that I want to send as the body' } )
Here is what I'm working with. I've been banging my head for a while here any help appreciated.
Would ideally be promise/async.
Maybe I should look to find a XMLHttp request npm module?
var request_promise_native_options = {
method: 'POST',
uri: 'http://requestbin.fullcontact.com/198flbs1',
headers: {
'User-Agent': 'Request-Promise',
'Content-Type': 'text/xml'
//'Content-Length': Buffer.byteLength(my_xml_string) //I've tried multiple ways to use this
},
body: {
my_xml_string //also tried many ways here Buffer, String() etc
},
json: false // automatically stringifys body to json if true
};
request_promise(request_promise_native_options)
.then(function (response) {
console.log("success");
})
.catch(function (err) {
console.log(err);
})
Kudos to #kevin-b to help me see the obvious. Simply remove {}
var request_promise_native_options = {
method: 'POST',
uri: 'http://requestbin.fullcontact.com/198flbs1',
headers: {
'User-Agent': 'Request-Promise',
'Content-Type': 'text/xml'
'Content-Length': Buffer.byteLength(my_xml_string)
},
body: my_xml_string,
json: false // automatically stringifys body to json if true
};

Request-Promise-Native ignores specified headers

I am somewhat new to Nodejs. I am working on a pair of Microservices, and I need one to post data to the other. I am using the request-promise-native library.
My code is to make the call is like this:
const options = {
method: 'POST',
uri: url,
formData: {
command: command,
version: version,
session_id: sid,
aicc_data: data
},
headers: {
'content-type' : 'application/x-www-form-urlencoded'
}
}
rp(options)
However, when I inspect the request as it come in to the other server, the header I have specified does not appear.
headers: { 'content-type': 'multipart/form-data; boundary=--------------------------395968157759002211606136',
host: 'localhost:9000',
'content-length': '513',
connection: 'close' }
What am I doing wrong?
options includes a formData object which enforces multipart/form-data.
You should add the form object instead when you want to use application/x-www-form-urlencoded.

Setting charset in post request on nodejs

I want to send post form data to some website using euc-kr charset by request module. And I use iconv-lite module too because nodejs supported charset isn't plenty.
Anyway, The website use euc-kr charset, So I have to handle form data's encoding(Node's default charset is utf-8). But it dosen't work well, I tried to change some options so many times but I stucked untill now, So could you tell me some hints.?
// added module request, iconv-lite(extendNodeEncoding) already.
function postDocumentForm() {
//Lets configure and request
request({
url: 'http://finance.naver.com/item/board_act.nhn', //URL to hit
headers: {
'Content-Type': 'content=text/html; charset=euc-kr'
},
method: 'POST',
encoding: 'euc-kr',
form: {
code:'000215',
mode: 'write',
temp: '',
keyCount: '0',
title: "폼 데이터 중 일부가 한글일 때",
opinion: '0',
body:'인코딩이 제대로 되지 않고 있음!'
}
}, function (error, response, body) {
if (error) {
console.log(error);
} else {
iconv.undoExtendNodeEncodings();
console.log(response.statusCode, response.body);
}
});
}
And here is result, odd characters.
I tried :
euc-kr to binary
euc-kr to null
euc-kr to utf-8
delete encoding option
delete request header
Finally I got a soultion, and I solved this problem.
If you send a data as a form using request module, the module change your form encoding to utf-8 by force. So even you setted your form encoding to another charset, the module changes your charset to utf8 again. You can see that at request.js on line 1120-1130.
So, You'd better send a data by 'body' option, not 'form' option.
(as a querystring)
body: "someKey=someValue&anotherKey=anotherValue...."
Notice how it handles receiving back the body encoding
iconv = require('iconv-lite');
function postDocumentForm() {
//Lets configure and request
request({
url: 'http://finance.naver.com/item/board_act.nhn', //URL to hit
headers: {
'Content-Type': 'content=text/html; charset=euc-kr'
},
method: 'POST',
encoding: null,
form: {
code:'000215',
mode: 'write',
temp: '',
keyCount: '0',
title: "폼 데이터 중 일부가 한글일 때",
opinion: '0',
body:'인코딩이 제대로 되지 않고 있음!'
}
}, function (error, response, body) {
if (error) {
console.log(error);
} else {
console.log(response.statusCode);
var utf8String = iconv.decode(new Buffer(body), "ISO-8859-1");
console.log(utf8String);
}
});
}
After reading the source code several hours, finally I found the simple solution.
First, write your encoding function, which input a string and output a encoded string:
const urlencode = require('urlencode');
function encoder(s){
return urlencode(s, 'gb2312');
}
Here is a Chinese encoder based on urlencode
Then just add a qsStringifyOptions option while posting:
request({
url: 'http://finance.naver.com/item/board_act.nhn', //URL to hit
headers: {
'Content-Type': 'content=text/html; charset=euc-kr'
},
method: 'POST',
encoding: null,
form: {
code:'000215',
mode: 'write',
temp: '',
keyCount: '0',
title: "폼 데이터 중 일부가 한글일 때",
opinion: '0',
body:'인코딩이 제대로 되지 않고 있음!'
},
qsStringifyOptions: {
encoder: encoder
}
});

Resources