Pipe gzipped content without decompression to response in NodeJS - node.js

I am using the node-fetch library and it by-default decompresses the response. I want to make a POST request which returns gzipped content and I want to pipe the POST request response to the response.
The code I am currently using is here:
router.post('/getData', async(request, response) => {
fetch(`http://url:port/path`, { method: 'POST', headers: {'Accept-Encoding': 'gzip'}, body: ''})
.then(data=> {
return data.body.pipe(response);
}
}
}
I understand that the node-fetch library decompresses the data by default. I do not need this. I want to pass the compressed data directly using streams. (Like a proxy)

What worked for me was setting compress=false but then adding the header to accept gzip encoding:
fetch(url, {
compress: false,
headers: { "accept-encoding": "gzip" },
});

Related

Request with request-promise and multipart/form-data

I have to implement the following workflow:
Make request no.1, get a JSON response. Manipulate this JSON object so that it can be used as payload in the second request. Make second request.
The first request is successfully implemented. I get the correct JSON object. The manipulation part is also done and tested.
I managed to create a correct postman collection for the second request. It uses the method POST, has several headers, Content-type is multipart/form-data and the payload from request no.1 is sent as Raw (in the body, options none, form-data etc ... I choose raw).
I am trying to implement this second request but get an error no matter what I try to tweak. The following is my code:
const manipulatedObject = await this._flattenPayload(payload);
const Request = require(request-promise);
const options = {
method: 'POST',
uri: 'https://second_request_url',
formData: {
file: {
value: manipulatedObject,
},
},
headers: {
Accept: 'application/json, text/plain, */*',
Connection: 'keep-alive',
'Content-Type': 'multipart/form-data; boundary=----WebKitFormBoundaryxxxxxxxxxxxxxxxx', // this is generated dynamically and matches the value in the manipulatedObject
cookie: this.cachedCookie,
'X-XSRF-TOKEN': this.cachedToken,
},
};
Request(options)
.then((body) => {
return body;
})
.catch((error) => {
return error;
});
* The parameter payload is the response from the first request.
The error I get is this:
message=source.on is not a function, stack=TypeError: source.on is not
a function
The variable manipulatedObject is a string. Currently I am copying it's value from the successful postman request to avoid errors from the manipulation process. The random token in the Content-Type header matches the ones in the manipulatedObject string. So the data are identical to the ones I use in the postman request.
I suspect the error is in the way I send the data, but I am having trouble figuring it out. I have read about this error and it seems to be generated when an object is passed to formData, but I am sending a string so I don't understand why it happens.
The values of formData accepts only three types of elements viz. string, Buffer and Stream. Refer to:request/issues/2366
U may change formData as follows:
formData: {
file: manipulatedObject,
},
or
formData: {
file: {
value: manipulatedObject,
options: {},
},
},

Convert Binary Media Data to Buffer in Node JS

I am using a third party api which returns the image in Binary Media Data format. After getting this data, I want to upload this to Google Cloud Storage. To do this, I need to convert this data into Buffer. I've tried multiple times but failed.
I am using NodeJS, npm request module to call api to save image to google cloud storage.
Here is the code:
var binaryData = data;
var bufferData = new Buffer(data);
request({
method: "POST",
url: '/endpoint/upload',
headers: {
'cache-control': 'no-cache',
'content-type': 'multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW'
},
formData: {
filename: {
value: fileBase64,
options: {
filename: 'test.jpg',
contentType: 'image/jpeg'
}
},
}
}, function(err, response, body){
console.log(body);
})
Your post request should follow the template described in the documentation. My post request looks like this:
req = https.request({
method: "POST",
protocol: 'https:',
hostname: 'www.googleapis.com',
path: '/upload/storage/v1/b/[bucket-name]/o?uploadType=media&name=[file-name]',
headers: {
'content-type': 'image/png',
'content-length': Buffer.byteLength(data),
'authorizatoin': Bearer [bearer-token]
}
}, (res) => {
console.log(res.statusCode);
console.log(res.statusMessage);
console.log(res.headers);
}
);
It also looks like you’re lacking authentication. You need to use OAuth 2.0 for Google Cloud Storage. Make sure the Cloud Storage JSON API is enabled too.
You need to obtain your file as a stream. Here's a useful post that specifies how to do that with axios. Once you download the file in the server, you can get it as a Buffer with fs.readFile.

Sending large xml as body in request module in nodejs

I have a very large xml which gets generated using wsjs node module. it looks like
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:diag="http://diagnostic.ws.fcrewards.test.com/" xmlns:ws="http://www.w3.org/2005/08/addressing" xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd"><soapenv:Header><ws:Action>http://diagnostic.ws.fcrewards.test.com/doEcho</ws:Action><ws:To>https://ws.mastercard.com/mtf/MRS/DiagnosticService</ws:To><ws:MessageID>35b479f1-5a54-e9a4-5551-0f15507f54b8</ws:MessageID><ws:ReplyTo><ws:Address>http://www.w3.org/2005/08/addressing/role/anonymous</ws:Address></ws:ReplyTo><o:Security><u:Timestamp wsu:Id="_0" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd"><u:Created>2017-02-24T17:13:13Z</u:Created><u:Expires>2017-02-24T17:18:13Z</u:Expires></u:Timestamp><o:BinarySecurityToken ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3" EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary" u:Id="sec_0">MIID1zCCAr+gAwIBAgIRAIVAIPeDa+CgeYorExNgwDQYJKoZIhvcNAQELBQAwgYUxCzAJBgNVBAYTAkJFMRwwGgYDVQQKExNNYXN0ZUNhcmQgV29ybGR3aWRlMSQwIgYDVQQLExtHbG9iYWwgSW5mb3JtYXRpb24gU2VjdXJpdHkxMjAwBgNVBAjdjdjjdjdjjdjjdQgUFJEIE1lc3NhZ2VzIFNpssssssssssssssssssssssssIxNjIxMzcwN1owZjEbMBkGA1UEAxMSbXRmLkNIUy5wYXlGb3J3jdjjdjdjdjjdjjndkwlklwlwlwww25pbmcgU1cxDjAMBgNVBAsTBUNIUyAxMQswCQYDVQQGEwJVUzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAM675qti1N/aMP+YsPe9UIaJasISYzIpEoysKhClmWfBTqJTrcw0UWnraSOGqdPvEpgcLdW2BwnZ+T2gmcLa/BY69hW+CYm0pR0px9MPNuP77YcJCS5d6ffuHvTlkVpBq2hWVhOck0X0uhu+m2I6G8Dshi8yiBJohs+jpuf63OxVtBTemFNM5pVDdB5z7sYhkrdcNzlSoPTKNcTFfZgnj0G6iwkb4XK1mSlXbQL8jYEPPZWDdxPFl1SrSnwUoyl3r84SZrV13JstZ4YrolAm8mmKB7pU+bJ46+zpeYDRDo/YH4cxSpkyPUuwB27PtLk1J/9FgBNLfY0XQnlk/b3Mm3MCAwEAAaNgMF4wHwYDVR0jBBgwFoAU66ASd4JT9FhYrwHp4f9kBT5H2aowDAYDVR0TBAUwAwIBADAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0OBBYEFJ1obzmA1M2PIL8GumtHSIdwUrZMMA0GCSqGSIb3DQEBCwUAA4IBAQASqfFjdTt74OHwmEJVlO3pYnnHcxvDU0v7+2xTTMpWSAKmfDGf5j71UDsa6E/ymclcz/mdbsHSvyYii8TsrYK8Nyq6o2nTAQ0mhRhSpip7kjv5eQ0PtJZhl0Je7W9b3GIS1M5IbNN7xusKNHaJSg7ZIf/gUwWuBOq1CxABE0xRGY5b5e6giglP2iJwgeOHukmeGiLck3HGqZjei+T/Fnvsyy7hQ6qzGTDZg6B6yd1ojMZddmjDV8QsQfUQWM0//rvORZup98VP7I9LmrDBQuu7cPPy0Vg8VRWZC9wrM82e6IHl1uMovU12PCth1vAcSyTjuorNn5NoQg8FIoLCdgLk</o:BinarySecurityToken><Signature xmlns="http://www.w3.org/2000/09/xmldsig#"><SignedInfo><CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/><SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/><Reference URI="#_0"><Transforms><Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/></Transforms><DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/><DigestValue>qNgtbxFDtGr1QCRG4ZnGSzfWqQ+GZJkuQAHgTxcI4B8=</DigestValue></Reference></SignedInfo><SignatureValue>ZqHJTwD91tcddddddddd3uZkuChVE0BHEx55YF3kGNw1D55VbunmbNZkmxxkuBvyAMjns/DJ8/duThXhlAmyPLYhRgj4djz8smJbksago1Tb+DFT8RQ4tOVPD2TPSaC5wrwA9g9zCCpPDHHv5FmZ4TIxwkPjG37MBqOIgUF8cUkL9kXU0sIJrUQddddddddddddddddddddddddddddddddddddddddddddddddddddddwsRtcjKDksvWpMBj0oSysWi+8/ce2HnO4nOuhrvX66BDOAyPL+avL2+JNTlA==</SignatureValue><KeyInfo><o:SecurityTokenReference><o:Reference URI="#sec_0" ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3"/></o:SecurityTokenReference></KeyInfo></Signature></o:Security></soapenv:Header><soapenv:Body><diag:doEcho>Cieran</diag:doEcho></soapenv:Body></soapenv:Envelope>
when I send this request using request module of node.js
request.post({
url: ctx.url,
body: ctx.request,
headers: {
"SOAPAction": ctx.action,
"Content-Type": ctx.contentType,
"ContentLength": Buffer.byteLength(ctx.request),
"MIME-Version": "1.0",
"Connection": "Keep-Alive",
"AcceptEncoding": "gzip"
},
encoding: null,
rejectUnauthorized: true,
agentOptions: ctx.agentOptions
},
function(error, response, body) {
ctx.response = body
if (response) {
ctx.resp_headers = response.headers
ctx.resp_contentType = response.headers["content-type"]
}
if (error) ctx.error = error
else ctx.statusCode = response.statusCode
callback(ctx)
})
ctx.request has the whole xml in it as 1 big string
I always get Authorization failed error because endpoint is out rightly rejecting my request.
But if I take the xml and put it in SOAPUI and do format xml then I get the right result.
So I concluded that since i'm sending the body as 1 big xml the endpoint doesn't like my request.
What's the workaround for this?
Is there any other node module available which can format the xml or break it up in small strings?

Node JS upload file streams over HTTP

I'm switching one of my projects from request over to something a bit more light-weight (such as got, axios, or fetch). Everything is going smoothly, however, I'm having an issue when attempting to upload a file stream (PUT and POST). It works fine with the request package, but any of the other three return a 500 from the server.
I know that a 500 generally means an issue on the server's end, but it is consistent only with the HTTP packages that I'm testing out. When I revert my code to use request, it works fine.
Here is my current Request code:
Request.put(`http://endpoint.com`, {
headers: {
Authorization: `Bearer ${account.token.access_token}`
},
formData: {
content: fs.createReadStream(localPath)
}
}, (err, response, body) => {
if (err) {
return callback(err);
}
return callback(null, body);
});
And here is one of the attempts using another package (in this case, got):
got.put(`http://endpoint.com`, {
headers: {
'Content-Type': 'multipart/form-data',
Authorization: `Bearer ${account.token.access_token}`,
},
body: {
content: fs.createReadStream(localPath)
}
})
.then(response => {
return callback(null, response.body);
})
.catch(err => {
return callback(err);
});
Per the got documentation, I've also tried using the form-data package in conjunction with it according to its example and I still get the same issue.
The only difference between these 2 I can gather is with got I do have to manually specify the Content-Type header otherwise the endpoint does give me a proper error on that. Otherwise, I'm not sure how the 2 packages are constructing the body with the stream, but as I said, fetch and axios are also producing the exact same error as got.
If you want any of the snippets using fetch or axios I'd be happy to post them as well.
I know this question was asked a while ago, but I too am missing the simple pipe support from the request package
const request = require('request');
request
.get("https://res.cloudinary.com/demo/image/upload/sample.jpg")
.pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))
// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
.pipe(request.post("http://127.0.0.1:8000/api/upload/stream"))
and had to do some experimenting to find similar features from current libraries.
Unfortunately, I haven't worked with "got" but I hope the following 2 examples help someone else that are interested in working with the Native http/https libraries or the popular axios library
HTTP/HTTPS
Supports piping!
const http = require('http');
const https = require('https');
console.log("[i] Test pass-through: http/https");
// Note: http/https must match URL protocol
https.get(
"https://res.cloudinary.com/demo/image/upload/sample.jpg",
(imageStream) => {
console.log(" [i] Received stream");
imageStream.pipe(
http.request("http://localhost:8000/api/upload/stream/", {
method: "POST",
headers: {
"Content-Type": imageStream.headers["content-type"],
},
})
);
}
);
// Or any readable stream
fs.createReadStream('/Users/file/path/localFile.jpeg')
.pipe(
http.request("http://localhost:8000/api/upload/stream/", {
method: "POST",
headers: {
"Content-Type": imageStream.headers["content-type"],
},
})
)
Axios
Note the usage of imageStream.data and that it's being attached to data in the Axios config.
const axios = require('axios');
(async function selfInvokingFunction() {
console.log("[i] Test pass-through: axios");
const imageStream = await axios.get(
"https://res.cloudinary.com/demo/image/upload/sample.jpg",
{
responseType: "stream", // Important to ensure axios provides stream
}
);
console.log(" [i] Received stream");
const upload = await axios({
method: "post",
url: "http://127.0.0.1:8000/api/upload/stream/",
data: imageStream.data,
headers: {
"Content-Type": imageStream.headers["content-type"],
},
});
console.log("Upload response", upload.data);
})();
Looks like this was a headers issue. If I use the headers directly from FormData (i.e., headers: form.getHeaders()) and just add in my additional headers afterwards (Authorization), then this ends up working just fine.
For me just works when I added other parameters on FormData.
before
const form = new FormData();
form.append('file', fileStream);
after
const form = new FormData();
form.append('file', fileStream, 'my-whatever-file-name.mp4');
So that way I can send stream from my backend to another backend in node, waiting a file in multipart/form-data called 'file'

How to set POST encoding with request module on node.js?

I'm using request module on node.js but there is something problem with encoding option. beneath codes are simple post request, but I don't know how to set up encoding of form field data. I already set headers to 'Content-Type': 'application/x-www-form-urlencoded; charset=euc-kr' But it doesn't works. field data is korean, like "안녕하세요", and I should post it with euc-kr encoding. (The site takes euc-kr, not utf8)
The same program on Java application, I coded like this :
PrintWriter wr = new PrintWriter(new OutputStreamWriter(conn.getOutputStream(), "euc-kr"));
But I don't know how to in nodejs. Can anyone give some solution...?
Code Sample
//Load the request module
var request = require('request');
//Lets configure and request
request({
url: 'http://google.com', //URL to hit
headers: { 'Content-Type': 'application/x-www-form-urlencoded; charset=euc-kr' },
method: 'POST',
form: {
field1: 'data',
field2: 'data'
}
}, function(error, response, body){
if(error) {
console.log(error);
} else {
console.log(response.statusCode, body);
}
});
Finally I got a soultion, and I solved this problem.
If you send a data as a form using request module, the module change your form encoding to utf-8 by force. So even you setted your form encoding to another charset, the module changes your charset to utf8 again. You can see that at request.js on line 1120-1130.
So, You'd better send a data by 'body' option, not 'form' option.
Node doesn't support EUC-KR so you can use iconv-lite to extend the native encodings available and set the encoding option in request.
List of Natively Supported Encodings
iconv.extendNodeEncodings(); only works for node pre v4+. See here to get this working for a newer version of node.
var iconv = require('iconv-lite');
var request = require('request');
// This will add to the native encodings available.
iconv.extendNodeEncodings();
request({
url: 'http://google.com', //URL to hit
method: 'POST',
form: {
field1: 'data',
field2: 'data'
},
encoding: 'EUC-KR'
}, function(error, response, body){
if(error) {
console.log(error);
} else {
console.log(response.statusCode, body);
}
});
iconv.undoExtendNodeEncodings();

Resources