I need to upload a gzipped file. For performance in case my string gets too big I decided to use streams but ran into an issue with the server requiring a content-length header which cannot be calculated as the gzipping is inline. I then decided to use chunked transfer but am not sure if I am either not doing this correctly or if the server will simply not accepting streams/chunks as it still returns an error about needing a content-length header.
Here's the bit of the code:
const gzip = zlib.createGzip()
let stream = createStream(string) // I also use files hence the streaming
.pipe(gzip) )
.pipe(request.put(url, {
headers: {
'Transfer-Encoding': 'chunked',
'x-ms-blob-type': 'blockblob'
}
}))
Response:
Content-Length HTTP header is missing
I've also played around with adding other headers such as:
'Content-Type': 'application/javascript'
'Content-Encoding': 'gzip'
Is my only option to just forgo streaming or gzip out of the flow and calculate length that way? I can't tell if I am missing something or of the server is being persnickety.
Related
I am trying to stream a large xml file from express to a client, and I have not yet found out how to send until the file is done processing on the server, and res.end() is called.
The xml file is build with xmlbuilder-js. It has a callback that receives document chunks, in which I am trying to send using response.write(chunk).
res.writeHead(200, {
'Content-Type': 'text/xml',
'Transfer-Encoding': 'chunked'
})
xmlToReturn = xmlBuilder.begin({
writer: {
pretty: true,
}
}, function(chunk) {
res.write(chunk)
}).dec('1.0', 'UTF-8', true)
...
res.end()
The callback works as expected, it shows the data chunks coming through.
I have tried:
changing the content-type on the response to, for example,
'application/octet-stream'
using res.flush() after calling
res.write(), or doing that periodically
experimenting with other headers
In all cases, if I can get the response to send, the client never receives the start of it until res.end() is called. What do I need to do so that express starts delivering the content as it flows through the callback?
I've explored questions and posts like this, which suggest my approach is correct but I am doing something wrong, or streaming is not working in express possibly due to other modules or middleware.
I am using the infamous request library to send requests.
One of those requests requires me to send the header multipart/mixed; boundary={myboundary}.
Request is using the form-data library for such requests but it does not set the Content-Type header properly. Therefore I would need to set it like this:
let req = request.post({url: "https://..."}, formData: formData)
req.setHeader('Content-Type', `multipart/mixed; boundary=${req.form().getBoundary()}`)
Sadly I can't add/alter any headers after firing the request. Therefore I want to know whether there is a way to intercept the sending so I can change the header?
You will need to use the multipart option instead of formData to use other, arbitrary multipart/* content types. Each object in the multipart array contains the headers to send in that part. The one exception is the body property which is used as the actual body of that part.
request.post({
url: 'https://...',
multipart: [
{ 'X-Foo-Header': 'bar', body: 'baz' },
// ...
],
headers: { 'Content-Type': 'multipart/mixed' }
});
The boundary should be automatically appended for an existing, explicit Content-Type header. This request test explicitly tests for this behavior.
I'm trying to upload node stream data to S3 via a signed url, therefore it is vital that the request contains the Content-Length and Content-Type header.
The data I want to upload is provided via a stream
var request = require('request')
var myReadableStream = ...fromSomeTranscoding()..;
myReadableStream.pipe(request({
method: 'PUT',
headers : {
'Content-Length' : ???????
'Content-Length' : 'image/png'
},
url: mySignedUploadUrl
}))
As I cannot access the stream length, I can do a workaround by piping it to a file to check its length and then in a second step pipe the file stream, but I guess there must be a better way to achieve this
I'm trying to send multiple binary files to a web service in a single multipart/mixed POST but can't seem to get it to work... target server rejects it. One thing I noticed is Node is trying to do the encoding as chunked, which I don't want:
POST /SomeFiles HTTP/1.1
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=123456789012345
Host: host.server.com
Connection: keep-alive
Transfer-Encoding: chunked
How do I get it to stop being chunked? Docs say that can be disabled by setting the Content-Length in the main request header but I can't set the Content-Length since, among other reasons, I'm loading one file at a time -- but it shouldn't be required either since it's multipart.
I think the rest is OK (excepting it's not chunked, unless the req_post.write() is doing that part), e.g. for the initial part I:
var req_post = https.request({
hostname: 'host.server.com',
path: '/ManyFiles',
method: 'POST',
headers: {
'MIME-Version': '1.0',
'Content-Type': 'multipart/mixed; boundary=' + boundary
}
},...);
and then pseudo-code:
while ( true ) {
// { get next file here }
req_post.write('\r\n--' + boundary + '\r\nContent-Type: ' + filetype + '\r\nContent-Length: ' + filesize + '\r\n\r\n');
req_post.write(chunk);// filesize
if ( end ) {
req_post.write('\r\n--' + boundary + '--\r\n');
req_post.end();
break;
}
}
Any help/pointers is appreciated!
The quick answer is you cannot disable chunked without setting content-length. The chunked encoding was introduced for cases where you do not know the size of the payload when you start transmitting. Originally, content-length was required and the recipient knew it had the full message when it received content-length bytes. Chunked encoding removed that requirement by transmitting mini-payloads each with a content-length, followed by a zero-size to denote completion of the payload. So, if you do not set the content-length, and you do not use the chunked methodology, the recipient will never know when it has the full payload.
To help solve your problem, if you cannot send chunked and do not want to read all the files before sending, take a look at fs.stat. You can use it to get the file size without reading the file.
I'm trying to query a webservice which answers with plain text. The text often has german umlauts in it. In the received stream the umlauts are broken. Any ideas what am I doing wrong?
Regards,
Torsten
Here is the sample code:
var request = require('request');
var uri = <anUriWithUserId>;
request(uri, {encoding: 'utf8','content-type': 'text/plain; charset=UTF-8'},
function (error, response, body)
{
console.log("encoding: " + response.headers['content-encoding']);
console.log("type: " + response.headers['content-type']);
console.log(body);
});
And the response:
encoding: undefined
type: text/plain
error=0
---
asin=
name=Eistee
detailname=Pfanner Der Gr�ne Tee, Zitrone - Kaktusfeige, 2,0 l
vendor=Hermann Pfanner Getr�nke GmbH, Lauterach, �sterreich
maincat=Getr�nke, Alkohol
When you set the encoding option in your request call, you advise the request module to decode the response body with this encoding. In this way you ignore the encoding used by the webservice, wich may or may not be utf-8. You need to find out wich encoding was used be the webservice and use that.
Depending on how complient the webservice you could also try to set the Accept-Charset: utf-8 header.
As your output shows, the webservice doesn't provide the used encoding in the Content-Type header, which is a bad habbit imho.
Sidenote: Content-Encoding isn't for charset, but for compression, gzip migh be a valid value for it.