I have a GET route in express that should return a binary png image stored in mongodb. However, when I enter the url into chrome to test, the image is downloaded but the request never completes. From the Network tab in Chrome DevTools the request is just stuck in the 'pending' state. I'm only getting this problem with binary data it seems. I have plenty of other json GET requests that work just fine with send().
I am using the send() function like this:
exports.getProjectPng = (req, res) => {
Project.findById(req.params.projectId).select('project.png')
.then(project => {
res.send(project.png.buffer);
});
If I simply replace send() with end() the request completes as expected. Also, perhaps significantly, the png image is actually rendered within the browser rather than downloading as a file.
So why does end() work but send() doesn't?
If you point curl at an express server and see what the response looks like for both methods it is quite interesting. The main difference is what when we call send, the Content-Type header is populated, which is consistent with the Express docs:
When the parameter is a Buffer object, the method sets the Content-Type response header field to “application/octet-stream”, unless previously defined
It's worthwhile noting that res.send() actually calls res.end() internally at the end of the call, so the different behaviour is likely down to something that res.send does in addition to res.end.
It might be worth populating the Content-Type Header in your example to "image/png" before sending.
e.g.
res.set('Content-Type', 'image/png');
For .end():
* Connected to localhost (::1) port 8081 (#0)
> GET /downloadpng_end HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:8081
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Date: Mon, 25 Jun 2018 13:14:58 GMT
< Connection: keep-alive
< Content-Length: 69040
<
And for send():
* Connected to localhost (::1) port 8081 (#0)
> GET /downloadpng_send HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:8081
> Accept: */*
>
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: application/octet-stream
< Content-Length: 69040
< ETag: W/"10db0-KwFSGG5Ib/DQNZChAbluTiKSP0o"
< Date: Mon, 25 Jun 2018 13:15:25 GMT
< Connection: keep-alive
<
Nodejs does not handle straight binary data very well.So , thats what buffer is used to handle binary data.
End() Method
end − This event is fired when there is no more data to read. while send has no guarantee whether its completed or not. You can read more about Buffer on offical docs here
Related
I have a simple Web server, which should send a file. I took the code from another answer.
#! /usr/bin/node
const FS = require ('fs');
const HTTP = require ('http');
const server = HTTP.createServer ();
server.on ('request', (request, response) => {
switch (request.url) {
case '/':
switch (request.method) {
case 'GET':
console.log ("GET /");
let stat = FS.statSync ('index.html');
console.log (stat.size);
response.writeHead (200, { 'Content-Type': 'text/html',
'Content-Lenght': stat.size });
let index = FS.createReadStream ('index.html', 'UTF-8');
index.pipe (response);
response.end ();
return;
}
break;
}
response.writeHead (400, {});
response.end ();
});
server.listen (8080);
When I try to send a GET request with curl, I get no content. My server reports, that the index.html file has 324 bytes:
$ ./server.js
GET /
324
But curl does not show the content. There header contains the content length, but the body is missing.
$ curl -v --noproxy \* http://localhost:8080/
[...]
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: text/html
< Content-Lenght: 324
< Date: Sat, 21 Nov 2020 19:24:31 GMT
< Connection: keep-alive
< Transfer-Encoding: chunked
<
* Connection #0 to host localhost left intact
I looks as if the connection got closed before the file has been piped. Is the the error and how can I avoid it?
Remove the response.end ();. You're prematurely closing the response BEFORE the .pipe() gets to do its work (it's asynchronous so it finishes over time and returns before it's done).
In the default configuration, .pipe() will end your response for you when it's done.
You will also notice that the other answer you took this idea from did not have a response.end().
Cannot save file larger than pipe chunk size (~64K).
Using mongodb 3.4.0, Relevant node dependencies
restify 4.2.0
mongodb ^2.2.12
lodash 4.16.6
bookeeppingData = {request, id, ...meta}
clone = lodash.cloneDeep(bookkeepingData)
const {
request: req,
id: _id,
meta: metadata,
} = clone
const bucket = new mongodb.GridFSBucket(
db,
{bucketName: 'my_gridfs_collection'}
);
const uploadSteam = bucket.openUploadStreamWithId(
_id,
undefined,
{metadata}
);
req.on('data', (chunk) => {
console.log(`Received ${chunk.length} bytes of data.`);
});
req.on('end', () => {
console.log('There will be no more data.');
});
req.on('error', (e) => {
console.log('req on error', e);
});
uploadSteam.on('finish', function() {
console.log('finsish');
keepThebooks(bookkeepingData);
});
uploadSteam.on('error', (e) => {
console.log('uploadstream on error', e);
});
req.pipe(uploadSteam);
}
When sending a file smaller than the ~64K, The console output is
Received 57259 bytes of data.
There will be no more data.
finish
This is the corresponding curl --verbose output (minus the json response):
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8060 (#0)
* Server auth using Basic with user 'driver'
> POST /artifact?branch=xyz&role=PHOTO&where.lat=55&where.long=77.2&where.acc=12&sys=system&id=1234&sys=system2&id=1234b&created=2018-11-01T18:41:50.850Z&tz=-600 HTTP/1.1
> Host: localhost:8060
> Authorization: Basic xxxxxxxxx
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: image/png
> Content-Length: 57259
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, must-revalidate
< Content-Type: application/json
< Content-Length: 388
< Date: Wed, 23 Jan 2019 20:58:16 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
When sending a file larger than ~64K, the console output is:
Received 65536 bytes of data.
(That's it -- no error)
The corresponding curl --verbose output is:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8060 (#0)
* Server auth using Basic with user 'driver'
> POST /artifact?branch=xyz&role=PHOTO&where.lat=55&where.long=77.2&where.acc=12&sys=system&id=1234&sys=system2&id=1234b&created=2018-11-01T18:41:50.850Z&tz=-600 HTTP/1.1
> Host: localhost:8060
> Authorization: Basic xxxxxxxxx
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: image/png
> Content-Length: 84801
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
I expected this to work, and for the console to be something like:
Received 65536 bytes of data.
Received 19274 bytes of data.
There will be no more data.
finish
I am on the same team.
Turns out we were deep cloning the object containing the stream. When we stopped cloning the entire file uploads fine.
I am making a Node.js/Express backend for a mobile app that makes requests to an API using the RESTful approach, where I can use the JSON data returned by this API for users to use in my mobile app.
My confusion lies in the differences that go on 'under to hood' and what automatically get handled when using specific headers that are intended for browsers and a server VS when making curl request through the terminal, that you'll run more than once, sending and responding with different headers
Headers/The Rules That Apply I Make Requests
I must use the conditional GETs convention where:
1) All responses return an HTTP Cache-Control header. It’s content indicates how long a cached response can be used to reduce unnecessary API requests.
2) In addition to that, each response returns an HTTP ETag header. It’s content is to be used in subsequent requests to the same resource in an HTTP If-None-Match header. The API will then return a status code 304 Not Modified if the cached information is still valid.
Clients accessing the This API MUST use this techniques, also known as conditional GET.
Using Curl Requests
Making Initial Request:
A) Returns the data I requested. B) Returns an Etag to use for future requests C) Returns a Cache-Control header to know how long the data is cache-able, like so:
First Request:
$ curl -v "https://api.example.com/data/3" -X GET \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-u token:secret
Responce:
< HTTP/1.1 200 OK
< Date: Wed, 01 Sep 2017 22:22:22 GMT
< ETag: "xxxxxxxxxxxxxxxxxxxxxxx" <-----Got Etag
< Last-Modified: Wed, 09 Sep 2015 11:11:11 GMT
< Content-Type: application/json; charset=utf-8
< Cache-Control: max-age=3600, private <-----and Cache-Control is set
{"data":{"id":1, "...": "..."}} to 3600 seconds (1 hour).
Here I understand the Cache-Control header specifies that the data I just got is good for 3,600 seconds, i.e. 1 hour. So I can use the returned data-information for the next hour without having to request the data from the API again. After that time period, another request can be made to the API. This time I include the returned xxx..... ETag value I got one hour ago (because thats how long the Cache-Control was set for), and use it in the If-None-Match header if I make another request like so:
Making Another Curl Request:
Future Request:
$ curl -v "https://api.example.com/data/3" -X GET \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "If-None-Match: xxxxxxxxxxxxxxxxxxxxxxx" <--Added here
-u token:secret
Future Responce:
< HTTP/1.1 304 Not Modified <--Got 304 because the
< Date: Wed, 23 Sep 2015 20:24:20 GMT
< ETag: "xxxxxxxxxxxxxxxxxxxxxxx" <-- `Etag` returned is the same
< Last-Modified: Wed, 09 Sep 2015 11:45:39 GMT (so I know it hasn't changed)
< Cache-Control: max-age=3600, private
< Connection: close
So this leads to me having a few questions:
Making None-Curl Requests (via a Node.js server):
I am using the pretty straight forward Node.js request module to make my request like so:
var request = require('request');
var options = {
url: 'https://api.example.com/data/3',
headers: { <------ Setting up Headers
'Content-Type': 'application/json', Note: no "If-None-Match"
'Accept': 'application/json'
'Authorization: Basic QQWERQWERWQER='
}
};
function callback(error, response, body) {
if (!error && response.statusCode == 200) {
var data = body <------ Got 200 response and loaded
} return data into data varible
}
request(options, callback);
So how is my Express server communicating with this API when making requests?
So after I make my first request on my Express server, it (my server) can then determine if it needs to fetch that content from the network or from cache based on the Cache-Control header and ETag. So how is this being done?
Do I need to programmatically/write code to do that every hour? So will I have to write code to grab the etag like this
if (!error && response.statusCode == 200) {
var data = body;
var etag = response.headers.etag <------ Grab this
}
and create a second set of headers like this
headers: {
'Content-Type': 'application/json',
'Accept': 'application/json'
'Authorization: Basic QQWERQWERWQER='
'if-none-match': etag <------ Adding it here
}
like I had to do when did when I did my second curl? (I did this and it will hold up on every requests not returning any status code, if this is how its done could it just be I did something wrong on my end?)
or does the initial response where Cache-Control headers set to max-age=3600, private and the Etag (set to xxxxxxxx in my example) automatically set the cache on my Express server and tells it to act accordingly with the it wants this resource where it check it again in an hour without having to do anything to tell it to?
Right now whenever I request a resource in the mobile app that will need a something (lets say the resource in my example https://api.example.com/data/3 ) from this API it triggers my Express server to fetch it on mobile app users behalf and return it. However when I request the same resource more than once, the API returns a 200 response every time. Where as it should be responding with a 304 for every request after the first. So what's going on here/ what am I doing/ understanding wrong? why would I not get 304 response?
I'm trying out restify, and though I'm more comfortable with Express, so far it's pretty awesome. I'm trying to set the content type header in the response like so:
server.get('/xml', function(req, res) {
res.setHeader('content-type', 'application/xml');
// res.header('content-type', 'application/xml'); // tried this too
// res.contentType = "application/xml"; // tried this too
res.send("<root><test>stuff</test></root>");
});
But the response I get back is instead application/octet-stream.
I also tried res.contentType('application/xml') but that actually threw an error ("Object HTTP/1.1 200 OK\ has no method 'contentType'").
What is the correct way to set the content type header to xml on the response?
Update:
When I do console.log(res.contentType); it actually outputs application/xml. Why is it not in the response headers?
Curl snippet:
* Hostname was NOT found in DNS cache
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /xml?params=1,2,3 HTTP/1.1
> User-Agent: curl/7.39.0
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/octet-stream
< Content-Length: 8995
< Date: Mon, 23 Feb 2015 20:20:14 GMT
< Connection: keep-alive
<
<body goes here>
Turns out the reason this was failing is because I was not sending the response using Restify's response handler; it was defaulting to the native Node.js handler.
Where I was doing this:
res.send(js2xmlparser("search", obj));
I should have been doing this:
res.end(js2xmlparser("search", o));
// ^ end, not send!
When I do console.log(res.contentType); it actually outputs application/xml. Why is it not in the response headers?
All you've done there is set a property on the res object. And because this is JavaScript, that works fine and you can read the property value back, but that's not the correct API for either node core or restify, so it is ignored by everything other than your code.
Your res.header("Content-Type", "application/xml"); looks correct to me based on the restify docs you linked to. Therefore my hunch is your tooling may be misleading you. Are you sure you are seeing the raw values in the response (many developer tools will unhelpfully "prettify" or otherwise lie to you) and you are hitting the route you really think you are? Output of curl -v or httpie --headers would be helpful.
It is possible to return application/xml by adding a formatter to the server instance at server creation:
var server = restify.createServer( {
formatters: {
'application/xml' : function( req, res, body, cb ) {
if (body instanceof Error)
return body.stack;
if (Buffer.isBuffer(body))
return cb(null, body.toString('base64'));
return cb(null, body);
}
}
});
Then at some part of the code:
res.setHeader('content-type', 'application/xml');
res.send('<xml>xyz</xml>');
Please, take a look at: http://restify.com/#content-negotiation
You can send the XML response using sendRaw instead of send. The sendRaw method doesn't use any formatter at all (you should preformat your response if you need it). See an example below:
server.get('/xml', function(req, res, next) {
res.setHeader('content-type', 'application/xml');
res.sendRaw('<xml>xyz</xml>');
next();
});
I have a simple API upload, it is used to accept upload file from client.
var flg=true;
app.post('/test', function(req, res){
flg=!flg;
var returnJson='{';
if(flg){
req.form.on('part', function (part) {
if(part){
part.resume();
}
returnJson=returnJson+',\"status\":\"0\"}';
res.send(returnJson);
});
}else{
console.log('close');
returnJson=returnJson+',\"status\":\"1\"}';
res.header('Connection', 'close');
res.send(413, returnJson);
}
});
I'd like to test this API with Jmeter. "status":"0" means success. "status":"1" means failure. I write Jmeter script like this:
http://i.imgur.com/vEUJKc8.jpg
Jmeter only displays all the samplers which response contains "status":"0". it seems Jmeter exclude failure sampler response which comes from else section.
http://imgur.com/bkFSpK2
How can I see all samplers which includes all success and failure samplers in Jmeter?
successful Sampler result is :
Thread Name: API 1-1
Sample Start: 2013-12-18 11:46:08 PST
Load time: 7
Latency: 6
Size in bytes: 178
Headers size in bytes: 163
Body size in bytes: 15
Sample Count: 1
Error Count: 0
Response code: 200
Response message: OK
Response headers:
HTTP/1.1 200 OK
X-Powered-By: Express
Content-Type: text/html; charset=utf-8
Content-Length: 15
Date: Wed, 18 Dec 2013 19:46:08 GMT
Connection: keep-alive
HTTPSampleResult fields:
ContentType: text/html; charset=utf-8
DataEncoding: utf-8
Any suggestion?
I don't like this stanza:
ContentType: text/html;
Correct ContentType for JSON will be application/json
You can try using HTTP Header Manager to set Content-Type header of your request to be application/json and see what happens.
Also there is a JSON plugin which provides JSON Path Extractor and JSON Path Assertion (Select “Extras with libs set” from the download list).