How to store request body in GridFS using Restify server? - node.js

Question

Cannot save file larger than pipe chunk size (~64K).
Using mongodb 3.4.0, Relevant node dependencies
restify 4.2.0
mongodb ^2.2.12
lodash 4.16.6
bookeeppingData = {request, id, ...meta}
clone = lodash.cloneDeep(bookkeepingData)
const {
request: req,
id: _id,
meta: metadata,
} = clone
const bucket = new mongodb.GridFSBucket(
db,
{bucketName: 'my_gridfs_collection'}
);
const uploadSteam = bucket.openUploadStreamWithId(
_id,
undefined,
{metadata}
);
req.on('data', (chunk) => {
console.log(`Received ${chunk.length} bytes of data.`);
});
req.on('end', () => {
console.log('There will be no more data.');
});
req.on('error', (e) => {
console.log('req on error', e);
});
uploadSteam.on('finish', function() {
console.log('finsish');
keepThebooks(bookkeepingData);
});
uploadSteam.on('error', (e) => {
console.log('uploadstream on error', e);
});
req.pipe(uploadSteam);
}
When sending a file smaller than the ~64K, The console output is
Received 57259 bytes of data.
There will be no more data.
finish
This is the corresponding curl --verbose output (minus the json response):
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8060 (#0)
* Server auth using Basic with user 'driver'
> POST /artifact?branch=xyz&role=PHOTO&where.lat=55&where.long=77.2&where.acc=12&sys=system&id=1234&sys=system2&id=1234b&created=2018-11-01T18:41:50.850Z&tz=-600 HTTP/1.1
> Host: localhost:8060
> Authorization: Basic xxxxxxxxx
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: image/png
> Content-Length: 57259
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store, must-revalidate
< Content-Type: application/json
< Content-Length: 388
< Date: Wed, 23 Jan 2019 20:58:16 GMT
< Connection: keep-alive
<
* Connection #0 to host localhost left intact
When sending a file larger than ~64K, the console output is:
Received 65536 bytes of data.
(That's it -- no error)
The corresponding curl --verbose output is:
* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8060 (#0)
* Server auth using Basic with user 'driver'
> POST /artifact?branch=xyz&role=PHOTO&where.lat=55&where.long=77.2&where.acc=12&sys=system&id=1234&sys=system2&id=1234b&created=2018-11-01T18:41:50.850Z&tz=-600 HTTP/1.1
> Host: localhost:8060
> Authorization: Basic xxxxxxxxx
> User-Agent: curl/7.61.1
> Accept: */*
> Content-Type: image/png
> Content-Length: 84801
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
* Empty reply from server
* Connection #0 to host localhost left intact
curl: (52) Empty reply from server
I expected this to work, and for the console to be something like:
Received 65536 bytes of data.
Received 19274 bytes of data.
There will be no more data.
finish

Answer 1

I am on the same team.
Turns out we were deep cloning the object containing the stream. When we stopped cloning the entire file uploads fine.