I'm trying to upload a file with node using this simple code:
UpdateController.prototype.uploadUpdateFile = function(req, res)
{
var w = fs.createWriteStream(settings.uploadFolder + settings.updateFile);
req.pipe(w);
w.on('finish', function()
{
res.send(JSON.stringify({
status:0,
filename:settings.uploadFolder + settings.updateFile
}));
}, function()
{
res.send(JSON.stringify({
status:1,
message:"error during file upload, operation failed"
}));
});
}
The file is correctly uploaded but it changes between the original because header (------WebKitForm ... /octet-stream....) and footer (..------WebKitFormBoundary9gOZjMubs9GivcUQ--..) are added to the content.
How to get only the file content ?
You would have to look at the headers of the client request to understand how the client decided to send you the file (how the file was encoded)
You will probably end up using busboy or another package that depends on it : https://www.npmjs.com/package/busboy
such a package will "decode" the data sent by the browser.
Related
I am a beginner with Node and I am trying to figure out how to create a zip file at the server then send it to the client and then download the zip file to the user's browser. I am using the Express framework and I am using Archiver to actually do the zipping. My server code is the following which was taken from Dynamically create and stream zip to client
router.get('/image-dl', function (req,res){
res.writeHead(200, {
'Content-Type': 'application/zip',
'Content-disposition': 'attachment; filename=myFile.zip'
});
var zip = archiver('zip');
// Send the file to the page output.
zip.pipe(res);
// Create zip with some files. Two dynamic, one static. Put #2 in a sub folder.
zip.append('Some text to go in file 1.', { name: '1.txt' })
.append('Some text to go in file 2. I go in a folder!', { name: 'somefolder/2.txt' })
.finalize();
});
So its zipping two text files and returning the result. On the client side I am using the following function in a service to actually call that endpoint
downloadZip(){
const headers = new Headers({'Content-Type': 'application/json'});
const token = localStorage.getItem('token')
? '?token=' + localStorage.getItem('token')
: '';
return this.http.get(this.endPoint + '/job/image-dl' + token, {headers: headers})
.map((response: Response) => {
const result = response;
return result;
})
.catch((error: Response) => {
this.errorService.handleError(error.json());
return Observable.throw(error.json());
});
}
and then I have another function which calls downloadZip() and actually downloads the zip file to the user's local browser.
testfunc(){
this.jobService.downloadZip().subscribe(
(blah:any)=>{
var blob = new Blob([blah], {type: "application/zip"});
FileSaver.saveAs(blob, "helloworld.zip");
}
);
}
When testfunc() is called, a zip file is downloaded to the user's browser however when I try to unzip it it creates a zip.cpgz file which then turns back into a zip file when clicked in an infinite loop which indicates that some kind of corruption happened. Can anyone see where I went wrong here?
I am trying to upload a file to a server (built using Java) by reading from a mongodb gridfs stream.
exports.upload = function(req, res, next) {
var IHUrl = config.api.url + "PhotosServlet";
var data = req.body;
var file1 = api.gfs.createReadStream({
_id: data.fileId
})
var formData = {
"imgTyp": data.imgTyp,
"listingid": data.listingid,
"scaleTech": data.scaleTech,
"SPC": data.SPC,
"SPCUID": data.SPCUID,
"varRand": data.varRand,
"file1": file1
};
var r = request.post({
url: IHUrl,
formData: formData
}, function(error, IHResponse, body) {
if (error) {
res.send(500, error);
console.log("Error occured uploading file1")
} else {
console.log("Upload successful", IHResponse);
res.send(200, IHResponse);
}
});
next();
};
File is already uploaded in mongodb gridfs before I start uploading the file to upstream server.
I am using following nodejs libraries:
request, gridfs-stream
I am getting the following error from upstream server:
javax.servlet.ServletException: Processing of multipart/form-data request failed. Stream ended unexpectedly
What could be going wrong here?
I realized that its been a while and its a problem some of you may encounter and the solution was not really what I posted in the comments as I found more problems later. The issue that we had was that the java servlet that we were posting the multipart form data was not able to handle chunked data and setting headers won't do you any good. using request library will not help you here. I had to make use of restler (https://www.npmjs.com/package/restler) to send the entire multipart data in a single chunk.
I am developping an file sharing platform with node.js and express.js, using busboy.
It works nicely at this time but uploading large file.
If I do, the server doesn't accept any new request while the large file is uploaded.
I that normal ? If it is, how to improve this behavior, even if that means the upload will take more time...
For now I develop and test on localhost on a pretty good PC (asus i7/8G) on ubuntu.
When I start uploadind a large file, and open a new tab to go to the app, the tab loads only when the upload is completed.
Application loading:
var app = express();
//...
app.use(bodyParser.urlencoded({
extended: true
}));
app.use(bodyParser.json());
app.use(methodOverride());
// Request multipart parsing middleware
app.use(busboy());
// default options, immediately start reading from the request stream and
// parsing
app.use(busboy({ immediate: true }));
My upload method in files controller:
exports.create = function(req, res) {
var _file = new File(req.body);
req.busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
file.on('data', function(data) {
_file.data += data;
});
file.on('end', function() {
_file.save(function(err) {
if (err) {
console.error(err);
return res.status(500).send({
message: errorHandler.getErrorMessage(err)
});
} else {
// doing some stuff on save
}
});
});
});
// req.busboy.on('field', function(key, value, keyTruncated, valueTruncated) {
// console.log('Field [' + key + ']: value: ' + value);
// });
// req.busboy.on('finish', function() {
// console.log('Done parsing form!');
// });
req.pipe(req.busboy);
};
There's at least a few things wrong here:
busboy is being loaded twice. You should remove the app.use(busboy({ immediate: true })); line from your app.js.
Next, the entirety of all files are being buffered in memory (_file.data += data;). You should instead stream the file(s) somewhere, whether that's to disk, to some network storage service like Amazon's S3, or any other place outside of the process.
This one is a moot point really once you switch to streaming, but technically with your current buffering code, you're concatenating multiple files together because the same File object is being used for the entire request. Perhaps that's not an issue if you know for sure only one file will ever be sent (for example, you are always the client), but it is worth noting.
I am using Flowjs, and its ng-flow directive to upload file with NodeJS as backend. When i try to upload file, only file in tem folder is uploaded, but it's note any type of file like JPG or PNG. (flow-135601-juicy_gustinektar02l_10185jpg.1). Here is code:
ANGULARJS
app.config(['flowFactoryProvider', function (flowFactoryProvider) {
flowFactoryProvider.defaults = {
target: 'http://localhost:8086/api/upload/',
permanentErrors: [500, 501],
maxChunkRetries: 1,
chunkRetryInterval: 5000,
simultaneousUploads: 1
};
flowFactoryProvider.on('catchAll', function (event) {
console.log('catchAll', arguments);
});
// Can be used with different implementations of Flow.js
//flowFactoryProvider.factory = fustyFlowFactory;
}]);
NODEJS
// Handle uploads through Flow.js
app.post('/api/upload', function(req, res){
flow.post(req, function(status, filename, original_filename, identifier){
console.log('POST', status, original_filename, identifier);
res.send(200, {
// NOTE: Uncomment this funciton to enable cross-domain request.
//'Access-Control-Allow-Origin': '*'
});
});
});
// Handle cross-domain requests
// NOTE: Uncomment this funciton to enable cross-domain request.
/*
app.options('/upload', function(req, res){
console.log('OPTIONS');
res.send(true, {
'Access-Control-Allow-Origin': '*'
}, 200);
});
*/
// Handle status checks on chunks through Flow.js
app.get('/api/upload', function(req, res){
flow.get(req, function(status, filename, original_filename, identifier){
console.log('GET', status);
res.send(200, (status == 'found' ? 200 : 404));
});
});
Reassembling all chunks is easy, just call this:
var stream = fs.createWriteStream(filename);
r.write(identifier, stream);
And that is it!
But other question is, when this method should be called?
Maybe when all chunks are uploaded and present at tmp folder.
But there is another issue with duplicate calls of the done.
This can be solved by creating and locking the file, once all chunks exists.
Then call
r.write(identifier, stream);
Then clean all chunks, release the lock and close the file.
Same approuch is done in php server side library: https://github.com/flowjs/flow-php-server/blob/master/src/Flow/File.php#L102
Hope this helps, and I hope someone could collaborate and update node.js sample with these fixes.
I am trying to send a file's content to the client in my request, but the only documentation Express has is it's download function which requires a physical file; the file I am trying to send comes from S3, so all I have is the filename and content.
How do I go about sending the content of the file and appropriate headers for content type and filename, along with the file's content?
For example:
files.find({_id: id}, function(e, o) {
client.getObject({Bucket: config.bucket, Key: o.key}, function(error, data) {
res.send(data.Body);
});
});
The type of file depends on the file obviously. Have a look at this:
http://en.wikipedia.org/wiki/Internet_media_type
If you know what exactly is your file, then assign one of these to response ( not mandatory though ). You should also add the length of the file to response ( if it is possible, i.e. if it is not a stream ). And if you want it to be downloadable as an attachment, then add Content-Disposition header. So all in all you only need to add this:
var filename = "myfile.txt";
res.set({
"Content-Disposition": 'attachment; filename="'+filename+'"',
"Content-Type": "text/plain",
"Content-Length": data.Body.length
});
NOTE: I'm using Express 3.x.
EDIT: Actually Express is smart enough to count content length for you, so you don't have to add Content-Length header.
This is a great situation to use streams. Use the knox library to simplify things. Knox should take care of setting the needed headers to pipe files to the client
var inspect = require('eyespect').inspector();
var knox = require('knox');
var client = knox.createClient({
key: 's3KeyHere'
, secret: 's3SecretHere'
, bucket: 's3BucketHer'
});
/**
* #param {Stream} response is the response handler provided by Express
**/
function downloadFile(request, response) {
var filePath = 's3/file/path/here';
client.getFile(filePath, function(err, s3Response) {
s3Response.pipe(response);
s3Response.on('error', function(err){
inspect(err, 'error downloading file from s3');
});
s3Response.on('progress', function(data){
inspect(data, 's3 download progress');
});
s3Response.on('end', function(){
inspect(filePath, 'piped file to remote client successfully at s3 path');
});
});
}
npm install knox eyespect