I am trying to receive a file on Node.js Express framework (via express-fileupload middleware), and then POST it directly to another server via the request package without saving it on the first device's disk and then re-reading it.
const streamifier = require('streamifier');
const fileUpload = require('express-fileupload');
app.use(fileUpload());
app.post('/upload', function(req, res) {
var fileBuffer = req.files.upload.data;
var fileReadStream = streamifier.createReadStream(fileBuffer);
//Use 'request' to send fileReadStream to another API for additional processing
}
However, the receiving API does not receive any file when I run this code.
When I run this other code, however, it works fine, but fs is only able to create a ReadStream from a file on the server, so it would involve me saving the file to some temporary hard disk location first, reading it again, and then sending it to the remote API, which I think is a bit wasteful.
var fs = require('fs');
app.post('/upload', function(req, res) {
var fileReadStream = fs.createReadStream('test.txt');
//Use 'request' to send fileReadStream to another API for additional processing
}
Any idea why the 1st code block results in no file received by the external API, while the 2nd one works perfectly fine? It seems that fileReadStream would be the same in both cases.
Turns out I should have just been sending the buffer by itself without transforming it into a ReadStream. However, I did have to add the name to it, or the receiving API wouldn't recognize that I was sending a file.
const fileUpload = require('express-fileupload');
app.use(fileUpload());
app.post('/upload', function(req, res) {
var fileBuffer = req.files.upload.data;
fileBuffer.name = 'test.png';
//Use 'request' to send fileBuffer to another API for additional processing
}
I am using dropzone to upload files from the UI. Here is my drozone configuration:
var imageUpload = new Dropzone('div#dataSection', {
url: 'api/image',
autoProcessQueue: true,
paramName: 'file'
});
I am using node and expressjs to build the server side endpoint based on this link
I am excuting below code on the server side.
fs.readFile(req.files.displayImage.path, function (err, data) {
// ...
var newPath = __dirname + "/uploads/uploadedFileName";
fs.writeFile(newPath, data, function (err) {
res.redirect("back");
});
});
The problem is req.files in coming as undefined. Am I missing anything?
The tutorial you were following is quite outdated, and certainly written for old version of express. With express 4.0, released in April 2014, (almost) all built-in middleware was dropped, including the one parsing uploaded files as req.files.
That's the problem, and the solution is to use separate module to handle uploaded files. Multer seems like a natural choice for that (kudos to #Ben Fortune for this suggestion).
First, install multer Node.js module:
npm install multer
Then, in the code on server side:
var express = require('express')
var multer = require('multer')
var app = express()
app.use(multer({ dest: './uploads/'})) // directory to store uploaded files in
After that, all POST-ed files should be available as req.files in your route handler functions, in the format of Multer file objects.
I have a working NodeJS+FFMPEG application. I am able to upload video files and have them converted on the server. I am using this NodeJS FFMPEG library
https://github.com/fluent-ffmpeg/node-fluent-ffmpeg
I get a message on the server when the job is complete, but how do I notify the client?? In this case a simple AIR application. Right now I can only 'hear' the initial response after a successful upload.
The initial video file was uploaded via a http POST request.
My primary node application without the dependencies is as follows
var ffmpeg = require('./lib/fluent-ffmpeg');
var express = require('express'),
multer = require('multer');
var app = express();
//auto save file to uploads folder
app.use(multer({ dest: './uploads/'}))
var temp;
app.post('/', function (req, res) {
console.log(req.body); //contains the variables
console.log("req.files ="+ req.files); //contains the file references
console.log("req.files.Filedata.path ="+ req.files.Filedata.path );
temp=req.files.Filedata.path;
// make sure you set the correct path to your video file
var proc = ffmpeg('./'+temp)
// set video bitrate
.videoBitrate(1024)
// set audio codec
.audioCodec('libmp3lame')
// set output format to force
.format('avi')
// setup event handlers
.on('end', function() {
console.log('file has been converted succesfully');
})
.on('error', function(err) {
console.log('an error happened: ' + err.message);
})
// save to file
.save('./converted/converted.avi');
res.send('Thank you for uploading!');
});
app.listen(8080);
There are two approaches you can use. The first is to poll the server from the client with AJAX GET requests every x seconds. The second approach is to use WebSockets and notify the client by sending a message to them directly.
Using Express with Node, I can upload a file successfully and pass it to Azure storage in the following block of code.
app.get('/upload', function (req, res) {
res.send(
'<form action="/upload" method="post" enctype="multipart/form-data">' +
'<input type="file" name="snapshot" />' +
'<input type="submit" value="Upload" />' +
'</form>'
);
});
app.post('/upload', function (req, res) {
var path = req.files.snapshot.path;
var bs= azure.createBlobService();
bs.createBlockBlobFromFile('c', 'test.png', path, function (error) { });
res.send("OK");
});
This works just fine, but Express creates a temporary file and stores the image first, then I upload it to Azure from the file. This seems like an inefficient and unnecessary step in the process and I end up having to manage cleanup of the temp file directory.
I should be able to stream the file directly to Azure storage using the blobService.createBlockBlobFromStream method in the Azure SDK, but I am not familiar enough with Node or Express to understand how to access the stream data.
app.post('/upload', function (req, res) {
var stream = /// WHAT GOES HERE ?? ///
var bs= azure.createBlobService();
bs.createBlockBlobFromStream('c', 'test.png', stream, function (error) { });
res.send("OK");
});
I have found the following blog which indicates that there may be a way to do so, and certainly Express is grabbing the stream data and parsing and saving it to the file system as well. http://blog.valeryjacobs.com/index.php/streaming-media-from-url-to-blob-storage/
vjacobs code is actually downloading a file from another site and passing that stream to Azure, so I'm not sure if it can be adapted to work in my situation.
How can I access and pass the uploaded files stream directly to Azure using Node?
SOLUTION (based on discussion with #danielepolencic)
Using Multiparty(npm install multiparty), a fork of Formidable, we can access the multipart data if we disable the bodyparser() middleware from Express (see their notes on doing this for more information). Unlike Formidable, Multiparty will not stream the file to disk unless you tell it to.
app.post('/upload', function (req, res) {
var blobService = azure.createBlobService();
var form = new multiparty.Form();
form.on('part', function(part) {
if (part.filename) {
var size = part.byteCount - part.byteOffset;
var name = part.filename;
blobService.createBlockBlobFromStream('c', name, part, size, function(error) {
if (error) {
res.send({ Grrr: error });
}
});
} else {
form.handlePart(part);
}
});
form.parse(req);
res.send('OK');
});
Props to #danielepolencic for helping to find the solution to this.
As you can read from the connect middleware documentation, bodyparser automagically handles the form for you. In your particular case, it parses the incoming multipart data and store it somewhere else then exposes the saved file in a nice format (i.e. req.files).
Unfortunately, we do not need (and necessary like) black magic primarily because we want to be able to stream the incoming data to azure directly without hitting the disk (i.e. req.pipe(res)). Therefore, we can turn off bodyparser middleware and handle the incoming request ourselves. Under the hood, bodyparser uses node-formidable, so it may be a good idea to reuse it in our implementation.
var express = require('express');
var formidable = require('formidable');
var app = express();
// app.use(express.bodyParser({ uploadDir: 'temp' }));
app.get('/', function(req, res){
res.send('hello world');
});
app.get('/upload', function (req, res) {
res.send(
'<form action="/upload" method="post" enctype="multipart/form-data">' +
'<input type="file" name="snapshot" />' +
'<input type="submit" value="Upload" />' +
'</form>'
);
});
app.post('/upload', function (req, res) {
var bs = azure.createBlobService();
var form = new formidable.IncomingForm();
form.onPart = function(part){
bs.createBlockBlobFromStream('taskcontainer', 'task1', part, 11, function(error){
if(!error){
// Blob uploaded
}
});
};
form.parse(req);
res.send('OK');
});
app.listen(3000);
The core idea is that we can leverage node streams so that we don't need to load in memory the full file before we can send it to azure, but we can transfer it as it comes along. The node-formidable module supports streams, hence piping the stream to azure will achieve our objective.
You can easily test the code locally without hitting azure by replacing the post route with:
app.post('/upload', function (req, res) {
var form = new formidable.IncomingForm();
form.onPart = function(part){
part.pipe(res);
};
form.parse(req);
});
Here, we're simply piping the request from the input to the output. You can read more about bodyParser here.
There are different options for uploading binary data (e.g. images) via Azure Storage SDK for Node, not using multipart.
Based on the Buffer and Stream definitions in Node and manipulating them, these could be handled using almost all the methods for BLOB upload: createWriteStreamToBlockBlob, createBlockBlobFromStream, createBlockBlobFromText.
References could be found here: Upload a binary data from request body to Azure BLOB storage in Node.js [restify]
People having trouble with .createBlockBlobFromStream trying to implement the solutions, note that this method has been changed slightly in newer versions
Old version:
createBlockBlobFromStream(containerName, blobName, part, size, callback)
New version
createBlockBlobFromStream(containerName, blobName, part, size, options, callback)
(if you don't care about options, try an empty array) for the parameter.
Oddly enough, "options" is supposed to be optional, but for whatever reason, mine fails if I leave it out.
All the PDF files are saved in the filesystem on the server, how to make the files to be downloadable in client side.
for Ex :
app.use('/pdfDownload', function(req, res){
var pathToTheFile = req.body.fileName;
readFile(pathToTheFile, function(data){
//code to make the data to be downloadable;
});
});
is the request made
function readFile(pathToTheFile, cb){
var fs = require('fs');
fs.readFile(pathToTheFile, function(err, data){
//how to make the file fetched to be downloadable in the client requested
//cb(data);
});
}
You can use express.static, set it up early in your app:
app.use('/pdf', express.static(__dirname + '/pathToPDF'));
And it will automatically do the job for you when browser navigates to e.g. '/pdf/fooBar.pdf'.