I am making a multer storage engine which makes stream connection between client and S3 Server.
At middle of the stream, my code examine chunks and send it to S3.
I could get a file stream from node.js server. But when I request file array upload, node inspector shows only one stream. What should I do?
Stream engine snippet
CustomStreamEngine.prototype._handleFile = function _handleFile (req, file, cb) {
// for inspect
req.files.length // 1
file;
};
request controller
var streamStorage = multer({
storage: streamEngine()
});
dev.post('/rec_test', streamStorage.array('source'), (req, res, next) => {
});
I just published this streaming multipart/form-data parser on npm as form-parser:
You should be able to do the following:
dev.post('/rec_test', async (req, res, next) => {
// Parse request
await parser(req, async ({ fieldType, fieldName, fieldContent }) => {
// Log all fields
console.log({ fieldType, fieldName, fieldContent });
// Handle 'source' file fields
if (fieldType === 'file' && fieldName === 'source[]') {
// Get file info
const { fileName, fileType, fileStream } = fieldContent;
// Upload fileStream to S3 :-)
}
});
});
Hope it's helpful.
K
I think you can add some logs to https://github.com/expressjs/multer/blob/master/lib/make-middleware.js to check.
Currently, I use axios on the client to send multi files to the server with multer. And I can see all files in the function
busboy.on('file', function (fieldname, fileStream, filename, encoding, mimetype), but there is only one file at a time, and this function will call the _handfile function of the custom storage, so that I think it is the reason for your issue.
Hope it can help you
Related
I upload an image from a react native expo app.
In the back end, I have this code :
const storage = multer.diskStorage({
destination: (req, file, callback) => {
callback(null, "avatars");
},
filename: (req, file, callback) => {
let imagePath = Date.now() + path.extname(file.originalname);
callback(null, imagePath);
},
});
const upload = multer({ storage: storage });
app.use("/uploadAvatar", upload.single("avatar"), (req, res) => {
res.status(200).json("Image enregistrée !");
});
It works fine and save the image into a folder.
What I need to do, is to save the image path in the database.
The image is an avatar for a user profile, so I need to add its path to the user table.
Suppose you are storing the image in your middleware:
app.use('/uploadAvatar', upload.single('avatar'), async (req, res) => {
// Assume that you're storing the image with a function,
// in this case you just need to pass the filename into
// your function. This is a pseudocode and is not expected to work
// as it is, so please adjust accordingly.
const result = await storeImage(req.file.filename);
// Return the response as JSON.
res.status(200).json('Image enregistrée!');
});
Basically, you have to inspect the req.file that would be populated after you call the upload.single('avatar') middleware. In this case, we're taking the filename to be stored into the database. You can do any path/file/property manipulations that you might want to do before storing that image, though.
For further information about req.file and what properties it contains, please see the Multer documentation.
So I'm trying to make the html form:
<form action="blahblah" encblah="multipart/form-data" whatever>
Thats not the problem, I need to make that form send the blob to express
app.post('/upload/avatars', async (req, res) => {
const body = req.body;
console.log(req.file);
console.log(body);
res.send(body);
});
So I can access the blob, create a read stream, pipe it to the cloud, and bam, upload the file without downloading anything on the express server it self.
Is that possible?
If yes, please tell me how.
If no, please tell me other alternatives.
On the client we do a basic multi-part form upload. This example is setup for a single image but you could call uploadFile in sequence for each image.
//client.ts
const uploadFile = (file: File | Blob) => {
const formData = new FormData();
formData.append("image", file);
return fetch("/upload", {
method: "post",
body: formData,
});
};
const handleUpload = (event: any) => {
return event.target.files.length ? uploadFile(event.target.files[0]) : null;
};
On the server we can use multer to read the file without persisting it to disk.
//server.js
const express = require("express");
const app = express();
const multer = require("multer");
const upload = multer();
app.post(
"/upload",
upload.fields([{ name: "image", maxCount: 1 }]),
(req, res, next) => {
console.log("/upload", req.files);
if (req.files.image.length) {
const image = req.files.image[0]; // { buffer, originalname, size, ...}
// Pipe the image.buffer where you want.
res.send({ success: true, count: req.files.image.originalname });
} else {
res.send({ success: false, message: "No files sent." });
}
}
);
For larger uploads I recommend socket.io, but this method works for reasonably sized images.
it is possible, but when you have a lot of traffic it would overwhelm your express server (in case you are uploading videos or big files ) but if it's for uploading small images (profile image, etc...) you're fine. either way you can use Multer npm
I'd recommend using client-side uploading on ex: s3-bucket, etc..., which returned a link, and therefore using that link.
This question already has answers here:
Nodejs POST request multipart/form-data
(5 answers)
Closed 5 years ago.
File uploading using multer is not happening
My code:
File read from html and pass to external url
router.post('/fileUpload',function (req,res) {
request({
uri: http//example.com/upload, // url of other server
method: "POST",
form: {
"name":req.body.name,
"image":? //image from html - no idea how to pass the req.files here
}
}, function (error, response, body) {
------------
------------
}
});
other server url : /example.com/upload
This is the code to upload image using multer
app.post('/upload',function(req,res){
var storage = multer.diskStorage({
destination: function (req, file, callback) {
callback(null, 'uploads');
},
filename: function (req, file, callback) {
callback(null, file.fieldname + '-' + Date.now());
}
});
var upload = multer({ storage : storage }).array('productImage');
upload(req,res,function(err) {
if(err) {
return res.json({'success':0,'result':{},'errorMessage':'Unknown Error'});
}
return res.json({'success':1,'result':{},'errorMessage':''});
});
});
Create readStream from file uploaded and pipe it to the another server.check this link https://www.npmjs.com/package/request, you will easily get this done.
The simple answer would be to create a read stream from the uploaded file and pipe it to the second server, like so:
fs.createReadStream(req.files[0].path).pipe(request.post('http//example.com/upload'))
However, there are many ways to make this work. The most efficient of which is to use a binary stream from the initial upload all the way to the second server. You want to avoid using your first server as a storage for all of the uploads.
Here's how I would do it instead:
Use jQuery file upload on client side upload
(Or any other library/approach to upload the raw binary stream)
$('#fileupload').fileupload({
url: 'https://server1.com/upload'
})
Use Formidable (or other library like multer) to handle upload server-side
(This will allow you to read the binary stream in parts, and handle each part as it comes)
app.post('/upload',function(req,res){
var form = new formidable.IncomingForm();
form.parse(req);
form.onPart = function(part) {
fs.createReadStream(part).pipe( request.post('http//example.com/upload'))
}
}
When each part of the binary upload is received, you can to stream the binary directly to the second server via pipe() to avoid having to store it on the first server whatsoever.
The key to making this work is to look at the file upload and transfer as a stream of binary bits. When the user uploads a file (req.body) you want to create a read stream from the file, and pipe the binary over the request.post().
I'm attempting to upload images to an Express server. I'm not exactly sure about how to do this, but heres what I've got from what I've been able to pick up from MDN, express, react-dropzone, and multer Documentation. Multer does not seem to pick up the FormData object from react-dropzone, when logging out req.file it returns undefined.
server.js
var storage = multer.diskStorage({
destination: './public/users',
filename: function (req, file, cb) {
switch (file.mimetype) {
case 'image/jpeg':
ext = '.jpeg';
break;
case 'image/png':
ext = '.png';
break;
}
cb(null, file.originalname + ext);
}
});
var upload = multer({storage: storage});
app.use(upload.single('photo'));
app.post('/uploadUserImage', function (req, res) {
console.log(JSON.stringify(req.body.photo)) // form fields
console.log(req.photo) // form files
console.log(req.file) // form files
res.send(req.body.photo);
});
client.js
function uploadImage (image) {
var formData = new FormData();
formData.append('photo', image);
fetch('http://localhost:8080/uploadUserImage/', {
method:'POST',
body: formData
});
}
When I make this request morgan logs out the following:
{ photo: '[object File]' } <--- from console.log(req.body');
undefined <--- from console.log(req.file);
multer creates the folder public/uploads but does not place the image in the location. How can I get the photo because I need to run it through sharp (to compress the filesize and resize the image) and then place it in the uploads folder?
The error occurs because you specified the 'Content-type' explicitly. To do this properly, you'd also need to specify the boundary. You can find a detailed explanation of multipart/form-data here: How does HTTP file upload work?
To solve the issue with the file upload, you should remove the 'Content-Type' specification from the fetch request. You can also refactor the uploadImage method to upload the form without parsing the inputs:
function uploadImage () {
// This assumes the form's name is `myForm`
var form = document.getElementById("myForm");
var formData = new FormData(form);
fetch('http://localhost:8000/uploadUserImage', {
method: 'POST',
body: formData
});
}
The problem for me was that firebase has an error and can't use multer. You need to use busboy and parse it manually. Also I needed to append the uri from react native imagePicker instead of the file blob. Like this:
data.append('fileData', {
uri : pickerResponse.uri,
type: pickerResponse.type,
name: pickerResponse.fileName
});
My goal is to accept an uploaded file and stream it to Wistia using the the Wistia Upload API. I need to be able to add fields to the HTTP request, and I don't want the file to touch the disk. I'm using Node, Express, Request, and Busboy.
The code below has two console.log statements. The first returns [Error: not implemented] and the second returns [Error: form-data: not implemented]. I'm new to streaming in Node, so I'm probably doing something fundamentally wrong. Any help would be much appreciated.
app.use("/upload", function(req, res, next) {
var writeStream = new stream.Writable();
writeStream.on("error", function(error) {
console.log(error);
});
var busboy = new Busboy({headers: req.headers});
busboy.on("file", function(fieldname, file, filename, encoding, mimetype) {
file.on("data", function(data) {
writeStream.write(data);
});
file.on("end", function() {
request.post({
url: "https://upload.wistia.com",
formData: {
api_password: "abc123",
file: new stream.Readable(writeStream)
}
}, function(error, response, body) {
console.log(error);
});
});
});
req.pipe(busboy);
});
I am not to familiar with the busboy module, but there errors you are getting are from attempting to use un-implemented streams. Whenever you create a new readable or writable stream directly from the stream module you have to create the _read and _write methods respectively Stream Implementors (node.js api). To give you something to work with the following example is using multer for handling multipart requests, I think you'll find multer is easier to use than busboy.
var app = require('express')();
var fs = require('fs');
var request = require('request');
app.use(multer());
app.post("/upload", function(req, res, next) {
// create a read stream
var readable = fs.createReadStream(req.files.myfile.path);
request.post({
url: "https://upload.wistia.com",
formData: {
api_password: "abc123",
file: readable
}
}, function(err, res, body) {
// send something to client
})
});
I hope this helps unfortunately I am not familiar with busboy, but this should work with multer, and as I said before there problem is just that you are using un-implemented streams I'm sure there is a way to configure this operation with busboy if you wanted.
If you want to use multipart (another npm) here is a tutorial:
http://qnimate.com/stream-file-uploads-to-storage-server-in-node-js/