I looking to use the form-data nodejs module to build up a multipart/form-data request. My HTTP endpoint I am posting to requires a file.
My "file" I want to attach is actually a base64encoded version of the file. I have the filename separately that I can use.
Looking at the form-data module - it looks like from the examples it relies on the file either being returned from fs or a request; is it possible to use options (field, value, options ) to make it accept either the base64encoded version of the file or do I need to decode it first? Ultimately the multipart is encoded anyway, or at least it can be.
var upload = multer({ storage: multer.memoryStorage({}) })
app.post('/', upload.single('test'), function (req, res, next) {
var raw = new Buffer(req.file.buffer.toString(), 'base64')
fs.writeFile('/tmp/upload.png', raw, function (err) {
if (err) return next(err)
res.end('Success!')
})
})
does this help ?
file name u can get from req.params or req.query.param , anywhere
Related
Current Situation
In the front-end side of the application I have a simple form, from where I can get 3 parameters for the data before uploading it to the back-end:
The form submition function
e.preventDefault();
const data = new FormData(e.currentTarget) //contains a string key called title
data.append("file", JSON.stringify(file)) //file object
data.append("description", JSON.stringify(editorState)) //description object
handleSubmit(e, data)
To prevent any error, I have some client-side validations (cannot upload if some parameters are empty). This is how I fetch the data:
handleSubmit
await fetch(process.env.NEXT_PUBLIC_DR_HOST, {
method: 'POST',
body: body, //body is the data argument
}).then(res =>
{
// window.location = res.url;
})
After that, the data will be sent to the server:
The server
router.post('/', upload.single('file'), validateDbData, tryAsync(async (req, res, next) =>
And using the multer middleware(upload.single('file')), the file will be uploaded respectively (storage).
The Problem
As you can see, after the upload.single('file') middleware, comes another one called validateDbData. This is the code for it:
const declarationSchema = Joi.object({
title: Joi.string().required(),
description: Joi.object().required(),
file: Joi.object()
})
const { error } = declarationSchema.validate(
req.body
)
console.log(error)
if (error)
{
const msg = error.details.map(e => e.message).join(',') //
throw new ServerError("Invalid Data", 400)
}
else
{
next()
}
And if there is something wrong (say the title is empty) an error is thrown, so the server-side validation works. But, the upload.single('file') middleware is called before the validateDbData validation midleware. This means that even if there is an error and everything got canceled, the file remained uploaded in the cloud (storage).
Question
How can I validate the data inside the FormData on the server before uploading the file parameter to the cloud (storage)? The multer middleware accepts only multipart/form-data, so reversing the middleware order will not work.
Potential solution: express-fileupload
file remained uploaded in the cloud.
What is meant by cloud here? Multer lets you store the file onto a storage or on memory, and then subsequent middleware calls get the file and its info in the req.file field.
Your validator validateDbData requires the presence of the file, which leaves you with the option of housekeeping(to be performed in validateDbData middleware) if the form is not valid, depending on the use-case you should carefully chose where multer middleware stores the file, on memory or on a storage.
I have an API in one nodejs project as below which receive multiple attachment from UI:
const upload = multer()
router.post('/upload', upload.array("attachments"),controller.getSomething);
getSomething is supposed to call another POST API using Axios only, which is in another NodeJs project which accept these attachments and process it. It as well accept multiple files via multer.
I am unsure how could i send multiple files as a request from one Nodejs project to another at once. could you please favour.
I had to set formdata as below:
const formData=new FormData();
for(let file of req.files){
formData.append("attachments",file.buffer,file.originalname);
}
And passed the formdata to other api via axios.
You can do the following steps:
When you upload the temporary files (coming from UI), save them in the temporary folder.
Pass all the files names to the POST API using Axios.
In the post API, read all the files from the temporary folder and stream them to the destination.
controller.getSomething = (req, res, next) => {
// get the file names
const fileNames = req.files.map(filename => filename);
// now post this filenames to the
axios.post('/pathname', {fileNames})
// or, you can use get request
}
Reading files in the post Request:
var promises= ['file1.css', 'file2.css'].map(function(_path){
return new Promise(function(_path, resolve, reject){
fs.readFile(_path, 'utf8', function(err, data){
if(err){
console.log(err);
resolve(""); //following the same code flow
}else{
resolve(data);
}
});
}.bind(this, _path));
});
Promise.all(promises).then(function(results){
//Put your callback logic here
response.writeHead(200, {"Content-Type": "text/css"});
results.forEach(function(content){response.write(content)});
response.end();
});
#copied from this link. You should check the different answers that can help you.
I need to serve local files from a different server using node. The api endpoint are being handled by express.
The goal is not to contain the entire file in memory instead stream the data so it shows the output to the enduser progressively.
By reading the stream api documentation i came up with this solution with a combination with expressjs response. Here is the example:
const open = (req, res) => {
const formattedUrl = new url.URL(
"https://dl.bdebooks.com/Old%20Bangla%20Books/Harano%20Graher%20Jantra%20Manob%20-%20Shaktimoy%20Biswas.pdf"
);
const src = fs.createReadStream(formattedUrl);
return src.pipe(res);
};
But when i hit this express endpoint http://localhost:3000/open it throws following error:
TypeError [ERR_INVALID_URL_SCHEME]: The URL must be of scheme file
I would like to display the file content inline! What I am doing wrong? Any suggestions will be greatly appreciated. :)
fs.createReadStream() operates on the file system. It does not accept an http or https URL. Instead, you need to use something like http.get() to make an http request and then return a readable stream that you can then pipe from.
const open = (req, res) => {
const formattedUrl = new url.URL("https://dl.bdebooks.com/Old%20Bangla%20Books/Harano%20Graher%20Jantra%20Manob%20-%20Shaktimoy%20Biswas.pdf");
http.get(formattedUrl, (stream) => {
stream.pipe(res);
}).on('error', (err) => {
// send some sort of error response here
});
};
This question already has answers here:
Nodejs POST request multipart/form-data
(5 answers)
Closed 5 years ago.
File uploading using multer is not happening
My code:
File read from html and pass to external url
router.post('/fileUpload',function (req,res) {
request({
uri: http//example.com/upload, // url of other server
method: "POST",
form: {
"name":req.body.name,
"image":? //image from html - no idea how to pass the req.files here
}
}, function (error, response, body) {
------------
------------
}
});
other server url : /example.com/upload
This is the code to upload image using multer
app.post('/upload',function(req,res){
var storage = multer.diskStorage({
destination: function (req, file, callback) {
callback(null, 'uploads');
},
filename: function (req, file, callback) {
callback(null, file.fieldname + '-' + Date.now());
}
});
var upload = multer({ storage : storage }).array('productImage');
upload(req,res,function(err) {
if(err) {
return res.json({'success':0,'result':{},'errorMessage':'Unknown Error'});
}
return res.json({'success':1,'result':{},'errorMessage':''});
});
});
Create readStream from file uploaded and pipe it to the another server.check this link https://www.npmjs.com/package/request, you will easily get this done.
The simple answer would be to create a read stream from the uploaded file and pipe it to the second server, like so:
fs.createReadStream(req.files[0].path).pipe(request.post('http//example.com/upload'))
However, there are many ways to make this work. The most efficient of which is to use a binary stream from the initial upload all the way to the second server. You want to avoid using your first server as a storage for all of the uploads.
Here's how I would do it instead:
Use jQuery file upload on client side upload
(Or any other library/approach to upload the raw binary stream)
$('#fileupload').fileupload({
url: 'https://server1.com/upload'
})
Use Formidable (or other library like multer) to handle upload server-side
(This will allow you to read the binary stream in parts, and handle each part as it comes)
app.post('/upload',function(req,res){
var form = new formidable.IncomingForm();
form.parse(req);
form.onPart = function(part) {
fs.createReadStream(part).pipe( request.post('http//example.com/upload'))
}
}
When each part of the binary upload is received, you can to stream the binary directly to the second server via pipe() to avoid having to store it on the first server whatsoever.
The key to making this work is to look at the file upload and transfer as a stream of binary bits. When the user uploads a file (req.body) you want to create a read stream from the file, and pipe the binary over the request.post().
I am trying to send a file's content to the client in my request, but the only documentation Express has is it's download function which requires a physical file; the file I am trying to send comes from S3, so all I have is the filename and content.
How do I go about sending the content of the file and appropriate headers for content type and filename, along with the file's content?
For example:
files.find({_id: id}, function(e, o) {
client.getObject({Bucket: config.bucket, Key: o.key}, function(error, data) {
res.send(data.Body);
});
});
The type of file depends on the file obviously. Have a look at this:
http://en.wikipedia.org/wiki/Internet_media_type
If you know what exactly is your file, then assign one of these to response ( not mandatory though ). You should also add the length of the file to response ( if it is possible, i.e. if it is not a stream ). And if you want it to be downloadable as an attachment, then add Content-Disposition header. So all in all you only need to add this:
var filename = "myfile.txt";
res.set({
"Content-Disposition": 'attachment; filename="'+filename+'"',
"Content-Type": "text/plain",
"Content-Length": data.Body.length
});
NOTE: I'm using Express 3.x.
EDIT: Actually Express is smart enough to count content length for you, so you don't have to add Content-Length header.
This is a great situation to use streams. Use the knox library to simplify things. Knox should take care of setting the needed headers to pipe files to the client
var inspect = require('eyespect').inspector();
var knox = require('knox');
var client = knox.createClient({
key: 's3KeyHere'
, secret: 's3SecretHere'
, bucket: 's3BucketHer'
});
/**
* #param {Stream} response is the response handler provided by Express
**/
function downloadFile(request, response) {
var filePath = 's3/file/path/here';
client.getFile(filePath, function(err, s3Response) {
s3Response.pipe(response);
s3Response.on('error', function(err){
inspect(err, 'error downloading file from s3');
});
s3Response.on('progress', function(data){
inspect(data, 's3 download progress');
});
s3Response.on('end', function(){
inspect(filePath, 'piped file to remote client successfully at s3 path');
});
});
}
npm install knox eyespect