I'm using Heroku to store my Express app and I can't use bodyParser, or formidable, because both uses temp files to store the posted file.
The file I'm posting is simply a large text file (csv). I don't really need to store the file anywhere, because the data will go into mongodb.
Is there anyway to parse the csv file as it is being uploaded?
Starting with Connect 2.9.0 (Express 3.4.0), bodyParser uses multiparty instead of formidable.
In multiparty's default configuration, it does not create temporary files. Read the docs to see exactly how to handle the incoming data, but you'll want to listen to the part event, which gives you a readable stream containing the uploaded file.
Related
I have a React user interface with a NodeJS express backend, through which users can upload files. To prevent tampering, I would like to generate a hash code for the file contents prior to the file upload, send that with the file to the express upload route, compute a hash code for the uploaded contents and then compare the two hash codes to ensure that the file contents are the same (within the limits of hash code uniqueness).
I'm able generate an MD5 hash code on the contents in the React UI and send that along with the file to the back end. But I can't figure out how to take the uploaded file contents and create a hash code for that.
I'm using Multer and tried Multer-MD5 as well as writing my own using CryptoJS. I suspect the issue lies in trying to convert the uploaded content (which seems to be a buffer) into something CryptoJS can process.
Or use another Hash generator.
Has anyone had to work through this before?
I worked out a solution, partly because I expect the uploaded content to be text.:
utils.fileHash(file.buffer.toString())
and
fileHash: (file) => {
return CryptoJS.MD5(file).toString();
},
I have read a lot of examples/tutorials on internet that explain how to upload and store files in PostgreSQL using NodeJS, Multer, Sequelize, Express, ...
With none of them I have understood how to do it in my project.
I have an Angular app where I have an input file, I have an API in NodeJS using pg-promise to link with my PostgreSQL database. I have done all my project api with pg-promise so I would like to use it and no Sequelize.
In the database I have read that the best is probably the BYTEA type for files like jpg, png, pdf, docx...
Here is my API hierarchy :
www file contains code to initiate the server
app.js file contains code to initiate express app and the require([library]) of what I need
index.js file contains code to link urls to function that are stored in queries.js
The only thing I want is How to store a file.
Thanks if you can help me
I strongly recommend you the use of multer library. It helps you with the storage of files in disk. You get the path where it is stored, and you could save the path on the database.
https://github.com/expressjs/multer
I'm using express js to serve files to an angular app and I'm deciding between these approaches:
res.status(201).json({ imagepath: '<URL>' });
res.sendFile('<URL>');
I'm thinking that sending the url will force an extra round trip so I should go with the sendFile but I want to doublecheck. Is there anything I'm missing?
The best option would be to store the image file on something like AWS S3 and then send the file URL using your Express application. You would use the aws-sdk package here to save the image to S3 (detailed article here). Then you would send the URL location of that image to your database to be stored.
I'm using sharp to process images on the server side and react dropzone to get the image file. When I post the file to the server from dropzone, I get the blob out of the request.body that looks like:
{ preview: 'blob:http%3A//127.0.0.1%3A3000/1451580a-b24f-478f-a9f7-338c0d790829' }
Optionally, before I send data to the server I can use FileReader (or something else) to do something with the image file instead of turning it into a blob.
Sharp takes:
A Buffer containing JPEG, PNG, WebP, GIF, SVG, TIFF
Raw pixel image data
A String containing the path to an image file, with most major formats supported.
How can I use what I have to provide sharp a supported format?
I recommend trying out a node.js module called Multer to help you access your photo file on your server.
https://github.com/expressjs/multer
First, on the client, you'll want to append your file to a FormData object like this:
// obtain the file from your react dropzone and save it to this file variable
const file = dropzone.file // not sure how you do this with react dropzone
const data = new FormData()
data.append('photo', file)
Then you'll send this FormData object to your server. On the server you'll use Multer on the route you're using for the photo for processing.
Make sure you npm install multer, and require it on your server or routes file. If you're sending a single file you'll use the multer 'single' method. If you want to do anything different check out the API documentation.
app.post('/photos', multer().single('photo'), controller.processPhoto);
In this example route, you're sending a POST request to /photos, multer is looking for a file with a FormData key of 'photo' and appending that to the request object.
Then in this made up 'controller.processPhoto' method you'll have access to the image as a property of the request object, on req.file. With this you can easily access a lot of good information including the image buffer req.file.buffer which it sounds like you need. (also mimetype, original name etc.)
This should be enough to get you started.
In a Node.js app, is it possible to upload a file (multipart/form-data) directly into a buffer on the server instead of saving the file to the filesystem?
In psuedo code you typically do this with a file upload form
router.post('/upload', function(req, res, next){
// file is saved to /tmp/dk3idkalel4kidkek
// do some processing on the file
// manually save file to /www/myapp/uploads/originafilename.ext
});
Is it possible to skip the part about saving the file to /tmp and just capturing the uploaded file as a Buffer and streaming it directly to the destination directory?
An example would be uploading an image, then optimizing the image, then saving it to it's intended destination. Can you just capture the incoming stream, optimize it then save it, all without the initial saving to the /tmp directory?
Multer is a node.js middleware for handling multipart/form-data, which is primarily used for uploading files. The memory storage engine stores the files in memory as Buffer objects. When using memory storage, the file info will contain a field called buffer that contains the entire file.
This should do exactly what you need.
You can read more here
Multer module uses busboy for upload, so an alternative would be to use busboy for express: connect-busboy