Im struggling to find material on this
I have a rest API, written in node.js, that uses mongoDB.
I want users to be able to upload images (profile pictures) and have them saved on the server (in mongoDB).
A few questions, Ive seen it is recommended to use GridFS, is this the best solution?
How do i send these files? Ive seen res.sendFile, but again is this the best solution?
If anyone has any material they can link me I would be appreciative
thanks
You won't be able to get the file object on the server directly. To get file object on the server, use connect-multiparty middleware. This will allow you to access the file on the server.
var multipart = require('connect-multiparty');
var multipartmiddleware = multipart();
var mv = require('mv');
var path = require('path');
app.post("/URL",multipartmiddleware,function(req,res){
var uploadedImage = req.files.file;
for (var i = 0; i < uploadedImage.length; i++) {
var tempPath = uploadedImage[i].path;
var targetPath = path.join(__dirname ,"../../../img/Ads/" + i + uploadedImage[i].name);
mv(tempPath, targetPath, function (err) {
if (err) { throw err; }
});
}
})
Use file system
Generally in any database you store the image location in the data as a string that tells the application where the image is stored on the file system.
Unless your database needs to be portable as a single unit, the storing of images inside of the database as binary objects generally adds unnecessary size and complexity to your database.
-Michael Stearne
In MongoDB, use GridFS for storing files larger than 16 MB.
- Mongo Documentation
Therefore unless your images will be over 16 MB, you should either store the file on a CDN (preferable) or the server's own file system and save its URL to user's document on the database.
Local file system implementation
This method uses Busboy to parse the photo upload.
in relevant html file:
<input type="file" title="Choose a file to upload" accept="image/*" autofocus="1">
Handler function for your photo upload route in server file (you will need to fill in the variables that apply to you and require the necessary modules):
function photoUploadHandlerFunction (req, res) {
var busboy = new Busboy({ headers: req.headers })
busboy.on('file', function (fieldname, file, filename, encoding, mimetype) {
const saveToDir = path.join(__dirname, uploadsPath, user.id)
const saveToFile = path.join(saveToDir, filename)
const pathToFile = path.join(uploadsPath, user.id, filename)
const writeStream = fs.createWriteStream(saveToFile)
createDirIfNotExist(saveToDir)
.then(pipeUploadToDisk(file, writeStream))
.then(findUserAndUpdateProfilePic(user, pathToFile))
.catch((err) => {
res.writeHead(500)
res.end(`Server broke its promise ${err}`)
})
})
busboy.on('finish', function () {
res.writeHead(200, { 'Connection': 'close' })
res.end("That's all folks!")
})
return req.pipe(busboy)
}
Where the promise functions createDirIfNotExist and pipeUploadToDisk could look like this:
function createDirIfNotExist (directory, callback) {
return new Promise(function (resolve, reject) {
fs.stat(directory, function (err, stats) {
// Check if error defined and the error code is "not exists"
if (err) {
if (err.code === 'ENOENT') {
fs.mkdir(directory, (err) => {
if (err) reject(err)
resolve('made folder')
})
} else {
// just in case there was a different error:
reject(err)
}
} else {
resolve('folder already existed')
}
})
})
}
function pipeUploadToDisk (file, writeStream) {
return new Promise((resolve, reject) => {
const fileWriteStream = file.pipe(writeStream)
fileWriteStream.on('finish', function () {
resolve('file written to file system')
})
fileWriteStream.on('error', function () {
reject('write to file system failed')
})
})
}
To answer your question 'How do I send these files?', I would need to know where to (MongoDB, to the client...). If you mean to the client, you could serve the static folder where they are saved.
If you still want to learn about implementing GridFs tutorialspoint have a good tutorial
More material
Good tutorial on handling form uploads
Tutorial using the node-formidable module
If you're using the mongoose odm you can use the mongoose-crate module and send the file wherever for storage.
Also, this is a good case for shared object storage like AWS S3 or Azure blob storage. If you are running a distributed setup in something like AWS, you usually don't want to store photos on the local server.
Store the url or key name in the database that points to the S3 object. This also integrates with CloudFront CDN pretty easily.
As suggested before. MultiPart for the actual upload.
Related
Looking for help on Uploading and Retrieving Images from MongoDb using multer.
My front end is ReactNative.(Not sure if this is needed but just to be sure.)
Multer
Problem: After looking and following tutorials i'm able to encode my path to base64 and upload it to my DB but now i'm confused how to retrieve the file from my DB. I saw some tutorials about decoding it from base64 but I don't quite understand how do I go about retrieving an image and displaying it in postman. (I tried looking but haven't found anything that gives me an answer. I'm sorry if this is a duplicated question. If you could point me in a direction or give me some advice I would be really greatful.)
**POST**
route.post("/sad", upload.single("image"), (req, res, next) => {
console.log(req.file);
const img = fs.readFileSync(req.file.path);
const img_enc = img.toString('base64');
const obj = {
usrImage: {
data: new Buffer.from(img_enc, 'base64'),
contentType: "image/jpg",
},
};
console.log(obj);
const newAccout = new account(obj);
newAccout.save();
});
**RETRIEVE**
route.get('/sad',(req,res)=>{
img.find({}).then((img)=>{
res.json(img)
//How do decode my buffer to show an image in Postman?
})
}
)
I am trying to create a userprofile where a username,password and image is saved. If you can help save an Image and then retrieve it from my accounts collection.
Hey I would advise that you start using a 3rd party for file upload like cloudinary very good way of managing files i.e images or video...
I am not that well of with multer but I can give a quick code example using Formidable does the same work as multer
Before you can start you'd need to make an account on cloudinary.com(don't worry its free)
Code below is how you could handle file upload
const Formidable = require("formidable"); //Meant for body parsing
const cloudinary = require("cloudinary").v2; // file uploader
//This below is your connection/configuration to get access to your cloudinary account so cloud_name, api_key and api_secret you'll get in your home dashboard(Cloudinary)
cloudinary.config({
cloud_name: process.env.CLOUD_NAME,
api_key: process.env.API_KEY,
api_secret: process.env.API_SECRET,
});
router.post('/api/file-upload', (req, res)=>{
const form = new Formidable.InconmingForm();
form.parse(req, (error, fields, files)=>{
const {file} = files
cloudinary.uploader.upload(file.path, {folder:"/"}, (err, res)=>{
const file_url = res.secure_url //This would be the url for your file given back by cloudinary
})
})
})
This script should upload your file and the file_url will be having the url of the file that you upload having ssl then after that you can now continue saving to mongoDB
Cloudinary docs for NodeJS
https://cloudinary.com/documentation/node_integration
Nice clear and understandable docs
Shameless plug
If you get lost you can check this video out on YouTube that I made handling file upload with cloudinary then save url given back to mongoDB
https://youtu.be/mlu-tbr2uUk
First call api find one
you will need fs module to complete following query
const fs = require('fs');
let data = await db.user.findOne({
where: {
id = req.body.id
}
})
// _________________ base 64 string data from findone query data
// |
let buff = new Buffer(data.image, 'base64');
let name = name.jpeg
let path = `tmp/${name}`; // <--- destination and file name you want to give to your file
fs.writeFileSync(path, buff);// < --this will write file to given path
fs.readFile(path, function (err, content) {// <------to send file in postman response
if (err) {
res.writeHead(400)
console.log(err);
res.end("No such image");
} else {
//specify the content type in the response will be an image
res.writeHead(200);
res.end(content);
}
});
fs.unlink(path, (err) => { // <-----to delete file from tmp directory
if (err) {
console.log(err)
}
})
Try this and switch to preview tab in postman.
I haven't tried it but maybe it helps.
route.get('/sad',(req,res)=>{
img.find({}).then((img)=>{
res.setHeader('contentType','image/jpg').send(img)
})
})
How do I create a file in express and node on my server and then download it to my client. I am using NextJS for my frontend and backend. I am confused on how I would download the file on the front end after the file is created on the root of the server folder. Since I am using React for my frontend whenever I try to visit that filepath it tries to take me to a page instead of the file
Here is what I have in my express route in node
var xls = json2xls(json, {
fields
});
// If there isn't a folder called /temp in the
// root folder it creates one
if (!fs.existsSync('./temp')) {
fs.mkdirSync('./temp');
}
const fileName = `temp/${req.user.first_name}${req.body._id + Date.now()}.xlsx`
// fs.writeFileSync(fileName, xls, 'binary');
fs.writeFile(fileName, xls, 'binary', function (err, result) {
if (err) {
return console.log(err);
}
console.log(result, 'this is result')
});
Here is what I have on my frontend
axios.post('api/download',payload)
.then(res => {
const link = document.createElement('a');
link.href = res.data.url;
link.download
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
})
.catch(err => {
throw err
})
Can you make request with GET on api, and.
Make request with GET.
Make temp directory to be static resources directory:
app.use(express.static('temp')); // app is your express instance.
// Maybe you have to correct temp's path
Response the post request with file url data
fs.writeFile(fileName, xls, 'binary', function (err, result) {
if (err) {
return console.log(err);
res.status(500).json({err});
}
console.log(result, 'this is result');
res.json({url: 'http://localhost:8080/temp/' + fileName}); // res is response object of you router handler.
// Maybe you have correct the server address
});
On other way, you can send the xls binary direct to client, in the client you create a BLOB object from the response, then create download link for the blob object.
I have a script that can call a RESTful API and retrieve CSV data from a report in chunks. I'm able to concatenate, parse, and display this data in the console. I am also able to write this CSV data to a local file and store it.
What I am trying to figure out is how to skip creating a file to store this data before uploading it to GCS and instead transfer it directly into Google Cloud Storage to save as a file. Since I am trying to make this a serverless cloud function, I am trying to stream it directly from memory into a Google Cloud Storage file.
I found this 'Streaming Transfers' documentation on google, but it only references doing this with 'gsutil' and I am struggling to find any examples or documentation on how to do this with node.js. I also tried to follow this answer on Stack overflow, but it's from 2013 and the methods seem a little out-dated. My script also isn't user-facing, so I don't need to hit any routes.
I am able to upload local files directly to my bucket using the function below, so Authentication isn't an issue. I'm just unsure how to convert a CSV blob or object in memory into a file in GCS. I haven't been able to find many examples so wasn't sure if anyone else has solved this issue in the past.
const { Storage } = require('#google-cloud/storage');
const storage = new Storage({
projectId,
keyFilename
});
function uploadCSVToGCS() {
const localFilePath = './test.csv';
const bucketName = "Test_Bucket";
const bucket = storage.bucket(bucketName);
bucket.upload(localFilePath);
};
I also found a 3rd party plugin that Google references called 'boto' that seems to do what I want, but this is for python, not node.js unfortunately.
Streaming object data to Cloud Storage is illustrated in the documentation. You will need to understand how node streams work, and make use of createWriteStream. The sample code is not exactly what you want, but you'll use the same pattern:
function sendUploadToGCS (req, res, next) {
if (!req.file) {
return next();
}
const gcsname = Date.now() + req.file.originalname;
const file = bucket.file(gcsname);
const stream = file.createWriteStream({
metadata: {
contentType: req.file.mimetype
},
resumable: false
});
stream.on('error', (err) => {
req.file.cloudStorageError = err;
next(err);
});
stream.on('finish', () => {
req.file.cloudStorageObject = gcsname;
file.makePublic().then(() => {
req.file.cloudStoragePublicUrl = getPublicUrl(gcsname);
next();
});
});
stream.end(req.file.buffer);
}
#doug-stevenson thanks for pushing me in the right direction. I was able to get it to work with the following code:
const { Storage } = require('#google-cloud/storage');
const storage = new Storage();
const bucketName = 'test_bucket';
const blobName = 'test.csv';
const bucket = storage.bucket(bucketName);
const blob = bucket.file(blobName);
const request = require('request');
function pipeCSVToGCS(redirectUrl) {
request.get(redirectUrl)
.pipe(blob.createWriteStream({
metadata: {
contentType: 'text/csv'
}
}))
.on("error", (err) => {
console.error(`error occurred`);
})
.on('finish', () => {
console.info(`success`);
});
};
Hey everyone so I am trying to make this type of request in nodejs. I assume you can do it with multer but there is one major catch I don't want to download the file or upload it from a form I want to pull it directly from s3, get the object and send it as a file along with the other data to my route. Is it possible to do that?
Yes it's completely possible. Assuming you know your way around the aws-sdk, you can create a method for retrieving the file and use this method to get the data in your route and do whatever you please with them.
Example: (Helper Method)
getDataFromS3(filename, bucket, callback) {
var params = {
Bucket: bucket,
Key: filename
};
s3.getObject(params, function(err, data) {
if (err) {
callback(true, err.stack); // an error occurred
}
else {
callback(false, data); //success in retrieving data.
}
});
}
Your Route:
app.post('/something', (req, res) => {
var s3Object = getDataFromS3('filename', 'bucket', (err, file) => {
if(err) {
return res.json({ message: 'File retrieval failed' });
}
var routeProperties = {};
routeProperties.file = file;
routeProperties.someOtherdata = req.body.someOtherData;
return res.json({routeProperties});
});
});
Of course, the code might not be totally correct. But this is an approach that you can use to get what you want. Hope this helps.
There are two ways that I see here, you can either:
pipe this request to user, it means that you still download it and pass it through but you don't save it anywhere, just stream it through your backend.
There is a very similar question asked here: Streaming file from S3 with Express including information on length and filetype
I'm just gonna copy & paste code snippet just for the reference how it could be done
function sendResponseStream(req, res){
const s3 = new AWS.S3();
s3.getObject({Bucket: myBucket, Key: myFile})
.createReadStream()
.pipe(res);
}
if the file gets too big for you to easily handle, create presigned URL in S3 and send it through. User then can download the file himself straight from S3 for a limited amount of time, more details here: https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html
I am receiving an image file sent from an Ajax request:
var data = canvas.toDataURL('image/jpg', 1.0);
$.post({
url: "/upload-image",
data: {
file: data
}
}).done(function(response) {
....
})
}
And on the server side, I want to transmit the image file to an API
function getOptions(buffer) {
return {
url: '.../face_detection',
headers: headers,
method: 'POST',
formData: {
filename: buffer
}
}
}
router.post('/upload-image', function(req, res, next) {
console.log('LOG 0' + Date.now());
var data_url = req.body.file;
var matches = data_url.match(/^data:.+\/(.+);base64,(.*)$/);
var ext = matches[1];
var base64_data = matches[2];
var buffer = new Buffer(base64_data, 'base64');
console.log('LOG 1' + Date.now());
request(getOptions(buffer), function(error, response, body) {
res.json(body);
console.log(Date.now());
});
});
The problem that I have is that the lines between LOG 0 and LOG 1 are very slow, a few seconds. But the image is only 650kb. Is there a way to accelerate this?
Using another method to read the header, avoid the buffer, change the uploading process. I don't know but I'd like to be faster.
Thank you very much.
I would suggest using a library to handle some of this logic. If you would prefer to keep a lean dependency list, you can take a look at the source of some of these modules and base your own solution off of them.
For converting a data URI to a buffer: data-uri-to-buffer
For figuring out a file type: file-type
I would especially recommend the file-type solution. A safer (can't say safest) way to ensure what kind of file a Buffer is is to inspect aspects of the file. file-type seems to at least take a look at the Magic Number of the file to check type. Not foolproof, but if you are accepting files from users, you have to accept the risks involved.
Also have a look at Security Stack Exchange questions for good practices. Although the following say PHP, all server software runs the risk of being vulnerable to user input:
Hacker used picture upload to get PHP code into my site
Can simply decompressing a JPEG image trigger an exploit?
Risks of a PHP image upload form
"use strict";
const dataUriToBuffer = require('data-uri-to-buffer'),
fileType = require("file-type"),
express = require("express"),
router = express.Router(),
util = require("util"),
fs = require("fs"),
path = require("path");
const writeFileAsync = util.promisify(fs.writeFile);
// Keep track of file types you support
const supportedTypes = [
"png",
"jpg",
"gif"
];
// Handle POSTs to upload-image
router.post("/upload-image", function (req, res, next) {
// Did they send us a file?
if (!req.body.file) {
// Unprocessable entity error
return res.sendStatus(422);
}
// Get the file to a buffer
const buff = dataUriToBuffer(req.body.file);
// Get the file type
const bufferMime = fileType(buff); // {ext: 'png', mime: 'image/png'}
// Is it a supported file type?
if (!supportedTypes.contains(bufferMime.ext)) {
// Unsupported media type
return res.sendStatus(415);
}
// Save or do whatever with the file
writeFileAsync(path.join("imageDir", `userimage.${bufferMime.ext}`), buff)
// Tell the user that it's all done
.then(() => res.sendStatus(200))
// Log the error and tell the user the save failed
.catch((err) => {
console.error(err);
res.sendStatus(500);
});
});