I want to implement file uploading via nginx and FileAPI on client side. So I have the following questions:
Which is module better suite for this task nginx-upload-module or native clientbodyinfileonly or something else?
How to check that user is authenticated before uploading starts (maybe touch backend and return some data back to nginx like user_id)
How to rename file to hash to be looks like the following www.mysite.com/files/011/b0f/639/011b0f639f69491e9e4cbaf41656297f.jpg ?
How to make and save three copy of uploaded image several sizes (128x128, 96x96, 30x30)?
Which is module better suite for this task nginx-upload-module or
native clientbodyinfileonly or something else?
I just had a look at nginx-upload-module and this is a 7 years old nginx module to handle multipart requests. Nginx has been supporting multipart uploads for years now, so you do not need to change anything to your nginx setup as long as you're not running a 7 years old version!
For the remaining questions I'll give you an example, using :
Thumbnail to generate the thumbnails (You will need graphitemagick installed, but you can replace it with any other lib)
Q to easily generate the different thumbnails concurrently and have a clean code with no callback pyramid
Multer to handle the file upload server-side.
You could use other libraries to do the same thing, but this will show you one way to do it easily. For the sake of this example it is all in one single file.
var express = require('express');
var multer = require('multer');
var md5 = require('MD5');
var Q = require('q');
var Thumbnail = require('thumbnail');
var app = express();
var targetFolder = '/var/www/mysite/files/full/';
var thumbnailFolder = '/var/www/mysite/files/thumbs/';
var thumbnail = new Thumbnail(targetFolder, thumbnailFolder);
var makeThumbnail = Q.nbind(thumbnail.ensureThumbnail,thumbnail);
var thumbnailSizes = [30,96,128];
app.post('/upload', function(req,res,next) {
//Check the authentication here, this is the first middleware.
if(authenticated) {
next();
} else {
res.send(401);
}
}, multer({
dest: targetFolder,
rename: function (fieldname, filename) {
//Rename the file with it's name + date hash
return md5(filename + Date.now());
},
onFileUploadComplete: function(file, req, res) {
var resizePromises = [];
for(var i in thumbnailSizes) {
var p = makeThumbnail(file.name,thumbnailSizes[i],thumbnailSizes[i]);
resizePromises.push(p);
}
Q.all(resizePromises).done(function(){
// The file has been resized to the specified format.
res.send(200);
},function(err) {
console.log(err);
res.send(500);
// Error !
});
}
});
Related
Making a basic blog with an admin section to learn the basics of node and express. I just implemented multer middleware to save images for a blog post to a folder ("images") on the server - not to mongo or an s3 bucket - keeping it simple for now to learn.
I am using EJS and using res.render to send and render the frontend. However, I want to put the image in the EJS file as well. I've tried simply passing in the filename like so:
res.render(path.resolve(__dirname, "../", "views", "posts.ejs"), {postData, file});
postData being the data on the post from the mongodb collection. All this does is send the filename itself which is not helpful.
I've looked around, but don't seem to find an answer to this, or I'm over thinking this?
Here is the rest of the code for the controller:
const path = require("path");
const fs = require('fs');
const Post = require('../models/modelPosts');
exports.getPost = (req, res, next) => {
const postPath = req.params.post;
Post.findOne({ postPath: postPath }, (error, postData) => {
if (error) { return next(error) };
if (postData.postReadyToView == true) {
// find the correct image
fs.readdirSync('./images').forEach(file => {
const stringOfFile = JSON.stringify(file);
const stringOfPathJPEG = JSON.stringify(postPath + ".jpeg");
const stringOfPathJPG = JSON.stringify(postPath + ".jpg");
const stringOfPathPNG = JSON.stringify(postPath + ".png")
// send the ejs file and image
if (stringOfFile == stringOfPathJPEG ||
stringOfFile == stringOfPathJPG ||
stringOfFile == stringOfPathPNG) {
res.render(path.resolve(__dirname, "../", "views", "posts.ejs"), {
postData, file
});
}
})
} else {
res.redirect(404, "/404");
}
})
};
Send the file path of the page to be rendered as data, register the image garden folder (ex: public/images) as a static folder using express.static in nodejs, and load the image when the file path is loaded in the rendering page. I think you can.
I am working on a project (web, node.js, Firebase functions, hosting and storage) that needs the following functionality:
user selects file via
<input type="file" multiple>
other button runs a script that aims to upload selected file(s) to storage.
The problem is that - according to the documentation of google storage admin with node.js a file can only be uploaded using a path e.g.:
const options = {
destination: bucket.file('existing-file.png'),
resumable: false
};
bucket.upload('local-img.png', options, function(err, newFile) {
// Your bucket now contains:
// - "existing-file.png" (with the contents of `local-img.png')
});
BUT file input field is not allowed to let the browser know what the exact path is.
It allows me to know:
the file name
the file URL (URL.createObjectURL(...) -» "blob:https://mis...")
the file itself as a blob
Summing it up the page cannot hand over the variant the upload mechanism can handle.
Does anybody know the way how to do this?
Google Function call:
var imageSelected = document.getElementById("imageSelected");
var imageList = imageSelected.files;
var imagePath = imageList.webkitRelativePath;
for (var i = 0; i < imageList.length; i++) {
imageFile = imageList[i];
imageFileURL = URL.createObjectURL(imageFile).toString();
var imageUpload = firebase.functions().httpsCallable('imageUpload');
imageUpload({ file: imageFile, fileName: imageFile.name, filePath: imageFileURL });
URL.revokeObjectURL(imageFileURL)
}
Function:
exports.imageUpload = functions.https.onCall((data, context) => {
const storage = new Storage();
var filePath = data.filePath;
var fileName = data.fileName;
var file = data.file;
const options = {
destination: bucket.file('TEST/' + fileName),
resumable: false
};
return new Promise((Resolve, Reject) => {
async function uploadFile() {
storage
.bucket("..............appspot.com")
.upload(file, options);
Resolve(fileName);
}
uploadFile().catch(error => {
Reject(error);
});
});
});
The problem here is that you are trying to upload the file with your Cloud Function as if the cloud function had access to the file in your local filesystem, which it has not being on the server side.
What you need to do is to base64 encode the file as a string, and put that string in the JSON payload, then decode it in the function so that the file can be upload. Note that this is only recommended for small files.
If you need to do this for bigger files a better alternative would to perform the upload in your client, which will then have access to your local filesystem, using the client SDK instead of the admin SDK.
I have this file/folder structure:
+--public
| +--/productsImg/ //folder where i want to move images
+--routes
| +--products.js //the part that tells the images to move to /public/productImg/
In order to move a specific image i have used EXPRESS-FILEUPLOAD module.
I want all the images to be in /public/productsImg/ folder. SEE the above for the folder/file structure.
In products.js i wrote the following code:
var image = req.files; // in VIEW <input type="file" name="image"/>
var uploadPath = path.resolve(__dirname, '../public/productsImg/' + image.name);
//console.log(uploadPath) gives nothing ;
image.mv(uploadPath, (err) => {
if (err) {
return res.status(500).send(err);
}
res.send('File uploaded to ' + uploadPath);
});
I have this error in nodemon console : POST /products/store 500 9.733 ms - 16104
I think may be i'm not using correctly the path.resolve part and my uploadPath variable (file) is unknow, i think.
Have seen other posts, i couldn't figure it out. Any ideas?
May be it can help someone: solution
1) Added var path = required('path'); in product.js eventhough i added this in my app.js. It is important otherwise it won't work.
2) Created /tmp/ folder (root of your project) and add this in app
app.use(fileUpload({
useTempFiles : true,
tempFileDir : '/tmp/'
}));
3) Changed the code to
var { image } = req.files;
var uploadPath = path.resolve(__dirname, '../public/productsImg/', image.name); // important
console.log(uploadPath);
image.mv(uploadPath, (err) => {
if (err) {
return res.status(500).send(err);
}
res.send('File uploaded to ' + uploadPath);
//or whatever you want to do
});
According to your example, your file object should be in req.files.image.
See express-fileupload documetnation:
Example:
You're uploading a file called car.jpg
Your input's name field is foo:
In your express server request, you can access your uploaded file from req.files.foo:
app.post('/upload', function(req, res) {
console.log(req.files.foo); // the uploaded file object
});
hie, in express-fileupload you access the image via the name attribute, so you might consider changing
var image = req.files
to
var image = req.files.image
hope it helps
I am trying to display file list from folders .
my folder structure is like below
Invoices
1. error
2. processed
3. unprocessed
I have created node api for same which i am calling on my html page. code for the same is as below
const fs = require('fs');
var express = require('express');
var cors = require('cors');
var app = express();
app.use(cors());
app.use(express.static(__dirname));
var flength;
var filename;
var currentFile;
var items = [];
var dir1 = 'Invoices';
var filepath = [];
var readFolder = function(dir1) {
var countt = function(filename) {
var currentFile = dir1 + '/' + filename;
fs.readdir(currentFile, (err, files) => {
flength = files.length;
var fileArrayList = [];
for (var f in files) {
var record = {
filename: files[f],
filepath: dir1 + '/' + filename + '/' + files[f]
}
fileArrayList.push(record);
}
items.push({
'file': filename,
'count': flength,
'files': fileArrayList
});
});
}
var ReadFirst = function(dir1) {
fs.readdir(dir1, (err, files) => {
for (var i in files) {
var filename = files[i];
var currentFile = dir1 + '/' + filename;
var stats = fs.statSync(currentFile);
if (stats.isDirectory()) {
countt(filename);
}
}
});
}
ReadFirst(dir1);
}
setTimeout(function(str1, str2) {
readFolder(dir1);
}, 1000);
app.get('/FileCount', function(req, res) {
res.send(items);
});
app.listen(4000);
console.log('Listening on port 4000');
When i add or delete files from any folder then its not reflecting on my html page.need help for this.
thank you.
This is happening because of the way you've implemented this.
Your client (the HTML page) requests the server (NodeJS) API for
some data. In this case, it is the list of files in a folder. The server sends the response based on the state of files at the time (plus ∆).
You display those results in the HTML page. Now, there is no live link between your HTML page and your backend server. This means any changes that happen after this point won't be automatically reflected in the page.
You can do two things here:
Call your API repeatedly after an interval of few seconds. (If you're using AngularJS, look into setTimeout function.
Use sockets for having a real-time link. One good documentation here:
https://loopback.io/doc/en/lb2/Realtime-socket-io.html
This is more of a design issue and your NodeJS API looks fine.
I want to export my data into csv file so for that purpose i used fast-csv in node js. its working fine my code is
var csv = require("fast-csv");
app.get('/file/exportToExcel', function(req, res) {
var whereCondi = {
'Id': req.query.id,
}
var response = {};
table.find(whereCondi, function(err, tableData) {
if (err) {
response.status = 400;
response.message = err.message;
res.send(response);
} else {
var csvStream = csv.createWriteStream({headers: true}),
writableStream = fs.createWriteStream("code.csv");
writableStream.on("finish", function() {
});
csvStream.pipe(writableStream);
for (var i = 0; i < tableData.length; i++) {
csvStream.write({allcodes: tableData[i].code});
}
csvStream.end();
}
});
});
but the problem is its saving that csv file in my root folder i want to download that csv file when user click on export to excel please help me.
writableStream = fs.createWriteStream("coupons.csv");
This looks to be your problem if I'm understanding you correctly. The current code saves the csv file relative to the app file (basically in the same directory in your case).
Try something like:
writableStream = fs.createWriteStream("./some/directory/coupons.csv");
You should create the csv file in your directory an then delete it in the same way like that
const express = require('express')
const objectstocsv = require('objects-to-csv')
const fs = require("fs")
const app = express()
var data = [
{code: 'CA', name: 'California'},
{code: 'TX', name: 'Texas'},
{code: 'NY', name: 'New York'},
];
const PORT = process.env.PORT || 5000
app.get('/',async(req,res) => {
const csv = new objectstocsv(data);
// Save to file:
await csv.toDisk('./test.csv');
// Download the file
res.download("./test.csv",() => {
//Then delete the csv file in the callback
fs.unlinkSync("./test.csv")
})
})
Very late to the game but wanted to add in case other people were encountering same hurdle. Not sure if this is an ideal solution since I just started learning, but I got around this problem by wrapping the csv creation in an async function, and having that function called when a user clicks on a button.
Essentially, user clicks button > GET request to specific path > export csv and render success page.
index.js or server js file
const exportToCsv = () => {
...some code to get data and generate csv...
};
app.get('/download', async (req, res) => {
exportToCsv();
res.render('<your desired view>');
};
view or html
<button type='submit' onclick='someNavFunction()'>Export</button>
The someNavFunction() can be a helper function to navigate to a new path or some other solution that helps you hit '/download' that you created route for in server file.
This solution worked for me because I wanted to render a success page after download. You can add additional validation to only render if exported successfully etc.