does anyone know how to use tinyPNG's API with multer? The docs seem deceptively simple:
var source = tinify.fromFile("unoptimized.jpg");
source.toFile("optimized.jpg");
though there's no clear indication of where this is meant to go, especially in something as convoluted as this:
var storage = multer.diskStorage(
{
destination: function (req, file, callback) {
callback(null, './uploads');
},
filename: function (req, file, callback) {
//use date to guarantee name uniqueness
callback(null, file.originalname + '-' + Date.now());
}
}
);
//.any() allows multiple file uploads
var upload = multer({ storage : storage}).any()
app.post('/api/photo', function(req,res){
upload(req,res,function(err) {
if(err) {
return res.end("Error uploading file.");
}
res.end("File is uploaded");
});
});
Where am I meant to "intercept" the file uploaded by multer so that I can compress it with tinyPNG?
Thanks in advance for the help!
Use following basic sample that changes uploaded photo/gallery files:
// Import express and multer.
var express = require('express');
var multer = require('multer');
// Setup upload.
var upload = multer({ dest: 'uploads/' });
var multipleFiles = upload.fields([{ name: 'photo', maxCount: 1 },
{ name: 'gallery', maxCount: 8 }]);
// Setup tinify.
var tinify = require("tinify");
tinify.key = "YOUR_API_KEY";
// Get request handler for '/' path.
var app = express();
app.get('/', function (req, res) {
res.setHeader("Content-Type", "text/html");
res.end(
"<form action='/api/photo' method='post' enctype='multipart/form-data'>" +
"<input type='file' name='photo' />" +
"<input type='file' name='gallery' multiple/>" +
"<input type='submit' />" +
"</form>"
);
});
// Upload file handler with '/api/photo' path.
app.post('/api/photo', multipleFiles, function (req, res) {
req.files['gallery'].forEach(function(file) {
// Your logic with tinify here.
var source = tinify.fromFile(file.path);
source.toFile(file.path + "_optimized.jpg");
});
res.end("UPLOAD COMPLETED!");
});
Feel free to change express middleware how you need it, just make sure you use upload.fields and authenticate using tinify.key = "YOUR_API_KEY";
https://github.com/expressjs/multer
https://tinypng.com/developers/reference/nodejs#compressing-images
I recently worked out a similar problem for myself using the tinify package and found the docs to be somewhat lacking.
I have a Vue front end collecting file uploads from the user using vue2dropzone. These are sent to a node / Express back end.
I have a need to compress the file and upload it to an S3 instance without storing on disk. That means using multer memory storage.
As a result there won’t be an ability to use tinify.fromFile() as there is no file stored locally.
In my images middleware:
Const multer = require(“multer”);
const tinify = require("tinify");
tinify.key = "your_key";
exports.singleFile = multer({ storage: multer.memoryStorage() }).fields([{ name: "file", maxCount: 1 }]);
exports.uploadCompImage = async (req, res, next) => {
try {
const fileName = `${req.params.name}${path.extname(req.files.file[0].originalname)}`;
const source = tinify.fromBuffer(req.files.file[0].buffer);
source.store({
service: "s3",
aws_access_key_id: "your_id",
aws_secret_access_key: "your_key
region: "your_region",
headers: {
"Cache-Control": "public"
},
path: `your_bucket/your_folder/${fileName}`
});
return res.status(200).send(`path_to_file/${fileName}`)
} catch (err) {
console.log(err);
next(err);
}
}
Then in my routes file:
Const images = require(“../middleware/images”);
// skipped several lines for brevity
productRouter
.route("/images/:name")
.post(images.singleFile, images.uploadCompImage)
This process creates a multer singleFile upload to memoryStorage, making the file available at req.files.file[0] (req.files[“file”] because I specified “file” as the name in multer fields, loop through this array if uploading multiple).
After setting that up I get the file name, set the source by using tinify to read from req.files.file[0].buffer as a buffer.
Then I set the source to my s3 instance and send back a public link to the file.
Hopefully this answer helps you. I could definitely see altering the process to change where the file goes or even write it to disk by altering the multer options.
Related
I am using the node.js as a backend and needs to upload the video on Amazon S3.
For that I am using the multer module but I need to know the efficient and standard way for uploading the video.
Generally we see that when we upload anything on the any good platfrom then there is a proper mechanism for uploading the video like:
When video is on uploading state, user get the response that how much percent is left for uploading the video
After uploading the video user get the response that video is uploaded successfully.
There is a handler which allow specific type of format to allow video.
There is a few limit size also which warn the user that maximum size is 20mb or 50 mb.
I am bit struggling about the good tutorial but unable to find as everywhere is sharing the tutorials about the image upload. So I thought I will raise the question which will help many others also regarding the same
I have implemented the process by which video will be uploaded on S3 but not getting the response after upload. It directly return the response and video will be uploaded in background.
I am sharing my implementation:
customapi.js file
const express = require('express');
const router = express.Router();
const helper = require('./file-upload');
const videoHandler = require('./videohandler');
// Post the video
router.post(
'/uploadvideo',
helper.single('media'),
videoHandler .uploadVideo
);
file-upload.js file
const AWS = require('aws-sdk')
const multer = require('multer')
const multerS3 = require('multer-s3')
const uuid = require('uuid/v1');
AWS.config.update({
accessKeyId: process.env.keyId,
secretAccessKey: process.env.accessKey,
});
const s3 = new AWS.S3();
const upload = multer({
storage: multerS3({
s3:s3,
bucket: process.env.bucketname,
acl: 'public-read',
contentType: multerS3.AUTO_CONTENT_TYPE,
metadata: function (req, file, cb) {
cb(null, {fieldName: file.fieldname})
},
key: function (req, file, cb) {
console.log(file) // This will print the filename which we can search in s3.
cb(null, uuid()+file.originalname)
console.log(uuid() + file.originalname)
}
})
})
videohandler.js file
module.exports = {
uploadVid: async (req, res) => {
try {
return res.send({message: "Done"})
} catch (error) {
console.log(error);
return res.send({message: "Error"})
}
}
}
I know what is happening exactly by which I am getting the response instantly instead of after the file upload.
When api calls -> helper.single('media') will invoke and process to start performing the task -> meanwhile videoHandler.uploadVideo will also called which directly send the response as there is no callback which understand the uploading process and return response accordingly.
Please share the best and efficient way as it supports many people also who is struggling for uploading the video.
Any help or suggestion is really appreciated.
Update Question
Many people sharing the way that video first upload on local disk and then it upload on S3. I need to know that is it a good behaviour. I mean for performing the task we need to do the double work instead of directly upload. It will consume the bandwidth and storage of the application on server.
I could handle it in this way. check whether is it applicable to your scenario.
App.js
router.post("/upload_service",
(req, res, next) => {
const upload = UploadController.upload.single('file')
upload(req, res, (err) => {
if (err) {
const error = new Error('Image upload error');
return next(error);
}
return next()
})
},
SomeController.createMethod)
UploadController.js
const multer = require('multer')
const multerS3 = require('multer-s3')
const AWS = require('aws-sdk')
const upload = multer({
storage: multerS3({
s3: new AWS.S3(),
bucket: 'bucket_name',
metadata: function (req, file, cb) {
cb(null, { fieldName: file.fieldname });
},
key: function (req, file, cb) {
const file_name_timestamp = Date.now().toString()
cb(null, `path/${file_name_timestamp}.${String(file.mimetype).split('/').pop()}`);
},
ContentType: "application/octet-stream",
})
})
module.exports = {
upload
}
SomeController.js
const createMethod = async function (req, res) {
if (!req.file) {
res.send('File missing')
}
}
Once file upload middleware was success, SomeController.createMethod will be triggered. using req.file, creation data can be visible.
In my MERN app, I am trying to access the images on the client-side that has been saved on the local database using multer.
The structure of my backend folder goes like this:
--- api
--- controllers
--- model
--- routes
--- config
--- db.js
--- appConfig.js
--- utils
--- uploads
--- multerMiddleware.js
--- app.js
The image uploading and storing to local DB works completely fine. When a new data is created, the data received by the client in API response contains the URL of the image uploaded so that it can be accessed again (like for displaying image thumbnail).
My code goes like:
App.js
const express = require("express");
const path = require('path');
const app = express();
const directory = path.join(__dirname, '/uploads');
app.use('/uploads', express.static(directory));
require("./config/db/db")();
require("./config/appRoutes/appRoutes")(app);
module.exports = app;
multerFile.js
const multer = require('multer');
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, __dirname + '/uploads')
},
filename: function (req, file, cb) {
const fileName = file.originalname.toLowerCase().split(' ').join('-');
cb(null, fileName);
}
});
const upload = multer({
storage
});
module.exports = upload;
controller
exports.createService = async (req, res) => {
const service_name = req.body.main_name;
const url = req.protocol + '://' + req.get('host');
let service_pic;
if (req.file) {
service_pic = url + '/utils/uploads/' + req.file.filename;
}
try {
const service = new Services({
_id: new mongoose.Types.ObjectId(),
service_name,
service_pic
});
const new_service = await service.save();
res.status(201).json({ message: "New data created", result: new_service });
} catch (error) {
console.log(error);
res.status(500).json({ message: "Internal server error", error });
}
}
With the route, http://localhost:5000/services/all, I get the JSON data as:
{
createdAt: "2020-09-07T08:25:11.581Z"
service_name: "TEST"
service_pic: "http://localhost:5000/utils/uploads/testio-logo-rgb1.png"
updatedAt: "2020-09-07T08:25:11.581Z"
}
When I try to access http://localhost:5000/utils/uploads/testio-logo-rgb1.png, it always returns an error: "error":{"message":"Route Not found"}}. The images are gettings stored properly into the /uploads folder, but still not accessible on the client.
I am not sure what thing is going wrong. Any help to resolve this is appreciated.
Change '/uploads' to 'utils/uploads'
// app.js
const directory = path.join(__dirname, 'utils/uploads');
app.use('/uploads', express.static(directory));
This block of code means: you've set up a static-assets serving endpoint at /uploads. Everytime a request hits this endpoint, your server will look up to the folder /utils/uploads.
An example request would be: http://localhost:5000/uploads/testio-logo-rgb1.png
I believe your URL is incorrect,
http://localhost:5000/utils/uploads/testio-logo-rgb1.png
should be
http://localhost:5000/uploads/testio-logo-rgb1.png
While uploading a file from ajax request Multer is giving an error that is given below.
TypeError [ERR_INVALID_ARG_TYPE]: The first argument must be one of
type string or Buffer. Received type object
at rite_ (_http_outgoing.js:595:11)
// code block for multer start
var Storage = multer.diskStorage({
destination: function(req, file, callback) {
callback(null, "./uploads/posts");
},
filename: function(req, file, callback) {
callback(null, file.fieldname + "_" + Date.now() + "_" + file.originalname);
}
});
var upload = multer({
storage: Storage
}).single('imgData');
//route Ajax Rquest URL Start
router.post('/blog/saveUploadImage',urlencoderParser,(req,res)=>{
upload(req, res, function(err) {
if (err) {
return res.end({UplaodStatus:true,type:'success',text:' 📷 Image Uploaded Now Saving Your Data It will take just a sec.'});
}
return res.end({UplaodStatus:false,type:'error',text:' ☹ Sorry There was some Problem Uploading Image '});
});
});
//route Ajax Rquest URL End
//JS code
// code for geting file
let fileUpload = document.getElementById('uploadFile').files;
//appending the file to formdata
var formData = new FormData();
formData.append('imgData', fileUpload);
//AJAX Request
$.ajax({
enctype:'multipart/form-data',
data:formData,
url:'/admin/blog/saveUploadImage',
type:'POST',
cache:false,
contentType:false,
processData:false,
timeout:10000,
});
You're passing an array of files to formData.append(...), instead you should pick just the first element from this array:
let fileUpload = document.getElementById('uploadFile').files[0];
The issue was I Imported this package ( Look Below ) because of this multer was not working.
const fileUpload = require('express-fileupload');
So I removed it now it works fine.
Thank you for ur help.
The issue was I Imported this package ( Look Below ) because of this multer was not working.
const fileUpload = require('express-fileupload');
So I removed it now it works fine.
I'm facing issues for uploading local images to my google cloud storage.
I've already tried two methods. The first one is uploading with multer
var storage = multer.diskStorage({
destination: (req, file, cb) => {
cb(null, './uploads/')
},
filename: (req, file, cb) => {
cb(null, file.fieldname + '-' + Date.now())
}
});
var upload = multer({storage: storage}).single('image');
app.post('/upload',function(req,res,next){
upload(req,res,(err) => {
if(err){
console.log(err)
}else{
console.log(req.file)
}
})
})
Then, i've tried directly with GCS
var bucket = admin.storage().bucket('mybucket')
app.post('/upload',function(req,res,next){
bucket
.save(file)
.then(() => {
})
for both of these solutions , req.files is always undefined whereas req.body is a buffer like this :
<Buffer 2d 2d 2d 2d ...>
when i try to save this buffer on my GCS bucket, i the .jpg/png file is created in my bucket but it is corrupted.
I'm browsing the web seeking for a solution but i found nothing that helped me to overcome this situation.
Any advice ?
You need multer, multer-google-storage and ofcourse bodyParser if you have additional form values. You need to sent data in multipart/form-data
In your .env file
GCS_BUCKET = <bucket name>
GCLOUD_PROJECT = <project id>
GCS_KEYFILE = <key file location>
You can download key file from GCP Console>Your Project>I AM & Admin>Service Accounts
In your route
const multer = require('multer');
const multerGoogleStorage = require("multer-google-storage");
var uploadHandler = multer({
storage: multerGoogleStorage.storageEngine()
});
router.post('/', uploadHandler.single('image'), function (req, res, next) {
const body = req.body;
res.json({fileName: req.file.filename});
res.end();
}
This will store file on to GCS with name [random-string-generated-by-gcs]_[YOUR FILE NAME WITH EXTENTION]. The same can be access under the route via req.file.filename.
Documentation
Make sure you have added enctype="multipart/form-data" attribute to your form. A probable reason for req.files being undefined.
I intended to use multer to upload multiple file and then rename them back to their original names. The below are the sample code:
var express = require('express');
var app = express();
var fs = require("fs");
var multer = require('multer');
app.use(express.static('public'));
var upload = multer({ dest: './upload/' });
app.get('/index.html', function (req, res) {
res.sendFile(__dirname + "/" + "index.html");
})
app.post('/file_upload', upload.array('theFile', 2), function (req, res, next) {
var errorcode = 0;
for (var i = 0; i < req.files.length; i++) {
fs.rename(req.files[i].path, req.files[i].destination + req.files[i].originalname, function (err) {
errorcode = err;
}(i));
}
if (errorcode != 0) {
console.log("errorcode is " + errorcode);
res.sendStatus(500);
return;
} else {
res.json({
message: 'File uploaded successfully',
});
}
})
var server = app.listen(8089, function () {
var host = server.address().address
var port = server.address().port
console.log("Example app listening at http://%s:%s", host, port)
})
I'm testing the above code on a windows server. And my observation is that the files can be uploaded successfully but the fs.rename() keeps returning error "1". And the renamed files in the targeted folder are always 1Kb. It seems that the rename function intends to fetch the files which might be still uploading. I'm not sure whether my understanding is correct. If so, is there a way to determine whether the files have been uploaded completely? Any suggestion for my problem?
Why not use Multer's built-in renaming functionality?
Adapted from the documentation:
var storage = multer.diskStorage({
destination: '/path/to/uploads/folder',
filename: function (req, file, cb) {
// Here we specify the file name to save it as
cb(null, file.originalname);
}
})
// And we can use it for example like this:
app.post('/upload', upload.single('image'), function (req, res, next) {
// req.file is the `image` file
// req.body will hold the text fields, if there were any
})
However, there are a couple things you should be aware of if you take this approach:
The client can send any type of file, with any (potentially incorrect) extension. This is a potential security risk.
If two files are uploaded with the same name, the second file will overwrite the first.
If you serve these files to other users, the security risk greatly increases. An attacker could create a script or HTML page and upload it, possibly giving it a different file name extension. There are several ways it could be run, such as if the user tries to open it in a new tab because an image didn't show up. The full implications of this, and how to deal with it, are a topic of their own.
Finally, make very, very, sure that the user cannot write to a directory other than the uploads folder. What happens when file.originalname is something like ../../index.js? It may be better to convert the file name to a slug.