Need your Help!
I deployed my nodejs app (only backend) to heroku and it includes a part where the user must insert documents in format jpg, jpeg, png! All pictures are located in the static folder (/assets/docs)! before pushing the changed code, I tried to create users and their pictures are shown! but when I do a new push to heroku, old files are gone and not showing!
I tried to ignore that static folder in new pushes - /assets/docs/* useless, /assets - useless, /assets/docs - useless
But as an experiment, I put 1 .jpg file in that static folder and pushed it. it always stays even in new pushes (with ignores also)
Using multer and mongoose
file uploading with multer
const multer = require('multer')
const path = require('path')
const storage = multer.diskStorage({
destination: 'assets/docs/',
filename: function(req, file, cb) {
cb(null, file.fieldname + '-' + Date.now() + path.extname(file.originalname))
}
})
const up = multer({
storage: storage,
limits: {
fileSize: 5 * 1024 * 1024
},
fileFilter(req, file, cb) {
if (!file.originalname.match(/\.(png|jpg|jpeg|docx|doc|pdf)$/)) {
return cb (Error("Allowed file types are png jpg jpeg docx doc pdf"), false)
}
cb (null, true)
}
})
saving files - mongoose
router.post('/register-employee', uploadDocument.single('passport'), async (req, res) => {
try {
const emp = await new Employee({
...req.body,
passport: `docs/${req.file.filename}`
})
await emp.save()
res.json(emp)
} catch (e) {
res.status(500).json({errorMessage: e.message})
}
})
gitignore file
/node_modules
/.vscode
/.idea
/config
/assets/docs
setting up a public folder
const publicDir = path.join(__dirname, '../assets')
app.use(express.static(publicDir))
The Heroku filesystem is ephemeral - that means that any changes to the filesystem whilst the dyno is running only last until that dyno is shut down or restarted.
Each dyno boots with a clean copy of the filesystem from the most recent deploy. This is similar to how many container based systems, such as Docker, operate.
You can get over this issue by using an external file storage system like Amazon s3 to persist data across restarts on a dyno.
You can read more here: https://help.heroku.com/K1PPS2WM/why-are-my-file-uploads-missing-deleted
Related
So I'm new to backend and node, and have been working on an app that should also support an image file upload when creating a new item.
I had issues at first getting this error ENOENT: no such file or directory but after following the answers here (I'm using a windows machine and was following a tutorial on a Mac)
ENOENT: no such file or directory .?
I've switched to using the __dirname together with path and I have no such error anymore.
What I do face now is another issue:
When I ask for the file.path, it is no longer relative like ./uploads but instead it is the full path on my computer
C:\Users\myuser\Documents\Coding\travel-market\api\src\uploads\2022-12-05T12-39-35.924Z-Screenshot 2022-11-02 193712.png
So when I pull that new item and try to render the image it doesn't show. Also I get this error Not allowed to load local resource.
Is that ok and would work just fine once the api is actually hosted on a server? Or is there a different way of doing things that would allow me to also view the image while I'm developing locally?
This is my entire code for saving right now:
const storage = multer.diskStorage({
destination: function(req, file, cb) {
cb(null, path.join(__dirname, "..", "uploads"));
},
filename: function(req, file, cb) {
const uniqueName =
new Date().toISOString().replace(/:/g, "-") + "-" + file.originalname;
cb(null, uniqueName);
},
});
const fileFilter = function(req, file, cb) {
if (!file.originalname.match(/\.(jpg|jpeg|png)$/)) {
return cb(new Error("Please upload an image file"));
}
cb(undefined, true);
};
const upload = multer({
storage,
limits: {
fileSize: 1024 * 1024 * 5, // This number is in bytes, so this is 5mb
},
fileFilter,
});
// Post a new plan
router.post("/plans", auth, upload.single("plan_image"), async(req, res) => {
console.log("this is the file", req.file);
const details = JSON.parse(req.body.details);
console.log("this is the body", details);
const plan = new Plan({
...details,
image: req.file.path,
author: req.user._id,
});
try {
await plan.save();
res.status(201).send(plan);
} catch (e) {
console.log("failed to save", e);
res.status(400).send(e);
}
});
Hello I'm on a (ReactJs Node Mysql Express) stack, my website has an events tab which the admin could upload to it events.
Each event has a cover photo, inner photos, Title, and a brief text.
So I send this whole form in a single post request (is it good?)
var fileUpload = multer({ storage:storage,
limits:{fileSize:150*1024*1024}
})
const multipleUpload = fileUpload.fields([{ name: 'coverPhoto', maxCount: 1 }, { name: 'innerPhotos', maxCount: 30 }])
const storage = multer.diskStorage({
destination: (req, files, cb) => {
cb(null, '../client/build/imgs/events')
},
filename: (req, files, cb) => {
cb(null, files.originalname)
}
})
The server does recieve the request perfectly but after multer receives the photos it takes so long to move them to the fs and the reverse proxy would send a 524 error that it took so long knowing that the format is webp and the photos are so small in size.
Any idea why is it taking so long?
I tried to move to formidable but I couldn't figure how to retrieve multiple input fields with multiple files in it ( I only succeeded in sending one input not multiple inputs)
Thank you
I have created file storage in nodejs application to store images in uploads folder. which works fine but now I want to display those images I have read some other posts but I am still quite unsure.
Here is my code:
var Storage = multer.diskStorage({
destination: function(req, file, callback) {
var pathname = file.originalname;
var path = pathname[0].replace('_','/');
cb(null,'./uploads'+pathname);
},
filename: function(req, file, callback) {
var pathname = file.originalname;
var filename = pathname[1];
if(pathname!=undefined)
callback(null, pathname);
}
});
var upload = multer({ storage: Storage }).single('file');
router.post('/upload', multer({dest:'./uploads/'}).single('file'), function(req,res)
{
return res.status(201).json({result:req.file});
});
from console log I get
{ fieldname: 'file',
originalname: '1569402978357.jpg',
encoding: '7bit',
mimetype: 'image/jpeg',
destination: './uploads/',
filename: 'ada9282345565e8a77c6adc4b2d15836',
path: 'uploads\\ada9282345565e8a77c6adc4b2d15836',
size: 170272 }
and in the uploads it is stored as filename
my problem is how can I display this image back? should call it from stored filename or should I change the storage to save it as original filename with file extension?
The choice to keep the original file name differs from use case to use case. In either case you will have to store the name somewhere in a persistence medium (like a database) so that while displaying it back you can look into your uploads directory and send the file back.
Depending on your use case, if you want to store the files in a different name, you need a way to find the files at a later stage. It can be achieved by storing the names in a database or you could even use some kind of formula to find out the file name of the stored file from the original one.
E.g:
name_of_stored_file = md5(name_of_original_file)
As files (images) are now stored in your backend, you can expose some API to send the files to client on request. Again this completely depends on the use case, but here's a simple example of using Expressjs res.sendFile method:
app.get('/file/:fileName', function (req, res) {
const filePath = // find out the filePath based on given fileName
res.sendFile(filePath);
});
http://expressjs.com/en/api.html#res.sendFile
I declare my multer configuration as follows:
const storage = multer.memoryStorage()
const upload = multer({
limits: {
fileSize: 1000 * 1000 * 10 // 10 Megabytes
},
storage: storage
});
And then in my express route:
...//On api.post() {}
console.log(file);
...
This outputs the file, with a buffer, which is great! However when I call:
createBlockBlobFromStream(containerName, fileName, file, size)
I get the error stream.pause is not a function which is called by the azure-storage at the start of the upload because the sdk wants to wait to get a 200 from the azure cloud blob service before it continues, which makes sense.
However it doesn't think that my file that I pass through has the property pause().
Anyone come across this before? I read somewhere that the stream created by multer is not the correct formant and I need to wrap it in streamifier or some other package.
The issue is that a buffer do not correspond with a stream. Better take a look to multer-azure-storage is easier.
e.g.
var multer = require('multer')
var MulterAzureStorage = require('multer-azure-storage')
var upload = multer({
storage: new MulterAzureStorage({
azureStorageConnectionString: 'https://mystorageaccount.blob.core.windows.net/',
azureStorageAccessKey: 'myaccesskey',
azureStorageAccount: 'mystorageaccount',
containerName: 'photos',
containerSecurity: 'blob'
})
})
When I use an npm package like google-cloud or multer-gcs to upload a file to firebase storage bucket (that uses google cloud storage under the hood..) the file gets uploaded. However, the file does not show and image type(MIME). Also, how do I use the image in my node.js application?
Here is the code to upload image to firebase bucket. Used multer to handle file upload..
var path = require('path');
var multer = require('multer');
var gcs = require( 'multer-gcs' );
var storage = gcs({
filename : function( req, file, cb ) {
cb( null, file.fieldname + '-' + Date.now() + path.extname(file.originalname).toLowerCase());
},
bucket : 'p****n-bcXX3.appspot.com', // Required : bucket name to upload
projectId : 'p****n-bcXX3', // Required : Google project ID
keyFilename: './p****n-5cbc725XXXXd.json', // Required : JSON credentials file for Google Cloud Storage
acl : 'publicread' // Optional : Defaults to private
});
In my routes:
router.post('/food/create', [authChecker, gcsUpload.single('food_image')], function(req, res, next) {
Controllers.create_food(req, res);
});
When I upload file directly to the bucket, type shows clearly. What's the catch here?
It looks like multer-gcs doesn't support the ability to add custom metadata to a file. If you use gcloud directly, you'd do something like this:
var options = {
destination: 'new-image.png',
metadata: {
contentType: 'image/png'
}
};
bucket.upload('local-image.png', options, function(err, file) {
// Your bucket now contains:
// - "new-image.png" (with the contents of `local-image.png')
// `file` is an instance of a File object that refers to your new file.
});
I'd file an issue with the developer of multer and ask them to support metadata uploads along side blob uploads, especially since gcloud already supports the ability to do this.
Here is what I am doing now to handle the file upload.
req.file.modifiedname = (Math.random().toString(36)+'00000000000000000').slice(2, 10) + Date.now() + path.extname(req.file.originalname);
var options = {
metadata: {
contentType: req.file.mimetype
},
predefinedAcl : 'publicread'
};
var blob = bucket.file(req.file.modifiedname);
var blobStream = blob.createWriteStream(options);
blobStream.on('error', function (err) {
return next(err);
});
blobStream.on('finish', function() {
publicUrl = format(
'https://storage.googleapis.com/%s/%s',
bucket.name, blob.name
);
req.file.gcsfileurl = publicUrl;
console.log(req.file.gcsfileurl);
next();
});
blobStream.end(req.file.buffer);