Cancel File Upload: Multer, MongoDB - node.js

I can't seem to find any up-to-date answers on how to cancel a file upload using Mongo, NodeJS & Angular. I've only come across some tuttorials on how to delete a file but that is NOT what I am looking for. I want to be able to cancel the file uploading process by clicking a button on my front-end.
I am storing my files directly to the MongoDB in chuncks using the Mongoose, Multer & GridFSBucket packages. I know that I can stop a file's uploading process on the front-end by unsubscribing from the subsribable responsible for the upload in the front-end, but the upload process keeps going in the back-end when I unsubscribe** (Yes, I have double and triple checked. All the chunks keep getting uploaded untill the file is fully uploaded.)
Here is my Angular code:
ngOnInit(): void {
// Upload the file.
this.sub = this.mediaService.addFile(this.formData).subscribe((event: HttpEvent<any>) => {
console.log(event);
switch (event.type) {
case HttpEventType.Sent:
console.log('Request has been made!');
break;
case HttpEventType.ResponseHeader:
console.log('Response header has been received!');
break;
case HttpEventType.UploadProgress:
// Update the upload progress!
this.progress = Math.round(event.loaded / event.total * 100);
console.log(`Uploading! ${this.progress}%`);
break;
case HttpEventType.Response:
console.log('File successfully uploaded!', event.body);
this.body = 'File successfully uploaded!';
}
},
err => {
this.progress = 0;
this.body = 'Could not upload the file!';
});
}
**CANCEL THE UPLOAD**
cancel() {
// Unsubscribe from the upload method.
this.sub.unsubscribe();
}
Here is my NodeJS (Express) code:
...
// Configure a strategy for uploading files.
const multerUpload = multer({
// Set the storage strategy.
storage: storage,
// Set the size limits for uploading a file to 120MB.
limits: 1024 * 1024 * 120,
// Set the file filter.
fileFilter: fileFilter
});
// Add new media to the database.
router.post('/add', [multerUpload.single('file')], async (req, res)=>{
return res.status(200).send();
});
What is the right way to cancel the upload without leaving any chuncks in the database?

So I have been trying to get to the bottom of this for 2 days now and I believe I have found a satisfying solution:
First, in order to cancel the file upload and delete any chunks that have already been uploaded to MongoDB, you need to adjust the fileFilter in your multer configuration in such a way to detect if the request has been aborted and the upload stream has ended. Then reject the upload by throwing an error using fileFilter's callback:
// Adjust what files can be stored.
const fileFilter = function(req, file, callback){
console.log('The file being filtered', file)
req.on('aborted', () => {
file.stream.on('end', () => {
console.log('Cancel the upload')
callback(new Error('Cancel.'), false);
});
file.stream.emit('end');
})
}
NOTE THAT: When canceling a file upload, you must wait for the changes to show up on your database. The chunks that have already been sent to the database will first have to be uploaded before the canceled file gets deleted from the database. This might take a while depending on your internet speed and the bytes that were sent before canceling the upload.
Finally, you might want to set up a route in your backend to delete any chunks from files that have not been fully uploaded to the database (due to some error that might have occured during the upload). In order to do that you'll need to fetch the all file IDs from your .chunks collection (by following the method specified on this link) and separate the IDs of the files whose chunks have been partially uploaded to the database from the IDs of the files that have been fully uploaded. Then you'll need to call GridFSBucket's delete() method on those IDs in order to get rid of the redundant chunks. This step is purely optional and for database maintenance reasons.

Try using try catch way.
There can be two ways it can be done.
By calling an api which takes the file that is currently been uploaded as it's parameter and then on backend do the steps of delete and clear the chunks that are present on the server
By handling in exception.
By sending a file size as a validation where if the backend api has received the file totally of it size then it is to be kept OR if the size of the received file is less that is due to cancellation of upload bin between then do the clearance steps where you just take the id and mongoose db of the files chuck and clear it.

Related

Uploading filedata from API Call to MongoDB

Im receiving filedata from an API call from an external service.
I want to save this filedata to my MongoDB. I was met with the error that the files are too large.
I went to research GridFS as an extra collection in my MongoDB.
I really cant find anything that solves my issuse. Ive tried to use multer to upload the data like this:
async function addFileDataToDB(fileData) {
const storage = new GridFsStorage({
url: mongoose.connection,
file: (req, file) => {
console.log(file.mimetype)
if (file.mimetype === 'application/pdf') {
return {
bucketName: 'fileBucket'
};
} else {
return null;
}
}
});
const upload = multer({ storage });
upload(fileData)
console.log('YAY! : - )')
}
Doesnt seem like something i can use. If i understand it correctly i cant use multer to transfer the data received by the endpoint to MongoDB. Multer seems more like something you would use to upload files from a form etc.
Im looking for any kind of help to point me in the right dirrection to upload this file data from the endpoint to a collection in mongoDB.
To clearify the file data is in the format of a buffer containing bytes, and im trying to do this in nodejs/express
Im new to GridFS, keep that in mind.

Returning image url's to the front-end from Postgres - Node / React

I have a function within my app that I have working that lets me submit a form with images that then stores the image in a file using Multer and uploads a URL to my Postgres database. When I return the file to the front end I am just left with filename that relates to the images but I don't know how to get the file path added so that the image displaying.
Should I be adding the prefix file path when inserting it into the database? Or is there a security issue if I display the full file path on the front-end. Obviously I know my front and backends should be decoupled and operating independently. I could also have a separate file for images outside of the backend but I am not sure if this is a recommended process. If this was in a professional environment would It just be handled by the likes of Google Cloud, AWS etc so I'm not sure if following the decoupling process is completely possible in this case.
I have seen process-cwd in some similar cases but I'm not sure if this is what I need to do. Could I hard code the file path in the front-end React component and then use the redux data that has the filename at the end?
I have the photo filename stored in my Redux store however I don't know to go from there. The image is in my backend/assets file at the minute.
API inserting the image into Postgres.
exports.createDiveSpot = async (req, res) => {
const fileNameWithExtension = `${req.file.filename}-${req.file.originalname}`
const newPath = `./assets/diveSpot/${fileNameWithExtension}`
fs.rename(req.file.path, newPath, function (err) {
if (err) {
console.log(err)
res.send(500)
}
console.log(req.body)
diveSpot.create({
diveLocation: req.body.diveLocation,
diveRegionID: req.body.diveRegionID,
diveSpotTypeID: req.body.diveSpotTypeID,
diveSpotDescription: req.body.diveSpotDescription,
photos: fileNameWithExtension,
}).then((data) => {
res.send(data)
})
Update
In the below method. I notice that I can enter a file name easily in the "const fileNameWithExtension" line. Can I enter a url for the public file on my front-end or should I make a file in my backend public with express. Should I be putting a file from my backend into my front-end again as it is technically already being submitted from the front-end then multer would move it to the front?
I have included my latest error message below.
exports.createArticle = async (req, res) => {
console.log(req.body)
const fileNameWithExtension = "../home/backend/assets/article/"`${req.file.filename}-${req.file.originalname}`
const newPath = `./assets/article/${fileNameWithExtension}`
console.log(req.body)
fs.rename(req.file.path, newPath, function (err) {
if (err) {
console.log(err)
res.send(500)
}
article.create({
articleTitle: req.body.articleTitle,
articleContent: req.body.articleContent,
userID: req.body.userID,
articleTypeID: req.body.articleTypeID,
photos: fileNameWithExtension,
}).then((data) => {
res.send(data)
})
.catch((err) => {
res.status(500).send({
message: err.message || 'Some error occurred while creating the post.',
})
})
}
)}
error message
[Error: ENOENT: no such file or directory, rename 'C:\Users\James Greene\WebstormProjects\softwaredevproject\SustainableScuba\backend\assets\article\678c78
193bb73ab287bbb6b644a0c86e' -> 'C:\Users\James Greene\WebstormProjects\softwaredevproject\WebApp\sustainable-scuba-web-app\public\article\678c78193bb73ab28
7bbb6b644a0c86e-sharkfeat.jpg'] {
errno: -4058,
code: 'ENOENT',
syscall: 'rename',
path: '...\\SustainableScuba\\backend\\assets\\article\\678c78193bb73ab287bbb6b644a0c86e',
dest: '...\\sustainable-scuba-web-app\\public\\article\\678c78193bb73ab287bbb6b644a0c86
e-sharkfeat.jpg'
}
express deprecated res.send(status): Use res.sendStatus(status) instead controllers\article.controller.js:15:21
Executing (default): INSERT INTO "articles" ("articleID","articleTitle","articleContent","photos","userID","articleTypeID") VALUES (DEFAULT,$1,$2,$3,$4,$5)
RETURNING *;
Unhandled rejection Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ServerResponse.setHeader (_http_outgoing.js:485:11)
at ServerResponse.header (...\backend\node_modules\express\lib\response.js:771:1
0)
It is being rendered on the front-end in Redux and is in the store. I am trying to display the image on the card component below.
<IconButton aria-label="settings">
<MoreVertIcon />
</IconButton>
}
title={articleList.articleTitle}
subheader={userID}/>
<CardMedia
id="Photos"
className={classes.media}
image={articleList.photos}
title="Dive Photos"/>
<CardContent>
<Typography className="Type" id="Type" variant="body2" color="textSecondary" component="p">
{articleType}
It is bad security practice to include full paths of your backend because it allows an attacker to traverse your installation and find exploits. It is also better to split up a complex application into different simpler responsibilities so that they are easier to solve.
This solution should 'separate the concerns' of the backend and frontend, as well as the two use cases to upload and download the file:
Use Case 1: Upload an image file to the server.
User selects image file from Client system
Client reads and uploads the file to the server
Server saves the file and associates it to content. (this is the sample code)
Use Case 2: Download and display an image file for a matching article on the UI.
Client requests an article
Server sends article content and ID for image files
Client requests image files and assigns the response to image components.
Server retrieves file by ID and returns it to the client.
Client image components render image files on the client system.
So, your backend needs to save the file in some way that it can publish it for later use. The best-practice as mentioned in comments is to put the file in a public folder and use the name as the ID, then let the webserver treat it as a static file. This allows the full benefit of web and cloud technologies to help you speed up and maintain your solution.
When the client uploads the image, the server should save it to a public folder and return its name/ID. The client path to the file is not enough in the upload, because the server can't access the client's system. Instead, the client needs to send a stream of bytes with metadata in the HTTP request. If you are only setting the filename in the upload, you need to read and send the bytes to the server like so:
JSON Example:
// read in the file bytes when opened:
onFileOpened(e) {
let data={};
let reader = new FileReader();
reader.onloadend = () => {
data.image= reader.result ;
};
reader.readAsDataURL(event.target.files[0]);
data.id=generateId();
}
// send the bytes on submit:
onSubmit(data){
POST(data);
}
Multipart Example:
With multer, the client is expected to send a Multipart/Form-Data request. This is falling back to older HTML tech and encoding the bytes similarly as above. However, it is still bytes in a wrapper.
<form action="/profile" method="post" enctype="multipart/form-data">
<input type="file" name="avatar" />
</form>
Then when the server receives the stream and invokes the createArticle method it should read the bytes and save them to a shared path, DB, or Content Delivery Network (CDN). The server should host or enable an endpoint to GET the same files from the recorded ID. Then the content can reference the file by appending the file's ID/name to the image endpoint.
Instead of renaming the file from the client path to the destination, the bytes need to be written to disk:
fs.createReadStream(req.file).pipe(fs.createWriteStream(newPath ));
This allows for the images to be hosted on multiple providers, allows many instances of your webserver to run in parallel, and allows you to size uploads and download services independently.

What to do to stop Express.js from duplicating long multipart requests?

Consider the following code:
var routes = function(app) {
app.post('/api/video', passport.authenticate('token', authentication), video.createVideo);
}
function createVideo(request, response) {
logger.info('starting create video');
upload(request, response, function(err) {
logger.info('upload finished', err);
//callback omitted for brevity
}
}
Upload is multer with multer-s3 middleware:
var upload = multer({
storage: s3({
dirname: config.apis.aws.dirname,
bucket: config.apis.aws.bucket,
secretAccessKey: config.apis.aws.secretAccessKey,
accessKeyId: config.apis.aws.accessKeyId,
region: config.apis.aws.region,
filename: function(req, file, cb) {
cb(null, req.user._id + '/' + uuid.v4() + path.extname(file.originalname));
}
}),
limits: {
fileSize: 1000000000
},
fileFilter: function(req, file, cb) {
if (!_.contains(facebookAllowedTypes, path.extname(file.originalname))) {
return cb(new Error('Only following types are allowed: ' + facebookAllowedTypes));
}
cb(null, true);
}
}).fields([{
name: 'video',
maxCount: 1
}]);
The code above does the following: it takes a file that is sent from somewhere and streams it to AWS S3 instance. multer-s3 uses s3fs in the background to create write stream and send the file as 5MB multiparts.
With big files, like 300MB it can take minutes to upload. And now something really strange happens. I can see in our frontend that it sends only one POST request on /api/video. Actually I also tried using Postman to make the request, not trusting our frontend.
It starts the upload, but after around 2 minutes it starts 2nd upload! If I try to upload smaller files, like 2-100MB then nothing of sorts happens. This is from my logs(from the code above):
{"name":"test-app","hostname":"zawarudo","pid":16953,"level":30,"msg":"starting create video","time":"2015-12-02T14:08:22.243Z","src":{"file":"/home/areinu/dev/projects/test-app-uploader/backend/app/services/videoService.js","line":169,"func":"createVideo"},"v":0}
{"name":"test-app","hostname":"zawarudo","pid":16953,"level":30,"msg":"starting create video","time":"2015-12-02T14:10:28.794Z","src":{"file":"/home/areinu/dev/projects/test-app-uploader/backend/app/services/videoService.js","line":169,"func":"createVideo"},"v":0}
{"name":"test-app","hostname":"zawarudo","pid":16953,"level":30,"msg":"upload finished undefined","time":"2015-12-02T14:12:46.433Z","src":{"file":"/home/areinu/dev/projects/test-app-uploader/backend/app/services/videoService.js","line":171},"v":0}
{"name":"test-app","hostname":"zawarudo","pid":16953,"level":30,"msg":"upload finished undefined","time":"2015-12-02T14:12:49.627Z","src":{"file":"/home/areinu/dev/projects/test-app-uploader/backend/app/services/videoService.js","line":171},"v":0}
As you can see both uploads end few ms after each other, but the second one starts after 2 minutes. The problem is - there should be only one upload!
All I did in postman was set my access token(so passport authorizes me) and added a file. This should create only 1 upload, meanwhile 2 happen, and both upload the same file.
Also, notice that both files get uploaded, both have different uuids(notice filename function creates the file names from uuid), both appear on s3, and both has proper size of 300MB, both can be downloaded and both work.
If the upload is smaller the duplication doesn't occur. What is the reason for this behavior? How to fix it?
The problem was very simple(I only spent whole day on figuring it out). It's just default timeout of node requests - 2 minutes. I don't know why it started another one nor why it actually worked, but setting default timeout on my server to 10 minutes fixed the issue.
If someone knows why the timed out requests actually did complete(and twice) please let me know. I'll improve the answer then.

Access uploaded image in Sails.js backend project

I am trying to do an upload and then accessing the image. The upload is going well, uploading the image to assets/images, but when I try to access the image from the browser like http://localhost:1337/images/image-name.jpg it gives me 404. I use Sails.js only for backend purposes - for API and the project is created with --no-front-end option. My front end is on AngularJS.
My upload function:
avatarUpload: function(req, res) {
req.file('avatar').upload({
// don't allow the total upload size to exceed ~10MB
maxBytes: 10000000,
dirname: '../../assets/images'
}, function whenDone(err, uploadedFiles) {
console.log(uploadedFiles);
if (err) {
return res.negotiate(err);
}
// If no files were uploaded, respond with an error.
if (uploadedFiles.length === 0) {
return res.badRequest('No file was uploaded');
}
// Save the "fd" and the url where the avatar for a user can be accessed
User
.update(req.userId, {
// Generate a unique URL where the avatar can be downloaded.
avatarUrl: require('util').format('%s/user/avatar/%s', sails.getBaseUrl(), req.userId),
// Grab the first file and use it's `fd` (file descriptor)
avatarFd: uploadedFiles[0].fd
})
.exec(function (err){
if (err) return res.negotiate(err);
return res.ok();
});
});
}
I see the image in the assets/images folder - something like this - 54cd1fc5-89e8-477d-84e4-dd5fd048abc0.jpg
http://localhost:1337/assets/images/54cd1fc5-89e8-477d-84e4-dd5fd048abc0.jpg - gives 404
http://localhost:1337/images/54cd1fc5-89e8-477d-84e4-dd5fd048abc0.jpg - gives 404
This happens because the resources your application accesses are not accessed directly from the assets directory but the .tmp directory in the project root.
The assets are copied to the .tmp directory when sails is lifted, so anything added after the lift isn't present in .tmp.
What I usually do is upload to .tmp and copy the file to assets on completion. This way assets isn't polluted in case the upload fails for any reason.
Let us know if this works. Good luck!
Update
Found a relevant link for this.

Publishing and subscribing to node-redis for an image resizing job after form POST

I've got a form submission that accepts an image. I'm creating thumbnails (resizing and pushing to S3) with the image upon submission, and it's taking awhile and blocking so I've decided I want to push it to a message queue and have that handle it.
I've decided to go with node-redis, since I'm already using redis in my stack. What I'm unclear on is how exactly the implementation would look (in its most basic form).
Consider some pseudocode below as:
var redis = require('redis'),
client = redis.createClient();
function listenForJob() {
client.on('message', function(msg) {
// msg is our temporary path name
// Kick of resize and push to s3 job
});
}
// Attached to my route where I POST to (e.g. /submit)
exports.form = function(req, res, next) {
// Input comes in, and image comes in as req.files
// and a temporary image is saved in /uploads
// which I'll save as the image submission for now
// until the process to resize and push to s3 happens.
listenForJob();
// Save to db
var data = {
title: req.body.title,
img: req.files.path // save temp file
}
connection.query('INSERT INTO submissions SET ?', data, function(err, rows) {
// Publish to our redis client?
client.publish('message', req.files.path);
connection.release();
res.redirect('/submissions');
});
};
Is this implementation even remotely the correct way to approach this? I'm new to taskworkers/message queues so I'm just wondering how to do implement it properly (given my use case).

Resources