Using Cloudinary, just like an image, I would like to limit the width and height of the pdfs that are uploaded.
This is how I upload the file:
const res = await new Promise((resolve, reject) =>
{
let cld_upload_stream = cloud.upload_stream(
{
folder: process.env.CLOUD_FOLDER,
},
function (err, res)
{
if (res)
{
resolve(res);
} else
{
reject(err);
}
}
);
streamifier.createReadStream(file.data).pipe(cld_upload_stream);
});
return {
url: res.url,
location: res.public_id
}
Are there any options to limit the width and height, that can work on pdf files?
I tried:
{ responsive_breakpoints:
{ create_derived: true,
bytes_step: 20000,
min_width: 200,
max_width: 1000 }}
but it deos not seem to work.
The responsive breakpoints feature you mentioned is related to analysing an image and deciding which sizes you should resize it to for a responsive design, balancing possible problems if you choose the sizes manually (which are that you may create 'too many' images with very similar sizes, or have large gaps between the byte sizes of the different sizes, so more bandwidth is used than necessary for often-requested files.)
There's a web interface here that uses that feature and provides examples of what it does:https://www.responsivebreakpoints.com/
This is not related to validating uploaded files or editing the original assets that you upload to Cloudinary. There's no server-side validation available related to the dimensions of an uploaded file, but you could either:
Use an Incoming Transformation to resize the asset before it's saved into your account: https://cloudinary.com/documentation/transformations_on_upload#incoming_transformations
Use the upload API response after the file is uploaded and validated, and if it's "too big", show an error to your user and delete the file again.
You could also use a webhooks notification to receive the uploaded file metadata: https://cloudinary.com/documentation/notifications
Related
I can't seem to find any up-to-date answers on how to cancel a file upload using Mongo, NodeJS & Angular. I've only come across some tuttorials on how to delete a file but that is NOT what I am looking for. I want to be able to cancel the file uploading process by clicking a button on my front-end.
I am storing my files directly to the MongoDB in chuncks using the Mongoose, Multer & GridFSBucket packages. I know that I can stop a file's uploading process on the front-end by unsubscribing from the subsribable responsible for the upload in the front-end, but the upload process keeps going in the back-end when I unsubscribe** (Yes, I have double and triple checked. All the chunks keep getting uploaded untill the file is fully uploaded.)
Here is my Angular code:
ngOnInit(): void {
// Upload the file.
this.sub = this.mediaService.addFile(this.formData).subscribe((event: HttpEvent<any>) => {
console.log(event);
switch (event.type) {
case HttpEventType.Sent:
console.log('Request has been made!');
break;
case HttpEventType.ResponseHeader:
console.log('Response header has been received!');
break;
case HttpEventType.UploadProgress:
// Update the upload progress!
this.progress = Math.round(event.loaded / event.total * 100);
console.log(`Uploading! ${this.progress}%`);
break;
case HttpEventType.Response:
console.log('File successfully uploaded!', event.body);
this.body = 'File successfully uploaded!';
}
},
err => {
this.progress = 0;
this.body = 'Could not upload the file!';
});
}
**CANCEL THE UPLOAD**
cancel() {
// Unsubscribe from the upload method.
this.sub.unsubscribe();
}
Here is my NodeJS (Express) code:
...
// Configure a strategy for uploading files.
const multerUpload = multer({
// Set the storage strategy.
storage: storage,
// Set the size limits for uploading a file to 120MB.
limits: 1024 * 1024 * 120,
// Set the file filter.
fileFilter: fileFilter
});
// Add new media to the database.
router.post('/add', [multerUpload.single('file')], async (req, res)=>{
return res.status(200).send();
});
What is the right way to cancel the upload without leaving any chuncks in the database?
So I have been trying to get to the bottom of this for 2 days now and I believe I have found a satisfying solution:
First, in order to cancel the file upload and delete any chunks that have already been uploaded to MongoDB, you need to adjust the fileFilter in your multer configuration in such a way to detect if the request has been aborted and the upload stream has ended. Then reject the upload by throwing an error using fileFilter's callback:
// Adjust what files can be stored.
const fileFilter = function(req, file, callback){
console.log('The file being filtered', file)
req.on('aborted', () => {
file.stream.on('end', () => {
console.log('Cancel the upload')
callback(new Error('Cancel.'), false);
});
file.stream.emit('end');
})
}
NOTE THAT: When canceling a file upload, you must wait for the changes to show up on your database. The chunks that have already been sent to the database will first have to be uploaded before the canceled file gets deleted from the database. This might take a while depending on your internet speed and the bytes that were sent before canceling the upload.
Finally, you might want to set up a route in your backend to delete any chunks from files that have not been fully uploaded to the database (due to some error that might have occured during the upload). In order to do that you'll need to fetch the all file IDs from your .chunks collection (by following the method specified on this link) and separate the IDs of the files whose chunks have been partially uploaded to the database from the IDs of the files that have been fully uploaded. Then you'll need to call GridFSBucket's delete() method on those IDs in order to get rid of the redundant chunks. This step is purely optional and for database maintenance reasons.
Try using try catch way.
There can be two ways it can be done.
By calling an api which takes the file that is currently been uploaded as it's parameter and then on backend do the steps of delete and clear the chunks that are present on the server
By handling in exception.
By sending a file size as a validation where if the backend api has received the file totally of it size then it is to be kept OR if the size of the received file is less that is due to cancellation of upload bin between then do the clearance steps where you just take the id and mongoose db of the files chuck and clear it.
Looking though the docs, there are two ways of uploading files (images, videos, ...) to cloudinary using the node.js sdk.
Is there some way of getting progress reports when using either of the below specified methods? E.g 1 of 100mb have been uploaded.
cloudinary.v2.uploader.upload_large(filePath, options, (err, results) => {});
cloudinary.v2.uploader.upload(filePath, options, (err, results) => {});
For assets larger than this limit (100MB) you must request that the derived versions are created before they're requested, which we call 'eagerly', and that the processing takes place in the background ('asynchronously'). When using asynchronous eager transformations you can manipulate the asset as large as your account's maximum video/image file size limit.
Eager transformations can be requested for new asset in the upload API call or configured in an upload preset, including an upload preset that is used when you upload to Media Library.
For existing videos, you can request eager transformations via the explicit API method.
Once the video is transformed eagerly/asynchronously it will be available via the URL as normal.
For example in node:
cloudinary.v2.uploader.upload("sample.jpg",
{ eager: [
{ width: 300, height: 300, crop: "pad" },
{ width: 160, height: 100, crop: "crop", gravity: "south"} ],
eager_async: true,
eager_notification_url: "https://mysite.example.com/eager_endpoint",
notification_url: "https://mysite.example.com/upload_endpoint" },
function(error, result) {console.log(result, error); });
What we do is take a request for an image like "media/catalog/product/3/0/30123/768x/lorem.jpg", then we use the original image located at "media/catalog/product/3/0/30123.jpg", resize it to 768px and webp if the browser supports that and then return the new image (if not already cached).
If you request: wysiwyg/lorem.jpg it will try to create a webp in maximum 1920 pixels (no enlargement).
This seems to work perfectly fine up to <= 1420 pixels wide image. However above that we only get HTTP 502: The Lambda function returned invalid json: The json output is not parsable.
There is a similar issue on SO that relates to GZIP, however as I understand you shouldn't really GZIP images: https://webmasters.stackexchange.com/questions/8382/gzipped-images-is-it-worth/57590#57590
But it's possible that the original image was uploaded to S3 GZIPPED already. But the gzip might be miss-leading because why would it work for smaller images then? We have GZIP disabled in Cloudfront.
I have given the Lamda#Edge Resize function maximum resources 3GB memory and timeout of 30 seconds.. Is this not sufficient for larger images?
I have deleted the already generated images, invalidated Cloudfront but it still behaves the same..
EDIT: UPDATE:
I simply tried a different image and then it works fine.. I have no idea why and how I should solve the broken image then... I guess Cloudfront has cached the 502 now.. I have invalidated using just "*" but didn't help.. Both original files are jpg.
The original source image for the working one is 6.1 MB and non working is 6.7 MB if that matters.
They have these limits:
https://docs.aws.amazon.com/lambda/latest/dg/limits.html
The response.body is about 512 MB when it stops working.
There are some low limits in Lambda, especially in Lambda#Edge on the response size. The limit is 1 MB for the entire response, headers and body included. If lambda function returns a bigger response it will be truncated which can cause HTTP 500 statuses. See documentation.
You can overcome that by saving result image on S3 (or maybe checking first if it's already there), and then instead of returning it just making a 301 redirect to CloudFront distribution integrated with that bucket - so image request will be redirected to result image.
For example in node.js with Origin-Response trigger:
'use strict';
exports.handler = (event, context, callback) => {
// get response
const response = event.Records[0].cf.response;
const headers = response.headers;
// create image and save on S3, generate target_url
// ...
// modify response and headers
response.status = 301;
response.statusDescription = 'Moved Permanently';
headers['Location'] = [{key: 'Location', value: target_url}];
headers['x-reason'] = [{key: 'X-Reason', value: 'Generated.'}];
// return modified response
callback(null, response);
};
Version for simple Lambda Gateway (without Origin-Response, replaces headers):
exports.handler = (event, context, callback) => {
// create image and save on S3, generate target_url
// ...
var response = {
status: 301,
headers: {
Location: [{
key: 'Location',
value: [],
}],
'X-Reason': [{
key: 'X-Reason',
value: '',
}],
},
};
callback(null, response);
}
Additional notes to #Zbyszek's answer, you can roughly estimate if the request is bigger than 1MB like this:
const isRequestBiggerThan1MB = (body, responseWithoutBody) => {
const responseSizeWithoutBody = JSON.stringify(responseWithoutBody).length;
return body.length + responseSizeWithoutBody >= 1000 * 1000;
};
the responseWithoutBody can't be too large or contain "recursive keys" (or what it's called) but in this case I can't imagine that you would have that. If it contains recursive keys then you can simply remove those. If the responseWithoutBody is too large you need to remove those values and measure them separatly - for example like I am doing with the response.body.
I have a small webapp built with nodejs and express (among other things) that has a route to resize images on the fly (using sharp). The route looks like this:
router.get('/image/:options/:basedir/:dir/:img', utilitiesController.getOptimizedImage);
In the utilities controller, I have the getOptimizedImage function checking for the existing image, returning the existing image content if it exists, or if it doesn't, performing some image processing tasks, then returning the resulting image.
exports.getOptimizedImage = async (req, res) => {
// Parse options from request...
// first, check to see if resized version exists
fs.readFile(processedImgPath, function (err, processedImg) {
if (err) {
//console.log('File does not yet exist.');
// If resized version doesn't exist, check for original
fs.readFile(originImgPath, function (err, originImg) {
if (err) {
// If origin image doesn't exist, return 400.
} else if (w || h) {
// If origin image does exist, process it...
// Once it's processed, return the processed image
res.end(newImg);
return;
}
}
} else {
//
res.end(processedImg);
//res.redirect(existingFileUrl);
return;
}
}
}
This code works. I can request something like so:
<img src="http://example.com/image/w800/foo/bar/imagename.jpg">
...and it returns the resized image as expected. The issue seems to be that because of the way the image is returned using res.end(), the browser cache (testing in Chrome) doesn't ever store the image, so reloading the page downloads the image fresh instead of loading it from memory or disk.
I can alternatively use res.redirect() to send back the url of the existing processed image file, which will be cached on refresh, but that feels like the wrong way to do this, since it ultimately doubles all the image requests using the image processing path.
I don't want to process the images prior to request; I'm specifically looking to only process the images the first time they're requested, then store a processed version to reference each consecutive time. I'm open to alternatives within these constraints, but hoping someone can explain how I might leverage browser caching with my current structure?
You should add http headers for caching before any res.end, the example below will set the Expires time to 1 day (86400000ms).
res.set({
"Cache-Control": "public, max-age=86400",
"Expires": new Date(Date.now() + 86400000).toUTCString()
})
I am creating an image uploading system (size usually >20MB<50MB) and i want to crop that images to various sizes (its for viewing mobile,web and desktop application),all images are stored into AWS s3.
Here is the snapshot of crop-sizes
[{
width:200,
height:200,
type:"small",
platform:"web"},
{
width:300,
height:400,
type:"small",
platform:"mobile-android"
}
....
....
]
Here is the think i am planned to do
1.First upload the image into S3.
2.Run all the crop operations in async task
upload:function(req,res){
//various cropsizes
var cropSizes = [];
//upload image to s3
uploadImageToS3(req.file,function(err,result){
if(!err){
//create crop
cropImage({
'cropsizes':cropSizes,
'file':req.file
},function(err,result){
console.log('all crop completed',result);
});
res.send('run crop in backgroud');
}
});
}
But is this correct method?? can anyone have better thing other than this???
Since you are already using s3 I would recommend trying aws lambda to resize your images and adding them back to s3 bucket with new sizes.
Here is detailed explanation in this link https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-lambda-and-amazon-api-gateway/