We are on a project with node js and flutter. We use an S3 bucket for storing images. We need to optimize images to a minimum, for easy loading. We use compression from the backend which results in random image sizes for different images. When those images are resized on photoshop with a preferred size of 500 * 500 px,(also used 200 * 200px, 300 * 300px) it results in the same multiple-sized images. What can we do to minimize image size and optimize the image loading performance?
There are many ways to approach this problem.
Try this way.
Create two folders in your S3 bucket: original & resized.
Create a lambda function and add this lambda function to the S3 trigger, so that whenever any image will be uploaded to S3, this trigger will be executed and the lambda function will run.
Upload images from your client-side to an original folder of S3 Bucket.
Once the image is uploaded lambda will run to do the processing on the image (Re-size/optimize ) and will save this processed(re-sized) image to the re-sized folder of the bucket.
Keep polling the resized folder from the client-side to get resized image. (Remember we upload the original image to the “original” folder & get the resized image from the “resized” folder)
Source : https://medium.com/#pushpendrachauhan/re-sizing-optimizing-images-on-the-go-using-amazon-lambda-amazon-s3-bucket-node-js-4fc933e6ddae
https://aws.amazon.com/blogs/compute/resize-images-on-the-fly-with-amazon-s3-aws-lambda-and-amazon-api-gateway/
Related
I am doing a mock content moderation labeling using AWS Rekognition on React, but no matter what available image format (bytes, S3 object) I submitted, Rekognition never recognizes it, giving Unhandled Rejection (InvalidParameterException): Requested image should either contain bytes or s3 object.
I have followed Rekognition's guideline here, StackOverflow solutions: 1. Converting into Bytes as an Image param here, 2. Using S3 Object here, 3. Using Buffer to read image here, 4. Using raw file input to upload image (this one definitely did not work), 5. Checking if my image is corrupted or not here – in fact I tried with more than 20 images and all failed, 6. Stripping EXIF data here.
Library I use: node-rekognition (I use an old package because my huge node app is still runs on version 12.13.0), React 16 (class component approach)
Feel free to request any additional info if I lack some. Thanks.
I am making a small Flask web application . I have uploaded images in media/upload_images folder .Now I want to get the uploaded image size when I will click a button in client side and after getting the image size, meanwhile it will be aslo deleted from media/upload_images folder. How can I implement this? ... Any suggestion? Help will be appriciated. Thanks
First, you will need to import os module.
import os
To get the size of an uploaded file you can use like below:
os.stat('media/upload_images/filename.extension').st_size # This will give you the size in bytes.
To remove a file, you can use like below:
os.remove('media/upload_images/filename.extension')
I will suggest you, when you are deleting a file, do the proper error handling to avoid exceptions.
Update:
To get the size of each file in a directory:
files_list = os.listdir(app.config['UPLOADED_PHOTOS_DEST'])
for file in files_list:
size = os.stat(app.config['UPLOADED_PHOTOS_DEST'] + file)
print('Filename: ', file, ', Size: ', size)
I am trying to create (what I thought was) a simple image classification pipeline between s3 and SageMaker.
Images are stored in an s3 bucket with their class labels in their file names currently, e.g.
My-s3-bucket-dir
cat-1.jpg
dog-1.jpg
cat-2.jpg
..
I've been trying to leverage several related example .py scripts, but most seem to be download data sets already in .rec format or containing special manifest or annotation files I don't have.
All I want is to pass the images from s3 to the SageMaker image classification algorithm that's located in the same region, IAM account, etc. I suppose this means I need a .lst file
When I try to manually create the .lst it doesn't seem to like it and it also takes too long doing manual work to be a good practice.
How can I automatically generate the .lst file (or otherwise send the images/classes for training)?
Things I read made it sound like im2rec.py was a solution, but I don't see how. The example I'm working with now is
Image-classification-fulltraining-highlevel.ipynb
but it seems to download the data as .rec,
download('http://data.mxnet.io/data/caltech-256/caltech-256-60-train.rec')
download('http://data.mxnet.io/data/caltech-256/caltech-256-60-val.rec')
which just skips working with the .jpeg files. I found another that converts them to .rec but again it has essentially the .lst already as .json and just converts it.
I have mostly been working in a Python Jupyter notebook within the AWS console (in my browser) but I have also tried using their GUI.
How can I simply and automatically generate the .lst or otherwise get the data/class info into SageMaker without manually creating a .lst file?
Update
It looks like im2py can't be run against s3. You'd have to completely download everything from all s3 buckets into the notebook's storage...
Please note that [...] im2rec.py is running locally,
therefore cannot take input from the S3 bucket. To generate the list
file, you need to download the data and then use the im2rec tool. - AWS SageMaker Team
There are 3 options to provide annotated data to the Image Classification algo: (1) packing labels in recordIO files, (2) storing labels in a JSON manifest file ("augmented manifest" option), (3) storing labels in a list file. All options are documented here: https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification.html.
Augmented Manifest and .lst files option are quick to do since they just require you to create an annotation file with a usually quick for loop for example. RecordIO requires you to use im2rec.py tool, which is a little more work.
Using .lst files is another option that is reasonably easy: you just need to create annotation them with a quick for loop, like this:
# assuming train_index, train_class, train_pics store the pic index, class and path
with open('train.lst', 'a') as file:
for index, cl, pic in zip(train_index, train_class, train_pics):
file.write(str(index) + '\t' + str(cl) + '\t' + pic + '\n')
I receive an image in base64 string and I wanted to save the image in 3 different sizes. My code for saving the image in my app is as following and it works, how can I set a sepcific size for the image ?
fs.writeFile(pathImage, new Buffer(base64String, "base64"), function (err) {}
You can't just save an image in different sizes by writing part of the file to disk. In order to resize your image you need to first know what image format you are working with and then use an appropriate library to resize the image, usually by reducing image quality or cropping the image.
For example if you are working with a JPEG, PNG, WebP, or TIFF images, you could use https://github.com/lovell/sharp
From its example page
const sharp = require('sharp');
sharp(inputBuffer)
.resize(320, 240)
.toFile('output.webp', (err, info) => ... );
So basically my front end is displaying images that are about 3-5mb. I need to find a way to make their size way smaller.
Basically I have the file object on node.js uploading to amazon aws. I just need to compress it before it uploads.
Also whats the best way to resize images on node.js?
The best way is use Canvas Node js module. This module is up to 3 times faster than ImageMagic.
Images manipulation comparation - https://github.com/ivanoff/images-manipulation-performance
author's results:
canvas.js : 4.001 img/sec;
gm-imagemagic.js : 1.206 img/sec;
gm.js : 1.536 img/sec;
lwip.js : 0.406 img/sec;