I am doing a mock content moderation labeling using AWS Rekognition on React, but no matter what available image format (bytes, S3 object) I submitted, Rekognition never recognizes it, giving Unhandled Rejection (InvalidParameterException): Requested image should either contain bytes or s3 object.
I have followed Rekognition's guideline here, StackOverflow solutions: 1. Converting into Bytes as an Image param here, 2. Using S3 Object here, 3. Using Buffer to read image here, 4. Using raw file input to upload image (this one definitely did not work), 5. Checking if my image is corrupted or not here – in fact I tried with more than 20 images and all failed, 6. Stripping EXIF data here.
Library I use: node-rekognition (I use an old package because my huge node app is still runs on version 12.13.0), React 16 (class component approach)
Feel free to request any additional info if I lack some. Thanks.
Related
I am trying to create (what I thought was) a simple image classification pipeline between s3 and SageMaker.
Images are stored in an s3 bucket with their class labels in their file names currently, e.g.
My-s3-bucket-dir
cat-1.jpg
dog-1.jpg
cat-2.jpg
..
I've been trying to leverage several related example .py scripts, but most seem to be download data sets already in .rec format or containing special manifest or annotation files I don't have.
All I want is to pass the images from s3 to the SageMaker image classification algorithm that's located in the same region, IAM account, etc. I suppose this means I need a .lst file
When I try to manually create the .lst it doesn't seem to like it and it also takes too long doing manual work to be a good practice.
How can I automatically generate the .lst file (or otherwise send the images/classes for training)?
Things I read made it sound like im2rec.py was a solution, but I don't see how. The example I'm working with now is
Image-classification-fulltraining-highlevel.ipynb
but it seems to download the data as .rec,
download('http://data.mxnet.io/data/caltech-256/caltech-256-60-train.rec')
download('http://data.mxnet.io/data/caltech-256/caltech-256-60-val.rec')
which just skips working with the .jpeg files. I found another that converts them to .rec but again it has essentially the .lst already as .json and just converts it.
I have mostly been working in a Python Jupyter notebook within the AWS console (in my browser) but I have also tried using their GUI.
How can I simply and automatically generate the .lst or otherwise get the data/class info into SageMaker without manually creating a .lst file?
Update
It looks like im2py can't be run against s3. You'd have to completely download everything from all s3 buckets into the notebook's storage...
Please note that [...] im2rec.py is running locally,
therefore cannot take input from the S3 bucket. To generate the list
file, you need to download the data and then use the im2rec tool. - AWS SageMaker Team
There are 3 options to provide annotated data to the Image Classification algo: (1) packing labels in recordIO files, (2) storing labels in a JSON manifest file ("augmented manifest" option), (3) storing labels in a list file. All options are documented here: https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification.html.
Augmented Manifest and .lst files option are quick to do since they just require you to create an annotation file with a usually quick for loop for example. RecordIO requires you to use im2rec.py tool, which is a little more work.
Using .lst files is another option that is reasonably easy: you just need to create annotation them with a quick for loop, like this:
# assuming train_index, train_class, train_pics store the pic index, class and path
with open('train.lst', 'a') as file:
for index, cl, pic in zip(train_index, train_class, train_pics):
file.write(str(index) + '\t' + str(cl) + '\t' + pic + '\n')
I am using Croppie to crop an image and it is giving me base64 which creates 413 error while sending it on server for large images, so I need proper image file. I tried to find converting it(base64) as a image file locally at client side but didn't get expected way for conversion using javascript/jquery.
Please guide me.
I am using this package
to upload the image into server ,i want to check the file size before upload but express-fileupload doesn't give any information about it
console.log(req.files.image); it returns only the name,data ,and image type
Assuming the name of your file in your HTML file is image, req.files.image.data.length will give you the buffer length in bytes. See this Node.js. API Documentation for more details. You can then do the math to convert the number of bytes into any thing you want.
This will allow you to get the actual number of bytes independent of filetype and doesn't require using the mv function. So you can use this on any file type not just images.
Hopefully this helps!
I think only way to get image info using express-fileupload is using mv function to move image on server then using image-size to get width and height
I'm trying to pull an image from s3, quantize it/manipulate it, and then store it back into s3 without saving anything to disk (entirely in-memory). I was able to do it once, but upon returning to the code and trying it again it did not work. The code is as follows:
import boto3
import io
from PIL import Image
client = boto3.client('s3',aws_access_key_id='',
aws_secret_access_key='')
cur_image = client.get_object(Bucket='mybucket',Key='2016-03-19 19.15.40.jpg')['Body'].read()
loaded_image = Image.open(io.BytesIO(cur_image))
quantized_image = loaded_image.quantize(colors=50)
saved_quantized_image = io.BytesIO()
quantized_image.save(saved_quantized_image,'PNG')
client.put_object(ACL='public-read',Body=saved_quantized_image,Key='testimage.png',Bucket='mybucket')
The error I received is:
botocore.exceptions.ClientError: An error occurred (BadDigest) when calling the PutObject operation: The Content-MD5 you specified did not match what we received.
It works fine if I just pull an image, and then put it right back without manipulating it. I'm not quite sure what's going on here.
I had this same problem, and the solution was to seek to the beginning of the saved in-memory file:
out_img = BytesIO()
image.save(out_img, img_type)
out_img.seek(0) # Without this line it fails
self.bucket.put_object(Bucket=self.bucket_name,
Key=key,
Body=out_img)
The file may need to be saved and reloaded before you send it off to S3. The file pointer seek also needs to be at 0.
My problem was sending a file after reading out the first few bytes of it. Opening a file cleanly did the trick.
I found this question getting the same error trying to upload files -- two scripts clashed, one creating, the other uploading. My answer was to create using ".filename" then:
os.rename(filename.replace(".filename","filename"))
The upload script then needs to ignore . files. This ensured the file was done being created.
To anyone else facing similar errors, this usually happens when content of the file gets modified during file upload, possibly due to file being modified by another process/thread.
A classic example would be to scripts modifying the same file at the same time, which throws the bad digest due to change in MD5 content. In the below example, the data file is being uploaded to s3, while it is being uploaded, if another process overwrites it, you will end up with this exception
random_uuid=$(uuidgen)
cat data
aws s3api put-object --acl bucket-owner-full-control --bucket $s3_bucket --key $random_uuid --body data
I am trying to upload and image to cloudinary by passing binary data of an image or base64 string. when i try to pass a base64 string as a data uri I am getting a error with the response of status code 502. but it works fine with small base64 strings.
This works fine
res = cloudinary.uploader.upload("data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUAAAAFCAYAAACNbyblAAAAHElEQVQI12P4//8/w38GIAXDIBKE0DHxgljNBAAO9TXL0Y4OHwAAAABJRU5ErkJggg==")
whereas when I pass some lengthy string it fails with 502 status code.
Might be because the uri cannot handle large value. Is there any other right way for passing lengthy strings?
or how can i pass a binary data as a input to the cloudinary?
I'm Itay from Cloudinary.
In order to understand this issue better we'll need to take a deeper look at your account. We'll need some more information like your cloud name, timestamp of the fault uploads and more.
If you prefer this to be handled more privately, you're more than welcomed to open a support ticket (http://support.cloudinary.com/tickets/new) and we'll be happy to assist.