I am working in node js. I want to execute the trigger when user upload the files to s3. So I created the script in node js which will upload the file to s3 bucket. But s3 event is not fired, however whenever I upload the file to s3 manually then trigger fires.
Please help
Since in your questions some things are unclear i.e. which method you are using in node js to upload file and what is your configuration in AWS Lambda to trigger the event.
I would recommend If you are using s3.upload() then try to use s3.putObject({}) to upload file in S3.
Check the trigger configuration is rightly created in AWS Lambda, Make sure the Event type as PUT is selected.
Check for IAM policy for the lambda function. It should have the below permission:
S3:PutBucketNotification
Related
I have 2 buckets on the S3 service. I have a lambda function "create-thumbnail" that triggered when an object is created into an original bucket, if it is an image, then resize it and upload it into the resized bucket.
Everything is working fine, but the function doesn't trigger when I upload files more than 4MB on the original bucket.
Function configurations are as follow,
Timeout Limit: 2mins
Memory 10240
Trigger Event type: ObjectCreated (that covers create, put, post, copy and multipart upload complete)
Instead of using the lambda function, I have used some packages on the server and resize the file accordingly and then upload those files on the S3 bucket.
I know this is not a solution to this question, but that's the only solution I found
Thanks to everyone who took their time to investigate this.
I saw different questions/answers but I could not find the one that worked for me. Hence, I am really new to AWS, I need your help. I am trying to download gzip file and load it to the json file then upload it to the S3 bucket using Lambda function. I wrote the code to download the file and convert it to json but having problem while uploading it to the s3 bucket. Assume that file is ready as x.json. What should I do then?
I know it is really basic question but still help needed :)
This code will upload to Amazon S3:
import boto3
s3_client = boto3.client('s3', region_name='us-west-2') # Change as appropriate
s3._client.upload_file('/tmp/foo.json', 'my-bucket', 'folder/foo.json')
Some tips:
In Lambda functions you can only write to /tmp/
There is a limit of 512MB
At the end of your function, delete the files (zip, json, etc) because the container can be reused and you don't want to run out of disk space
If your lambda has proper permission to write a file into S3, then simply use boto3 package which is an AWS SDK for python.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html
Be aware that if the lambda locates inside of VPC then lambda cannot access to the public internet, and also boto3 API endpoint. Thus, you may require a NAT gateway to proxy lambda to the public.
The context here is simple, there's a lambda (lambda1) that creates a file asynchronously and then uploads it to S3.
Then, another lambda (lambda2) receives the soon-to-exist file name and needs to keep checking S3 until the file exists.
I don't think S3 triggers will work because lambda2 is invoked by a client request
1) Do I get charged for this kind of request between lambda and S3? I will be polling it until the object exists
2) What other way could I achieve this that doesn't incur charges?
3) What method do I use to check if a file exists in S3? (just try to get it and check status code?)
This looks like you should be using an S3 objectCreated trigger on the Lambda. That way, whenever an object gets created, it will trigger your Lambda function automatically with the file metadata.
See here for information on configuring an S3 event trigger
Let me make sure I understand correctly.
Client calls Lambda1. Lambda1 creates a file async and uploads to S3
the call to lambda one returns as soon as lambda1 has started it's async processing.
Client calls lambda2, to pull the file from s3 that lambda1 is going to push there.
Why not just wait for Lambda one to create the file and return it to client? Otherwise this is going to be an expensive file exchange.
I would like to upload invoking a REST endpoint in multi-part.
In particular, I am looking at this API: Google Cloud Storage: Objects: insert
I did read about using multer, however I did not find any complete example showing me how to perform this operation.
Could someone help me with that?
https://cloud.google.com/nodejs/getting-started/using-cloud-storage#uploading_to_cloud_storage
^^ this is a a good example of how to use multer to upload a single image to Google Cloud Storage. Use multer to create filestream for each file ( storage: multer.memoryStorage() ), and handle the file stream by sending it to your GCS bucket in your callback.
However link only shows an example for one image. If you want to do an array of images, create a for-loop, where you create a stream for each file in your request, but only put the next() function after the for loop ends. If you keep the next(); in each loop cycle you will get the error: Error: Can't set headers after they are sent.
There is an example for uploading files with the nodejs client library and multer. You can modify this example and set the multipart option:
Download the sample code and cd into the folder:
git clone https://github.com/GoogleCloudPlatform/nodejs-docs-samples/
cd nodejs-docs-samples/appengine/storage
Edit the app.yaml file and include your bucket name:
GCLOUD_STORAGE_BUCKET: YOUR_BUCKET_NAME
Then in the source code, you can modify the publicUrl variable according to Objects: insert example:
const publicUrl = format(`https://www.googleapis.com/upload/storage/v1/b/${bucket.name}/o?uploadType=multipart`);
Download a key file for your service account and set the environment variable:
Go to the Create service account key page in the GCP Console.
From the Service account drop-down list, select New service account.
Input a name into the Service account name field.
From the Role drop-down list, select Project > Owner.
Click Create. A JSON file that contains your key downloads to your computer. And finally export the environment variable:
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/your/key/file
After that, yo're ready to run npm start and go to the app's frontend and upload your file:
I am attempting to set a NodeJS Lambda function to be triggered when an image is uploaded to an Amazon S3 bucket. I have seen multiple tutorials and have the yml file set up as shown. Below is the YML config file:
functions:
image-read:
handler: handler.imageRead
events:
- s3:
bucket: <bucket-name-here>
event: s3:ObjectCreated:*
Is there something I am missing for the configuration? Is there something I need to do in an IAM role to set this up properly?
The YAML that you have here looks good but there may be some other problems.
Just to get you started:
are you deploying the function using the right credentials? (I've seen it many times that people are deploying in some other account etc. than they think - verify in the web console that it's there)
can you invoke the function in some other way? (from the serverless command line, using http trigger etc.)
do you see anything in the logs of that function? (add console.log statements to see if anything is being run)
do you see the trigger installed in the web console?
can you add trigger manually on the web console?
Try to add a simple function that would only print some logs when it is run and try to add a trigger for that function manually. If it works then try to do the same with the serverless command line but start with a simple function with just one log statement and if it works then go from there.
See also this post for more hints - S3 trigger is not registered after deployment:
https://forum.serverless.com/t/s3-trigger-is-not-registered-after-deployment/1858