I have a thumbnailing function running on lambda, and I want to deploy it on elastic beanstalk. Lambda did a lot of background jobs for me so that when I deploy my function to elastic beanstalk, it won't work for me properly as I expected.
My lambda function can thumbnail all images in a given folder of given s3 bucket, and store them into different size images in the same location when it's triggered. However, when I deployed that to beanstalk, it will not be triggered by any of s3 events.
I know the rough step to fix it, but I need to know few specific things:
Before creating a lambda function, we need to configure event resourses:
I want to know if I can somehow pass them in beanstalk, I'm thinking about passing a json into my node.js function, but I don't know exactly how.
I don't know if I should add my function in an infinite loop to monitor events notifications from s3.
I want to combine this node.js function with other independent node.js service by express. And I want to display summary message about how many images has been thumbnailed in browser. But currently, with lambda package structure, I'm exporting a function handler to other js files. How can I export internal data to another static hjs/jade page?
How can I get the notification from s3?
In brief, if it isn't worthy of adding such complexity to deploy lambda function to beanstalk, should I just leave it as a lambda function?
Regarding Elastic Beanstalk vs. AWS Lambda, I think Lambda is going to be more scalable, as well as cheaper, for this sort of task. And I think saving status information to a DynamoDB table would be a quick and easy way to make statistics available that you can display in your web application, while preventing those statistics from disappearing if you redeploy or restart your application. Saving that data in DynamoDB would also allow you to have more than one EC2 instance serving your website in Elastic Beanstalk without having to worry about synchronizing that data across servers somehow.
Regarding sending S3 notifications to Elastic Beanstalk, you would need to do the following:
Configure S3 to send notifications to an SNS topic
Configure the SNS topic to send notifications to an HTTP endpoint
Configure an HTTP endpoint in your Beanstalk application to receive and process those notifications
Related
I have a node app that takes an url, scrape some text with puppeteer and translate it using deepl before sending me back the result in a txt file. It works as expected locally but having a lot of urls to visit and wanting to learn, I'm trying to make this app works with AWS Lambda and a docker image.
I was thinking about using a GET/POST request to send the url to API Gateway to trigger my lambda and wait for it to send me back the txt file. The issue is the whole process takes 2/3 minutes to complete and send back the file. It is not a problem locally but I know you should not have an http request wait for 3 minutes before returning.
I don't really know how to tackle this problem. Should I create a local server and make the lambda post a request to my ip adress once it is done?
I'm a loss here.
Thanks in advance!
One can see a few alternatives to what is seemingly asynchronous processing concern.
Poke the lambda with the data it needs (via an API, SDK, or CLI) then have it write its results to an S3 bucket. One could poll the s3 bucket for the results asynchronously and pull them down, obviously this requires some scripting.
An another approach would be to have the lambda post the results to a SNS topic that you've subscribed to.
That said, I'm not entirely sure what is meant by local IP, but I would avoid pushing data directly to a self-managed server (or your local IP), rather I would want to use one of the AWS "decoupling" services like SNS, SQS, or even S3 to split apart processing steps. This way it's possible to make many requests and pull down the data as needed.
I have containerized node.js code that runs on ECS. When multiple users use node.js to call a .py image generating problem, only 1 user gets the image, the rest get errors. I wonder if it appropriate to use Lambda so that the image generation multithreads.
For some reason, the containerized code which uses docker works locally, but not on AWS when multiple users access the .py function.
If users send images from mobile application, you can use aws-sdk to upload images from mobile device to AWS S3, and configure Lambda trigger for image upload.
This Lambda will process image data and return you the result.
Since Lambda from serverless world, it can handle really big amount of invocations.
So, in case if you/your team can add aws-sdk to mobile application, it's nice approach to upload image directly from device to S3, trigger Lambda to image processing and change user's data in some storage.
If you have untrusted environment, like user's browser, it's not O.K. to upload image directly from browser to S3, since to achieve this goal you have to provide AWS access keys.
So, in this case, it's O.K. to upload image to server and transfer image from server to S3.
After this, the logic keeps the same: trigger AWS Lambda, process data and update it in storage.
This behaviour will reduce load to the server and allows you to work on features, instead of working on image storing and other stuff, which will just bother your server.
I need my nodejs application to receive a http request with a file name, when a file is uploaded in my S3 bucket.
I would like some recommendations on the most simple/straight forward way to achieve this.
So far I see 3 ways to do this, but I feel Im overthinking this, and there surely exist better options:
1/ file uploaded on s3 -> S3 send a notification to SNS -> SNS sends a http request to my application
2/ file uploaded on s3 -> lambda function is triggered and sends a http request to my application
3/ make my application watch the bucket on regular basis and do something when a file is uploaded
thanks
ps. yes, Im really new to amazon services :)
SNS: Will work OK, but you'll have to manage the SNS topic subscription. You also won't have any control over the HTTP post's format.
Lambda: This is what I would go with. It gives you the most control.
How would you efficiently check for new objects exactly? This isn't a good solution.
You could also have S3 post the new object events to SQS, and configure your application to poll the SQS queue instead of listening for an HTTP request.
SNS- If you want to call multiple services on updating S3 then I would suggest SNS. You can create a topic for SNS and there can have multiple subscribers to that topic. Later if you want to add more HTTP then it would be as simple as subscribing the topic.
Lambda - If you need to sent notification to only one HTTP endpoint then I would strongly recommend this.
SQS - You don't need SQL in this scenario. SQS is mainly for decoupling components and would be the best fit for microservices but you can use with other messaging systems as well
You don't need to build something on your on to regularly monitor the bucket for changes as already services there like Lambda, SNS etc.
I'm new to GraphQL, Apollo, AWS S3, and Redux. I've read the tutorials for each and I'm familiar with React Native, Node, Heroku, and Mongo. I'm having trouble understanding the following:
how a "GraphQL Server" is hosted for a mobile device using React Native?
can I create the GraphQL server with Node and host it on AWS S3?
how to grab that data by using Apollo/GraphQL in my React Native code and store that data locally using Apollo/Redux?
do I have to use Graphcool as the endpoint from the start instead? All I'm trying to do is pulling data from my database when the app loads (not looking to stream it, so that I am able to use the data offline).
Where should I look to get a better understanding?
I have a couple comments for you in your exploration of new territory.
GraphQL is simply the query language the talks to your database. So you are free to run any type of api (on a server, serverless, etc.) that will use graphql to take in a graphql query/mutation and interact with your database.
GraphCool is a "production-ready backend" basically back-end as a service. So you wouldn't worry about running a server (as I believe they run most everything on serverless infrastructure) or managing where your DB is housed.
You can run an HTTP server on AWS EC2 or serverless using AWS Lambda. (Or the same flavor with Google or Azure). Whatever you decide to use to accept requests, your endpoint will accept graphql query strings and then do stuff with the db. AWS S3 is more of static storage. You can store files there to be retrieved, or scripts that can be pulled, but S3 probably isn't where you would want any server-like code to run.
Apollo would be a tool to use on your frontend for easily interacting with your graphql server. React-Apollo
Apollo/Redux may help you then manage the state throughout the app. You'll simply be loading the data into the app state on load then interacting with that state without needing to make any more external calls it sounds like.
Hopefully this was helpful.
I have a lambda function that is invoked by a get/post request to an API Gateway endpoint and then based on the request it reads/writes from/to a DynamoDB table.How can I unit test the function?I found this library called lambda-tester but it doesn't play well when dealing with Dynamo db.I also tried the Amazon's unit and load testing blueprint, but am not a big fan because it writes the results to yet another dynamo db table and I would rather just test locally and see the output on the terminal.
We are working on a set of tools for just this. Take a look at this blog:
http://docs.bespoken.tools/en/latest/tutorials/tutorial_bst_emulator_nodejs
Our primary intention is to help Alexa development, but you can "talk" to any lambdas. Is your lambda an Alexa skill (and dynamo is for persistence)?
I actually was able to use my non-local dev database after setting up AWS CLI, which required the necessary AWS credentials. Once you do that and set the region in the code, you can access your non-local dev database without a problem and test the code.