I am totally new to Lambda (or AWS) and am still to build knowledge and experience around it.
Now, I was building an app where in it requires to fetch data from twitter Hashtag.
If I got it correctly, Twitter restricts number of API calls we make every minute(?) hence we need to have a backend and needs to have oAuth2 authentication.
In a simple express app, I would have done an API call in the global scope to get the data and use setInterval to hit that API after every x minute so as to not exceed number of limits.
Now based on the very vague understanding, I guess Lambda runs function when we need it, Hence is it right to assume that we can't use lambda for such use cases?
The old-school way of doing this is to run a cron job that fires a particular script every so often. The AWS way of running code periodically is using CloudWatch scheduled events. You can configure how often you want to run a given target, and set the target as a lambda function.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
Related
Is there a way with the AWS CLI to tell that you are running your lambda locally programmatically? I'm trying to avoid adding extra data in the request.
I have some functionality that I don't want kicked off when I'm running locally, but I do once its up in the AWS cloud.
Thanks
A first option is to use one of the environment variables that are available when a Lambda function is executed. The AWS_EXECUTION_ENV - like you stated in your comment - can be a good pick for this.
A second option is using the context object which is passed in as a second parameter to your handler function. This contains very specific information about the request, such as the awsRequestId which could also help you in determining whether or not your code is running on the cloud or locally.
I am new to serverless and NodeJS.Could you please guide me how can I create a automated test cases for
lambda to lambda invoke
API Gateway to Lambda Invoke
DynamoDB insertion test
Please help. Thanks in advance.
If you want full end-to-end test of a lambda function, you will have to handle that outside the function itself.
If you use unit testing tools you will be able to run them locally or even inside the function, but you won't have the ability to actually query the function and go through the whole process.
I'd create a second lambda function with any unit test library, like mocha, and write functional tests that invoke the first lambda function, through API gateway, with a simple http-request package (like request).
EDIT:
Here's more clarification on each one of your points:
1) Lambda to lambda invoke
If by lambda-to-lambda you mean you want to call another function WITHOUT using API GW, then I guess you're planning to use the AWS SDK to trigger a function.
If that is the case, it's like any other test. You will create a test function which will get the SDK to trigger the second lambda, and then check the result of the SDK function. It will probably indicate if it's a success or not, or even give you the result.
2) API gateway to lambda invoke
If you are looking to test if the connection between API GW and lambda works, I'd say, why bother? It's a setup-once-and-use kind of deal.. But if you still want to test this, it will be similar to item 1), with the exception that instead of using the SDK, you'd use an API gateway URL.
So you can use an npm package such as axios or request to make a request to such URL and see if the content is the expected.
I'd even say you can run the test in the lambda function and call the very same lambda function, no need to create separate lambdas.
3) Dynamo insertion
This one is the easiest, just create a unit test that writes something into dynamo. Then, in order to know if the test passes or not, just read the DB trying to find what you wrote.
If you're in the fence between testing libraries, I'd suggest going for mocha and chai.
If I can help you answering something more specific, let me know.
I want to build a nodejs application to scrape data from a website every 20mins and store it in firebase. Can you please tell me which product of google( compute engine, app engine or cloud functions ) is effective for this requirement as below are the things i am expecting to do,
1. Run Nodejs, cheerio to scrape data from website and store in firebase
2. Schedule it to run 20mins initially later may change it to 30mins or 1hr.
After reading the docs, i know that there are too many ways to implement this, but i am looking for a cost/resource effective way.
Pointers and ideas would be good.
Host the Node.js application within the App Engine[1] as Cloud Functions are event-driven[2]. You can use App Engine standard[3] or App Engine flexible[4] environment. For the scheduling part, Google Cloud Platform has a Cron Service[5] and you can create a cron job for your task hitting App Engine[6]. You can find a sample design here[7].
It depends on how much time your script spends waiting on requests. During that time the script is idle but you're getting charged at a super-high rate.
If you're doing a lot of concurrency then I would say do it with cloud functions.Another pro of doing it that way is your ip won't get blocked because it will be different every time.
Regarding scheduling, I'm not sure if Google lets do that, but I know AWS does.
A cost effective/simple way would be to use cronjob.org and have it send an http request to your cloud functions url to trigger it. If you're worried about other people triggering it, tell your cronjob to send an http header w/ an api key. Check this api key in your cloud function code to verify cronjob.org sent the request. I don't think it gets any more easy/cheap than this.
I am trying to create an AWS Lambda transform function for a Firehose stream that sends records directly to an Elasticsearch cluster.
Currently, there is no way to specify an ES document id in a Firehose stream record, so all records, even duplicates, are inserted. However, Firehose does support transformation functions hosted in Lambda, which gave me an idea:
My solution is to create a Lambda transform function that executes a DELETE request to Elasticsearch for every record during transformation, then returning all records unmodified, thereby achieving "delete-insert" behaviour (I am ok with the record disappearing for a short period).
However, I know very little about Nodejs and even though this is such a simple thing, I can't figure out how to do it.
Is there a Node package available to Lambda that I can use to do this? (Preferably an AWS Elasticsearch API, but a simple HTTP package would do).
Do I have to package up some other module to get this done?
Can something like Apex help me out here? My preferred language is Go, but so far I have been unable to get apex functions to execute or log anything to Cloudwatch...
Thanks in advance.
It seems a simple task to do, so I guess no framework is needed.
A few lines of code in Node.js would get the things done.
There are two packages that can help you:
elasticsearch-js
http-aws-es (If your ES domain is protected and you need to sign the requests)
The API doc: https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html
I am trying to deploy the Webhook Example for Google Actions found here onto Lambda AWS.
I was successful deploying and making the POST calls using ngrok. So, no problems there.
But the issue i found is it uses Express node module for POST request calls. Lambda AWS fails when the request is made to Express module. So is there a way to make the POST call successful.
I tried using Lambda-Express node module to deploy it, but it seems to have some issue as well.
Lambda AWS does not directly support an HTTP interface.
One solution would be for you to use API Gateway which would allow you to translate the HTTPS POST that AoG sends, into a call to AWS Lambda.
In your lambda you will handle the request which comes in via the standard Lambda handler:
function( event, context, callback );
instead of via Express. You would probably also want to remove Express from your code, which might sound like a lot of work, but I took a brief look at it when it was released and my impression was that the dependence on Express was minor and quite unnecessary.
The alternative would be to switch from Lambda to something HTTP based like Google App Engine which is also serverless (to a degree). I guess that might be easier but I don't know what your other factors might be.
I have come across this tutorial that explains the step by step process of connecting API.ai intent using the Lambda function.
And if you follow the Google web hook example to deploy it to Lambda, then it's a wrong direction. Completely eliminate the express usage and also the Assistant class is not necessary when Lambda deployment is necessary.