I am trying to create an AWS Lambda transform function for a Firehose stream that sends records directly to an Elasticsearch cluster.
Currently, there is no way to specify an ES document id in a Firehose stream record, so all records, even duplicates, are inserted. However, Firehose does support transformation functions hosted in Lambda, which gave me an idea:
My solution is to create a Lambda transform function that executes a DELETE request to Elasticsearch for every record during transformation, then returning all records unmodified, thereby achieving "delete-insert" behaviour (I am ok with the record disappearing for a short period).
However, I know very little about Nodejs and even though this is such a simple thing, I can't figure out how to do it.
Is there a Node package available to Lambda that I can use to do this? (Preferably an AWS Elasticsearch API, but a simple HTTP package would do).
Do I have to package up some other module to get this done?
Can something like Apex help me out here? My preferred language is Go, but so far I have been unable to get apex functions to execute or log anything to Cloudwatch...
Thanks in advance.
It seems a simple task to do, so I guess no framework is needed.
A few lines of code in Node.js would get the things done.
There are two packages that can help you:
elasticsearch-js
http-aws-es (If your ES domain is protected and you need to sign the requests)
The API doc: https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/api-reference.html
Related
I am attempting to write an Azure CosmosDB integration (Core SQL Api) that integrates with an external service to provide some of the query data. As an example, I need a query made on Cosmos DB to convert some of the data returned (e.g. ID's) by the query into real data by calling an external service via a REST API. This should only happen when querying certain columns.
I initially investigated using a JS stored procedure and/or a UDF to make this external call, but the JS environment seems to be extremely limited and doesn't provide any way to make external calls. I then tried using this https://github.com/Oblarg/cosmosdb-storedprocs-ts repository, which uses webpack to bundle all of node.js into the stored procedure, allowing node modules to be used in stored procedures. Whilst this does allow some node modules to be used, whenever I try and use "https", "fetch", or "axios" modules to make an HTTP GET request I get errors (the same code works fine in a normal node environment, but I'm not a JS expert and can't seem to work past these errors). After a day of attempts it seems like the stored procedure approach is not possible.
Is this the case or is there some way of making HTTP GET requests from a JS stored procedure? If not possible with stored procedures, are there any other techniques to achieve the requirement of reading data from a remote API when querying cosmos DB?
Thanks
There is no way to achieve this from CosmosDB directly, for queries you also cannot use the change feed as the document dont change, so really your only option is to use a function or some preprocessor app to handle it, as you say its not ideal but there is no other solution here. If it was an insert or an update then change feed would allow you to do this but for plain queries its not possible.
I have 5 HTTP microservices written with NestJS. I have to convert them into lambda function where each service will have its own lambda function. The purpose of this is to completely turn my service to serverless.
I am using API Gateway to map requests to the right lambda by the given request path.
Now creating an MVC pattern from scratch that receives an URL path and resolves and controller & function needed (including url params, and such) is something that has already been done by both express and nestjs.
Is there a way to implement nesstjs's abstraction functionality without the actual server listening? So I can simply pass nestjs the URI and request data and it will work upon it?
Any other solutions for running an MVC serverless process on lambda?
After researching the network and looking at these great answers, I thought of collecting that information all together and bring something more detailed, things that I was misunderstanding from these articles described and the ones I found.
How to create a serverless NestJS application using Lambda function AWS provider
Let's take a look at this repository article: https://github.com/rdlabo/serverless-nestjs
Thanks to him, he basically packed a ready-to-go NestJS project configured with the Serverless framework.
I am not experienced enough to say a lot about the serverless framework, but I can explain how it works in our case.
First of all, there is a serverless.yml as explained the the github repository article, basically here you describe the name of your lambda function, what to exclude in the package and the trigger http events.
functions:
index:
handler: dist/index.handler
events:
- http:
cors: true
path: '/'
method: any
- http:
cors: true
path: '{proxy+}'
method: any
If we take a look at this yml file, you can see that we have 2 HTTP events, one for root path and one for proxy path.
A question that I asked myself when reading through it, so what does this event part do?
This part of the yml basically creates your endpoints in the API Gateway service of AWS (if you are using AWS as a provider).
API Gateway is a service made by AWS that lets you map requests into other services of AWS, such as a lambda function.
When you run the command sls deploy
after entering your credentials using sls config credentials the serverless framework will create OR modify if exists a new lambda function based on the config you set up, and set up API Gateway endpoints linked to that lambda.
After deploying you will receive a link that activates that lambda.
The example I use basically uses an express serverless solution that someone created, basically a proxy code that knows how to receive API Gateway request object and transform it to express and activate it without having a server running.
Note: Serverless is using CloudFormation to create a stack in order to upload and deploy the lambda function, I think with this way you can upload more than 250mb unzipped, because my project currently is 450mb unzipped, I am not sure about this but when I tried uploading a bigger zip, my lambda started overflowing, as in saying that it is missing some modules, I guess because of the size.
Or maybe serverless really optimizes the modules so the uploaded package is much smaller than what you expect. If someone knows about this, +1!
You must provide more info about your architecture.
Have you used Blueprint with microservice?
Then choose microservice-http-endpoint.
Take a look at [microservice-http-endpoint] example.
1
You can convert you whole express app in a lambda function and process request as they come in.
This blog shows how to do it for express: https://serverless.com/blog/serverless-express-rest-api/
Similar thing can be tried for NestJS as well. So lambda function is basically a piece of code that gets executed as the request come in. There are limitation on the size of the code and time to execute a job which is platform specific.
I am totally new to Lambda (or AWS) and am still to build knowledge and experience around it.
Now, I was building an app where in it requires to fetch data from twitter Hashtag.
If I got it correctly, Twitter restricts number of API calls we make every minute(?) hence we need to have a backend and needs to have oAuth2 authentication.
In a simple express app, I would have done an API call in the global scope to get the data and use setInterval to hit that API after every x minute so as to not exceed number of limits.
Now based on the very vague understanding, I guess Lambda runs function when we need it, Hence is it right to assume that we can't use lambda for such use cases?
The old-school way of doing this is to run a cron job that fires a particular script every so often. The AWS way of running code periodically is using CloudWatch scheduled events. You can configure how often you want to run a given target, and set the target as a lambda function.
https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/RunLambdaSchedule.html
I'm attempting to bulk import a lot of data into DocumentDB using Node. I've created the recommended stored procedure (the one that has been posted numerous times here and on MSDN). However, when I try executing it via Node, I get the following response: "Requests originating from scripts cannot reference partition keys other than the one for which client request was submitted."
The partitionKey on the Collection is "/id", and my request in Node looks like this:
client.executeStoredProcedure('dbs/blah/blah/sprocs/blah', [docs], {partitionKey: '/id'}, ...);
I can't find any concise documentation on this as it related to Node specifically. Has anyone encountered this issue and is there a solution? I'm completely open to the idea that I'm making a silly mistake, but this is my first rodeo, so to speak, with DocumentDB. Thank you all
I suspect that you have a data in the set you are trying to import with an /id other than the one you specify. If that's the case, I can think of two options:
Not use the sproc and load them one at a time from your node.js script. I recommend using one of the async.js functions or the equivalent from some other parallelization library so you can have many requests going in parallel.
You can separate the data client side by /id and then call the sproc for each set.
I am trying to deploy the Webhook Example for Google Actions found here onto Lambda AWS.
I was successful deploying and making the POST calls using ngrok. So, no problems there.
But the issue i found is it uses Express node module for POST request calls. Lambda AWS fails when the request is made to Express module. So is there a way to make the POST call successful.
I tried using Lambda-Express node module to deploy it, but it seems to have some issue as well.
Lambda AWS does not directly support an HTTP interface.
One solution would be for you to use API Gateway which would allow you to translate the HTTPS POST that AoG sends, into a call to AWS Lambda.
In your lambda you will handle the request which comes in via the standard Lambda handler:
function( event, context, callback );
instead of via Express. You would probably also want to remove Express from your code, which might sound like a lot of work, but I took a brief look at it when it was released and my impression was that the dependence on Express was minor and quite unnecessary.
The alternative would be to switch from Lambda to something HTTP based like Google App Engine which is also serverless (to a degree). I guess that might be easier but I don't know what your other factors might be.
I have come across this tutorial that explains the step by step process of connecting API.ai intent using the Lambda function.
And if you follow the Google web hook example to deploy it to Lambda, then it's a wrong direction. Completely eliminate the express usage and also the Assistant class is not necessary when Lambda deployment is necessary.