I want to create a lambda function that will be triggered upon a kinesis Record and store data in the MongoDB database. Data will be added to the kinesis stream through a REST API that uses the same MongoDB database mentioned above. So, can I implement that idea according to the following folder structure? If not what is the best method to do it?
Related
I've just built out my first elasticsearch datastore using the documentation and that was great. I've also set myself up a lambda to make PUT and GET calls to the elasticsearch datastore but having read multiple resources, I'm still unclear as to how to to connect my lambda up to the elasticsearch to PUT data onto it.
I am dividing the load on my Database and want to retrieve data from ES and write data to MongoDB. Can I sync them real time? I have checked the Transporter library but I want to do it for realtime.
There are several ways to achieve that :
Using your own application server. Whenever you are inserting a new
document in the mongo, put it in the ES as well at the same time.
That way you will maintain the consistency with minimum latency.
Use logstash. It has near realtime pipelining capabilities.
You can use elasticsearch mongodb river. Its a plugin used for data synchronization between mongo and elasticsearch.
I want to deploy a Node.JS script to Firebase that listens for new tweets using Twitter's streaming API, and then stores each tweet in the Firebase Database. How do I deploy this script and make sure that it runs 24/7?
As an example, consider the following (simplified) code, based on NPM Twit package:
var stream = T.stream('statuses/sample')
stream.on('tweet', function (tweet) {
// write data to Firebase Database here
})
I looked into Firebase Cloud functions, but that's not what I want, because they are triggered by an event. I also saw there is Firebase Queue, but I don't really want to queue the process: I just want it to run all the time. Is this possible currently with Firebase hosting?
An alternative would be to create a Twitter recipe with IFTTT that calls a Webbook. However, I want to store tweets from many different users, so it's not a very flexible solution.
Firebase Hosting can't run any code. Cloud Functions can run code, but not indefinitely, and only in response to defined triggers.
Consider instead using multiple IFTTT with a Cloud Function as a webhook to receive tweets that meet some search criteria, as implemented in this project. You can set up multiple IFTTT applets if you need multiple queries to be aggregated in one place. This may be your best bet if you don't want to maintain your own server.
I have a lambda function that is invoked by a get/post request to an API Gateway endpoint and then based on the request it reads/writes from/to a DynamoDB table.How can I unit test the function?I found this library called lambda-tester but it doesn't play well when dealing with Dynamo db.I also tried the Amazon's unit and load testing blueprint, but am not a big fan because it writes the results to yet another dynamo db table and I would rather just test locally and see the output on the terminal.
We are working on a set of tools for just this. Take a look at this blog:
http://docs.bespoken.tools/en/latest/tutorials/tutorial_bst_emulator_nodejs
Our primary intention is to help Alexa development, but you can "talk" to any lambdas. Is your lambda an Alexa skill (and dynamo is for persistence)?
I actually was able to use my non-local dev database after setting up AWS CLI, which required the necessary AWS credentials. Once you do that and set the region in the code, you can access your non-local dev database without a problem and test the code.
I have Lambda function that runs every 5 mins and queries for new tweets for a particular user ID. This uses REST API now and works pretty well.
Using streaming API might have been a better way instead of running a nodeJs lambda function every 5 mins, right? Is there a way to use streaming API on Lambda nodeJs? Or any other code hosting service?
Check the code in the following repo; It's connected to Twitter API stream and monitor some keywords and respond and insert to dynamoDB.
https://github.com/PBXDom/Twitter-Marketing-Nodejs
We use this code in our company for monitoring keywords in Twitter and find related leads.