I have Lambda function that runs every 5 mins and queries for new tweets for a particular user ID. This uses REST API now and works pretty well.
Using streaming API might have been a better way instead of running a nodeJs lambda function every 5 mins, right? Is there a way to use streaming API on Lambda nodeJs? Or any other code hosting service?
Check the code in the following repo; It's connected to Twitter API stream and monitor some keywords and respond and insert to dynamoDB.
https://github.com/PBXDom/Twitter-Marketing-Nodejs
We use this code in our company for monitoring keywords in Twitter and find related leads.
Related
I have a node app that takes an url, scrape some text with puppeteer and translate it using deepl before sending me back the result in a txt file. It works as expected locally but having a lot of urls to visit and wanting to learn, I'm trying to make this app works with AWS Lambda and a docker image.
I was thinking about using a GET/POST request to send the url to API Gateway to trigger my lambda and wait for it to send me back the txt file. The issue is the whole process takes 2/3 minutes to complete and send back the file. It is not a problem locally but I know you should not have an http request wait for 3 minutes before returning.
I don't really know how to tackle this problem. Should I create a local server and make the lambda post a request to my ip adress once it is done?
I'm a loss here.
Thanks in advance!
One can see a few alternatives to what is seemingly asynchronous processing concern.
Poke the lambda with the data it needs (via an API, SDK, or CLI) then have it write its results to an S3 bucket. One could poll the s3 bucket for the results asynchronously and pull them down, obviously this requires some scripting.
An another approach would be to have the lambda post the results to a SNS topic that you've subscribed to.
That said, I'm not entirely sure what is meant by local IP, but I would avoid pushing data directly to a self-managed server (or your local IP), rather I would want to use one of the AWS "decoupling" services like SNS, SQS, or even S3 to split apart processing steps. This way it's possible to make many requests and pull down the data as needed.
I want to deploy a Node.JS script to Firebase that listens for new tweets using Twitter's streaming API, and then stores each tweet in the Firebase Database. How do I deploy this script and make sure that it runs 24/7?
As an example, consider the following (simplified) code, based on NPM Twit package:
var stream = T.stream('statuses/sample')
stream.on('tweet', function (tweet) {
// write data to Firebase Database here
})
I looked into Firebase Cloud functions, but that's not what I want, because they are triggered by an event. I also saw there is Firebase Queue, but I don't really want to queue the process: I just want it to run all the time. Is this possible currently with Firebase hosting?
An alternative would be to create a Twitter recipe with IFTTT that calls a Webbook. However, I want to store tweets from many different users, so it's not a very flexible solution.
Firebase Hosting can't run any code. Cloud Functions can run code, but not indefinitely, and only in response to defined triggers.
Consider instead using multiple IFTTT with a Cloud Function as a webhook to receive tweets that meet some search criteria, as implemented in this project. You can set up multiple IFTTT applets if you need multiple queries to be aggregated in one place. This may be your best bet if you don't want to maintain your own server.
I have a Node app/script that needs to constantly be running (it's a discord bot, done with discord.js, but I think that's mostly irrelevant), and I'd like to do it on Firebase.
It has its own client.on('event', ()=>{}) events system, so I don't believe that I could use Firebase's cloud functions. There's also what seems to be a website-hosting based way to have a node.js server, but that seems triggered by HTTP requests.
Is there any other way I could do it?
There is no way to run arbitrary node.js code on Firebase. Unless your script can run within Cloud Functions "triggered execution" mode, you'll need your own app server to run it.
You can of course create a service that maps Discord.js events to Firebase events, such as writes to the Realtime Database, Cloud Firestore, even just direct HTTPS calls to a Cloud Functions endpoint. You could even bypass Firebase there and have your mapping service write to Cloud PubSub and use that to trigger Cloud Functions.
One thing that looks promising in the Discord.js documentation is their mention of web hooks, which is just another way of describing HTTP endpoints. But from my quick scan I couldn't figure out if those would allow you to call your HTTP triggered Cloud Function.
I'm developing a chat-bot using api.ai for NLP and i'm stucked where i need to query some data from a database. And also i need to do some processing of those data. I'm not much clear how can I do this using the webhook. Any thoughts and tips would be really appreciated.
You might want to check out API.AI's getting started guide, the last part of which involves the basics of adding fulfillment to your agent which pulls from this getting started sample
We use Cloud Functions for Firebase in the sample but you can't call non-Google APIs or URLs without setting up billing. You can also use any other hosting providers (Google's AppEngine, Compute Engine, Amazon's Lambda, EC2, Heroku, Azure, etc.) for fulfilling API.AI agents, as long as you have a server that can respond to a HTTPS request, it'll work with API.AI's webhook fulfillment. To get started you can even use your local machine and tools like ngrok to tunnel the connection from a HTTPS URL to your local development machine: https://ngrok.com/. If your just starting out I'd recommend doing whatever your most comfortable with.
Am currently using Node-RED to process the data received through the webhook. The webhook receives the data in JSON and if you’ve setup your intents and entities correctly, you’ll get data that you can act on.
This can be achieved as a stand-alone nodeJS app that you host somewhere, like Heroku or OpenShift. But important point is that you need a server running a program somewhere to process the data that api.ai sends.
Am using firebase as my database and is integrated into my Node-RED setup and now I can do sophisticated queries like “how much did I walk last Wednesday”
I'm working on a facebook chat bot, and I'm developing it using the serverless framework (Node.js) and deploying it to aws lambda. For the first few weeks, I just ran a local serverless lambda simulator using the serverless offline plugin and everything was working great. Yesterday, I finally decided to deploy it to AWS lambda, and now I see a significant drop in performance and consistency. Sometimes the bot takes 10 seconds to respond and sometimes it is instantaneous. The weird part is, on the lambda cloud logs, it always says the function completes in around 150 ms, which seems super fast, but the facebook bot simply doesn't mirror that speed. I am hitting a database, but the queries are definitely not taking anywhere near 10 seconds to run.
UPDATE:
I decided to try to test the bot my manually sending requests to the API endpoint using postman (which is basically curl). Every time the api responded instantly, even when I send the exact same request body that the messenger does. So it seems like the request is just taking a long time to reach the lambda api, but when it gets there it runs as it should. Any ideas of how to fix this?
If the API is responding quickly to your curl request, then the problem isn't on AWS end. Try matching when you send your request via Facebook to your app and when your app recieves it.
If it's getting held up on Facebooks end, Im afraid there isnt much you can do to solve it.
Another issue could be the datacenter your lambda is running in versus where facebook is. For example, using chkutil.com, you can see facebook.com seems particularly slow from the Asia-Pacific datacenters.
As it turns out, Facebook was experiencing DNS issues and has since remedied the issue.