I have a cronjob which processes lakhs of data with API calls and runs very frequently. Now the issue is sometimes the API calls are timed out and the cronjob gets stuck in running state. When next trigger time comes a new cronjob is started on a different thread. This results in duplicate data and multiple instances of same job running on multiple thread. How do i stop it.?
Related
I need to build a Node.js API that, for each different user that calls it, starts running some piece of code (a simple script that sets up a Telegram client, listens to new messages and performs a couple of tasks here) that'd then continuously run in the background.
My ideas so far have been a) launching a new child process for each API call and b) for each call automatically deploying the script on the cloud.
I assume the first idea wouldn't be scalable, as for the second I have no experience on the matter.
I searched a dozen of keyword and haven't found anything relevant so far. Is there any handy way to implement this? In which direction can I search?
I look forward to any hint
Im not a node dev, but as a programmer you can do something like:
When user is active, it calls a function
this function must count the seconds that has passed to match the 24h (86400 seconds == 24 hours) and do the tasks;
When time match, the program stops
Node.js is nothing more that an event loop (libuv) whose execution stack run on v8 (javascript). The process will keep running until the stack is empty.
Keep it mind that there is only one thread executing your code (the event loop) and everything will happen as callback.
As long as you set up your telegram client with some listeners, node.js will wait for new messages and execute related listener.
Just instantiate a new client on each api call and listen to it, no need to spam a new process.
Anyway you'll eventually end in out of memory if you don't limit the number of parallel client of if you don't close them after some time (eg. using setInterval()).
A Node.js app running, which only processes queues from Redis using Bull.js. The job performs a task and then it adds another entry in Redis with same data. Like this:
job({dataId}) {
// Task
queue.add({dataId}, {delay});
}
There is only one job running per dataId and it's been made like this because I need the delay of the queue to be random every time it's processed.
The problem:
Given time with the app running, the jobs will be duplicated as there will be more than one job with the same dataId being processed by Bull.js. I only got realized that when I used the Task Force to see the status.
I can check if there are any jobs running with the dataId before adding it into the queue but that definitely would not be optimal, so I'd like to know what's causing it so I can prevent it from happening.
I am using PM2 in cluster mode and have 2 instances of my node.js application running. I have some long executing cron jobs (about 30 seconds) that I am trying to run. I am placing an if statement before the execution of the cron jobs to ensure that they only run on the first process via
if (process.env.NODE_APP_INSTANCE === 0) {
myCronFunction()
}
The goal was that since there are two processes, and PM2 should be load balancing them, if the cron job executes on process one, then process two would still be available to respond to the request. I am not sure what's going on, if PM2 is not successfully load balancing them, or what. But when my cron job executes on instance one, instance two is still not responding to requests until after the job on instance one finishes executing.
I'm not sure why that is. It is my understanding that they are supposed to be completely independent of one another.
Anyone have any ideas?
I've changed a long running process on an Node/Express route to use Bull/Redis.
I've pretty much copied this tutorial on Heroku docs.
The gist of that tutorial is: The Express route schedules the Job, immediately returns 200 to the client, and browser long polls for the job status (a ping route on Express). When the client gets a completed status, it displays it in the UI. The Worker is a separate file and is run with an addtional yarn run worker.js.
Notice the end where it recommends using Throng for clustering your workers.
I'm using this Bull Dashboard to monitor jobs/queues. This dashboard shows the available Workers, and their status (Idle when not running/ Not in Idle when running).
I've got the MVP working, but the operation is super slow. The average time to complete is 1 minute 30 second. Whereas before (before adding Bull) it was seconds to complete.
Another strange it seems to take at least 30 seconds for a Workers status to change from Idle to not Idle. Seems that a lot of the latency is waiting for the worker.
Being that the new operation is a separate file (worker.js) and throng is enabling clustering, I was expecting this operation to be super fast, but it is just the opposite.
Does anyone have an experience with this? Or pointers to help figure out what is causing this to be so slow?
I am using nodejs on google app engine with an end point for a cron job. When the rest end point is called I want to proceed with my cron job after returning the response back to the caller.The cron task will continue for about an hour. Will GAE terminate the task if it runs for an hour or more ? I suppose GAE should not kill my nodejs server process because that way my application will stop. I want to know if there is any possibility for the task to end prematurely due to some restrictions on GAE.
It depends on which type of scaling you have selected: https://cloud.google.com/appengine/docs/standard/java/an-overview-of-app-engine
Requests on Basic & Manual Scaling can run indefinitely, Automatic scaling has a 60 second deadline for http requests & 10 minutes for task queue requests. If you're not sure which type of scaling you have you probably have Automatic.
You could setup a micro-service with Basic scaling specifically for tasks like this; so that your primary service can stay on Automatic scaling.
You could also split up your cron task into several tasks, and then daisy-chain them using push queues (i.e. you cron task launches, does some work, and then launches task2 and dies. task2 launches, does some work, launches task3 and dies. etc)