Our node app gets quite big and one job takes quite some time to execute. We run this job with a cronjob, but by calling the URL. Now Heroku has problems with this, because the job takes more than 30 seconds to finish. So we receive a time-out and after that it tries to execute it immediately again, and again, till our Memory quota is about 300% and the app crashes.
Now I want to fix this. Locally we don't have any problems running this script at all. It takes about a minute (for now, but in the future if we have more users it may take more time) to finish and memory stays stable.
Now running this script on the background should fix the problem according https://devcenter.heroku.com/articles/request-timeout#debugging-request-timeouts
Overe here https://devcenter.heroku.com/articles/asynchronous-web-worker-model-using-rabbitmq-in-node#getting-started I read about JackRabbit. But it seems like it's used for systems like RabbitMQ https://github.com/hunterloftis/jackrabbit
So my question: anyone who has experience with background tasks in node? Can and should I use JackRabbit for my background tasks, or are there better solutions? My background task just contains a very complex ExpressJS task, which takes some time to execute so....
I'm the Node.js platform owner at Heroku (and I actually wrote the web worker article you referenced).
Your use case sounds like it may fit the scheduler very well:
https://devcenter.heroku.com/articles/scheduler
It's a great replacement for cron-type jobs.
Related
I have been looking for a time based persistent scheduler. I looked into some applications (Agenda, node-cron, node-schedule). But I couldn't find anything that satisfies my criteria.
So my applications sends out reminders to our customers based on their event timings. I am hesitating to run a regular cronjob because I have to run every 15 mins or so in this case. And for each cronjob, I have to make a database call. I am trying not to use resources unnecessarily.
In addition to that, I am already running a lot of cronjobs. But in my case, when the job is completed, I want the cron to get cancelled/finished; not live on memory until the server restart happens.
I tried using the above specified applications by setting exact timestamps (agenda, node-cron, node-schedule). But the cron lives on forever even after the job is completed, and if i restart the server, all the scheduled jobs are cron. So persistence is also an issue I am facing.
My server uses node js. If there are any other languages/tools to make this work, I am all ears.
Looking forward to your help.
I tried following this solution. But this solution is for one predefined event. In my case, the number of reminders to be sent out are dynamic and jobs are to be scheduled on the fly.
I am working on a nestjs app that makes heavy use of task scheduling using the #nestjs/schedule package that integrates with the node-cron npm lib.
At the moment, the app has been in development for over 6 months and has over 30 cron tasks running in the background simultaneously. although most of them have distinct intervals, some crons have the same interval (e.g. runs EVERY 30 SECONDS).
All cron tasks more or less follow the same behavior;
send request to external APIs to get data.
query mongo db to run some checks and update records accordingly.
some crons emit events to the client when certain condition is met by other cron tasks.
my question is:
How can I measure the performance of the node process while running all these background tasks in my local development PC? and what effect it might have on requests that comes from the client?
another point is: Is it possible to detect a memory leak before it happen?
basically I have concerns about the app performance and I want to try to prevent the problem before it happen.
Thanks.
So, I see that heroku provides the option to run a command at a specific time. Information on the scheduler here.
LINK: https://elements.heroku.com/addons/scheduler
However, if you go through the steps when setting it up, they do not provide a lot of flexibility on when you can run your code daily. For example, you can only run code at 4:00pm or 4:30pm, not 4:10pm.
How can I make it so that a node.js file would run on heroku at a specific time (like 4:10pm or 2:15pm, some time outside the options given on heroku) on a daily basis?
There appears to be no support for node.js explaining this either on their website.
Might be just a work around, but you could start the process at the nearest time slot, that is before your desired time, let it run and wait passively until your desired time and let it do the actual task just then.
However notice as Heroku mentions in the documentation Heroku Scheduler isn't guaranteed to run the task, even though it's very reliable. If you need something critical or have something that has to been run everyday for sure, you should probably make a separate process, which handles the scheduling.
We've added a simple Heroku add-on called Cron To Go that does exactly that - you can use Cron expressions for accuracy and schedule one-off Dynos, just like with Heroku Scheduler.
There's also a simple Node example here.
I've setup a node.js server with cron jobs via node-cron, which is js-land cron implementation. I've noticed that sometimes the jobs are not launching, aparently without errors and following an unknown pattern.
Well, since this server is a free one, I was thinking that maybe it goes to sleep when there is no activity, so that the jobs are not launching. I've looked the docs and I haven't seen any clear indication on this.
I've already seen the Cron addon, but I'm not interested on it. I'll like to make it work within a js process.
Thanks!
cloudControl uses Container idling (https://www.cloudcontrol.com/dev-center/Platform%20Documentation#deploying-new-versions) for free deployments.
If a free deployment (1 container with 128MB) does not get any requests within a timeframe of one hour the container is idled ("server goes to sleep").
I'm planning to host an express app on Heroku (I'm already experimenting with a single dyno).
I want to use node-cron for maintenance tasks (doing some MongoDB updates). The question is, what's the simplest way to make sure the maintenance only runs once? All dynos would try to run the maintenance at the same time.
My current approach uses MongoDB's atomic upserts as some sort of semaphore (every dyno tries to set the flag for the current maintenance). But that's kinda ugly.
I'd like to not have a separate worker instance since it's really just a simple task that needs to be run once a day.
I think that's what the Heroku Scheduler is good for (https://devcenter.heroku.com/articles/scheduler). If the execution of your update isn't taking too long it's the way to go. You write a JS script (e.g. schedule.js) that knows how to update your MongoDB, put it in your root and schedule it using the Heroku Scheduler (it comes with a trivial frontend) to invoke it (node scheduler.js) at the time desired.