I need to deploy few cron jobs for my nodejs app hosted on google cloud.
I have 2 options
cron-schedule npm package.
cron.yaml provided by app engine.
I am not sure which has better performance and why ?
adding to it, I want my cron jobs to work on preemptible instance as they are meant for it, i search a lot but didn't find anything related.
I can speak for cron.yaml.
1. Cron.yaml has timeout limit of 60 mins.
2. Cron.yaml has retry limit of 5
3. Cron are vm level so multiple apps with their own cron is out of question.
4. From fail or time out log entry doesn't include clear number of retry.
Related
I have been looking for a time based persistent scheduler. I looked into some applications (Agenda, node-cron, node-schedule). But I couldn't find anything that satisfies my criteria.
So my applications sends out reminders to our customers based on their event timings. I am hesitating to run a regular cronjob because I have to run every 15 mins or so in this case. And for each cronjob, I have to make a database call. I am trying not to use resources unnecessarily.
In addition to that, I am already running a lot of cronjobs. But in my case, when the job is completed, I want the cron to get cancelled/finished; not live on memory until the server restart happens.
I tried using the above specified applications by setting exact timestamps (agenda, node-cron, node-schedule). But the cron lives on forever even after the job is completed, and if i restart the server, all the scheduled jobs are cron. So persistence is also an issue I am facing.
My server uses node js. If there are any other languages/tools to make this work, I am all ears.
Looking forward to your help.
I tried following this solution. But this solution is for one predefined event. In my case, the number of reminders to be sent out are dynamic and jobs are to be scheduled on the fly.
I've recently started using an App Engine on Google Cloud Platform and have set up some cron jobs to get some scheduled work done. However recently one of my tasks took more than 10 minutes and it timed out... obviously I could break this work into batches or find another way around the problem, however I'm keen to not always be mindful of how long a job might take and want future jobs to run until completed or failed.
I've looked into various services that Google offer but with no success; Task Queue is Python 2.x only and Cloud Tasks has the same 10 minute limit unless you manually manage scaling (which I would prefer to stay automatic as that's the point of App Engine for me).
Am I missing something? This 10 minute limit seems like a big unnecessary blocker and I have no idea where to look.
https://cloud.google.com/tasks/docs/creating-appengine-handlers
Thanks for your time.
Google services such as App Engine are designed to model a web server HTTP Request / Response design. You are trying to use them as task/execute engines.
Use the correct service if you require long execution times, which usually means requests that take longer than a few minutes to complete. Use Cloud Tasks and Compute Engine. Otherwise you will need to architect your application to fit with App Engine's requirements and limitations.
Cloud Tasks for Asynchronous task execution
If you want to use App Engine, you need to use either Basic Scaling or Manual Scaling. I understand that manual scaling isn't your favorite, I also don't like this mode. But the basic scaling is acceptable.
In addition, it's more designed to perform background task, exactly what you try to achieve.
If you accept this change, you can use Cloud Task. You have up to 24H of timeout if your App Engine service is in basic scaling (or manual)
You have this same information, on the scaling description on App Engine documentation.
When you use basic scaling, your instance type needs to be updated to BXXX.
I have used Azure Schedulers before for quick jobs before. It targets a URL which is ASPX page or WebApi and it did the job.
Now I have a job that takes up to 15-20 minutes. Of course, I am getting timeout error after 30 seconds.
I'm trying to avoid creating a Windows Service or some console application that would run on Azure VM, rather have a non-UI application that runs in the background.
Do you have any suggestion what should I do?
You should use an Azure WebJob for this. WebJobs support simple scheduling via a cron expression (details here). Basically you upload a simple script file or exe that performs the work you want done, upload it to your WebApp along with a cron schedule expression, and Azure WebJobs will make sure it runs on schedule.
For your scenario, you'll want to create a "Continuous" WebJob and ensure you've enabled "Always On" which ensures the background job continues running (i.e. it isn't request triggered).
WebJobs sure is a good solutions, but it will share resources with its attached Web App.
You could consider using an Azure Cloud Service. I do that myself for longer running tasks, that are more CPU intensive.
Read more
For long running WebJobs, you have to tinker with the Timeout value (by default 2 minutes) or make sure your Webjob makes some Console.Writes.
To achieve that, go to the Web App Settings > Application Settings and add the following configurations:
WEBJOBS_IDLE_TIMEOUT - Time in seconds after which we'll abort a running triggered job's process if it's in idle, has no cpu time or output.
SCM_COMMAND_IDLE_TIMEOUT - Time in milisecods. By default, when your build process launches some command, it's allowed to run for up to 60 seconds without producing any output. If that is not long enough, you can make it longer, e.g. to make it 10 minutes:
I've setup a node.js server with cron jobs via node-cron, which is js-land cron implementation. I've noticed that sometimes the jobs are not launching, aparently without errors and following an unknown pattern.
Well, since this server is a free one, I was thinking that maybe it goes to sleep when there is no activity, so that the jobs are not launching. I've looked the docs and I haven't seen any clear indication on this.
I've already seen the Cron addon, but I'm not interested on it. I'll like to make it work within a js process.
Thanks!
cloudControl uses Container idling (https://www.cloudcontrol.com/dev-center/Platform%20Documentation#deploying-new-versions) for free deployments.
If a free deployment (1 container with 128MB) does not get any requests within a timeframe of one hour the container is idled ("server goes to sleep").
I've seen that Heroku charges $15/mo to run Delayed Job, and $3/mo to run cron tasks daily. Is it possible to skip that entirely and run my own cron tasks manually? Or are they somehow figuring out that I'm running cron tasks?
I'm not entirely sure what you mean by "run my own cron tasks manually". For cron specifically, you need access to crontab, which they can control, as they're their servers. If you have another way of doing it, it would probably be fine, but bear in mind that your app is not tied to a specific server when running under Heroku, and that the server will change between executions.
Also, unless they've changed it since last time I checked, you can run daily cron tasks for free, but hourly costs $3/mo.
EDIT: Yes, daily crons are free. See http://addons.heroku.com/.
If you install the Heroku gem on your computer, you can then run your cron tasks manually as follows:
$ heroku rake cron
(in /disk1/home/slugs/xxxxxx_aa515b2_6c4f/mnt)
Running cron at 2010/04/25 10:28:54...
This will execute the exact same code as Heroku's daily/hourly cron add-on does; that is, for this to work, your application must have a Rakefile with a cron task, for example:
desc "Runs cron maintenance tasks."
task :cron do
puts "Running cron at #{Time.now.strftime('%Y/%m/%d %H:%M:%S')}..."
# TODO: your cron code goes here
end
Now, just add the heroku rake cron command to a crontab on any Unix server of yours, or even directly to your personal computer's crontab if you're running Linux or Mac OS X, and you can be scheduling cron jobs for your Heroku application as you please and without being charged for it.
Updating the answer for 2020:
You can use Heroku Scheduler which is Heroku's own add-on that lets you schedule commands using one-off dynos (so that you only pay for the run time of your jobs). The add-on itself is free, but doesn't really allow you to use cron but rather plain frequency: every day, every hour or every 10 minutes. Also, there's no guarantee that your job will execute at the scheduled time or at all..
There are other 3rd party add-ons that can help you run one-off dynos using cron expressions for better flexibility and are more resilient than Heroku Scheduler (proper disclosure, my company is the creator of one such add-on).
You can also use custom clock process (see here for more info) which essentially means that you have one dyno or process spawn tasks that run on other dynos. This usually costs more than using the aforementioned add-ons, but you have more granular control over your processes and since you only rely on Heroku, it may be more stable.
Yes, I've successfully used a cron job on my local server which essentially runs
$ heroku rake <rake task>
at whatever intervals I've required. I've used in on both aspen and bamboo stacks.
You can also just install a gem like rufus-scheduler if you're running a rails app and setup scheduling that way. I don't know if this is bad practice for some reason, but it's what I do with my app, and it seems to work fine.
If you want to have scheduled jobs you can also use http://guardiano.getpeople.in that is a free service (for 10 jobs) for job scheduling.
You just need to setup an HTTP endpoint in your application to recieve event notifications on POST or GET and you can also set some additional params to prevent not authorized actions.
So you set a job in Guardiano that will cal http://yourapp.com/youraction and leave "minutes" blank if you want you action to run once in the future or set minutes to what you want to run your action every X minutes. In that way you only have to create your endpoint for you app and when this EP is called you execute something.
So your app can sleep and you don't need to spend money and time setting up jobs and taking care that they are working properly.
IMHO if you need something fast for an MVP or you need to setup a lot of jobs for different apps than a free service like that where you can actually outsource cronjobs is quite good.
There was aslo an Heroku Addon called Temporize to do that but I'm not sure is still alive and working