I have been using Quartz for over a year to schedule jobs in my app. I now have a new requirement to get rid of quarts, and instead use crontab. I have no experience in this area. How do I design something in my app that allows a job to be called by the operating system? Do I need to create a URL for each job that the OS can call?
you could define simple rest or soap services (using apache-cxf or axis2 for example) and call the from a ruby- or pythonscript
My company makes CloudQuartz (www.thecloudblocks.com) which allows you to schedule the jobs through an API and get callbacks when they are due to run.
It might be useful to what you're trying to do.
Related
I'm trying to find a tool where you can easily monitor cronjobs for the company I work at. Rundeck seems like the perfect tool for this but I can't figure out if it's possible to integrate the existing cronjobs into Rundeck. It's too much work to do this manually because there are hundreds of them.
If you now another tool that can do this feel free to recommend!
I'd rather want it to be open source but if it's paid and it works correct I'm open to it.
Rundeck works as a "very enhanced cron replacement" maybe the best approach in your case is to migrate your cron jobs to Rundeck and use the notification for monitoring. This looks like a good starting point for you, don´t forget to visit the official documentation.
What Mega mentions is correct, rundeck can be used as a replacement for cron.
Even better, there is no need to configure all of those manually: Rundeck allows you to import job definitions via API call.
Steps:
Get rundeck installed
Set up a job to run one of your cron jobs manually
Export that job
Use a script to make many copies of that file, that each contain a different item from cron*
Import all those files via API call
*You'll need to change at least the name and workflow (called sequence commands file - you'll be able to see based on the workflow step you configured)
Me and my team are using a shared hosting service with a limited linux container (without root and privileged user) and we need to develop a new feature that involves long running tasks (> 600ms).
We thought of two possible solutions:
Breaking apart the task, and via the frontend, make one separate http request to the server.
Use bash screen to run a bash script with a infinite loop calling php artisan schedule:run (mimicking cronjob)
I don't feel very confortable with the first solution, moving server logic to the browser seens wrong in my opinion.
The second solution is only a supposition (not tested), we are not sure if the bash screen would randomly stop at any time.
What would be the least unstable way to achive our goal? Thx
Assuming you already explored this because you mention that a CRON would not be an option, but even unprivileged users can setup a CRON, which is the simplest solution in combination with the Laravel scheduler.
If an actual CRON using the scheduler is really out of the question I do think making an HTTP endpoint you could call from the browser is the next best thing. Just don't think an endpoint you can call from a browser that you can only call it from a browser ;)
Take for example https://www.easycron.com/ (no affiliation but the first Google result). You can setup a CRON job there to call a URL to trigger those tasks on a CRON interval. Internally at my company called the "poor mans CRON" :)
I would agree that running a "screen" session is the most unreliable since on a server reboot those are not started again and if you "infinite loop" crashes it will not automatically restart.
If you go that route (or any CRON route) you can add some monitoring using for example https://healthchecks.io/ (again no affiliation, Google). This allows you to define a CRON schedule and gives you a URL to call after the CRON finishes, if your CRON does not call that URL according to the CRON schedule you will be notified. Good to have as insurance.
i have to do a reptitive task in nodeJS and i've seen there is existing package like this one.
https://www.npmjs.com/package/node-cron
and the actual platform where i'm hosted propose inside cronjob.
https://www.netlify.com/docs/webhooks/
so my question is when it's more interessant to use the platform or a package.
thanks.
From the URL posted i didn't see any method of setting up a cron job using webhooks. Unless you were thinking of setting up a webhook that listens for a post which is sent using a linux cron job or the like?
Regardless, the actual question about using a platform or a package. They have pros and cons, but based purely on your question I would go with the platform.
If you choose to use a package you will have to write the code to call the package (which you need to test, maintain and run). You need to ensure that the node process is always up and running, if it dies or exits that it is re-spawned, that if the operating system reboots the node process gets kicked off again. All these problems are can be easily solved (PM2 for instance) but the fact is you need to think of the problems and solve them yourself or the cron job might not run when you want it to.
When using the platform you know that it is well tested, that it will work as documented, and that it will be resilient to failure modes that you might not be aware of.
My question is in regards to clarification and/or anybodies previous experience with NodeJS and Google Cloud Platform (GCP).
I have developed numerous NodeJS scripts that read and transform serveral JSON sports feed in order to populate a Google Firebase database backend.
The NodeJS scripts work exactly as desired; with the exception that I need to run/execute the NodeJs script manually in order to populate the backend. I obviously want this to be automatically, lets say an interval of every 2 mins.
I am unclear on how to achieve this!? Does GCP offer a cron job that can execute my NodeJS on a specific time interval? If so how should I implement this?!
If you are planning on using Compute Engine you can just use a cron job which comes with the both the Debian and Red Hat Linux public images available within Google Cloud Platform.
You could create an enrty like this to run the script every 2 hours.
* /2* * * * /usr/local/bin/node /home/example/script.js
Here are two examples of how to do this using cron and appengine:
https://github.com/firebase/functions-cron
https://mhaligowski.github.io/blog/2017/05/25/scheduled-cloud-function-execution.html
The basic idea is the same: one appengine app for cron, where you tell it what URL to get, at what frequency. What is serving at the URL is immaterial here, you would obviously have your nodejs app in an appengine instance, serving URLs that match those given to cron. The cron portion of the examples is independent of language, it is REST based.
So the steps for you would be:
Setup your nodejs app in GAE the standard way (regardless of the
fact that you want your app URLs called at intervals)
Setup your cron app in GAE as explained in those examples
Notice your nodejs app of step 1 being called as you specified in step 2!
I am looking for a front end for our operator to run our Talend jobs. We do not want him to have the ability to delete or modify jobs. Only to run them and monitor their results. Any suggestions for tools for doing this?
Thanks
The subscription version of Talend (called Talend Integration Suite, or TIS) has precisely that. It's a web-based console called Talend Administration Center (TAC) and it allows an operator to run jobs and monitor their results -- among many other things. The permissions can be set in the way you described, so that the operator is not able to delete or modify the jobs.
In my company we use SOS Job Scheduler, which you can use with TOS out-of-the-box as a general diy scheduling solution. Just these days I'm working on a Talend Custom Component to integrate deeply between the first and the latter (like log-propagation, context parameter sharing and so on. Basically it let you use SOS API from a TOS job). I can speed-up and put on github in a few days, if you need :)
You can export job as shall script (.sh) and anyone can execute that and will not able to delete those jobs.
To see the execution details put some tlog component into your talend jobs.
This is just a work around not the solution if you only have open studio (free version).
I'm using Rundeck for manual and scheduled execution of Talend jobs . I find the job setup and scheduling to be far easier in Rundeck than in SpagoBI (which I frequently use for reports and BI).
You could also use something like automic (uc4) automation engine. It has fine grained security and is a commercial product . I've used it in the past for similar work.