I have a cron running hourly inside a docker container (swarm) in Microsoft Azure.
The container is currently running all the time, and the (unix) cron keeps triggering the import script.
I would be interested if there is an elegant way of scheduling job. e.g using the https://learn.microsoft.com/en-us/azure/scheduler/
Say the scheduler spins up the container (starting the import script) and ending when done. It would handle execution intervals, failed jobs etc.
Any thoughts?
Cheers,
Tibor
Related
I am currently trying Apache Airflow on my system (Ubuntu 18) and I set it up with postgreSQL and RabbitMQ to use the CeleryExecutor.
I run airflow webserver and airflow scheduler on separate consoles, but the scheduler is only putting tasks as queued but no worker is actually running them.
I tried opening a different terminal and run airflow worker on its own and that seemed to do the trick.
Now the scheduler puts tasks on a queue and the worker I ran manually actually executes them.
As I have read, that should not be the case. The scheduler should run the workers on its own right? What could I do to make this work?
I have checked the logs from the consoles and I don't see any errors.
This is expected. If you look at the docs for airflow worker, it is specifically to bring up a Celery worker when you're using the CeleryExecutor, while the other executors do not require a separate process for tasks to run.
LocalExecutor: uses multiprocessing to run tasks within the scheduler.
SequentialExecutor: just runs one task at a time so that happens within the scheduler as well.
CeleryExecutor: scales out by having N workers, so having it as a separate command lets you run a worker on as many machines as you'd like.
KubernetesExecutor: I imagine talks to your Kubernetes cluster to tell it to run tasks.
I am trying to optimize the time how many times my cron job runs to create a python script.
I want to know how can I find or know before a process dies by itself so that way I can run my cron job accordingly
Trying to setup the time schedule of my cron job to run a python script on linux server
no code
I want to run a python cron-type scheduler and schedule a job for one time execution at a particular date and time. Once the job is done, the job has to disappear, while the scheduler still continues to run. Also looking for examples of using scheduler and storing jobs in mongodb or redis.
You can use celery for this.Celery helps you run tasks in background and schedule cron jobs as well.
These docs state that you can
schedule tasks to execute at a specific time
other useful resources.
I want to automatically start a job on an Azure Batch AI cluster once a week. The jobs are all identical except for the starting time. I thought of writing a PowerShell Azure Function that does this, but Azure Functions v2 doesn't support PowerShell and I don't want to use v1 in case it will be phased out. I would prefer not to do this in C# or Java. How can I do this?
Currently, there's no option available to trigger a job on Azure Batch AI cluster. Maybe you want to run a shell script which in turn can create a regular schedule using system's task scheduler. Please see if this doc by Said Bleik helps:
https://github.com/saidbleik/batchai_mm_ad#scheduling-jobs
I assume this way you can add multiple schedules for the job!
Azure Batch portal has "Job schedules" tab. You can go there, add a Job, and set a schedule for the Job. You can specify the recurrence in the Schedule
Scheduled jobs
Job schedules enable you to create recurring jobs within the Batch service. A job schedule specifies when to run jobs and includes the specifications for the jobs to be run. You can specify the duration of the schedule--how long and when the schedule is in effect--and how frequently jobs are created during the scheduled period.
I am using node-cron to schedule some tasks inside my node app. This package has some API to create, start and stop cron jobs. However, I can't seem to find these cron jobs when I run crontab -l command in my OS terminal. I tried both on my mac os as well as on centos.
Specific question:
Does such node packages create cron jobs at the OS level?
If answer to 1 is yes, then will these cron jobs execute irregardless my node app is running or not?
If answer to 2 is yes, then how do I stop and clear out all such schedules cron jobs?
Giving a fast look at the node-cron source code, you can check that,
node-cron does not create any cron at the OS Level.
It looks like just a long time out functionality..
I suppose that if the node process will be restarted you lost the launched cronjobs.