I have a Map/Reduce script running in every 15 minutes. While I check the schedule status, it seems few times the execution skips. I have set the priority to "High", Concurrency limit to "2". Please see the screenshot for reference. I am not getting why this skips for few cases. Can anyone please help me on this? Thanks in advance.
From the above example, the script skips to execute on 3/16/2021 5:01.42 pm
You can try listing all your scheduled and map/reduce scripts. It is likely that some other scheduled or map/reduce script was running at that time. Netsuite will not interrupt a running task so if something else was running through the next slot Netsuite won't have run it.
The other thing I've seen is scripts scheduled for near midnight California time are often skipped or terminated. I haven't looked at this lately so they may have fixed this but if that time is midnight California then that's likely the reason.
Related
I have an uploader service which needs to run every 5minutes and it definitely finished within 5 minutes so there are never two parallel session.
Wondering what would be a good strategy to run this, either to schedule this as a cron job on host or start a go program with infinite loop which execute the program and sleeps(Golang: Implementing a cron / executing tasks at a specific time)
If your task is...
On Unix
Stand alone
Periodic
Has an acceptable startup time
cron will be better than rolling your own scheduler just for the one service. It will guarantee the process will always run at the correct time and has rudimentary error reporting. There's no need to add a watchdog in case your infinite loop has an error, cron will run the process again in 5 minutes.
If cron is insufficient, look into other job schedulers before rolling your own.
I have an uploader service which needs to run every 5minutes and it definitely finished within 5 minutes so there are never two parallel session.
These are famous last words. I would suggest adding in some form of locking. For example, write your PID to a file in /var/run and check if that process is running. There's even a little pidfile library for Go.
Take a look on Systemd, you can execute a script with timers and set max execution time for the script.
https://wiki.archlinux.org/index.php/Systemd/Timers
I Have crontab set up (on my local MacOSX system) to run a job on a per hourly basis. It runs fine. I am not sure if it is possible, but is there a way that I can run the job for 'missing' hours (in case my computer sleeps or I shut it down)?
For example if cronjob ran fine for hours (1-13) before I shut down the system. I start up the system again after, lets say 2 hours. Is there a way to tell cron to run the job for missing hours (14,15) too before executing hour 16?
The cronjob currently running fills up some data in my local MySQL DB with hour information in one of the columns. Any tips, tricks or libraries will be helpful.
Thanks.
Maybe have a look at anacron for Mac that can catch up jobs missed while your Mac was sleeping the next time it wakes up. See here.
I'm doing a research project that requires I monitor cron jobs on a Ubuntu Linux system. I have collected data about the jobs' tasks and when they are started, I just don't know of a way to monitor how long they take to finish running.
I could calculate the time of finishing the task minus starting it with something like this but that would require doing that on the Shell scripts of each cron job. That's not necessarily difficult by any means but it seems a little silly that cron wouldn't in some way log this, so I'm trying to find an easier way :P
tl;dr Figure out time cron jobs take from start to finish
You could just put time in front of your crontabs, and if you're getting notifications about cron script outputs, it'll get sent to you.
For example, if you had:
0 1,13 * * * /maint/run_webalizer.sh
add time in front
0 1,13 * * * time /maint/run_webalizer.sh
and you'll get some output that looks like (the "real" is the time you want):
real 3m1.255s
user 0m37.890s
sys 0m3.492s
If you don't get cron notifications, you can just pipe the output to a file.
man time. Maybe you can create a wrapper and tell Cron to use it as your "shell" or something like that.
Cronitor (https://cronitor.io) is a tool I built exactly for this purpose. It uses http requests to record the start and end of your jobs.
You'll be notified if your job doesn't run on schedule, or if it runs for too long/too short. You can also configure it to send alerts to you via email, sms, but also Slack, Hipchat, Pagerduty and others.
I use the Jenkins CI to do this via its external-monitor-job plugin. Jenkins can track start and end times, track overall execution time over time, save the output of all jobs it tracks, and present success/failure conditions graphically.
https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+external+jobs
I have an application that has to launch jobs repeatingly. But (yes, that would have been to easy without a but...) I would like users to define their backup frequency in application.
In worst case, they would have to choose between :
weekly,
daily,
every 12 hours,
every 6 hours,
hourly
In best case, they should be able to use crontab expressions (see documentation for example)
How to do this? Do I launch a job every minutes that check for last execution time, frequency and then launches another job if needed? Do I create a sort of queue that will be executed by a masterjob?
Any clues, ideas, opinions, best pratices, experiences are welcome!
EDIT : Solved this problem using Akka scheduler. Ok, this is a technical solution not a design answer but still everything works great.
Each user defined repetition is an actor that send messages every period to a new actor to execute the actual job.
There may be two ways to do this depending on your requirements/architecture:
If you can only use Play:
The user creates the job and the frequency it will run (crontab, whatever).
On saving the job, you calculate the first time it will have to be run. You then add an entry to a table JOBS with the execution time, job id, and any other information required. This is required as Play is stateless and information must be stored in the DB for later retrieval.
You have a job that queries the table for entries whose execution date is less than now. Retrieves the first, runs it, removes it from the table and adds a new entry for next execution. You should keep some execution counter so if a task fails (which means the entry is not removed from DB) it won't block execution of the other tasks by the job trying again and again.
The frequency of this job is set to run every second. That way while there is information in the table, you should execute the request around as often as they are required. As Play won't spawn a new job while the current one is working if you have enough tasks this one job will serve all. If not, it will be killed at some point and restored when required.
Of course, the crons of the users will not be too precise, as you have to account for you own cron delays plus execution delays on all the tasks in queue, which will be run sequentially. Not the best approach, unless you somehow disallow crons which run every second or more often than every minute (to be safe). Doing a check on execution time of the crons to kill them if they are over a certain amount of time would be a good idea.
If you can use more than Play:
The better alternative I believe is to use Quartz (see this) to create a future execution when the user creates the job, and reproram it once the execution is over.
There was a discussion on google-groups about it. As far as I remember you must define a job which start every 6 hours and check which backups must be done. So you must remember when the last backup job was finished and make the control yourself. I'm unsure if Quartz can handle such a requirement.
I looked in the source-code (always a good source ;-)) and found a method every, where I think this should be do what you want. How ever I'm unsure if this is a clever design, because if you have 1000 user you will have then 1000 Jobs. I'm unsure if Play was build to handle such a large number of jobs.
[Update] For cron-expressions you should have a look into JobPlugin.scheduleForCRON()
There are several ways to solve this.
If you don't have a really huge load of jobs, I'd just persist them to a table using the required flexibility. Then check all of them every hour (or the lowest interval you support) and run those eligible. Simple.
Or, if you prefer to use cron syntax anyway, just write (export) jobs to a user crontab using a wrapper which calls back to your running app, or starts the job in a standalone process if that's possible.
Need some advice, I'm after a decent process/task manager for Ubuntu.
Basically I have a few scripts/programs which I want to run as long running processes, but I want to shut them down at various periods (say over the weekend or every day for a few hours). During the time that the process needs to be up and crashes, I would like it so that the task scheduler will automatically restart the process.
SO for example, I want to run program X between 9:00-17:00 every day. If the process is still running it should be killed at 17:00. If the process crashes between 9AM and 5PM then the process should be automatically restarted.
Are there any easy to use tools which can do this? I would like to avoid having to manage PID files and having cron jobs which do the start and stop...
Any thing anyone recommends? Any advice appreciated!
Cheers.
I do not know if a tool exists for this, but except if you have many interactive tasks, it really does not a that big issue to manage for a few jobs :
1) You can start your cronjobs whenever you like thanks to the crontab,
2) You can insert a "commit suicide" within these scripts under a time condition for example.
# your script doing things
# Then it commit suicide
if [ your_condition ];then
kill $$
fi
Please note that if you want to allow users login only at certain periods of time, then it's a different question.