I've got about a dozen servers that each have crontabs with anywhere from 20-50 crontab entries. My single most common cause of a process failure is someone commenting out jobs in cron during a fix or patch and then forgetting to uncomment the jobs.
I'd like to do two things to solve this:
Start using our schedule suppression process that allows users to suppress schedules without actually touching crontab. Nothing magical - just touch a file in a directory dedicated to the process. The process checks that directory on start-up.
Implement a process that will send out alerts if crontab doesn't match its backup or current version in svn.
Can anyone recommend an existing solution for #2 (alert when crontab changes)?
In this case i would suggest to compare hashvalues of the file you want to have and the actual file.
Just write a little bashscript that sends out a emailnotification or creates a notification file or whatever you want and let this script be run automatically every x seconds / minutes / hours.
A possible script could be
if [[ $(md5sum path/to/crontab.backup | cut -d' ' -f1) == $(md5sum /etc/crontab | cut -d' ' -f1) ]]
then
# send your notification
fi
This is a very simple solution to check if a file was changed since the last backup was made.
Related
Multiple scripts are running in my Linux server which are generating huge data and I realise that it will eat all my 500GB of storage size in next 2-5 days and scripts require 10 more days to finish the process means they need more space. So most likely I am going to have a space issue problem and I will have to restart the entire process again.
Process is like this -
script1.sh content is like below
"calling an api" > /tmp/output1.txt
script2.sh content is like below
"calling an api" > /tmp/output2.txt
Executed like this -
nohup ./script1.sh & ### this create file in /tmp/output1.txt
nohup ./script2.sh & ### this create file in /tmp/output2.txt
My understand initially was, if I will follow below steps, it should work --
when scripts are running with nohup in background execute this command -
mv /tmp/output1.txt /tmp/output1.txt_bkp; touch /tmp/output1.txt
And then transfer this file /tmp/output1.txt_bkp to another server via ftp and remove it after that to get space on server and script will keep on writing in /tmp/output1.txt file.
But this assumption was wrong and script is keep on writing in /tmp/output1.txt_bkp file. I think script is writing based on inode number that is why it is keep on writing in old file.
Now the question is how to avoid space issue without killing/restart scripts?
Essentially what you're trying to do is pull a file out from under a script that's actively writing into it. I'm not sure how nohup would let you do that.
May I suggest a different approach?
Why don't you move an x number of lines from your /tmp/output[x].txt to /tmp/output[x].txt_bkp? You can do so without much trouble while your script is running and dumping stuff into /tmp/output[x].txt. That way you can free up space by shrinking your output[x] files.
Try this as a test. Open 2 terminals (or use screen) to your Linux box. Make sure both are in the same directory. Run this command in one of your terminals:
for line in `seq 1 2000000`; do echo $line >> output1.txt; done
And then run this command in the other before the first one finishes:
head -1000 output1.txt > output1.txt_bkp && sed -i '1,+999d' output1.txt
Here is what's going to happen. The first command will start producing a file that looks like this:
1
2
3
...
2000000
The second command will chop off the first 1000 lines of output1.txt and put them into output1.txt_bkp and it will do so WHILE the file is being generated.
Afterwards, look inside output1.txt and output1.txt_bkp, you will see that the former looks like this:
1001
1002
1003
1004
...
2000000
While the latter will have the first 1000 lines. You can do the same exact thing with your logs.
A word of caution: Based on your description, your box is under a heavy load from all that dumping. This may negatively impact the process outlined above.
I am currently working on project to automate a manual task in my office. We have a process that we have to re-trigger some of our ID's when they fall in repair. As part of the process, we have to extract those ID's from a oracle DB table and then put in a file on our Linux server and run the command like this-
Example file:
$cat /task/abc_YYYYMMDD_1.txt
23456
45678
...and so on
cat abc_YYYYMMDD_1.txt | scripttoprocess -args
I am using an existing java based code called 'scripttoprocess'. I can't see what's inside this code as it is encrypted( it seems) in my script. I simply go to the location where my files are present present and then use it like this:
cd /export/incoming/task
for i in `ls abc_YYYYMMDD*.txt`;do
cat $i | scripttoprocess -args
if [ $? -eq 0];then
mv $i /export/incoming/HIST/
fi
done
scripttoprocess is and existing script. I am just calling it in my own script. My script is running continuously in a loop in the background. It simply searches for abc_YYYYMMDD_1.txt file in /task directory and if it detects such a file then it starts processing the file. But I have noticed that my script starts processing the file well before it is fully written and sometime moves the file to HIST without fully processing it.
How can handle this situation. I want to be fully sure that file is completely written before I start processing it. Secondly, Is there any way to take control of the file like preparing a control file which contains list of the files which are present in the /task directory. And then I can cat this control file and pick up file names from inside of it ? Your guidance will be much appreciated.
I used
iwatch -e close_write -c "/usr/bin/pdflatex -interaction batchmode %f" document.tex
To run a command (Latex to PDF conversion) when a file (document.tex) is closed after writing to it, which you could do as well.
However, there is a caveat: This was only meant to catch manual edits to the file and failure was not critical. Therefore, this ignores the case that immediately after closing, it is opened and written again. Ask yourself if that is good enough for you.
I agree with #TenG, normally you shouldn't move a file until it is fully written. If you know for sure that the file is finished (like a file from yesterday) then you can move it safely, otherwise you can process it, but not move it. You can for example process a part of it and remember the number of processed rows so that you don't restart from scratch next time.
If you really really want to work with files that are "in progress", sometimes tail -F works for this case, but then your bash script is an ongoing process as well, not a job, and you have to manage it.
You can also check if a file is currently open (and thus unfinished) using lsof (see https://superuser.com/questions/97844/how-can-i-determine-what-process-has-a-file-open-in-linux ; check if file is open with lsof ).
Change the process, that extracts the ID's from the oracle DB table.
You can use the mv as commented by #TenG, or put something special in the file that shows the work is done:
#!/bin/bash
source file_that_runs_sqlcommands_with_credentials
output=$(your_sql_function "select * from repairjobs")
# Something more for removing them from the table and check the number of deleted records
printf "%s\nFinished\n" "${output}" >> /task/abc_YYYYMMDD_1.txt
or
#!/bin/bash
source file_that_runs_sqlcommands_with_credentials
output=$(your_sql_function "select * from repairjobs union select 'EOF' from dual")
# Something more for removing them from the table and check the number of deleted records
printf "%s\n" "${output}" >> /task/abc_YYYYMMDD_1.txt
This is a tomcat7 installation with the default logging configuration, the catalina.out is only being rolled out when we restart the server. As it is prod server we can not restart it very often. We have a huge number of entries going to that file which causes our catalina.out to grow very high in few days until it consumes the whole diskspace.
As we don't want to change the logging configuration as it is puppetized and we would need to create devops tickets and all that slow stuff, I wrote a bash script that is being run every 5 min via crontab that will cut the log file by half when a limit is reached, the script is like the following:
if [ $catalinaSize -gt $catalinaThreshold ]; then
middle=$(wc -l $catalinaLoc | awk '{ print $1 }')
middle=$(( $middle / 2 ))
sed -i -e 1,${middle}d $catalinaLoc
echo "+++ catalina.out was cut by half"
Basically this script checks the current size of the file and compares it to a threshold value, then it uses wc and awk to retrieve the number of lines in that file so it can use then sed for cutting the file by half.
I tested the script in other environments and it worked. The problem is that after some successful runs for several days in production, suddenly the catalina.out is not getting any log entries from tomcat since some days ago.
The explanation I think about is that Tomcat is not able to write into that file anymore because of the cut by half operation.
Is it possible to know what is preventing Tomcat to write into that file?
I suspect it is sed -i doing the damage: behind the scenes, it writes the output stream to a temp file, then moves the temp file to the original name. I suspect the file handle held by catalina no longer points to any file.
You'll have to find a way to actually edit the file, not replace it. This might be a valid replacement for sed:
printf "%s\n" "1,${middle}d" "wq" | ed "$catalinaLoc"
Tangentially, an easier way to get the number of lines:
middle=$(( $(wc -l < "$catalinaLoc") / 2 ))
When you redirect to wc, it no longer prints out the filename.
I need to stop the overlapping of cron jobs for example:If a cron job is scheduled at morning 2 o clock for DB backup and other cron job is scheduled at morning 7 o clock for DB backup again.So i need to stop the 7 o clock scheduled cron job if the DB backup for 2 o clock is not completed.
This is what flock is for. From man flock:
...
The third form is convenient inside shell scripts, and is usually used
the following manner:
(
flock -n 9 || exit 1
# ... commands executed under lock ...
) 9>/var/lock/mylockfile
...
In this case, flock will try to get a lock on fd 9 for /var/lock/myfile. If it can't (because the earlier job is still running), it will exit.
You need to put it in a script.
The most straightforward way to do this would be to have your 2am and 7am tasks coordinate. So for instance, the DB backup script could create a lock file when it executes, and check for the existence of this file when it starts up. If the file exists it could simply exit without doing any work (or sleep for a bit, or whatever you decide).
Doing this in a completely task-agnostic way might be possible, but it's going to be trickier and unnecessary in this particular case.
Assuming you are on a Unix system: The easiest way to do this is
create a file and write the cron job's PID to it. If the file can't be
created using open(2) with O_CREAT|O_EXCL flags, it already
exists, thus the previous job is still running or has crashed without
cleaning up. The PID can be used to determine whether the job is still
running.
In your case, the safest approach is probably for the cron job to die
if the PID file is found on startup.
Since you put the perl tag to the question, I suggest to have a look
at the File::Pid and Proc::PID::File modules.
You just need something that you can check to see if the first task has finished running, and this check should be done right before the job runs again.
I'd implement this with a database (if available) or a registery key or even a text file. So your task would look like this:
read job_flag
if job_flag == 0
step job_flag = 1
run job
step job_flag = 0
else
stop job
end
Why not just do this in a single cron job?
/path/to/job1 && /path/to/job2
Here is my solution:
$ crontab -l
* * * * * cd /patch/to/script; . /patch/to/.profile; overlap.pl process.sh -t Device || process.sh -t Device
$
$
$ cat overlap.pl
#!/usr/bin/env perl
use warnings;
use strict;
my $cmd = join(' ',#ARGV);
my #result = `pgrep -f \"$cmd\"`;
scalar #result > 2 ? exit 0 : exit 1;
EXPLANATION:
this job runs script every minute to check if it works (* * * * *)
cron job is complex, fisrt change directory, load profile so I need to launch script directly from this job not from script that is checking if overlapped
and then check if this particular process with the same parameters is already running (overlap.pl with exact command that I want to check)
if "no", return 0, if "yes", return >0
then there is logical "or" checking result and launch desired script or not
Pay attention to the array length in overlap.pl, you can check it by yourself by doing `ps -elf | grep \"$cmd\" >> /tmp/croncheck`;
You can do that just using cron. You are going to have to add some logic to your script or code to check if the other one is running or not. A total hack would be to have your script check for the exists of a particular file. If it does not exist then you should create this file, perform the backup, then remove it. If the file does exist, do nothing.
That is a hack and only really works because your jobs are spaced so far apart.
I want to programatically change the hour which certain cron jobs execute.
I'm not sure whether I'm going about this the right way, but here's the plan.
Jobs which are subject to change will have a comment on the end of the line "#change-enabled".
15 5 7,14,21,28 * * /path/to/executable1 #change-enabled
15 * * * * /path/to/executable2
45 5 */2 * * /path/to/executable3 #change-enabled
Then I want to pipe the output from "crontab -l" through sed to catch and alter jobs with that comment on the end, and pipe the results of that through "crontab -".
// Edit
My gut tells me to stay away from doing this as root because the new hour is determined using data from a 3rd party. (Google Analytics Data Export API) So /etc/cron.d/ is something I'm going to stay away from for this application.
Here is the command which has been working with all of my current cron jobs during debugging. I've been using "date +%l" in place of "/home/user/slow-hour" and leaving off the "| cron -" to see what my gets printed.
crontab -l | sed -e "s/^\([^ ]*\) [0-9]* \(.*#change-enabled\)/\1`/home/user/slow-hour` \2/" | crontab -
The contents of ~/slow-hour will be a script which fetches hourly data for website profiles at Google Analytics and looks for the slowest hour. In the event of any error whatsoever that script will return 1AM, mail me a notice and then bail.
In order the reduce the likelihood of a days execution being skipped due to the slow hour being earlier than the current hour, the automated cron rescheduler will be set to run at midnight and the earliest hour ~/slow-hour will return will be 1AM.
Based on my manual cron job adjustments in the past, I don't anticipate excluding midnight from being the slowest hour being a problem in my case.
crontab -l | sed '/change-enabled$/s/pat/repl/' | crontab -
Assuming you want to set the hour to 8, one command would be
sed -e 's/\([0-9]*\) [0-9]* \(.*#change-enabled\)/\1 8 \2/'
You can do that, but there's a better way (provided, you've got root access).
Instead of altering crontab, split jobs into related groups. For each group put a new file in /etc/cron.d/{jobname}. This way you can just regenerate one script without problems - no sed magic needed. Split everything that can change into separate files, so that you just write a new file in place of the old one and don't mess with the entries that don't need changing.
The only difference is the syntax. Instead of:
<time> <command>
you have to put there:
<time> <user> <command>
so that the job is run with the correct rights.
Example - instead of 'crontab -l | sed -e 's/....somejob.../.../ | crontab ..' just run 'echo .... > /etc/cron.d/somejob'
crontab -l | sed -re '/# *change-enabled *$/s/^([^ ]+) [^ ]+/\1 new_hour/' | crontab -