I'm having an issue with a simple shell script that merely moves a file from one location to another. It takes the source & target paths & names as parameters. Code is as follows...
#!/bin/sh
if [ ! -f $1 ]; then
exit 1
else
ls -l $1
mv -v $1 $2
X=$?
echo 'Move completion code = ' $X
ls -l $2
exit $X
fi
This script is called multiple times in a sequential job schedule, loading data to (replacing) the same file. Recently seeing instances where the job after the move script runs with data from the file that existed before - indicating that it was not overwritten - yet the messages in the move script show a successful move - and checking the file itself shows it with the expected data. Any ideas? Could the script be completing before the mv command has fully completed? I've put the completion code message in to check for this, I've also tried sleeping for 5 sec. What am I missing?
Thanks!
Update...
Here's the output after adding the md5sum checks....
-rw-r--r--. 1 appworx dba 22 Aug 10 12:15 UP.dat
Before move - file 1
9bf394b473cf389d704357e4da46fcfa UP.dat
`UP.dat' -> `ids.dat'
removed `UP.dat'
After move - file 2
9bf394b473cf389d704357e4da46fcfa
Move completion code = 0
-rw-r--r--. 1 appworx dba 22 Aug 10 12:15 load_ids.dat
Related
Linux: Red Hat Enterprise Linux Server release 6.5 (Santiago)
BASH: GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
One of the VM that I just provisioned is still waiting for the DNS to update it's node entries and one the node has a NAS which is mounted on at /project_support folder. It takes around 5-10 minutes for /project_support folder to be accessible, yea bad I know.
For my automation not break due to this issue and I didn't want to add a constant sleep NNN, I used until loop so that until the mount is mounted and /project_support folder is available, I wanted to sleep for 5 seconds and check for folder existence before proceeding further in the automation script/logic.
Why the first until loop is continuing and not exiting on the condition check when cd command to a given folder is successful aka exit status 0? The second until is working as expected.
For the 2nd until loop, even if I create the folder (from a second terminal, while until loop is running), it still doesn't break the until loop.
[root#nukaserver01 ~]# cd /project_support; echo $?
0
[root#nukaserver01 project_support]# pwd
/project_support
[root#nukaserver01 project_support]# until [ `cd /project_support` ]; do echo '-- sleep 5'; sleep 5; done
-- sleep 5
-- sleep 5
-- sleep 5
^C
[root#nukaserver01 project_support]# until [ `cd /project_support-I-DONT-EXIST-AT-ALL` ]; do echo '-- sleep 5'; sleep 5; done
-bash: cd: /project_support-I-DONT-EXIST-AT-ALL: No such file or directory
-- sleep 5
-bash: cd: /project_support-I-DONT-EXIST-AT-ALL: No such file or directory
-- sleep 5
^C
[root#nukaserver01 project_support]#
Why is this issue getting fixed when I do: ?
until [ `cd /project_support && pwd` ]; do ...; ...; done
until [ `cd /project_support` ]
The backticks capture cd's output, which is an empty string whether or not the named directory exists (error messages are printed to stderr and aren't captured). The square brackets then test if that string is empty, which it always is.
Unnecessary square brackets is a common shell scripting malady. Get rid of them and the backticks.
until cd /project_support; do
...
done
Note that since cd is no longer being executed in a sub-shell it will actually take effect when it succeeds. To prevent that you can add an explicit sub-shell:
until (cd /project_support); do
...
done
Or if you're just waiting until the directory exists, check that directly:
until [[ -d /project_support ]]; do
...
done
You are returning and testing an empty string, which always evaluates as false.
until [ "" ] ; do sleep 1; date; done
Tue, Nov 06, 2018 12:03:17 PM
Tue, Nov 06, 2018 12:03:18 PM
Tue, Nov 06, 2018 12:03:19 PM
. . .
By adding the pwd it returns the directory name/path when the cd succeeds, which as a string eval's as true.
try this -
until [[ -d /project_support ]] # ...
that will test for the existence of a directory.
I have a cron job that copies its log file daily to my home folder.
Everyday it overrides the existing file in the destination folder, which is expected. I want to preserve the log from previous dates so that next time it copies the file to destination folder, it preserves the files from previous dates.
How do I do that?
The best way to manage cron logs is to have a wrapper around each job. The wrapper could do these things, at the minimum:
initialize environment
redirect stdout and stderr to log
run the job
perform checks to see if job succeeded or not
send notifications if necessary
clean up logs
Here is a bare bones version of a cron wrapper:
#!/bin/bash
log_dir=/tmp/cron_logs/$(date +'%Y%m%d')
mkdir -p "$log_dir" || { echo "Can't create log directory '$log_dir'"; exit 1; }
#
# we write to the same log each time
# this can be enhanced as per needs: one log per execution, one log per job per execution etc.
#
log_file=$log_dir/cron.log
#
# hitherto, both stdout and stderr end up in the log file
#
exec 2>&1 1>>"$log_file"
#
# Run the environment setup that is shared across all jobs.
# This can set up things like PATH etc.
#
# Note: it is not a good practice to source in .profile or .bashrc here
#
source /path/to/setup_env.sh
#
# run the job
#
echo "$(date): starting cron, command=[$*]"
"$#"
echo "$(date): cron ended, exit code is $?"
Your cron command line would look like:
/path/to/cron_wrapper command ...
Once this is in place, we can have another job called cron_log_cleaner which can remove older logs. It's not a bad idea to call the log cleaner from the cron wrapper itself, at the end.
An example:
# run the cron job from command line
cron_wrapper 'echo step 1; sleep 5; echo step 2; sleep 10'
# inspect the log
cat /tmp/cron_logs/20170120/cron.log
The log would contain this after running the wrapped cron job:
Fri Jan 20 04:35:10 UTC 2017: starting cron, command=[echo step 1; sleep 5; echo step 2; sleep 10]
step 1
step 2
Fri Jan 20 04:35:25 UTC 2017: cron ended, exit code is 0
Insert
`date +%F`
to your cp command, like this:
cp /path/src_file /path/dst_file_`date +%F`
so it will copy src_file to dst_file_2017-01-20
upd:
As #tripleee noticed, % character should be escaped in cron, so your cron job will look like this:
0 3 * * * cp /path/src_file /path/dst_file_`date +\%F`
I have a script that is running every 2 minutes by looking into a folder and checking if new files have been delivered
the problem is that sometimes the script starts during the files are copied (the files being quite big) so an email is sent to the customer with "file x is empty" but of course the file is good. the correct message email is received afterwards.
for avoiding the overlapping between different processing I have set up a CRON_RUN file like this. If there are files that are currently processing then my script exits. when the program finishes, the CRON_RUN file is removed for allowing to the next process to be started if exists of course
Now the processing is not overlapping any more but I have observed that
if I deliver a file in the input and during the copy I start the processing the program identifies an EMPTY file so it is still not as expected.
is there a command like?:
if copy process is in run or checksum file is not completed then
exit
else
do my program
fi
my current implementation so far is:
#!/bin/bash
MYDIR=`dirname $0`
ext_dir="/server/oracle/apps/delivery"
if test -f $ext_dir/CRON_RUN
then
echo $0 CRON RUN >&2
exit 1
fi
touch $ext_dir/CRON_RUN
export CRON_RUN=CRON_RUN
for......
my program....
export CRON_RUN=""
rm -f $ext_dir/CRON_RUN
could you please tell me if I understood correctly how to use flock command?
#!/bin/bash
MYDIR=`dirname $0`
ext_dir="/server/oracle/apps/delivery"
if test -f $ext_dir/CRON_RUN
then
echo $0 CRON RUN >&2
exit 1
fi
touch $ext_dir/CRON_RUN
export CRON_RUN=CRON_RUN
#running the program for each input folder
#inside the input some files might be in copying
#if a process copying is in progress program should exit
# if not then the load-input script should start
for input_folder in electro food comp rof
do
(
# Wait for lock on /server/oracle/apps/delivery/$input_folder/.load-input.exclusivelock(fd 200) for 10 seconds
flock -x -w 10 200 || exit 1
# Do stuff
$MYDIR/load-input $input_folder $mylog > $mylog 2>&1
) 200>/server/oracle/apps/delivery/$input_folder/.load-input.exclusivelock
done
export CRON_RUN=""
rm -f $ext_dir/CRON_RUN
My crontab entry as "ins" user is this:
* * * * * /usr/bin/ksh /apps/swbkp/swbkp.sh
The script is:
#! /usr/bin/ksh
. /apps/ins/.profile
cdate=$(date +'%y%m%d')
/apps/omni/bin/swmml -e "backup-node:" >> /apps/swbkp/swerrr1.$cdate
#
if [[ -f /apps/omni/conf/archive.C7M3UAA.500.$cdate ]]
then
mv -f /apps/omni/conf/archive.C7M3UAA.500.$cdate /apps/swbkp/
elif [[ -f /apps/omni/labeir1/dffile/archive.C7M3UAA.500.$cdate ]]
then
mv -f /apps/omni/labeir1/dffile/archive.C7M3UAA.500.$cdate /apps/swbkp/
else
printf "Backup archive File not present to move"
fi >> /apps/swbkp/swerrr1.$cdate
#
Note: /apps/omni/bin/swmml -e "backup-node:" this line simply creates a backup file on my system of type archive.C7M3UA.500.<current date>
2 weird things happening:
the backup file generated is:
-rw-r--r-- 1 root root 165 Aug 28 21:55 /apps/omni/labeir1/dffile/archive.C7M3UAA.500.130828
When getting moved to /apps/swbkp, the timestamp is 1 min before:
-rw-r--r-- 1 root root 165 Aug 28 21:54 archive.C7M3UAA.500.130828
Nothing is getting redirected to /apps/swbkp/swerrr1.$cdate file
-rw-r--r-- 1 ins ins 0 Aug 28 21:24 swerrr1.130828
Whereas when I run the script from terminal, everything works perfect, i.e the file has the same timestamp in backup folder as well as moved folder, and outputs are getting logged in the log file as well.
Kindly help
This may already have been answered, but for anyone looking for help:
You'll need to escape the 'percent' like below when using date/time stamps in crons:
cdate=$(date +'\%y\%m\%d') instead of cdate=$(date +'%y%m%d')
if using this in yaml/ Ansible playbooks, you'll need to double escape or escape the escape like:
$(date +'\\%y\\%m\\%d')
Eventually the cron should look something like
* * * * * script.sh > /tmp/script_$(date +\%y\%m\%d).log 2>&1
Take a look at /apps/ins/.profile and the files it executes - such files usually have a conditional that exits the script early if it's not run in a terminal.
For example:
[ -z "$PS1" ] && return
This may change the behavior of your script (or even skip it, if exit is used in place of return). At the very least, you will miss aliases, possible PATH changes, and other things set up in the .profile script, which will affect if and how your main script is run.
Try to comment the line . /apps/ins/.profile in the script above, and see if it still runs in the terminal. If it does, run it like that from crontab and see if it fixes your problem.
I want to execute a script and make it schedule the next execution. A sample would be:
#!/bin/bash
TMP=/tmp/text.txt
SCRIPT=$(readlink -f $0)
date >>$TMP
at -f $SCRIPT now + 1 minutes >>$TMP 2>&1
echo -e "\n" >>$TMP
A sample execution would do as follows:
First execution OK. Schedules to next minute
Second execution writes OK but doesn't schedule
Resulting output would be:
tue mar 5 14:34:01 CET 2013
job 15 at 2013-03-05 14:35
tue mar 5 14:35:00 CET 2013
job 16 at 2013-03-05 14:36
[now at 2013-03-05 14:38]
atq outputs nothing and I don't see any /var/at/jobs (In fact, ls /var/at* outputs nothing. There is no message in any user in /var/mail/. I'm trying on a CentOS release 5.6 x86_64
Anyone has any hint as to what may be happening?
suspectus, you have hit the point... echo $SCRIPT gives '/bin/bash'... I've manually written the full path and now it works