I usually work with BASH, but I'm trying to setup a cronjob from a machine and user account that is configured with zsh. When the cronjob runs, the date variable does not contain the date, just the string for the command to return the date.
DATE=$(date +%Y%m%d)
55 15 * * 1-5 scp user#host:/path/to/some/file/$DATE.log /tmp
I've tried using backticks rather than $() around the command, but that did not work either. Is there a special way to do command substitution in zsh?
If this is a line from the user's crontab, then the line to set the variable isn't set by the shell, it's set by cron itself - and as you've discovered, it doesn't do a lot of interpretation on the values!
The values do get set, but set literal values in the process's environment - and environment variables don't get interpreted further after being expanded - so the $(date) stays as $(date), as you've seen.
The easiest thing to do would be:
55 15 * * 1-5 scp user#host:/path/to/file/`date +%Y%m%d`.log /tmp
The backticks `` are another way to interpolate the result of a command, that works in more shells than $(command) (though it is less flexible, since you can't nest it).
I don't think the cron way of setting up environment variables supports the $() stuff. It doesn't even support variable substitutions.
I think you'll either have to put the date calculation into the cron command:
55 15 * * 1-5 scp user#host:/path/to/some/file/$(date +%Y%m%d).log /tmp
or create another script which sets the variable then calls scp with it (and of course you call that script from within your crontab).
Related
I want to know the use of . ./.profile whenever we execute cron jobs. I have seen many scripts having this included. The question is, what is the use and what if I don't add it?
Example:
00 1-22 * * 1-5 . ./.profile ; /global/u1/sie/rox/Scripts/Calls.ksh >/dev/null 2>&1
. somefile is the POSIX-compliant equivalent to the bash builtin source: Running source somefile in bash, or . somefile in any POSIX-compliant shell, executes every command inside that script in that existing shell.
In terms of why this is useful in a crontab: cron runs with a very minimal environment -- it may not even have a PATH set, and is unlikely to have many other facilities. If your scripts depend on environment variables being present, it can be necessary to either specify them in the crontab or to source in (that is, execute in the existing shell) a script which defines them.
That said, I advise against this idiom:
.profile is used by login sessions -- sessions with a user interacting with the shell in real-time -- and folks intending to customize their interactive session's behavior are liable to make modifications without keeping scheduled jobs in mind.
It's not obvious by reading your crontab which environment variables ~/.profile will or won't set, and thus difficult to reason about the state of the environment.
Instead, you should set environment variables at the top of your crontab:
PATH=/bin:/usr/bin:/usr/local/bin
VARNAME=VALUE
# ...etc...
0 1-22 * * 1-5 /global/u1/sie/rox/Scripts/Calls.ksh >/dev/null 2>&1
The profile files are the shell profiles, you can add code to it that will run as soon as the shell start up, ./profile is the profile file for Ksh and Bourne, /.bash_profile is for bash /.login is for Tcsh and Csh.
When a script calls the profile it's because it needs something from it, i.e $path variables or even specific commands that it might not have access to. In this case, since cron doesn't have access to much since it runs in a minimal enviroment that script will pull the .profile because it depends on something that's in there.
More info here
and here
I have my php script file in /var/www/html/dbsync/index.php. When cd /var/www/html/dbsync/ and run php index.php it works perfectly.
I want to call PHP file through sh file, the location of SH file is as below
/var/www/html/dbsync/dbsync.sh
This is the content of the dbsync.sh file is:
/usr/bin/php /var/www/html/dbsync/index.php >> /var/www/html/dbsync/myscript.log 2>&1 -q -f
When I cd /var/www/html/dbsync/ and run ./dbsync.sh it works perfectly as well.
Now if I set up crontab as below:
1 * * * * /var/www/html/dbsync/dbsync.sh /var/www/html/dbsync
However, this crontab is not working as expected.
What can be wrong?
As seen in comments, the problem is that you are not defining what program should be used to execute the script. Take into account that a cronjob is executed in a tiny environment; there, not much can be assumed. This is why we define full paths, etc.
So you need to say something like:
1 * * * * /bin/sh /var/www/html/dbsync/dbsync.sh /var/www/html/dbsync
# ^^^^^^^
/bin/sh being the binary you want to use to execute the script.
Otherwise, you can set execution permissions to the script and add a shell-script header telling it what interpreter to use:
#!/bin/sh
If you do this, adding the path of the binary is not necessary.
From Troubleshooting common issues with cron jobs:
Using relative paths. If your cron job is executing a script of some
kind, you must be sure to use only absolute paths inside that script.
For example, if your script is located at /path/to/script.phpand
you're trying to open a file called file.php in the same directory,
you cannot use a relative path such as fopen(file.php). The file must
be called from its absolute path, like this: fopen(/path/to/file.php).
This is because cron jobs do not necessarily run from the directory in
which the script is located, so all paths must be called specifically.
Also, I understand you want to run this every minute. If so, 1 * * * * won't do. Intead, it will run at every 1st minute past every hour. So if you want to run it every minute, say * * * * *.
It is important to understand "login shell" and "interactive shell" what they means.
login shell: is briefly when you sign in with ssh session and get a terminal window where you can enter shell commands. After login the system executes some files(.bashrc) and sets some environment variables such as the PATH variable for you.
interactive shell :After login on a system, you can startup manually shell terminal(s). The system executes some profile file assigned to your account (.bash_profile, .bash_login,.profile). This files also sets some environment variables and initialize PATH variable for your manually opened shell session.
By OS started shell scripts and cron jobs does not fit in above mentioned way for starting a shell. Therefore no any system scripts(.bashrc) or user profiles are executed. This means our PATH variable is not initialized. Shell commands could not found because PATH variable does not point to right places.
This explains why your script runs successfully if you start it manually but fails when you start it via crontab.
Solution-1:
Use absolute path of every shell command instead of only the command name used in your script file(s).
instead of "awk" use "/usr/bin/awk"
instead of "sed" use "/bin/sed"
Solution-2: Initialize environment variables and especially the PATH variable before executing shell scripts!
method 1, add this header in your dbsync.sh:
#!/bin/bash -l
method 2, add bash -l in your cron file:
1 * * * * bash -l /var/www/html/dbsync/dbsync.sh /var/www/html/dbsync
I have been "toying" around with this for some time now, for most issues i just worked around, but now i need this solved.
Why do basically all my cron jobs dont work with "If tests"
lets take this one
if [ "$line" == "downloads.php" ]
works absolutely fine when i run it in the shell, when i start it as cron job it just never works. The workaround
if echo "$line" | grep -q "downloads.php"
works both ways. Why is that? For the first one the [ ] basically stand for "test", and the second one well, its just a grep. But why are the "tests" not working in my cron jobs? (with or without redirection to >null)
i currently need this one in a cron job, and now i don't really know how to work around, or basically i just finally want to understand how to solve this, what I am doing wrong.
while [[ "${ofile: -1}" != "_" ]]
this one just produces an error "87: Bad substitution", do until first character is "_"
i managed to overcome all issues with cron, from full paths, to environment, this one is still a puzzle for me. any help is appreciated.
It sounds like you are running the script with bash, while cron is running it under some other shell; thus, all of the bash extensions you're using are failing. Make sure the shebang on your script (i.e. the first line) requests bash (#!/bin/bash), and that the cron entry runs it directly rather than specifying a shell (e.g. 0 0 * * * /path/to/script NOT 0 0 * * * /bin/sh /path/to/script).
EDIT: there are several different ways of controlling which shell will be used to interpret a script, with a definite precedence order:
If you run the script with an explicit shell (e.g. /bin/sh /path/to/script), the shell you told to run it will be used, and any shebang will be ignored.
If you run the script directly (e.g. /path/to/script or ./script, or place it in your PATH and run it as script), the system will use the shebang to determine which shell (or other interpreter) to run it with.
If you run it directly and there's no shebang, the program you're running it from (e.g. bash, sh, or crond) might choose to do something else. In this situation, bash will run the script with bash. I'm not sure what crond will do, and it might even depend on which version it is.
In general, using a proper shebang and running the script directly is the best way to go; the script should "know" what the proper interpreter is, and that should be respected. For example, if you write a script in portable shell code (with a #!/bin/sh shebang), and then later need to use some bash-only features, you can simply change the shebang and not have to track down all the places it's run from.
Specifying the shell explicitly should be reserved for cases where the shebang is wrong or missing (in which case the better solution is to fix the script), or you don't have execute permission (again, fix the script). The third option is an unreliable fallback, and should be avoided whenever possible.
P.s. If I'm reading your last comment (above) correctly, you want to test whether $line contains "downloads.php", not whether it equals that; but the [ x == y] comparison tests for equality, not containment. To test for containment, use the bash-only [[ string == pattern ]] form:
if [[ "$line" == *"downloads.php"* ]]
OS: Amazon Linux
I have a Ruby script that connects to a site, then it searches with an XPath request for a div block where is the stats counter I want to parse.
Then it compares the number from the site with the current value in the database, if the number has increased it sends me an email.
The problem is that, then I run the script from the current directory it works.
The script parses the block of text which contains a value.
I extract the value with Regex like this (/\d/)
...
But when it the script executes by crontab it gets some strange value like
...041704300440043504330438044104420440043804400430432043004304304304304304404370430432043004420435043043504390447043504400435043704320430044804430430...
I don't know how to debug it because, when I run the script manually it works, but fails with strange value when executed by crontab.
The text in the site is russian, encoded with Windows-1251.
Maybe there is something wrong with that.
I have set # encoding: utf-8, in the .rb file.
That could be an environment problem, which could include bad paths, etc. You can compare your ENV from the command-line to the environment when launched by crontab.
Try:
ruby -rpp -e 'pp ENV' > /tmp/crontab_env.out
from crontab, then:
ruby -rpp -e 'pp ENV' > /tmp/cmd_env.out
from the command-line, then:
vimdiff /tmp/*env.out
or use a regular editor.
If you're using RVM, note that it is typically only available to interactive shells. There's a whole section in the RVM manual dedicated to this topic: RVM: Ruby Version Manager - Using Cron with RVM
It could be that this is simply a problem of the wrong Ruby version, including its Gems, being used. Try removing the hashbang line in your script, and calling it like this in your crontab:
1 0 * * * /usr/local/rvm/bin/ruby-1.9.3-p362 /path/to/script.rb
This should make sure the proper environment is loaded with the Ruby binary.
If the actual problem is that RVM isn't even available for non-interactive scripts, you could also go one step further and do what your shell does when it's loading RVM—scroll to the right, this is a big line:
1 0 * * * /bin/bash -l -c 'source "$HOME/.rvm/scripts/rvm" && rvm use 1.9.3-p362 && ruby /path/to/script.rb
Problems with cron jobs are often caused by having the wrong environment. The script probably depends on an environment variable that's set when you start an interactive shell (through ~/.profile, ~/.bashrc or similar), but not when your program is started directly, by cron.
Get a list of environment variables and their current values by typing env. Add a cron job that simply runs env. Compare the outputs and chip away until you find the culprit.
I'd say LANG and friends are a good place to start. Get a list of language and encoding-related environment variables by typing locale.
If I run a script from /home/<user>/<dir>/script.sh, as root, the cron works pretty well. But If I run the script from /root/<dir>/script.sh (as root, again), the cron does not seem to work.
Having run afoul of various default $PATHs in the past when using 'cron', I always spell in full the absolute $PATH for each executable file and each target file. I always assume that 'cron' has NO $PATH set and has NO current-working-directory.
In other words don't use a command like
"myprocess abc*.txt"
but do it in full like
"/usr/localbin/myprocess /home/jvs/abc*.txt".
Alternatively, create a bash script which does the job, and call that bash script with a full absolute path, such as
"/usr/local/bin/myprocess_abc_txts".
If you need to have some flexibility in the script, use environment variables which are set specifically within the bash script you call with 'cron'.
I think you need to add a little more information. I'd guess it is a permissions thing though. Add the permissions of the file, the directories, and the line in your crontab so we can help. Also, if you are putting this in /root, are you running this in root's crontab?
Remember the environment - especially when run by cron rather than by root. When cron runs something, you probably don't have anything much set of your environment, unlike when you run a command via at. It is also not clear what your current directory will be. So, for commands that will be run by cron, use a script (as you're already doing) and make sure it sets enough of the environment for it to run. And make sure your environment setting code is not interactive!
On my machines, I have a mechanism such that the cron entry reads (for example):
23 1 * * 1-5 /usr/bin/ksh /work1/jleffler/bin/Cron/weekday
The weekday script in the Cron directory is a link to a standard script that first sets the environment and then runs the command /work1/jleffler/bin/weekday (in this case - it uses the name of the command to determine what to run).
The actual script in the Cron directory is:
: "$Id: runcron.sh,v 2.1 2001/02/27 00:53:22 jleffler Exp $"
#
# Commands to be performed by Cron (no debugging options)
# Set environment -- not done by cron (usually switches HOME)
. $HOME/.cronfile
base=`basename $0`
cmd=${REAL_HOME:-/real/home}/bin/$base
if [ ! -x $cmd ]
then cmd=${HOME}/bin/$base
fi
exec $cmd ${#:+"$#"}
I've been using it a while now - this version since 2001 - and it works a treat for me. I'm using a basic (Sun Solaris 10) implementation of cron; there may be new features in new versions of cron on other platforms to make some of this unnecessary. (The $REAL_HOME stuff is a weirdness of mine; pretend it says $HOME - though that makes some of the script unnecessary for you.) The .cronfile is responsible for the environment setting - it does quite a lot, but that's my problem, not yours.
It could be because you're looking for relative directories/files in the script which are located when running it from /home/ but not from /root, because /root is not in /home/root nor would it look like a users homefolder in /home/
Can you check and see if it is looking for relative files, or post the script?
On another note, why don't you just set it to run from a user's homefolder then?
Another way to run sh script is place your bash script in /usr/bin directory and simply run command bash yourscript.sh without adding /usr/bin/ directory