I'm trying to upload files to Google Cloud Storage using cron in Linux, but it fails.
I have also set path and configuration in my script file as :
PATH=/bin/gsutil/
export BOTO_CONFIG="/home/ashu/.boto"
# rest of script
But still nothing works.
And if you are using default installation provided by Google Cloud - Compute Engine, most probably gsutil is in /snap/bin
PATH=$PATH:/snap/bin
It's a bit safer to do
PATH="$PATH":/bin/gsutil/
so you don't kill access to the usual places like /bin and /usr/bin and etc. You may not use them directly, but scripts you call might!
update: #ComputerDruid rightly points out that quotes keep spaces from causing trouble.
I had the same issue on Ubuntu 20.04 and I found that the simplest solution was to create a softlink between my gsutil installation and the system bin folder like this:
sudo ln -s /snap/bin/gsutil /usr/bin/gsutil
Apart from modifying the PATH, as suggested by pjz, did you try to look at the actual output from gsutil / cron?
Which reason is given for the commend failing? In case you need to catch the output of gsutil,
you can redirect standard output and error (stdout and stderr) to a file, and save it there.
E.g. if you're using Bash, you could redirect the output to gsutil_log.txt by modifying your crontab as:
*/1 * * * * /mypath/myscript.sh >> $HOME/gsutil_log.txt
This will redirect stdout and stderr and append any output to gsutil_log.txt in $HOME for myscript.sh that is called every minute by cron.
If the output is helpful, that should advance you a bit with debugging.
I removed my pip install and used following link for installation :
https://cloud.google.com/storage/docs/gsutil_install#specifications.
Also use of sudo should be avoided for path and export as it can lead to some issues.
PATH=$PATH:/root/gsutil/
export BOTO_CONFIG="/root/.boto"
# rest of script
Above code works well.
you can try using the full path gsutil command to use in crontab
/root/gcloud/gsutil cp ...
Related
I have this crontab #reboot "/home/pi/Desktop/TV Scraper 2.0/run.sh" set up and for whatever reason it doesn't seem to run the bash file on reboot.
Typing "/home/pi/Desktop/TV Scraper 2.0/run.sh" on the terminal actually runs the script, so I know it's correct.
This is what's inside run.sh just in case:
#!/bin/bash
cd "/home/pi/Desktop/TV Scraper 2.0"
node ./app.js
I've also tried using #reboot root sh "/home/pi/Desktop/TV Scraper 2.0/run.sh" as well, but it doesn't work either.
How can I move forward with this? My knowledge of Linux is very limited. All I need is to have some Node and Python3 scripts run on every reboot. On Windows that's such an easy task: I've tried CRON, rc.local and autostart, nothing works.
My guess is that node is not available via cronjob, since its containing directory is not in your PATH environment variable. When you execute the script manually, it's probably available via PATH.
An easy fix for this is to use the full path, which you can get by executing which node. The result should be something like /usr/bin/node. Then you can use that, instead of just node.
For debugging purpose you can also redirect stdout and stderr to a file, so the last line in your script would look like this:
/usr/bin/node ./app.js &>/tmp/cron-debug.log
If that doesn't fix it, i would rename the directory "TV Scraper 2.0" and replace the whitespace characters with something like an underscore. Directory and file names are less likely to cause problems if you avoid whitespaces.
So I'm sure this is written somewhere, and I'm just wording my searches wrong, but I haven't been able to find it.
I write shell scripts. Many of those scripts require superuser permissions. Rather than putting the sudo command inside the script, I want to invoke the entire script from the start with sudo.
So instead of:
#!/bin/bash
# Sample Script 1
sudo mkdir /usr/tempDir
It would be:
#!/bin/bash
# Sample Script 2
mkdir /usr/tempDir
And I would just invoke it like this: sudo ./sampleScript.sh
My problem is that when I write a script, I simlink it to my bin, so I don't need to worry about the path to the script when I execute it.
If I have the file simlinked to my bin, and I run it as just: sampleScript.sh, it is recognized (and will fail). However, if I do: sudo sampleScript.sh, I get a "sudo: sampleScript.sh: command not found" message.
This happens if it's in both /home/myName/bin and /usr/local/bin.
I know there has to be a trick to making this work that I'm missing. And I also know I'm probably just not wording my google searches properly. Just looking for some guidance to figure this part out.
Thanks.
When you call sudo the PATH changes ie it's not using yours. Look here
sudo changes PATH - why?
It does this because it's now running as root, not as your user.
I have a script used for zipping a database and site files, then dumps the output into a backup folder on the server. The script runs fine from the command line, but it will not work through cron.
After much research, I am thinking that cron cannot run it in its current form because it runs in a different environment.
Here is the script, saved as file_name.sh
#!/bin/bash
NOW=$(date +"%Y-%m-%d-%H%M")
FILE="website.com.$NOW.tar"
BACKUP_DIR="/backupfolder"
WWW_DIR="/var/www/website/"
DB_USER="dbuser"
DB_PASS="dbpw"
DB_NAME="dbname"
DB_FILE="website.com.$NOW.sql"
WWW_TRANSFORM='s,^var/www/website,www,'
DB_TRANSFORM='s,^backupfolder,database,'
tar -cvf $BACKUP_DIR/$FILE --transform $WWW_TRANSFORM $WWW_DIR
mysqldump -u$DB_USER -p$DB_PASS $DB_NAME > $BACKUP_DIR/$DB_FILE
tar --append --file=$BACKUP_DIR/$FILE --transform $DB_TRANSFORM $BACKUP_DIR/$DB_FILE
rm $BACKUP_DIR/$DB_FILE
gzip -9 $BACKUP_DIR/$FILE
I currently have the script stored in /usr/local/scripts/
Is there something wrong with the above code that does not allow it to run through cron?
Which crontab should it go in? crontab -e from terminal, or /etc/crontab? They are two different files.
Several things come to mind: first, one of the most common problems with cron jobs is that generally crond runs things with a very minimal PATH (usually just /usr/bin:/bin), so if the script uses any commands from some other binaries directory, it'll fail. Where is mysqldump on your system (run which mysqldump if you aren't sure)? If this is the problem, adding PATH=/usr/local/bin:/usr/bin:/bin (or whatever's appropriate in your case) at the beginning of your script should fix it. Alternately, you can set PATH in the crontab file (put this line before the entry that runs your script).
If that's not the problem, my next step would be to capture the script's output, with something like:
1 1 * * * /usr/local/scripts/file_name.sh >/tmp/file_name.log 2>&1
... and see if the output is informative. BTW, as #tripleee mentioned, the format of your cron entry is suitable for the files crontab -e edits, but not for /etc/crontab. The /etc version has an additional field specifying which user to run the job as, e.g.
1 1 * * * eric /usr/local/scripts/file_name.sh >/tmp/file_name.log 2>&1
Best practice is to always use crontab -e (the resultant files are usually in /var/spool/cron/) and this works on every unix and linux platform I ever worked on.
Other common issues with cron execution are missing environment variables. Any environment variables set in .bash_profile (or .profile if you use korn shell) will not necessarily be present in the cron environment. This can be overcome by including them in your script.
As Gordon said, paths are another suspect. You can always full path you executables in your script (eg /bin/mysqldump). Some of the more cynical of us do this anyway to make sure we are executing what we intended as apposed to some other file of the same name in the current path.
I can only guess at your specific problem since you fixed it by creating /scripts, that perhaps the permissions on /usr/local/scripts directory did not allow execution by the cron user?
I have had to remove the extension (.sh) for cron to run in some instances.
So I fixed it. Not sure what the problem was, but this worked for me.
I originally had the scripts located in /usr/local/scripts/
I created a new directory here - /scripts/ and moved the scripts there. The new crontab -e command looked like this:
1 1 * * * bash /scripts/file_name.sh
Works perfectly. Again, I am not sure what the issue was before, but it works now.
I have tried exporting my paths and variables and crontab still will not run my script. I'm sure I am doing something wrong.
I have a shell script which runs a jar file. This is not working correctly.
After reading around I have read this is commonly due to incorrect paths due to cron running via its own shell instance and therefore does not have the same preferences setup as my profile does.
Here is what my script looks like today after several modifications:
#!/bin/bash --
. /root/.bash_profile
/usr/bin/java -jar Pharmagistics_auto.jar -o
...
those are the most important pieces of the script, the rest are straightforward shell based.
Can someone tell me what I am doing wrong?
Try specifying the full path to the jar file:
/usr/bin/java -jar /path/to/Pharmagistics_auto.jar -o
I would just tell you what you have already ruled out: Check your path and environment.
Since you have alredy done this, start debugging. Like write checkpoints into a logfile to see how far your script gets (if even started at all), check the cronjob log file for errors, check your mail (cron sends mails on errors) and so on ...
Not very specific, sorry.
"exporting my paths and variables" won't work since crontab runs in a different shell by a different user.
Also, not sure if this is a typo in how you entered the question, but I see:
usr/bin/java
...and I can't help but notice you're not specifying the fully qualified path. It's looking for a directory named "usr" in the current working directory. Oft times for crontab, the cwd is undefined, hence your reference goes nowhere.
Try specifying the full path from root, like so:
/usr/bin/java
Or, if you want to see an example of relative pathing in action, you could also try:
cd /
usr/bin/java
A few thoughts.
Remove the -- after the #!/bin/bash
Make sure to direct script output seen by cron to mail or somewhere else where you can view it (e.g. MAILTO=desiredUser)
Confirm that your script is running and not blocked by a different long-running script (e.g. on the second line, add touch /tmp/MY_SCRIPT_RAN && exit)
Debug the script using set -x and set -v once you know it's actually running
Do you define necessary paths and env vars in your personal .profile (or other script)? Have you tried sourcing that particular file (or is that what you're doing already with /root/.bash_profile?)
Another way of asking this is: are you certain that whatever necessary paths and env vars you expect are actually available?
If nothing else, have you tried echo'ing individual values or just using the "env" command in your script and then reviewing the stdout?
provide full paths to your jar file, and what user are you running the crontab in? If you set it up for a normal user, do you think that user has permission to source the root's profile?
If I run a script from /home/<user>/<dir>/script.sh, as root, the cron works pretty well. But If I run the script from /root/<dir>/script.sh (as root, again), the cron does not seem to work.
Having run afoul of various default $PATHs in the past when using 'cron', I always spell in full the absolute $PATH for each executable file and each target file. I always assume that 'cron' has NO $PATH set and has NO current-working-directory.
In other words don't use a command like
"myprocess abc*.txt"
but do it in full like
"/usr/localbin/myprocess /home/jvs/abc*.txt".
Alternatively, create a bash script which does the job, and call that bash script with a full absolute path, such as
"/usr/local/bin/myprocess_abc_txts".
If you need to have some flexibility in the script, use environment variables which are set specifically within the bash script you call with 'cron'.
I think you need to add a little more information. I'd guess it is a permissions thing though. Add the permissions of the file, the directories, and the line in your crontab so we can help. Also, if you are putting this in /root, are you running this in root's crontab?
Remember the environment - especially when run by cron rather than by root. When cron runs something, you probably don't have anything much set of your environment, unlike when you run a command via at. It is also not clear what your current directory will be. So, for commands that will be run by cron, use a script (as you're already doing) and make sure it sets enough of the environment for it to run. And make sure your environment setting code is not interactive!
On my machines, I have a mechanism such that the cron entry reads (for example):
23 1 * * 1-5 /usr/bin/ksh /work1/jleffler/bin/Cron/weekday
The weekday script in the Cron directory is a link to a standard script that first sets the environment and then runs the command /work1/jleffler/bin/weekday (in this case - it uses the name of the command to determine what to run).
The actual script in the Cron directory is:
: "$Id: runcron.sh,v 2.1 2001/02/27 00:53:22 jleffler Exp $"
#
# Commands to be performed by Cron (no debugging options)
# Set environment -- not done by cron (usually switches HOME)
. $HOME/.cronfile
base=`basename $0`
cmd=${REAL_HOME:-/real/home}/bin/$base
if [ ! -x $cmd ]
then cmd=${HOME}/bin/$base
fi
exec $cmd ${#:+"$#"}
I've been using it a while now - this version since 2001 - and it works a treat for me. I'm using a basic (Sun Solaris 10) implementation of cron; there may be new features in new versions of cron on other platforms to make some of this unnecessary. (The $REAL_HOME stuff is a weirdness of mine; pretend it says $HOME - though that makes some of the script unnecessary for you.) The .cronfile is responsible for the environment setting - it does quite a lot, but that's my problem, not yours.
It could be because you're looking for relative directories/files in the script which are located when running it from /home/ but not from /root, because /root is not in /home/root nor would it look like a users homefolder in /home/
Can you check and see if it is looking for relative files, or post the script?
On another note, why don't you just set it to run from a user's homefolder then?
Another way to run sh script is place your bash script in /usr/bin directory and simply run command bash yourscript.sh without adding /usr/bin/ directory