I am new to shell scripting. I am trying to work through this.
> script to execute in cron (util.sh)
#!/bin/sh
HOST='ahostname'
PORT='3306'
USER='auser'
PASS='apassword'
DB='adatabase'
. /mnt/stor/backups/backup.sh
(I also tried source /mnt/stor/backups/backup.sh)
> script to execute (backup.sh)
When backup.sh is called (it does get called) it appears to simply be parsed and not executed. So no matter what I put in it I get messages like:
/mnt/stor/backups/backup.sh: line 8: date: command not found
/mnt/stor/backups/backup.sh: line 8: mysqldump: command not found
/mnt/stor/backups/backup.sh: line 8: tar: command not found
/mnt/stor/backups/backup.sh: line 8: rm: command not found
The idea is to have a domain localized file, execute it with variables, and call a master script that uses the variable to do the dirty work. Because of limitations with one of my hosts and multiple domains this is the best method.
The script with the Problem seems to be /mnt/stor/backups/backup.sh. Try setting the PATH to include all the usual directories with binaries, so the script can find its tools. Or, even better, change /mnt/stor/backups/backup.sh and use absolute paths in the commands like /bin/rm instead of just 'rm'.
When running from cron, you can't rely on any variables that are normally in your login shell's profile (e.g.: PATH, CLASSPATH, etc). You have to set explicitly what you need. In your case, i'm guessing that it's the lack of a PATH variable that's causing your troubles.
It's also good practice to put full paths to the programs you're executing from an unattended script, just to make sure you really are going to run that specific command, i.e., don't rely on the path.
So instead of
date
for example, use
/bin/date
etc.
Related
I have encountered a bit of a conundrum while working on an automation project.
When I try to run:
program = subprocess.run("mode")
I get:
FileNotFoundError: [WinError 2] The system cannot find the file specified
However, when I replace mode with ipconfig:
program = subprocess.run("ipconfig")
it runs perfectly fine.
Anyone got an explanation? I am currently using a batch file to run the mode command, but I would like to change the arguments without editing a batch file.
Edit 1:
I also just tried using os.system:
os.system("mode")
and that also worked.
Edit 2:
Now I would just like answer to the original problem just to understand what was going on.
In Actual meaning of 'shell=True' in subprocess it pretty much says that shell=True is something that you should shy away from.
FileNotFoundError: [WinError 2] The system cannot find the file specified
Is what tipped me off that you might want shell=True in your subprocess call. If the file can't be found that means one of two things:
It's not on your path.
It's not actually a file.
For instance, in Linux:
$ which echo
echo: shell built-in command
That makes it pretty obvious that there is no echo file. It's just a command that's built into the shell. This may be the same thing when it comes to mode on Windows. Though this site seems to suggest that it's a MODE.COM file. You may try invoking that, as in
subprocess.run('MODE.COM')
That may work - at least according to one of the answers that I linked to
Invoking via the shell does allow you to expand environment variables and file globs according to the shell's usual mechanism. On POSIX systems, the shell expands file globs to a list of files. On Windows, a file glob (e.g., ".") is not expanded by the shell, anyway (but environment variables on a command line are expanded by cmd.exe).
So in your case, perhaps mode isn't a file, but MODE.COM is, and since Windows has a spotty relationship with casing, it seems possible that by passing shell=True, the Windows shell happily takes mode and converts it to MODE.COM for you, but without it, it tries to execute the file literally named mode, which doesn't exist.
new to linux's inner workings.
I have accrued a lot of executable scripts since I started, I was told I wasted a lot of time typing their full paths every time I wanted to use them so it was suggested to add the paths into my .bash_profile so I did as follows
PATH=$HOME/bin/Tools/cif2cell-1.1.5:$HOME/bin/Tools/cteprouts:$PATH
PATH=$PATH:$HOME/bin:$HOME/bin/Tools
export $PATH
. $HOME/bin/AtomsScriptsNG/bin/src/settings.sh
source $HOME/bin/AtomsScriptsNG/bin/src/settings.sh
I am connecting to a cluster computer which runs on the Unix language using putty.exe (not sure if it makes a difference).
When I login with the above .bash_profile I get the following errors
-bash: export: `/home/eg205/bin/Tools/cif2cell-1.1.5:/home/eg205/bin/Tools/ctepro uts:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/opt/pbs/default/bin:/home/eg205/bin:/home/eg205/bin:/home/eg205/bin/Tools': not a valid identifier
-bash: /home/eg205/bin/AtomsScriptsNG/bin/src/settings.sh: No such file or directory
-bash: /export71/home/eg205/bin/AtomsScriptsNG/bin/src/settings.sh: No such file or directory
I'm sure I'm making some glaring mistake... how do I set it up correctly to load the AtomsScruptsNG environment from the settings.sh and look in the directories bin, cif2cell-1.1.5 and cteprouts for the scripts I run often?
try removing the $ from your export line like this:
export PATH
this will tell the shell to export the variable 'PATH' and not it's "contents/value".
One thing to have in mind concerning shells: every line you pass into a shell, either by typing or by feeding it scripts, will be executed AFTER all substitutions have been made. so everything which looks like $FOOBAR will be replaced by the contents of the variable FOOBAR.
if every variable has been succesfully replaced the whole commandline will be executed by the shell.
I am trying to create a script that will run a program on each file in a list. I have been trying to do this using a .csh file (I have no clue if this is the best way), and I started with something as simple as hello world
echo "hello world"
The problem is that I cannot execute this script, or verify that it works correctly. (I was trying to do ./testscript.csh which is obviously wrong). I haven't been able to find anything that really explains how to run C Scripts, and I'm guessing there's a better way to do this too. What do I need to change to get this to work?
You need to mark it as executable; Unix doesn't execute things arbitrarily based on extension.
chmod +x testscript.csh
Also, I strongly recommend using sh or bash instead of csh, or you will soon learn about the idiosyncrasies of csh's looping and control flow constructs (some things only work inside them if done a particular way, in particular with the single-line versions things are very limited).
You can use ./testscript.csh. You will however need to make it executable first:
chmod u+x testscript.csh
Which means set testscript to have execute permissions for the user (who ever the file is owned by - which in this case should be yourself!)
Also to tell the OS that this is a csh script you will need put
#! /path/to/csh
on the first line (where /path/to/csh is the full path to csh on your system. You can find that out by issuing the command which csh).
That should give you the behvaiour you want.
EDIT As discussed in some of the comments, you may want to choose an alternative shell to C Shell (csh). It is not the friendliest one for scripting.
You have several options.
You can run the script from within your current shell. If you're running csh or tcsh, the syntax is source testscript.csh. If you're running sh, bash, ksh, etc., the syntax is . ./testscript.sh. Note that I've changed the file name suffix; source or . runs the commands in the named file in your current shell. If you have any shell-specific syntax, this won't work unless your interactive shell matches the one used by the script. If the script is very simple (just a sequence of simple commands), that might not matter.
You can make the script an executable program. (I'm going to repeat some of what others have already written.) Add a "shebang" as the first line. For a csh script, use #!/bin/csh -f. The -f avoids running commands in your own personal startup scripts (.cshrc et al), which saves time and makes it more likely that others will be able to use it. Or, for a sh script (recommended), used #!/bin/sh (no -f, it has a completely different meaning). In either case, run chmod +x the_script, then ./the_script.
There's a trick I often use when I want to perform some moderately complex action. Say I want to delete some, but not all, files in the current directory, but the criterion can't be expressed conveniently in a single command. I might run ls > tmp.sh, then edit tmp.h with my favorite editor (mine happens to be vim). Then I go through the list of files and delete all the ones that I want to leave alone. Once I've done that, I can replace each file name with a command to remove it; in vim, :%s/.*/rm -f &/. I add a #!/bin/sh at the top save it, chmod +x foo.sh, then ./foo.sh. (If some of the file names might have special characters, I can use :%s/.*/rm -f '&'/.)
Recently I created a bash script which I am supposed to run in cron.
After preparing the bash script and its normal working, I put it in Cron and found that it was failing. As as second step , I removed all the environment dependencies i.e instead of just file.txt, I specified /home/blah-blah/file.txt
I still found the script to be failing still at one step. The step was a data processing tool.
The command i executed was /bin/blah-blah/processing_tool -parameter $INDEX where $INDEX is a variable calculated within the bash script.
Third step was to add the bash profile as source at the beginning of the bash script. Voila!!!! The script started executing perfectly from cron.
My question is why is this happening even after I removed all the environment dependencies from my script. Also I have heard that sourcing a cron job to a bash profile is not recommended. If so, Is there any other way in which I can avoid doing this.
Basicly: Anything started from cron starts with a totally clean slate.
You can make no assumptions whatsoever about the content of environment variables or whichever folder is the current folder at the start of any script run from cron.
Easiest solution:
cd to the desired directory to make sure your path is in the desired location.
source /etc/profile to mak sure you get the system wide environment variables setup.
source ~myuserid/.profile to read your personal environment settings. (~/.profile won't work as that would indicate the cron user.)
Then start executing the actual script.
Of course the approach above requires the cron process to have read access to your home-dir adn it's probably doing a lot more work thatn is actually required.
Slightly more complicated: Figure out which environment variables are required by the script and anything that gets called by the script.
Explicitly export these at the beginning of the cron script.
(P.s. replace /etc/profile and ~myuserid/.profile with whatever are the corresponding files for your shell of choice.)
A cron can be thought of as a separate user. So, this "user" may not "see" or "read" the same files as you do. It is thus essential that all path names etc. be defined in the absolute.
Every script runs within its own process. So, when you run a script, you can change the $SHELL and any other variable within but it will be lost once you get out of it. My guess is that the $INDEX variable computation may have had been computed within the script successfully but its use outside of the script may have failed. Without more information about what job it was, or what you wanted to do, it is hard to tell.
There are two ways to run a cron job:
As root, you can run su -user -c < job > in root crontab.
Sourcing your profile explicitly, as you have done.
You can also set environment variables within the crontab.
As user in the user crontab, you can run it like so: "/home/blah/.profile && myScript"
That said, there HAS to be something in your environment variables (apart from file extensions) that is not present when you run the cron job. You will have to execute that script with -x flag (in bash) and then pore over the output. Using a diff between your environment variables and that of root/cron might be a pointer. Also, check if there are some utilities that are being used in your scripts whose locations are not part of the $PATH variable for cron/root.
I have tried exporting my paths and variables and crontab still will not run my script. I'm sure I am doing something wrong.
I have a shell script which runs a jar file. This is not working correctly.
After reading around I have read this is commonly due to incorrect paths due to cron running via its own shell instance and therefore does not have the same preferences setup as my profile does.
Here is what my script looks like today after several modifications:
#!/bin/bash --
. /root/.bash_profile
/usr/bin/java -jar Pharmagistics_auto.jar -o
...
those are the most important pieces of the script, the rest are straightforward shell based.
Can someone tell me what I am doing wrong?
Try specifying the full path to the jar file:
/usr/bin/java -jar /path/to/Pharmagistics_auto.jar -o
I would just tell you what you have already ruled out: Check your path and environment.
Since you have alredy done this, start debugging. Like write checkpoints into a logfile to see how far your script gets (if even started at all), check the cronjob log file for errors, check your mail (cron sends mails on errors) and so on ...
Not very specific, sorry.
"exporting my paths and variables" won't work since crontab runs in a different shell by a different user.
Also, not sure if this is a typo in how you entered the question, but I see:
usr/bin/java
...and I can't help but notice you're not specifying the fully qualified path. It's looking for a directory named "usr" in the current working directory. Oft times for crontab, the cwd is undefined, hence your reference goes nowhere.
Try specifying the full path from root, like so:
/usr/bin/java
Or, if you want to see an example of relative pathing in action, you could also try:
cd /
usr/bin/java
A few thoughts.
Remove the -- after the #!/bin/bash
Make sure to direct script output seen by cron to mail or somewhere else where you can view it (e.g. MAILTO=desiredUser)
Confirm that your script is running and not blocked by a different long-running script (e.g. on the second line, add touch /tmp/MY_SCRIPT_RAN && exit)
Debug the script using set -x and set -v once you know it's actually running
Do you define necessary paths and env vars in your personal .profile (or other script)? Have you tried sourcing that particular file (or is that what you're doing already with /root/.bash_profile?)
Another way of asking this is: are you certain that whatever necessary paths and env vars you expect are actually available?
If nothing else, have you tried echo'ing individual values or just using the "env" command in your script and then reviewing the stdout?
provide full paths to your jar file, and what user are you running the crontab in? If you set it up for a normal user, do you think that user has permission to source the root's profile?