I have 100 scripts with .bash extension and I want to submit all of them in one shot in pbs script. I used
qsub *.bash
But it is not working. Can anyone please help me
One way it to simply loop over the files:
for i in `ls -1 *.bash` ; do qsub $i ; done
Related
I have an sh file (named a.sh) and I process it with the command sbatch for my project, so I type a lot of:
sbatch a.sh
There are 2 critical lines for a.sh (rest of them are irrelevant I guess). They are
source /okyanus/progs/gaussian/g16/bsd/g16.sariyer.profile
and
g16 dtn-b3-0-0.gjf
The second one is at the end of the file (further will be mentioned as aaa.com) and it is what needs to be changed. and aaa.com should be in the same directory with a.sh to submit the job aaa.com.
In the a.sh file, there is a name of a file (lets say aaa.com) which the data is taken for the sbatch process. So a standard operation for me to do lets say next 4 jobs is:
modify the a.sh file to change the name of the file addressed (say change aaa.com to aab.com), write and quit
type: sbatch a.sh (to start operation)
modify the a.sh file to change the name of the file addressed (say change aab.com to aac.com), write and quit
type: sbatch a.sh (to start operation)
modify the a.sh file to change the name of the file addressed (say change aac.com to aad.com), write and quit
type: sbatch a.sh (to start operation)
modify the a.sh file to change the name of the file addressed (say change aad.com to aae.com), write and quit
type: sbatch a.sh (to start operation)
However, I used to have a command template with sed -i. That command could do these 8 operations in one tick. From my old notes, I could find some parts of my old template.
my old template is a short version but works effectively to execute first two operations in one step with the command:
sed -i 's/aaa.com/aab.com/g' a.sh ; sbatch a.sh
above command does first and second step at once. I knew I used a command which could execute all 8 steps at once. I was something like:
sed -i 's/aaa.com/aab.com???aab.com/aac.com???aac.com/aad.com???aad.com/aae.com/g' a.sh ; sbatch a.sh
The above command could do all 8 steps at once, submitting next 4 jobs. However, I could not remember what should have been written on the ??? parts to successfully execute the command.
I am sure the command I propose worked with its correct state. Any other ideas and helps will be appreciated.
P.S.: a.sh file is something generated from the system. It sources a program related with chemistry, then submits the values of the .com (or .gjf) file to the chemistry program it runs for.
Thanks!
what should have been written on the ??? parts to successfully execute the command
s or s/<regex>/<replacement>/<flags> is a substitute command.
The default and ultimate command separator in sed is the newline. Optionally when possible you can separate commands with a ;. So it looks like:
sed 's/regex/replacement/; s/regex2/replacement2/; : another_command ; b another_command'
I was something like:
sed -i 's/aaa.com/aab.com???aab.com/aac.com???aac.com/aad.com???aad.com/aae.com/g'
Doing:
sed 's/aaa.com/aab.com/; s/aab.com/aac.com/;`
makes no sense, first aaa is replaced by aab, then aab by aac. It's the same as:
sed 's/aaa.com/aac.com/;`
You can do it from the back:
sed 's/aab.com/aac.com/; s/aaa.com/aab.com/;'
But really save yourself the trouble of dealing with some uncretain state and create a template file:
# template.a.sh
work on UNIQUE_STRING
then iterate over values and replace recreating the whole file each time, that way you do not have to care what "was" in the file and what "will be" in the file. Just create the file from template:
for i in aaa.com aab.com aac.com; do
sed 's/UNIQUE_STRING/'"$i"'/' template.a.sh > a.sh
sbatch a.sh
done
I am trying to write a script so I can use the 'qsub' command to submit a job to the cluster.
Pretty much, once I get into the cluster, I go to the directory with my files and I do these steps:
export PATH=$PATH:$HOME/program/bin
Then,
program > run.log&
Is there any way to make this into a script so I am able to submit the job to the queue?
Thanks!
Putting the lines into a bash script and then running qsub myscript.sh should do it.
Is there a way to stop sbatch from copying the script to the compute node. For example when I run:
sbatch --mem=300 /shared_between_all_nodes/test.sh
test.sh is copied to /var/lib/slurm-llnl/slurmd/etc/ on the executing compute node. The trouble with this is there are other scripts in /shared_between_all_nodes/ that test.sh needs to use and I would like to avoid hard coding the path.
In sge I could use qsub -b y to stop it from copying the script to the compute node. Is there a similar option or config in slurm?
Using sbatch --wrap is a nice solution for this
sbatch --wrap /shared_between_all_nodes/test.sh
quotes are required if the script has parameters
sbatch --wrap "/shared_between_all_nodes/test.sh param1 param2"
from sbatch docs http://slurm.schedmd.com/sbatch.html
--wrap=
Sbatch will wrap the specified command string in a simple "sh" shell script, and submit that script to the slurm controller. When --wrap is used, a script name and arguments may not be specified on the command line; instead the sbatch-generated wrapper script is used.
The script might be copied there, but the working directory will be the directory in which the sbatch command is launched. So if the command is launched from /shared_between_all_nodes/ it should work.
To be able to lauch sbatch form anywhere, use this option
-D, --workdir=<directory>
Set the working directory of the batch script to directory before
it is executed.
like
sbatch --mem=300 -D /shared_between_all_nodes /shared_between_all_nodes/test.sh
I have a cron script that I run nightly from the system-wide /etc/crontab like this:
00 01 * * * www-data /root/scripts/image-auto-process.sh 1> /var/data/logs/cron.out 2> /var/data/logs/cron.err
It's a bash script that backs up (with rsync) some directories full of scanned jpegs, runs a php-program to batch process those jpegs (preview/thumbnails), uploads them to a server and upon success cleans out the first mentioned directories.
Everything but the last clean-out step works like a charm. However, if I run it from the commandline as the www-data user, it works. When it runs via cron as same user, it doesn't.
The last, clean-out step looks like the following:
echo "remove original scans"
for i in `find $SCAN_ORIG_DIR -mindepth 2 -type d -print | grep -v -f $EXCLUDES`; do rm -rvf $i; done
echo "job Done..."
$SCAN_ORIG_DIR is the directory to search. $EXCLUDES is a file (readable to www-data) containing lines with directories to ignore (same file is used to tell rsync what not to backup). -mindepth 2 is used in find because I only want to return subdir's of $SCAN_ORIG_DIR that also have subdirs, like $SCAN_ORIG_DIR/subdir/subsubdir.
I tried putting the above code into its own script and running it on commandline and via cron just to make sure no prior code in the script was causing the problem.
Results in commandline (not exact representation, but just to illustrate):
remove original scans
removed '$SCAN_ORIG_DIR/subdir1/subsubdir'
removed '$SCAN_ORIG_DIR/subdir2/subsubdir'
removed '$SCAN_ORIG_DIR/subdir3/subsubdir'
job Done...
Results via cron:
remove original scans
job Done...
So, I'm stumped. I sincerely hope anyone can help shine a light on what's wrong here.
Thank you very much for you time and efforts :-)
A common problem with scripts when running in cron, is that the user login scripts (.bashrc, ,bash_profile) are not executed, so some variables are missing.
BTW, it is not good practice to use the system-wide /etc/crontab. Use crontab -e to add cron jobs.
Please kindly consider the following sample code snippet:
function cogstart
{
nohup /home/michael/..../cogconfig.sh
nohup /home/michael/..../freshness_watch.sh
watch -n 15 -d 'tail -n 1 /home/michael/nohup.out'
}
Basically the freshness_watch.sh and the last watch commands are supposed to be executed in parallel, i.e., the watch command doesn't have to wait till its prequel to finish. I am trying to work out a way like using xterm but since the freshness_watch.sh is a script that would last 15 minutes at the most(due to my bad way of writing a file monitoring script in Linux), I definitely want to trigger the last watch command while this script is still executing...
Any thoughts? Maybe in two separate/independent terminals?
Many thanks in advance for any hint/help.
As schemathings indicates indirectly, you probably want to append the '&' character to the end of the line with freshness_watch.sh. (without the single-quotes). I don't see any reason to use '&' for your final watch command, unless you add more commands after that.
'&' at the end of a unix command-line indicates 'run in the back-ground'.
You might want to insert a sleep ${someNumOfSecs} after your call to freshness_watch, to give it some time to have the CPU to it's self.
Seeing as you mention xterm, do you know about the crontab facility that allows you to schedule a job to run anytime you want, and is done without the user having to login? (Maybe this will help with your issue). I like setting jobs to run in crontab, because then you can capture any trace information you care to capture AND any possible output from stderr into a log/trace file.
( nohup wc -l * || nohup ls -l * ) &
( nohup wc -l * ; nohup ls -l * ) &
I'm not clear on what you're attempting to do - the question seems self contradictory.