I use -N option to specify the name for the job when submitting through qsub. The qsub, however, adds some numeric string after thejob name as described in the man page:
By default the file name for standard output has the
form job_name.ojob_id and job_name.ojob_id.task_id for
array job tasks (see -t option below).
Therefore, whenever I submit a new job with same job name, a new suffix .ojob_id is added to the job name and a new output file is created.
What I trying to achieve is to have same output file each time a job is submitted through qsub. How can I do that? I have to run a job several time and I want the output from a run to overwrite the output file generated in the previous run. How can I achieve that?
See the example below:
First time command is given to run script hello_world to output in log_hello_world:
qsub -cwd -N log_hello_world hello_world.sh
It creates two output files:
log_hello_world.e7584345
log_hello_world.o7584345
Second time the same command is given: It creates two more output files
log_hello_world.e7584366
log_hello_world.o7584366
What can I do to get the output in just one file log_hello_world.
I was able to resolve this issue by using options -o and -e to save the log and the error files respectively. With these options, the log and the errors from the job are written into same file every time from this command.
qsub -cwd -N job_hello_world -o log.hello_world -e error.hello_world hello_world.sh
You should append a file with a fixed name. This is done in the code which you run. You create a file in your directory, and then append it with the new results each time you run your code. So in your code (not in the qsub line), explicitly add a line of code which asks the results to be written to a file in your directory, in Mathematica this would be
str = OpenAppend["/home/my_name/Scratch/results.dat"];
Write[str,results];
Close[str];
Where results is a variable which contains the results of your computation. Then just run the job using qsub -N log_hello_world hello_world.sh. This will write the results to the same file every time you run the job without changing the name of the file.
(If you're looking to write both -o and -e files to the same file, you can just add -j y to the qsub after having specified the file path for the error file)
Related
In my main script, I call a function and within that function I call stata (with system command) where it takes input from matlab and then matlab calls the output of this stata command. I want each of these input and output files have a unique JobID. (I can give other unique IDs with datetime etc, but I don't want this.) I want JobIDs like 1,2,3...
I am using a cluster environment and PBS file to add details about my submit. I want this PBS file take the JOBID from my first step and then send it to the matlab script.
My steps are as follows:
1- write a .py script for multiple jobs (I first try for #2 jobs)
#!/bin/env python
#PBS -A open
import csv, subprocess
for i in range(1,3)
qsub_cmd = """qsub -F JOBID = {jobid} submit_m3.pbs""".format(jobid=i)
print(qsub_cmd)
exit = subprocess.call(qsub_cmd, shell=True)
2 - Write pbs file called submit_m3.pbs (I need to pass JOBID here)
#!/bin/bash
#PBS -l nodes=1:ppn=8
#PBS -l walltime=48:00:00
#PBS -l pmem=10gb
#PBS -A open
# Load in matlab
module purge
module load matlab
cd ~/code
matlab -nodisplay -nosplash < myScript.m > log.myLog
# JOBID must be placed somewhere in the above line.
3- Call the .py script in Linux terminal
python myPyFile.py
In the step 2, instead of the line starting with matlab, I also tried this (following other community questions)
matlab -nodisplay -nosplash -nodesktop -r "job_id=${JOBID};run('myScript.m');exit;"
but this code didn't pass JOBID to my script.
So, 2 things need to be solved. Passing JOBID to .pbs file and then to matlab file. Once it is in matlab, the rest (stata part) is OK!
I am running a hive script from within WinSCP (as it allows me to do simple edits to my file as well as run them using file custom commands or via the terminal).
Currently, I'm using the following as a custom command (just right clicking on the file and running this, makes the task very convenient):
nohup /location/platform/cloak/bin/cloak-hive -f ! >> script_temp.log 2>&1 &
which creates a log file the first time and for each subsequent run, the same file gets appended. Ideally I would want the log file to take the name from the ! pattern and expand it to filename.log. If that is not possible, I'd want to have a timestamp added to the filename.
Aim is to get different logs for different runs, but without changing the custom command each time. I have seen various solutions to append the timestamp after the file is created, but I need to get the name correct with the custom command itself for convenience and tracking. The server being accessed by WinSCP is running RHEL 7.8.
So use the ! pattern in the log name:
nohup /location/platform/cloak/bin/cloak-hive -f ! >> !.log 2>&1 &
I have an sh file (named a.sh) and I process it with the command sbatch for my project, so I type a lot of:
sbatch a.sh
There are 2 critical lines for a.sh (rest of them are irrelevant I guess). They are
source /okyanus/progs/gaussian/g16/bsd/g16.sariyer.profile
and
g16 dtn-b3-0-0.gjf
The second one is at the end of the file (further will be mentioned as aaa.com) and it is what needs to be changed. and aaa.com should be in the same directory with a.sh to submit the job aaa.com.
In the a.sh file, there is a name of a file (lets say aaa.com) which the data is taken for the sbatch process. So a standard operation for me to do lets say next 4 jobs is:
modify the a.sh file to change the name of the file addressed (say change aaa.com to aab.com), write and quit
type: sbatch a.sh (to start operation)
modify the a.sh file to change the name of the file addressed (say change aab.com to aac.com), write and quit
type: sbatch a.sh (to start operation)
modify the a.sh file to change the name of the file addressed (say change aac.com to aad.com), write and quit
type: sbatch a.sh (to start operation)
modify the a.sh file to change the name of the file addressed (say change aad.com to aae.com), write and quit
type: sbatch a.sh (to start operation)
However, I used to have a command template with sed -i. That command could do these 8 operations in one tick. From my old notes, I could find some parts of my old template.
my old template is a short version but works effectively to execute first two operations in one step with the command:
sed -i 's/aaa.com/aab.com/g' a.sh ; sbatch a.sh
above command does first and second step at once. I knew I used a command which could execute all 8 steps at once. I was something like:
sed -i 's/aaa.com/aab.com???aab.com/aac.com???aac.com/aad.com???aad.com/aae.com/g' a.sh ; sbatch a.sh
The above command could do all 8 steps at once, submitting next 4 jobs. However, I could not remember what should have been written on the ??? parts to successfully execute the command.
I am sure the command I propose worked with its correct state. Any other ideas and helps will be appreciated.
P.S.: a.sh file is something generated from the system. It sources a program related with chemistry, then submits the values of the .com (or .gjf) file to the chemistry program it runs for.
Thanks!
what should have been written on the ??? parts to successfully execute the command
s or s/<regex>/<replacement>/<flags> is a substitute command.
The default and ultimate command separator in sed is the newline. Optionally when possible you can separate commands with a ;. So it looks like:
sed 's/regex/replacement/; s/regex2/replacement2/; : another_command ; b another_command'
I was something like:
sed -i 's/aaa.com/aab.com???aab.com/aac.com???aac.com/aad.com???aad.com/aae.com/g'
Doing:
sed 's/aaa.com/aab.com/; s/aab.com/aac.com/;`
makes no sense, first aaa is replaced by aab, then aab by aac. It's the same as:
sed 's/aaa.com/aac.com/;`
You can do it from the back:
sed 's/aab.com/aac.com/; s/aaa.com/aab.com/;'
But really save yourself the trouble of dealing with some uncretain state and create a template file:
# template.a.sh
work on UNIQUE_STRING
then iterate over values and replace recreating the whole file each time, that way you do not have to care what "was" in the file and what "will be" in the file. Just create the file from template:
for i in aaa.com aab.com aac.com; do
sed 's/UNIQUE_STRING/'"$i"'/' template.a.sh > a.sh
sbatch a.sh
done
Tried to run a bash script for this as I want to put this into cron and run every night to pull out errors from a file and write to a file in same directory:
My script which just hangs and pulls out nothing:
#!/bin/bash
tail -f /var/log/syslog | grep -i "error" > /var/log/syserrorlog.log
When this runs I would like this to write to/update the same file and not override it.
Change your > to >> (the latter means append). Also, tail -f will hang by definition: it monitors files for new data.
If your syslog is rotated every day, then you can simply use
#!/bin/bash
cat /var/log/syslog | grep -i "error" >> /var/log/syserrorlog.log
If it is not rotated, you can add a grep to the pipeline which filters out the relevant date
Try using logrotate. This is daemon used to periodically rotate logs (e.g. archive logs files and then clear them every night). It supports many configuration options, one of them is postrotate - custom script executed after rotation is done. Description copied from logrotate's man page:
postrotate/endscript
The lines between postrotate and endscript (both of which must appear on lines by themselves) are executed (using /bin/sh) after the
log file is rotated. These directives may only appear inside a log
file definition. Normally, the absolute path to the log file is passed
as first argument to the script. If sharedscripts is specified, whole
pattern is passed to the script. See also prerotate. See sharedscripts
and nosharedscripts for error handling.
Syslog is a standard daemon too, so it should have configuration file in /etc/logrotate.d/. You could add your commands there.
I am using a PBS queuing system and submit a job with the following bash jobscript
#PBS -l nodes=1:ppn=1
#PBS -l walltime=1:30:00
./aprogram $input
using qsub -v "input=myinputfile" script.job
This works fine except for the fact that if I run the job, change something in the input file without renaming it and rerun the job shortly thereafter, the input file the program aprogram gets is still the old input file.
Obviously the file is hiding somewhere and is not reread if the time is to short (waiting a few minutes does the trick). Does anybody have an idea where I could wipe the file out such that it is correctly read?
I can't speak for all PBS queuing systems, but Torque only makes a local copy of the script; it doesn't parse the script to figure out what the script uses and make local copies of those things. In other words, if you are using Torque it is not doing this. I'm not sure what it could be, although I would try to track down what is refreshed every few minutes to see if that could be it.