I am new in Parallel processing. I have some python scripts which I should not change them for some reasons. Each of these python scripts only use one cpu core and does some processing on an input image. I run these python scripts with a shell script one after another. Can I do paralel threading in shell script without thouching python scripts so that each python script uses multiple cpu cores and the processing speed on each image gets increasd?
Yes, start them with GNU Parallel.
So, if you want to run your script 10 times, with parameters 0..9:
parallel python yourScript.py {} ::: {0..9}
If you want to see what would run without actually running anything:
parallel --dry-run ...
If you want a progress meter:
parallel --progress ...
Related
I'm executing several python scripts using SLURM in the following format:
General SLURM header (partition, wall-time, etc.)
python script.py scenario_1 &
python script.py scenario_2 &
python script.py scenario_3 &
python script.py scenario_4
I'm discovering that the order I specify these jobs to be completed matters. If scenario_4 (or, more generally, the last job) finishes before other jobs, the remaining jobs will not complete. I am able to simply organize my jobs by duration, although in many cases I do not know the relative compute time which makes this estimation imperfect. Is there a way to ensure that SLURM doesn't prematurely kill jobs?
Written like this, this submission script will ensure only termination of scenario_4.
Slurm will consider a job to be finished when the submission script is finished; and the submission script will consider itself done whenever all foreground jobs are done.
Add the wait command at the end, like this:
python script.py scenario_1 &
python script.py scenario_2 &
python script.py scenario_3 &
python script.py scenario_4 &
wait
The wait command will fore the submission script to wait for all background jobs to be done before considering itself finished.
I have some fairly time consuming python scripts to run ~3 hours or so on my machine. I don't want to run them concurrently since it might crash my machine. Alone I have more than enough memory but running 5 or so might cause an issue. I am running them remotely so I ssh into my server and run them like this:
nohup python my_script.py > my_output.txt &
That way if my connection gets interrupted I can re-establish the connection and my result is right there. I want to run the same python script a couple times with different command line arguments sequentially so I can run everything I need without me needing to set up the next one every few hours. I could manually code all of the arguments into a python script and do it that way but it seems inelegant. I don't want to have to fiddle with my python script every time I do this. Is there some sort of listener I could use to trigger the next one when one of them finishes?
I'd suggest writing a bash script that runs the python jobs sequentially:
#!/bin/bash
python3 my_script1.py > my_output1.txt
python3 my_script2.py > my_output2.txt
Then nohup that:
nohup ./driver.sh &
You really want to read up on utilities like tmux or screen and just script the while thing.
I would like to run parallel multiple run ncverilog.
Normally, we use ncverilog run script like this.
Run.scr-
ncveriog blar~blar~
But this is run at once. It means that if I want to run 100 scripts , I have to run new after previous script end. But I want to run at simultant 100 script.
How do I run the scripts at simultant?
Use GNU Parallel
parallel ncverilog bla bla
Normally, that will run as many ncverilogs in parallel as you have CPU cores, but you can add -j 25, for example, if you specifically want 25 to run in parallel.
If you need to supply lots of different parameters, just make a loop or script that generates the parameters and feeds a list of jobs into GNU Parallel like this:
for i in {0..99}; do
echo ncverilog $i <and some other parameters>
done | parallel -j 25
Example
Example
Documentation
there is a process running that will write some output to a set of files. I wrote a python script that will make a copy of these output files in another directory. Right now I can simply run the python script when I notice the other process is done. How can I get this script to be run automatically when the other process is done?
I don't have control over the other process's source code. Messing with the source code might make the results file inadmissible, so I'd rather not touch it at all.
I am running Python 2.7.1 in an Ubuntu 11.x machine.
You don't tell much about what is the program running before the Python script, but if it is or you can convert to a shell script, you can use this syntax:
$ first-script.sh && python-script.sh
The && operator means that if the first script finished successfully, run the second afterwards.
Of course, you could invoke the python interpreter with your script directly as the 2nd script. Here I assume that it is wrapped in a Bash script.
I have a testcase containing multiple Execute Commands (SSH Library) which are calling different commands in Linux environment. The main thing I would like to do is to run some of them in parallel. By default Robot performs one command and after it finishes, performs the next one.
As for me it is not a good behavior, I would like to have my command executed during execution of previous one. For example:
Execute Command ./script.sh
Execute Command ./script_parallel.sh
What I would like Robot to do:
Execute script.sh
During execution perform script_parallel.sh (which will finish before script.sh finishes)
Finish script.sh
Will it be possible to use GNU Parallel?
Execute Command parallel ::: ./script.sh ./script_parallel.sh
Have you tried Start command? It starts the command in background and returns immediately. To verify successful execution of commands you need Read Command Output.