Parallel seems to be installed properly as I am able to run basic parallel check commands.
# parallel --version
GNU parallel 20161222
Copyright (C) 2007,2008,2009,2010,2011,2012,2013,2014,2015,2016
Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.
Web site: http://www.gnu.org/software/parallel
When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.
But when I'm trying to execute a query on remote servers it doesn't give any output also doesn't returns the prompt.. eg :-
parallel ssh {} hostname :::: hosts
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
To silence this citation notice: run 'parallel --citation'.
in above eg , hosts file contains a list of remote server IP's
Even a simple parallel echo ::: A B C command doens's gives output..
I tried running with /usr/bin/parallel but issue remains..
when I ran the command using bash -x I get the following error :-
bash -x parallel ssh {} hostname :::: hosts
+ use IPC::Open3
/usr/bin/parallel: line 22: use: command not found
/usr/bin/parallel: parallel: line 24: syntax error near unexpected token `('
/usr/bin/parallel: parallel: line 24: `use POSIX qw(:sys_wait_h setsid ceil :errno_h);'
I've checked the file /usr/bin/parallel seems fine to me.
Not able to find solution for this. Any help is really appreciated.
Thanks.
UPDATE_1 :-
root#sg-server:[~]:# parallel -Dall echo ::: 1
shell? perl /usr/bin/parallel -Dall echo ::: 1
shell? -bash
which -bash => shell path /bin/bash
Academic tradition requires you to cite works you base your article on.
When using programs that use GNU Parallel to process data for publication
please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
To silence this citation notice: run 'parallel --citation'.
Open file descriptors: 2 1
echo Context: Non: 4 Ctxgrp: NonCtxGrp: 1
echo ▒<▒> Context: Non: 4 Ctxgrp: 1 NonCtxGrp: 1
Wanted procs: 16
MultifileQueue->empty RecordQueue->empty CommandLineQueue->empty JobQueue->empty MultifileQueue->empty RecordQueue->empty eval 1
noncontext + command: 5
context+noncontext + command: 5
cmd_line->number_of_args 1
Replace echo ▒<▒>
Target: echoTarget: ▒<▒>Replacing in ▒<▒>
Return echo 1
noncontext + command: 5
context+noncontext + command: 5
noncontext + command: 5
context+noncontext + command: 5
6 == 6 echo 1
Time to fork 1 procs: 0 (processes so far: 1)
MultifileQueue->empty 1 RecordQueue->empty 1CommandLineQueue->empty 1JobQueue->empty 1 RecordQueue-unget ''
JobQueue->empty Limited to procs: 1
JobQueue->empty Running jobs before on ::
: has out of 1 jobs running. Start another.
JobQueue->empty Command to run on ':': 'echo 1'
processes . Starting (1): echo 1
bash -c testfun\(\)\ \{\ rm\ /tmp/parTMasE.tst\;\ \}\;\ export\ -f\ testfun\;\ perl\ -MIPC::Open3\ -e\ if\\\(\\\$pid\\\=::open3\\\(\\\$i,\\\$o,\\\$e,\\\"-\\\"\\\)\\\)\\\ \\\{\\\ wait\\\;\\\ \\\}\\\ else\\\ \\\{\\\ exec\\\(\\\"bash\\\",\\\"-c\\\",\\\"testfun\\\ \\\&\\\&\\\ true\\\"\\\)\\\;\\\ \\\} 2>/dev/null
^Ckill_sleep TERM
kill_sleep TERM
kill_sleep TERM
kill_sleep KILL
Even on running parallel -Dall echo ::: 1 command, I had to CTRL+c to get the prompt back.
As we debugged in the comments your (/usr/local)/bin/rm is not the normal (/usr/local)/bin/rm.
Related
I am using two scripts say script1 and script2. I call script2 from script1 which generates lots of text data. I am redirecting this text to a file using tee. But tee is stuck and doesn't move forward.
I cannot give the exact example as scripts are way too long to be included here. So giving a snippet of what's going on.
# script1
# Do some task
/bin/bash script2.sh 2>&1 | tee script2.log
capture=$(cat script2.log | grep "Successfully completed" | wc -l)
# Do some more work
#script2
# Execute some code
# Some python scripts
echo "Script End here"
Now when I execute script1.sh I am getting Script End here indicating second script completed successfully. But script1 is stuck at tee. When I removed tee and didn't capture output it worked out fine.
Did anyone face this before. Any idea what might be going on.
# System details
$ uname -a
Linux jetson-nx-jetpack461 4.9.253-tegra #1 SMP PREEMPT Sat Feb 19 08:58:27 PST 2022 aarch64 aarch64 aarch64 GNU/Linux
# Bash version
$ bash --version
GNU bash, version 4.4.20(1)-release (aarch64-unknown-linux-gnu)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
From script2
When the script has a large number of lines (1000/2000/...), it is difficult to debug using set -x.
Either proceed based on comment from Barmar, or you include:
echo read1
read
command12
echo read2
read
command23
View the location where it is hanging.
If it is hanging exactly due to tee, use the following command:
echo read5
read
echo type tee
type tee
echo read6
read
Using this we can identify exactly where the script is hanging.
You can also use read1.1 read1.0 read1.-1 ... when debugging.
I have seven licenses of a particular software. Therefore, I want to start 7 jobs simultaneously. I can do that using '&'. Now, 'wait' command waits till the end of all of those 7 processes to be finished to spawn the next 7. Now, I would like to write the shell script where after I start the first seven, as and when a job gets completed I would like to start another. This is because some of those 7 jobs might take very long while some others get over really quickly. I don't want to waste time waiting for all of them to finish. Is there a way to do this in linux? Could you please help me?
Thanks.
GNU parallel is the way to go. It is designed for launching multiples instances of a same command, each with a different argument retrieved either from stdin or an external file.
Let's say your licensed script is called myScript, each instance having the same options --arg1 --arg2 and taking a variable parameter --argVariable for each instance spawned, those parameters being stored in file myParameters :
cat myParameters | parallel -halt 1 --jobs 7 ./myScript --arg1 --argVariable {} --arg2
Explanations :
-halt 1 tells parallel to halt all jobs if one fails
--jobs 7 will launch 7 instances of myScript
On a debian-based linux system, you can install parallel using :
sudo apt-get install parallel
As a bonus, if your licenses allow it, you can even tell parallel to launch these 7 instances amongst multiple computers.
You could check how many are currently running and start more if you have less than 7:
while true; do
if [ "`ps ax -o comm | grep process-name | wc -l`" -lt 7 ]; then
process-name &
fi
sleep 1
done
Write two scripts. One which restarts a job everytime it is finished and one that starts 7 times the first script.
Like:
script1:
./script2 job1
...
./script2 job7
and
script2:
while(...)
./jobX
I found a fairly good solution using make, which is a part of the standard distributions. See here
What will happen when I a script file and save it when it's still running, and will it print my needed results.
Let's test it.
Create a script test.sh:
#!/usr/bin/env bash
sleep 1
echo 'echo "executed overwritten"' >> "$0" # append to self
sleep 1
echo 'executed original'
and execute it:
$ bash --version
GNU bash, version 4.2.24(1)-release (i686-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
$ chmod +x test.sh
$ ./test.sh
executed original
executed overwritten
$
Notice that bash continued reading the modified file. It maintains its current position in the file (in bytes) when the file changes.
As a demonstration, the script
#!/usr/bin/env bash
sleep 1
dd if=/dev/urandom bs=1024 count=1 of="$0" &>/dev/null # overwrite self
sleep 1
echo 'executed original'
gives the output
$ ./test.sh
./test.sh: line 6: syntax error near unexpected token `$'\311\262\203''
./test.sh: line 6: `��z�eп9)�v��▒y�a��44'{�d��4\:�A����˷���&�$�����l�
#(ɲ��4��OϹI�n>��7��P�M�a��X.�S�a���V�m�~O<��{}������J��$��TOtRd��Nw�&��B�Dz�▒��-��<`�P<?N��▒rT�Jq�L����JY�*hz���M�����i�⫣��S+�����\��c���m�NKV�8|��xvX}�V����PTd䊄�9��7���|��/��X��
��0¤k��_�R���e�*���(qu:UUɭp/j��n��bŇ_�UR?3▒�▒�%Rn�|DE$8�QbaK)A�{ ��O>9��A�����lt�����g)s��O��M��#���w��|�����N��,W'
Notice that it attempted to execute the random gibberish.
(This is Ubuntu 12.04. Behavior may vary with other shells.)
I have a script - a linear list of commands - that takes a long time to run sequentially. I would like to create a utility script (Perl, Bash or other available on Cygwin) that can read commands from any linear script and farm them out to a configurable number of parallel workers.
So if myscript is
command1
command2
command3
I can run:
threadpool -n 2 myscript
Two threads would be created, one commencing with command1 and the other command2. Whichever thread finishes its first job first would then run command3.
Before diving into Perl (it's been a long time) I thought I should ask the experts if something like this already exists. I'm sure there should be something like this because it would be incredibly useful both for exploiting multi-CPU machines and for parallel network transfers (wget or scp). I guess I don't know the right search terms. Thanks!
If you need the output not to be mixed up (which xargs -P risks doing), then you can use GNU Parallel:
parallel -j2 ::: command1 command2 command3
Or if the commands are in a file:
cat file | parallel -j2
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel
In Perl you can do this with Parallel::ForkManager:
#!/usr/bin/perl
use strict;
use warnings;
use Parallel::ForkManager;
my $pm = Parallel::ForkManager->new( 8 ); # number of jobs to run in parallel
open FILE, "<commands.txt" or die $!;
while ( my $cmd = <FILE> ) {
$pm->start and next;
system( $cmd );
$pm->finish;
}
close FILE or die $!;
$pm->wait_all_children;
There is xjobs which is better at separating individual job output then xargs -P.
http://www.maier-komor.de/xjobs.html
You could also use make. Here is a very interesting article on how to use it creatively
Source: http://coldattic.info/shvedsky/pro/blogs/a-foo-walks-into-a-bar/posts/7
# That's commands.txt file
echo Hello world
echo Goodbye world
echo Goodbye cruel world
cat commands.txt | xargs -I CMD --max-procs=3 bash -c CMD
I have several init.d scripts that I'm using to start some daemons. Most of these scripts I've found on the internet and they all use start-stop-daemon. My understanding is that "start-stop-daemon" is a command that is specific to Linux or BSD distros and is not available on Solaris.
What is the best way to translate my init.d scripts from Linux to Solaris? Is there a command equivalent to start-stop-daemon that I can use, roughly?
Since I'm not much of a Solaris user, I'm willing to admit upfront that I don't even know if my question is inherently invalid or not.
start-stop-daemon is a Linux thing, and not used that much on Solaris. I guess you can port the command though, if you want to reuse your init scripts.
Otherwise it depends on what version of Solaris you are using. Starting with Solaris 10 and also OpenSolaris they use a new startup script framework "Solaris Service Management Facility", which you configure with the commands svcs, svccfg and svcadm.
See http://www.oracle.com/technetwork/server-storage/solaris/overview/servicemgmthowto-jsp-135655.html for more information.
For older Solaris releases most init scripts are written in pure shell without any helper commands like start-stop-daemon.
On Solaris 10 or later using SMF is recommended, but on an earlier release you'd create an init script in /etc/init.d and link to it from the rcX.d directories. Here's a bare-bones example of an init script for launching an rsync daemon:
#!/sbin/sh
startcmd () {
/usr/local/bin/rsync --daemon # REPLACE WITH YOUR COMMANDS
}
stopcmd () {
pkill -f "/usr/local/bin/rsync --daemon" # REPLACE WITH YOUR COMMANDS
}
case "$1" in
'start')
startcmd
;;
'stop')
stopcmd
;;
'restart')
stopcmd
sleep 1
startcmd
;;
*)
echo "Usage: $0 { start | stop | restart }"
exit 1
;;
esac
Create a link to the script from each rcX.d directory (following the "S"/"K" convention).
ln rsync /etc/rc3.d/S91rsync
for i in `ls -1d /etc/rc*.d | grep -v 3`; do ln rsync $i/K02rsync; done
See the README in each rcX.d directory and check the man page for init.d. Here's a bit of the man page:
File names in rc?.d directories are of the form
[SK]nn, where S means start this job, K means
kill this job, and nn is the relative sequence number for killing or
starting the job.
When entering a state (init S,0,2,3,etc.) the rc[S0-6] script
executes those scripts in /etc/rc[S0-6].d that are prefixed with K
followed by those scripts prefixed with S. When executing each
script in one of the /etc/rc[S0-6] directories, the /sbin/rc[S0-6]
script passes a single argu- ment. It passes the argument 'stop'
for scripts prefixed with K and the argument 'start' for scripts
prefixed with S. There is no harm in applying the same sequence
number to multiple scripts.