I would like to make program with option -j after checking how many cores cpu has got.
#!/bin/bash
x="grep -c ^processor /proc/cpuinfo"
make -j${x}
x variable display number of cores. make -j${x} not working
You'll want to capture the output of the command into a string through a subshell:
#!/bin/bash
x=$(grep -c ^processor /proc/cpuinfo)
make -j"${x}"
Related
I'm writing a small bash script where I am compiling 2 programs then executing them in the background within a bash script. These 2 programs do output some generic text. However I need to prefix these outputs like PROGRAM1: xxxxx. How do I achieve that? I have found several answers here however they weren't exactly applicable to this situation.
Here's the code:
#!/bin/bash
echo "This program compiles 2 programs, executes them (executes the 2nd one first, then sleeps for 0.01 second which then executes program 1), then observes the outputs"
gcc -O3 -std=c11 one.c -o one
gcc -O3 -std=c11 two.c -o two
./two &
sleep 0.01
TWO_PID=$(pgrep two)
./one $TWO_PID
#"Prefix output statements here"
#add code here
#Cleanup
rm -f one two
You can do something like this
#! /bin/bash
# ...
label() {
while read -r l; do
echo "$1: $l"
done
}
./two | label "two" &
two_pid=$(pgrep two)
./one $two_pid | label "one"
You don't need the sleep and be careful as pgrep could match more than one process.
Also, instead of compiling and removing you should use make.
I am trying to run a script (1.sh)
spin -a /home/files/1/1.pml;
gcc -O2 -DXUSAFE -DSAFETY -DNOCLAIM -w -o pan pan.c >log1.txt;
./pan -m100000 >log2.txt;
spin -p -s -r -X -v -n123 -l -g -k /home/files/1/1.pml.trail \
-u10000 /home/files/1/1.pml >log3.txt;
The command spin -a ...; generates temporary files (pan.c, pan.h) which is used by the next gcc -O2.. command. If I run the script in terminal it creates the temporary files in the same location.
I want to run multiple scripts parallelly. I tried two things, first to write a script to run then in a loop in background (parallel.sh)
for((i=1;i<1800;i++))
do
/home/files/$i/$i.sh &
done
and secondly use parallel gnu parallel -j0 sh /home/files/{}/{}.sh ::: {1..1800}.
Both method created temp file in the location from where they were called from instead of the script location.
For example if I run the script 'parallel.sh' from home/files the temp file are created in "home/files" instead of the location "home/files/1","home/files/2", etc.
Please suggest a method so that the temporary file generated by the script 1.sh,2.sh,.. are created in the directory /home/file/1/, /home/files/2/,.. respectively while I run the parallel script parallel.sh or parallel GNU in terminal from location /home.
The trick is to change the working directory for each command.
When your computer can really run up to 1800 such processes at the same time without heating up the climate:
for i in {1..1800}; do (cd $i && ./$i.sh) & done
When running in parallel, and your processes are cpu-bound, it usually does not gain throughput when running more than the number of processors:
seq 1 1800 | xargs -n1 -P8 -I% sh -c 'cd % && ./%.sh'
Try:
parallel 'cd /home/files/{}; sh {}.sh' ::: {1..1800}
It will run one process per core, and may be faster than '-j0' (only testing can tell with certainty).
If your scripts only vary by the number, consider rewriting it as a general script or bash function that takes the number as an argument:
spinit() {
num=$1
spin -a /home/files/$num/$num.pml;
gcc -O2 -DXUSAFE -DSAFETY -DNOCLAIM -w -o pan pan.c >log1.txt;
./pan -m100000 >log2.txt;
spin -p -s -r -X -v -n123 -l -g -k /home/files/$num/$num.pml.trail \
-u10000 /home/files/$num/$num.pml >log3.txt;
}
export -f spinit
parallel 'cd /home/files/{}; spinit {}' ::: {1..1800}
TASK - SSH to 650 Servers and fetch few details from them and then write the completed server name in different file. How can do it in faster way? If I do normal ssh it takes 7 Minutes. So, I read about awk and wrote following 2 codes.
Could you please explain me the difference in the following codes?
Code 1 -
awk 'BEGIN{done_file="/home/sarafa/AWK_FASTER/done_status.txt"}
{
print "blah"|"ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=1 -o ConnectionAttempts=1 "$0" uname >/dev/null 2>&1";
print "$0" >> done_file
}' /tmp/linux
Code 2 -
awk 'BEGIN{done_file="/home/sarafa/AWK_FASTER/done_status.txt"}
{
"ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=1 -o ConnectionAttempts=1 "$0" uname 2>/dev/null"|getline output;
print output >> done_file
}' /tmp/linux
When I run these codes for 650 Servers, Code 1 takes - 30 seconds and Code 2 takes 7 Minutes ?
Why is there so much time difference ?
File - /tmp/linux is a list of 650 servers
Updated Answer - with thanks to #OleTange
This form is preferable to my suggestion:
parallel -j 0 --tag --slf /tmp/linux --nonall 'hostname;ls'
--tag Tag lines with arguments. Each output line will be prepended
with the arguments and TAB (\t). When combined with --onall or
--nonall the lines will be prepended with the sshlogin
instead.
--nonall --onall with no arguments. Run the command on all computers
given with --sshlogin but take no arguments. GNU parallel will
log into --jobs number of computers in parallel and run the
job on the computer. -j adjusts how many computers to log into
in parallel.
This is useful for running the same command (e.g. uptime) on a
list of servers.
Original Answer
I would recommend using GNU Parallel for this task, like this:
parallel -j 64 -k -a /tmp/linux 'echo ssh user#{} "hostname; ls"'
which will ssh into 64 hosts in parallel (you can change the number), run hostname and ls on each and then give you all the results in order (-k switch).
Obviously remove the echo when you see how it works.
I've been trying to display the type of terminal being used as the name only. For instance if I was using konsole it would display konsole. So far I've been using this command.
pstree -A -s $$
That outputs this.
systemd---konsole---bash---pstree
I have the following that can extract konsole from that line
pstree -A -s $$ | sed 's/systemd---//g;s/---.*//g' | head -1
and that outputs konsole properly. But some people have output from just the pstree command that can look like this.
systemd---kdeinit4---terminator---bash---pstree
or this
systemd---kdeinit4---lxterminal---bash---pstree
and then when I add the sed command it extracts kdeinit4 instead of terminator. I can think of a couple scenarios to extract the type of terminal but none that don't contain conditional statements to check for specific types of terminals. The problem I'm having is I can't accurately predict how many non or non-relative things may be infront or behind of the terminal name or what they will be nor can I accurately predict what the terminal name will be. Does anyone have any ideas on a solution to this?
You could use
ps -p "$PPID" -o comm=
Or
ps -p "$PPID" -o fname=
If your shell does not have PPID variable set you could get it with
ps -p "$(ps -p "$$" -o ppid= | sed 's|\s\+||')" -o fname=
Another theory is that the parent process of the current shell that doesn't belong to the same tty as the shell could actually be the one that produces the virtual terminal, so we could find it like this as well:
#!/bin/bash
shopt -s extglob
SHELLTTY=$(exec ps -p "$$" -o tty=)
P=$$
while read P < <(exec ps -p "$P" -o ppid=) && [[ $P == +([[:digit:]]) ]]; do
if read T < <(exec ps -p "$P" -o tty=) && [[ $T != "$SHELLTTY" ]]; then
ps -p "$P" -o comm=
break
fi
done
I don't know how to isolate the terminal name on your system, but as a parsing exercise, and assuming the terminal is directly running your bash you could pipe the pstree output through:
awk -F"---bash---" ' NF == 2 { count = split( $1, arr, "---" ); print arr [count]; }'
This will find the word prior to the "---bash---" which in your examples is
konsole
terminator
lxterminal
If you want different shell types, you could expand the field separator to include them like:
awk -F"---(bash|csh)---" ' NF == 2 { count = split( $1, arr, "---" ); print arr[count]; }'
Considering an imaginary line like:
systemd---imaginary---monkey---csh---pstree
the awk would find "monkey" as the terminal name as well as anything from your test set.
No guarantees here, but I think this will work most of the time, on linux:
ps -ocomm= $(lsof -tl /proc/$$/fd/0 | grep -Fxf <(lsof -t /dev/ptmx))
A little explanation is probably in order, but see man ps, man lsof and (especially) man pts for information.
/dev/ptmx is a pseudo-tty master (on modern linux systems, and some other unix(-like) systems). A program will have one of these open if it is a terminal emulator, a telnet/ssh daemon, or some other program which needs a captive terminal (screen, for example). The emulator writes to the pseudo-tty master what it wants to "type", and reads the result from the pseudo-tty slave.
/proc/$$/fd/0 is stdin of process $$ (i.e. the shell in which the command is executed). If stdin has not been redirected, this will be a symlink to some slave pseudotty, /dev/pts/#. That is the other side of the /dev/ptmx device, and consequently all of the programs listed above which have /dev/ptmx open will also have some /dev/pts/# slave open as well. (You might think that you could use /dev/stdin or /dev/fd/0 instead of /proc/$$/fd/0, but those would be opened by lsof itself, and consequently would be its stdin; because of the way lsof is implemented, that won't work.) The -l option to lsof causes it to follow symlinks, so that will cause it to show the process which have the same pts open as the current shell.
The -t option to lsof causes it to produce "terse" output, consisting only of pids, one per line. The -Fx options to grep cause it to match strings, rather than regex, and to force a full line match; the -f FILE option causes it to accept the strings to match from FILE (which in this case is a process substitution), one per line.
Finally, ps -ocomm= prints out the "command" (chopped, by default, to 8 characters) corresponding to a pid.
In short, the command finds a list of terminal emulators and other master similar programs which have a master pseudo-tty, and a list of processes which use the pseudo-tty slave; finds the intersection between the two, and then looks up the command name for whatever results.
curTerm=$(update-alternatives --query x-terminal-emulator | grep '^Best:')
curTerm=${curTerm##*/}
printf "%s\n" "$curTerm"
And the result is
terminator
Of course it can be different.
Now you can use $curTerm variable in your sed command.
But I am not sure if this is going to work properly with symlinks.
I tried to use qsub -N "compile-$*" in Makefile and it gives the following error
because $* equals to "compile-obj/linux/flow" in this case.
qsub: ERROR! argument to -N option must not contain /
The whole command which I am using is:-
qsub -P bnormal -N "compile-obj/linux/flow" -cwd -now no -b y -l cputype=amd64 -sync yes -S /bin/sh -e /remote//qsub_files/ -o /remote/qsub_files/
Any idea how to include slash in naming while running qsub?
Thanks
I'm not familiar with qsub, but make just executes what command you supply it. So I suspect you constructed illegal qsub command.
Maybe Automatic-Variables section of GNU make can help you too.
Adding a whole rule to question can help.
I resolved the problem by manipulating the name passed to -N option by replacing / with -. It works for me. Thanks.