I am new to Linux and was introduced to the "&" recently. I have to run several traceroutes and store them in a single file, and I am curious if I am able to kick off these traceroutes in parallel?
I tried the following but the results in the generated file, are not kept apart? Well, that is what it seems to me.
traceroute -n -z 100 www.yahoo.com >> theLog.log &
traceroute -n -z 100 www.abc.com >> theLog.log &
Is what I am asking even possible to do? If so what commands should I be using?
Thanks for any direction given.
Perhaps you could investigate parallel (and tell us about your experience)?
If you are on Ubuntu, you can do sudo apt-get install moreutils to obtain parallel.
If you want it to run parallel is better to keep the intermediary results in separated files them join them at the end. The steps would be to start each trace to it's log file and store their pid, wait for them all to stop, them join the results, something like the following:
traceroute -n -z 100 www.yahoo.com > theLog.1.log & PID1=$!
traceroute -n -z 100 www.abc.com > theLog.2.log & PID2=$!
wait $PDI1 $PDI2
cat theLog.1.log theLog.2.log > theLog.log
rm theLog.2.log theLog.1.log
With the following command they are not really in parallel, but you can continue using your terminal, and the results are taken apart:
{ traceroute -n -z 100 www.yahoo.com; traceroute -n -z 100 www.abc.com; } >> theLog.log &
As you have it written, the behavior is undefined. You might try what enzotib posted, or try to have each write to their own file and cat them together at the end.
traceroutes in the #enzotib's answer are executed one at a time in sequence.
You can execute traceroutes in parallel using suggested by #rmk the parallel utility.
$ /usr/bin/time parallel traceroute -n -z 100 <hosts.txt >> parallel.log
24.78user 0.63system 1:24.04elapsed 30%CPU (0avgtext+0avgdata 37456maxresident)k
72inputs+72outputs (2major+28776minor)pagefaults 0swaps
Sequential analog is 5 times slower:
$ /usr/bin/time ./sequential.sh
24.63user 0.51system 7:19.09elapsed 5%CPU (0avgtext+0avgdata 5296maxresident)k
112inputs+568outputs (1major+8759minor)pagefaults 0swaps
Where sequential.sh is:
#!/bin/bash
( while read host; do traceroute -n -z 100 $host; done; ) <hosts.txt >>sequential.log
And hosts.txt is:
www.yahoo.com
www.abc.com
www.google.com
stackoverflow.com
facebook.com
youtube.com
live.com
baidu.com
wikipedia.org
blogspot.com
qq.com
twitter.com
msn.com
yahoo.co.jp
taobao.com
google.co.in
sina.com.cn
amazon.com
google.de
google.com.hk
Related
I'm wrote a bash script and I don't have a chance to download pssh. However, I need to run multiple ssh commands in parallel. There is no problem so far, but when I run ssh commands, I want to see the outputs sequentially from the remote machine. I mean one ssh has multiple outputs and they get mixed up because more than one ssh is running.
#!/bin/bash
pid_list=""
while read -r list
do
ssh user#$list 'commands'&
c_pid=$!
pid_list="$pid_list $c_pid"
done < list.txt
for pid in $pid_list
do
wait $pid
done
What should I add to the code to take the output unmixed?
The most obvious way to me would be to write the outputs in a file and cat the files at the end:
#!/bin/bash
me=$$
pid_list=""
while read -r list
do
ssh user#$list 'hostname; sleep $((RANDOM%5)); hostname ; sleep $((RANDOM%5))' > /tmp/log-${me}-$list &
c_pid=$!
pid_list="$pid_list $c_pid"
done < list.txt
for pid in $pid_list
do
wait $pid
done
cat /tmp/log-${me}-*
rm /tmp/log-${me}-* 2> /dev/null
I didn't handle stderr because that wasn't in your question. Nor did I address the order of output because that isn't specified either. Nor is whether the output should appear as each host finishes. If you want those aspects covered, please improve your question.
I want to create script, which one logging information from different IP's and in the same time it writes logs to different file, it should run like while:true, but when i start script it logs only first ip address in text file, what i already tried:
#!/bin/bash
IP=`cat IP.txt`
for i in $IP
do
/usr/bin/logclient -l all -f /root/$i.log $i 19999
done
IP.txt file contains:
x.x.x.x
x.x.x.x
x.x.x.x
x.x.x.x
It looks like your script should work as-is, and if logclient works like I think, it'll just create a number of different logs for each IP address. Doing a ls /root/*.log should reveal all the logs generated.
Parallelizing execution isn't something bash is particularly good at. It has job control for backgrounding tasks, but keeping track of those processes and not overloading your CPU/RAM can be tough.
GNU Parallel
If your system has it installed, I'd greatly suggest using GNU parallel. It will kick off one process for each CPU core to make parellizing jobs much easier. parallel only exits when all the children exit.
parallel /usr/bin/logclient -l all -f /root/{}.log {} 19999 ::::+ IP.txt
# all jobs finished, post-process the log (if wanted)
cat /root/*.log >> /root/all-ips.log
rather use while, than for. Something like this:
while read LINE; do /usr/bin/logclient -l all -f /root/$LINE.log $LINE 19999; done < IP.txt
I have an instrument that streams data out a tcp port, and I'd like to use standard tools in a script to determine if the stream is available.
Manually I use ncat, which promptly exits if the data stream isn't available.
Here's my initial bash script:
#!/bin/bash
ncat somehost 1234 >/dev/null &
pid=$!
sleep 1
if [ -d /proc/$pid/ ]; then
kill -KILL $pid
echo "It's alive, ALIVE\!"
# Launch clients
else
echo "He's dead, Jim."
# Perform resurrection
fi
It works, but I'm wondering if there is a simpler or better way to accomplish this that doesn't rely on job control or procfs or even ncat. I'd also like to know how much data was sent before a timeout exires.
And, yes, in bash ncat host port can be replaced by cat </dev/tcp/host/port, but I'd also like to avoid bash-isms (so it can work under busybox).
Another approach would be to use wc to count the lines/chars output by ncat, since it outputs only one line and exits if the connection can't be made. But I can't wait forever for termination if the stream is up, so I'd need to use something like timeout, which has its own complications when trying to access the output of a command.
Is there a "plain" approach that works with minimal dependencies?
Or should I write a simple tool instead? It would return the number of bytes read if a connection was made to host/port, a negative errno otherwise, and would support a wait time and protocol spec (tcp/udp). A return value of 0 would mean the connection was made, but no data arrived before the wait time expired (a good thing to know).
Or maybe patch ncat to do the above?
I use netcat in FreeBSD, which has a -z option that simply checks whether a port is open. This eliminates the background and sleep you're using in your script.
-z Specifies that nc should just scan for listening daemons, without
sending any data to them.
The option exists in netcat on an old Ubuntu box here, so that might be an option for you.
$ nc -z somehost 1234 && echo "It's alive, ALIVE!"
I don't see an equivalent option in ncat. You may be able to compile netcat into busybox; it's much smaller than ncat, at least on my systems:
Linux:
$ ls -lH `which nc ncat`
-rwxr-xr-x 1 root root 31296 2010-02-21 01:32 /bin/nc
-rwxr-xr-x 1 root root 130448 2009-11-06 04:39 /usr/bin/ncat
FreeBSD:
ls -l `which nc ncat`
-r-xr-xr-x 1 root wheel 28112 Jan 15 14:53 /usr/bin/nc
-rwxr-xr-x 1 root wheel 182775 Mar 19 2012 /usr/local/bin/ncat
Of course, this doesn't help you check the amount of traffic that has come through the stream, or to analyse its content. For that, I think your solution is innovative and reasonable, though the challenge you've presented may not be resolvable easily.
You can use ps to avoid the procfs dependency, and you can store things in a temp file for analysis rather. Note that busybox should include a mktemp, but you should probably check the options. I haven't tested this:
#!/bin/sh
TMPFILE=`mktemp /tmp/str.XXXX`
trap "rm -f $TMPFILE" 0 1 2 3 15
nc somehost 1234 > $TMPFILE &
pid=$!
sleep 1
if ps $pid >/dev/null; then
kill -KILL $pid
echo -n "It's alive, "
if [ -s $TMPFILE ]; then
echo "ALIVE!"
else
echo "but unresponsive."
fi
else
echo "He's dead, Jim."
fi
That's all I've got. And it's not much different from what you already have.
Another option might be to build a custom tool that you might be able to compile into your own busybox, but that'll be a c question rather than a shell question. :-)
This is the first question that I post here. I tried to do a throughout search, but if I haven't (and the answer is obvious somewhere else), please just let me know.
I have a script that runs a program for me, here it is:
csv_file=../data/teste_nohup.csv
trace_file=../data/gnp.trace
declare -i n=100
declare -i p=1
declare -i counter=0
while [ $counter -lt 3 ];
do
n=100
while true
do
nice -19 sage gnptest.py ${n} ${p} | tee -a ${csv_file}
notify-send "finished test gnp ${n} ${p}"
done
done
So, what I'm trying to do is run the gnptest.py program a few times, and have the result be written to the csv_file.
The problem is, that depending on the input, the program may take a long time to complete. So I'd like to connect to the server over ssh, start the program, close the terminal, and check the output file from time to time.
I've tried nohup and disown. Nohup creates a huge nohup.out file, full with errors that I don't get while normally running the script (it complains about using the -lt operand, for example). But the biggest problem that I'm facing is that no command (nohup ou disown -h) is executing the program and sending the output to the file that I've specified in the csv_file variable, which is being done using the tee command. Also, none of them seem to continue running after I logout...
Any help will be much appreciated.
Thanks in advance!!
i hv just joined so cannt add comment
Please try by using redirection instead of tee in script
And to get rid of Nohup.out use following to run script
nohup script.sh > /dev/null 2>&1 &
If above produces error use
nohup script.sh > /dev/null 2>&1 </dev/null &
Hope this will help.
I want to create bash script that will verify by ping list of IP’s
The problem is that ping to any address take few seconds ( in case no ping answer ) in spite I defined the ping as the following:
Ping –c 1 126.78.6.23
The example above perform ping only one time – but the problem is the time , waiting few seconds until ping ended ( if no answer )
In my case this is critical because I need to check more than 150 IP’s ( usually more 90% of the IP’s are not alive )
So to check 150 IP’s I need more than 500 seconds
Please advice if there is some good idea how to perform ping quickly
remark my script need to run on both OS ( linux and solaris )
The best idea is to run ping in parallel
and then save the result in a file.
In this case your script will run not longer than a second.
for ip in `< list`
do
( ping -c1 $ip || echo ip >> not-reachable ) &
done
Update. In Solaris -c has other meaning, so for solaris you need
run ping other way:
ping $ip 57 1
(Here, 57 is the size of the packet and 1 is the number of the packets to be sent).
Ping's syntax in Solaris:
/usr/sbin/ping -s [-l | -U] [-adlLnrRv] [-A addr_family]
[-c traffic_class] [-g gateway [ -g gateway...]]
[-F flow_label] [-I interval] [-i interface] [-P tos]
[-p port] [-t ttl] host [data_size] [npackets]
You can make a function that aggregates the two methods:
myping()
{
[ `uname` = Linux ] && ping -c 1 "$i" || ping "$ip" 57 1
}
for ip in `< list`
do
( myping $ip || echo ip >> not-reachable ) &
done
Another option, don't use ping directly but use ICMP module from some language.
You can use for example Perl + Net::Ping module from Perl:
perl -e 'use Net::Ping; $timeout=0.5; $p=Net::Ping->new("icmp", $timeout) or die bye ; print "$host is alive \n" if $p->ping($host); $p->close;'
Does Solaris ship with coreutils OOTB these days? Then you can use timeout to specify an upper limit:
timeout 0.2s ping -c 1 www.doesnot.exist >/dev/null 2>&1
You could use hping3, which is scriptable (in Tcl).
As already stated, a simple way is to overcome the timing issue run the ping commands in parallel.
You already have the syntax for Linux (iputils) ping.
With Solaris, the proper option to send a single ping would be
ping -s 126.78.6.23 64 1
Installing nmap from sources would provide a more powerful alternative though.