Run ssh in shell script in parallel and set remote variables - linux

Question

I'm writing a script to read from a input file, which contains ~1000 lines of host info. The script ssh to each host, cd to the remote hosts log directory and cat the latest daily log file. Then I redirect the cat log file locally to do some pattern matching and statistics.
The simplified structure of my program is a while loop looks like this:
while read host
do
ssh -n name#$host "cd TO LOG DIR AND cat THE LATEST LOGFILE" | matchPattern
done << EOA
$(awk -F, '{print &7}' $FILEIN)
EOA
where matchPattern is a function to match pattern and do statistics.
Right now I got 2 questions for this:
1) How to find the latest daily log file remotely? The latest log file name matches xxxx2012-05-02.log and is newest created, is it possible to do ls remotely and find the file matching the xxxx2012-05-02.log file name?(I can do this locally but get jammed when appending it to ssh command) Another way I could come up with is to do
cat 'ls -t | head -1' or
cat $(ls -t | head -1)
However if I append this to ssh, it will list my local newest created file name, can we set this to a remote variable so that cat will find the correct file?
2) As there are nearly 1000 hosts, I'm wondering can I do this in parallel (like to do 20 ssh at a time and do the next 20 after the first 20 finishes), appending & to each ssh seems not suffice to accomplish it.
Any ideas would be greatly appreciated!
Follow up:
Hi everyone, I finally find a crappy way do solve the first problem by doing this:
ssh -n name#$host "cd $logDir; cat *$logName" | matchPattern
Where $logName is "today's date.log"(2012-05-02.log). The problem is that I can only use local variables within the double quotes. Since my log file ends with 2012-05-02.log, and there is no other files ends with this suffix, I just do a blindly cat *2012-05-02.log on remote machine and it will cat the desired file for me.

Answer 1

For your first question,
ssh -n name#$host 'cat $(ls -t /path/to/log/dir/*.log | head -n 1)'
should work. Note single quotes around the remote command.
For your second question, wrap all the ssh | matchPattern | analyse stuff into its own function, then iterate over it by
outstanding=0
while read host
do
sshMatchPatternStuff &
outstanding=$((outstanding + 1))
if [ $outstanding -ge 20 ] ; then
wait
outstanding=$((outstanding - 1))
fi
done << EOA
$(awk -F, '{print &7}' $FILEIN)
EOA
while [ $outstanding -gt 0 ] ; do
wait
outstanding=$((outstanding - 1))
done
(I assume you're using bash.)
It may be better to separate the ssh | matchPattern | analyse stuff into its own script, and then use a parallel variant of xargs to call it.

Answer 2

for your second question, take a look at parallel distributed shell:
http://sourceforge.net/projects/pdsh/

Answer 3

If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you can do this:
parallel -j0 --nonall --slf <(awk -F, '{print $7}' servers.txt) 'cd logdir; cat `ls -t | head -1` | grep pattern'
This way you get the matching done on the remote server. If you prefer to transfer the full log file and do the matching locally, simply move the grep outside:
parallel -j0 --nonall --slf <(awk -F, '{print $7}' servers.txt) 'cd logdir; cat `ls -t | head -1`' | grep pattern
You can install GNU Parallel simply by:
wget http://git.savannah.gnu.org/cgit/parallel.git/plain/src/parallel
chmod 755 parallel
cp parallel sem
Watch the intro videos for GNU Parallel to learn more:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1