LocationDevice array includes all the devices in the location (from a text file)
what I want to do is to to save the statistics results in the PingValue[$k] array and to print them out in the next loop (it will be something other than printing of course) I want to find away to do send the declaration command to the background so the 1000 devices pings happen for less than 10 seconds (-c 5 for each device) then i get to call my arrays in process them in the second loop.
I tried to echo PingValue[$k]=`ping -c 5 -q ${LocationDevice[$k]}` but didn't work and the array didn't have the value and ALL the results printed after couple or seconds
for ((k=0;k<${#LocationDevice[#]};k++)) do
PingValue[$k]=`ping -c 5 -q ${LocationDevice[$k]}`
done
sleep 10
for ((i=0;i<${#LocationDevice[#]};i++)) do
echo "${PingValue[$i]}"
done
Combining the output from multiple parallel background processes into a single array is tricky because those processes can't easily access any shared memory. The best way, I think, is to build a large pipeline to synchronize the data and to use a null value to delineate the output from each process. The best way to launch such a pipeline is recursively. However there won't be any way to quickly read 1000 results from the pipeline into the final array, that takes a while. Instead you can do your processing as you read each item from the pipeline (while building the array if you still need it after). Try this:
PingHosts() {
[ "$1" ] || return
host="$1"; shift
rest=("$#")
(
r=$(ping -c 5 -q "$host")
echo "$r"
printf "\0"
cat
) < <(PingHosts "${rest[#]}" &)
}
for ((i=0; i<${#LocationDevice[#]}; ++i)) do
read -r -d $'\0' pingresult
PingValue[$i]="$pingresult"
echo "Processing result for $i:"
echo "${PingValue[$i]}"
echo "-----------------------"
done < <(PingHosts "${LocationDevice[#]}")
Related
My goal is to run a script every time I push the R key on my keyboard:
#!/bin/bash
tail -50 .local/share/binding\ of\ isaac\ afterbirth+/log.txt | grep "on seed" | awk '{print $(NF-1) $NF}' > /home/fawk/seed
I already tried to do this with a program called xbindkeys but it would interfere and pause the game.
Then I tried it with another bash script using the read command to execute the first script but my knowledge isn't that great so far. Reading and experimenting for hours didn't lead anywhere.
The purpose is to feed a game-seed to OBS. The easiest way seems to create a (text-)file containing the seed for OBS to pick up. All I want is that seed (1st script) to be written into a file.
I tried
#!/bin/bash
while :
do
read -r 1 k <&1
if [[ $k = r ]] ; then
exec /home/fawk/seedobs.sh
fi
done
and many other variations but didn't get closer to a solution.
I am trying to create a wordlist consisting of the same password followed by a 4-digit numeric pin. The pin goes through every possible combination of 10,000 variations. The desired output should be like this:
UoMYTrfrBFHyQXmg6gzctqAwOmw1IohZ 1111
UoMYTrfrBFHyQXmg6gzctqAwOmw1IohZ 1112
UoMYTrfrBFHyQXmg6gzctqAwOmw1IohZ 1113
and so on.
I created a shell script that almost get this, but awk doesn't seem to like having a variable passed through it, and seems to just print out every combination when called. This is the shell script:
#!/bin/bash
# Creates 10,000 lines of the bandit24pass and every possible combination
# Of 4 digits pin
USER="UoMYTrfrBFHyQXmg6gzctqAwOmw1IohZ"
PASS=$( echo {1,2,3,4,5,6,7,8,9}{1,2,3,4,5,6,7,8,9}{1,2,3,4,5,6,7,8,9}{1,2,3,4,5,6,7,8,9} | awk '{print $I}' )
for I in {1..10000};
do
echo "$USER $PASS"
done
I though $I would translate to $1 for the first run of the loop, and increment upwards through each iteration. Any help would be greatly appreciated.
I though $I would translate to $1 for the first run of the loop, and increment upwards through each iteration.
No, command substitutions are expanded once; like, when you do foo=$(echo), foo is an empty line, not a reference to echo.
This whole task could be achieved by a single call to printf btw.
printf 'UoMYTrfrBFHyQXmg6gzctqAwOmw1IohZ %s\n' {1111..9999}
Tyr this
$echo $user
UoMYTrfrBFHyQXmg6gzctqAwOmw1IohZ
$for i in {1000..9999}; do echo $user $i; done;
Assuming that I have files with 100 lines. There are a lot of lines that repeat themselves in the file, and only one line that does not.
I want to find the line that shows only once. Is there a command for that or do I have to build some complicated loop as below?
My code so far:
#!/bin/bash
filename="repeat_lines.txt"
var="$(wc -l <$filename )"
echo "length:" $var
#cp ex4.txt ex4_copy.txt
for((index=0; index < var; index++));
do
one="$(head -n $index $filename | tail -1)"
counter=0
for((index2=0; index2 < var; index2++));
do
two="$(head -n $index2 $filename | tail -1)"
if [ "$one" == "$two" ]; then
counter=$((counter+1))
fi
done
echo $one"is "$counter" times in the text: "
done
If I understood your question correctly, then
sort repeat_lines.txt | uniq -u should do the trick.
e.g. for file containing:
a
b
a
c
b
it will output c.
For further reference, see sort manpage, uniq manpage.
You've got a reasonable answer that uses standard shell tools sort and uniq. That's probably the solution you want to use, if you want something that is portable and doesn't require bash.
But an alternative would be to use functionality built into your bash shell. One method might be to use an associative array, which is a feature of bash 4 and above.
$ cat file.txt
a
b
c
a
b
$ declare -A lines
$ while read -r x; do ((lines[$x]++)); done < file.txt
$ for x in "${!lines[#]}"; do [[ ${lines["$x"]} -gt 1 ]] && unset lines["$x"]; done
$ declare -p lines
declare -A lines='([c]="1" )'
What we're doing here is:
declare -A creates the associative array. This is the bash 4 feature I mentioned.
The while loop reads each line of the file, and increments a counter that uses the content of a line of the file as the key in the associative array.
The for loop steps through the array, deleting any element whose counter is greater than 1.
declare -p prints the details of an array in a predictable, re-usable format. You could alternately use another for loop to step through the remaining array elements (of which there might be only one) in order to do something with them.
Note that this solution, while fine for small files (say, up to a few thousand lines), may not scale well for very large files of, say, millions of lines. Bash isn't the fastest at reading input this way, and one must be cognizant of memory limits when using arrays.
The sort alternative has the benefit of memory optimization using files on disk for extremely large files, at the expense of speed.
If you're dealing with files of only a few hundred lines, then it's hard to predict which solution will be faster. In the end, the form of output may dictate your choice of solution. The sort | uniq pipe generates a list to standard output. The bash solution above generates the same list as keys in an array. Otherwise, they are functionally equivalent.
I need to parse 70 identically formatted files (different data), repeatedly, to process some information on demand from each file. I.e. (as a simplified example)...
find /dir -name "MYDATA.bam" | while read filename; do
dir=$(echo ${filename} | awk -F"/" '{ print $(NF-1)}')
ARRAY[$dir]=$(samtools view ${filename} | head -1)
done
Since it's 70 files, I wanted each samtools view command to run as an independent thread...so I didn't have to wait for each command to finish (each command takes around 1 second.) Something like...
# $filename will = "/dir/DATA<id#>/MYDATA.bam"
# $dir then = "DATA<id#>" as the ARRAY key.
find /dir -name "MYDATA.bam" | while read filename; do
dir=$(echo ${filename} | awk -F"/" '{ print $(NF-1)}')
command="$(samtools view ${filename} | head -1)
ARRAY[$dir]=$command &
done
wait # To get the array loaded
(... do stuff with $ARRAY...)
But I can't seem to find the syntax to get all the commands called in the background, but still have "result" receive the (correct) output.
I'd be running this on a slurm cluster, so I WOULD actually have 70 cores available to run each command independently (theoretically making that step take 1-2 seconds concurrently, instead of 70 seconds consecutively).
You can do this simply with GNU Parallel like this:
#!/bin/bash
doit() {
dir=$(echo "$1" | awk -F"/" '{print $(NF-1)}')
result=$(samtools view "$1" | head -1)
echo "$dir:$result"
}
# export doit() function for subshells of "parallel" to use
export -f doit
# find the files and pass, null-terminated, to GNU Parallel
find somewhere -name "MYDATA.bam" -print0 | parallel -0 doit {}
It will run one copy of samtools per CPU core you have available, but you can easily change that, with parallel -j 8 if you just want 8 at a time, for example.
If you want the outputs in order, use parallel -k ...
I am not familiar with slurm clusters, so you may have to read up on how to tell GNU Parallel about your nodes, or let it just run 8 at a time or however many cores your main node has.
Capturing the output of a process even when spawned in the background blocks the shell. Here a small example:
echo "starting to sleep in the background"
sleep 2 &
echo "some printing in the foreground"
wait
echo "done sleeping"
This will produce the following output:
starting to sleep in the background
some printing in the foreground
<2 second wait>
done sleeping
If however you capture like this:
echo "starting to sleep in the background"
output=$(sleep 2 &)
echo "some printing in the foreground"
wait
echo "done sleeping"
The following happens:
starting to sleep in the background
<2 second wait>
some printing in the foreground
done sleeping
The actual waiting happened on the assignment of the output. By the time the wait statement is reached there is no more background process and thus no waiting.
So one way would be to pipe the output into files and stitch them back together
after the wait. This is a bit awkward.
A simpler solution might be to use GNU Parallel, a tool that deals with
collecting the output of parallel processes. It works particularly well when the output is line based.
You should be able to do this with just Bash. This snippet show how you can run each command in the background and write the results to stdout. The inner loop reads in these results and adds them to your array. You'll probably have to tweak this to make it work.
while read -r dir && read -r data; do
ARRAY[$dir]="$data"
done < <(
# sub shell level one
find /dir -name "MYDATA.bam" | while read filename; do
(
# sub shell level two
# run each task in parallel, output will be in the following format
# "directory"
# "result"
# ...
dir=$(awk -F"/" '{ print $(NF-1)}' <<< "$filename")
printf "%s\n%s\n" \
"$dir" "$(samtools view "$filename" | head -1)"
) &
done
)
The key is that ( command; command ) & runs each command in a new sub shell in the background, so the top level shell can continue to the next task.
The < <(command) allows us to redirect the stdout of a subprocess to the stdin of another shell command. This is how we can read the results into our variable and have the variable be available later.
I have a program which will be running continuously writing output to stdout, and I would like to continuously take the last set of output from it and write this to a file. Each time a new set of output comes along, I'd like to overwrite the same file.
An example would be the following command:
iostat -xkd 5
Every 5 seconds this prints some output, with a blank line after each one. I'd like the last "set" to be written to a file, and I think this should be achievable with something simple.
So far I've tried using xargs, which I could only get to do this with a single line, rather than a group of lines delimited by something.
I think it might be possible with awk, but I can't figure out how to get it to either buffer the data as it goes along so it can write it out using system, or get it to close and reopen the same output file.
EDIT: To clarify, this is a single command which will run continuously. I do NOT want to start a new command and read the output of that.
SOLUTION
I adapted the answer from #Bittrance to get the following:
iostat -xkd 5 | (while read r; do
if [ -z "$r" ]; then
mv -f /tmp/foo.out.tmp /tmp/foo.out
else
echo "$r" >> /tmp/foo.out.tmp
fi
done)
This was basically the same, apart from detecting the end of a "section" and writing to a temp file so that whenever an external process tried to read the output file it would be complete.
Partial answer:
./your_command | (while read r ; do
if ... ; then
rm -f foo.out
fi
echo $r >> foo.out
done)
If there is a condition (the ...) such that you know that you are receiving the first line of the "set" this would work.
Why not:
while :; do
iostat -xkd > FILE
sleep 5
done
If you're set on using awk the following writes each output from iostat to a numbered file:
iostat -xkd 5 | awk -v RS=$'\n'$'\n' '{ print >NR; close(NR) }'
Note: the first record is the header that iostat outputs.
Edit In reply to comment, I tested this perl solution to work:
#!/usr/bin/perl
use strict;
use warnings;
$|=1;
open(STDOUT, '>', '/tmp/iostat.running') or die $!;
while(<>)
{
seek (STDOUT, 0, 0) if (m/^$/gio);
print
}
So now you can
iostat -xkd 5 | myscript.pl&
And watch the snapshots:
watch -n10 cat /tmp/iostat.running
If you wanted to make sure that enough space is overwritten at the end (since the output length might vary slightly each time), you could print some padding at the end, like e.g.:
print "***"x40 and seek (STDOUT, 0, 0) if (m/^$/gio);
(snipped old answer text since apparently it was confusing people)