I have such textfile:
313 "88.68.245.12"
189 "87.245.108.11"
173 "84.134.230.11"
171 "87.143.88.4"
158 "77.64.132.10"
....
I want to grep only the IP from the first 10 lines, run whois over the IP adress and from that output I want to grep the line where it says netname.
How can I achieve this?
Just loop through the file with while - read:
while IFS='"' read -r a ip c
do
echo "ip: $ip"
whois "$ip" | grep netname
done < <(head -10 file)
This is giving IFS='"' so that the field separator is a double quote ". This way, the values within double quotes will be stored in $ip.
Then, we print the ip and perform the whois | grep thing.
Finally, we feed the loop with head -10 file, so that we just read the first 10 lines.
Related
I'm studying bash scripting and I'm stuck fixing an exercise of this site: https://ryanstutorials.net/bash-scripting-tutorial/bash-variables.php#activities
The task is to write a bash script to output a random word from a dictionary whose length is equal to the number supplied as the first command line argument.
My idea was to create a sub-dictionary, assign each word a number line, select a random number from those lines and filter the output, which worked for a similar simpler script, but not for this.
This is the code I used:
6 DIC='/usr/share/dict/words'
7 SUBDIC=$( egrep '^.{'$1'}$' $DIC )
8
9 MAX=$( $SUBDIC | wc -l )
10 RANDRANGE=$((1 + RANDOM % $MAX))
11
12 RWORD=$(nl "$SUBDIC" | grep "\b$RANDRANGE\b" | awk '{print $2}')
13
14 echo "Random generated word from $DIC which is $1 characters long:"
15 echo $RWORD
and this is the error I get using as input "21":
bash script.sh 21
script.sh: line 9: counterintelligence's: command not found
script.sh: line 10: 1 + RANDOM % 0: division by 0 (error token is "0")
nl: 'counterintelligence'\''s'$'\n''electroencephalograms'$'\n''electroencephalograph': No such file or directory
Random generated word from /usr/share/dict/words which is 21 characters long:
I tried in bash to split the code in smaller pieces obtaining no error (input=21):
egrep '^.{'21'}$' /usr/share/dict/words | wc -l
3
but once in the script line 9 and 10 give error.
Where do you think is the error?
problems
SUBDIC=$( egrep '^.{'$1'}$' $DIC ) will store all words of the given length in the SUBDIC variable, so it's content is now something like foo bar baz.
MAX=$( $SUBDIC | ... ) will try to run the command foo bar baz which is obviously bogus; it should be more like MAX=$(echo $SUBDIC | ... )
MAX=$( ... | wc -l ) will count the lines; when using the above mentioned echo $SUBDIC you will have multiple words, but all in one line...
RWORD=$(nl "$SUBDIC" | ...) same problem as above: there's only one line (also note #armali's answer that nl requires a file or stdin)
RWORD=$(... | grep "\b$RANDRANGE\b" | ...) might match the dictionary entry catch 22
likely RWORD=$(... | awk '{print $2}') won't handle lines containing spaces
a simple solution
doing a "random sort" over the all the possible words and taking the first line, should be sufficient:
egrep "^.{$1}$" "${DIC}" | sort -R | head -1
MAX=$( $SUBDIC | wc -l ) - A pipe is used for connecting a command's output, while $SUBDIC isn't a command; an appropriate syntax is MAX=$( <<<$SUBDIC wc -l ).
nl "$SUBDIC" - The argument to nl has to be a filename, which "$SUBDIC" isn't; an appropriate syntax is nl <<<"$SUBDIC".
This code will do it. My test dictionary of words is in file file. It's a good idea to get all words of a given length first but put them in an array not in var. And then get a random index and echo it.
dic=( $(sed -n "/^.\{$1\}$/p" file) )
ind=$((0 + RANDOM % ${#dic[#]}))
echo ${dic[$ind]}
I am also doing this activity and I create one simple solution.
I create the script.
#!/bin/bash
awk "NR==$1 {print}" /usr/share/dict/words
Here if you want a random string then you have to run the script as per the below command from the terminal.
./script.sh $RANDOM
If you want the print any specific number string then you can run as per the below command from the terminal.
./script.sh 465
cat /usr/share/dict/american-english | head -n $RANDOM | tail -n 1
$RANDOM - Returns a different random number each time is it referred to.
this simple line outputs random word from the mentioned dictionary.
Otherwise as umläute mentined you can do:
cat /usr/share/dict/american-english | sort -R | head -1
I have a file with unknown number of lines(but even number of lines). I want to print them side by side based on total number of lines in that file. For example, I have a file with 16 lines like below:
asdljsdbfajhsdbflakjsdff235
asjhbasdjbfajskdfasdbajsdx3
asjhbasdjbfajs23kdfb235ajds
asjhbasdjbfajskdfbaj456fd3v
asjhbasdjb6589fajskdfbaj235
asjhbasdjbfajs54kdfbaj2f879
asjhbasdjbfajskdfbajxdfgsdh
asjhbasdf3709ddjbfajskdfbaj
100
100
150
125
trh77rnv9vnd9dfnmdcnksosdmn
220
225
sdkjNSDfasd89asdg12asdf6asdf
So now i want to print them side by side. as they have 16 lines in total, I am trying to get the results 8:8 like below
asdljsdbfajhsdbflakjsdff235 100
asjhbasdjbfajskdfasdbajsdx3 100
asjhbasdjbfajs23kdfb235ajds 150
asjhbasdjbfajskdfbaj456fd3v 125
asjhbasdjb6589fajskdfbaj235 trh77rnv9vnd9dfnmdcnksosdmn
asjhbasdjbfajs54kdfbaj2f879 220
asjhbasdjbfajskdfbajxdfgsdh 225
asjhbasdf3709ddjbfajskdfbaj sdkjNSDfasd89asdg12asdf6asdf
paste command did not work for me exactly, (paste - - - - - - - -< file1) nor the awk command that I used awk '{printf "%s" (NR%2==0?RS:FS),$1}'
Note: The number of lines in a file dynamic. The only known thing in my scenario is, they are even number all the time.
If you have the memory to hash the whole file ("max" below):
$ awk '{
a[NR]=$0 # hash all the records
}
END { # after hashing
mid=int(NR/2) # compute the midpoint, int in case NR is uneven
for(i=1;i<=mid;i++) # iterate from start to midpoint
print a[i],a[mid+i] # output
}' file
If you have the memory to hash half of the file ("mid"):
$ awk '
NR==FNR { # on 1st pass hash second half of records
if(FNR>1) { # we dont need the 1st record ever
a[FNR]=$0 # hash record
if(FNR%2) # if odd record
delete a[int(FNR/2)+1] # remove one from the past
}
next
}
FNR==1 { # on the start of 2nd pass
if(NR%2==0) # if record count is uneven
exit # exit as there is always even count of them
offset=int((NR-1)/2) # compute offset to the beginning of hash
}
FNR<=offset { # only process the 1st half of records
print $0,a[offset+FNR] # output one from file, one from hash
next
}
{ # once 1st half of 2nd pass is finished
exit # just exit
}' file file # notice filename twice
And finally if you have awk compiled into a worms brain (ie. not so much memory, "min"):
$ awk '
NR==FNR { # just get the NR of 1st pass
next
}
FNR==1 {
mid=(NR-1)/2 # get the midpoint
file=FILENAME # filename for getline
while(++i<=mid && (getline line < file)>0); # jump getline to mid
}
{
if((getline line < file)>0) # getline read from mid+FNR
print $0,line # output
}' file file # notice filename twice
Standard disclaimer on getline and no real error control implemented.
Performance:
I seq 1 100000000 > file and tested how the above solutions performed. Output was > /dev/null but writing it to a file lasted around 2 s longer. max performance is so-so as the mem print was 88 % of my 16 GB so it might have swapped. Well, I killed all the browsers and shaved off 7 seconds for the real time of max.
+------------------+-----------+-----------+
| which | | |
| min | mid | max |
+------------------+-----------+-----------+
| time | | |
| real 1m7.027s | 1m30.146s | 0m48.405s |
| user 1m6.387s | 1m27.314 | 0m43.801s |
| sys 0m0.641s | 0m2.820s | 0m4.505s |
+------------------+-----------+-----------+
| mem | | |
| 3 MB | 6.8 GB | 13.5 GB |
+------------------+-----------+-----------+
Update:
I tested #DavidC.Rankin's and #EdMorton's solutions and they ran, respectively:
real 0m41.455s
user 0m39.086s
sys 0m2.369s
and
real 0m39.577s
user 0m37.037s
sys 0m2.541s
Mem print was about the same as my mid had. It pays to use the wc, it seems.
$ pr -2t file
asdljsdbfajhsdbflakjsdff235 100
asjhbasdjbfajskdfasdbajsdx3 100
asjhbasdjbfajs23kdfb235ajds 150
asjhbasdjbfajskdfbaj456fd3v 125
asjhbasdjb6589fajskdfbaj235 trh77rnv9vnd9dfnmdcnksosdmn
asjhbasdjbfajs54kdfbaj2f879 220
asjhbasdjbfajskdfbajxdfgsdh 225
asjhbasdf3709ddjbfajskdfbaj sdkjNSDfasd89asdg12asdf6asdf
if you want just one space between columns, change to
$ pr -2ts' ' file
You can also do it with awk simply by storing the first-half of the lines in an array and then concatenating the second half to the end, e.g.
awk -v nlines=$(wc -l < file) -v j=0 'FNR<=nlines/2{a[++i]=$0; next} j<i{print a[++j],$1}' file
Example Use/Output
With your data in file, then
$ awk -v nlines=$(wc -l < file) -v j=0 'FNR<=nlines/2{a[++i]=$0; next} j<i{print a[++j],$1}' file
asdljsdbfajhsdbflakjsdff235 100
asjhbasdjbfajskdfasdbajsdx3 100
asjhbasdjbfajs23kdfb235ajds 150
asjhbasdjbfajskdfbaj456fd3v 125
asjhbasdjb6589fajskdfbaj235 trh77rnv9vnd9dfnmdcnksosdmn
asjhbasdjbfajs54kdfbaj2f879 220
asjhbasdjbfajskdfbajxdfgsdh 225
asjhbasdf3709ddjbfajskdfbaj sdkjNSDfasd89asdg12asdf6asdf
Extract the first half of the file and the last half of the file and merge the lines:
paste <(head -n $(($(wc -l <file.txt)/2)) file.txt) <(tail -n $(($(wc -l <file.txt)/2)) file.txt)
You can use columns utility from autogen:
columns -c2 --by-columns file.txt
You can use column, but the count of columns is calculated in a strange way from the count of columns of your terminal. So assuming your lines have 28 characters, you also can:
column -c $((28*2+8)) file.txt
I do not want to solve this, but if I were you:
wc -l file.txt
gives number of lines
echo $(($(wc -l < file.txt)/2))
gives a half
head -n $(($(wc -l < file.txt)/2)) file.txt > first.txt
tail -n $(($(wc -l < file.txt)/2)) file.txt > last.txt
create file with first half and last half of the original file. Now you can merge those files together side by side as it was described here .
Here is my take on it using the bash shell wc(1) and ed(1)
#!/usr/bin/env bash
array=()
file=$1
total=$(wc -l < "$file")
half=$(( total / 2 ))
plus1=$(( half + 1 ))
for ((m=1;m<=half;m++)); do
array+=("${plus1}m$m" "${m}"'s/$/ /' "${m}"',+1j')
done
After all of that if just want to print the output to stdout. Add the line below to the script.
printf '%s\n' "${array[#]}" ,p Q | ed -s "$file"
If you want to write the changes directly to the file itself, Use this code instead below the script.
printf '%s\n' "${array[#]}" w | ed -s "$file"
Here is an example.
printf '%s\n' {1..10} > file.txt
Now running the script against that file.
./myscript file.txt
Output
1 6
2 7
3 8
4 9
5 10
Or using bash4+ feature mapfile aka readarray
Save the file in an array named array.
mapfile -t array < file.txt
Separate the files.
left=("${array[#]::((${#array[#]} / 2))}") right=("${array[#]:((${#array[#]} / 2 ))}")
loop and print side-by-side
for i in "${!left[#]}"; do
printf '%s %s\n' "${left[i]}" "${right[i]}"
done
What you said The only known thing in my scenario is, they are even number all the time. That solution should work.
Is it possible to concatenate the headers lines in a file with the output from a filter using grep? Perhaps using the cat command or something else from GNU's coreutils?
In particular, I have a tab delimited file that roughly looks like the following:
var1 var2 var3
1 MT 500
30 CA 40000
10 NV 1240
40 TX 500
30 UT 35000
10 AZ 1405
35 CO 500
15 UT 9000
1 NV 1505
30 CA 40000
10 NV 1240
I would like to select from lines 2 - N all lines that contain "CA" using grep and also to place the first row, the variable names, in the first line of the output file using GNU/Linux commands.
The desired output for the example would be:
var1 var2 var3
30 CA 40000
35 CA 65000
15 CA 2500
I can select the two sets of desired output with the following lines of code.
head -1 filename
grep -E CA filename
My initial idea is to combine the output of these commands using cat, but I have not been successful so far.
If you're running the commands from a shell (including shell scripts), you can run each command separately and redirect the output:
head -1 filename > outputfile
grep -E CA filename >> outputfile
The first line will overwrite outputfile, because a single > was used. The second line will append to outputfile, because >> was used.
If you want to do this in a single command, the following worked in bash:
(head -1 filename && grep -E CA filename) > outputfile
If you want the output to go to standard output, leave off the parenthesis and redirection:
head -1 filename && grep -E CA filename
It's not clear what you're looking for, but perhaps just:
{ head -1 filename; grep -E CA filename; } > output
or
awk 'NR==1 || /CA/' filename > output
But another interpretation of your question is best addressed using sed or awk.
For example, to print lines 5-9 and line 14, you can do:
sed -n -e 5,9p -e 14p
or
awk '(NR >=5 && NR <=9) || NR==14'
I just came across a method that uses the cat command.
cat <(head -1 filename) <(grep -E CA filename) > outputfile
This site, tldp.org, calls the <(command) syntax "process substitution."
It is unclear to me what method would be more efficient in terms of memory / speed, but this is testable.
I've got script in crontab which creates every 30 minutes files with list of Offline peers in asterisk:
now=$(date +"%Y%m%d%H%M")
/usr/sbin/asterisk -rx 'sip show peers' | grep "Unspec" | sed 's/[/].*//' >> /var/log/asterisk/offline/offline_$now
I need to parse theese files and find extensions that were always offline, i.e. stings in files that were constant.
How can I do this?
Output is:
/usr/sbin/asterisk -rx 'sip show peers' | grep "Unspec" | sed 's/[/].*//' | tail -3
891
894
899
ls /var/log/asterisk/offline/
offline_201309051400 offline_201309051418 offline_201309051530 offline_201309051700
offline_201309051830 offline_201309052000 offline_201309052130
offline_201309051405 offline_201309051430 offline_201309051600 offline_201309051730
offline_201309051900 offline_201309052030 offline_201309052200
offline_201309051406 offline_201309051500 offline_201309051630 offline_201309051800
offline_201309051930 offline_201309052100 offline_201309052230
This awk script will print the lines that are present in all of the files:
awk 'FNR==1{f++}{a[$0]++}END{for (i in a) if (a[i]==f) print i}' offline_*
How it works:
With FNR==1{f++} we count the number of files that are parsed (FNR is equal to one for the first line of each file)
with {a[$0]++} we count how many times each line has appeared.
the END block prints the elements of the array that have been found f times.
I want to ping a bunch of locations but not at the same time, in order so they don't timeout.
The input is for example: ping google.com -n 10 | grep Minimum >> output.txt
This will make the output of: Minimum = 29ms, Maximum = 46ms, Average = 33ms
But there are extra spaces in front of it which I don't know how to cut off, and when it outputs to the txt file it doesn't go to a new line. What I am trying to do is make it so I can copy and paste the input and ping a bunch of places once the previous finishes and log it in a .txt file and number them so it would look like:
Server 1: Minimum = 29ms, Maximum = 46ms, Average = 33ms
Server 2: Minimum = 29ms, Maximum = 46ms, Average = 33ms
Server 3: Minimum = 29ms, Maximum = 46ms, Average = 33ms
Server 4: Minimum = 29ms, Maximum = 46ms, Average = 33ms
Well, first of all, ping on linux limits packet number to send with -c, not -n.
Secondly, output of ping is not Minimum = xx ms, Maximum = yy ms, Avrage = zz ms, but rtt min/avg/max/mdev = 5.953/5.970/5.987/0.017 ms
So basically if you do something in lines of:
for server in google.com yahoo.com
do
rtt=`ping $server -c 2 | grep rtt`
echo "$server: $rtt" >> output.txt
done
You should achieve what you want.
[edit]
If cygwin is your platform, the easiest way to strip the spaces would be either what people are suggesting, sed, or then just | awk '{print $1}', will trim your line as well.
I think you might be able to solve this using sed two times and a while loop at the end:
N=1; ping google.com -n 10 | grep Minimum | sed -r 's/(Average = [[:digit:]]+ms)/\1\n/g' | sed -r s'/[[:space:]]+(Minimum)/\1/g' | while read file; do echo Server "$N": "$file"; N=$((N+1)); done >> output.txt
The steps:
The first sed fixes the newline issue:
Match the final part of the string after which you want a new line, in this case Average = [[:digit:]]+ms and put it into a group using the parenthesis
Then replace it with the same group (\1) and insert a newline character (\n) after it
The second sed removes the whitespaces, by matching the word Minimum and all whitespaces in front of it after which it only returns the word Minimum
The final while statement loops over each line and adds Server "$N": in front of the ping results. The $N was initialized to 1 at the start, and is increased with 1 after each read line
You can use sed to remove first 4 spaces :
ping google.com -n 10 | grep Minimum | sed s/^\ \ \ \ //