I'm new to writing scripts in bash and I am having an issue with performing a while read loop when trying to access data from the star wars api. My problem is that I am trying to get the name of all the characters and the name of all their associated starships (can be more than one). My below code will get the name of the character, then get the url to be passed to the spacecraft and retrieve the name of the spacecraft.The issue is when I'm trying to put this all together the output seems to overwrite what person belongs to what spacecraft rather than assigning each person to that spacecraft. I would expect to see taking Luke Skywalker as the example:
Luke Skywalker X-Wing
Luke Skywalker Imperial Shuttle
But the output I am getting is blank for Luke Skywalker as the X-Wing seems to attach to another character as the code progresses through all people associated with it. Any guidance would be greatly appreciated.
Bash Shell Script:
#!/usr/bin/env bash
url='https://swapi.co/api'
n=1
while true
do
response=$( curl -sL -H 'Accept: application/json' ${url}/people
page=${n} )
if [[ ${response} =~ .*detail.*Not.* ]]; then
break
fi
n=$((n + 1))
name=$( echo ${response} | jq -r '.results[] | "\(.name)"' | tr -d '"')
echo ${response} | jq -r '.results[] | { starships: .starships }' |
egrep /starships/ |\
sed -e 's/^[ \t]*//' | tr -d '"' | tr -d ',' |\
while read shipurl
do
curl -s ${shipurl} | jq '.name' | tr -d '"'
done |\
while read shipname
do
echo "${name}" " ${shipname}"
done
done
Thanks!
oh boy! This should be done using python... but here, I gave it a try. Not a hardcore Star Wars fan, so don't know if this turns out ok, but this will give you some more idea...
Note, I have disabled while true for my test case. I am sure some of those jq related stuff can be cleaned up more. I use pilots for each of these starships and match according to that because each character name has more than one matched starship.
$ cat crazystarwars.sh
#!/usr/bin/env bash
url='https://swapi.co/api'
n=1
#while true
#do
response=$(curl -sL -H 'Accept: application/json' "${url}/people?page=${n}")
if [[ ${response} =~ .*"detail".*"Not".* ]]; then
break
fi
readarray -t name < <( echo "${response}" | jq -r '.results[] | "\(.name)"')
readarray -t shipurls < <(echo ${response} | jq -r '.results[] | { starships: .starships }' | egrep /starships/ | sed -e 's/^[ \t]*//' | tr -d '"' | tr -d ',')
for p in "${shipurls[#]}"
do
shipname=$(curl -s ${p} | jq '.name' | tr -d '"')
readarray -t _pilots < <(curl -s ${p} | jq -r '.pilots[]' | xargs -I {} sh -c 'curl -s {}' | jq '.name')
for _name in "${name[#]}"
do
if [[ "${_pilots[#]}" =~ "${_name}" ]];
echo "${_name} ${shipname}"
break
fi
done
done
# ((n++))
#done
Output is something like (not entirely sure if its the right one):
$ ./crazystarwars.sh
Luke Skywalker X-wing
Luke Skywalker Imperial shuttle
Darth Vader TIE Advanced x1
Luke Skywalker X-wing
Obi-Wan Kenobi Jedi starfighter
Obi-Wan Kenobi Trade Federation cruiser
Obi-Wan Kenobi Naboo star skiff
Obi-Wan Kenobi Jedi Interceptor
Obi-Wan Kenobi Belbullab-22 starfighter
Related
I would like to make array which put users in a time using for loop. For example:
y[1]="user1"
y[2]="user2"
...
y[n]="usern"
I tried to do it like this
#!/bin/bash
x=$(who | cut -d " " -f1 | sort | uniq | wc -l)
for (( i=1; i<=$x; i++ )); do
y[$i]=$(who | cut -d " " -f1 | sort | uniq | sed -n '$ip')
p[$i]=$(lsof -u ${y[$i]} | wc -l)
echo "Users:"
echo ${y[$i]}
echo -e "Number of launched files:\n" ${p[$i]}
done
Most likely I'm using command "sed" wrong.
Can you help me?
Indeed your sed command seems to be a bit off. I can't really guess what you're trying to do there. Besides that, I'm wondering why you're executing who twice. You can make use of the data first obtained in the following manner.
#!/bin/bash
# define two arrays
y=()
p=()
#x=0
while read -r username; do
y+=("$username")
p+=($(lsof -u $(id -u "$username") | wc -l))
echo -e "User:\n${y[-1]}"
echo -e "Open files:\n${p[-1]}"
# The -1 index is the last index in the array, but you
# could uncomment the x=0 variable and the line below:
#((x++))
done <<< $(who | cut -d " " -f1 | sort | uniq)
echo "Amount of users: $x"
exit 0
When I run terraform plan it shows a changed resource, which happens to be JSON data in an aws_s3_bucket_object. But the JSON is long and it's difficult to see what changed. How can I display this as a diff?
https://github.com/coinbase/terraform-landscape can help with this.
gem install terraform_landscape (may need sudo on macOS)
terraform plan | landscape
This shows JSON changes as a diff. Here's an example from the github site:
I wrote a bash script to format terraforms ugly policy output:
#!/bin/bash
input=$( xclip -o )
old=$( echo "$input" | awk -F' => ' '{ print $1 }' | sed 's/\\n/\n\r/g' | sed 's/\\"/"/g' | sed 's/"{/{/' | sed 's/}"/}/' )
new=$( echo "$input" | awk -F' => ' '{ print $2 }' | sed 's/\\n/\n\r/g' | sed 's/\\"/"/g' | sed 's/"{/{/' | sed 's/}"/}/' )
echo "----------------------------------------------------------------------------------------------"
echo "old:"
echo "$old" | jq '.'
echo "----------------------------------------------------------------------------------------------"
echo "new:"
echo "$new" | jq '.'
echo "----------------------------------------------------------------------------------------------"
echo "diff:"
diff -u --color <( echo "$old" | jq '.' ) <( echo "$new" | jq '.' )
echo "----------------------------------------------------------------------------------------------"
It shows three blocks of output, the old, then the new and then the diff. It makes use of xclip, jq and diff. Usage is (on Linux) to highlight the terrafrom output and then invoke the script (I call it tf-diff and it lives in ~/bin).
I am trying to get the max version number from a directory where i have several versions of one program
for example if output of ls is
something01_1.sh
something02_0.1.2.sh
something02_0.1.sh
something02_1.1.sh
something02_1.2.sh
something02_2.0.sh
something02_2.1.sh
something02_2.3.sh
something02_3.1.2.sh
something.sh
I am getting the max version number with the following -
ls somedir | grep some_prefix | cut -d '_' -f2 | sort -t '.' -k1 -r | head -n 1
Now if at the same time i want to check it with the version number which i already have in the system, whats the best way to do it...
in bash i got this working (if 2.5 is the current version)
(ls somedir | grep some_prefix | cut -d '_' -f2; echo 2.5) | sort -t '.' -k1 -r | head -n 1
is there any other correct way to do it?
EDIT: In the above example some_prefix is something02.
EDIT: Actual Problem here is
(ls smthing; echo more) | sort
is it the best way to merge output of two commands/program for piping into third.
I have found the solution. The best way it seems is using process substitution.
cat <(ls smthing) <(echo more) | sort
for my version example
cat <(ls somedir | grep some_prefix | cut -d '_' -f2) <(echo 2.5) | sort -t '.' -k1 -r | head -n 1
for the benefit of future readers, I recommend - please drop the lure of one-liner and use glob as chepner suggested.
Almost similar question is asked on superuser.
more info about process substitution.
Is the following code more suitable to what you're looking for:
#/bin/bash
highest_version=$(ls something* | sort -V | tail -1 | sed "s/something02_\|\.sh//g")
current_version=$(echo $0 | sed "s/something02_\|\.sh//g")
if [ $current_version > $highest_version ]; then
echo "Uh oh! Looks like we need to update!";
fi
You can try something like this :
#! /bin/bash
lastversion() { # prefix
local prefix="$1" a=0 b=0 c=0 r f vmax=0
for f in "$prefix"* ; do
test -f "$f" || continue
read a b c r <<< $(echo "${f#$prefix} 0 0 0" | tr -C '[0-9]' ' ')
v=$(((a*100+b)*100+c))
if ((v>vmax)); then vmax=$v; fi
done
echo $vmax
}
lastversion "something02"
It will print: 30102
I want to extract from the command ping -c 4 www.stackoverflow.com | tail -1| awk '{print $4}'
the average time.
107.921/108.929/110.394/0.905 ms
Output should be: 108.929
One way is to just add a cut to what you have there.
ping -c 4 www.stackoverflow.com | tail -1| awk '{print $4}' | cut -d '/' -f 2
ping -c 4 www.stackoverflow.com | tail -1| awk -F '/' '{print $5}' would work fine.
"-F" option is used to specify the field separator.
This might work for you:
ping -c 4 www.stackoverflow.com | sed '$!d;s|.*/\([0-9.]*\)/.*|\1|'
The following solution uses Bash only (requires Bash 3):
[[ $(ping -q -c 4 www.example.com) =~ \ =\ [^/]*/([0-9]+\.[0-9]+).*ms ]] \
&& echo ${BASH_REMATCH[1]}
For the regular expression it's easier to read (and handle) if it is stored in a variable:
regex='= [^/]*/([0-9]+\.[0-9]+).*ms'
[[ $(ping -q -c 4 www.example.com) =~ $regex ]] && echo ${BASH_REMATCH[1]}
Promoting luissquall's very elegent comment to an answer:
ping -c 4 www.stackoverflow.com | awk -F '/' 'END {print $5}'
Direct extract mean time from ping command:
ping -w 4 -q www.duckduckgo.com | cut -d "/" -s -f5
Options:
-w time out 4 seconds
-q quite mode
-d delimiter
-s skip line without delimiter
-f No. of field - depends on your system - sometimes 5th, sometimes 4th
I personly use is this way:
if [ $(ping -w 2 -q www.duckduckgo.com | cut -d "/" -s -f4 | cut -d "." -f1) -lt 20 ]; then
echo "good response time"
else
echo "bad response time"
fi
Use these to get current ping as a single number:
123.456:
ping -w1 -c1 8.8.8.8 | tail -1| cut -d '=' -f 2 | cut -d '/' -f 2
123:
ping -w1 -c1 8.8.8.8 | tail -1| cut -d '=' -f 2 | cut -d '/' -f 2 | cut -d '.' -f 1
Note that this displays the average of only 1 ping (-c1), you can increase the sample size by increasing this number (i.e. -c1337)
This avoids using awk (like #Buggabill posted), which doesn't play nice in bash aliases + takes a nanosecond longer
None of these worked well for me due to various issues such as when a timeout occurs. I only wanted to see bad ping times or timeouts and wanted PING to continue quickly, and none of these solutions worked. Here's my BASH script that works well to do both. Note that in the ping command, response time is limited to 1 second.
I realize this does not directly answer the OP's question, however it does provide a good way to deal with some issues that occur with some of the incomplete "solutions" provided here, thus going beyond the scope of the OPs question, which others coming here are looking for (I cite myself as an example), so I decided to share for those people, not specifically OP's question.
while true
do
###Set your IP amd max milliseconds###
ip="192.168.1.53"
maxms=50
###do not edit below###
err="100% packet loss"
out="$(ping -c 1 -i 1 -w 1 $ip)"
t="$(echo $out | awk -F '/' 'END {print $5}')"
t=${t%.*}
re='^[0-9]+$'
if ! [[ $t =~ $re ]] ; then
if [[ $out == *"$err"* ]] ; then
echo "`date` | ${ip}: TIMEOUT"
else
echo "error: Not a number: ${t} was found in: ${out}"
fi
else
if [ "$t" -gt $maxms ]; then
echo "`date` | ${ip}: ${t} ms"
fi
fi
done
Say I have 8b1f 0008 0231 49f6 0300 f1f3 75f4 0c72 f775 0850 7676 720c 560d 75f0 02e5 ce00 0861 1302 0000 0000, how can I easily get a binary file from that without copying+pasting into a hex editor?
Use:
% xxd -r -p in.txt out.bin
See xxd.
This version will work with binary format too:
cat /bin/sh \
| od -A n -v -t x1 \
| tr -d '\r' \
| xxd -r -g 1 -p1 \
| md5sum && md5sum /bin/sh
The extra '\r' is just if you're dealing with DOS text files...
And process byte by byte to prevent endianness difference if running parts of a pipe on different systems.
All the present answers refer to the convenient xxd -r approach, but for situations where xxd is not available or convenient here is a more portable (and more flexible, but more verbose and less efficient) solution, using only POSIX shell syntax (it also compensates for odd-number of digits in input):
un_od() {
printf -- "$(
tr -d '\t\r\n ' | sed -e 's/^(.(.{2})*)$/0\1/' -e 's/\(.\{2\}\)/\\x\1/g'
)"
}
By the way: you don't specify whether your input is big-endian or little-endian, or whether you want big/little-endian output. Usually input such as in your question would be big-endian/network-order (e.g., as created by od -t x1 -An -v), and would be expected to transform to big-endian output. I presume xxd just assumes that default if not told otherwise, and this solution does that too. If byte-swapping is needed, how you do the byte-swapping also depends on the word-size of the system (e.g., 32 bit, 64 bit) and very rarely the byte-size (you can almost always assume 8-bit bytes - octets - though).
The below functions use a more complex version of the binary -> od -> binary trick to portably byteswap binary data, conditional on system endianness, and accounting for system word-size. The algorithm works for anything up to 72-bit word size (because seq -s '' 10 -> 12345678910 doesn't work):
if { sed --version 2>/dev/null || :; } | head -n 1 | grep -q 'GNU sed'; then
_sed() { sed -r "${#}"; }
else
_sed() { sed -E "${#}"; }
fi
sys_bigendian() {
return $(
printf 'I' | od -t o2 | head -n 1 | \
_sed -e 's/^[^ \t]+[ \t]+([^ \t]+)[ \t]*$/\1/' | cut -c 6
)
}
sys_word_size() { expr $(getconf LONG_BIT) / 8; }
byte_swap() {
_wordsize=$1
od -An -v -t o1 | _sed -e 's/^[ \t]+//' | tr -s ' ' '\n' | \
paste -d '\\' $(for _cnt in $(seq $_wordsize); do printf -- '- '; done) | \
_sed -e 's/^/\\/' -e '$ s/\\+$//' | \
while read -r _word; do
_thissize=$(expr $(printf '%s' "$_word" | wc -c) / 4)
printf '%s' "$(seq -s '' $_thissize)" | tr -d '\n' | \
tr "$(seq -s '' $_thissize -1 1)" "$_word"
done
unset _wordsize _prefix _word _thissize
}
You can use the above to output file contents in big-endian format regardless of system endianness:
if sys_bigendian; then
cat /bin/sh
else
cat /bin/sh | byte_swap $(sys_word_size)
fi
Here is the way to reverse "od" output:
echo "test" | od -A x -t x1 | sed -e 's|^[0-f]* ?||g' | xxd -r
test