I have a shell script that pulls the number of online players, but I need a little help.
The script:
#!/usr/bin/bash
wget --output-document=- http://runescape.com/title.ws 2>/dev/null \
| grep PlayerCount \
| head -1l \
| sed 's/^[^>]*>//' \
| sed "s/currently.*$/$(date '+%r %b %d %Y')/"
It outputs the following:
<p class="top"><span>69,215</span> people 06:31:37 PM Nov 22 2011
What I would like it to say is this:
69,215 people 06:31:37 PM Nov 22 2011
Can any of you help me? :)
This is one of many different ways to do this. Used cut and sed (cut -d">" -f 3,4 | sed 's/<\/span>//'):
[ 15:40 jon#hozbox.com ~ ]$ echo "<p class="top"><span>69,215</span> people 06:31:37 PM Nov 22 2011" | cut -d">" -f 3,4 | sed 's/<\/span>//'
69,215 people 06:31:37 PM Nov 22 2011
#!/usr/bin/bash
wget --output-document=- http://runescape.com/title.ws 2>/dev/null \
| grep PlayerCount \
| head -1l \
| sed 's/^[^>]*>//' \
| sed "s/currently.*$/$(date '+%r %b %d %Y')/" \
| cut -d">" -f 3,4 \
| sed 's/<\/span>//'
I think what you're after is code that removes any tags. Your sed 's/^[^>]*>//' only replaces the up to the first >.
You may want to consider sed 's/<[^>]*>//g' instead.
Pipe the output to:
sed 's%<p class="top"><span>\(.*\)</span>%\1%'
Or combine the two separate sed scripts you already have with this one, giving you:
sed -e 's/^[^>]*>//' \
-e "s/currently.*$/$(date '+%r %b %d %Y')/" \
-e 's%<p class="top"><span>\(.*\)</span>%\1%'
In fact, the grep and head commands are also superfluous; you could do the lot with a single sed command. Note that putting the | on the end of the line means you don't need a backslash.
#!/usr/bin/bash
wget --output-document=- http://runescape.com/title.ws 2>/dev/null |
sed -e '/PlayerCount/!{d;n}' \
-e 's/^[^>]*>//' \
-e "s/currently.*$/$(date '+%r %b %d %Y')/" \
-e 's%<p class="top"><span>\(.*\)</span>%\1%' \
-e 'q'
The /PlayerCount/!n means skip to the next input line unless the input matches 'PlayerCount'. The next three lines do what they always did. The last line implements head -1l by printing (implicitly) and quitting.
(As a matter of idle interest, the wget command produces some 790 lines of data if it runs to completion. I get a 'cannot write to "-" (Broken pipe)' error if I don't redirect standard error to /dev/null (plus some progress reporting that isn't wanted). There are probably options to handle that; it also appears there's only one line with 'PlayerCount' so you could omit the '-e q' command.)
Related
Given a string with multiple words like below, all in one line:
first-second-third-201805241346 first-second-third-201805241348 first-second-third-201805241548 first-second-third-201705241540
I am trying to the maximum number from the string, in this case the answer should be 201805241548
I have tried using awk and grep, but I am only getting the answer as last word in the string.
I am interested in how to get this accomplished.
echo 'first-second-third-201805241346 first-second-third-201805241348 first-second-third-201805241548 first-second-third-201705241540' |\
grep -o '[0-9]\+' | sort -n | tail -1
The relevant part is grep -o '[0-9]\+' | sort -n | tail -n 1.
Using single gnu awk command:
s='first-second-third-201805241346 first-second-third-201805241348 first-second-third-201805241548 first-second-third-201705241540'
awk -F- -v RS='[[:blank:]]+' '$NF>max{max=$NF} END{print max}' <<< "$s"
201805241548
Or using grep + awk (if gnu awk is not available):
grep -Eo '[0-9]+' <<< "$s" | awk '$1>max{max=$1} END{print max}'
Another awk
echo 'first-...-201705241540' | awk -v RS='[^0-9]+' '$0>max{max=$0} END{print max}'
Gnarly pure bash:
n='first-second-third-201805241346 \
first-second-third-201805241348 \
first-second-third-201805241548 \
first-second-third-201705241540'
z="${n//+([a-z-])/;p=}"
p=0 m=0 eval echo -n "${z//\;/\;m=\$((m>p?m:p))\;};m=\$((m>p?m:p))"
echo $m
Output:
201805241548
How it works: This code constructs code, then runs it.
z="${n//+([a-z-])/;p=}" substitutes non-numbers with some pre-code
-- setting $p to the value of each number, (useless on its own). At this point echo $z would output:
;p=201805241346 \ ;p=201805241348 \ ;p=201805241548 \ ;p=201705241540
Substitute the added ;s for more code that sets $m to the
greatest value of $p, which needs eval to run it -- the actual
code the whole line with eval runs looks like this:
p=0 m=0
m=$((m>p?m:p));p=201805241346
m=$((m>p?m:p));p=201805241348
m=$((m>p?m:p));p=201805241548
m=$((m>p?m:p));p=201705241540
m=$((m>p?m:p))
Print $m.
I'm new to writing scripts in bash and I am having an issue with performing a while read loop when trying to access data from the star wars api. My problem is that I am trying to get the name of all the characters and the name of all their associated starships (can be more than one). My below code will get the name of the character, then get the url to be passed to the spacecraft and retrieve the name of the spacecraft.The issue is when I'm trying to put this all together the output seems to overwrite what person belongs to what spacecraft rather than assigning each person to that spacecraft. I would expect to see taking Luke Skywalker as the example:
Luke Skywalker X-Wing
Luke Skywalker Imperial Shuttle
But the output I am getting is blank for Luke Skywalker as the X-Wing seems to attach to another character as the code progresses through all people associated with it. Any guidance would be greatly appreciated.
Bash Shell Script:
#!/usr/bin/env bash
url='https://swapi.co/api'
n=1
while true
do
response=$( curl -sL -H 'Accept: application/json' ${url}/people
page=${n} )
if [[ ${response} =~ .*detail.*Not.* ]]; then
break
fi
n=$((n + 1))
name=$( echo ${response} | jq -r '.results[] | "\(.name)"' | tr -d '"')
echo ${response} | jq -r '.results[] | { starships: .starships }' |
egrep /starships/ |\
sed -e 's/^[ \t]*//' | tr -d '"' | tr -d ',' |\
while read shipurl
do
curl -s ${shipurl} | jq '.name' | tr -d '"'
done |\
while read shipname
do
echo "${name}" " ${shipname}"
done
done
Thanks!
oh boy! This should be done using python... but here, I gave it a try. Not a hardcore Star Wars fan, so don't know if this turns out ok, but this will give you some more idea...
Note, I have disabled while true for my test case. I am sure some of those jq related stuff can be cleaned up more. I use pilots for each of these starships and match according to that because each character name has more than one matched starship.
$ cat crazystarwars.sh
#!/usr/bin/env bash
url='https://swapi.co/api'
n=1
#while true
#do
response=$(curl -sL -H 'Accept: application/json' "${url}/people?page=${n}")
if [[ ${response} =~ .*"detail".*"Not".* ]]; then
break
fi
readarray -t name < <( echo "${response}" | jq -r '.results[] | "\(.name)"')
readarray -t shipurls < <(echo ${response} | jq -r '.results[] | { starships: .starships }' | egrep /starships/ | sed -e 's/^[ \t]*//' | tr -d '"' | tr -d ',')
for p in "${shipurls[#]}"
do
shipname=$(curl -s ${p} | jq '.name' | tr -d '"')
readarray -t _pilots < <(curl -s ${p} | jq -r '.pilots[]' | xargs -I {} sh -c 'curl -s {}' | jq '.name')
for _name in "${name[#]}"
do
if [[ "${_pilots[#]}" =~ "${_name}" ]];
echo "${_name} ${shipname}"
break
fi
done
done
# ((n++))
#done
Output is something like (not entirely sure if its the right one):
$ ./crazystarwars.sh
Luke Skywalker X-wing
Luke Skywalker Imperial shuttle
Darth Vader TIE Advanced x1
Luke Skywalker X-wing
Obi-Wan Kenobi Jedi starfighter
Obi-Wan Kenobi Trade Federation cruiser
Obi-Wan Kenobi Naboo star skiff
Obi-Wan Kenobi Jedi Interceptor
Obi-Wan Kenobi Belbullab-22 starfighter
I currently have many text files over several directories that I am sorting and storing the results in text file. The issues is not the sorting part but formatting the output that gets placed in the text file. I am looking to output in this format file '/path/to/file1' currently it shows /path/to/file1. I want to do this all within one process(not have to run an additional loop or fine to change the format).
$ target=~/tmp/shuf
$ destination=/filepath/
$ find $target -iname "*.txt" -type f | shuf | awk -F- '{printf("%s:%s\n", $0, $NF)}' | sort -t : -k 2 -s | cut -d : -f 1 | xargs -n1 basename | sed "s,^,$destination," > $destination/results.txt
Current results.txt:
/path/to/cs650-software_methodologies-fname_lname-001.txt
/path/to/s630-linux_research_paper-fname_lname-001.txt
Desired results.txt:
file '/path/to/cs650-software_methodologies-fname_lname-001.txt'
file '/path/to/s630-linux_research_paper-fname_lname-001.txt'
I find awk is often easier for this kind of formatting, if you don't have substitutions. This also allows us to skip the basename call and leave that part to awk as well. Just note that this will not work if you have any forward slashes in your actual filenames.
find $target -type f -iname "*.txt" \
| shuf \
| awk -F- '{printf("%s:%s\n", $0, $NF)}' \
| sort -t : -k 2 -s \
| cut -d : -f 1 \
| awk -F / '{printf("file '\''%s'\''\n", $0)}' \
> $destination/results.txt
I get the following error:
> echo "${$(qstat -a | grep kig):0:7}"
-bash: ${$(qstat -a | grep kig):0:7}: bad substitution
I'm trying to take the number before. of
> qstat -a | grep kig
1192530.perceus- kigumen lr_regul pbs.sh 27198 2 16 -- 24:00:00 R 00:32:23
and use it as an argument to qdel in openPBS so that I can delete all process that I started with my login kigumen
so ideally, this should work:
qdel ${$(qstat -a | grep kig):0:7}
so far, only this works:
str=$(qstat -a | grep kig); qdel "${str:0:7}"
but I want a clean one-liner without a temporary variable.
The shell substring construct you're using (:0:7) only works on variables, not command substitution. If you want to do this in a single operation, you'll need to trim the string as part of the pipeline, something like one of these:
echo "$(qstat -a | grep kig | sed 's/[.].*//')"
echo "$(qstat -a | awk -F. '/kig/ {print $1}')"
echo "$(qstat -a | awk '/kig/ {print substr($0, 1, 7)}')"
(Note that the first two print everything before the first ".", while the last prints the first 7 characters.) I don't know that any of them are particularly cleaner, but they do it without a temp variable...
qstat -u palle | cut -f 1 -d "." | xargs qdel
Kills all my jobs... normally I grep out the jobname(s) before cut'ing...
So I use a small script "idlist":
qstat -u palle | grep -E "*.in" | grep -E "$1" | cut -f 1 -d "." | xargs
To see all my "map_..." jobs:
idlist "map_*"
For killing all my "map_...." jobs:
idlist "map_*" | xargs qdel
yet another ways :
foreach m1 in $(qstat -a );do
if [[ $m1 =~ kig ]];then
m2=${m1%.kig}
echo "kig found $m2 "
break
fi
done
Say I have 8b1f 0008 0231 49f6 0300 f1f3 75f4 0c72 f775 0850 7676 720c 560d 75f0 02e5 ce00 0861 1302 0000 0000, how can I easily get a binary file from that without copying+pasting into a hex editor?
Use:
% xxd -r -p in.txt out.bin
See xxd.
This version will work with binary format too:
cat /bin/sh \
| od -A n -v -t x1 \
| tr -d '\r' \
| xxd -r -g 1 -p1 \
| md5sum && md5sum /bin/sh
The extra '\r' is just if you're dealing with DOS text files...
And process byte by byte to prevent endianness difference if running parts of a pipe on different systems.
All the present answers refer to the convenient xxd -r approach, but for situations where xxd is not available or convenient here is a more portable (and more flexible, but more verbose and less efficient) solution, using only POSIX shell syntax (it also compensates for odd-number of digits in input):
un_od() {
printf -- "$(
tr -d '\t\r\n ' | sed -e 's/^(.(.{2})*)$/0\1/' -e 's/\(.\{2\}\)/\\x\1/g'
)"
}
By the way: you don't specify whether your input is big-endian or little-endian, or whether you want big/little-endian output. Usually input such as in your question would be big-endian/network-order (e.g., as created by od -t x1 -An -v), and would be expected to transform to big-endian output. I presume xxd just assumes that default if not told otherwise, and this solution does that too. If byte-swapping is needed, how you do the byte-swapping also depends on the word-size of the system (e.g., 32 bit, 64 bit) and very rarely the byte-size (you can almost always assume 8-bit bytes - octets - though).
The below functions use a more complex version of the binary -> od -> binary trick to portably byteswap binary data, conditional on system endianness, and accounting for system word-size. The algorithm works for anything up to 72-bit word size (because seq -s '' 10 -> 12345678910 doesn't work):
if { sed --version 2>/dev/null || :; } | head -n 1 | grep -q 'GNU sed'; then
_sed() { sed -r "${#}"; }
else
_sed() { sed -E "${#}"; }
fi
sys_bigendian() {
return $(
printf 'I' | od -t o2 | head -n 1 | \
_sed -e 's/^[^ \t]+[ \t]+([^ \t]+)[ \t]*$/\1/' | cut -c 6
)
}
sys_word_size() { expr $(getconf LONG_BIT) / 8; }
byte_swap() {
_wordsize=$1
od -An -v -t o1 | _sed -e 's/^[ \t]+//' | tr -s ' ' '\n' | \
paste -d '\\' $(for _cnt in $(seq $_wordsize); do printf -- '- '; done) | \
_sed -e 's/^/\\/' -e '$ s/\\+$//' | \
while read -r _word; do
_thissize=$(expr $(printf '%s' "$_word" | wc -c) / 4)
printf '%s' "$(seq -s '' $_thissize)" | tr -d '\n' | \
tr "$(seq -s '' $_thissize -1 1)" "$_word"
done
unset _wordsize _prefix _word _thissize
}
You can use the above to output file contents in big-endian format regardless of system endianness:
if sys_bigendian; then
cat /bin/sh
else
cat /bin/sh | byte_swap $(sys_word_size)
fi
Here is the way to reverse "od" output:
echo "test" | od -A x -t x1 | sed -e 's|^[0-f]* ?||g' | xxd -r
test