Note: Question does not duplicate Ignoring specific errors in a shell script .
Suppose it is needed to capture the leading characters of a encoded representation of a file.
In shell (tested in Bash), it is easy to use the following form:
encoded="$(< file base64 | head -c16)"
The statement functions desired, except under certain alterations to the environment.
Consider the following:
set -o errexit -o pipefail
shopt -s inherit_errexit
encoded="$(< file base64 | head -c16)"
The final line would cause termination of a script, because of the non-zero return status (141) given by base64, unhappy with closed pipe. The return status is propagated to the pipe and then to the invoking shell.
The undesired effect requires a workaround, such as follows:
set -o errexit -o pipefail
shopt -s inherit_errexit
encoded="$((< file base64 || :) | head -c16)"
The : has the same effect as would have the keyword true, to evaluate as a non-error.
However, this approach leads to a further unwanted effect.
The following shows a variation with a different error:
set -o errexit -o pipefail
shopt -s inherit_errexit
encoded="$((< /not/a/real/file base64 || :) | head -c16)"
echo $?
The printed code is zero. Now, a true error has been masked.
The most obvious solution, as follows, is rather verbose
set -o errexit -o pipefail
shopt -s inherit_errexit
encoded="$((< /not/a/real/file base64 || [ $? == 141 ]) | head -c16)"
echo $?
Is a more compact form available? Is any environment alteration available such that statements masks only the particular status code, without the explicit inline expression?
First off, apparently, to actually provoke the 141 error the file needs to be fairly, large, e.g.
head -c 1000000 /dev/urandom > file
now, as you said, this script sh.sh will terminate before showing encoded: ...:
#!/bin/bash
set -o errexit -o pipefail
shopt -s inherit_errexit
encoded="$(< file base64 | head -c16)"
echo "encoded: $encoded"
instead of checking for the error code, you could let base64 continue to pipe the rest of its data to /dev/null by invoking cat > /dev/null after head is done:
#!/bin/bash
set -o errexit -o pipefail
shopt -s inherit_errexit
encoded="$(< file base64 | ( head -c16 ; cat > /dev/null ) )"
echo "encoded: $encoded"
now you will get encoded: NvyX2Zx4nTDjtQO8 or whatever.
And this does not mask other errors like the file not existing:
$ ./sh.sh
./sh.sh: line 5: file: No such file or directory
However, it will be less efficient because the whole file will be read.
For your specific example program, you could also just change your approach. 16 base 64 characters represent 16 * 6 = 96 bits, so 96/8 = 12 bytes of data from the file are needed:
encoded="$(head -c12 file | base64)"
this will not cause SIGPIPE.
Related
I am trying to write a shell script to identify broken urls from a list of urls.
here is input_url.csv sample:
https://www.google.com/
https://www.nbc.com
https://www.google.com.hksjkhkh/
https://www.google.co.jp/
https://www.google.ca/
Here is what I have which works:
wget --spider -nd -nv -H --max-redirect 0 -o run.log -i input_url.csv
and this gives me '2019-09-03 19:48:37 URL: https://www.nbc.com 200 OK' for valid urls, and for broken ones it gives me '0 redirections exceeded.'
what i expect is that i only want to save those broken links into my output file.
sample expect output:
https://www.google.com.hksjkhkh/
I think I would go with:
<input.csv xargs -n1 -P10 sh -c 'wget --spider --quiet "$1" || echo "$1"' --
You can use -P <count> option to xargs to run count processes in parallel.
xargs runs the command sh -c '....' -- for each line of the input file appending the input file line as the argument to the script.
Then sh inside runs wget ... "$1". The || checks if the return status is nonzero, which means failure. On wget failure, echo "$1" is executed.
Live code link at repl.
You could filter the output of wget -nd -nv and then regex the output, well like
wget --spider -nd -nv -H --max-redirect 0 -i input 2>&1 | grep -v '200 OK' | grep 'unable' | sed 's/.* .//; s/.$//'
but this looks not expendable, is not parallel so probably is slower and probably not worth the hassle.
I have the following bash code (running on Red Hat) that is exiting when I enable set -o errexit and the variable in the code is empty, BUT works fine when the variable is set; the code is designed to test if a screen session matching .monitor_* exists, and if so do something.
I have the following turned on:
set -o errexit
set -x xtrace; PS4='$LINENO: '
If there is a session matching the above pattern it works; however, if nothing matches it just exits with no information other than the following output from xtrace
someuser:~/scripts/tests> ./if_test.sh
+ ./if_test.sh
+ PS4='$LINENO: '
4: set -o errexit
5: set -o pipefail
6: set -o nounset
88: /usr/bin/ls -lR /var/run/uscreens/S-storage-rsync
88: grep '.monitor_*'
88: awk '{ print $9 }'
88: /usr/bin/grep -Ev 'total|uscreens'
8: ms=
I tested the command I am using to set the ms var and it agrees with the xtrace output, it's not set.
someuser:~/scripts/tests> test -n "${mn}"
+ test -n ''
I have tried using a select statement and got the same results... I can't figure it out, anyone able to help? Thanks.
I read through all the possible solution recommendations, nothing seems to address my issue.
The code:
#!/usr/bin/env bash
set -o xtrace; PS4='$LINENO: '
set -o errexit
set -o pipefail
set -o nounset
ms="$(/usr/bin/ls -lR /var/run/uscreens/S-"${USER}" | /usr/bin/grep -Ev "total|uscreens" | grep ".monitor_*" | awk '{ print $9 }')"
if [[ -z "${ms}" ]]; then
echo "Handling empty result"
elif [[ -n "${ms}" ]]; then
echo "Handling non-empty result"
fi
The following answer was proposed: Test if a variable is set in bash when using "set -o nounset"; however, it doesn't address the issue at all. In my case the variable being tested is set and as stated in my detail, it's set to "", or nothing. Thank you; however, it doesn't help.
It really seems to be the variable declaration that it isn't liking.
ms="$(/usr/bin/ls -lR /var/run/uscreens/S-"${USER}" | /usr/bin/grep -Ev "total|uscreens" | grep ".monitor_*" | awk '{ print $9 }')"
You're running set -o pipefail, so if any component in a pipeline has a nonzero exit status, the entire pipeline is treated as having a nonzero exit status.
Your pipeline runs grep. grep has a nonzero status whenever no matches are found.
You're running set -o errexit (aka set -e). With errexit enabled, the script terminates whenever any command fails (subject to a long and complicated set of exceptions; some of these are presented in the exercises section of BashFAQ #105, and others touched on in this excellent reference).
Thus, when you have no matches in your grep command, your script terminates on the command substitution running the pipeline in question.
If you want to exempt a specific command from set -e's behavior, the easiest way to do it is to simply append ||: (shorthand for || true), which marks the command as "checked".
"You're running set -o pipefail. When grep doesn't match anything, it has a nonzero exit status, and with pipefail, that fails the entire pipeline. This is all behaving exactly the way you're telling your shell it should behave." – Charles Duffy
Charles above comment was exactly what was going on, my script was working as intended and I need to adjust the logic to work differently if I wish to keep set -o pipefail set.
Thank you for the help.
I use a command which parses video files for certain frames and returning their timecode, when found. At the moment, I have to execute the command, wait, until the values printed to stdout reach the desired position and then abort the execution using Ctrl+C.
As I have to watch the process and to abort the execution in the right moment to get the information I need, I thought, I could automate this to some degree by creating a bash script.
I am not certain, if it can be done in bash, as I don't exactly know, how to abort the execution in connection with the values it writes to stdout.
The output of the command looks like
0.040000
5.040000
10.040000
15.040000
18.060000
(...)
I tried
until [[ "$timecode" -gt 30 ]]; do
timecode=$(mycommand)
sleep 0.1
done
echo "Result: $timecode"
or
while [[ "$timecode" -le 30 ]]; do
timecode=$(mycommand)
sleep 0.1
done
echo "Result: $timecode"
which both seem to result in the command being executed until it finishes and afterwards the rest of the loop is being processed. But I want to evaluate the output while the command executes and break execution depending on the output.
Additional information
The command has no capability to be stopped at a certain point in the stream. It parses the whole file and gives the results unless signalled to stop. This was my first shot.
The execution time of the command is very long as the files I parse are ~2GB. As I don't need all frames of the file but only a few around a given timecode, I never let it execute until it finished.
The output of the command varies from file to file, so I can't look for an exact value. If I knew the exact value, I probably wouldn't have to look for it.
The destination time code - in the example it is specified by "-gt 30" - is different for every file I will have to parse, so I will have to put this into a command line parameter once the script works. I would also have to make sure to get back more than the last value of the execution but about the last 5 values. For these two I already have Ideas.
I'm totally stuck on that one and have not even an idea what to google for.
Thank you for your input!
Manuel
With the answers of PSkocik and Kyle Burton, I was able to integrate the suggested solution into my script. It doesn't work and I don't see, why.
Here the complete script including the external command providing the output:
#!/usr/bin/env bash
set -eu -o pipefail
parser () {
local max="$1"
local max_int
max_int="${max%.*}"
while read tc;
do
local tc_int
tc_int="${tc%.*}"
echo $tc
if (( "$tc_int" >= "$max_int" )); then
echo "Over 30: $tc";
exec 0>&-
return 0
fi
done
}
ffprobe "$1" -hide_banner -select_streams v -show_entries frame=key_frame,best_effort_timestamp_time -of csv=nk=1:p=0:s="|" -v quiet | sed -ne "s/^1|//p" | parser 30
I don't get any output from the "echo $tc" but the ffprobe is running - I can see it in top. It runs until I stop the script using Ctrl+C.
Thank you Kyle for your big efforts in this. I'd never come to such a conclusion. I changed the commandline of ffprobe to your suggestion
ffprobe "$1" -hide_banner -select_streams v -show_entries frame=key_frame,best_effort_timestamp_time -of csv=nk=1:p=0:s="|" -v quiet | cut -f2 -d\| | parser 30
and now, I'm getting results while ffprobe runs. But... the way you changed the command returns all frames, ffprobe finds and not only the Keyframes. The original output of the ffprobe command looks like
1|0.000000
0|0.040000
0|0.080000
0|0.120000
0|0.160000
0|0.200000
(...)
The 0 at the beginning of the line means: this is no keyframe.
The 1 at the beginning of the line means: this is a keyframe.
The script is intended to provide only the keyframes around a certain timecode of the video file. The way you changed the command, it now provides all frames of the video file what makes the resulting output useless. It has to be filtered for all lines starting with zero to be dropped.
As I don't exactly understand, why this doesn't work with sed, I can only try to find a solution by try and error, facilitating different tools to filter the output. But if the filtering itself causes the problem, we might have hit a wall here.
If you have process a that's outputting stuff to stdout and process b that reads the outputted stuff via a pipe:
a | b
all b has to usually do to kill a when a certain item is outputted
is to close its standard input.
A sample b:
b()
{
while read w;
do case $w in some_pattern)exec 0>&-;; esac;
echo $w
done
}
This closing of stdin (filedescriptor 0) will cause the producer process to be killed by SIGPIPE the moment it tries to make its next write.
I think PSkocik's approach makes sense. I think all you need to do is run your mycommand and pipe it into your while loop. If you put PSkocik's code in a file wait-for-max.sh then you should be able to run it as:
mycommand | bash wait-for-max.sh
After working with M. Uster in comments above, we've come up with the following solution:
#!/usr/bin/env bash
set -eu -o pipefail
# echo "bash cutter.sh rn33.mp4"
# From: https://stackoverflow.com/questions/45304233/execute-command-in-bash-script-until-output-exceeds-certain-value
# test -f stack_overflow_q45304233.tar || curl -k -O https://84.19.186.119/stack_overflow_q45304233.tar
# test -f stack_overflow_q45304233.tar || curl -k -O https://84.19.186.119/stack_overflow_q45304233.tar
# test -f rn33.mp4 || curl -k -O https://84.19.186.119/rn33.mp4
function parser () {
local max="$1"
local max_int
# NB: this removes everything after the decimal point
max_int="${max%.*}"
# I added a line number so I could match up the ouptut from this function
# with the output captured by the 'tee' command
local lnum="0"
while read -r tc;
do
lnum="$(( 1 + lnum ))"
# if a blank line is read, just ignore it and continue
if [ -z "$tc" ]; then
continue
fi
local tc_int
# NB: this removes everything after the decimal point
tc_int="${tc%.*}"
echo "Read[$lnum]: $tc"
if (( "$tc_int" >= "$max_int" )); then
echo "Over 30: $tc";
# This closes stdin on this process, which will cause an EOF on the
# process writing to us across the pipe
exec 0>&-
return 0
fi
done
}
# echo "bash version: $BASH_VERSION"
# echo "ffprobe version: $(ffprobe -version | head -n1)"
# echo "sed version: $(sed --version | head -n1)"
# NB: by adding in the 'tee ffprobe.out' into the pipeline I was able to see
# that it was producing lines like:
#
# 0|28.520000
# 1|28.560000
#
#
# changing the sed to look for any single digit and a pipe fixed the script
# another option is to use cut, see below, which is probalby more robust.
# ffprobe "$1" \
# -hide_banner \
# -select_streams v \
# -show_entries frame=key_frame,best_effort_timestamp_time \
# -of csv=nk=1:p=0:s="|" \
# -v quiet 2>&1 | \
# tee ffprobe.out |
# sed -ne "s/^[0-9]|//p" | \
# parser 30
ffprobe "$1" \
-hide_banner \
-select_streams v \
-show_entries frame=key_frame,best_effort_timestamp_time \
-of csv=nk=1:p=0:s="|" \
-v quiet 2>&1 | \
cut -f2 -d\| | \
parser 30
The answer to my question has finally been found by the help of PSkocik and intense support of Kyle Burton. Thanks to both of you!
I didn't know, that it is possible to pipe the output of commands executed in a script to a function that belongs to the script. This was the first piece of information necessary.
And I didn't know, how to evaluate the piped information inside the function properly and how to signal from inside the function, that the execution of the command generating the values should be terminated.
Additionally, Kyle found, that the filtering I did by piping the original output to sed and the resulting data to the function inside the script prohibited the script to function as designed. I'm still uncertain, why - but it definitively does.
The original command generating the output is now being piped as it is to the internal function of the script. The filtering is being done inside the function to avoid the problem with sed. Now everything works as expected and I can continue completing the script.
This is the working code of the soultion:
#!/usr/bin/env bash
set -eu -o pipefail
function parser () {
local max="$1"
local max_int
max_int="${max%.*}"
while read tc;
do
#If line is empty, continue
if [ -z "$tc" ]; then
continue
fi
#If first char is 0 (=non-Index Frame), continue
local iskey="${tc:0:1}";
if [ $iskey == "0" ]; then
continue
fi
#Return timecode if intended maximum has been reached
local val="${tc:2:10}"
local tc_int
tc_int="${val%.*}"
if (( "$tc_int" >= "$max_int" )); then
echo "First index frame at/after given Timecode: $tc";
exec 0>&-
return 0
fi
done
}
ffprobe "$1" -hide_banner -select_streams v -show_entries frame=key_frame,best_effort_timestamp_time -of csv=nk=1:p=0:s="|" -v quiet | parser "$2"
Usage:
./script.sh "Name of Movie.avi" 30
where 30 represents the timecode at which the next found index frame is being searched and returned.
I'm writing a program that prints the username and the number of times that the user has logged in, or prints "Unknown user" otherwise.
My code is the following:
iden=$1
c='last | grep -w -c $iden'
if (( $c > 1 ))
then
echo "$iden $c"
else
echo "Unknown user"
fi
And I keep getting this error:
-bash: ((: last | grep -w -c 123: expression recursion level exceeded (error token is "c 123")
To store the output of a command in a variable you need to say var=$(command). Hence, use:
c=$(last | grep -w -c "$iden") # always good to quote the variables
instead of
c='last | grep -w -c $iden'
If you are learning Bash scripting, it is always handy to paste your code in ShellCheck to see the problems you may have.
You can use this too:
c=`last | grep -w -c "$iden"`
I am writing a script which you can pass a file name into as an argument and it'll only run if it's a certain file extension.
flac2mp3 "01 Song.flac"
or
flac2mp3 "01 Song.FLAC"
I know there a lot of scripts out there showing you how to convert flac to mp3, but this is my script and I want to learn how to write the script using this method.
It's so I can learn arguments and for when I feel like converting only 1 individual file. (for multiple files I just wrote a for loop with *.flac inside the script)
I just want to learn how to check if the $1 argument contains *.[Ff][Ll][Aa][Cc]
Here's what I cobbled up together from the internet so far (which I know is embarrassingly wrong but I wanted to show what I was going for) :
#!/bin/bash
#flac2mp3
if [ -z $1 ] && [[$1 !=~ *.[Ff][Ll][Aa][Cc]]];then echo "Give FLAC File Name"; exit 0;fi
OUTF=${1%.flac}.mp3
ARTIST=$(metaflac "$1" --show-tag=ARTIST | sed s/.*=//g)
TITLE=$(metaflac "$1" --show-tag=TITLE | sed s/.*=//g)
ALBUM=$(metaflac "$1" --show-tag=ALBUM | sed s/.*=//g)
GENRE=$(metaflac "$1" --show-tag=GENRE | sed s/.*=//g)
TRACKNUMBER=$(metaflac "$1" --show-tag=TRACKNUMBER | sed s/.*=//g)
DATE=$(metaflac "$1" --show-tag=DATE | sed s/.*=//g)
flac -c -d "$1" | lame -m j -q 0 --vbr-new -V 0 -s 44.1 - "$OUTF"
id3 -t "$TITLE" -T "${TRACKNUMBER:-0}" -a "$ARTIST" -A "$ALBUM" -y "$DATE" -g "${GENRE:-12}" "$OUTF"
done
Please and Thank Your for the help.
Try the following code:
shopt -s nocasematch
if [[ $1 == *flac ]]; then
echo "ok"
fi
This is case insensitive.
EDIT
$ LANG=C help shopt
shopt: shopt [-pqsu] [-o] [optname ...]
Set and unset shell options.
Change the setting of each shell option OPTNAME. Without any option
arguments, list all shell options with an indication of whether or not each
is set.
Options:
-o restrict OPTNAMEs to those defined for use with `set -o'
-p print each shell option with an indication of its status
-q suppress output
-s enable (set) each OPTNAME
-u disable (unset) each OPTNAME
Exit Status:
Returns success if OPTNAME is enabled; fails if an invalid option is
given or OPTNAME is disabled.
If you run shopt alone in a shell, you will see al options available :
$ shopt
autocd on
cdable_vars on
cdspell off
checkhash off
checkjobs off
checkwinsize off
cmdhist on
compat31 off
compat32 off
compat40 off
compat41 off
direxpand off
dirspell off
dotglob on
execfail off
expand_aliases on
extdebug off
extglob on
extquote on
failglob off
force_fignore on
globstar on
gnu_errfmt off
histappend on
histreedit off
histverify off
hostcomplete off
huponexit off
interactive_comments on
lastpipe off
lithist off
login_shell off
mailwarn off
no_empty_cmd_completion off
nocaseglob off
nocasematch off
nullglob off
progcomp on
promptvars on
restricted_shell off
shift_verbose off
sourcepath on
xpg_echo off
To know what does all these options :
man bash | less +/'^SHELL BUILTIN COMMANDS'
then search `shopt from within this section.