BASH getopts Multiple Scripts with Same Options - linux

I have a series of BASH scripts.
I am using getopts to parse arguments from the cmd line (although open to alternatives).
There are a series of common options to these scripts call this options set A
ie queue, ncores etc.
Each script then has a series of extra options ie set B1,B2,B3.
What I want is for script
"1 to be able to take options A+B1"
"2 to be able to take options A+B2"
"3 to be able to take options A+B2"
But I want to be able to store the code for options A in a central location (library/function) with having to write out in each script.
What I want is a way to insert generic code in getopts. Or alternatively a way to run getopts twice.
In fact I've done this by having getopts as a function which is sourced.
But the problem is I cant get the unrecognised option to work them.
I guess one way would be to remove the arguements from options A from the string before passing to a getopts for B1, B2 , B3 etc ?
Thanks Roger

That's a very nice question, to answer which we need to have a good understanding of how getopts works.
The key point here is that getopts is designed to iterate over the supplied arguments in a single loop. Thus, the solution to the question is to split the loop between different files rather then running the command twice:
#!/usr/bin/env bash
# File_1
getopts_common() {
builtin getopts ":ab:${1}" ${2} ${#:3} || return 1
case ${!2} in
'a')
echo 'a triggered'
continue
;;
'b')
echo "b argument supplied -- ${OPTARG}"
continue
;;
':')
echo "MISSING ARGUMENT for option -- ${OPTARG}" >&2
exit 1
;;
esac
}
#!/usr/bin/env bash
# File_2
# source "File_1"
while getopts_common 'xy:' OPTKEY ${#}; do
case ${OPTKEY} in
'x')
echo 'x triggered'
;;
'y')
echo "y argument supplied -- ${OPTARG}"
;;
'?')
echo "INVALID OPTION -- ${OPTARG}" >&2
exit 1
;;
':')
echo "MISSING ARGUMENT for option -- ${OPTARG}" >&2
exit 1
;;
*)
echo "UNIMPLEMENTED OPTION -- ${OPTKEY}" >&2
exit 1
;;
esac
done
Implementation notes
We start with File_2 since that's where the execution of the script starts:
Instead of invoking getopts directly, we call it via it's proxy: getopts_common, which is responsible for processing all common option.
getopts_common function is invoked with:
A string that defines which options to expect, and where to expect their arguments. This string only covers options defined in File_2.
The name of the shell-variable to use for option reporting.
A list of the command line arguments. (This simplifies accessing them from inside getopts_common function.)
Moving on to the sourced file (File_1) we need to bear in mind that getopts_common function runs inside the while loop defined in File_2:
getopts returns false if there is nothing left to parse, || return 1 bit insures that getopts_common function does the same.
The execution needs to move on to the next iteration of the loop when a valid option is processed. Hence, each valid option match ends with continue.
Silent error reporting (enabled when OPTSPEC starts with :) allows us to distinguish between INVALID OPTION and MISSING ARGUMENT. The later error is specific to the common options defined in File_1, thus it needs to be trapped there.
For more in-depth information on getopts, see Bash Hackers Wiki: Getopts tutorial

Related

Redefine echo for several scripts by calling a script that redefines echo

I have scripts A, B, C, etc. that I execute independently. I want to redefine how echo works for all scripts I am using.
I made the following script, preamble.sh, that I call from each script A, B, C:
#!/bin/bash
# Change the color of our output
# TODO: How to redefine echo for the parent caller?
PURPLE='\033[1;35m'
NC='\033[0m' # No Color
function echo() { builtin echo -e '\033[1;35m'$0'\033[0m'; }
unameOut="$(uname -s)"
case "${unameOut}" in
Linux*) machine=Linux ;;
Darwin*) machine=Mac ;;
CYGWIN*) machine=Cygwin ;;
MINGW*) machine=MinGw ;;
*) machine="UNKNOWN:${unameOut}" ;;
esac
echo "[Running on ${machine}...]"
When I do bash preamble.sh from another script, all I get is
preamble.sh
in purple (the color that I want echo to use in each script A, B, C, etc.).
I guess what ends up happening is echo is redefined correctly within preamble.sh but this is not how I expected it to work. When I call bash preamble.sh, preamble.sh gets executed but instead of telling me what machine it runs on, it just prints preamble.sh in purple because that must be the argument $0.
I realize I might be doing something that is not possibly to do directly.
How do I achieve what I am trying to achieve?
The arguments to a function are $1, $2, ...
You want:
function echo() { builtin echo -e '\033[1;35m'$1'\033[0m'; }
not:
function echo() { builtin echo -e '\033[1;35m'$0'\033[0m'; }
Whether inside a function of not, $0 will remain the name of the script itself.
Edit: For your other question, you would need to run the script within the current shell for the changes to persist. You can do this using either
source preamble.sh
or
. preamble.sh
This is necessary since by default, the script will run in a new shell and any variables, functions, etc you define will not be visible.

getopts: unable to identify arguments

This is the script I tried:
#!/bin/bash
while getopts ":u:p:" option; do
case $option in
u) USER=$OPTARG;;
p) PASS=$OPTARG;;
\?) echo "Invalid Option: $OPTARG"
exit 1;;
:) echo "Please provide an argument for $OPTARG!"
exit 1;;
esac
done
echo "Username/Password: $USER/$PASS"
If command for running the script is:
./test9.sh -u test -p -a
Then I am getting an output:
Username/Password: test/-a
-a is an invalid argument but the script is taking -a as password. I would like to display a message Please enter a password and exit the script. Please help me in fixing this.
There are three kinds of parameters: options, option arguments, and positional parameters. If a parameter is an option that requires an argument, then the next parameter will be treated as an an argument no matter what. It may start with a dash or even coincide with a valid option, it will still be treated as an option argument.
If your program wants to reject arguments that start with a dash, you need to program it yourself. Passwords that start with a dash are perfectly legitimate; a program that checks passwords must not reject them.
Option that accept optional arguments are extremely confusing and non-standard. Getopt in general doesn't support them. There's a GNU extension for that, but don't use it.
TL;DR there's nothing to fix, your script is fine.
I haven't tested your script, but I think that if you use getopt instead of getopts you'll get the result you expect, an error because -a is not a valid option.

If I create a menu in Bash is command style history possible

I have no problem creating the menu - that is not what this question is about.
What happens however is I go from having the arrow keys be useful (scroll up and down to get access to previous commands I've run at the command line) to completely useless (^[[A^[[A^[[A^[[B^[[C^[[D^[[C)
Is there any way to encapsulate that behaviour into a menu?
E.g. can I use the scroll up and down keys to access previous options I've selected. (It's a BIG menu and I have MANY options like dev.client.alpha or dev.otherclient.beta etc...)
I supposed I could break each one into separate files and just use the command line diredtly OR I could pass an augment to the menu so as to call: ~/menu dev.clint.alpha directly from the command line.
Just curious is anyone else has (had) this itch and if anything has ever been done about it?
Menu I'm presently using is done basically as follows:
while :
clear
do
echo "$MENU"
read CHOICE ARG1 ARG2 ARG3 ARG4 overflow
case $CHOICE in
command.a)
# do stuff here
;;
command.b)
# do different stuff here
;;
*) # catch all...
continue
;;
esac
done
clear
You can do what you want by enabling readline on your read,
and appending each choice reply to the internal history. You can even save
the history to a file so when you rerun the script you have the old
history. For example:
HISTFILE=/tmp/myhistory
history -r # read old history
while :
do echo "MENU. a b q"
read -e # sets REPLY, enables readline
history -s "$REPLY" # add to history
history -w # save to file
set -- $REPLY
CHOICE=$1; shift
ARG1=$1 ARG2=$2 ARG3=$3 ARG4=$4; shift 4
overflow="$*"
case $CHOICE in
a) echo do stuff here $ARG1 ;;
b) echo do different stuff here ;;
q) exit ;;
*) echo catch all...
continue ;;
esac
done

RHEL6 getopts doesn't seem to be working

I have a new RHEL6 machine and I'm trying to run a script to generate some output. The script uses getopts which I've never used in the past. This should have worked on other machines but is my first time trying it. Below is the beginning of the script. Is there anything wrong with the syntax? When I try to output the variables it displays nothing:
#! /bin/sh
while getopts "h:u:g:o:e:y:bf" c
do
case "$c" in
u) USER=$OPTARG;;
g) GROUP=$OPTARG;;
o) OUT=$OPTARG;;
b) BATCH=1;;
f) FORCE=1;;
h) FQDN=$OPTARG;;
e) ENTITYID=$OPTARG;;
y) YEARS=$OPTARG;;
\?) echo "keygen [-o output directory (default .)] [-u username to own keypair] [-g owning groupname] [-h hostname for cert] [-y years to issue cert] [-e entityID to embed in cert]"
exit 1;;
esac
done
echo $FQDN
The echo displays a blank line.
You can't use question mark with the bash getopts (you also can't use the colon). In the case of question mark, getopts sets the value of the argument ($c in your case) to a question mark when the end of options has been encountered. It also uses question mark and colon as the value for the argument name when there's an error (specifically, ? is used when an invalid option is encountered or when in non-silent mode and a required option is not provided; colon is used in silent mode when a required option is not provided). In those error cases, OPTARG contains the offending argument. This is how POSIX getopts works as well.
The KSH getopts behaves differently, but it also excludes ? : (as well as - [ ] and only allowing # as the first option). It does, however, show a usage message when you provide -?. Basically, don't use -? with shell getopts. :)
Typically, I write a small function called "usage" and call it from both *) and by checking $? immediately after the case statement for non-zero value.

Why should eval be avoided in Bash, and what should I use instead?

Time and time again, I see Bash answers on Stack Overflow using eval and the answers get bashed, pun intended, for the use of such an "evil" construct. Why is eval so evil?
If eval can't be used safely, what should I use instead?
There's more to this problem than meets the eye. We'll start with the obvious: eval has the potential to execute "dirty" data. Dirty data is any data that has not been rewritten as safe-for-use-in-situation-XYZ; in our case, it's any string that has not been formatted so as to be safe for evaluation.
Sanitizing data appears easy at first glance. Assuming we're throwing around a list of options, bash already provides a great way to sanitize individual elements, and another way to sanitize the entire array as a single string:
function println
{
# Send each element as a separate argument, starting with the second element.
# Arguments to printf:
# 1 -> "$1\n"
# 2 -> "$2"
# 3 -> "$3"
# 4 -> "$4"
# etc.
printf "$1\n" "${#:2}"
}
function error
{
# Send the first element as one argument, and the rest of the elements as a combined argument.
# Arguments to println:
# 1 -> '\e[31mError (%d): %s\e[m'
# 2 -> "$1"
# 3 -> "${*:2}"
println '\e[31mError (%d): %s\e[m' "$1" "${*:2}"
exit "$1"
}
# This...
error 1234 Something went wrong.
# And this...
error 1234 'Something went wrong.'
# Result in the same output (as long as $IFS has not been modified).
Now say we want to add an option to redirect output as an argument to println. We could, of course, just redirect the output of println on each call, but for the sake of example, we're not going to do that. We'll need to use eval, since variables can't be used to redirect output.
function println
{
eval printf "$2\n" "${#:3}" $1
}
function error
{
println '>&2' '\e[31mError (%d): %s\e[m' "$1" "${*:2}"
exit $1
}
error 1234 Something went wrong.
Looks good, right? Problem is, eval parses twice the command line (in any shell). On the first pass of parsing one layer of quoting is removed. With quotes removed, some variable content gets executed.
We can fix this by letting the variable expansion take place within the eval. All we have to do is single-quote everything, leaving the double-quotes where they are. One exception: we have to expand the redirection prior to eval, so that has to stay outside of the quotes:
function println
{
eval 'printf "$2\n" "${#:3}"' $1
}
function error
{
println '&2' '\e[31mError (%d): %s\e[m' "$1" "${*:2}"
exit $1
}
error 1234 Something went wrong.
This should work. It's also safe as long as $1 in println is never dirty.
Now hold on just a moment: I use that same unquoted syntax that we used originally with sudo all of the time! Why does it work there, and not here? Why did we have to single-quote everything? sudo is a bit more modern: it knows to enclose in quotes each argument that it receives, though that is an over-simplification. eval simply concatenates everything.
Unfortunately, there is no drop-in replacement for eval that treats arguments like sudo does, as eval is a shell built-in; this is important, as it takes on the environment and scope of the surrounding code when it executes, rather than creating a new stack and scope like a function does.
eval Alternatives
Specific use cases often have viable alternatives to eval. Here's a handy list. command represents what you would normally send to eval; substitute in whatever you please.
No-op
A simple colon is a no-op in bash:
:
Create a sub-shell
( command ) # Standard notation
Execute output of a command
Never rely on an external command. You should always be in control of the return value. Put these on their own lines:
$(command) # Preferred
`command` # Old: should be avoided, and often considered deprecated
# Nesting:
$(command1 "$(command2)")
`command "\`command\`"` # Careful: \ only escapes $ and \ with old style, and
# special case \` results in nesting.
Redirection based on variable
In calling code, map &3 (or anything higher than &2) to your target:
exec 3<&0 # Redirect from stdin
exec 3>&1 # Redirect to stdout
exec 3>&2 # Redirect to stderr
exec 3> /dev/null # Don't save output anywhere
exec 3> file.txt # Redirect to file
exec 3> "$var" # Redirect to file stored in $var--only works for files!
exec 3<&0 4>&1 # Input and output!
If it were a one-time call, you wouldn't have to redirect the entire shell:
func arg1 arg2 3>&2
Within the function being called, redirect to &3:
command <&3 # Redirect stdin
command >&3 # Redirect stdout
command 2>&3 # Redirect stderr
command &>&3 # Redirect stdout and stderr
command 2>&1 >&3 # idem, but for older bash versions
command >&3 2>&1 # Redirect stdout to &3, and stderr to stdout: order matters
command <&3 >&4 # Input and output!
Variable indirection
Scenario:
VAR='1 2 3'
REF=VAR
Bad:
eval "echo \"\$$REF\""
Why? If REF contains a double quote, this will break and open the code to exploits. It's possible to sanitize REF, but it's a waste of time when you have this:
echo "${!REF}"
That's right, bash has variable indirection built-in as of version 2. It gets a bit trickier than eval if you want to do something more complex:
# Add to scenario:
VAR_2='4 5 6'
# We could use:
local ref="${REF}_2"
echo "${!ref}"
# Versus the bash < 2 method, which might be simpler to those accustomed to eval:
eval "echo \"\$${REF}_2\""
Regardless, the new method is more intuitive, though it might not seem that way to experienced programmed who are used to eval.
Associative arrays
Associative arrays are implemented intrinsically in bash 4. One caveat: they must be created using declare.
declare -A VAR # Local
declare -gA VAR # Global
# Use spaces between parentheses and contents; I've heard reports of subtle bugs
# on some versions when they are omitted having to do with spaces in keys.
declare -A VAR=( ['']='a' [0]='1' ['duck']='quack' )
VAR+=( ['alpha']='beta' [2]=3 ) # Combine arrays
VAR['cow']='moo' # Set a single element
unset VAR['cow'] # Unset a single element
unset VAR # Unset an entire array
unset VAR[#] # Unset an entire array
unset VAR[*] # Unset each element with a key corresponding to a file in the
# current directory; if * doesn't expand, unset the entire array
local KEYS=( "${!VAR[#]}" ) # Get all of the keys in VAR
In older versions of bash, you can use variable indirection:
VAR=( ) # This will store our keys.
# Store a value with a simple key.
# You will need to declare it in a global scope to make it global prior to bash 4.
# In bash 4, use the -g option.
declare "VAR_$key"="$value"
VAR+="$key"
# Or, if your version is lacking +=
VAR=( "$VAR[#]" "$key" )
# Recover a simple value.
local var_key="VAR_$key" # The name of the variable that holds the value
local var_value="${!var_key}" # The actual value--requires bash 2
# For < bash 2, eval is required for this method. Safe as long as $key is not dirty.
local var_value="`eval echo -n \"\$$var_value\""
# If you don't need to enumerate the indices quickly, and you're on bash 2+, this
# can be cut down to one line per operation:
declare "VAR_$key"="$value" # Store
echo "`var_key="VAR_$key" echo -n "${!var_key}"`" # Retrieve
# If you're using more complex values, you'll need to hash your keys:
function mkkey
{
local key="`mkpasswd -5R0 "$1" 00000000`"
echo -n "${key##*$}"
}
local var_key="VAR_`mkkey "$key"`"
# ...
How to make eval safe
eval can be safely used - but all of its arguments need to be quoted first. Here's how:
This function which will do it for you:
function token_quote {
local quoted=()
for token; do
quoted+=( "$(printf '%q' "$token")" )
done
printf '%s\n' "${quoted[*]}"
}
Example usage:
Given some untrusted user input:
% input="Trying to hack you; date"
Construct a command to eval:
% cmd=(echo "User gave:" "$input")
Eval it, with seemingly correct quoting:
% eval "$(echo "${cmd[#]}")"
User gave: Trying to hack you
Thu Sep 27 20:41:31 +07 2018
Note you were hacked. date was executed rather than being printed literally.
Instead with token_quote():
% eval "$(token_quote "${cmd[#]}")"
User gave: Trying to hack you; date
%
eval isn't evil - it's just misunderstood :)
I’ll split this answer in two parts, which, I think, cover a large proportion of the cases where people tend to be tempted by eval:
Running weirdly built commands
Fiddling with dynamically named variables
Running weirdly built commands
Many, many times, simple indexed arrays are enough, provided that you take on good habits regarding double quotes to protect expansions while defining the array.
# One nasty argument which must remain a single argument and not be split:
f='foo bar'
# The command in an indexed array (use `declare -a` if you really want to be explicit):
cmd=(
touch
"$f"
# Yet another nasty argument, this time hardcoded:
'plop yo'
)
# Let Bash expand the array and run it as a command:
"${cmd[#]}"
This will create foo bar and plop yo (two files, not four).
Note that sometimes it can produce more readable scripts to put just the arguments (or a bunch of options) in the array (at least you know at first glance what you’re running):
touch "${args[#]}"
touch "${opts[#]}" file1 file2
As a bonus, arrays let you, easily:
Add comments about a specific argument:
cmd=(
# Important because blah blah:
-v
)
Group arguments for readability by leaving blank lines within the array definition.
Comment out specific arguments for debugging purposes.
Append arguments to your command, sometimes dynamically according to specific conditions or in loops:
cmd=(myprog)
for f in foo bar
do
cmd+=(-i "$f")
done
if [[ $1 = yo ]]
then
cmd+=(plop)
fi
to_be_added=(one two 't h r e e')
cmd+=("${to_be_added[#]}")
Define commands in configuration files while allowing for configuration-defined whitespace-containing arguments:
readonly ENCODER=(ffmpeg -blah --blah 'yo plop')
# Deprecated:
#readonly ENCODER=(avconv -bloh --bloh 'ya plap')
# […]
"${ENCODER[#]}" foo bar
Log a robustly runnable command, that perfectly represents what is being run, using printf’s %q:
function please_log_that {
printf 'Running:'
# From `help printf`:
# “The format is re-used as necessary to consume all of the arguments.”
# From `man printf` for %q:
# “printed in a format that can be reused as shell input,
# escaping non-printable characters with the proposed POSIX $'' syntax.”
printf ' %q' "$#"
echo
}
arg='foo bar'
cmd=(prog "$arg" 'plop yo' $'arg\nnewline\tand tab')
please_log_that "${cmd[#]}"
# ⇒ “Running: prog foo\ bar plop\ yo $'arg\nnewline\tand tab'”
# You can literally copy and paste that ↑ to a terminal and get the same execution.
Enjoy better syntax highlighting than with eval strings, since you don’t need to nest quotes or use $-s that “will not be evaluated right away but will be at some point”.
To me, the main advantage of this approach (and conversely disadvantage of eval) is that you can follow the same logic as usual regarding quotation, expansion, etc. No need to rack your brain trying to put quotes in quotes in quotes “in advance” while trying to figure out which command will interpret which pair of quotes at which moment. And of course many of the things mentioned above are harder or downright impossible to achieve with eval.
With these, I never had to rely on eval in the past six years or so, and readability and robustness (in particular regarding arguments that contain whitespace) were arguably increased. You don’t even need to know whether IFS has been tempered with! Of course, there are still edge cases where eval might actually be needed (I suppose, for example, if the user has to be able to provide a full fledged piece of script via an interactive prompt or whatever), but hopefully that’s not something you’ll come across on a daily basis.
Fiddling with dynamically named variables
declare -n (or its within-functions local -n counterpart), as well as ${!foo}, do the trick most of the time.
$ help declare | grep -- -n
-n make NAME a reference to the variable named by its value
Well, it’s not exceptionally clear without an example:
declare -A global_associative_array=(
[foo]=bar
[plop]=yo
)
# $1 Name of global array to fiddle with.
fiddle_with_array() {
# Check this if you want to make sure you’ll avoid
# circular references, but it’s only if you really
# want this to be robust.
# You can also give an ugly name like “__ref” to your
# local variable as a cheaper way to make collisions less likely.
if [[ $1 != ref ]]
then
local -n ref=$1
fi
printf 'foo → %s\nplop → %s\n' "${ref[foo]}" "${ref[plop]}"
}
# Call the function with the array NAME as argument,
# not trying to get its content right away here or anything.
fiddle_with_array global_associative_array
# This will print:
# foo → bar
# plop → yo
(I love this trick ↑ as it makes me feel like I’m passing objects to my functions, like in an object-oriented language. The possibilities are mind-boggling.)
As for ${!…} (which gets the value of the variable named by another variable):
foo=bar
plop=yo
for var_name in foo plop
do
printf '%s = %q\n' "$var_name" "${!var_name}"
done
# This will print:
# foo = bar
# plop = yo

Resources