zsh doesn't recognize options set with -o, but only when its in a shebang, and on Linux.
The following script fails on zsh 5.0.2, zsh 5.6, and the latest git:
#!/bin/zsh -o pipefail
thiswillfail | echo 'hello, world'
echo $?
exit
Expected Output
hello, world
/home/prajjwal/script:2: command not found: thiswillfail
127
Actual Output
/bin/zsh: no such option: pipefail
What works
The script on zsh 5.3 on MacOS Mojave. This appears to be failing on every Linux version I've tried so far, though.
Manually invoking /bin/zsh -o pipefail on a terminal
Setting the option with set -o pipefail after the shebang in the script.
What I've tried
Emptying my .zshrc's to ensure one of my settings isn't causing this.
Aside
While I'm only trying to get pipefail to work, this refuses to work with any other options that I try to set, even though all of them are mentioned in zshoptions.
This is a pitfall of trying to set options on the #! line. The kernel only splits the shebang line at the first space.
When you say
#!/bin/zsh -o pipefail
the command that ends up being executed is "/bin/zsh" "-o pipefail" "/home/prajjwal/script".
The error message says that " pipefail" (note the leading space) is not a valid option, which is correct: Only "pipefail" is valid.
In this case a possible workaround is to cram everything into a single command line argument:
#!/bin/zsh -opipefail
But in general that's not possible and the #! interface doesn't let you pass more than one extra argument, so your choices are to find other workarounds (e.g. using set manually) or to write a wrapper script:
#!/bin/zsh
exec /bin/zsh -o pipefail /path/to/the/real/script
On some systems another alternative is available: env -S. This option is not specified by POSIX and is not available on all systems. It works on systems using the GNU tools (including all standard Linux distributions) and FreeBSD, but not OpenBSD or NetBSD. I couldn't find any information about MacOS.
(Note: GNU env also supports --split-string as an alternative (long) option name, but FreeBSD env does not.)
Usage:
#!/usr/bin/env -S /bin/zsh -o pipefail
This invokes env with a single long argument ('-S /bin/zsh -o pipefail'). Standard option processing treats it as the -S option followed by an argument (' /bin/zsh -o pipefail').
In a simple case like this, env -S splits the argument on spaces/tabs and treats the resulting list as if it had been part of the original command line in the first place:
env -S ' /bin/zsh -o pipefail'
# works the same as:
env /bin/zsh -o pipefail
In less simple cases you'll have to quote some characters (env -S treats spaces, ", ', \, $ specially, among others). In particular, it does not work like shell quoting. For details refer to the manual pages linked above.
Related
I'm writing a bash script and it throws an error when using "sh" command in Ubuntu (it seems it's not compatible with dash, I'm learning on this subject). So I would like to detect if dash is being used instead of bash to throw an error.
How can I detect it in a script context?. Is it even possible?
You can check for the presence of shell-specific variables:
For instance, bash defines $BASH_VERSION.
Since that variable won't be defined while running in dash, you can use it to make the distinction:
[ -n "$BASH_VERSION" ] && isBash=1
Afterthought: If you wanted to avoid relying on variables (which, conceivably, could be set incorrectly), you could try to obtain the ultimate name of the shell executable running your script, by determining the invoking executable and, if it is a symlink, following it to its (ultimate) target.
The shell function getTrueShellExeName() below does that; for instance, it would return 'dash' on Ubuntu for a script run with sh (whether explicitly or via shebang #!/bin/sh), because sh is symlinked to dash there.
Note that the function's goal is twofold:
Be portable:
Work with all POSIX-compatible (Bourne-like) shells,
across at least most platforms, with respect to what utilities and options are used - see caveats below.
Work in all invocation scenarios:
sourced (whether from a login shell or not)
executed stand-alone, via the shebang line
executed by being passed as a filename argument to a shell executable
executed by having its contents piped via stdin to a shell executable
Caveats:
On at least one platform - macOS - sh is NOT a symlink, even though it is effectively bash. There, the function would return 'sh' in a script run with sh.
The function uses readlink, which, while not mandated by POSIX, is present on most modern platforms - though with differing syntax and features. Therefore, using GNU readlink's -f option to find a symlink's ultimate target is not an option.
(The only modern platform I'm personally aware of that does not have a readlink utility is HP-UX - see https://stackoverflow.com/a/24114056/45375 for a recursive-readlink implementation that should work on all POSIX platforms.)
The function uses the which utility (except in zsh, where it's a builtin), which, while not mandated by POSIX, is present on most modern platforms.
Ideally, ps -p $$ -o comm= would be sufficient to determine the path of the executable underlying the process, but that doesn't work as intended when directly executing shell scripts with shebang lines on Linux, at least when using the ps implementation from the procps-ng package, as found on Ubuntu, for instance: there, such scripts report the script's file name rather than the underlying script engine's.Tip of the hat to ferdymercury for his help.
Therefore, the content of special file /proc/$$/cmdline is parsed on Linux, whose first NUL-separated field contains the true executable path.
Example use of the function:
[ "$(getTrueShellExeName)" = 'bash' ] && isBash=1
Shell function getTrueShellExeName():
getTrueShellExeName() {
local trueExe nextTarget 2>/dev/null # ignore error in shells without `local`
# Determine the shell executable filename.
if [ -r /proc/$$/cmdline ]; then
trueExe=$(cut -d '' -f1 /proc/$$/cmdline) || return 1
else
trueExe=$(ps -p $$ -o comm=) || return 1
fi
# Strip a leading "-", as added e.g. by macOS for login shells.
[ "${trueExe#-}" = "$trueExe" ] || trueExe=${trueExe#-}
# Determine full executable path.
[ "${trueExe#/}" != "$trueExe" ] || trueExe=$([ -n "$ZSH_VERSION" ] && which -p "$trueExe" || which "$trueExe")
# If the executable is a symlink, resolve it to its *ultimate*
# target.
while nextTarget=$(readlink "$trueExe"); do trueExe=$nextTarget; done
# Output the executable name only.
printf '%s\n' "$(basename "$trueExe")"
}
Use $0 (that is the name of the executable of the shell being called).The command for example
echo $0
gives
/usr/bin/dash
for the dash and
/bin/bash
for a bash.The parameter substitution
${0##*/}
gives just 'dash' or 'bash'. This can be used in a test.
An alternative approach might be to test if a shell feature is available, for example to give an idea...
[[ 1 ]] 2>/dev/null && echo could be bash || echo not bash, maybe dash
echo $0 and [[ 1 ]] 2>/dev/null && echo
could be bash || echo not bash, maybe bash worked for me running Ubuntu 19.
Done slight Pascal, Fortran and C in school, but need to become fluent in shell script.
So I have this shell script that checks and then concats an environmental variable to /etc/environment, then reloads the file without having to logout/login:
#!/bin/sh
portvar="PORT=5000"
echo $portvar
grep -q $portvar /etc/environment && echo "EV already in" || echo $portvar >> /etc/environment
set -a; source /etc/environment; set +a;
When I run it, I get the error ./test.sh: 5: ./test.sh: source: not found. However, if I run set -a; source /etc/environment; set +a; directly in the terminal it updates the environmental variable just fine. I have no idea what the set command does, I just found it in another stack overflow question.
Any idea why it runs in the terminal directly but not in the .sh file?
Thanks
/bin/sh on your system is likely some shell that isn't bash and doesn't implement the source command. On my Ubuntu 20.04 system /bin/sh is actually dash.
The source command is not defined by POSIX as part of the shell command language nor is it one of the required special built-in utilities. It's a non-standard feature provided by bash. However, the . command, which does the same thing, is specified by POSIX.
So you can use . instead, e.g. . /etc/environment. Or if you want to keep using source, then you need to have your script run by bash or some other shell that supports it, by changing the shebang line to #!/bin/bash.
There is a tool called checkbashisms that can help you find unintentional uses of bash-specific features in your scripts. When run on your script, it flags this:
possible bashism in foo.sh line 5 (should be '.', not 'source'):
I have been working on a shell script to automate some tasks. What is the best way to make sure the shell script would run without any issues in most of the platforms. For ex., I have been using echo -n command to print some messages to the screen without a trailing new line and the -n switch doesn't work in some ksh shells. I was told the script must be POSIX compliant. How do I make sure that the script is POSIX compliant. Is there a tool? Or is there a shell that supports only bare minimum POSIX requirements?
POSIX
One first step, which gives you indications of what works or not and why, is to set the shebang to /bin/sh and use shellcheck site to analyze your script.
For example, paste this script in the shellcheck editor window:
#!/bin/sh
read -r a b <<<"$1"
echo $((a+b))
to get an indication that: "In POSIX sh, here-strings are undefined".
As a second step, you can use a shell that is as compatible with POSIX as possible.
One shell that is compatible with most other simple shells, is dash, Debian default system shell, which is a derivative of the older BSD ash.
Another shell compatible with posix is posh.
However, dash and/or posh may not be available for some systems.
There is lksh (with a ksh flavor), with the goal to be compatible with legacy (old) shell scripts. From its manual:
lksh is a command interpreter intended exclusively for running legacy shell scripts.
But there is the need to use options when calling lksh, like -o posix and -o sh:
Note that it's strongly recommended to invoke lksh with at least the -o posix option, if not both that and -o sh, to fully enjoy better compatibility to the POSIX standard (which is probably why you use lksh over mksh in the first place) or legacy scripts, respectively.
You would call lksh -o posix -o sh instead of the simple lksh.
Using options is a way to make other shells become POSIX compatible. Like lksh, using the option -o posix, like bash -o posix.
In bash, it is even possible to turn the POSIX option inside an script, with:
shopt -o posix # also with: set -o posix
It is also possible to make a local link to bash or zsh that makes both act like an old sh shell. Like this:
$ ln -s /bin/bash ./sh
$ ./sh
There are plenty of alternatives (dash, posh, lksh, bash, zsh, etc.) to get a shell that will work as a POSIX shell.
Portable
However, even so, all the above does not ensure "portability".
Unfortunately, making a shell script 'POSIX-compliant' is usually easier than making it run on any real-world shell.
The only real-world sensible recommendation is test your script in several shells.
Like the list above: dash, posh, lksh, and bash --posix.
Solaris is a world on its own, probably you will need to test against /bin/sh and xpg4/sh.
Followup:
How can I test for POSIX compliance for shell scripts?
Starting Bash with the --posix command-line option or executing ‘set -o posix’ while Bash is running will cause Bash to conform more closely to the POSIX standard by changing the behavior to match that specified by POSIX in areas where the Bash default differs.
Reference
Note:
This answer complements user8017719's great answer.
As requested in the question, a tool is discussed below: while it does not directly check for POSIX compliance, it runs a given script in multiple shells, notably including /bin/sh.
/bin/sh, the system default shell, should not be assumed to support any features other than POSIX-prescribed ones, though in practice it does, to varying degrees, depending on the specific implementation. Therefore, successfully running via /bin/sh on one platform does not guarantee that the script will work on another. Among widely used shells, dash comes closest to being a POSIX-features-only shell.
Running successfully in multiple shells is important:
if you're authoring a script that needs to be sourced in various shells.
if you know that your script will encounter only a limited set of known-in-advance shells.
For a proof-of-the-pudding-is-in-the-eating approach, consider using shall (a utility I wrote), which allows you to invoke a given script or command with multiple shells at once, with feedback about which of the targeted shells the script/command executed successfully with.
If you have Node.js installed, you can easily install it with npm install -g shall (if not, follow the above link to the GitHub repo for manual installation instructions) and then use it as follows:
shall scriptFile
or, with an ad-hoc command:
shall -c '<shell-commands>'
By default, it invokes sh, and, if installed, dash, bash, zsh, and ksh, but you can target any set of shells that you have installed by using the SHELLS environment variable.
Using the example of the echo -n command on macOS to only target shells sh and bash:
$ SHELLS=sh,bash shall -c 'echo -n hi'
✓ sh (bash variant) [0.00s]
-n hi
✓ bash [0.00s]
hi
OK - All 2 shells (sh, bash) report success.
On macOS, bash (effectively) acts as sh, and while echo -n didn't fail when used with sh, you can also see that -n wasn't recognized as an option when bash ran as sh.
Another macOS example that shows that bash permits certain Bash-specific extensions even when running as sh, such as using nonstandard [[ ... ]] conditionals (assumes that dash - which acts as sh on Ubuntu systems - was installed via Homebrew):
$ SHELLS=sh,bash,dash shall -c '[[ -n nonempty ]] && echo nonempty'
✓ sh (bash variant) [0.00s]
nonempty
✓ bash [0.00s]
nonempty
✗ dash [0.01s]
dash: 1: [[: not found
FAILED - 1 shell (dash) reports failure, 2 (sh, bash) report success.
As you can see, Bash running as sh still accepted [[ ... ]], whereas dash, which is a (mostly) POSIX-features-only shell, failed, because POSIX only mandates [ ... ] conditionals (as an alias of test ... commands).
Someone can explain me why when I copy and paste the following command in the terminal it displays the colorful test correctly, but when I run it via sh myscript.sh it does not display the colored text?
blue='\e[1;34m'
NC='\e[0m'
echo -e "${blue}Test${NC}"
EDIT
Sudo is not the problem. If I copy the above and paste directly into the terminal, everything works. If you run through file, sh myscript.sh not work
Probably because sh isn't bash on your system.
$ file /bin/sh
/bin/sh: symbolic link to `dash'
Try
bash myscript.sh
Your interactive shell seems to be GNU Bash, while sh is a generic POSIX shell, which actually may be dash, busybox sh or something else. The problem is that neither -e option for echo nor \e are POSIX-compliant.
But you can easily use printf instead of echo -e (do not forget to explicitly specify newline character \n) and \033 instead of \e:
blue='\033[1;34m'
NC='\033[0m'
printf "${blue}%s${NC}\n" 'Test'
Or, of course, you can just use bash (as Elliott Frisch suggested) if you are sure that it would be available on target system.
Also I should point out, that what you done is not right way to run shell scripts at all. If you’re writing a standalone script, then you’d better to use hashbang and set execution bit to file.
$ cat myscript
#!/bin/sh
blue='\033[1;34m'
NC='\033[0m'
printf "${blue}%s${NC}\n" 'Test'
$ chmod +x myscript
$ ./myscript
But if you’re writing a command sequence (a macros, if you will) for interactive shell, there is source (or simply .) command:
$ source myscript
(Then all of above about POSIX-compliance does not matter of course.)
I am using bash and this works on Linux:
read -r -d '' VAR<<-EOF
Hello\nWorld
EOF
echo $VAR > trail
i.e the contents of the file on Linux is
Hello\nWorld
When i run on Solaris
trial file has
Hello
World
The newline(\n) is being replaced with a newline. How can i avoid it?
Is it a problem with heredoc or the echo command?
[UPDATE]
Based on the explanation provided here:
echo -E $VAR > trail
worked fine on Solaris.
The problem is with echo. Behavior is defined in POSIX, where interpretting \n is part of XSI but not basic POSIX itself.
You can avoid this on all platforms using printf (which is good practice anyways):
printf "%s\n" "$VAR"
This is not a problem for bash by the way. If you had used #!/usr/bin/env bash as the shebang (and also not run the script with sh script), behavior would have been consistent.
If you use #!/bin/sh, you'll get whichever shell the system uses as a default, with varying behaviors like this.
To complement #that other guy's helpful answer:
Even when it is bash executing your script, there are several ways in which the observed behavior - echo by default interpreting escape sequences such as \n - can come about:
shopt -s xpg_echo could be in effect, which makes the echo builtin interpret \ escape sequences by default.
enable -n echo could be in effect, which disables the echo builtin and runs the external executable by default - and that executable's behavior is platform-dependent.
These options are normally NOT inherited when you run a script, but there are still ways in which they could take effect:
If your interactive initialization files (e.g., ~/.bashrc) contain commands such as the above and you source (.) your script from an interactive shell.
When not sourcing your script: If your environment contains a BASH_ENV variable that points to a script, that script is sourced before your script runs; thus, if that script contains commands such as the above, they will affect your script.