By using zsh for some time along with oh-my-zsh framework, I noticed that which command behaves different in zsh, than in bash.
What I mean:
# on zsh
ilias#ilias-pc ~ ➜ which ls
ls: aliased to ls --color=auto
ilias#ilias-pc ~ ➜ which which
which: shell built-in command
ilias#ilias-pc ~ ➜ bash
[ilias#ilias-pc ~]$ which ls
/usr/bin/ls
[ilias#ilias-pc ~]$ which which
/usr/bin/which
[ilias#ilias-pc ~]$
Why does this happen and how can I "fix" it?
PS. I reproduce this on Arch Linux (not sure whether it matters but I mention it).
$ zsh -c 'type which'
which is a shell builtin
$ bash -c 'type which'
which is /usr/bin/which
The solution is to not use which(1), which is a non-standard and not very useful command. The question of what you should use instead isn't the most straightforward due to the alternatives being poorly specified and inconsistently implemented, but they are better than which.
Depending on your requirements, you want command (see the -v option), type, or whence. See POSIX for the former two, or your shell manual for the latter. (Bash doesn't support whence, but it is supported by most other ksh derivatives, albeit inconsistently. It typically has the most features).
In ZSH, which is equivalent to whence -c (showing function's definitions), not whence -p (which tells executable path). If you want to change that, make an alias.
Related
I am using iterm2 on Mac OS Catalina, however, I prefer all the GNU utils over the FreeBSD ones. Specifically, the cp command in FreeBSD lacks the -t option to specify the target, which I like to use when piping find | xargs cp -t <dest>.
So I used brew to install the GNU core utils as described in this post: https://apple.stackexchange.com/questions/69223/how-to-replace-mac-os-x-utilities-with-gnu-core-utilities
And so, I now have the GNU versions of the common shell tools, my ls is now using the /usr/local/opt/coreutils/libexec/gnubin/ls. The downside is that my ls colors are now gone. See below:
env and command outputs
I can obviously get around this by aliasing my ls command with the /bin/ls but I am wondering if there is a better way. How can I get the GNU ls to recognize my environment settings for colors?
You need to use dircolors to change the coreutil ls output. This link has details on how to use it. https://www.gnu.org/software/coreutils/manual/html_node/dircolors-invocation.html
I had the same issue.
Looks like all GNU ls needed was --color=auto
So this alias in my .zshrc worked -
alias ls='ls -G --color=auto'
I assume the same should work in .bashrc (not tested).
On linux systems when you type a command in a shell like rm * -rf, the order of the * and the -rf doesn't matter. My shell interpret it the same way. Now, on my Mac when I type rm -rf * everything works fine, but if I do rm * -rf an error shows up rm: -rf: No such file or directory
I tried that on a macOS and a linux both with fish and bash shells. Same problems.
Does anyone have any idea why the command interpreter on macOS thinks that -rf at the end of the command is not interpreted as parameters of the command ?
It's not about the shell, it's about the commands.
The parsing of command line arguments is not a feature and responsibility of the shell, but of the actual commands.
In both systems the shell faithfully passes the command line arguments in whatever order they were specified, and then it's up to the implementation of the commands to parse them as they see fit.
In linux, the core utilities are typically of the GNU implementation,
while on osx, the core utilities are typically of the BSD implementation.
The man page of the commands should tell you which implementation it is.
For example the last line of man rm in Linux is something like this:
GNU coreutils 8.21 March 2016 RM(1)
On osx:
BSD January 28, 1999 BSD
Order of the arguments in any shell has historically been relevant in unix.
rm incidentally even has an option -- to stop parsing options (to be able to remove files that start with "-" e.g.)
See rm(1) and getopt(3) man pages
if the shell doesn't respect order of the arguments it is given just what would the result be of this sequence:
$ touch a b
$ mv a b
what file would remain ?
I have been working on a shell script to automate some tasks. What is the best way to make sure the shell script would run without any issues in most of the platforms. For ex., I have been using echo -n command to print some messages to the screen without a trailing new line and the -n switch doesn't work in some ksh shells. I was told the script must be POSIX compliant. How do I make sure that the script is POSIX compliant. Is there a tool? Or is there a shell that supports only bare minimum POSIX requirements?
POSIX
One first step, which gives you indications of what works or not and why, is to set the shebang to /bin/sh and use shellcheck site to analyze your script.
For example, paste this script in the shellcheck editor window:
#!/bin/sh
read -r a b <<<"$1"
echo $((a+b))
to get an indication that: "In POSIX sh, here-strings are undefined".
As a second step, you can use a shell that is as compatible with POSIX as possible.
One shell that is compatible with most other simple shells, is dash, Debian default system shell, which is a derivative of the older BSD ash.
Another shell compatible with posix is posh.
However, dash and/or posh may not be available for some systems.
There is lksh (with a ksh flavor), with the goal to be compatible with legacy (old) shell scripts. From its manual:
lksh is a command interpreter intended exclusively for running legacy shell scripts.
But there is the need to use options when calling lksh, like -o posix and -o sh:
Note that it's strongly recommended to invoke lksh with at least the -o posix option, if not both that and -o sh, to fully enjoy better compatibility to the POSIX standard (which is probably why you use lksh over mksh in the first place) or legacy scripts, respectively.
You would call lksh -o posix -o sh instead of the simple lksh.
Using options is a way to make other shells become POSIX compatible. Like lksh, using the option -o posix, like bash -o posix.
In bash, it is even possible to turn the POSIX option inside an script, with:
shopt -o posix # also with: set -o posix
It is also possible to make a local link to bash or zsh that makes both act like an old sh shell. Like this:
$ ln -s /bin/bash ./sh
$ ./sh
There are plenty of alternatives (dash, posh, lksh, bash, zsh, etc.) to get a shell that will work as a POSIX shell.
Portable
However, even so, all the above does not ensure "portability".
Unfortunately, making a shell script 'POSIX-compliant' is usually easier than making it run on any real-world shell.
The only real-world sensible recommendation is test your script in several shells.
Like the list above: dash, posh, lksh, and bash --posix.
Solaris is a world on its own, probably you will need to test against /bin/sh and xpg4/sh.
Followup:
How can I test for POSIX compliance for shell scripts?
Starting Bash with the --posix command-line option or executing ‘set -o posix’ while Bash is running will cause Bash to conform more closely to the POSIX standard by changing the behavior to match that specified by POSIX in areas where the Bash default differs.
Reference
Note:
This answer complements user8017719's great answer.
As requested in the question, a tool is discussed below: while it does not directly check for POSIX compliance, it runs a given script in multiple shells, notably including /bin/sh.
/bin/sh, the system default shell, should not be assumed to support any features other than POSIX-prescribed ones, though in practice it does, to varying degrees, depending on the specific implementation. Therefore, successfully running via /bin/sh on one platform does not guarantee that the script will work on another. Among widely used shells, dash comes closest to being a POSIX-features-only shell.
Running successfully in multiple shells is important:
if you're authoring a script that needs to be sourced in various shells.
if you know that your script will encounter only a limited set of known-in-advance shells.
For a proof-of-the-pudding-is-in-the-eating approach, consider using shall (a utility I wrote), which allows you to invoke a given script or command with multiple shells at once, with feedback about which of the targeted shells the script/command executed successfully with.
If you have Node.js installed, you can easily install it with npm install -g shall (if not, follow the above link to the GitHub repo for manual installation instructions) and then use it as follows:
shall scriptFile
or, with an ad-hoc command:
shall -c '<shell-commands>'
By default, it invokes sh, and, if installed, dash, bash, zsh, and ksh, but you can target any set of shells that you have installed by using the SHELLS environment variable.
Using the example of the echo -n command on macOS to only target shells sh and bash:
$ SHELLS=sh,bash shall -c 'echo -n hi'
✓ sh (bash variant) [0.00s]
-n hi
✓ bash [0.00s]
hi
OK - All 2 shells (sh, bash) report success.
On macOS, bash (effectively) acts as sh, and while echo -n didn't fail when used with sh, you can also see that -n wasn't recognized as an option when bash ran as sh.
Another macOS example that shows that bash permits certain Bash-specific extensions even when running as sh, such as using nonstandard [[ ... ]] conditionals (assumes that dash - which acts as sh on Ubuntu systems - was installed via Homebrew):
$ SHELLS=sh,bash,dash shall -c '[[ -n nonempty ]] && echo nonempty'
✓ sh (bash variant) [0.00s]
nonempty
✓ bash [0.00s]
nonempty
✗ dash [0.01s]
dash: 1: [[: not found
FAILED - 1 shell (dash) reports failure, 2 (sh, bash) report success.
As you can see, Bash running as sh still accepted [[ ... ]], whereas dash, which is a (mostly) POSIX-features-only shell, failed, because POSIX only mandates [ ... ] conditionals (as an alias of test ... commands).
While I was playing around in my shell investigating the answer to this question, I noticed that, even though /bin/sh was pointing to /bin/bash on my system, the two commands behave differently. First of all, the output of
ls -lh /bin/sh
is:
lrwxrwxrwx 1 root root 4 Apr 22 2013 /bin/sh -> bash*
However, invoking the following command through /bin/sh:
/bin/sh -c "script.sh 2> >( grep -v FILTER 2>&1 )"
returns this error:
/bin/sh: -c: line 0: syntax error near unexpected token '>'
/bin/sh: -c: line 0: 'script.sh 2> >( grep -v FILTER 2>&1 )'
While running the same command through /bin/bash:
/bin/bash -c "script.sh 2> >( grep -v FILTER 2>&1 )"
executes successfully, here is the output:
This should be on stderr
For reference, here is the contents of script.sh:
#!/bin/sh
echo "FILTER: This should be filtered out" 1>&2
echo "This should be on stderr" 1>&2
echo "FILTER: This should be filtered out" 1>&2
Why do the two invocations behave differently?
bash looks at the value of $argv[0] (bash is implemented in C) to determine how it was invoked.
Its behavior when invoked as sh is documented in the manual:
If Bash is invoked with the name sh, it tries to mimic the startup
behavior of historical versions of sh as closely as possible, while
conforming to the POSIX standard as well.
When invoked as an interactive login shell, or as a non-interactive
shell with the -login option, it first attempts to read and execute
commands from /etc/profile and ~/.profile, in that order. The
--noprofile option may be used to inhibit this behavior. When invoked as an interactive shell with the name sh, Bash looks for the variable
ENV, expands its value if it is defined, and uses the expanded value
as the name of a file to read and execute. Since a shell invoked as sh
does not attempt to read and execute commands from any other startup
files, the --rcfile option has no effect. A non-interactive shell
invoked with the name sh does not attempt to read any other startup
files.
When invoked as sh, Bash enters POSIX mode after the startup files are
read
There's a long list (currently 46 items) of things that change when bash is in POSIX mode, documented here.
(POSIX mode is probably useful mostly as a way to test scripts for portability to non-bash shells.)
Incidentally, programs that change their behavior depending on the name under which they were invoked are fairly common. Some versions of grep, fgrep, and egrep are implemented as a single executable (though GNU grep doesn't do this). view is typically a symbolic link to vi or vim; invoking it as view causes to open in read-only mode. The Busybox system includes a number of individual commands that are all symlinks to the master busybox executable.
Invoking bash as sh causes it to enter posix mode after reading the startup files it would normally read (as opposed to the startup files a POSIX sh would read.) Bash has many different invocation modes. You can find out about these modes from the INVOCATION section of the manual. Here is some detail about the POSIX mode.
POSIX mode
This mode means bash will try, in various degrees, to conform to POSIX expectations. As explained here, bash has a few different invocations for this mode, with slightly different implications:
sh: Bash enters POSIX mode after reading startup files.
bash --posix: Bash enters POSIX mode before reading startup files.
set -o posix: Bash switches to POSIX mode.
POSIXLY_CORRECT: If this variable is in the environment when bash starts, the shell enters posix mode before reading the startup files, like bash --posix. If it is set while bash is running, like set -o posix.
From the Bash Reference Manual:
If Bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.
Because the bash binary checks how it was invoked (via argv[0]) and enters a compatibility mode if it's being run as sh.
I have just started using Linux and I am curious how shell built-in commands such as cd are defined.
Also, I'd appreciate if someone could explain how they are implemented and executed.
If you want to see how bash builtins are defined then you just need to look at Section 4 of The Bash Man Page.
If, however, you want to know how bash bultins are implemented, you'll need to look at the Bash source code because these commands are compiled into the bash executable.
One fast and easy way to see whether or not a command is a bash builtin is to use the help command. Example, help cd will show you how the bash builtin of 'cd' is defined. Similarly for help echo.
The actual set of built-ins varies from shell to shell. There are:
Special built-in utilities, which must be built-in, because they have some special properties
Regular built-in utilities, which are almost always built-in, because of the performance or other considerations
Any standard utility can be also built-in if a shell implementer wishes.
You can find out whether the utility is built in using the type command, which is supported by most shells (although its output is not standardized). An example from dash:
$ type ls
ls is /bin/ls
$ type cd
cd is a shell builtin
$ type exit
exit is a special shell builtin
Re cd utility, theoretically there's nothing preventing a shell implementer to implement it as external command. cd cannot change the shell's current directory directly, but, for instance, cd could communicate new directory to the shell process via a socket. But nobody does so because there's no point. Except very old shells (where there was not a notion of built-ins), where cd used some dirty system hack to do its job.
How is cd implemented inside the shell? The basic algorithm is described here. It can also do some work to support shell's extra features.
Manjari,
Check the source code of bash shell from ftp://ftp.gnu.org/gnu/bash/bash-2.05b.tar.gz
You will find that the definition of shell built-in commands in not in a separate binary executable but its within the shell binary itself (the name shell built-in clearly suggests this).
Every Unix shell has at least some builtin commands. These builtin commands are part of the shell, and are implemented as part of the shell's source code. The shell recognizes that the command that it was asked to execute was one of its builtins, and it performs that action on its own, without calling out to a separate executable. Different shells have different builtins, though there will be a whole lot of overlap in the basic set.
Sometimes, builtins are builtin for performance reasons. In this case, there's often also a version of that command in $PATH (possibly with a different feature set, different set of recognized command line arguments, etc), but the shell decided to implement the command as a builtin as well so that it could save the work of spawning off a short-lived process to do some work that it could do itself. That's the case for bash and printf, for example:
$ type printf
printf is a shell builtin
$ which printf
/usr/bin/printf
$ printf
printf: usage: printf [-v var] format [arguments]
$ /usr/bin/printf
/usr/bin/printf: missing operand
Try `/usr/bin/printf --help' for more information.
Note that in the above example, printf is both a shell builtin (implemented as part of bash itself), as well as an external command (located at /usr/bin/printf). Note that they behave differently as well - when called with no arguments, the builtin version and the command version print different error messages. Note also the -v var option (store the results of this printf into a shell variable named var) can only be done as part of the shell - subprocesses like /usr/bin/printf have no access to the variables of the shell that executed them.
And that brings us to the 2nd part of the story: some commands are builtin because they need to be. Some commands, like chmod, are thin wrappers around system calls. When you run /bin/chmod 777 foo, the shell forks, execs /bin/chmod (passing "777" and "foo") as arguments, and the new chmod process runs the C code chmod("foo", 777); and then returns control to the shell. This wouldn't work for the cd command, though. Even though cd looks like the same case as chmod, it has to behave differently: if the shell spawned another process to execute the chdir system call, it would change the directory only for that newly spawned process, not the shell. Then, when the process returned, the shell would be left sitting in the same directory as it had been in all along - therefore cd needs to be implemented as a shell builtin.
A Shell builtin -- http://linux.about.com/library/cmd/blcmdl1_builtin.htm
for eg. -
which cd
/usr/bin/which: no cd in (/usr/bin:/usr/local/bin......
Not a shell builtin but a binary.
which ls
/bin/ls
http://ss64.com/bash/ this will help you.
and here is shell scripting guide
http://www.freeos.com/guides/lsst/