Arithmetic expression in redirection - linux

What is the difference between these two:
cnt=1
head -n $((++cnt)) /etc/passwd >/dev/null
echo $cnt # prints 2
and
cnt=1
date >$((++cnt)).txt # creates file "2.txt"
echo $cnt # prints 1
My question is why in the second example 1 is printed.
Note:
cnt=1
(cnt=5)
echo $cnt # prints 1
I know why this will print 1. Is the redirection executed in a subshell too? If yes, where is that described?

I don't have a concrete citation for why this behavior exists, but going off the notes in SC2257* there are some interesting points to note in the manual.
When a simple command other than a builtin or shell function is to be executed, it is invoked in a separate execution environment
§3.7.3 Command Execution Environment
This reflects what SC2257 notes, though it's unclear about which environment the redirection's value is evaluated in. However §3.1.1 Shell Operation seems to say that redirection happens before this execution (sub)environment is invoked:
Basically, the shell does the following:
...
Performs the various shell expansions....
Performs any necessary redirections and removes the redirection operators and their operands from the argument list.
Executes the command.
We can see that this isn't limited to arithmetic expansions but also other state-changing expansions like :=:
$ bash -c 'date >"${word:=wow}.txt"; echo "word=${word}"'
word=
$ bash -c 'echo >"${word:=wow}.txt"; echo "word=${word}"'
word=wow
Interestingly, this does not appear to be a (well-defined) subshell environment, because BASH_SUBSHELL remains set to 0:
$ date >"${word:=$BASH_SUBSHELL}.txt"; ls
0.txt
We can also check some other shells, and see that zsh has the same behavior, though dash does not:
$ zsh -c 'date >"${word:=wow}.txt"; echo "word=${word}"'
word=
$ zsh -c 'echo >"${word:=wow}.txt"; echo "word=${word}"'
word=wow
$ dash -c 'date >"${word:=wow}.txt"; echo "word=${word}"'
word=wow
$ dash -c 'echo >"${word:=wow}.txt"; echo "word=${word}"'
word=wow
I skimmed the zsh guide but didn't find an exact mention of this behavior there either.
Needless to say, this does not appear to be well-documented behavior, so it's fortunate that ShellCheck can help catch it. It does however appear to be long-standing behavior, it's reproducible in Bash 3, 4, and 5.
* Unfortunately the commit that added SC2257 doesn't link to an Issue or any other further context.

Shellcheck's advice is sound; sometimes redirections are performed in subshells. However, the crux of this behavior is when expansions occur:
bind_int_variable variables.c:3410 cnt = 2, late binding
expr_bind_variable expr.c:336
exp0 expr.c:1040
exp1 expr.c:1007
exppower expr.c:962
expmuldiv expr.c:887
exp3 expr.c:861
expshift expr.c:837
exp4 expr.c:807
exp5 expr.c:785
expband expr.c:767
expbxor expr.c:748
expbor expr.c:729
expland expr.c:702
explor expr.c:674
expcond expr.c:627
expassign expr.c:512
expcomma expr.c:492
subexpr expr.c:474
evalexp expr.c:439
param_expand subst.c:9498 parameter expansion, including arith subst
expand_word_internal subst.c:9990
shell_expand_word_list subst.c:11335
expand_word_list_internal subst.c:11459
expand_words_no_vars subst.c:10988
redirection_expand redir.c:287 expansions post-fork()
do_redirection_internal redir.c:844
do_redirections redir.c:230 redirections are done in child process
execute_disk_command execute_cmd.c:5418 fork to run date(1)
execute_simple_command execute_cmd.c:4547
execute_command_internal execute_cmd.c:842
execute_command execute_cmd.c:394
reader_loop eval.c:175
main shell.c:805
When execute_disk_command() is called, it forks and then executes date(1). After the fork() and before the execve(), redirections and additional expansions are done (via do_redirections()). Variables expanded and bound post-fork will not reflect in the parent shell.
From BASH's perspective, however, this is just a simple command rather than a subshell command. This is an implicit subshell.
See execute_disk_command() in execute_cmd.c
Execute a simple command that is hopefully defined in a disk file
somewhere.
1) fork ()
2) connect pipes
3) look up the command
4) do redirections
5) execve ()
6) If the execve failed, see if the file has executable mode set.
If so, and it isn't a directory, then execute its contents as
a shell script.
(references taken from commit 9e49d343e3cd7e20dad1b86ebfb764e8027596a7 [browse tree])

try this
let cnt=1
echo `date` > "$((++cnt))".txt
echo $cnt
(editing my answer)
#alaniwi pointed out that strace clearly shows a clone. And I agree with P.P that it does not indicate a subshell being there and having scoured the documentation I did not find a direct reference for it.
EDIT:command substitution, process substitutions forks a subshell. I do want to quote the following from https://www.gnu.org/software/bash/manual/bash.html#Command-Execution-Environment
Command substitution, commands grouped with parentheses, and asynchronous commands are invoked in a subshell environment ...
This may hold some more answers :
https://www.gnu.org/software/bash/manual/bash.html (search for command substitution)
https://www.tldp.org/LDP/abs/html/subshells.html
https://tldp.org/LDP/abs/html/process-sub.html

Related

Recover after "kill 0"

I have a script that invokes kill 0. I want to invoke that script from another script, and have the outer script continue to execute. (kill 0 sends a signal, defaulting to SIGTERM, to every process in the process group of the calling process; see man 2 kill.)
kill0.sh:
#!/bin/sh
kill 0
caller.sh:
#!/bin/sh
echo BEFORE
./kill0.sh
echo AFTER
The current behavior is:
$ ./caller.sh
BEFORE
Terminated
$
How can I modify caller.sh so it prints AFTER after invoking kill0.sh?
Modifying kill0.sh is not an option. Assume that kill0.sh might read from stdin and write to stdout and/or stderr before invoking kill 0, and I don't want to interfere with that. I still want the kill 0 command to kill the kill0.sh process itself; I just don't want it to kill the caller as well.
I'm using Ubuntu 16.10 x86_64, and /bin/sh is a symlink to dash. That shouldn't matter, and I prefer answers that don't depend on that.
This is of course a simplified version of a larger set of scripts, so I'm at some risk of having an XY problem, but I think that a solution to the problem as stated here should let me solve the actual problem. (I have a wrapper script that invokes a specified command, capturing and displaying its output, with some other bells and whistles.)
One solution
You need to trap the signal in the parent, but enable it in the child. So a script like run-kill0.sh could be:
#!/bin/sh
echo BEFORE
trap '' TERM
(trap 15; exec ./kill0.sh)
echo AFTER
The first trap disables the TERM signal. The second trap in the sub-shell re-enables the signal (using the signal number instead of the name — see below) before running the kill0.sh script. Using exec is a minor optimization — you can omit it and it will work the same.
Digression on obscure syntactic details
Why 15 instead of TERM in the sub-shell? Because when I tested it with TERM instead of 15, I got:
$ sh -x run-kill0.sh
+ echo BEFORE
BEFORE
+ trap '' TERM
+ trap TERM
trap: usage: trap [-lp] [arg signal_spec ...]
+ echo AFTER
AFTER
$
When I used 15 in place of TERM (twice), I got:
$ sh -x run-kill0.sh
+ echo BEFORE
BEFORE
+ trap '' 15
+ trap 15
+ exec ./kill0.sh
Terminated: 15
+ echo AFTER
AFTER
$
Using TERM in place of the first 15 would also work.
Bash documentation on trap
Studying the Bash manual for trap shows:
trap [-lp] [arg] [sigspec …]
The commands in arg are to be read and executed when the shell receives signal sigspec. If arg is absent (and there is a single sigspec) or equal to ‘-’, each specified signal’s disposition is reset to the value it had when the shell was started.
A second solution
The second sentence is the key: trap - TERM should (and empirically does) work.
#!/bin/sh
echo BEFORE
trap '' TERM
(trap - TERM; exec ./kill0.sh)
echo AFTER
Running that yields:
$ sh -x run-kill0.sh
+ echo BEFORE
BEFORE
+ trap '' TERM
+ trap - TERM
+ exec ./kill0.sh
Terminated: 15
+ echo AFTER
AFTER
$
I've just re-remembered why I use numbers and not names (but my excuse is that the shell — it wasn't Bash in those days — didn't recognize signal names when I learned it).
POSIX documentation for trap
However, in Bash's defense, the POSIX spec for trap says:
If the first operand is an unsigned decimal integer, the shell shall treat all operands as conditions, and shall reset each condition to the default value. Otherwise, if there are operands, the first is treated as an action and the remaining as conditions.
If action is '-', the shell shall reset each condition to the default value. If action is null ( "" ), the shell shall ignore each specified condition if it arises.
This is clearer than the Bash documentation, IMO. It states why trap 15 works. There's also a minor glitch in the presentation. The synopsis says (on one line):
trap n [condition...]trap [action condition...]
It should say (on two lines):
trapn[condition...]
trap [action condition...]

'less' the file specified by the output of 'which'

command 'which' shows the link to a command.
command 'less' open the file.
How can I 'less' the file as the output of 'which'?
I don't want to use two commands like below to do it.
=>which script
/file/to/script/fiel
=>less /file/to/script/fiel
This is a use case for command substitution:
less -- "$(which commandname)"
That said, if your shell is bash, consider using type -P instead, which (unlike the external command which) is built into the shell:
less -- "$(type -P commandname)"
Note the quotes: These are important for reliable operation. Without them, the command may not work correctly if the filename contains characters inside IFS (by default, whitespace) or can be evaluated as a glob expression.
The double dashes are likewise there for correctness: Any argument after them is treated as positional (as per POSIX Utility Syntax Guidelines), so even if a filename starting with a dash were to be returned (however unlikely this may be), it ensures that less treats that as a filename rather than as the beginning of a sequence of options or flags.
You may also wish to consider honoring the user's pager selection via the environment variable $PAGER, and using type without -P to look for aliases, shell functions and builtins:
cmdsource() {
local sourcefile
if sourcefile="$(type -P -- "$1")"; then
"${PAGER:-less}" -- "$sourcefile"
else
echo "Unable to find source for $1" >&2
echo "...checking for a shell builtin:" >&2
type -- "$1"
fi
}
This defines a function you can run:
cmdsource commandname
You should be able to just pipe it over, try this:
which script | less

Internal Variable PIPESTATUS

I am new to linux and bash scripting and i have query about this internal variable PIPESTATUS which is an array and stores the exit status of individual commands in pipe.
On command line:
$ find /home | /bin/pax -dwx ustar | /bin/gzip -c > myfile.tar.gz
$ echo ${PIPESTATUS[*]}
$ 0 0 0
working fine on command line but when I am putting this code in a bash script it is showing only one exit status. My default SHELL on command line is bash only.
Somebody please help me to understand why this behaviour is changing? And what should I do to get this work in script?
#!/bin/bash
cmdfile=/var/tmp/cmd$$
backfile=/var/tmp/backup$$
find_fun() {
find /home
}
cmd1="find_fun | /bin/pax -dwx ustar"
cmd2="/bin/gzip -c"
eval "$cmd1 | $cmd2 > $backfile.tar.gz " 2>/dev/null
echo -e " find ${PIPESTATUS[0]} \npax ${PIPESTATUS[1]} \ncompress ${PIPESTATUS[2]} > $cmdfile
The problem you are having with your script is that you aren't running the same code as you ran on the command line. You are running different code. Namely the script has the addition of eval. If you were to wrap your command line test in eval you would see that it fails in a similar manner.
The reason the eval version fails (only gives you one value in PIPESTATUS) is because you aren't executing a pipeline anymore. You are executing eval on a string that contains a pipeline. This is similar to executing /bin/bash -c 'some | pipe | line'. The thing actually being run by the current shell is a single command so it has a single exit code.
You have two choices here:
Get rid of eval (which you should do anyway as eval is generally something to avoid) and stop using a string for a command (see Bash FAQ 050 for more on why doing this is a bad idea.
Move the echo "${PIPESTATUS[#]}" into the eval and then capture (and split/parse) the resulting output. (This is clearly a worse solution in just about every way.)
Instead of ${PIPESTATUS[0]} use ${PIPESTATUS[#]}
As with any array in bash PIPESTATUS[0] contains the first command exit status. If you want to get all of them you have to use PIPESTATUS[#] which returns all the contents of the array.
I'm not sure why it worked for you when you tried it in the command line. I tested it and I didn't get the same result as you.

Bash variable defaulting doesn't work if followed by pipe (bash bug?)

I've just discovered a strange behaviour in bash that I don't understand. The expression
${variable:=default}
sets variable to the value default if it isn't already set. Consider the following examples:
#!/bin/bash
file ${foo:=$1}
echo "foo >$foo<"
file ${bar:=$1} | cat
echo "bar >$bar<"
The output is:
$ ./test myfile.txt
myfile.txt: ASCII text
foo >myfile.txt<
myfile.txt: ASCII text
bar ><
You will notice that the variable foo is assigned the value of $1 but the variable bar is not, even though the result of its defaulting is presented to the file command.
If you remove the innocuous pipe into cat from line 4 and re-run it, then it both foo and bar get set to the value of $1
Am I missing somehting here, or is this potentially a bash bug?
(GNU bash, version 4.3.30)
In second case file is a pipe member and runs as every pipe member in its own shell. When file with its subshell ends, $b with its new value from $1 no longer exists.
Workaround:
#!/bin/bash
file ${foo:=$1}
echo "foo >$foo<"
: "${bar:=$1}" # Parameter Expansion before subshell
file $bar | cat
echo "bar >$bar<"
It's not a bug. Parameter expansion happens when the command is evaluated, not parsed, but a command that is part of a pipeline is not evaluated until the new process has been started. Changing this, aside from likely breaking some existing code, would require extra level of expansion before evaluation occurs.
A hypothetical bash session:
> foo=5
> bar='$foo'
> echo "$bar"
$foo
# $bar expands to '$foo' before the subshell is created, but then `$foo` expands to 5
# during the "normal" round of parameter expansion.
> echo "$bar" | cat
5
To avoid that, bash would need some way of marking pieces of text that result from the new first round of pre-evaluation parameter expansion, so that they do not undergo a second
round of evaluation. This type of bookkeeping would quickly lead to unmaintainable code as more corner cases are found to be handled. Far simpler is to just accept that parameter expansions will be deferred until after the subshell starts.
The other alternative is to allow each component to run in the current shell, something that is allowed by the POSIX standard, but is not required, either. bash made the choice long ago to execute each component in a subshell, and reversing that would break too much existing code that relies on the current behavior. (bash 4.2 did introduce the lastpipe option, allowing the last component of a pipeline to execute in the current shell if explicitly enabled.)

Unable to run BASH script in current environment multiple times

I have a bash script that I use to move from source to bin directories from anywhere I currently am (I call this script, 'teleport'). Since it basically is just a glorified 'cd' command, I have to run it in the current shell (i.e. . ./teleport.sh ). I've set up an alias in my .bashrc file so that 'teleport' matches '. teleport.sh'.
The first time I run it, it works fine. But then, if I run it again after it has run once, it doesn't do anything. It works again if I close my terminal and then open a new one, but only the first time. My intuition is that there is something internally going on with BASH that I'm not familiar with, so I thought I would run it through the gurus here to see if I can get an answer.
The script is:
numargs=$#
function printUsage
{
echo -e "Usage: $0 [-o | -s] <PROJECT>\n"
echo -e "\tMagically teleports you into the main source directory of a project.\n"
echo -e "\t PROJECT: The current project you wish to teleport into."
echo -e "\t -o: Teleport into the objdir.\n"
echo -e "\t -s: Teleport into the source dir.\n"
}
if [ $numargs -lt 2 ]
then
printUsage
fi
function teleportToObj
{
OBJDIR=${HOME}/Source/${PROJECT}/obj
cd ${OBJDIR}
}
function teleportToSrc
{
cd ${HOME}/Source/${PROJECT}/src
}
while getopts "o:s:" opt
do
case $opt in
o)
PROJECT=$OPTARG
teleportToObj
;;
s)
PROJECT=$OPTARG
teleportToSrc
;;
esac
done
My usage of it is something like:
sjohnson#corellia:~$ cd /usr/local/src
sjohnson#corellia:/usr/local/src$ . ./teleport -s some-proj
sjohnson#corellia:~/Source/some-proj/src$ teleport -o some-proj
sjohnson#corellia:~/Source/some-proj/src$
<... START NEW TERMINAL ...>
sjohnson#corellia:~$ . ./teleport -o some-proj
sjohnson#corellia:~/Source/some-proj/obj$
The problem is that getopts necessarily keeps a little bit of state so that it can be called in a loop, and you're not clearing that state. Each time it's called, it processes one more argument, and it increments the shell's OPTIND variable so it'll know which argument to process the next time it's called. When it's done with all the arguments, it returns 1 (false) every time it's invoked, which makes the while exit.
The first time you source your script, it works as expected. The second (and third, fourth...) time, getopts does nothing but return false.
Add one line to reset the state before you start looping:
unset OPTIND # clear state so getopts will start over
while getopts "o:s:" opt
do
# ...
done
(I assume there's a typo in your transcript, since it shows you invoking the script -- not sourcing it -- on the second try, but that's not the real problem here.)
The problem is that the first time you call is you are sourcing the script (thats what ". ./teleport") does which runs the script in the current shell thus preserving the cd. The second time you call it, it isn't sourced so you create a subshell, cd to the appropriate directory, and then exit the subshell putting you right back where you called the script from!
The way to make this work is simply to make teleportToSrc and teleportToObj aliases or functions in the current shell (i.e. outside a script)

Resources