I am executing a ksh script from another ksh script. The called script ends by executing ksh which simply stops the caller script from continuing.
MRE
#!/bin/ksh
# Caller Script
. ~/called
# Does not continue to next echo.
echo "DONE!"
#!/bin/ksh
#Called script
# Some exports..
ENV=calledcalled ksh
Output with set -x
++ ksh
++ ENV=calledcalled
.kshrc executed
If I run calledcalled directly in my caller it works fine (i.e. continues with next commands. Why does this happen? I checked $? and it is 0. I tried ./called || true. Please let me know if more information is needed.
Note: Called script is outside my control.
This is completely normal and expected. Remember, when you run cmd1; cmd2, cmd2 doesn't run until cmd1 exits.
When your script runs ksh (and is invoked from a terminal or other context where reading from stdin doesn't cause an immediate EOF), nothing is making that new copy of ksh exit -- it waits for code to run to be given to it on stdin as normal -- so that script is just sitting around waiting for the copy of ksh to exit before it does anything else.
There are plenty of ways you can work around this. A few easy ones:
Ensure that stdin is empty so the child interpreter can't wait for input
. ~/called </dev/null
Define a function named ksh that doesn't do anything at all.
ksh() { echo "Not actually running ksh" >&2; }
. ~/called
Set ENV (a variable which, when defined, tells any shell to run the code in that file before doing anything else) to the filename of a script that, when run, causes any interactive shell to exit immediately.
exit_script=$(mktemp -t exit_script.XXXXXX)
printf '%s\n' 'case $- in *i*) exit 0;; esac' >"$exit_script"
ENV=$exit_script . ~/called
rm -f -- "$exit_script"
The above are just a few approaches; you can surely imagine many more with just a little thought and experimentation.
Related
I'm trying to run multiple subshells in a bash script and capture the stdout result to a variable. When I run the subshell in the background I would expect I can use wait to let the subshell complete and then use the variable the result is assigned to later in the program.... but it doesn't seem to work.
Simple example script:
l=$(ls) &
wait $!
echo "L=$l"
Then when I run it:
$ bash -x test2.sh
+ wait 16821
++ ls
+ l='test1.sh test2.sh'
+ echo L=
L=
The output from my test program would suggest the variable l should be assigned the result of the subshell, but when I use echo it is empty...
If I don't background the subshell (or use wait) then it works as expected...
l=$(ls)
echo "L=$l"
Results in:
$ bash -x test1.sh
++ ls
+ l='test1.sh test2.sh'
+ echo 'L=test1.sh test2.sh'
L=test1.sh test2.sh
Am I missing something obvious or ... ?
From bash manpage (emphasis mine):
Command substitution, commands grouped with parentheses, and
asynchronous commands are invoked in a subshell environment that is
a duplicate of the shell environment, except that traps caught by the
shell are re‐set to the values that the shell inherited from its
parent at invocation. Builtin commands that are invoked as part of a
pipeline are also executed in a subshell environment. Changes made to the
subshell environment cannot affect the shell's execution environment.
So, l=$(ls) & would be like (l=$(ls)) if not backgrounded.
Usually on unix systems you can suppress command output by redirecting STDIN and/or STDERR to a file or /dev/null. But what, if you need to pass content to a piped command via STDIN in a bash script?
The example below should make clear what is meant. It's just an example, though - I'm not searching for a solution to this command in specific but to that kind of situation in general. Sadly there are numerous situations where you would want to suppress output in a script, but need to pass content via STDIN, when a command has no switch to submit the information in an other way.
My "problem" is that I wrote a function to execute commands with proper error handling and in which I would like to redirect all output produced by the executed commands to a log file.
Example problem:
[18:25:35] [V] root#vbox:~# echo 'test' |read -p 'Test Output' TMP &>/dev/null
[18:25:36] [V] root#vbox:~# echo $TMP
[18:25:36] [V] root#vbox:~#
Any ideas on how to solve my problem?
What user000001 is saying is that all commands in a bash pipeline are executed in subshells. So, when the subshell handling the read command exits, the $TMP variable disappears too. You have to account for this and either:
avoid subshells (examples given in comment above)
do all your work with variables in the same subshell
echo test | { read value; echo subshell $value; }; echo parent $value
use a different shell
$ ksh -c 'echo test | { read value; echo subshell $value; }; echo parent $value'
subshell test
parent test
I wrote a lot of bash scripts that should work with the current bash session, because I often used fg, jobs, etc.
I always starts my scripts with . script.sh but one of my friends startet it with ./script.sh and got error that fg "couldn't be executed".
Is it possible to force a . script.sh or anything else what I can do to prevent errors? Such as cancel the whole script and print an error with echo or something else.
Edit:
I think bashtraps have problems when executing sourced, is there any way to use fg, jobs and bashtraps in one script?
Looks like you're trying to determine if a script is being run interactively or not. The bash manual says that you can determine this with the following test:
#! /bin/bash
case "$-" in
*i*) echo interactive ;;
*) echo non-interactive ;;
esac
sleep 2 &
fg
If you run this with ./foo.sh, you'll see "non-interactive" printed and an error for the fg built-in. If you source it with . foo.sh or source foo.sh you won't get that error (assuming you're running those from an interactive shell, obviously).
For your use-case, you can exit with an error message in the non-interactive mode.
If job control is all you need, you can make it work both ways with #!/bin/bash -i:
#!/bin/bash -i
sleep 1 &
fg
This script works the same whether you . myscript or ./myscript.
PS: You should really adopt your friend's way of executing scripts. It's more robust and most people write their scripts to work that way (e.g. assuming exit will just exit the script).
There are a couple of simple tricks to remind people to use source (or .) to run your script: First, remove execute permission from it (chmod -x script.sh), so running it with ./script.sh will give a permission error. Second, replace the normal shebang (first line) with something like this:
#!/bin/echo please run this with the command: source
This will make the script print something like "please run this with the command: source ./script.sh" (and not run the actual script) if someone does manage to execute it.
Note that neither of these helps if someone runs the script with bash script.sh.
I have a shell script which writes all output to logfile
and terminal, this part works fine, but if I execute the script
a new shell prompt only appear if I press enter. Why is that and how do I fix it?
#!/bin/bash
exec > >(tee logfile)
echo "output"
First, when I'm testing this, there always is a new shell prompt, it's just that sometimes the string output comes after it, so the prompt isn't last. Did you happen to overlook it? If so, there seems to be a race where the shell prints the prompt before the tee in the background completes.
Unfortunately, that cannot fixed by waiting in the shell for tee, see this question on unix.stackexchange. Fragile workarounds aside, the easiest way to solve this that I see is to put your whole script inside a list:
{
your-code-here
} | tee logfile
If I run the following script (suppressing the newline from the echo), I see the prompt, but not "output". The string is still written to the file.
#!/bin/bash
exec > >(tee logfile)
echo -n "output"
What I suspect is this: you have three different file descriptors trying to write to the same file (that is, the terminal): standard output of the shell, standard error of the shell, and the standard output of tee. The shell writes synchronously: first the echo to standard output, then the prompt to standard error, so the terminal is able to sequence them correctly. However, the third file descriptor is written to asynchronously by tee, so there is a race condition. I don't quite understand how my modification affects the race, but it appears to upset some balance, allowing the prompt to be written at a different time and appear on the screen. (I expect output buffering to play a part in this).
You might also try running your script after running the script command, which will log everything written to the terminal; if you wade through all the control characters in the file, you may notice the prompt in the file just prior to the output written by tee. In support of my race condition theory, I'll note that after running the script a few times, it was no longer displaying "abnormal" behavior; my shell prompt was displayed as expected after the string "output", so there is definitely some non-deterministic element to this situation.
#chepner's answer provides great background information.
Here's a workaround - works on Ubuntu 12.04 (Linux 3.2.0) and on OS X 10.9.1:
#!/bin/bash
exec > >(tee logfile)
echo "output"
# WORKAROUND - place LAST in your script.
# Execute an executable (as opposed to a builtin) that outputs *something*
# to make the prompt reappear normally.
# In this case we use the printf *executable* to output an *empty string*.
# Use of `$ec` is to ensure that the script's actual exit code is passed through.
ec=$?; $(which printf) ''; exit $ec
Alternatives:
#user2719058's answer shows a simple alternative: wrapping the entire script body in a group command ({ ... }) and piping it to tee logfile.
An external solution, as #chepner has already hinted at, is to use the script utility to create a "transcript" of your script's output in addition to displaying it:
script -qc yourScript /dev/null > logfile # Linux syntax
This, however, will also capture stderr output; if you wanted to avoid that, use:
script -qc 'yourScript 2>/dev/null' /dev/null > logfile
Note, however, that this will suppress stderr output altogether.
As others have noted, it's not that there's no prompt printed -- it's that the last of the output written by tee can come after the prompt, making the prompt no longer visible.
If you have bash 4.4 or newer, you can wait for your tee process to exit, like so:
#!/usr/bin/env bash
case $BASH_VERSION in ''|[0-3].*|4.[0-3]) echo "ERROR: Bash 4.4+ needed" >&2; exit 1;; esac
exec {orig_stdout}>&1 {orig_stderr}>&2 # make a backup of original stdout
exec > >(tee -a "_install_log"); tee_pid=$! # track PID of tee after starting it
cleanup() { # define a function we'll call during shutdown
retval=$?
exec >&$orig_stdout # Copy your original stdout back to FD 1, overwriting the pipe to tee
exec 2>&$orig_stderr # If something overwrites stderr to also go through tee, fix that too
wait "$tee_pid" # Now, wait until tee exits
exit "$retval" # and complete exit with our original exit status
}
trap cleanup EXIT # configure the function above to be called during cleanup
echo "Writing something to stdout here"
From Bash Reference Manual I get the following about exec bash builtin command:
If command is supplied, it replaces the shell without creating a new process.
Now I have the following bash script:
#!/bin/bash
exec ls;
echo 123;
exit 0
This executed, I got this:
cleanup.sh ex1.bash file.bash file.bash~ output.log
(files from the current directory)
Now, if I have this script:
#!/bin/bash
exec ls | cat
echo 123
exit 0
I get the following output:
cleanup.sh
ex1.bash
file.bash
file.bash~
output.log
123
My question is:
If when exec is invoked it replaces the shell without creating a new process, why when put | cat, the echo 123 is printed, but without it, it isn't. So, I would be happy if someone can explain what's the logic of this behavior.
Thanks.
EDIT:
After #torek response, I get an even harder to explain behavior:
1.exec ls>out command creates the out file and put in it the ls's command result;
2.exec ls>out1 ls>out2 creates only the files, but do not put inside any result. If the command works as suggested, I think the command number 2 should have the same result as command number 1 (even more, I think it should not have had created the out2 file).
In this particular case, you have the exec in a pipeline. In order to execute a series of pipeline commands, the shell must initially fork, making a sub-shell. (Specifically it has to create the pipe, then fork, so that everything run "on the left" of the pipe can have its output sent to whatever is "on the right" of the pipe.)
To see that this is in fact what is happening, compare:
{ ls; echo this too; } | cat
with:
{ exec ls; echo this too; } | cat
The former runs ls without leaving the sub-shell, so that this sub-shell is therefore still around to run the echo. The latter runs ls by leaving the sub-shell, which is therefore no longer there to do the echo, and this too is not printed.
(The use of curly-braces { cmd1; cmd2; } normally suppresses the sub-shell fork action that you get with parentheses (cmd1; cmd2), but in the case of a pipe, the fork is "forced", as it were.)
Redirection of the current shell happens only if there is "nothing to run", as it were, after the word exec. Thus, e.g., exec >stdout 4<input 5>>append modifies the current shell, but exec foo >stdout 4<input 5>>append tries to exec command foo. [Note: this is not strictly accurate; see addendum.]
Interestingly, in an interactive shell, after exec foo >output fails because there is no command foo, the shell sticks around, but stdout remains redirected to file output. (You can recover with exec >/dev/tty. In a script, the failure to exec foo terminates the script.)
With a tip of the hat to #Pumbaa80, here's something even more illustrative:
#! /bin/bash
shopt -s execfail
exec ls | cat -E
echo this goes to stdout
echo this goes to stderr 1>&2
(note: cat -E is simplified down from my usual cat -vET, which is my handy go-to for "let me see non-printing characters in a recognizable way"). When this script is run, the output from ls has cat -E applied (on Linux this makes end-of-line visible as a $ sign), but the output sent to stdout and stderr (on the remaining two lines) is not redirected. Change the | cat -E to > out and, after the script runs, observe the contents of file out: the final two echos are not in there.
Now change the ls to foo (or some other command that will not be found) and run the script again. This time the output is:
$ ./demo.sh
./demo.sh: line 3: exec: foo: not found
this goes to stderr
and the file out now has the contents produced by the first echo line.
This makes what exec "really does" as obvious as possible (but no more obvious, as Albert Einstein did not put it :-) ).
Normally, when the shell goes to execute a "simple command" (see the manual page for the precise definition, but this specifically excludes the commands in a "pipeline"), it prepares any I/O redirection operations specified with <, >, and so on by opening the files needed. Then the shell invokes fork (or some equivalent but more-efficient variant like vfork or clone depending on underlying OS, configuration, etc), and, in the child process, rearranges the open file descriptors (using dup2 calls or equivalent) to achieve the desired final arrangements: > out moves the open descriptor to fd 1—stdout—while 6> out moves the open descriptor to fd 6.
If you specify the exec keyword, though, the shell suppresses the fork step. It does all the file opening and file-descriptor-rearranging as usual, but this time, it affects any and all subsequent commands. Finally, having done all the redirections, the shell attempts to execve() (in the system-call sense) the command, if there is one. If there is no command, or if the execve() call fails and the shell is supposed to continue running (is interactive or you have set execfail), the shell soldiers on. If the execve() succeeds, the shell no longer exists, having been replaced by the new command. If execfail is unset and the shell is not interactive, the shell exits.
(There's also the added complication of the command_not_found_handle shell function: bash's exec seems to suppress running it, based on test results. The exec keyword in general makes the shell not look at its own functions, i.e., if you have a shell function f, running f as a simple command runs the shell function, as does (f) which runs it in a sub-shell, but running (exec f) skips over it.)
As for why ls>out1 ls>out2 creates two files (with or without an exec), this is simple enough: the shell opens each redirection, and then uses dup2 to move the file descriptors. If you have two ordinary > redirects, the shell opens both, moves the first one to fd 1 (stdout), then moves the second one to fd 1 (stdout again), closing the first in the process. Finally, it runs ls ls, because that's what's left after removing the >out1 >out2. As long as there is no file named ls, the ls command complains to stderr, and writes nothing to stdout.