I have this shell script which I use to back up my system. There is a line:
tar -Pzcpf /backups/backup.tar.gz --directory=/ --exclude=proc --exclude=sys --exclude=dev/pts --exclude=backups --exclude=var/log / 2> >(grep -v 'socket ignored' >&2)
As you can see, I have been trying to filter out the annoying, useless "socket ignored" error by tar, using this blog post.
What I get from shell upon execution is:
/bin/sysback: line 45: syntax error near unexpected token >'
/bin/sysback: line 45:tar -Pzcpf /backups/backup --directory=/
--exclude=proc --exclude=sys --exclude=dev/pts --exclude=backups --exclude=var/log / 2> >(grep -v 'socket ignored' >&2)'
The syntax you've used is a bash extension to the basic shell syntax, so you must take care to run your script with bash. (Ksh also has >(…) process substitution but doesn't support it after a redirection. Zsh would be fine.)
Given the error message you're getting, you are running this script in bash, but in its POSIX compatibility mode, not in full bash mode. Take care to invoke your script with an explicit #!/bin/bash line. #!/bin/sh won't do, even if /bin/sh is a symbolic link to bash, because bash runs in POSIX mode if it's invoked under the name sh. Always invoke bash by name if you use bash features.
Also take care not to set the environment variable POSIXLY_CORRECT or to pass the --posix option on the command line if you want to use bash features.
Alternatively, don't use this bash-specific syntax; use a portable construct such as the one proposed by Stephane Rouberol.
How about:
tar -Pzcpf /backups/backup.tar.gz --directory=/ \
--exclude=proc --exclude=sys --exclude=dev/pts \
--exclude=backups --exclude=var/log 2>&1 | grep -v 'socket ignored'
I have found that on gentoo also if sh is a link to /bin/bash if you call the script with 'sh "scriptname"' it doesn't run it as a bash script and fail with:
matchmorethan.sh: line 34: syntax error near unexpected token `<'
matchmorethan.sh: line 34: `done < <( cat $matchfile )'
So if you need to use the Process Substitution feature you need to specifically run it with bash. But I didn't find any reference to this.
Actually you don't have to go for such a redirection in std error when GNU tar provides options to ignore the "socket ignored" warning.
tar --warning='no-file-ignored' -Pzcpf /backups/backup.tar.gz --directory=/ --exclude=proc --exclude=sys --exclude=dev/pts --exclude=backups --exclude=var/log / 2> new.err
You could find the original link with more ignore options here
Related
Is it possible to check a bash script syntax without executing it?
Using Perl, I can run perl -c 'script name'. Is there any equivalent command for bash scripts?
bash -n scriptname
Perhaps an obvious caveat: this validates syntax but won't check if your bash script tries to execute a command that isn't in your path, like ech hello instead of echo hello.
Time changes everything. Here is a web site which provide online syntax checking for shell script.
I found it is very powerful detecting common errors.
About ShellCheck
ShellCheck is a static analysis and linting tool for sh/bash scripts. It's mainly focused on handling typical beginner and intermediate level syntax errors and pitfalls where the shell just gives a cryptic error message or strange behavior, but it also reports on a few more advanced issues where corner cases can cause delayed failures.
Haskell source code is available on GitHub!
I also enable the 'u' option on every bash script I write in order to do some extra checking:
set -u
This will report the usage of uninitialized variables, like in the following script 'check_init.sh'
#!/bin/sh
set -u
message=hello
echo $mesage
Running the script :
$ check_init.sh
Will report the following :
./check_init.sh[4]: mesage: Parameter not set.
Very useful to catch typos
sh -n script-name
Run this. If there are any syntax errors in the script, then it returns the same error message.
If there are no errors, then it comes out without giving any message. You can check immediately by using echo $?, which will return 0 confirming successful without any mistake.
It worked for me well. I ran on Linux OS, Bash Shell.
I actually check all bash scripts in current dir for syntax errors WITHOUT running them using find tool:
Example:
find . -name '*.sh' -print0 | xargs -0 -P"$(nproc)" -I{} bash -n "{}"
If you want to use it for a single file, just edit the wildcard with the name of the file.
null command [colon] also useful when debugging to see variable's value
set -x
for i in {1..10}; do
let i=i+1
: i=$i
done
set -
For only validating syntax:
shellcheck [programPath]
For running the program only if syntax passes, so debugging both syntax and execution:
shellproof [programPath]
Bash shell scripts will run a syntax check if you enable syntax checking with
set -o noexec
if you want to turn off syntax checking
set +o noexec
There is BashSupport plugin for IntelliJ IDEA which checks the syntax.
If you need in a variable the validity of all the files in a directory (git pre-commit hook, build lint script), you can catch the stderr output of the "sh -n" or "bash -n" commands (see other answers) in a variable, and have a "if/else" based on that
bashErrLines=$(find bin/ -type f -name '*.sh' -exec sh -n {} \; 2>&1 > /dev/null)
if [ "$bashErrLines" != "" ]; then
# at least one sh file in the bin dir has a syntax error
echo $bashErrLines;
exit;
fi
Change "sh" with "bash" depending on your needs
I'm trying to run the command from this question:
exec 2> >(logger -t myscript)
It works great on my desktop linux system, however, on my embedded linux device the same command presents the following error:
-sh: syntax error near unexpected token `>'
So I'm guessing my shell doesn't like part of the command syntax - most likely this portion:
exec 2>>(logger -t myscript)
In fact, while I understand that the 2> is redirecting stderr I don't actually understand the syntax of the second > character in this case, is it another way of representing a pipe?
If I can understand what it is doing then perhaps I can modify my command to work with my limited shell on the embedded linux device.
The syntax in question only works with bash (or other shells with ksh extensions). In the error
-sh: syntax error near unexpected token `>'
...you're trying to use that syntax with /bin/sh.
Be sure your script starts with #!/bin/bash, and that you invoke it with bash yourscript rather than sh yourscript.
A bit more explanation:
>(foo) gets replaced with a filename (of the form /dev/fd/## if supported, or a named pipe otherwise) which receives output from a process named foo. This is the part that requires bash or ksh extensions.
exec <redirection> applies a redirection to the current shell process (thus, exec 2>stderr.log redirects all stderr from the current command and its children to the file stderr.log).
Thus, exec 2> >(foo) modifies the stderr file descriptor (of your current shell session) to go to the stdin of command foo; in this case, foo is logger -t myscript, thus sending the process's stderr to syslog.
To perform the same operation on a more limited (but still POSIX-compliant) shell:
# note: if any security concerns exist, put the FIFO in a directory
# created by mktemp -d rather than hardcoding its name
mkfifo /tmp/log.fifo # create the FIFO
logger -t myscript </tmp/log.fifo & # start the reader in the background first!
exec 2>/tmp/log.fifo # then start writing
rm -f /tmp/log.fifo # safe to delete at this point
>( command ) is a bash construct called "process substitution". man bash says:
Process substitution is supported on systems that support named pipes (FIFOs) or the /dev/fd method of naming open files.
Your shell doesn't seem to be bash, anyway.
While I was playing around in my shell investigating the answer to this question, I noticed that, even though /bin/sh was pointing to /bin/bash on my system, the two commands behave differently. First of all, the output of
ls -lh /bin/sh
is:
lrwxrwxrwx 1 root root 4 Apr 22 2013 /bin/sh -> bash*
However, invoking the following command through /bin/sh:
/bin/sh -c "script.sh 2> >( grep -v FILTER 2>&1 )"
returns this error:
/bin/sh: -c: line 0: syntax error near unexpected token '>'
/bin/sh: -c: line 0: 'script.sh 2> >( grep -v FILTER 2>&1 )'
While running the same command through /bin/bash:
/bin/bash -c "script.sh 2> >( grep -v FILTER 2>&1 )"
executes successfully, here is the output:
This should be on stderr
For reference, here is the contents of script.sh:
#!/bin/sh
echo "FILTER: This should be filtered out" 1>&2
echo "This should be on stderr" 1>&2
echo "FILTER: This should be filtered out" 1>&2
Why do the two invocations behave differently?
bash looks at the value of $argv[0] (bash is implemented in C) to determine how it was invoked.
Its behavior when invoked as sh is documented in the manual:
If Bash is invoked with the name sh, it tries to mimic the startup
behavior of historical versions of sh as closely as possible, while
conforming to the POSIX standard as well.
When invoked as an interactive login shell, or as a non-interactive
shell with the -login option, it first attempts to read and execute
commands from /etc/profile and ~/.profile, in that order. The
--noprofile option may be used to inhibit this behavior. When invoked as an interactive shell with the name sh, Bash looks for the variable
ENV, expands its value if it is defined, and uses the expanded value
as the name of a file to read and execute. Since a shell invoked as sh
does not attempt to read and execute commands from any other startup
files, the --rcfile option has no effect. A non-interactive shell
invoked with the name sh does not attempt to read any other startup
files.
When invoked as sh, Bash enters POSIX mode after the startup files are
read
There's a long list (currently 46 items) of things that change when bash is in POSIX mode, documented here.
(POSIX mode is probably useful mostly as a way to test scripts for portability to non-bash shells.)
Incidentally, programs that change their behavior depending on the name under which they were invoked are fairly common. Some versions of grep, fgrep, and egrep are implemented as a single executable (though GNU grep doesn't do this). view is typically a symbolic link to vi or vim; invoking it as view causes to open in read-only mode. The Busybox system includes a number of individual commands that are all symlinks to the master busybox executable.
Invoking bash as sh causes it to enter posix mode after reading the startup files it would normally read (as opposed to the startup files a POSIX sh would read.) Bash has many different invocation modes. You can find out about these modes from the INVOCATION section of the manual. Here is some detail about the POSIX mode.
POSIX mode
This mode means bash will try, in various degrees, to conform to POSIX expectations. As explained here, bash has a few different invocations for this mode, with slightly different implications:
sh: Bash enters POSIX mode after reading startup files.
bash --posix: Bash enters POSIX mode before reading startup files.
set -o posix: Bash switches to POSIX mode.
POSIXLY_CORRECT: If this variable is in the environment when bash starts, the shell enters posix mode before reading the startup files, like bash --posix. If it is set while bash is running, like set -o posix.
From the Bash Reference Manual:
If Bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.
Because the bash binary checks how it was invoked (via argv[0]) and enters a compatibility mode if it's being run as sh.
I'm writing a shell script called myShellScript.sh inside of which I have the following text:
echo *** Print out this line ****
diff <(./myProgram) <(./otherProgram)
However, when I run sh myShellScript.sh I get the following error:
-bash-4.2$ sh myShellScript.sh
myShellScript.sh **** Print out this line **** myShellScript.sh
myShellScript.sh: line 2: syntax error near unexpected token `('
myShellScript.sh: line 2: `diff <(./myProgram) <(./otherProgram)'
Process substitution with the <(...) operator is a bash feature. You're getting that error message because your script is getting executed as something else (for example dash), or an older version of bash, or bash running in POSIX-compatibility mode with non-POSIX features like process substitution disabled (thanks, #chepner!)
If you want to execute the script with a full-featured bash, you have two options:
Run the script with bash:
bash myShellScript.sh
Set the first line of the script to #!/bin/bash (or whatever is the path to bash in your system), and run the script like this:
./myShellScript.sh
You need to execute your script with bash, not with sh.
You are using process substitution, which is not a standard POSIX shell feature. sh is a POSIX-compatible shell, so it does not support language extensions like process substitution. Bash will run with POSIX compatibility enabled if it is invoked as sh.
Therefore, you should execute scripts that require Bash-specific features using bash.
You clearly seem to be using bash, but for anyone reading this that needs to use a shell without support for process substitution, you can use the following:
# Instead of diff <(./myProgram) <(./otherProgram)
# A pair of named pipes to avoid using disk space
# for the output of myProgram and otherProgram
mkfifo myProgram.output
mkfifo otherProgram.output
./myProgram > myProgram.output &
./otherProgram > otherProgram.output &
diff myProgram.output otherProgram.output
rm myProgram.output otherProgram.output
This is nearly identical to how bash might perform process substitution on some platforms.
Is it possible to check a bash script syntax without executing it?
Using Perl, I can run perl -c 'script name'. Is there any equivalent command for bash scripts?
bash -n scriptname
Perhaps an obvious caveat: this validates syntax but won't check if your bash script tries to execute a command that isn't in your path, like ech hello instead of echo hello.
Time changes everything. Here is a web site which provide online syntax checking for shell script.
I found it is very powerful detecting common errors.
About ShellCheck
ShellCheck is a static analysis and linting tool for sh/bash scripts. It's mainly focused on handling typical beginner and intermediate level syntax errors and pitfalls where the shell just gives a cryptic error message or strange behavior, but it also reports on a few more advanced issues where corner cases can cause delayed failures.
Haskell source code is available on GitHub!
I also enable the 'u' option on every bash script I write in order to do some extra checking:
set -u
This will report the usage of uninitialized variables, like in the following script 'check_init.sh'
#!/bin/sh
set -u
message=hello
echo $mesage
Running the script :
$ check_init.sh
Will report the following :
./check_init.sh[4]: mesage: Parameter not set.
Very useful to catch typos
sh -n script-name
Run this. If there are any syntax errors in the script, then it returns the same error message.
If there are no errors, then it comes out without giving any message. You can check immediately by using echo $?, which will return 0 confirming successful without any mistake.
It worked for me well. I ran on Linux OS, Bash Shell.
I actually check all bash scripts in current dir for syntax errors WITHOUT running them using find tool:
Example:
find . -name '*.sh' -print0 | xargs -0 -P"$(nproc)" -I{} bash -n "{}"
If you want to use it for a single file, just edit the wildcard with the name of the file.
null command [colon] also useful when debugging to see variable's value
set -x
for i in {1..10}; do
let i=i+1
: i=$i
done
set -
For only validating syntax:
shellcheck [programPath]
For running the program only if syntax passes, so debugging both syntax and execution:
shellproof [programPath]
Bash shell scripts will run a syntax check if you enable syntax checking with
set -o noexec
if you want to turn off syntax checking
set +o noexec
There is BashSupport plugin for IntelliJ IDEA which checks the syntax.
If you need in a variable the validity of all the files in a directory (git pre-commit hook, build lint script), you can catch the stderr output of the "sh -n" or "bash -n" commands (see other answers) in a variable, and have a "if/else" based on that
bashErrLines=$(find bin/ -type f -name '*.sh' -exec sh -n {} \; 2>&1 > /dev/null)
if [ "$bashErrLines" != "" ]; then
# at least one sh file in the bin dir has a syntax error
echo $bashErrLines;
exit;
fi
Change "sh" with "bash" depending on your needs