echo Command of Shell Script -e option writing on file [duplicate] - linux

I have a line of code that works fine in my terminal:
for i in *.mp4; do echo ffmpeg -i "$i" "${i/.mp4/.mp3}"; done
Then I put the exact same line of code in a script myscript.sh:
#!/bin/sh
for i in *.mp4; do echo ffmpeg -i "$i" "${i/.mp4/.mp3}"; done
However, now I get an error when running it:
$ sh myscript.sh
myscript.sh: 2: myscript.sh: Bad substitution
Based on other questions I tried changing the shebang to #!/bin/bash, but I get the exact same error. Why can't I run this script?

TL;DR: Since you are using Bash specific features, your script has to run with Bash and not with sh:
$ sh myscript.sh
myscript.sh: 2: myscript.sh: Bad substitution
$ bash myscript.sh
ffmpeg -i bar.mp4 bar.mp3
ffmpeg -i foo.mp4 foo.mp3
See Difference between sh and Bash. To find out which sh you are using: readlink -f $(which sh).
The best way to ensure a bash specific script always runs correctly
The best practices are to both:
Replace #!/bin/sh with #!/bin/bash (or whichever other shell your script depends on).
Run this script (and all others!) with ./myscript.sh or /path/to/myscript.sh, without a leading sh or bash.
Here's an example:
$ cat myscript.sh
#!/bin/bash
for i in *.mp4
do
echo ffmpeg -i "$i" "${i/.mp4/.mp3}"
done
$ chmod +x myscript.sh # Ensure script is executable
$ ./myscript.sh
ffmpeg -i bar.mp4 bar.mp3
ffmpeg -i foo.mp4 foo.mp3
(Related: Why ./ in front of scripts?)
The meaning of #!/bin/sh
The shebang suggests which shell the system should use to run a script. This allows you to specify #!/usr/bin/python or #!/bin/bash so that you don't have to remember which script is written in what language.
People use #!/bin/sh when they only use a limited set of features (defined by the POSIX standard) for maximum portability. #!/bin/bash is perfectly fine for user scripts that take advantage of useful bash extensions.
/bin/sh is usually symlinked to either a minimal POSIX compliant shell or to a standard shell (e.g. bash). Even in the latter case, #!/bin/sh may fail because bash will run in compatibility mode as explained in the man page:
If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.
The meaning of sh myscript.sh
The shebang is only used when you run ./myscript.sh, /path/to/myscript.sh, or when you drop the extension, put the script in a directory in your $PATH, and just run myscript.
If you explicitly specify an interpreter, that interpreter will be used. sh myscript.sh will force it to run with sh, no matter what the shebang says. This is why changing the shebang is not enough by itself.
You should always run the script with its preferred interpreter, so prefer ./myscript.sh or similar whenever you execute any script.
Other suggested changes to your script:
It is considered good practice to quote variables ("$i" instead of $i). Quoted variables will prevent problems if the stored file name contains white space characters.
I like that you use advanced parameter expansion. I suggest to use "${i%.mp4}.mp3" (instead of "${i/.mp4/.mp3}"), since ${parameter%word} only substitutes at the end (for example a file named foo.mp4.backup).

The ${var/x/y/} construct is not POSIX. In your case, where you just remove a string at the end of a variable and tack on another string, the portable POSIX solution is to use
#!/bin/sh
for i in *.mp4; do
ffmpeg -i "$i" "${i%.mp4}.mp3"
done
or even shorter, ffmpeg -i "$i" "${i%4}3".
The definitive dope for these constructs is the chapter on Parameter Expansion for the POSIX shell.

Related

Different outputs using ./ and sh [duplicate]

I have a line of code that works fine in my terminal:
for i in *.mp4; do echo ffmpeg -i "$i" "${i/.mp4/.mp3}"; done
Then I put the exact same line of code in a script myscript.sh:
#!/bin/sh
for i in *.mp4; do echo ffmpeg -i "$i" "${i/.mp4/.mp3}"; done
However, now I get an error when running it:
$ sh myscript.sh
myscript.sh: 2: myscript.sh: Bad substitution
Based on other questions I tried changing the shebang to #!/bin/bash, but I get the exact same error. Why can't I run this script?
TL;DR: Since you are using Bash specific features, your script has to run with Bash and not with sh:
$ sh myscript.sh
myscript.sh: 2: myscript.sh: Bad substitution
$ bash myscript.sh
ffmpeg -i bar.mp4 bar.mp3
ffmpeg -i foo.mp4 foo.mp3
See Difference between sh and Bash. To find out which sh you are using: readlink -f $(which sh).
The best way to ensure a bash specific script always runs correctly
The best practices are to both:
Replace #!/bin/sh with #!/bin/bash (or whichever other shell your script depends on).
Run this script (and all others!) with ./myscript.sh or /path/to/myscript.sh, without a leading sh or bash.
Here's an example:
$ cat myscript.sh
#!/bin/bash
for i in *.mp4
do
echo ffmpeg -i "$i" "${i/.mp4/.mp3}"
done
$ chmod +x myscript.sh # Ensure script is executable
$ ./myscript.sh
ffmpeg -i bar.mp4 bar.mp3
ffmpeg -i foo.mp4 foo.mp3
(Related: Why ./ in front of scripts?)
The meaning of #!/bin/sh
The shebang suggests which shell the system should use to run a script. This allows you to specify #!/usr/bin/python or #!/bin/bash so that you don't have to remember which script is written in what language.
People use #!/bin/sh when they only use a limited set of features (defined by the POSIX standard) for maximum portability. #!/bin/bash is perfectly fine for user scripts that take advantage of useful bash extensions.
/bin/sh is usually symlinked to either a minimal POSIX compliant shell or to a standard shell (e.g. bash). Even in the latter case, #!/bin/sh may fail because bash will run in compatibility mode as explained in the man page:
If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.
The meaning of sh myscript.sh
The shebang is only used when you run ./myscript.sh, /path/to/myscript.sh, or when you drop the extension, put the script in a directory in your $PATH, and just run myscript.
If you explicitly specify an interpreter, that interpreter will be used. sh myscript.sh will force it to run with sh, no matter what the shebang says. This is why changing the shebang is not enough by itself.
You should always run the script with its preferred interpreter, so prefer ./myscript.sh or similar whenever you execute any script.
Other suggested changes to your script:
It is considered good practice to quote variables ("$i" instead of $i). Quoted variables will prevent problems if the stored file name contains white space characters.
I like that you use advanced parameter expansion. I suggest to use "${i%.mp4}.mp3" (instead of "${i/.mp4/.mp3}"), since ${parameter%word} only substitutes at the end (for example a file named foo.mp4.backup).
The ${var/x/y/} construct is not POSIX. In your case, where you just remove a string at the end of a variable and tack on another string, the portable POSIX solution is to use
#!/bin/sh
for i in *.mp4; do
ffmpeg -i "$i" "${i%.mp4}.mp3"
done
or even shorter, ffmpeg -i "$i" "${i%4}3".
The definitive dope for these constructs is the chapter on Parameter Expansion for the POSIX shell.

Linux Script is failing [duplicate]

#!/bin/bash
jobname="job_201312161447_0003"
jobname_pre=${jobname:0:16}
jobname_post=${jobname:17}
This bash script gives me Bad substitution error on ubuntu. Any help will be highly appreciated.
The default shell (/bin/sh) under Ubuntu points to dash, not bash.
me#pc:~$ readlink -f $(which sh)
/bin/dash
So if you chmod +x your_script_file.sh and then run it with ./your_script_file.sh, or if you run it with bash your_script_file.sh, it should work fine.
Running it with sh your_script_file.sh will not work because the hashbang line will be ignored and the script will be interpreted by dash, which does not support that string substitution syntax.
I had the same problem. Make sure your script didnt have
#!/bin/sh
at the top of your script. Instead, you should add
#!/bin/bash
For others that arrive here, this exact message will also appear when using the env variable syntax for commands, for example ${which sh} instead of the correct $(which sh)
Your script syntax is valid bash and good.
Possible causes for the failure:
Your bash is not really bash but ksh or some other shell which doesn't understand bash's parameter substitution. Because your script looks fine and works with bash.
Do ls -l /bin/bash and check it's really bash and not sym-linked to some other shell.
If you do have bash on your system, then you may be executing your script the wrong way like: ksh script.sh or sh script.sh (and your default shell is not bash). Since you have proper shebang, if you have bash ./script.sh or bash ./script.sh should be fine.
Try running the script explicitly using bash command rather than just executing it as executable.
Also, make sure you don't have an empty string for the first line of your script.
i.e. make sure #!/bin/bash is the very first line of your script.
Not relevant to your example, but you can also get the Bad substitution error in Bash for any substitution syntax that Bash does not recognize. This could be:
Stray whitespace. E.g. bash -c '${x }'
A typo. E.g. bash -c '${x;-}'
A feature that was added in a later Bash version. E.g. bash -c '${x#Q}' before Bash 4.4.
If you have multiple substitutions in the same expression, Bash may not be very helpful in pinpointing the problematic expression. E.g.:
$ bash -c '"${x } multiline string
$y"'
bash: line 1: ${x } multiline string
$y: bad substitution
Both - bash or dash - work, but the syntax needs to be:
FILENAME=/my/complex/path/name.ext
NEWNAME=${FILENAME%ext}new
I was adding a dollar sign twice in an expression with curly braces in bash:
cp -r $PROJECT_NAME ${$PROJECT_NAME}2
instead of
cp -r $PROJECT_NAME ${PROJECT_NAME}2
I have found that this issue is either caused by the marked answer or you have a line or space before the bash declaration
Looks like "+x" causes problems:
root#raspi1:~# cat > /tmp/btest
#!/bin/bash
jobname="job_201312161447_0003"
jobname_pre=${jobname:0:16}
jobname_post=${jobname:17}
root#raspi1:~# chmod +x /tmp/btest
root#raspi1:~# /tmp/btest
root#raspi1:~# sh -x /tmp/btest
+ jobname=job_201312161447_0003
/tmp/btest: 4: /tmp/btest: Bad substitution
in my case (under ubuntu 18.04), I have mixed $( ${} ) that works fine:
BACKUPED_NB=$(ls ${HOST_BACKUP_DIR}*${CONTAINER_NAME}.backup.sql.gz | wc --lines)
full example here.
I used #!bin/bash as well tried all approaches like no line before or after #!bin/bash.
Then also tried using +x but still didn't work.
Finally i tried running the script ./script.sh it worked fine.
#!/bin/bash
jobname="job_201312161447_0003"
jobname_post=${jobname:17}
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts# sh jaru.sh
jaru.sh: 3: jaru.sh: Bad substitution
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts# ./jaru.sh
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts#

Error when doing string substitution in bash [duplicate]

I have a line of code that works fine in my terminal:
for i in *.mp4; do echo ffmpeg -i "$i" "${i/.mp4/.mp3}"; done
Then I put the exact same line of code in a script myscript.sh:
#!/bin/sh
for i in *.mp4; do echo ffmpeg -i "$i" "${i/.mp4/.mp3}"; done
However, now I get an error when running it:
$ sh myscript.sh
myscript.sh: 2: myscript.sh: Bad substitution
Based on other questions I tried changing the shebang to #!/bin/bash, but I get the exact same error. Why can't I run this script?
TL;DR: Since you are using Bash specific features, your script has to run with Bash and not with sh:
$ sh myscript.sh
myscript.sh: 2: myscript.sh: Bad substitution
$ bash myscript.sh
ffmpeg -i bar.mp4 bar.mp3
ffmpeg -i foo.mp4 foo.mp3
See Difference between sh and Bash. To find out which sh you are using: readlink -f $(which sh).
The best way to ensure a bash specific script always runs correctly
The best practices are to both:
Replace #!/bin/sh with #!/bin/bash (or whichever other shell your script depends on).
Run this script (and all others!) with ./myscript.sh or /path/to/myscript.sh, without a leading sh or bash.
Here's an example:
$ cat myscript.sh
#!/bin/bash
for i in *.mp4
do
echo ffmpeg -i "$i" "${i/.mp4/.mp3}"
done
$ chmod +x myscript.sh # Ensure script is executable
$ ./myscript.sh
ffmpeg -i bar.mp4 bar.mp3
ffmpeg -i foo.mp4 foo.mp3
(Related: Why ./ in front of scripts?)
The meaning of #!/bin/sh
The shebang suggests which shell the system should use to run a script. This allows you to specify #!/usr/bin/python or #!/bin/bash so that you don't have to remember which script is written in what language.
People use #!/bin/sh when they only use a limited set of features (defined by the POSIX standard) for maximum portability. #!/bin/bash is perfectly fine for user scripts that take advantage of useful bash extensions.
/bin/sh is usually symlinked to either a minimal POSIX compliant shell or to a standard shell (e.g. bash). Even in the latter case, #!/bin/sh may fail because bash will run in compatibility mode as explained in the man page:
If bash is invoked with the name sh, it tries to mimic the startup behavior of historical versions of sh as closely as possible, while conforming to the POSIX standard as well.
The meaning of sh myscript.sh
The shebang is only used when you run ./myscript.sh, /path/to/myscript.sh, or when you drop the extension, put the script in a directory in your $PATH, and just run myscript.
If you explicitly specify an interpreter, that interpreter will be used. sh myscript.sh will force it to run with sh, no matter what the shebang says. This is why changing the shebang is not enough by itself.
You should always run the script with its preferred interpreter, so prefer ./myscript.sh or similar whenever you execute any script.
Other suggested changes to your script:
It is considered good practice to quote variables ("$i" instead of $i). Quoted variables will prevent problems if the stored file name contains white space characters.
I like that you use advanced parameter expansion. I suggest to use "${i%.mp4}.mp3" (instead of "${i/.mp4/.mp3}"), since ${parameter%word} only substitutes at the end (for example a file named foo.mp4.backup).
The ${var/x/y/} construct is not POSIX. In your case, where you just remove a string at the end of a variable and tack on another string, the portable POSIX solution is to use
#!/bin/sh
for i in *.mp4; do
ffmpeg -i "$i" "${i%.mp4}.mp3"
done
or even shorter, ffmpeg -i "$i" "${i%4}3".
The definitive dope for these constructs is the chapter on Parameter Expansion for the POSIX shell.

Unix shell scripting: pass shell options (-x etc.) to nested scripts

How can I run nested shell scripts with the same option? For example,
parent.sh
#!/bin/sh
./child.sh
child.sh
#!/bin/sh
ls
How can I modify parent.sh so that when I run it with sh -x parent.sh, the -x option is effective in child.sh as well and the execution of ls is displayed on my console?
I'm looking for a portable solution which is effective for rare situations such as system users with /bin/false as their registered shell. Will the $SHELL environment variable be of any help?
Clarification: I sometimes want to call parent.sh with -x, sometimes with -e, depending on the situation. So the solution must not involve hard-coding the flags.
If you use bash, i can recommend the following:
#!/bin/bash
export SHELLOPTS
./child.sh
You can propagate as many times as you need, also you can use echo $SHELLOPTS in every script down the line to see what is happening and how options are propagated if you need to understand it better.
But for /bin/sh it will fail with /bin/sh: SHELLOPTS: readonly variable because of how POSIX is enforced on /bin/sh in various systems, more info here: https://lists.gnu.org/archive/html/bug-bash/2011-10/msg00052.html
it's looks like a hack and seems it's not the best way.
But it will do exact what you want
One of the ways how you can do it - it's to create aliases to create wrappers for sh:
alias saveShell='cp /bin/sh $some_safe_place'
alias shx='cp $some_safe_place /bin/x_sh; rm /bin/sh; echo "/bin/x_sh -x $#" > /bin/sh; chmod 755 /bin/sh '
alias she='cp $some_safe_place /bin/e_sh; rm /bin/sh; echo "/bin/e_sh -e $#" > /bin/sh; chmod 755 /bin/sh '
alias restoreShell='cp $some_safe_place /bin/sh'
How to Use:
run saveShell and then use shx or she , if you would change -x on -e run restoreShell and then run shx or she
run script as usually
sh ./parent.sh
BE VERY CAREFUL WITH MOVING SH
Other solution
replace #!/bin/sh to #!/bin/sh -x or #!/bin/sh -e with sed in all sh files before running script.

bash script: bad substitution [duplicate]

#!/bin/bash
jobname="job_201312161447_0003"
jobname_pre=${jobname:0:16}
jobname_post=${jobname:17}
This bash script gives me Bad substitution error on ubuntu. Any help will be highly appreciated.
The default shell (/bin/sh) under Ubuntu points to dash, not bash.
me#pc:~$ readlink -f $(which sh)
/bin/dash
So if you chmod +x your_script_file.sh and then run it with ./your_script_file.sh, or if you run it with bash your_script_file.sh, it should work fine.
Running it with sh your_script_file.sh will not work because the hashbang line will be ignored and the script will be interpreted by dash, which does not support that string substitution syntax.
I had the same problem. Make sure your script didnt have
#!/bin/sh
at the top of your script. Instead, you should add
#!/bin/bash
For others that arrive here, this exact message will also appear when using the env variable syntax for commands, for example ${which sh} instead of the correct $(which sh)
Your script syntax is valid bash and good.
Possible causes for the failure:
Your bash is not really bash but ksh or some other shell which doesn't understand bash's parameter substitution. Because your script looks fine and works with bash.
Do ls -l /bin/bash and check it's really bash and not sym-linked to some other shell.
If you do have bash on your system, then you may be executing your script the wrong way like: ksh script.sh or sh script.sh (and your default shell is not bash). Since you have proper shebang, if you have bash ./script.sh or bash ./script.sh should be fine.
Try running the script explicitly using bash command rather than just executing it as executable.
Also, make sure you don't have an empty string for the first line of your script.
i.e. make sure #!/bin/bash is the very first line of your script.
Not relevant to your example, but you can also get the Bad substitution error in Bash for any substitution syntax that Bash does not recognize. This could be:
Stray whitespace. E.g. bash -c '${x }'
A typo. E.g. bash -c '${x;-}'
A feature that was added in a later Bash version. E.g. bash -c '${x#Q}' before Bash 4.4.
If you have multiple substitutions in the same expression, Bash may not be very helpful in pinpointing the problematic expression. E.g.:
$ bash -c '"${x } multiline string
$y"'
bash: line 1: ${x } multiline string
$y: bad substitution
Both - bash or dash - work, but the syntax needs to be:
FILENAME=/my/complex/path/name.ext
NEWNAME=${FILENAME%ext}new
I was adding a dollar sign twice in an expression with curly braces in bash:
cp -r $PROJECT_NAME ${$PROJECT_NAME}2
instead of
cp -r $PROJECT_NAME ${PROJECT_NAME}2
I have found that this issue is either caused by the marked answer or you have a line or space before the bash declaration
Looks like "+x" causes problems:
root#raspi1:~# cat > /tmp/btest
#!/bin/bash
jobname="job_201312161447_0003"
jobname_pre=${jobname:0:16}
jobname_post=${jobname:17}
root#raspi1:~# chmod +x /tmp/btest
root#raspi1:~# /tmp/btest
root#raspi1:~# sh -x /tmp/btest
+ jobname=job_201312161447_0003
/tmp/btest: 4: /tmp/btest: Bad substitution
in my case (under ubuntu 18.04), I have mixed $( ${} ) that works fine:
BACKUPED_NB=$(ls ${HOST_BACKUP_DIR}*${CONTAINER_NAME}.backup.sql.gz | wc --lines)
full example here.
I used #!bin/bash as well tried all approaches like no line before or after #!bin/bash.
Then also tried using +x but still didn't work.
Finally i tried running the script ./script.sh it worked fine.
#!/bin/bash
jobname="job_201312161447_0003"
jobname_post=${jobname:17}
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts# sh jaru.sh
jaru.sh: 3: jaru.sh: Bad substitution
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts# ./jaru.sh
root#ip-10-2-250-36:/home/bitnami/python-module/workflow_scripts#

Resources