When I write:
echo 2*3>5 is a valid inequality
In my bash terminal, a new file named 5 is created in my directory which contains:
2*3 is a valid inequality
I want to know what exactly is going on here and why am I getting this output?
I believe it's obvious that I'm new to Linux!
Thanks
In bash, redirections can occur anywhere in the line (but you shouldn't do it! --- see the bash-hackers tutorial). Bash takes the >5 as a redirection, creates output file 5, and then processes the rest of the arguments. Therefore, echo 2*3 is a valid inequality happens, which gives you the output you see in the output file 5.
What you probably want is
echo "2*3>5 is a valid inequality"
or
echo '2*3>5 is a valid inequality'
(with single-quotes), either of which will give you the message you specify as a printout on the command line. The difference is that, within "", variables (such as $foo) will be filled in, but not within ''.
Edit: The bash man page says that the
redirection operators may precede or appear anywhere within a simple command or may follow a command. Redirections are processed in the order they appear, from left to right.
bash does the output redirection first i.e. >5 is done first and a file named 5 is created (or truncated if it already exists). The resultant file descriptor remains open for the runtime of the echo command.
Then the remaining portion, 2*3 is a valid inequality, runs as the argument to echo and standard output is saved in the (already-open) file 5 eventually.
To get the whole string as the output, use single or double quotes:
echo '2*3>5 is a valid inequality'
This is an example of output redirection. You're instructing the echo statement to, instead of writing to standard out, write to a filename. That filename happens to be "5".
You can avoid that behavior by quoting:
echo "2*3>5 is a valid inequality"
Related
I am trying write a script to compare one string variable to a list of values, i.e. if the variable matches (exact) to one of the values, then some action needs to be done.
The script is trying to match Unix pathnames, i.e. if the user enters / , /usr, /var etc, then to give an error, so that we do not get accidental corruption using the script. The list of values may change in future due to the application requirements. So I cannot have huge "if" statement to check this.
What I intend to do is that in case if the user enters, any of the forbidden path to give an error but sub-paths which are not forbidden should be allowed, i.e. /var should be rejected but /var/opt/app should be accepted.
I cannot use regex as partial match will not work
I am not sure of using a where loop and an if statement, is there any alternative?
thanks
I like to use associative arrays for this.
declare -A nonoList=(
[/foo/bar]=1
["/some/other/path with spaces"]=1
[/and/so/on]=1
# as many as you need
)
This can be kept in a file and sourced, if you want to separate it out.
Then in your script, just do a lookup.
if [[ -n "${nonoList[$yourString]}" ]] # -n checks for nonzero length
This also prevents you from creating a big file ad grep'ing over it redundantly, though that also works.
As an alternative, if you KNOW there will not be embedded newlines in any of those filenames (it's a valid character, but messy for programming) then you can do this:
$: cat foo
/foo/bar
/some/other/path with spaces
/and/so/on
Just a normal file with one file-path per line. Now,
chkSet=$'\n'"$(<foo)"$'\n' # single var, newlines before & after each
Then in your processing, assuming f=/foo/bar or whatever file you're checking,
if [[ "$chkSet" =~ $'\n'"$f"$'\n' ]] # check for a hit
This won't give you accidental hits on /some/other/path when the actual filename is /some/other/path with spaces because the pattern explicitly checks for a newline character before and after the filename. That's why we explicitly assure they exist at the front and end of the file. We assume they are in between, so make sure your file doesn't have any spaces (or any other characters, like quotes) that aren't part of the filenames.
If you KNOW there will also be no embedded whitespace in your filenames, it's a lot easier.
mapfile -t nopes < foo
if [[ " ${nopes[*]} " =~ " $yourString " ]]; then echo found; else echo no; fi
Note that " ${nopes[*]} " embeds spaces (technically it uses the first character of $IFS, but that's a space by default) into a single flattened string. Again, literal spaces before and behind key and list prevent start/end mismatches.
Paul,
Your alternative work around worked like a charm. I don't have any directories which need embedded space in them. So as long as my script can recognize that there are certain directories to avoid, it does its job.
Thanks
I'm new to linux and have been coding some beginenr level shell scripts.
What I want to do is write 2 scripts. The first script will read input from user and the 2nd script will display this input in a loop till it detects an "exit" from the user.
This is how I've coded the 2 shell scripts.
File1.sh:
read var1
echo $var1
File2.sh:
while [ "$var2" != "exit" ]
do
echo $1
read var2
done
Now, I want to use a named pipe to pass the output of File1.sh as input to var1 of File2.sh. I probably will have to modify code in File2.sh so that it will accept argument from a named pipe (as in instead of $1 the input will be from the named pipe), but I'm not at all sure how to go about it.
Giving the output of File1.sh as input to the named pipe can be given as follows:
mkfifo pipe
./File1.sh > pipe
This command keeps asking for input until i break out using ctrl + c. I don't know why that is.
Also how do I make the File2.sh read from this pipe?
will this be correct?
pipe|./File2.sh
I'm very new to linux but I've searched quite a lot online and there isn't even one example of doing this in shell script.
As for your original question, the syntax to read from a named pipe (or any other object in the file system) is
./File2.sh <pipe
Also, your script needs to echo "$var2" with the correct variable name, and double quotes to guard the value against wildcard expansion, variable substitution, etc. See also When to wrap quotes around a shell variable?
The code in your own answer has several new problems.
In File1.sh, you are apparently attempting to declare a variable pipe1, but the assignment syntax is wrong: You cannot have whitespace around the equals sign. Because you never use this variable for anything, this is by and large harmless (but will result in pipe1: command not found which is annoying, of course).
In File2.sh, the while loop's syntax is hopelessly screwed; you dropped the read; the echo still lacks quotes around the variable; and you repeatedly reopen the pipe.
while [ "$input" != "exit" ]
do
read -r input
echo "$input"
done <pipe1
Redirecting the entire loop once is going to be significantly more efficient.
Notice also the option -r to prevent read from performing any parsing of the values it reads. (The ugly default behavior is legacy from the olden days, and cannot be fixed without breaking existing scripts, unfortunately.)
First in File1.sh, echo var1 should be echo $var1.
In order to get input from pipe, try:
./File2.sh < pipe
This is how I solved it.
First mistake I made was to declare the pipe outside the programs. What I was expecting was there is a special way in which a program accepts input parameters of the type "pipe". Which as far as I've figured is wrong.
What you need to do is declare the pipe inside the program. So in the read program what you do is,
For File1.sh:
pipe1=/Documents
mkfifo pipe1
cat > pipe1
This will send the read input from the user to the pipe.
Now, when the pipe is open, it will keep accepting input. You can read from the pipe only when its open. So you need to open a 2nd terminal window to run the 2nd program.
For File2.sh:
while("$input" != "exit")
do
read -r input < pipe1
echo "$input"
done
So whenever you input some string in the first terminal window, it will be reflected in the 2nd terminal window until "exit" is detected.
I'm just starting to learn linux scripting, and with them user output/inputs. One of the things i need to learn and keep trying to do to no avail is append user input to a file output. Something like
read text > text.dat
or
read text
$text > text.dat
Typically ends up in failure, or the creation of text.dat which ends up empty no matter what is typed in by the user. What am i missing?
The read command, as documented in it's manual file, will take a line of user input and assign it to a variable which you can name as an argument. It will also split the user input and assign it to multiple variables if you pass more than one name. It will do this all in the background without printing any kind of confirmation to the standard out. We also know that the > operator will redirect the standard out of a command to a file descriptor. It is also important to note that unless bash is explicitly told that a line contains multiple commands (by using a semi-colon or similar) it will assume it is all one command with multiple arguments.
So lets have a look at your examples and see what is happening:
read text > text.dat
This will run the read command, which will silently assign the user input to a variable called $text. It will then redirect the output of the command (nothing, as it is silent) to a file called text.dat. End result: an empty text.dat and an unused $text variable.
read text $text > text.dat
Bash will parse this command and first attempt to get the value assigned to the $text variable, at this point it is undefined and so it will be ignored. So it will run the read command, which will silently assign the user input to a variable called $text. It will then redirect the output of the command (nothing, as it is silent) to a file called text.dat. End result: an empty text.dat and an unused $text variable.
So how can we resolve this? The first command is fine, we use read text to allow the user to input a line and have that line assigned to a variable called $text. Then, we need a way to send that variable to standard out so we can redirect it. To do that, we can use the echo command, which we can redirect.
So for example:
read text
echo $text > text.dat
Another thing to note is that the > operator will overwrite the file, to append to it you can use the >> operator.
So to take a user input and append it to a file we have:
read text
echo $text >> text.dat
I am using Install Anywhere 2012 and would like to be able to parse a batch or shell script for a give value and have that value stored in an IA variable. For instance, if I have the following shell file:
MY_VAR1=123
MY_VAR2=a\b\c
ECHO $MY_VAR1
I would like to pass in the path to the file and the variable name (ex. MY_VAR1) and have the result, 123, stored in an IA variable of my choosing (lets say $OUTPUT$). I could achieve this through writing some java custom code but was wondering if there was an alternative approach built into IA that would make this much easier. The variable will not be initialized when I need to figure out its value so essentially just echoing it's value or something similar will not work. Any help would be greatly appreciated!
example in Windows batch:
#echo off &setlocal
for /f "tokens=2delims==" %%a in ('findstr "MY_VAR1" "ShellFile"') do set "output=%%a"
if defined output (echo MY_VAR1: %output%) else echo MY_VAR1 not found!
On Linux/Unix, you could use perl or awk (both are standard utilities in most distros). Python or Ruby are also candidates, but may not be installed on your target system. You could even write your own targeted parser using Lex and Yacc and ship it with your installer. However, for your needs, that's surely overkill.
Here's an example of a possible awk solution in an Execute Script/Batch File Action:
#!/bin/bash
awk '
# Process lines that begin with our variable name,
# preceded by optional spaces or tabs.
/^[ \t]*$TARGET_VARIABLE_NAME$=.+/ {
# Split the current line on "=" into
# an array called result
split($0, result, "=")
value = result[1]
# Look for trailing comments and remove them.
offset = index(value, "#")
if (offset > 0) {
value = substr(value, 1, offset - 1)
}
# Remove any possible leading spaces and quotes.
# Note that the single-quote is escaped. That escape
# is for bash, not for awk. I am doing this from
# memory and do not have access to IA right now.
# you may have to play with the escaping.
gsub(/^[\'" ]*/, "", value)
# Remove any possible trailing spaces and quotes.
# See above regarding the escaped single-quote.
gsub(/[\'" ]*$/, "", value)
# send "value" to stdout
print value
}
' < $SHELL_INPUT_FILE$
The print value line (near the end) sends value to stdout.
In the Execute Script/Batch File Action settings you can designate variables that receive the stdout and stderr streams produced by the script action. By default, the stdout stream is stored in $EXECUTE_STDOUT$. You can change this to a variable name of your choosing.
In the example, above, $TARGET_VARIABLE_NAME$ and $SHELL_INPUT_FILE$ are InstallAnywhere variables that hold the name of the variable to find and the name of the file to parse, respectively. These variables will be replaced by their values before the Action executes.
Assume we have a script called /home/fred/hello.sh, which contains the following code:
#!/bin/bash
WIFE='Wilma'
NEIGHBOR="Barney Rubble"
echo "Hello to $WIFE and $NEIGHBOR from $PWD"
Before the Execute Script/Batch File Action runs, stuff the name of the script file into $SHELL_INPUT_FILE$ (/home/fred/hello.sh). Then set the value of $TARGET_VARIABLE_NAME$ to the variable you wish to find (say, NEIGHBOR). After the action completes, $EXECUTE_STDOUT$ in InstallAnywhere will contain Barney Rubble.
You can build on this idea to parse arbitrarily complex files in an Execute Script/Batch File Action. Just make your awk (or perl/Ruby/Python) script as complex as needed.
NOTE: when scripting Unix shell scripts in InstallAnywhere ALWAYS check the "Do not replace unknown variables" option. If you don't, InstallAnywhere will quietly convert anything that looks vaguely like an InstallAnywhere variable into blanks... It's very annoying.
For a Windows solution, find a standalone Windows version of awk or perl and include it with your installation. Then extend the above solution to work for batch files.
You'd want to create two Execute Script/Batch File Actions, one with a rule for Linux/Unix and one with a rule for Windows. You'd have to install the Windows awk or perl executable before calling this action though. Also, you'd need to fully qualify the path to the awk/perl executable. Finally, the actual script will need to be sensitive to differences in batch syntax versus shell syntax.
Below is an awk script modified to look for batch variable definitions. The pattern changes and you won't have to worry about embedded comments:
$PATH_TO_AWK_EXE$ '
# This pattern looks for optional spaces, the word SET
# with any capitalization, the target variable, more
# optional spaces and the equals sign.
/^[ \t]*[Ss][Ee][Tt][ \t]*$TARGET_VARIABLE_NAME$[ \t]*=.+/ {
# Split the current line on "=" into
# an array called result
split($0, result, "=")
value = result[1]
# No trailing comments in Batch files.
# Remove any possible leading spaces and quotes.
# Note that the single-quote is escaped. That escape
# is for bash, not for awk. I am doing this from
# memory and do not have access to IA right now.
# you may have to play with the escaping.
gsub(/^[\'" ]*/, "", value)
# Remove any possible trailing spaces and quotes.
# See above regarding the escaped single-quote.
gsub(/[\'" ]*$/, "", value)
# send "value" to stdout
print value
}
' < $SHELL_INPUT_FILE$
Above, the IA variable $PATH_TO_AWK_EXE$ points to the location where awk was installed. It would be set as some combination of $USER_INSTALL_FOLDER$, possibly other directory names, and the name of the awk.exe file. $PATH_TO_AWK_EXE$ can later be used to remove the awk executable, if desired.
You can try to get variable from output of script
"Execute Script" -> Store process's stdout in: $EXECUTE_OUTPUT$
Than you can use $EXECUTE_OUTPUT$ as variable after that
What idiom should one use in Bash scripts (no Perl, Python, and such please) to build up a command line for another program out of the script's arguments while handling filenames correctly?
By correctly, I mean handling filenames with spaces or odd characters without inadvertently causing the other program to handle them as separate arguments (or, in the case of < or > — which are, after all, valid if unfortunate filename characters if properly escaped — doing something even worse).
Here's a made-up example of what I mean, in a form that doesn't handle filenames correctly: Let's assume this script (foo) builds up a command line for a command (bar, assumed to be in the path) by taking all of foo's input arguments and moving anything that looks like a flag to the front, and then invoking bar:
#!/bin/bash
# This is clearly wrong
FILES=
FLAGS=
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags string
FLAGS="$FLAGS $ARG"
else
# Looks like a file, add it to the files string
FILES="$FILES $ARG"
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD="bar $FLAGS $FILES"
echo "Issuing: $CMD"
$CMD
(Note that this just an example; there are lots of other times one needs to do this and that to a bunch of args and then pass them onto other programs.)
In a naive scenario with simple filenames, that works great. But if we assume a directory containing the files
one
two
three and a half
four < five
then of course the command foo * fails miserably in its task:
foo: Handling four < five
foo: Handling one
foo: Handling three and a half
foo: Handling two
Issuing: bar four < five one three and a half two
If we actually allow foo to issue that command, well, the results won't be what we're expecting.
Previously I've tried to handle this through the simple expedient of ensuring that there are quotes around each filename, but I've (very) quickly learned that that is not the correct approach. :-)
So what is? Constraints:
I want to keep the idiom as simple as possible (not least so I can remember it).
I'm looking for a general-purpose idiom, hence my making up the bar program and the contrived example above instead of using a real scenario where people might easily (and reasonably) go down the route of trying to use features in the target program.
I want to stick to Bash script, I don't want to call out to Perl, Python, etc.
I'm fine with relying on (other) standard *nix utilities, like xargs, sed, or tr provided we don't get too obtuse (see #1 above). (Apologies to Perl, Python, etc. programmers who think #3 and #4 combine to draw an arbitrary distinction.)
If it matters, the target program might also be a Bash script, or might not. I wouldn't expect it to matter...
I don't just want to handle spaces, I want to handle weird characters correctly as well.
I'm not bothered if it doesn't handle filenames with embedded nul characters (literally character code 0). If someone's managed to create one in their filesystem, I'm not worried about handling it, they've tried really hard to mess things up.
Thanks in advance, folks.
Edit: Ignacio Vazquez-Abrams pointed me to Bash FAQ entry #50, which after some reading and experimentation seems to indicate that one way is to use Bash arrays:
#!/bin/bash
# This appears to work, using Bash arrays
# Start with blank arrays
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS+=("$ARG")
else
# Looks like a file, add it to the files array
FILES+=("$ARG")
fi
done
# Call bar with the flags and files
echo "Issuing (but properly delimited, not exactly as this appears): bar ${FLAGS[#]} ${FILES[#]}"
bar "${FLAGS[#]}" "${FILES[#]}"
Is that correct and reasonable? Or am I relying on something environmental above that will bite me later. It seems to work and it ticks all the other boxes for me (simple, easy to remember, etc.). It does appear to rely on a relatively recent Bash feature (FAQ entry #50 mentions v3.1, but I wasn't sure whether that was arrays in general of some of the syntax they were using with it), but I think it's likely I'll only be dealing with versions that have it.
(If the above is correct and you want to un-delete your answer, Ignacio, I'll accept it provided I haven't accepted any others yet, although I stand by my statement about link-only answers.)
Why do you want to "build up" a command? Add the files and flags to arrays using proper
quoting and issue the command directly using the quoted arrays as arguments.
Selected lines from your script (omitting unchanged ones):
if [[ ${ARG:0:1} == - ]]; then # using a Bash idiom
FLAGS+=("$ARG") # add an element to an array
FILES+=("$ARG")
echo "Issuing: bar \"${FLAGS[#]}\" \"${FILES[#]}\""
bar "${FLAGS[#]}" "${FILES[#]}"
For a quick demo of using arrays in this manner:
$ a=(aaa 'bbb ccc' ddd); for arg in "${a[#]}"; do echo "..${arg}.."; done
Output:
..aaa..
..bbb ccc..
..ddd..
Please see BashFAQ/050 regarding putting commands in variables. The reason that your script doesn't work is because there's no way to quote the arguments within a quoted string. If you were to put quotes there, they would be considered part of the string itself instead of as delimiters. With the arguments left unquoted, word splitting is done and arguments that include spaces are seen as more than one argument. Arguments with "<", ">" or "|" are not a problem in any case since redirection and piping is performed before variable expansion so they are seen as characters in a string.
By putting the arguments (filenames) in an array, spaces, newlines, etc., are preserved. By quoting the array variable when it's passed as an argument, they are preserved on the way to the consuming program.
Some additional notes:
Use lowercase (or mixed case) variable names to reduce the chance that they will collide with the shell's builtin variables.
If you use single square brackets for conditionals in any modern shell, the archaic "x" idiom is no longer necessary if you quote the variables (see my answer here). However, in Bash, use double brackets. They provide additional features (see my answer here).
Use getopts as Let_Me_Be suggested. Your script, though I know it's only an example, will not be able to handle switches that take arguments.
This for ARG in "$#" can be shortened to this for ARG (but I prefer the readability of the more explicit version).
See BashFAQ #50 (and also maybe #35 on option parsing). For the scenario you describe, where you're building a command dynamically, the best option is to use arrays rather than simple strings, as they won't lose track of where the word boundaries are. The general rules are: to create an array, instead of VAR="foo bar baz", use VAR=("foo" "bar" "baz"); to use the array, instead of $VAR, use "${VAR[#]}". Here's a working version of your example script using this method:
#!/bin/bash
# This is clearly wrong
FILES=()
FLAGS=()
for ARG in "$#"; do
echo "foo: Handling $ARG"
if [ x${ARG:0:1} = "x-" ]; then
# Looks like a flag, add it to the flags array
FLAGS=("${FLAGS[#]}" "$ARG") # FLAGS+=("$ARG") would also work in bash 3.1+, as Dennis pointed out
else
# Looks like a file, add it to the files string
FILES=("${FILES[#]}" "$ARG")
fi
done
# Call bar with the flags and files (we don't care that they'll
# have an extra space or two)
CMD=("bar" "${FLAGS[#]}" "${FILES[#]}")
echo "Issuing: ${CMD[*]}"
"${CMD[#]}"
Note that in the echo command I used "${VAR[*]}" instead of the [#] form because there's no need/point to preserving word breaks here. If you wanted to print/record the command in unambiguous form, this would be a lot messier.
Also, this gives you no way to build up redirections or other special shell options in the built command -- if you add >outfile to the FILES array, it'll be treated as just another command argument, not a shell redirection. If you need to programmatically build these, be prepared for headaches.
getopts should be able to handle spaces in arguments correctly ("file name.txt"). Weird characters should work as well, assuming they are correctly escaped (ls -b).