I have a variable in a shell script in which I'd like to format the data. The variable stores new data during every iteration of a loop. Each time the new data is stored, I'd like to insert a new line character. Here is how I'm trying to store the data into the variable.
VARIABLE="$VARIABLE '\n' SomeData"
Unfortunately, the output includes the literal '\n' Any help would be appreciative.
Try $'\n':
VAR=a
VAR="$VAR"$'\n'b
echo "$VAR"
gives me
a
b
A common technique is:
nl='
'
VARIABLE="PreviousData"
VARIABLE="$VARIABLE${nl}SomeData"
echo "$VARIABLE"
PreviousData
SomeData
Also common, to prevent inadvertently having your string start with a newline:
VARIABLE="$VARIABLE${VARIABLE:+$nl}SomeData"
(The expression ${VARIABLE:+$nl} will expand to a newline if and only if VARIABLE is set and non-empty.)
VAR="one"
VAR="$VAR.\n.two"
echo -e $VAR
Output:
one.
.two
Other than $'\n' you can use printf also like this:
VARIABLE="Foo Bar"
VARIABLE=$(printf "${VARIABLE}\nSomeData")
echo "$VARIABLE"
OUTPUT:
Foo Bar
SomeData
I had a problem with all the other solutions: when using a # followed by SPACE (quite common when writing in Markdown) both would get split onto a new line.
So, another way of doing it would involve using single quotes so that the "\n" get rendered.
FOO=$'# Markdown Title #\n'
BAR=$'Be *brave* and **bold**.'
FOOBAR="$FOO$BAR"
echo "$FOOBAR"
Output:
# Markdown Title #
Be *brave* and **bold**.
Single quote All special characters between these quotes lose their
special meaning.https://www.tutorialspoint.com/unix/unix-quoting-mechanisms.htm
So the syntax you use does something different that you want to achieve.
This is what you need:
The $'\X' construct makes the -e option in echo unnecessary.
https://linux.die.net/abs-guide/escapingsection.html
echo -e "something\nsomething"
or
echo "something"$'\n'"something"
It's a lot simpler than you think:
VARIABLE="$VARIABLE
SomeData"
Building upon the first two solutions, I'd do like shown below. Concatenating strings with the '+=' operator, somehow looks clearer to me.
Also rememeber to use printf as opposed to echo, you will save yourself so much trouble
sometext="This is the first line"
sometext+=$'\n\n'
sometext+="This is the second line AFTER the inserted new lines"
printf '%s' "${sometext}"
Outputs:
This is the first line
This is the third line AFTER the inserted new line
Your problem is in the echo command, in ash you have to use the option -e to expand special characters. This should work for you:
VAR="First line"
VAR="$VAR\nSecond line"
echo -e $VAR
This outputs
First line
Second line
Related
I need to extract path from a string. I found examples in another post, but missing additional steps.
I have a string as below:
title="test test good dskgkdh hdfyr /rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL kdlkfg nsfgf trhrnrt"
cobsrc=$(awk '{match($0,/\/[^"]*/,a);print a[0]}' <<< $title)
echo $cobsrc
Output is
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL kdlkfg nsfgf trhrnrt
I need only
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
What modification is required?
An existing post on similar query:
how to extract path from string in shell script
Four solutions, in order of my own preference.
First option would be simple parameter expansion, in two steps:
$ title="/${title#*/}"
$ title="${title%% *}"
$ echo "$title"
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
The first line removes everything up to the first slash (while prepending a slash to replace the one that's stripped", the second line removes everything from the first bit of whitespace that remains.
Or, if you prefer, use a regex:
$ [[ $title =~ ^[^/]*(/[^ ]+)\ ]]
$ echo ${BASH_REMATCH[1]}
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
The regex translates as:
null at the beginning of the line,
a run of zero or more non-slashes,
an atom:
a slash followed by non-space characters
a space, to end the previous atom.
The $BASH_REMATCH array contains the content of the bracketed atom.
Next option might be grep -o:
$ grep -o '/[^ ]*' <<<"$title"
(Result redacted -- you know what it'll be.)
You could of course assign this output to a variable using command substitution, which you already know about.
Last option is another external tool...
$ sed 's:^[^/]*::;s/ .*//' <<<"$title"
This is the same functionality as is handled by the parameter expansion (at the top of the answer) only in a sed script, which requires a call to an external program. Included only for pedantry. :)
Could you please try following.
echo "$title" | awk 'match($0,/\/.*\/[^ ]*/){print substr($0,RSTART,RLENGTH)}'
Output will be as follows.
/rlsmodules/svnrepo/SOURCE/CBL/MQ/BASELINE/MQO000.CBL
Solution 2nd: Considering that your variable don't have space in between its value then following may help you too.
echo "$title" | awk '{sub(/[^/]* /,"");sub(/ .*/,"")} 1'
I need my script to send an email from terminal. Based on what I've seen here and many other places online, I formatted it like this:
/var/mail -s "$SUBJECT" "$EMAIL" << EOF
Here's a line of my message!
And here's another line!
Last line of the message here!
EOF
However, when I run this I get this warning:
myfile.sh: line x: warning: here-document at line y delimited by end-of-file (wanted 'EOF')
myfile.sh: line x+1: syntax error: unexpected end of file
...where line x is the last written line of code in the program, and line y is the line with /var/mail in it. I've tried replacing EOF with other things (ENDOFMESSAGE, FINISH, etc.) but to no avail. Nearly everything I've found online has it done this way, and I'm really new at bash so I'm having a hard time figuring it out on my own. Could anyone offer any help?
The EOF token must be at the beginning of the line, you can't indent it along with the block of code it goes with.
If you write <<-EOF you may indent it, but it must be indented with Tab characters, not spaces. So it still might not end up even with the block of code.
Also make sure you have no whitespace after the EOF token on the line.
The line that starts or ends the here-doc probably has some non-printable or whitespace characters (for example, carriage return) which means that the second "EOF" does not match the first, and doesn't end the here-doc like it should. This is a very common error, and difficult to detect with just a text editor. You can make non-printable characters visible for example with cat:
cat -A myfile.sh
Once you see the output from cat -A the solution will be obvious: remove the offending characters.
Please try to remove the preceeding spaces before EOF:-
/var/mail -s "$SUBJECT" "$EMAIL" <<-EOF
Using <tab> instead of <spaces> for ident AND using <<-EOF works fine.
The "-" removes the <tabs>, not <spaces>, but at least this works.
Note one can also get this error if you do this;
while read line; do
echo $line
done << somefile
Because << somefile should read < somefile in this case.
May be old but I had a space after the ending EOF
<< EOF
blah
blah
EOF <-- this was the issue. Had it for years, finally looked it up here
For anyone stumbling here who googled "bash warning: here-document delimited by end-of-file", it may be that you are getting the
warning: here-document at line 74 delimited by end-of-file
...type warning because you accidentally used a here document symbol (<<) when you meant to use a here string symbol (<<<). That was my case.
Here is a flexible way to do deal with multiple indented lines without using heredoc.
echo 'Hello!'
sed -e 's:^\s*::' < <(echo '
Some indented text here.
Some indented text here.
')
if [[ true ]]; then
sed -e 's:^\s\{4,4\}::' < <(echo '
Some indented text here.
Some extra indented text here.
Some indented text here.
')
fi
Some notes on this solution:
if the content is expected to have simple quotes, either escape them using \ or replace the string delimiters with double quotes. In the latter case, be careful that construction like $(command) will be interpreted. If the string contains both simple and double quotes, you'll have to escape at least of kind.
the given example print a trailing empty line, there are numerous way to get rid of it, not included here to keep the proposal to a minimum clutter
the flexibility comes from the ease with which you can control how much leading space should stay or go, provided that you know some sed REGEXP of course.
When I want to have docstrings for my bash functions, I use a solution similar to the suggestion of user12205 in a duplicate of this question.
See how I define USAGE for a solution that:
auto-formats well for me in my IDE of choice (sublime)
is multi-line
can use spaces or tabs as indentation
preserves indentations within the comment.
function foo {
# Docstring
read -r -d '' USAGE <<' END'
# This method prints foo to the terminal.
#
# Enter `foo -h` to see the docstring.
# It has indentations and multiple lines.
#
# Change the delimiter if you need hashtag for some reason.
# This can include $$ and = and eval, but won't be evaluated
END
if [ "$1" = "-h" ]
then
echo "$USAGE" | cut -d "#" -f 2 | cut -c 2-
return
fi
echo "foo"
}
So foo -h yields:
This method prints foo to the terminal.
Enter `foo -h` to see the docstring.
It has indentations and multiple lines.
Change the delimiter if you need hashtag for some reason.
This can include $$ and = and eval, but won't be evaluated
Explanation
cut -d "#" -f 2: Retrieve the second portion of the # delimited lines. (Think a csv with "#" as the delimiter, empty first column).
cut -c 2-: Retrieve the 2nd to end character of the resultant string
Also note that if [ "$1" = "-h" ] evaluates as False if there is no first argument, w/o error, since it becomes an empty string.
make sure where you put the ending EOF you put it at the beginning of a new line
Along with the other answers mentioned by Barmar and Joni, I've noticed that I sometimes have to leave a blank line before and after my EOF when using <<-EOF.
I have a file stv.txt containing some names
For example stv.txt is as follows:
hello
world
I want to generate another file by using these names and adding some extra text to them.I have written a script as follows
for i in `cat stvv.txt`;
do printf 'if(!strcmp("$i",optarg))' > my_file;
done
output
if(!strcmp("$i",optarg))
desired output
if(!strcmp("hello",optarg))
if(!strcmp("world",optarg))
how can I get the correct result?
This is a working solution.
1 All symbols inside single quotes is considered a string. 2 When using printf, do not surround the variable with quotes. (in this example)
The code below should fix it,
for i in `cat stvv.txt`;
printf 'if(!strcmp('$i',optarg))' > my_file;
done
basically, break the printf statement into three parts.
1: the string 'if(!strcmp('
2: $i (no quotes)
3: the string ',optarg))'
hope that helps!
To insert a string into a printf format, use %s in the format string:
$ for line in $(cat stvv.txt); do printf 'if(!strcmp("%s",optarg))\n' "$line"; done
if(!strcmp("hello",optarg))
if(!strcmp("world",optarg))
The code $(cat stvv.txt) will perform word splitting and pathname expansion on the contents of stvv.txt. You probably don't want that. It is generally safer to use a while read ... done <stvv.txt loop such as this one:
$ while read -r line; do printf 'if(!strcmp("%s",optarg))\n' "$line"; done <stvv.txt
if(!strcmp("hello",optarg))
if(!strcmp("world",optarg))
Aside on cat
If you are using bash, then $(cat stvv.txt) could be replaced with the more efficient $(<stvv.txt). This question, however, is tagged shell not bash. The cat form is POSIX and therefore portable to all POSIX shells while the bash form is not.
I'm writing a shell script that should be somewhat secure, i.e., does not pass secure data through parameters of commands and preferably does not use temporary files. How can I pass a variable to the standard input of a command?
Or, if it's not possible, how can I correctly use temporary files for such a task?
Passing a value to standard input in Bash is as simple as:
your-command <<< "$your_variable"
Always make sure you put quotes around variable expressions!
Be cautious, that this will probably work only in bash and will not work in sh.
Simple, but error-prone: using echo
Something as simple as this will do the trick:
echo "$blah" | my_cmd
Do note that this may not work correctly if $blah contains -n, -e, -E etc; or if it contains backslashes (bash's copy of echo preserves literal backslashes in absence of -e by default, but will treat them as escape sequences and replace them with corresponding characters even without -e if optional XSI extensions are enabled).
More sophisticated approach: using printf
printf '%s\n' "$blah" | my_cmd
This does not have the disadvantages listed above: all possible C strings (strings not containing NULs) are printed unchanged.
(cat <<END
$passwd
END
) | command
The cat is not really needed, but it helps to structure the code better and allows you to use more commands in parentheses as input to your command.
Note that the 'echo "$var" | command operations mean that standard input is limited to the line(s) echoed. If you also want the terminal to be connected, then you'll need to be fancier:
{ echo "$var"; cat - ; } | command
( echo "$var"; cat - ) | command
This means that the first line(s) will be the contents of $var but the rest will come from cat reading its standard input. If the command does not do anything too fancy (try to turn on command line editing, or run like vim does) then it will be fine. Otherwise, you need to get really fancy - I think expect or one of its derivatives is likely to be appropriate.
The command line notations are practically identical - but the second semi-colon is necessary with the braces whereas it is not with parentheses.
This robust and portable way has already appeared in comments. It should be a standalone answer.
printf '%s' "$var" | my_cmd
or
printf '%s\n' "$var" | my_cmd
Notes:
It's better than echo, reasons are here: Why is printf better than echo?
printf "$var" is wrong. The first argument is format where various sequences like %s or \n are interpreted. To pass the variable right, it must not be interpreted as format.
Usually variables don't contain trailing newlines. The former command (with %s) passes the variable as it is. However tools that work with text may ignore or complain about an incomplete line (see Why should text files end with a newline?). So you may want the latter command (with %s\n) which appends a newline character to the content of the variable. Non-obvious facts:
Here string in Bash (<<<"$var" my_cmd) does append a newline.
Any method that appends a newline results in non-empty stdin of my_cmd, even if the variable is empty or undefined.
I liked Martin's answer, but it has some problems depending on what is in the variable. This
your-command <<< """$your_variable"""
is better if you variable contains " or !.
As per Martin's answer, there is a Bash feature called Here Strings (which itself is a variant of the more widely supported Here Documents feature):
3.6.7 Here Strings
A variant of here documents, the format is:
<<< word
The word is expanded and supplied to the command on its standard
input.
Note that Here Strings would appear to be Bash-only, so, for improved portability, you'd probably be better off with the original Here Documents feature, as per PoltoS's answer:
( cat <<EOF
$variable
EOF
) | cmd
Or, a simpler variant of the above:
(cmd <<EOF
$variable
EOF
)
You can omit ( and ), unless you want to have this redirected further into other commands.
Try this:
echo "$variable" | command
If you came here from a duplicate, you are probably a beginner who tried to do something like
"$variable" >file
or
"$variable" | wc -l
where you obviously meant something like
echo "$variable" >file
echo "$variable" | wc -l
(Real beginners also forget the quotes; usually use quotes unless you have a specific reason to omit them, at least until you understand quoting.)
Alright, I know this is a simple question, but I can't seem to get this sed command to work. I'm trying to get a text file and replace one bit of it from placeholder text to a study code. The study code that it is going to replace it with is passed into the script using arguments when the script is first ran. The problem is, when I try to replace the placeholder text with the variable $study, it replaces it with a literally "$study".
Right now my arguments set like this:
export study=$1
export tag=$2
export mode=$3
export select=$4
My sed command looks like this:
sed -i.backup -e 's/thisisthestudycodereplacethiswiththestudycode/$study/' freq.spx
Is there some easy way of getting sed to not look at the literal $study, or would it be better at this point to do it another way?
Use double quotes instead of single quotes.
Because ' quoting prevents shell variable expansions, and " quoting does not.
You probably won't run into this issue, but just in case...
Paul's answer is slightly suboptimal if $study might contain slashes or other characters with special meaning to sed.
mv freq.spx freq.spx.backup && \
awk -v "study=$study" '{
gsub(/thisisthestudycodereplacethiswiththestudycode/, study);
print;
}' freq.spx.backup > freq.spx
Although awkward (hah, pun!), this will always work regardless of $study's contents.
try this...................
sed -e 's/%d/'$a'/g' amar.htm , amar.htm having the string "%d" which is indeed to be replaced and "a" is having the string to replace.