How can I write a here document to a file in Bash script?
Read the Advanced Bash-Scripting Guide Chapter 19. Here Documents.
Here's an example which will write the contents to a file at /tmp/yourfilehere
cat << EOF > /tmp/yourfilehere
These contents will be written to the file.
This line is indented.
EOF
Note that the final 'EOF' (The LimitString) should not have any whitespace in front of the word, because it means that the LimitString will not be recognized.
In a shell script, you may want to use indentation to make the code readable, however this can have the undesirable effect of indenting the text within your here document. In this case, use <<- (followed by a dash) to disable leading tabs (Note that to test this you will need to replace the leading whitespace with a tab character, since I cannot print actual tab characters here.)
#!/usr/bin/env bash
if true ; then
cat <<- EOF > /tmp/yourfilehere
The leading tab is ignored.
EOF
fi
If you don't want to interpret variables in the text, then use single quotes:
cat << 'EOF' > /tmp/yourfilehere
The variable $FOO will not be interpreted.
EOF
To pipe the heredoc through a command pipeline:
cat <<'EOF' | sed 's/a/b/'
foo
bar
baz
EOF
Output:
foo
bbr
bbz
... or to write the the heredoc to a file using sudo:
cat <<'EOF' | sed 's/a/b/' | sudo tee /etc/config_file.conf
foo
bar
baz
EOF
Instead of using cat and I/O redirection it might be useful to use tee instead:
tee newfile <<EOF
line 1
line 2
line 3
EOF
It's more concise, plus unlike the redirect operator it can be combined with sudo if you need to write to files with root permissions.
Note:
the following condenses and organizes other answers in this thread, esp the excellent work of Stefan Lasiewski and Serge Stroobandt
Lasiewski and I recommend Ch 19 (Here Documents) in the Advanced Bash-Scripting Guide
The question (how to write a here document (aka heredoc) to a file in a bash script?) has (at least) 3 main independent dimensions or subquestions:
Do you want to overwrite an existing file, append to an existing file, or write to a new file?
Does your user or another user (e.g., root) own the file?
Do you want to write the contents of your heredoc literally, or to have bash interpret variable references inside your heredoc?
(There are other dimensions/subquestions which I don't consider important. Consider editing this answer to add them!) Here are some of the more important combinations of the dimensions of the question listed above, with various different delimiting identifiers--there's nothing sacred about EOF, just make sure that the string you use as your delimiting identifier does not occur inside your heredoc:
To overwrite an existing file (or write to a new file) that you own, substituting variable references inside the heredoc:
cat << EOF > /path/to/your/file
This line will write to the file.
${THIS} will also write to the file, with the variable contents substituted.
EOF
To append an existing file (or write to a new file) that you own, substituting variable references inside the heredoc:
cat << FOE >> /path/to/your/file
This line will write to the file.
${THIS} will also write to the file, with the variable contents substituted.
FOE
To overwrite an existing file (or write to a new file) that you own, with the literal contents of the heredoc:
cat << 'END_OF_FILE' > /path/to/your/file
This line will write to the file.
${THIS} will also write to the file, without the variable contents substituted.
END_OF_FILE
To append an existing file (or write to a new file) that you own, with the literal contents of the heredoc:
cat << 'eof' >> /path/to/your/file
This line will write to the file.
${THIS} will also write to the file, without the variable contents substituted.
eof
To overwrite an existing file (or write to a new file) owned by root, substituting variable references inside the heredoc:
cat << until_it_ends | sudo tee /path/to/your/file
This line will write to the file.
${THIS} will also write to the file, with the variable contents substituted.
until_it_ends
To append an existing file (or write to a new file) owned by user=foo, with the literal contents of the heredoc:
cat << 'Screw_you_Foo' | sudo -u foo tee -a /path/to/your/file
This line will write to the file.
${THIS} will also write to the file, without the variable contents substituted.
Screw_you_Foo
To build on #Livven's answer, here are some useful combinations.
variable substitution, leading tab retained, overwrite file, echo to stdout
tee /path/to/file <<EOF
${variable}
EOF
no variable substitution, leading tab retained, overwrite file, echo to stdout
tee /path/to/file <<'EOF'
${variable}
EOF
variable substitution, leading tab removed, overwrite file, echo to stdout
tee /path/to/file <<-EOF
${variable}
EOF
variable substitution, leading tab retained, append to file, echo to stdout
tee -a /path/to/file <<EOF
${variable}
EOF
variable substitution, leading tab retained, overwrite file, no echo to stdout
tee /path/to/file <<EOF >/dev/null
${variable}
EOF
the above can be combined with sudo as well
sudo -u USER tee /path/to/file <<EOF
${variable}
EOF
When root permissions are required
When root permissions are required for the destination file, use |sudo tee instead of >:
cat << 'EOF' |sudo tee /tmp/yourprotectedfilehere
The variable $FOO will *not* be interpreted.
EOF
cat << "EOF" |sudo tee /tmp/yourprotectedfilehere
The variable $FOO *will* be interpreted.
EOF
For future people who may have this issue the following format worked:
(cat <<- _EOF_
LogFile /var/log/clamd.log
LogTime yes
DatabaseDirectory /var/lib/clamav
LocalSocket /tmp/clamd.socket
TCPAddr 127.0.0.1
SelfCheck 1020
ScanPDF yes
_EOF_
) > /etc/clamd.conf
For those looking for a pure bash solution (or a need for speed), here's a simple solution without cat:
# here-doc tab indented
{ read -r -d '' || printf >file '%s' "$REPLY"; } <<-EOF
foo bar
EOF
or for an easy "mycat" function (and avoid leaving REPLY in environment):
mycat() {
local REPLY
read -r -d '' || printf '%s' "$REPLY"
}
mycat >file <<-EOF
foo bar
EOF
Quick speed comparison of "mycat" vs OS cat (1000 loops >/dev/null on my OSX laptop):
mycat:
real 0m1.507s
user 0m0.108s
sys 0m0.488s
OS cat:
real 0m4.082s
user 0m0.716s
sys 0m1.808s
NOTE: mycat doesn't handle file arguments, it just handles the problem "write a heredoc to a file"
As instance you could use it:
First(making ssh connection):
while read pass port user ip files directs; do
sshpass -p$pass scp -o 'StrictHostKeyChecking no' -P $port $files $user#$ip:$directs
done <<____HERE
PASS PORT USER IP FILES DIRECTS
. . . . . .
. . . . . .
. . . . . .
PASS PORT USER IP FILES DIRECTS
____HERE
Second(executing commands):
while read pass port user ip; do
sshpass -p$pass ssh -p $port $user#$ip <<ENDSSH1
COMMAND 1
.
.
.
COMMAND n
ENDSSH1
done <<____HERE
PASS PORT USER IP
. . . .
. . . .
. . . .
PASS PORT USER IP
____HERE
Third(executing commands):
Script=$'
#Your commands
'
while read pass port user ip; do
sshpass -p$pass ssh -o 'StrictHostKeyChecking no' -p $port $user#$ip "$Script"
done <<___HERE
PASS PORT USER IP
. . . .
. . . .
. . . .
PASS PORT USER IP
___HERE
Forth(using variables):
while read pass port user ip fileoutput; do
sshpass -p$pass ssh -o 'StrictHostKeyChecking no' -p $port $user#$ip fileinput=$fileinput 'bash -s'<<ENDSSH1
#Your command > $fileinput
#Your command > $fileinput
ENDSSH1
done <<____HERE
PASS PORT USER IP FILE-OUTPUT
. . . . .
. . . . .
. . . . .
PASS PORT USER IP FILE-OUTPUT
____HERE
If you want to keep the heredoc indented for readability:
$ perl -pe 's/^\s*//' << EOF
line 1
line 2
EOF
The built-in method for supporting indented heredoc in Bash only supports leading tabs, not spaces.
Perl can be replaced with awk to save a few characters, but the Perl one is probably easier to remember if you know basic regular expressions.
In addition, if you're writing to a file, it can be a good idea to check whether or not your write succeeded for failed. For example:
if ! echo "contents" > ./file ; then
echo "ERROR: failed to write to file" >& 2
exit 1
fi
To do the same with heredoc, there are two possible approaches.
1)
if ! cat > ./file << EOF
contents
EOF
then
echo "ERROR: failed to write to file" >& 2
exit 1
fi
if ! cat > ./file ; then
echo "ERROR: failed to write to file" >& 2
exit 1
fi << EOF
contents
EOF
You can test the error case in the above code by replacing the destination file ./file with /file (assuming you're not running as root).
I like the following method of basic redirection for its concision, readability and presentation in an indented script:
<<-End_of_file >file
→ foo bar
End_of_file
Where → is a tab character.
This is standard Bourne shell redirection without forking any cat or tee process.
But it is not working with current bash even when called through /bin/sh.
It is still working with /bin/zsh since more than 20 years.
I know this much:
$ command 2>> error
$ command 1>> output
Is there any way I can output the stderr to the error file and output stdout to the output file in the same line of bash?
Just add them in one line command 2>> error 1>> output
However, note that >> is for appending if the file already has data. Whereas, > will overwrite any existing data in the file.
So, command 2> error 1> output if you do not want to append.
Just for completion's sake, you can write 1> as just > since the default file descriptor is the output. so 1> and > is the same thing.
So, command 2> error 1> output becomes, command 2> error > output
Try this:
your_command 2>stderr.log 1>stdout.log
More information
The numerals 0 through 9 are file descriptors in bash.
0 stands for standard input, 1 stands for standard output, 2 stands for standard error. 3 through 9 are spare for any other temporary usage.
Any file descriptor can be redirected to a file or to another file descriptor using the operator >. You can instead use the operator >> to appends to a file instead of creating an empty one.
Usage:
file_descriptor > filename
file_descriptor > &file_descriptor
Please refer to Advanced Bash-Scripting Guide: Chapter 20. I/O Redirection.
Like that:
$ command >>output 2>>error
Or if you like to mix outputs (stdout & stderr) in one single file you may want to use:
command > merged-output.txt 2>&1
Multiple commands' output can be redirected. This works for either the command line or most usefully in a bash script. The -s directs the password prompt to the screen.
Hereblock cmds stdout/stderr are sent to seperate files and nothing to display.
sudo -s -u username <<'EOF' 2>err 1>out
ls; pwd;
EOF
Hereblock cmds stdout/stderr are sent to a single file and display.
sudo -s -u username <<'EOF' 2>&1 | tee out
ls; pwd;
EOF
Hereblock cmds stdout/stderr are sent to separate files and stdout to display.
sudo -s -u username <<'EOF' 2>err | tee out
ls; pwd;
EOF
Depending on who you are(whoami) and username a password may or may not be required.
Is there a way for a script to log both, the command line run (including piped ones) plus its output without duplicating the line for the command?
The intention is that the script should have a clean output, but should log verbosely into a log file (so no set -x). Apart from the output, it shall also log the command line causing the output, which could be a piped command-one liner.
The most basic approach is to duplicate the command line in the script and then dump it into the log followed by the captured output of the actual command being run:
echo "command argument1 \"quoted argument2\" | grep -oE \"some output\"" >> file.log
output="$(command argument1 "quoted argument2" 2>&1 | grep -oE "some output")"
echo "${output}" >> file.log
This has the side effect that quoted sections would need to be escaped for the log, which can lead to errors resulting in confusion.
If none of the commands were piped, one could store the command line in an array and then "run" the array.
command=(command argument1 "quoted argument2")
echo "${command[#]}" >> file.log
output="$("${command[#]}" 2>&1)"
echo "${output}" >> file.log
Though with this approach "quoted argument2" would become quoted argument2 in the log.
Is there a way (in bash) to realize this without having to duplicate the commands?
You could play with redirections, switch the x option on and off on demand, unset PS4 to get rid of the leading + , and define log_on and log_off functions for easier coding. Something like this:
$ cat script.sh
#!/usr/bin/env bash
function log_on {
exec 3>&1 4>&2
exec &> >( sed -E '/^(set \+x|log_off)$/d' >> file.log )
ps4=$PS4
PS4=
set -x
}
function log_off {
set +x
exec 1>&3 2>&4
PS4=$ps4
}
echo something not logged
log_on
echo something logged
log_off
echo something else not logged
$ rm -f file.log
$ ./script.sh
something not logged
something else not logged
$ cat file.log
echo something logged
something logged
The exec <redirection> commands look a bit cryptic (as most redirections) but they are rather simple:
exec 3>&1 4>&2 makes copies of file descriptors fd1 and fd2 (stdout and stderr by default) to be able to restore these in log_off. After this fd3 and fd4 are copies of fd1 and fd2, respectively. Pick other fd than 3 or 4 if you already use them.
exec &> >( sed ... ) redirect fd1 and fd2 to the standard input of a sed command.
The sed command sed -E '/^(set \+x|log_off)$/d' >> file.log deletes lines containing only set + or log_off and appends its output to file.log. Without this sed command you would always see the two following lines:
log_off
set +x
in your logs, after a group of logged commands.
exec 1>&3 2>&4 restores fd1 and fd2 from their copies in fd3 and fd4.
The rest is straightforward: save PS4 in ps4 such that it can be restored, enable/disable the x option. This should be easy to adapt or extend if needed.
The x option displays the simple commands separately. It breaks pipes, for instance. If you prefer a command log that looks more like the commands you wrote you can replace set -/+x by set -/+v.
IMHO this has already been answered here:
For simplicity the set linux command is what you need.
set -x or set -v
I'm trying to redirect a program's stdin and stdout. I'm currently experimenting with a bash mockup of this, but I'm getting some odd behavior.
I have the following:
mkfifo in
mkfifo out
I also have the following script, test.sh
#!/bin/bash
while read line; do
echo "I read ${line}"
done < /dev/stdin
In terminal 1, I do the following:
tail -f out
In terminal 2, I do the following:
./test.sh < in > out
In terminal 3, I do the following:
echo "foo" > in
echo "bar > in
However, instead of seeing "I read foo" followed by "I read bar" in terminal 1, I get nothing after the first echo, both lines after the second echo, and then the test.sh program in terminal 2 exits. How can I prevent the exit so I can keep sending test.sh input? Also, instead of buffering and then dumping when the program terminates, how can I get the output from test.sh to flush to the tail -f in terminal 1?
Use the redirection on a single compound command that contains your two echo commands.
{
echo "foo"
echo "bar"
} > in
If, as seems likely on a closer reading, you want in to stay open while you are executing commands interactively, use exec to open in on another file descriptor:
exec 3> in # Open in on file descriptor 3
echo "foo" >&3 # Write to file descriptor 3 instead of standard output
echo "bar" >&3 # "
exec 3>&- # Close file descriptor 3
Note that exec 3> in will block until something (test.sh in your case) opens in for reading, and due to buffering, you may not see any output from tail -f out until you close file descriptor 3.
I am trying to log something in the console, but it doesn't work. The output is:
Resolving...
+ /keyboard
+ /keyboard/windows.keylayout
+ /keyboard/hardware.txt
Addition count 3 src: 60.42KB
61868 / 61868 [====================================================] 100.00% 0s
I want that into a .log file.
But it only takes "Resolving...", when I execute "script >> script.log" as the first line, I guess that's because of the "+", how can I fix that, that it will log ALL of the output into that file?
cd /home/andre/test/ && /usr/local/bin/drive_armv7 pull -no-prompt keyboard 2>&1 >> error.log
Thanks.
Order of assignment?
cd /home/andre/test/ && /usr/local/bin/drive_armv7 pull -no-prompt keyboard 2>&1 >> error.log
tells stderr to go where stdout is going, which is the terminal. THEN you tell stdout to go into the log. Switch them -
cd /home/andre/test/ &&
/usr/local/bin/drive_armv7 pull -no-prompt keyboard >> error.log 2>&1
That tells stdout to go into the log, THEN tells stderr to go where stdout is going...which is now the log.
It could be that you are running a default program called script
user#apollo:~$ script test
Script started, file is test
user#apollo:~$ which script
/usr/bin/script
If "script" is a local executable called "script.sh", you should run it as following
./script.sh
Trying to reproduce your issue gives no unexpected output
cat > script.sh <<EOF
echo "Resolving..."
echo "+ /keyboard"
echo "+ /keyboard/windows.keylayout"
echo "+ /keyboard/hardware.txt"
echo "Addition count 3 src: 60.42KB"
echo "61868 / 61868 [====================================================] 100.00% 0s"
EOF
chmod +x script.sh
./script.sh >> script.log
cat script.log
The unexpected behaviour depends on what your script is trying to do