I have a program which takes input and standard input until eof is given (CTRL-D on linux). I want to run this program with a lot of default input, then continue entering things until I manually hit CTRL-D to stop it. Is there any way to remove the EOF that a bash pipeline puts in?
IE: cat somedata.dat | <insert answer here> | ./myprogram such that myprogram never receives EOF on stdin.
Bash doesn't actually add an "end-of-file" character; there isn't such a thing. Rather, the problem is that ./myprogram reaches the end of its standard input (which is hooked up to the pipe), so the next time it tries to read a character, it gets end-of-file instead. There's no way to have it suddenly switch over to "stealing" the standard input from the terminal, because it's not hooked up to that input at all.
Instead, to feed more input to ./myprogram than just what's in somedata.dat, you can ask cat to start reading (and forwarding) its own standard input after it's finished reading somedata.dat:
cat somedata.dat - | ./myprogram
or
cat somedata.dat /dev/stdin | ./myprogram
Edited to add (per further question in the comments): If you have a more complicated pipeline feeding into ./myprogram, rather than just a file, then you can run your main command and then cat, piping the whole thing to ./myprogram:
{
reallyConfusingTransform < somedata.dat
cat
} | ./myprogram
or in one line:
{ reallyConfusingTransform < somedata.dat ; cat ; } | ./myprogram
(Note that I've also eliminated a "useless use of cat" (UUOC), but if you really prefer to use cat that way, you can still write cat somedata.dat | reallyConfusingTransform instead of reallyConfusingTransform < somedata.dat.)
Related
I have a printf command that will write a file but won't print to stdout. I would like to have both so I can let the user see what's happening, and at the same time, write a record to a log file.
printf "%s\n" "This is some text" "That will be written to a file" "There will be several lines" | tee -a bin/logfile.log > bin/newfile.conf
That command appends to the log file and writes to the new file, but writes no output to the screen :(
OS: Centos 7
It's because you're redirecting the screen output with > bin/newfile.conf in addition to what you're doing with tee. Just drop the > and everything after it. If you want to output to both of those files at once in addition to the screen, you can use tee twice, e.g.:
printf ... | tee -a bin/logfile.log | tee bin/newfile.conf
That appends to logfile.log and overwrites newfile.conf, and also writes out to the screen. Use or omit the -a option as needed.
As John1024 points out you can also use tee once since it accepts multiple filenames, although in that case -a applies to all filenames, but it can be useful in the case where you want the append vs. overwrite behavior to be the same for all files.
As the image. All commands are similar.
I know how to use that, but I don't really know the detail.
Would anyone know that? Thank you very much.
# does `cat` read fd and print?
$ cat file
# does `cat` read from stdin and print?
$ cat < file
$ cat - < file
# with heredoc or herestring, what methods `cat` command use to read from heredoc?stdin?
$ cat << EOF
heredoc> test
heredoc> EOF
test
$ cat <<< "test"
$ cat - << EOF
heredoc> test
heredoc> EOF
test
$ cat - <<< "test"
# and I dont why these commands works?
$ cat <(echo "test")
$ cat - <<(echo "test")
# why this command doesn't work?
$ cat - <(echo "test")
Some reading material, all from the very useful Bash manual:
Redirection (<filename) -- causes standard input to be redirected to the file filename
Here documents (<<WORD) -- causes standard input to be redirected to the script source from the next line, up to but not including the line WORD
Here strings (<<<"string") -- causes standard input to be redirected to the string string (as though the string were written to a temporary file and then standard input redirected to that file)
Process substitution (<(command)) -- starts a process executing command and inserts a name onto the command line which acts like a filename, such that reading from that "file" produces the output of the command
The use of - to indicate the the source file is standard input is common to many commands and recommended by Posix. Many commands read from standard input if no file is specified. Some, like cat, implement both ways of indicating that the intention is to read from standard input.
Note that - and <(command) are both filename arguments, while <filename, <<WORD and <<<"string" are redirections. So while they superficially look similar, they are quite different under the hood. What they have in common is that they have to do with input; some of them (but not here-docs/strings) have analogues that have to do with output, using > instead of <.
The formal definition of pipe states that the STDOUT of the left file will be immediately piped to the STDIN of the right file.I have two files, hello.txt and human.txt. cat hello.txt returns Hello and cat human.txt returns I am human.Now if I do cat hello.txt | cat human.txt, shouldn't that return Hello I am human?Instead I'm seeing command not found.I am new to shell scripting.Can someone explain?
Background: A pipe arranges for the output of the command on the left (that is, contents written to FD 1, stdout) to be delivered as input to the command on the right (on FD 0, stdin). It does this by connecting the processes with a "pipe", or FIFO, and executing them at the same time; attempts to read from the FIFO will wait until the other process has written something, and attempts to write to the FIFO will wait until the other process is ready to read.
cat hello.txt | cat human.txt
...feeds the content of hello.txt into the stdin of cat human.txt, but cat human.txt isn't reading from its stdin; instead, it's been directed by its command line arguments to read only from human.txt.
Thus, that content on the stdin of cat human.txt is ignored and never read, and cat hello.txt receives a SIGPIPE when cat human.txt exits, and thereafter exits as well.
cat hello.txt | cat - human.txt
...by contrast will have the second cat read first from stdin (you could also use /dev/stdin in place of - on many operating systems, including Linux), then from a file.
You don't need to pipe them rather you can read from multiple file like below which will in-turn concatenate the content of both file content
cat hello.txt human.txt
| generally used when you want to fed output of first command to the second command in pipe. In this case specifically your second command is reading from a file and thus don't need to be piped. If you want to you can do like
echo "Hello" | cat - human.txt
First thing the command will not give an error it will print I m human i.e the contents of human.txt
Yeah you are right about pipe definition , but on the right side of pipe there should be some command.
If the command is for receiving the input and providing the output than it will give you output,otherwise the command will do its own behaviour
But here there is a command i.e cat human.txt on the right side but it will print its own contents and does no operation on the received input .
And also this error comes when when you write like
cat hello.txt | human.txt
bash will give you this error :
human.txt: command not found
In bash, calling foo would display any output from that command on the stdout.
Calling foo > output would redirect any output from that command to the file specified (in this case 'output').
Is there a way to redirect output to a file and have it display on stdout?
The command you want is named tee:
foo | tee output.file
For example, if you only care about stdout:
ls -a | tee output.file
If you want to include stderr, do:
program [arguments...] 2>&1 | tee outfile
2>&1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.
Furthermore, if you want to append to the log file, use tee -a as:
program [arguments...] 2>&1 | tee -a outfile
$ program [arguments...] 2>&1 | tee outfile
2>&1 dumps the stderr and stdout streams.
tee outfile takes the stream it gets and writes it to the screen and to the file "outfile".
This is probably what most people are looking for. The likely situation is some program or script is working hard for a long time and producing a lot of output. The user wants to check it periodically for progress, but also wants the output written to a file.
The problem (especially when mixing stdout and stderr streams) is that there is reliance on the streams being flushed by the program. If, for example, all the writes to stdout are not flushed, but all the writes to stderr are flushed, then they'll end up out of chronological order in the output file and on the screen.
It's also bad if the program only outputs 1 or 2 lines every few minutes to report progress. In such a case, if the output was not flushed by the program, the user wouldn't even see any output on the screen for hours, because none of it would get pushed through the pipe for hours.
Update: The program unbuffer, part of the expect package, will solve the buffering problem. This will cause stdout and stderr to write to the screen and file immediately and keep them in sync when being combined and redirected to tee. E.g.:
$ unbuffer program [arguments...] 2>&1 | tee outfile
Another way that works for me is,
<command> |& tee <outputFile>
as shown in gnu bash manual
Example:
ls |& tee files.txt
If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.
For more information, refer redirection
You can primarily use Zoredache solution, but If you don't want to overwrite the output file you should write tee with -a option as follow :
ls -lR / | tee -a output.file
Something to add ...
The package unbuffer has support issues with some packages under fedora and redhat unix releases.
Setting aside the troubles
Following worked for me
bash myscript.sh 2>&1 | tee output.log
Thank you ScDF & matthew your inputs saved me lot of time..
Using tail -f output should work.
In my case I had the Java process with output logs. The simplest solution to display output logs and redirect them into the file(named logfile here) was:
my_java_process_run_script.sh |& tee logfile
Result was Java process running with output logs displaying and
putting them into the file with name logfile
You can do that for your entire script by using something like that at the beginning of your script :
#!/usr/bin/env bash
test x$1 = x$'\x00' && shift || { set -o pipefail ; ( exec 2>&1 ; $0 $'\x00' "$#" ) | tee mylogfile ; exit $? ; }
# do whaetever you want
This redirect both stderr and stdout outputs to the file called mylogfile and let everything goes to stdout at the same time.
It is used some stupid tricks :
use exec without command to setup redirections,
use tee to duplicates outputs,
restart the script with the wanted redirections,
use a special first parameter (a simple NUL character specified by the $'string' special bash notation) to specify that the script is restarted (no equivalent parameter may be used by your original work),
try to preserve the original exit status when restarting the script using the pipefail option.
Ugly but useful for me in certain situations.
Bonus answer since this use-case brought me here:
In the case where you need to do this as some other user
echo "some output" | sudo -u some_user tee /some/path/some_file
Note that the echo will happen as you and the file write will happen as "some_user" what will NOT work is if you were to run the echo as "some_user" and redirect the output with >> "some_file" because the file redirect will happen as you.
Hint: tee also supports append with the -a flag, if you need to replace a line in a file as another user you could execute sed as the desired user.
< command > |& tee filename # this will create a file "filename" with command status as a content, If a file already exists it will remove existed content and writes the command status.
< command > | tee >> filename # this will append status to the file but it doesn't print the command status on standard_output (screen).
I want to print something by using "echo" on screen and append that echoed data to a file
echo "hi there, Have to print this on screen and append to a file"
tee is perfect for this, but this will also do the job
ls -lr / > output | cat output
I've looked but can't find anything. A program for example, a TTS I use lets you do the following:
~#festival -tts | echo "I am to be spoken"
Which is really nice, but for programs I use such as hexdump, I don't know how to pipe text into it. I can REALLY use some of these things, some examples I tried (but failed) are like so:
~#gtextpad < dmesg
//(how do I put the contents into the text pad for display? not just into a file)
~#hexdump | echo "I am to be put into hexdump"
//(How do I get hexdump to read the echo? It normally reads a file such as foo.txt..)
here are some ways to pass text to hexdump
Stdin:
echo "text" | hexdump
Here document
hexdump <<EOF
text
EOF
to process entire file
hexdump file
On bash, you can also do
hexdump <<< "Pipe this text into hexdump"
The data flow in a pipeline (series of commands separated by pipe symbols) flows from left to right. Thus, the output from command1 below goes to the input of command2, and so on.
command1 |
command2 |
command3 arg1 arg2 arg3 |
sort |
more
So, to get the output of 'echo' into 'hexdump', use:
echo "I am to be dumped" | hexdump
I don't see how the 'festival' command you show can work. The shell does enough plumbing that without making unwarranted assumptions and doing a lot of skulduggery and then still relying on scheduling decisions in the kernel to get things 'right', I don't see how it can be made to work.
From the hexdump(1) man page:
The hexdump utility is a filter which displays the specified files, or the standard input, if no files are specified, in a user specified format.
So:
echo "I am to be put into hexdump" | hexdump