This question already has answers here:
pass stdout as file name for command line util?
(6 answers)
Closed 5 years ago.
Some utilities can not output to stdout.
Example
util out.txt
It works. But sometimes I want to pipe the output to some other program like:
util out.txt | grep test
Does any magic "stdout" file in linux exists, so when I will replace the out.txt above, it will work redirect the data to stdout pipe?
Note: I know util out.txt && cat out.txt | grep test, so please do not post answers like this.
You could use /dev/stdout. But that won't always work if a program needs to lseek(2) (or mmap(2)) it.
Usually /dev/stdout is a symlink to /proc/self/fd/1 (see proc(5)).
IIRC some version of some programs (probably GNU awk) are handling specifically the /dev/stdout filename (e.g. to be able to work without /proc/ being mounted).
A common, but not universal, convention for program arguments is to consider -, when used as a file name, to represent the stdout (or the stdin). For example, see tar(1) used with -f -.
If you write some utility, I recommend following that - convention when possible and document if stdout needs to be seekable.
Some programs are testing if stdout or stdin is a terminal (e.g. using isatty(3)) to behave differently, e.g. by using ncurses. If you write such a program, I recommend providing a program option to disable that detection.
Related
This question already has answers here:
How to 'grep' a continuous stream?
(13 answers)
Closed 6 years ago.
I have a log file A that is constantly updated (but it is rolled over) and I need to constantly filter it's content and write to a persistent file.
TL;DR
I need to:
tail -f A.log | grep "keyword" >> B.log
But this command does not write anything to B.log.
Research only got me complex stuff that is not my case. My guess is that I'm missing some simple concept.
This is not the same question marked as possible duplicate, as the grep works and I have it's output if I don't try to write it to a file. The problem is the file.
If just grep, without writing to the file, works, you encountered a buffering "problem". I/O buffering, unless manually implemented by the program will get handled by the libc. If the program's stdout is a termial, buffering will be line-based. If not, the libc buffers output until the buffer reached a size limit.
On Linux, meaning with glibc you can use the stdbuf command to configure that buffering:
tail -f A.log | stdbuf -oL grep "keyword" >> B.log
-oL specifies that the output stream should be line-buffered.
Grep command syntax is as:
grep "literal_string" filename --> search from string in filename.
So I am assuming the order of is like this
-- keyword(grep) --> string to be searched --> filename/source string and command is interpreted from left to right.
My question is how the commands such as this got processed:
ps -ef | grep rman
Do the order is optional?
How grep is able to know that source is on left and not on right? Or I am missing something here.
When using Unix Pipes, most system commands will take the output from the previous command (to the left of the pipe ) and then pass the output onto the command to the right of the pipe.
The order is important when using grep with or without a pipe.
Thus
grep doberman /file/about/dogs
is the same as
cat /file/about/dogs | grep doberman
See Pipes on http://linuxcommand.org/lts0060.php for some more information.
As step further down from Kyle's answer regarding pipes is that most shell commands read their input from stdin and write their output to stdout. Now many of the commands will also allow you to specify a filename to read from or write too, or allow you to redirect a file to stdin as input and redirect the commands stdout to a file. But regardless how you specify what to read, the command process input from it's stdin and provides output on stdout (errors on stderr). stdin, stdout, and stderr are the designation of file descriptors 0, 1 & 2, respectively.
This basic function is what allows command to be piped together. Where a pipe (represented by the | character) does nothing more that take the stdout from the first command (on left) and direct it to the next commands stdin. As such, yes, the order is important.
Another point to remember is that each piped process is run in its own subshell. Put another way, each | will spawn another shell to run the following command in. This has implications if you are relying on the environment of one process for the next.
Hopefully, these answers will give you a better feel for what is taking place.
This question already has answers here:
How to redirect output to a file and stdout
(11 answers)
Closed 10 years ago.
I am running a script called upgrade.sh
ANd upgrade.sh calls a script called roll.sh
roll.sh >> logfile.text
But roll.sh has some questions and prompts, and the redirect is preventing those outputs from hitting the screen. I cannot edit roll.sh.
I also tried `results=$(roll.sh)
Even then, the output was not coming onto the screen
Use tee, it was created specifically for this purpose: to forward standard input to the screen and one or more files. Make sure to use the -a option to append to logfile.text if you don't want to overwrite it.
roll.sh | tee -a logfile.text
You want tee:
TEE(1) User Commands TEE(1)
NAME
tee - read from standard input and write to standard output and files
A common way to handle that is to have the script write its prompts to stderr instead of stdout.
I would like my code to handle the output coming from pipe.
for example, ls -l | mycode
how to achieve this under Linux?
Just read from stdin, such as with scanf().
The pipe in Linux/Unix will transfer the output of the first program to the standard input of the second. How you access the standard input will depend on what language you are using.
When you type "ls -l | mycode" into the shell, it is the shell program itself (e.g. bash, zsh) that does all the trickery with pipes. It simply provides the output from ls -l to mycode on standard input. Similarly, anything you write on standard output or error can be redirected or piped by the shell to some other process or file. Exactly how to read and write to those files depends on the language.
What exactly are the uses of '-' in bash? I know they can be used for
cd - # to take you to the old 'present working directory'
some stream generating command | vim - # somehow vim gets the text.
My question is what exactly is - in bash? In what other contexts can I use it?
Regards
Arun
That depends on the application.
cd -
returns to the last directory you were in.
Often - stands for stdin or stdout. For example:
xmllint -
does not check an XML file but checks the XML on stdin. Sample:
xmllint - <<EOF
<root/>
EOF
The same is true for cat:
cat -
reads from stdin. A last sample where - stands for stdout:
wget -O- http://google.com
will receive google.com by HTTP and send it on stdout.
By the way: That has nothing to do with your shell (e.g. bash). It's only semantics of the called application.
- in bash has no meaning as a standalone argument (I would not go as far as to say it it does not have a meaning in shell at all - it's for example used in expansion, e.g. ls [0-9]* lists all files starting with a digit).
As far as being a standalone parameter value, bash will do absolutely nothing special with it and pass to a command as-is.
What the command does with it is up to each individual program - can be pretty much anything.
There's a commonly used convention that - argument indicates to a program that the input needs to be read from STDIN instead of a file. Again, this is merely how many programs are coded and technically has nothing to do with bash.
From tldp:
This can be done for instance using a hyphen (-) to indicate that a program should read from a pipe
This explains how your vim example gets its data.
There is no universal rule here.
According to the context it changes
It is pretty much useful when you have something to do repeatedly in two directories. Refer #4 here: http://www.thegeekstuff.com/2008/10/6-awesome-linux-cd-command-hacks-productivity-tip3-for-geeks/
In many places it means STDIN.