Does running something like below cause the textfile lines to be directed to the STDIN of program.sh?
cat textfile | program.sh
Yes; and the rest of this answer comes to satisfy SO's requirement of minimum 30 characters per answer (excluding links).
http://en.wikipedia.org/wiki/Pipeline_(Unix)
Yes. You're writing the stdout from cat to the stdin of program.sh. Because cat isn't doing much except reading the file, you can also write it as:
program.sh < textfile
...which does the same thing.
From a technical standpoint, stdin is accessed through file descriptor 0, while stdout is file descriptor 1 and stderr is file descriptor 2. With this information, you can make more complicated redirections, such as redirecting stderr to the same place (or a different place!) than stdout. For a cheat sheet about redirections, see Peteris Krumins's Bash Redirections Cheat Sheet.
Yes.
You are running the command sort on a text file. The output goes to program.sh
Related
When running dos2unix on a file I get the following printed to the terminal
dos2unix: converting file <filename> to UNIX format ...
In my attempt to suppress the output by sending it to /dev/null I noticed that this is sent out on stderr instead of stdout as I'd expected (since it seems like a normal message, not an error). Is there a reason for this?
In Unix-like environments it is common to chain processes: The result of one program is used as input for another program. Mixing results with diagnostics would confuse the next processing stage. It would also hide the diagnostics from a potential user watching the terminal, where processing results piped to the next program don't show.
This is the reason for the separation of results and diagnostics in stdout and stderr. Diagnostics are not restricted to errors but should contain everything which is not a processing result which subsequent programs would expect.
With respect to the actual question: dos2unix is often used to transform files in-place but can also output to stdout (when called without a file name, it reads from stdin and outputs to stdout). stdout then can be redirected independently from stderr. Consider cat blados | dos2unix > blaunix. You would still see the diagnostics (which may contain error messages!), but the result of the processing will go to blaunix.
It's not so common to print diagnostics at all in case of success — probably a little nod to DOS users. It would be pretty bad if the processing result contained the informational message; for example, it would break a C file.
There is no reason, but normally stderr is not just for error output. It is another stream that is often used for logging or informational messages. Since a log message is not output, it is not sent to stdout, which is for the results of the program.
The reason its printed on your terminal is a consequence of your shell, and not really controlled by the application.
Try dos2unix -q <filename>
-q, --quiet
Quiet mode. Suppress all warnings and messages. The return value is zero. Except when wrong command-line options are used.
Simply because that's the way it was implemented...
If you'll check out the source-code you'll see:
...
if (!pFlag->Quiet)
fprintf(stderr, _("dos2unix: converting file %s to file %s in UNIX format ...\n"), argv[ArgIdx-1], argv[ArgIdx]);
...
I use this one-liner to redirect stderr to stdout, skip the resulting first irrelevant line, and send the rest back into stderr.
dos2unix thefile 2>&1|tail -n+2 1>&2
How translate this:
echo "test" | tee -a test1 test2
To a pure UNIX descriptor redirection solution (oneliner better and no PIPES).
Is it possible?
If you want a byte written to one file descriptor (pipe, socket etc.) to show up as readable data on more than one file descriptor which are not dup()s of each other (but e.g. they correspond to two different regular files), then it's not possible on a generic Unix system. Even if the two file descriptors are dup()s, after reading the byte from one of them it would make the byte disappear from the other one, so it can't be read twice.
If you want to do it in Bash without using a |, then it's not possible.
If you want to do it in Zsh without using a |, then just follow chepner's comment: do setopt multios, and then echo test >>test1 >>test2. In the background Zsh will create a helper process to do the copying equivalent to what tee -a does.
I am creating a NASM assembly code to read 2d array of numbers present in file from stdin
i am running the executable like this -> ./abc < input.txt .
and after that i will display the read 2d array on terminal then i want to get keys codes of arrow keys (which normal appear in terminal as special characters) i wrote code for it but its not working. ( I did echo off in termios setting for that)
Although it was working when i am taking file name as an argument & reading and not from stdin but using fopen with proper fd.
./abc abc.txt
in this case after displaying the read 2d array i am able to get arrow keys codes in program but not in earlier case.
Please help me in this matter.
By using input redirection you disconnect stdin from your terminal and instead connect it to a pipe that your shell is reading the file into.
You could use cat input.txt - | ./abc, but you would have to pres Enter to flush the line buffer and make cat pipe the current line into your program.
I would suggest not messing with stdin and just taking the input file as an argument, like you already did before.
I'm trying to empty a file in linux while in use, it's a log file so it is continuosly written.
Right now I've used:
echo -n > filename
or
cat /dev/null > filename
but all of this produce an empty file with a newline character (or strange character that I can see as ^#^#^#^#^#^#^#^#^#^#^#^.. on vi) and I have to remove manually with vi and dd the first line and then save.
If I don't use vi adn dd I'm not able to manipulate file with grep but I need an automatic procedure that i can write in a shell script.
Ideas?
This should be enough to empty a file:
> file
However, the other methods you said you tried should also work. If you're seeing weird characters, then they are being written to the file by something else - most probably whatever process is logging there.
What's going on is fairly simple: you are emptying out the file.
Why is it full of ^#s, then, you ask? Well, in a very real sense, it is not. It does not contain those weird characters. It has a "hole".
The program that is writing to the file is writing a file that was opened with O_WRONLY (or perhaps O_RDWR) but not O_APPEND. This program has written, say, 65536 bytes into the file at the point when you empty out the file with cp /dev/null filename or : > filename or some similar command.
Now the program goes to write another chunk of data (say, 4096 or 8192 bytes). Where will that data be written? The answer is: "at the current seek offset on the underlying file descriptor". If the program used O_APPEND the write would be, in effect, preceded by an lseek call that did a "seek to current end-of-file, i.e., current length of file". When you truncate the file that "current end of file" would become zero (the file becoming empty) so the seek would move the write offset to position 0 and the write would go there. But the program did not use O_APPEND, so there is no pre-write "reposition" operation, and the data bytes are written at the current offset (which, again, we've claimed to be 65536 above).
You now have a file that has no data in byte offsets 0 through 65535 inclusive, followed by some data in byte offsets 65536 through 73727 (assuming the write writes 8192 bytes). That "missing" data is the "hole" in the file. When some other program goes to read the file, the OS pretends there is data there: all-zero-byte data.
If the program doing the write operations does not do them on block boundaries, the OS will in fact allocate some extra data (to fit the write into whole blocks) and zero it out. Those zero bytes are not part of the "hole" (they're real zero bytes in the file) but to ordinary programs that do not peek behind the curtain at the Wizard of Oz, the "hole" zero-bytes and the "non-hole" zero bytes are indistinguishable.
What you need to do is to modify the program to use O_APPEND, or to use library routines like syslog that know how to cooperate with log-rotation operations, or perhaps both.
[Edit to add: not sure why this suddenly showed up on the front page and I answered a question from 2011...]
Another way is the following:
cp /dev/null the_file
The advantage of this technique is that it is a single command, so in case it needs sudo access only one sudo call is required.
Why not just :>filename?
(: is a bash builtin having the same effect as /bin/true, and both commands don't echo anything)
Proof that it works:
fg#erwin ~ $ du t.txt
4 t.txt
fg#erwin ~ $ :>t.txt
fg#erwin ~ $ du t.txt
0 t.txt
If it's a log file then the proper way to do this is to use logrotate. As you mentioned doing it manually does not work.
I have not a linux shell here to try ir, but have you try this?
echo "" > file
I just want to know the difference between:
sort < output
and
sort output
in Linux. How does it work exactly?
This has been discussed on unix.stackexchange here: Performance difference between stdin and command line argument
In sort < file the shell performs redirection. It opens the file and passes the stdin file descriptor to the sort command which reads it.
In sort file, the sort command opens the file and then reads it.
sort < output is telling the shell to use the contents of the file output and dump it to standard in for the command sort.
sort output is telling the command sort to use the file output on disk as it's source.
Many unix commands will accept either standard in or a file as input. The acceptance of standard in allows easier chaining of commands, often for things like ps aux | grep "my process" | sort. (List all processes, filter by "my process", sort lines).
With sort < input the shell will run the sort command, and attach its input to the file 'input'.
With sort input the shell will run the sort command, and give it as parameter the string input. The sort command will then open the file to read it- content.
Effectively there is no difference.
sort < output uses a feature of the shell called file redirection (see e.g. here)
The shell opens tile file output and attaches that open file as stdin to the sort program.
sort output gives the output filename as an command line argument to sort.
sort, as many utilities that takes a filename as an argument, will try to read input from stdin if you do not give it a filename as an argument, such as in the first case here. In both cases, sort will read the content of the output file, sort it, and write the result to stdout.