Can somebody explain what actually happens internally(the system calls called) for the command ls | grep 'xxx' ?
First, pipe(2,3p) is called in order to create the pipe with read and write ends. fork(2,3p) is then called twice, once for each command. Then dup2(2,3p) is used to replace the appropriate file descriptor in each forked child with each end of the pipe. Finally exec(3) is called in each child to actually run the commands.
The standard output of the first command is fed as standard input to the second command in the pipeline. There are a couple of system calls that you may be interested to understand what is happening in more detail, in particular, fork(2), execve(2), pipe(2), dup2(2), read(2) and write(2).
In effect the shell arranges STDIN_FILENO and STDOUT_FILENO to be the read end and the write end of the pipe respectively. When the first process in the pipeline performs a write(2) the standard output of that process is duplicated to be the write end of the pipe, similarly when the second process does a read(2) on the standard input it ends up reading from the read end of the pipe.
There are of course more details to be considered, please check out a book such as Advanced programming in the UNIX environment by Richard Stevens.
Related
I have a Bash script that gets invoked like this:
script.sh < input_file.txt
All script.sh does is run some other program:
#!/bin/bash
otherprogram
Now when "otherprogram" reads from stdin, it gets text from input_file.txt, without any need to explicitly redirect the standard input of script.sh into otherprogram.
I don't know a lot about how processes get started, but I have read that when fork() gets called, all file descriptors from the parent process, including stdin, are shared with the child--which makes sense, since fork() just makes an identical copy of everything in the parent process' memory. But why would all file descriptions still be shared after the child process replaces the copy of the parent with a new program (presumably by calling exec...())?
If child processes do always inherit all file descriptors from their parent, can someone explain why that actually makes sense and is a good idea?
When a fork is called () Most fields of the PCB (process control block) are copied from the original to the newly created PCB (and so also the open files, as you said).
To summarize, immediately after executing a fork:
There are 2 processes that are exactly the same, except for the differences described in the fork(2) man page if you want to have a look.
Both processes are at the same line of code (the line immediately after the fork).
In the child process, the return value of the fork is 0.
In the parent process, the return value of the fork is greater than 0.
Let's move to the exec:
So we now have two copies of the shell. But they are still both running the shell program; we want the child to run any program. The child uses exec, to replace itself with the the program you passes as argument. Exec does not create a new process; it just changes the program file that an existing process is running.
So exec first wipes out the memory state of the calling process. It then goes to the filesystem to find the program file requested and copies this file into the program's memory and initializes register state, including the PC.
exec doesn't alter most of the other fields in the PCB - this is important, because it means the process calling exec can set things up if it wants to, for example changing the open files as in your case where the child copy inherit the the file descriptor of the stdin which is pointing to your input file.
Another example can be:
You want that the child process when it prints on the standard output (2) for example by means of an echo, it actually prints on a file. What you can do is before calling the exec (father process) changing the place where the file descriptor 2 points to (using for example dup2()) and then call the fork() ad the exec in the child process.
In Unix and related OSes, the general way to launch a program is:
The process forks.
The child process makes whatever changes to the environment are needed, such as (perhaps):
Changing where file descriptors point (especially stdin/stdout/stderr).
Changing the running user.
Changing the present working directory.
Changing/adding/removing environment variables. (Though you can also just do this as part of the next step.)
The child process then replaces itself with the desired program.
This approach allows for a huge amount of flexibility; the parent program can decide exactly what should and should not be inherited. (It also fits well with the Unix philosophy, of doing one thing and doing it well: exec doesn't need 50 bajillion arguments for different things you might want the new program to have, because you just set those up in the process before switching over to that program.)
It also makes it very easy to write "wrapper" programs that delegate almost of all their functionality to other programs. For example, a script like this:
#!/bin/bash
cd /directory/that/foo/requires/
foo "$#"
which is a drop-in replacement for foo that just changes the directory it's run in, doesn't have to worry about all the things that foo should inherit, only the one thing that it shouldn't. (Which, again, fits very well with the Unix philosophy.)
What is the most straightforward way to create a "virtual" file in Linux, that would allow the read operation on it, always returning the output of some particular command (run everytime the file is being read from)? So, every read operation would cause an execution of a command, catching its output and passing it as a "content" of the file.
There is no way to create such so called "virtual file". On the other hand, you would be
able to achieve this behaviour by implementing simple synthetic filesystem in userspace via FUSE. Moreover you don't have to use c, there
are bindings even for scripting languages such as python.
Edit: And chances are that something like this already exists: see for example scriptfs.
This is a great answer I copied below.
Basically, named pipes let you do this in scripting, and Fuse let's you do it easily in Python.
You may be looking for a named pipe.
mkfifo f
{
echo 'V cebqhpr bhgchg.'
sleep 2
echo 'Urer vf zber bhgchg.'
} >f
rot13 < f
Writing to the pipe doesn't start the listening program. If you want to process input in a loop, you need to keep a listening program running.
while true; do rot13 <f >decoded-output-$(date +%s.%N); done
Note that all data written to the pipe is merged, even if there are multiple processes writing. If multiple processes are reading, only one gets the data. So a pipe may not be suitable for concurrent situations.
A named socket can handle concurrent connections, but this is beyond the capabilities for basic shell scripts.
At the most complex end of the scale are custom filesystems, which lets you design and mount a filesystem where each open, write, etc., triggers a function in a program. The minimum investment is tens of lines of nontrivial coding, for example in Python. If you only want to execute commands when reading files, you can use scriptfs or fuseflt.
No one mentioned this but if you can choose the path to the file you can use the standard input /dev/stdin.
Everytime the cat program runs, it ends up reading the output of the program writing to the pipe which is simply echo my input here:
for i in 1 2 3; do
echo my input | cat /dev/stdin
done
outputs:
my input
my input
my input
I'm afraid this is not easily possible. When a process reads from a file, it uses system calls like open, fstat, read. You would need to intercept these calls and output something different from what they would return. This would require writing some sort of kernel module, and even then it may turn out to be impossible.
However, if you simply need to trigger something whenever a certain file is accessed, you could play with inotifywait:
#!/bin/bash
while inotifywait -qq -e access /path/to/file; do
echo "$(date +%s)" >> /tmp/access.txt
done
Run this as a background process, and you will get an entry in /tmp/access.txt each time your file is being read.
I have come across strange scenario where when I am trying to redirect stdout logs of perl script into a log file, all the logs are getting written at the end of execution when script completes instead of during execution of the script.
While running script when I do tail -f "filename", I could able to see log only when script has completed its execution not during execution of script.
My script details are given below:
/root/Application/download_mornings.pl >> "/var/log/file_manage/file_manage-$(date +\%Y-\%m-\%d).txt"
But when I run without redirecting log file, I can see logs on command prompt as when script progresses.
Let me know if you need any other details.
Thanks in advance for any light that you all might be able shed whats going on.
Santosh
Perl would buffer the output by default. You can say:
$| = 1;
(at the beginning of the script) to disable buffering. Quoting perldoc perlvar:
$|
If set to nonzero, forces a flush right away and after every write or
print on the currently selected output channel. Default is 0
(regardless of whether the channel is really buffered by the system or
not; $| tells you only whether you've asked Perl explicitly to flush
after each write). STDOUT will typically be line buffered if output is
to the terminal and block buffered otherwise. Setting this variable is
useful primarily when you are outputting to a pipe or socket, such as
when you are running a Perl program under rsh and want to see the
output as it's happening. This has no effect on input buffering. See
getc for that. See select on how to select the output channel. See
also IO::Handle.
You might also want to refer to Suffering from Buffering?.
Whenever I need to limit shell command output, I use less to paginate the results:
cat file_with_long_content | less
which works fine and dandy, but what I'm curious about is, less still works even if the output is never ending, consider having the following script in inf.sh file:
while true; do date; done
then I run
sh inf.sh | less
And it's still able to again paginate the results, so is it correct to say that pipe streams the result rather than waiting for the command to finish before outputting the result?
Yes, when you run sh inf.sh | less the two commands are run in parallel. Data written into the pipe by the first process is buffered (by the kernel) until it is read by the second. If the buffer is full (i.e., if the first command writes to the pipe faster than the second can read) then the next write operation will block until further space is available. A similar condition occurs when reading from an empty pipe: if the pipe buffer is empty but the input end is still open, a read will block for more data.
See the pipe(7) manual for details.
It is correct. Pipes are streams.
You can code your own version of the less tool in very few lines of C code. Take the time to do it, including a short research on files and pipes, and you'll emerge with the understanding to answer your own question and more :).
When I do this:
find . -name "pattern" | grep "another-pattern"
Are the processes, find and grep, spawned together? My guess is yes. If so, then how does this work?:
yes | command_that_prompts_for_confirmations
If yes is continuously sending 'y' to stdout and command_that_prompts_for_confirmations reads 'y' whenever it's reading its stdin, how does yes know when to terminate? Because if I run yes alone without piping its output to some other command, it never ends.
But if piping commands don't spawn all the processes simultaneously, then how does yes know how many 'y's to output? It's catch-22 for me here. Can someone explain me how this piping works in *NIX?
From the wikipedia page: "By itself, the yes command outputs 'y' or whatever is specified as an argument, followed by a newline, until stopped by the user or otherwise killed; when piped into a command, it will continue until the pipe breaks (i.e., the program completes its execution)."
yes does not "know" when to terminate. However, at some point outputting "y" to stdout will cause an error because the other process has finished, that will cause a broken pipe, and yes will terminate.
The sequence is:
other program terminates
operating system closes pipe
yes tries to output character
error happens (broken pipe)
yes terminates
Yes, (generally speaking) all of the processes in a pipeline are spawned together. With regard to yes and similar situations, a signal is passed back up the pipeline to indicate that it is no longer accepting input. Specifically: SIGPIPE, details here and here. Lots more fun infomation on *nix pipelining is available on wikipedia.
You can see the SIGPIPE in action if you interrupt a command that doesn't expect it, as you get a Broken Pipe error. I can't seem to find an example that does it off the top of my head on my Ubuntu setup though.
Other answers have covered termination. The other facet is that yes will only output a limited number of y's - there is a buffer in the pipe, and once that is full yes will block in its write request. Thus yes doesn't consume infinite CPU time.
The stdout of the first process is connected to the stdin of the second process, and so on down the line. "yes" exits when the second process is done because it no longer has a stdout to write to.