For every recipe in target, make invokes a different subshell for recipe execution .
So , I tried this command strace make all 2>&1 | grep fork,
but did not get any matches for the fork system call. Where I am wrong ?
You don't see calls to fork because the actual system call is clone. You'll see this if you inspect the output of strace. I like to save the strace output in a file and look at it afterwards:
strace -o trace make all
If I have a Makefile that looks like this:
three: one two
cat one two > three
one:
date > one
two:
date > two
Then after running the strace command above, I have:
$ grep clone trace
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6a3570ce50) = 29836
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6a3570ce50) = 29838
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f6a3570ce50) = 29840
From the fork man page:
C library/kernel differences
Since version 2.3.3, rather than invoking the kernel's fork() system
call, the glibc fork() wrapper that is provided as part of the NPTL
threading implementation invokes clone(2) with flags that provide the
same effect as the traditional system call. (A call to fork() is
equivalent to a call to clone(2) specifying flags as just SIGCHLD.)
The glibc wrapper invokes any fork handlers that have been
established using pthread_atfork(3).
Related
I looked at this page and can't understand how this works.
This command "exponentially spawns subprocesses until your box locks up".
But why? What I understand less are the colons.
user#host$ :(){ :|:& };:
:(){ :|:& };:
..defines a function named :, which spawns itself (twice, one pipes into the other), and backgrounds itself.
With line breaks:
:()
{
:|:&
};
:
Renaming the : function to forkbomb:
forkbomb()
{
forkbomb | forkbomb &
};
forkbomb
You can prevent such attacks by using ulimit to limit the number of processes-per-user:
$ ulimit -u 50
$ :(){ :|:& };:
-bash: fork: Resource temporarily unavailable
$
More permanently, you can use /etc/security/limits.conf (on Debian and others, at least), for example:
* hard nproc 50
Of course that means you can only run 50 processes, you may want to increase this depending on what the machine is doing!
That defines a function called : which calls itself twice (Code: : | :). It does that in the background (&). After the ; the function definition is done and the function : gets started.
So every instance of : starts two new : and so on... Like a binary tree of processes...
Written in plain C that is:
fork();
fork();
Just to add to the above answers, the behavior of pipe | is to create two processes at once and connect them with pipe(pipe is implemented by the operating system itself), so when we use pipe, each parent processes spawn two other processes, which leads to utilization of system resource exponentially so that resource is used up faster. Also & is used to background the process and in this case prompts returns immediately so that the next call executes even faster.
Conclusion :
|: To use system resource faster( with exponential growth)
&: background the process to get process started faster
This defines a function called : (:()). Inside the function ({...}), there's a :|:& which is like this:
: calls this : function again.
| signifies piping the output to a command.
: after | means pipe to the function :.
&, in this case, means run the preceding in the background.
Then there's a ; that is known as a command separator.
Finally, the : starts this "chain reaction", activating the fork bomb.
The C equivalent would be:
#include <sys/types.h>
#include <unistd.h>
int main()
{
fork();
fork();
}
All of the literature that I have read so far on setuid talks about seteuid in a way that implies it is a system call. The section 2 man pages never say if a function is a system call or not, so seteuid(2) is no help. And if it isn't a system call, meaning the functionality is not provided by the kernel, then how can "set effective UID" be achieved?
The section 2 man pages are all system calls -- that's what section 2 is for. The section 3 man pages are all library calls, as that's what section 3 is for. See man(1) (the manual page for man itself) for the list of sections and what they are:
1 Executable programs or shell commands
2 System calls (functions provided by the kernel)
3 Library calls (functions within program libraries)
4 Special files (usually found in /dev)
5 File formats and conventions eg /etc/passwd
6 Games
7 Miscellaneous (including macro packages and conventions), e.g.
man(7), groff(7)
8 System administration commands (usually only for root)
9 Kernel routines [Non standard]
You can easily verify if it is a system call or if it is defined in libc by writing a little program and running strace on it. For example,
int main() {
seteuid();
}
gcc -o main main.c
-bash-4.2$ strace ./main 2>&1 | grep set
setresuid(-1, 1, -1) = -1 EPERM (Operation not permitted)
So in this case seteuid is implemented in libc. See here for implementation
Is it possible to capture the arguments to userspace function in a precompiled binary with the linux perf tool ? I couldn't figure this out via the documentation or google ?
If not any other suggestions ?
Thanks...
I don't know how to do it with perf but there other ways. If gdb is suitable then use it. If it is suitable for example because of performance problems then use SystemTap:
1) your precompiled binary has debug information and you can use gdb
just attach to a running process, put breakpoint and possibly add command for it:
break your_function
command
info args
continue
end
2) your precompiled binary does not have debug information and you can use gdb
In this situation you need to know a calling convetion. For example for x64_86
break your_function
command
info register rdi
continue
end
3) your precompiled binary has debug information, you cannot use gdb but can use SystemTap
Then something like this:
sudo stap params.stp -x <PID> 'process("your-process-name").function("your_function")'
> cat params.stp
function trace(extra)
{
printf("params:%s\n", extra)
}
probe $1.call { trace($$parms$$) }
If I use a command like this one:
./program >> a.txt &
, and the program is a long running one then I can only see the output once the program ended. That means I have no way of knowing if the computation is going well until it actually stops computing. I want to be able to read the redirected output on file while the program is running.
This is similar to opening a file, appending to it, then closing it back after every writing. If the file is only closed at the end of the program then no data can be read on it until the program ends. The only redirection I know is similar to closing the file at the end of the program.
You can test it with this little python script. The language doesn't matter. Any program that writes to standard output has the same problem.
l = range(0,100000)
for i in l:
if i%1000==0:
print i
for j in l:
s = i + j
One can run this with:
./python program.py >> a.txt &
Then cat a.txt .. you will only get results once the script is done computing.
From the stdout manual page:
The stream stderr is unbuffered.
The stream stdout is line-buffered
when it points to a terminal.
Partial lines will not appear until
fflush(3) or exit(3) is called, or
a new‐line is printed.
Bottom line: Unless the output is a terminal, your program will have its standard output in fully buffered mode by default. This essentially means that it will output data in large-ish blocks, rather than line-by-line, let alone character-by-character.
Ways to work around this:
Fix your program: If you need real-time output, you need to fix your program. In C you can use fflush(stdout) after each output statement, or setvbuf() to change the buffering mode of the standard output. For Python there is sys.stdout.flush() of even some of the suggestions here.
Use a utility that can record from a PTY, rather than outright stdout redirections. GNU Screen can do this for you:
screen -d -m -L python test.py
would be a start. This will log the output of your program to a file called screenlog.0 (or similar) in your current directory with a default delay of 10 seconds, and you can use screen to connect to the session where your command is running to provide input or terminate it. The delay and the name of the logfile can be changed in a configuration file or manually once you connect to the background session.
EDIT:
On most Linux system there is a third workaround: You can use the LD_PRELOAD variable and a preloaded library to override select functions of the C library and use them to set the stdout buffering mode when those functions are called by your program. This method may work, but it has a number of disadvantages:
It won't work at all on static executables
It's fragile and rather ugly.
It won't work at all with SUID executables - the dynamic loader will refuse to read the LD_PRELOAD variable when loading such executables for security reasons.
It's fragile and rather ugly.
It requires that you find and override a library function that is called by your program after it initially sets the stdout buffering mode and preferably before any output. getenv() is a good choice for many programs, but not all. You may have to override common I/O functions such as printf() or fwrite() - if push comes to shove you may just have to override all functions that control the buffering mode and introduce a special condition for stdout.
It's fragile and rather ugly.
It's hard to ensure that there are no unwelcome side-effects. To do this right you'd have to ensure that only stdout is affected and that your overrides will not crash the rest of the program if e.g. stdout is closed.
Did I mention that it's fragile and rather ugly?
That said, the process it relatively simple. You put in a C file, e.g. linebufferedstdout.c the replacement functions:
#define _GNU_SOURCE
#include <stdlib.h>
#include <stdio.h>
#include <dlfcn.h>
char *getenv(const char *s) {
static char *(*getenv_real)(const char *s) = NULL;
if (getenv_real == NULL) {
getenv_real = dlsym(RTLD_NEXT, "getenv");
setlinebuf(stdout);
}
return getenv_real(s);
}
Then you compile that file as a shared object:
gcc -O2 -o linebufferedstdout.so -fpic -shared linebufferedstdout.c -ldl -lc
Then you set the LD_PRELOAD variable to load it along with your program:
$ LD_PRELOAD=./linebufferedstdout.so python test.py | tee -a test.out
0
1000
2000
3000
4000
If you are lucky, your problem will be solved with no unfortunate side-effects.
You can set the LD_PRELOAD library in the shell, if necessary, or even specify that library system-wide (definitely NOT recommended) in /etc/ld.so.preload.
If you're trying to modify the behavior of an existing program try stdbuf (part of coreutils starting with version 7.5 apparently).
This buffers stdout up to a line:
stdbuf -oL command > output
This disables stdout buffering altogether:
stdbuf -o0 command > output
Have you considered piping to tee?
./program | tee a.txt
However, even tee won't work if "program" doesn't write anything to stdout until it is done. So, the effectiveness depends a lot on how your program behaves.
If the program writes to a file, you can read it while it is being written using tail -f a.txt.
Your problem is that most programs check to see if the output is a terminal or not. If the output is a terminal then output is buffered one line at a time (so each line is output as it is generated) but if the output is not a terminal then the output is buffered in larger chunks (4096 bytes at a time is typical) This behaviour is normal behaviour in the C library (when using printf for example) and also in the C++ library (when using cout for example), so any program written in C or C++ will do this.
Most other scripting languages (like perl, python, etc.) are written in C or C++ and so they have exactly the same buffering behaviour.
The answer above (using LD_PRELOAD) can be made to work on perl or python scripts, since the interpreters are themselves written in C.
The unbuffer command from the expect package does exactly what you are looking for.
$ sudo apt-get install expect
$ unbuffer python program.py | cat -
<watch output immediately show up here>
I looked at this page and can't understand how this works.
This command "exponentially spawns subprocesses until your box locks up".
But why? What I understand less are the colons.
user#host$ :(){ :|:& };:
:(){ :|:& };:
..defines a function named :, which spawns itself (twice, one pipes into the other), and backgrounds itself.
With line breaks:
:()
{
:|:&
};
:
Renaming the : function to forkbomb:
forkbomb()
{
forkbomb | forkbomb &
};
forkbomb
You can prevent such attacks by using ulimit to limit the number of processes-per-user:
$ ulimit -u 50
$ :(){ :|:& };:
-bash: fork: Resource temporarily unavailable
$
More permanently, you can use /etc/security/limits.conf (on Debian and others, at least), for example:
* hard nproc 50
Of course that means you can only run 50 processes, you may want to increase this depending on what the machine is doing!
That defines a function called : which calls itself twice (Code: : | :). It does that in the background (&). After the ; the function definition is done and the function : gets started.
So every instance of : starts two new : and so on... Like a binary tree of processes...
Written in plain C that is:
fork();
fork();
Just to add to the above answers, the behavior of pipe | is to create two processes at once and connect them with pipe(pipe is implemented by the operating system itself), so when we use pipe, each parent processes spawn two other processes, which leads to utilization of system resource exponentially so that resource is used up faster. Also & is used to background the process and in this case prompts returns immediately so that the next call executes even faster.
Conclusion :
|: To use system resource faster( with exponential growth)
&: background the process to get process started faster
This defines a function called : (:()). Inside the function ({...}), there's a :|:& which is like this:
: calls this : function again.
| signifies piping the output to a command.
: after | means pipe to the function :.
&, in this case, means run the preceding in the background.
Then there's a ; that is known as a command separator.
Finally, the : starts this "chain reaction", activating the fork bomb.
The C equivalent would be:
#include <sys/types.h>
#include <unistd.h>
int main()
{
fork();
fork();
}