I'm writing a Bash script to monitor a process and detect when it has crashed. To do this, I am monitoring the /proc directory;
start_my_process;
my_process_id=$!;
until [[ ! -d "/proc/$my_process_pid" ]]; do
# alert the process is dead and restart it...
done
Can I be guaranteed that the process's entry in /proc/ will be created BEFORE Bash finishes executing the command to start the process? Or is it possible that by time my check above is executed, the entry for start_my_process might not yet be created?
EDIT:
In the end I actually went against a custom solution and chose monit which is an excellent watchdog tool.
/proc/<pid> is never created. It is not a real directory.
/proc is a virtual filesystem. When you open one of its "files" and read from its output stream, the data are being provided by the kernel. Since the kernel is also responsible for managing process <pid>, the kernel will tell you that /proc/<pid> directory exists as soon as and for as long as the kernel is keeping track of it.
Since bash won't be able to set $! until the process exists, you are definitely safe checking for the process's virtual directory under /proc after that time.
Related
I am trying to write a script that keeps checking if process "XXX" gets killed and if it does, show error message with text from /proc/XXX_ID/fd/1 and start again.
I am trying to make it work on custom thinclient distro where is no more packages than needed. Linux 4.6.3TS_SMP i686
I am new in scripting in Bash and I can't seem to get it working. I were googling and trying different things last two days and I moved nowhere. What am I doing wrong?
#!/bin/bash
while [ true ] ; do
process_ID="$(pgrep XXX)"
tail -f /proc/${process_ID}/fd/1 > /bin/test2.txt
Everything works till now. Now I need to somehow check if test2.txt is empty or not and if its not, use text from it as error message and start checking again. I tried something like this
if [ -s /bin/test2.txt ]
then
err="$(cat /bin/test2.txt)"
notify-send "${err}"
else
fi
done
How about this:
output_filepath=$(readlink /proc/$pid/fd/1)
while ps $pid > /dev/null
do
sleep 1
done
tail "$output_filepath"
The whole idea only works if the stdout (fd 1) of the process is redirected into a file which can be read by someone else. Such a redirect will result in /proc/<pid>/fd/1 being a symlink. We read and memorize the target of that symlink in the first line.
Then we wait until the ps of the given PID fails which typically means the process has terminated. Then we tail the file at the memorized path.
This approach has several weaknesses. The process (or its caller) could remove, modify, or rename the output file at termination time, or reading it can somehow be impossible (permissions, etc.). The output could be redirected to something not being a file (e.g. a pipe, a socket, etc.), then tailing it won't work.
ps not failing does not necessarily mean that the same process is still there. It's next to impossible that a PID is reused that quickly, but not completely impossible.
I am quite new to shell programming on Linux and in my Linux instance, I am redirecting the stdout and stderr of a program to two files in following manner and run it in background
myprog > run.log 2>> err.log &
This works fine, and I get my desired behavior
Now there is a another background process that monitors the run.log and err.log, and moves them to other file names, if the log files grow beyond a certain threshold.
e.g. mv err.log err[date-time].log
my expectation is that after this file move happens, err.log will be created again by the myprog output redirection and new output will be written to that new file. However, after my log file monitoring process moves the file, err.log or run.log never get created again although myprog continues to run without any issues.
Is this the normal behavior in Linux? If it is, what should I do to get my expected behavior working?
Yes, it is. Unless you first program reopen the files, it will keep writing to the old file, even if you can't access it anymore. In fact, the space used by that removed file will only be available after every process closes it. If reopening it is not possible (ie. you can't change the executable nor restart it), then a solution like http://httpd.apache.org/docs/2.4/programs/rotatelogs.html is your best bet.
It can rotate logs based on filesize or time, and even call a custom script after a rotation.
Example usage:
myprog | rotatelogs logname.log 50M
This way the log will be rotated whenever the size reaches 50 megabytes.
[EDIT: pointed to a newer version of rotatelogs]
If I had to guess, it actually associates the process that is logging with a file descriptor, not a file name. When you rename it, you only change the file name. So the process just keeps logging to the file. Just a guess. If I were tasked with fixing it, I would stop the logging process and restart it at that point to re-associate it with the right file.
Just a guess.
Software with support for log rotation actually has support written in for this rotation. If you look at man logrotate, you'll notice that a typical configuration looks like this:
"/var/log/httpd/access.log" /var/log/httpd/error.log {
rotate 5
mail www#my.org
size 100k
sharedscripts
postrotate
/usr/bin/killall -HUP httpd
endscript
}
...which is to say that it sends a HUP signal to the program whose log has been rotated; that program has a signal handler that reopens its output files.
You can do this in your shell scripts too:
reopen_logs() {
exec >>run.log 2>>err.log
}
trap reopen_logs HUP
...then, after rotating your logs, run kill -HUP pid_of_yourscript; on the next occasion when the script itself is executing a command (since signal handlers only run between foregrounded executables), it will reopen its output to recreate the log file without needing to restart.
While writing a perl script intended to fully automate the setup of virtual machines (Xen pv) I hit a small maybe very simple problem.
Using perl's chroot function I do my things on the guest file system and then I need to get back to my initial real root. How the hell I do that?
Script example:
`mount $disk_image $mount_point`;
chdir($mount_point);
chroot($mount_point);
#[Do my things...]
#<Exit chroot wanted here>
`umount $mount_point`;
#[Post install things...]
I've tried exit; but obviously that exit the whole script.
Searching for a way to exit the chroot I've found a number of scripts who aim to exit an already setup chroot (privilege escalation). Since I do the chroot here theses methods do not aplies.
Tried some crazy things like:
opendir REAL_ROOT, "/";
chdir($mount_point);
chroot($mount_point);
chdir(*REAL_ROOT);
But no go.
UPDATE
Some points to consider:
I can't split the script in multiple files. (Silly reasons, but really, I can't)
The chrooted part involve using a lot of data gathered earlier by the script (before the chroot), enforcing the need of not lunching another script inside the chroot.
Using open, system or backticks is not good, I need to run commands and based on the output (not the exit code, the actual output) do other things.
Steps after the chroot depends on what was done inside the chroot, hence I need to have all the variables I defined or changed while inside, outside.
Fork is possible, but I don't know a good way to handle correctly the passing of informations from and to the child.
The chrooted process() cannot "unchroot" itself by exiting (which would just exit).
You have to spawn a children process, which will chroot.
Something along the lines of the following should do the trick:
if (fork())
{
# parent
wait;
}
else
{
# children
chroot("/path/to/somewhere/");
# do some Perl stuff inside the chroot...
exit;
}
# The parent can continue it's stuff after his chrooted children did some others stuff...
It stills lacks of some error checking thought.
You can't undo a chroot() on a process - that's the whole point of the system call.
You need a second process (a child process) to do the work in the chrooted environment. Fork, and have the child undergo the chroot and do its stuff and exit, leaving the parent to do the cleanup.
Try spawning a child process that does the chroot, e.g. with system or fork depending on your needs, and waiting for the child to return the main program continues.
This looks like it might be promising:
Breaking Out of a Chroot Jail Using PERL
Save the original root as the current working directory or as a file descriptor:
chdir "/";
chroot "/mnt";
# Do something
chroot ".";
OR
open DIR, "<", "/";
chroot "/mnt";
# Do something
chdir DIR;
chroot ".";
close DIR;
I have started a service daemon , by running the binary(written in C++) through script file stored rc5.d .
But I am not sure how to capture the pid of the daemon process and store it in pid file in /var/run/.pid . So that I can use the pid for termination.
How can I do this?
Try using start-stop-daemon(8) with the --pidfile argument in your init script. Have your program write its PID to a specified location (usually determined in a configuration file).
What you have to look out for is stale PID files, for instance, if a lock file persisted across a reboot. That logic is best implemented in the init script itself, hence the --exec option to start-stop-daemon.
E.g, if /var/run/foo.pid is 1234, and /proc/1234/exe isn't your service, the lock file is stale and should be quietly removed, allowing the service to start normally.
As far as your application goes, just make sure the location of the lockfile is configurable, and some means exists to tell the init script where to put it.
For instance: (sample: /etc/default/foo) :
PIDFILE=/var/run/foo.pid
OTHEROPTION=foo
Then in /etc/init.d/foo :
[ -f /etc/default/foo ] && . /etc/default/foo
Again, other than writing to the file consistently, all of this logic should be handled outside of your application.
If you know the port the program has open, use fuser command to determine the pid.
You could go about more than one way:
In your program use getpid to write it to a configurable file (perhaps looking in ENV)
Use $! after starting the program (this doesn't work for me on archlinux though :-?)
After starting the program, use pidof
I think my server has been compromised and it has many perl processes running. However, I don't know what file they are being launched from so I can delete it. How can I find this information?
If your system has been hacked, you cannot trust any of the software, not even the kernel. Format the disk and re-install everything. There is just no way to be sure you've cleaned out the infection, because you can't trust the very tools you would use to clean things. You can't copy new tools onto the box, because you can't trust the SSH daemon or the /bin/cp command. Anything -- ls, vi, ps, cat, dd, etc. -- could have been replaced with a trojan that works to hide the infected files.
You could check the symbolic link /proc/pid/cwd, also check the ppid from ps(1).
The first thing I would do is look at the parent process id (PPID). That said, if the PPID is 1, that doesn't tell you anything.
Auditing the filesystem could help see here
pstree could also help
If you run the command "ps -ef" you should get a list of all processes running on your machine. Each process will have a process id number (PID), and also a parent PID. Find the offending process(es) and check their parent PIDs. Then find the process with a matching PID, and it should be your culprit.
Try ls -l /proc/<pid>/exe, or ls -l /proc/<pid>/fd. I don't remember if perl keeps the script file open after the program starts, but if it does, it will be one of the process's file descriptors.
But if your system is pwned, don't expect anything to make sense.