Consider the following, observed with bash 4.4 on a Linux 3.19 kernel:
# in reality, this may access files "nobody" isn't allowed
get_a_secret() { printf '%s\n' "This is a secret"; }
# attach a process substitution reading the secret to FD 10
exec 10< <(get_a_secret)
# run a less-privileged program that needs the secret, passing it the file descriptor.
chpst -u nobody:nobody -- cat /dev/fd/10
...or the shorter/simpler:
chpst -u nobody:nobody -- cat <(get_a_secret)
Either fails in a manner akin to the following:
cat: /proc/self/fd/10: Permission denied
So, two branches to this question:
What's going on here?
Is there a way to get the desired behavior (passing the ability to read the secret through to the single child process being invoked in a way that doesn't persistently expose that secret to other processes running as "nobody") without exposing the FIFO's output to other processes?
(Yes, I'm well aware that I need to lock down ptrace and /proc/*/mem to prevent another process running as "nobody" from pulling the secret out of the client as it's being read; that said, that's (1) something I can do, and (2) when the process is only run before any potentially-attacker-controlled executables are invoked, less exposure than allowing any process running as nobody to pull the secret out of /proc/*/environ for the full duration of that process).
The following workaround avoids this issue:
exec 10< <(get_a_secret)
chpst -u nobody:nobody -- sh -c 'cat <&10'
Note the redirection being written as <&10 -- not </dev/fd/10 or </proc/self/fd/10 (on platforms which provide /dev/fd -- on platforms without this facility, bash rewrites it into a fdup2() call).
An answer with an explanation of the behavior (and perhaps a workaround that allows programs that don't accept a FD number as input to act on the read side?) would be in a position to supercede this one. :)
I want to be able to reboot a bunch of servers all at the same time (in a bash script).
Currently, what I do is something like that:
function reboot_servers() {
echo "Rebooting servers..."
for server in "${servers[#]}"
do
sshpass -p 'password' ssh -o StrictHostKeyChecking=no root#$server 'reboot'
done
}
(servers is an array of 4 servers, sometimes 8, and in the future probably more)
Now, I am aware that in theory I cannot really have them rebooted all at the exact same time, but I'd like it to be as simultaneously as possible, and the above solution is far from optimal for me.
In my current script, if every iteration takes (say) few hundreds milliseconds in average (the ssh login sometimes lags and is unpredictable), the time passed from when the first server launches the reboot command until the last one does could amount to seconds, which is completely ineffective.
I should also mention that the clocks in all the servers are synced, and also to give you some context, the above function is being run over and over again in something similar to this;
function main() {
iteration=0
while true
do
echo "------> Iteration $((++iteration)) <------"
wait_random_time
reboot_servers
wait_for_servers
if bug_reproduced
then
echo "Bug was reproduced."
exit 0
else
echo "No reproduction, trying again..."
fi
done
}
I read a little bit about the at command, but I'm not sure how to use it for my benefit here.
I would recommend using parallel-ssh.
I end up using pdsh, which gave quite impressive results...
$> pdsh -l root -w server0[0-3] date "+%T.%3N"
server00: 12:29:45.845
server01: 12:29:45.830
server02: 12:29:45.870
server03: 12:29:45.893
Writing a small bash script.
I have a command that spawns a process then returns, leaving the spawned process running. I need to wait for the spawned process to terminate, then run some commands. How can I do this?
The specific case is:
VBoxManage startvm "my_vm"
#when my_vm closes
do_things
However, I've encountered this issue before in other contexts, so if possible I'm looking for a general solution, rather than just one relating to virtualbox vm's.
I have an answer, and it's not pretty
tail --pid=$pid -f /dev/null
In the context of a VBox vm, the following heroic one-liner has proved successful.
VBoxManage startvm "my_vm"; tail --pid=$(awk '/Process ID:/ {print $4;}' /path_to/my_vm/Logs/VBox.log) -f /dev/null; echo hello
Running this, I was able to see that 'hello' was not output until after my_vm had shut down.
Good grief, there needs to be a better way of doing that than a side effect of an option of a command out of left-field. Any better answers? Please....
Unfortunately, I do not think you can generalize VirtualBox (or any process that spawns processes in the background) to one-size-fits-all solution.
Here is a VirtualBox specific answer.
As you noticed VBox runs the machine (and a bunch other things) in the background. But you can query VBoxManage for what is running:
vm_name="my_vm"
VBoxManage startvm $vm_name
while [[ `VBoxManage list runningvms | grep $vm_name` ]]; do
echo Sleeping ...
sleep 5
done
echo Done!
do_things
Hopefully you can tune this to your specific needs.
Services default to starting as root at boot time on my RHEL box. If I recall correctly, the same is true for other Linux distros which use the init scripts in /etc/init.d.
What do you think is the best way to instead have the processes run as a (static) user of my choosing?
The only method I'd arrived at was to use something like:
su my_user -c 'daemon my_cmd &>/dev/null &'
But this seems a bit untidy...
Is there some bit of magic tucked away that provides an easy mechanism to automatically start services as other, non-root users?
EDIT: I should have said that the processes I'm starting in this instance are either Python scripts or Java programs. I'd rather not write a native wrapper around them, so unfortunately I'm unable to call setuid() as Black suggests.
On Debian we use the start-stop-daemon utility, which handles pid-files, changing the user, putting the daemon into background and much more.
I'm not familiar with RedHat, but the daemon utility that you are already using (which is defined in /etc/init.d/functions, btw.) is mentioned everywhere as the equivalent to start-stop-daemon, so either it can also change the uid of your program, or the way you do it is already the correct one.
If you look around the net, there are several ready-made wrappers that you can use. Some may even be already packaged in RedHat. Have a look at daemonize, for example.
After looking at all the suggestions here, I've discovered a few things which I hope will be useful to others in my position:
hop is right to point me back
at /etc/init.d/functions: the
daemon function already allows you
to set an alternate user:
daemon --user=my_user my_cmd &>/dev/null &
This is implemented by wrapping the
process invocation with runuser -
more on this later.
Jonathan Leffler is right:
there is setuid in Python:
import os
os.setuid(501) # UID of my_user is 501
I still don't think you can setuid
from inside a JVM, however.
Neither su nor runuser
gracefully handle the case where you
ask to run a command as the user you
already are. E.g.:
[my_user#my_host]$ id
uid=500(my_user) gid=500(my_user) groups=500(my_user)
[my_user#my_host]$ su my_user -c "id"
Password: # don't want to be prompted!
uid=500(my_user) gid=500(my_user) groups=500(my_user)
To workaround that behaviour of su and runuser, I've changed my init script to something like:
if [[ "$USER" == "my_user" ]]
then
daemon my_cmd &>/dev/null &
else
daemon --user=my_user my_cmd &>/dev/null &
fi
Thanks all for your help!
Some daemons (e.g. apache) do this by themselves by calling setuid()
You could use the setuid-file flag to run the process as a different user.
Of course, the solution you mentioned works as well.
If you intend to write your own daemon, then I recommend calling setuid().
This way, your process can
Make use of its root privileges (e.g. open log files, create pid files).
Drop its root privileges at a certain point during startup.
Just to add some other things to watch out for:
Sudo in a init.d script is no good since it needs a tty ("sudo: sorry, you must have a tty to run sudo")
If you are daemonizing a java application, you might want to consider Java Service Wrapper (which provides a mechanism for setting the user id)
Another alternative could be su --session-command=[cmd] [user]
on a CENTOS (Red Hat) virtual machine for svn server:
edited /etc/init.d/svnserver
to change the pid to something that svn can write:
pidfile=${PIDFILE-/home/svn/run/svnserve.pid}
and added option --user=svn:
daemon --pidfile=${pidfile} --user=svn $exec $args
The original pidfile was /var/run/svnserve.pid. The daemon did not start becaseu only root could write there.
These all work:
/etc/init.d/svnserve start
/etc/init.d/svnserve stop
/etc/init.d/svnserve restart
Some things to watch out for:
As you mentioned, su will prompt for a password if you are already the target user
Similarly, setuid(2) will fail if you are already the target user (on some OSs)
setuid(2) does not install privileges or resource controls defined in /etc/limits.conf (Linux) or /etc/user_attr (Solaris)
If you go the setgid(2)/setuid(2) route, don't forget to call initgroups(3) -- more on this here
I generally use /sbin/su to switch to the appropriate user before starting daemons.
Why not try the following in the init script:
setuid $USER application_name
It worked for me.
I needed to run a Spring .jar application as a service, and found a simple way to run this as a specific user:
I changed the owner and group of my jar file to the user I wanted to run as.
Then symlinked this jar in init.d and started the service.
So:
#chown myuser:myuser /var/lib/jenkins/workspace/springApp/target/springApp-1.0.jar
#ln -s /var/lib/jenkins/workspace/springApp/target/springApp-1.0.jar /etc/init.d/springApp
#service springApp start
#ps aux | grep java
myuser 9970 5.0 9.9 4071348 386132 ? Sl 09:38 0:21 /bin/java -Dsun.misc.URLClassPath.disableJarChecking=true -jar /var/lib/jenkins/workspace/springApp/target/springApp-1.0.jar
I'm trying to ensure a script remains running on a development server. It collates stats and provides a web service so it's supposed to persist, yet a few times a day, it dies off for unknown reasons. When we notice we just launch it again, but it's a pain in the rear and some users don't have permission (or the knowhow) to launch it up.
The programmer in me wants to spend a few hours getting to the bottom of the problem but the busy person in me thinks there must be an easy way to detect if an app is not running, and launch it again.
I know I could cron-script ps through grep:
ps -A | grep appname
But again, that's another hour of my life wasted on doing something that must already exist... Is there not a pre-made app that I can pass an executable (optionally with arguments) and that will keep a process running indefinitely?
In case it makes any difference, it's Ubuntu.
I have used a simple script with cron to make sure that the program is running. If it is not, then it will start it up. This may not be the perfect solution you are looking for, but it is simple and works rather well.
#!/bin/bash
#make-run.sh
#make sure a process is always running.
export DISPLAY=:0 #needed if you are running a simple gui app.
process=YourProcessName
makerun="/usr/bin/program"
if ps ax | grep -v grep | grep $process > /dev/null
then
exit
else
$makerun &
fi
exit
Then add a cron job every minute, or every 5 minutes.
Monit is perfect for this :)
You can write simple config files which tell monit to watch e.g. a TCP port, a PID file etc
monit will run a command you specify when the process it is monitoring is unavailable/using too much memory/is pegging the CPU for too long/etc. It will also pop out an email alert telling you what happened and whether it could do anything about it.
We use it to keep a load of our websites running while giving us early warning when something's going wrong.
-- Your faithful employee, Monit
Notice: Upstart is in maintenance mode and was abandoned by Ubuntu which uses systemd. One should check the systemd' manual for details how to write service definition.
Since you're using Ubuntu, you may be interested in Upstart, which has replaced the traditional sysV init. One key feature is that it can restart a service if it dies unexpectedly. Fedora has moved to upstart, and Debian is in experimental, so it may be worth looking into.
This may be overkill for this situation though, as a cron script will take 2 minutes to implement.
#!/bin/bash
if [[ ! `pidof -s yourapp` ]]; then
invoke-rc.d yourapp start
fi
If you are using a systemd-based distro such as Fedora and recent Ubuntu releases, you can use systemd's "Restart" capability for services. It can be setup as a system service or as a user service if it needs to be managed by, and run as, a particular user, which is more likely the case in OP's particular situation.
The Restart option takes one of no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always.
To run it as a user, simply place a file like the following into ~/.config/systemd/user/something.service:
[Unit]
Description=Something
[Service]
ExecStart=/path/to/something
Restart=on-failure
[Install]
WantedBy=graphical.target
then:
systemctl --user daemon-reload
systemctl --user [status|start|stop|restart] something
No root privilege / modification of system files needed, no cron jobs needed, nothing to install, flexible as hell (see all the related service options in the documentation).
See also https://wiki.archlinux.org/index.php/Systemd/User for more information about using the per-user systemd instance.
I have used from cron "killall -0 programname || /etc/init.d/programname start". kill will error if the process doesn't exist. If it does exist, it'll deliver a null signal to the process (which the kernel will ignore and not bother passing on.)
This idiom is simple to remember (IMHO). Generally I use this while I'm still trying to discover why the service itself is failing. IMHO a program shouldn't just disappear unexpectedly :)
Put your run in a loop- so when it exits, it runs again... while(true){ run my app.. }
I couldn't get Chris Wendt solution to work for some reason, and it was hard to debug. This one is pretty much the same but easier to debug, excludes bash from the pattern matching. To debug just run: bash ./root/makerun-mysql.sh. In the following example with mysql-server just replace the value of the variables for process and makerun for your process.
Create a BASH-script like this (nano /root/makerun-mysql.sh):
#!/bin/bash
process="mysql"
makerun="/etc/init.d/mysql restart"
if ps ax | grep -v grep | grep -v bash | grep --quiet $process
then
printf "Process '%s' is running.\n" "$process"
exit
else
printf "Starting process '%s' with command '%s'.\n" "$process" "$makerun"
$makerun
fi
exit
Make sure it's executable by adding proper file permissions (i.e. chmod 700 /root/makerun-mysql.sh)
Then add this to your crontab (crontab -e):
# Keep processes running every 5 minutes
*/5 * * * * bash /root/makerun-mysql.sh
The supervise tool from daemontools would be my preference - but then everything Dan J Bernstein writes is my preference :)
http://cr.yp.to/daemontools/supervise.html
You have to create a particular directory structure for your application startup script, but it's very simple to use.
first of all, how do you start this app? Does it fork itself to the background? Is it started with nohup .. & etc? If it's the latter, check why it died in nohup.out, if it's the first, build logging.
As for your main question: you could cron it, or run another process on the background (not the best choice) and use pidof in a bashscript, easy enough:
if [ `pidof -s app` -eq 0 ]; then
nohup app &
fi
You could make it a service launched from inittab (although some Linuxes have moved on to something newer in /etc/event.d). These built in systems make sure your service keeps running without writing your own scripts or installing something new.
It's a job for a DMD (daemon monitoring daemon). there are a few around; but I usually just write a script that checks if the daemon is running, and run if not, and put it in cron to run every minute.
Check out 'nanny' referenced in Chapter 9 (p197 or thereabouts) of "Unix Hater's Handbook" (one of several sources for the book in PDF).
A nice, simple way to do this is as follows:
Write your server to die if it can't listen on the port it expects
Set a cronjob to try to launch your server every minute
If it isn't running it'll start, and if it is running it won't. In any case, your server will always be up.
I think a better solution is if you test the function, too. For example, if you had to test an apache, it is not enough only to test, if "apache" processes on the systems exist.
If you want to test if apache OK is, then try to download a simple web page, and test if your unique code is in the output.
If not, kill the apache with -9 and then do a restart. And send a mail to the root (which is a forwarded mail address to the roots of the company/server/project).
It's even simplier:
#!/bin/bash
export DISPLAY=:0
process=processname
makerun="/usr/bin/processname"
if ! pgrep $process > /dev/null
then
$makerun &
fi
You have to remember though to make sure processname is unique.
One can install minutely monitoring cronjob like this:
crontab -l > crontab;echo -e '* * * * * export DISPLAY=":0.0" && for
app in "eiskaltdcpp-qt" "transmission-gtk" "nicotine";do ps aux|grep
-v grep|grep "$app";done||"$app" &' >> crontab;crontab crontab
disadvantage is that the app names you enter have to be found in ps aux|grep "appname" output and at same time being able to be launched using that name: "appname" &
also you can use the pm2 library.
sudo apt-get pm2
And if its a node app can install.
Sudo npm install pm2 -g
them can run the service.
linux service:
sudo pm2 start [service_name]
npm service app:
pm2 start index.js