Puppet start on device with no clock? - puppet

I'm starting puppet agent using upstart on a Debian 14.04 embedded system:
description "Puppet Agent"
start on started 2klic-gateway
stop on runlevel [!2345]
respawn
pre-start script
if [ ! -f /var/lib/sc2klic/system.json ]; then
stop ; exit 0
fi
puppet config set certname "$(hostname)"
end script
exec /usr/local/bin/puppet agent --no-daemonize
The device, or at least this version, doesn't have a hardware clock. So the system starts with a date of January 1, 1970.
When I look in /var/log/upstart/puppet-agent.log I see this error over and over again:
ESC[1;31mError: Could not parse application options: copyright with a year after 1972 is very strange; did you accidentally add or subtract two years?ESC[0m
Is it possible to initialize puppet agent without having a correct date?
Puppet Agent Version 4.10.1

This error seems to be related to the documentation so I don't know if it should error here. I proposed a pull-request to make an exception to not trigger in this situation.
That said we know if year is 1970 the device is not online, thus puppet can't be used anyways.
As a work around I added the following to the upstart pre-start script:
while [[ $(date +%Y) == "1970" ]]
do
sleep 30
done
This way we just keep puppet waiting until the device has connected to an NTP server.

Related

Using "Deep Sleep" / "Power Saving" mode through SSH in NAS OS?

I'm an user of a LACIE 2-BIG-NAS. Until the NAS OS 4.1.9.2 version I had the "Deep Sleep" option in the Home menu, but after the next upgrade this option was removed.
I tried to downgrade to the previous version following the manual steps but it was not able, only upgrades are available.
I asked to the support service of Lacie but the their solution is to backup my data and do a fresh install and upgrade until 4.1.9.2. This isn't a solution from my point of view.
Now I tried to get into deep sleep mode from a SSH conection because NAS OS is a linux-based SO. I tried all the posibilities with initng command (sudo ngc -0 and -1) which is used by the NAS OS, but it's imposible to wake on lan the NAS (the OS powers off but no answer from the wake-on-lan request).
The code for wake on lan is correct because when I schedule the deep sleep mode I can do it, but I don`t know how to get deep sleep mode on-demand.
I googled and try other options but I think these were the closest to the solution.
Please, can you help me to find the correct ssh command line to get the deep sleep mode in the Lacie 2-big-nas?
Best regards.
I found the solution in cron. There is a scheduled command in /sbin/smart_shutdown so, if you execute that script as root, the 2-big-nas go into the Deep sleep mode.
This is the content of the script "smart_shutdown":
#!/bin/sh
#
# This script is intended to handle a user shutdown request.
# It will probably (but not necesseraly) called from a crontab.
#
PATH=/bin:/sbin:/usr/bin:/usr/sbin
valid_runlevels="shutdown halt sleep reboot"
runlevel="sleep"
check_runlevel()
{
req_runlevel=$1
for valid in ${valid_runlevels}; do
[ "${req_runlevel}" = "${valid}" ] && return 0
done
logger "smart_shutdown: request invalid runlevel ${req_runlevel}"
return 1
}
request_runlevel()
{
dbus-send --system --dest=com.lacie.Unicorn --type=method_call --print-reply --reply-timeout=1000 /com/lacie/Unicorn com.lacie.Unicorn.switch_runlevel string:"$1"
}
if [ ! -z "$1" ]; then
check_runlevel "$1" || exit 1
runlevel=$1
fi
request_runlevel ${runlevel}
exit 0
I hope you can take advantage of this in the future.

Ubuntu upstart gets incorrect PID from Play 1.3

The Upstart script using the start-stop-daemon we've been using for Play 1.2.7 is now unable to stop/restart Play since Play 1.3 due to it having an incorrect PID.
Framework version: 1.3.0 on Ubuntu 12.04.5 LTS
Reproduction steps:
Setup an upstart script (playframework.conf) for a Play application
Play application starts successfully on server reboot Run 'sudo
status playframework' will return playframework start/running,
process 28912 - At this point process 28912 doesn't exist
vi {playapplicationfolder}/server.pid shows 28927
'stop playframework'
then fails due to unknown pid 28912 'status playframework' results in
playframework stop/killed, process 28912
Only way to restart play framework after this point is to either find the actual process and kill it then start play using the usual 'play start' command manually. Or restart the server.
This has broken our deployments scripts now as we used to install the new version of our app, then do play restart before reconnecting to the load balancer.
Upstart Script:
#Upstart script for a play application that binds to an unprivileged user.
# put this into a file like /etc/init/playframework
# you can then start/stop it using either initctl or start/stop/restart
# e.g.
# start playframework
description "PlayApp"
author "-----"
version "1.0"
env PLAY_BINARY=/opt/play/play
env JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
env HOME=/opt/myapp/latest
env USER=ubuntu
env GROUP=admin
env PROFILE=prod
start on (filesystem and net-device-up IFACE=lo) or runlevel [2345]
stop on runlevel [!2345]
limit nofile 65536 65536
respawn
respawn limit 10 5
umask 022
expect fork
pre-start script
test -x $PLAY_BINARY || { stop; exit 0; }
test -c /dev/null || { stop; exit 0; }
chdir ${HOME}
rm ${HOME}/server.pid || true
/opt/configurer.sh
end script
pre-stop script
exec $PLAY_BINARY stop $HOME
end script
post-stop script
rm ${HOME}/server.pid || true
end script
script
exec start-stop-daemon --start --exec $PLAY_BINARY --chuid $USER:$GROUP --chdir $HOME -- start $HOME -javaagent:/opt/newrelic/newrelic.jar --%$PROFILE -Dprecompiled=true --http.port=8080 --https.port=4443
end script
We've tried specifying the PID file in the start-stop-daemon as per: http://man.he.net/man8/start-stop-daemon however this also didnt seem to have any effect.
I have found some threads on similar issues https://askubuntu.com/questions/319199/upstart-tracking-wrong-pid-of-process-not-respawning but have been unable to find a way round this so far. I have tried changing fork to daemon but the same issue remains. I also can't see what has changed between Play 1.2.7 and 1.3 to cause this.
Another SO post has also asked a similar question but not had an answer as yet: https://stackoverflow.com/questions/23117345/upstart-gets-wrong-pid-after-launching-celery-with-start-stop-daemon
This is because getJavaVersion() spawns a subprocess, which bumps the PID count, which breaks Upstart, the latter which expects Play to fork exactly none, once or twice, depending on which expect stanza you use.
I've fixed this in a pull request.

Automatically starting Celery from within Django app

I am getting a Django 1.6 set up started on a Linux (Debian Whiskey) server on Google Compute Engine. I've got Celery 3.1 running in the background to help with some processes. When I start a new instance (using a snapshot I've created), I always need to start Celery. I am looking for a way to start Celery automatically on server-load. This is particularly helpful if the server decides to restart, as they seem to do now and then. To achieve this, I've edited the rc.local file:
$ sudo nano /etc/rc.local
It used to contain the following:
exit 0
[ -x /sbin/initctl ] && initctl emit --no-wait google-rc-local-has-run || true
I've edited the file such that it now reads:
cd /home/user/gce_app celery -A myapp.tasks --concurrency=1 --loglevel=info worker > output.log 2> errors.log &
exit 0
[ -x /sbin/initctl ] && initctl emit --no-wait google-rc-local-has-run || true
The directory:
/home/user/gce_app
is where my Django project resides and the directory I need to be in to start Celery. However, after restarting the instance, when I type in:
$ celery status
Error: No nodes replied within time constraint.
Opening the errors.log file, I see:
/etc/rc.local: 14: /etc/rc.local: celery: not found
Surely the cd at the start of that code string should address this? Is there a way (within the Django project itself) to start the Celery instance when the project is started to make the code more platform-independent and immune to inevitable OS updates?
I think you're missing a semicolon between your 'cd' and celery invocations. Also, I suspect rc.local may not be searching your path, so you may need to give an absolute path to celery. e.g.
cd /home/user/gce_app; /usr/bin/celery ...
Alternatively, you might look at using a startup script from the GCE metadata to avoid needing to modify rc.local.
Since you seem to be using upstart this might help you:
description "runs celery"
start on runlevel [2345]
stop on runlevel [!2345]
console log
env VENV='/srv/myvirtualenv'
env PROJECT='/srv/run/mydjangoproject'
exec su -s /bin/sh -c 'exec "$0" "$#"' www-data -- /usr/bin/env PATH=$VENV:$PATH $VENV/python $PROJECT/manage.py celeryd
respawn
respawn limit 10 5

How can I trigger a delayed system shutdown from in a shell script?

On my private network I have a backup server, which runs a bacula backup every night. To save energy I use a cron job to wake the server, but I haven't found out, how to properly shut it down after the backup is done.
By the means of the bacula-director configuration I can call a script during the processing of the last backup job (i.e. the backup of the file catalog). I tried to use this script:
#!/bin/bash
# shutdown server in 10 minutes
#
# ps, 17.11.2013
bash -c "nohup /sbin/shutdown -h 10" &
exit 0
The script shuts down the server - but apparently it returns just during the shutdown,
and as a consequence that last backup job hangs just until the shutdown. How can I make the script to file the shutdown and return immediately?
Update: After an extensive search I came up with a (albeit pretty ugly) solution:
The script run by bacula looks like this:
#!/bin/bash
at -f /root/scripts/shutdown_now.sh now + 10 minutes
And the second script (shutdown_now.sh) looks like this:
#!/bin/bash
shutdown -h now
Actually I found no obvious method to add the required parameters of shutdown in the syntax of the 'at' command. Maybe someone can give me some advice here.
Depending on your backup server’s OS, the implementation of shutdown might behave differently. I have tested the following two solutions on Ubuntu 12.04 and they both worked for me:
As the root user I have created a shell script with the following content and called it in a bash shell:
shutdown -h 10 &
exit 0
The exit code of the script in the shell was correct (tested with echo $?). The shutdown was still in progress (tested with shutdown -c).
This bash function call in a second shell script worked equally well:
my_shutdown() {
shutdown -h 10
}
my_shutdown &
exit 0
No need to create a second BASH script to run the shutdown command. Just replace the following line in your backup script:
bash -c "nohup /sbin/shutdown -h 10" &
with this:
echo "/sbin/poweroff" | /usr/bin/at now + 10 min >/dev/null 2>&1
Feel free to adjust the time interval to suit your preference.
If you can become root: either log in as, or sudo -i this works (tested on ubuntu 14.04):
# shutdown -h 20:00 & //halts the machine at 8pm
No shell script needed. I can then log out, and log back in, and the process is still there. Interestingly, if I tried this with sudo in the command line, then when I log out, the process does go away!
BTW, just to note, that I also use this command to do occasional reboots after everyone has gone home.
# shutdown -r 20:00 & //re-boots the machine at 8pm

How to make sure an application keeps running on Linux

I'm trying to ensure a script remains running on a development server. It collates stats and provides a web service so it's supposed to persist, yet a few times a day, it dies off for unknown reasons. When we notice we just launch it again, but it's a pain in the rear and some users don't have permission (or the knowhow) to launch it up.
The programmer in me wants to spend a few hours getting to the bottom of the problem but the busy person in me thinks there must be an easy way to detect if an app is not running, and launch it again.
I know I could cron-script ps through grep:
ps -A | grep appname
But again, that's another hour of my life wasted on doing something that must already exist... Is there not a pre-made app that I can pass an executable (optionally with arguments) and that will keep a process running indefinitely?
In case it makes any difference, it's Ubuntu.
I have used a simple script with cron to make sure that the program is running. If it is not, then it will start it up. This may not be the perfect solution you are looking for, but it is simple and works rather well.
#!/bin/bash
#make-run.sh
#make sure a process is always running.
export DISPLAY=:0 #needed if you are running a simple gui app.
process=YourProcessName
makerun="/usr/bin/program"
if ps ax | grep -v grep | grep $process > /dev/null
then
exit
else
$makerun &
fi
exit
Then add a cron job every minute, or every 5 minutes.
Monit is perfect for this :)
You can write simple config files which tell monit to watch e.g. a TCP port, a PID file etc
monit will run a command you specify when the process it is monitoring is unavailable/using too much memory/is pegging the CPU for too long/etc. It will also pop out an email alert telling you what happened and whether it could do anything about it.
We use it to keep a load of our websites running while giving us early warning when something's going wrong.
-- Your faithful employee, Monit
Notice: Upstart is in maintenance mode and was abandoned by Ubuntu which uses systemd. One should check the systemd' manual for details how to write service definition.
Since you're using Ubuntu, you may be interested in Upstart, which has replaced the traditional sysV init. One key feature is that it can restart a service if it dies unexpectedly. Fedora has moved to upstart, and Debian is in experimental, so it may be worth looking into.
This may be overkill for this situation though, as a cron script will take 2 minutes to implement.
#!/bin/bash
if [[ ! `pidof -s yourapp` ]]; then
invoke-rc.d yourapp start
fi
If you are using a systemd-based distro such as Fedora and recent Ubuntu releases, you can use systemd's "Restart" capability for services. It can be setup as a system service or as a user service if it needs to be managed by, and run as, a particular user, which is more likely the case in OP's particular situation.
The Restart option takes one of no, on-success, on-failure, on-abnormal, on-watchdog, on-abort, or always.
To run it as a user, simply place a file like the following into ~/.config/systemd/user/something.service:
[Unit]
Description=Something
[Service]
ExecStart=/path/to/something
Restart=on-failure
[Install]
WantedBy=graphical.target
then:
systemctl --user daemon-reload
systemctl --user [status|start|stop|restart] something
No root privilege / modification of system files needed, no cron jobs needed, nothing to install, flexible as hell (see all the related service options in the documentation).
See also https://wiki.archlinux.org/index.php/Systemd/User for more information about using the per-user systemd instance.
I have used from cron "killall -0 programname || /etc/init.d/programname start". kill will error if the process doesn't exist. If it does exist, it'll deliver a null signal to the process (which the kernel will ignore and not bother passing on.)
This idiom is simple to remember (IMHO). Generally I use this while I'm still trying to discover why the service itself is failing. IMHO a program shouldn't just disappear unexpectedly :)
Put your run in a loop- so when it exits, it runs again... while(true){ run my app.. }
I couldn't get Chris Wendt solution to work for some reason, and it was hard to debug. This one is pretty much the same but easier to debug, excludes bash from the pattern matching. To debug just run: bash ./root/makerun-mysql.sh. In the following example with mysql-server just replace the value of the variables for process and makerun for your process.
Create a BASH-script like this (nano /root/makerun-mysql.sh):
#!/bin/bash
process="mysql"
makerun="/etc/init.d/mysql restart"
if ps ax | grep -v grep | grep -v bash | grep --quiet $process
then
printf "Process '%s' is running.\n" "$process"
exit
else
printf "Starting process '%s' with command '%s'.\n" "$process" "$makerun"
$makerun
fi
exit
Make sure it's executable by adding proper file permissions (i.e. chmod 700 /root/makerun-mysql.sh)
Then add this to your crontab (crontab -e):
# Keep processes running every 5 minutes
*/5 * * * * bash /root/makerun-mysql.sh
The supervise tool from daemontools would be my preference - but then everything Dan J Bernstein writes is my preference :)
http://cr.yp.to/daemontools/supervise.html
You have to create a particular directory structure for your application startup script, but it's very simple to use.
first of all, how do you start this app? Does it fork itself to the background? Is it started with nohup .. & etc? If it's the latter, check why it died in nohup.out, if it's the first, build logging.
As for your main question: you could cron it, or run another process on the background (not the best choice) and use pidof in a bashscript, easy enough:
if [ `pidof -s app` -eq 0 ]; then
nohup app &
fi
You could make it a service launched from inittab (although some Linuxes have moved on to something newer in /etc/event.d). These built in systems make sure your service keeps running without writing your own scripts or installing something new.
It's a job for a DMD (daemon monitoring daemon). there are a few around; but I usually just write a script that checks if the daemon is running, and run if not, and put it in cron to run every minute.
Check out 'nanny' referenced in Chapter 9 (p197 or thereabouts) of "Unix Hater's Handbook" (one of several sources for the book in PDF).
A nice, simple way to do this is as follows:
Write your server to die if it can't listen on the port it expects
Set a cronjob to try to launch your server every minute
If it isn't running it'll start, and if it is running it won't. In any case, your server will always be up.
I think a better solution is if you test the function, too. For example, if you had to test an apache, it is not enough only to test, if "apache" processes on the systems exist.
If you want to test if apache OK is, then try to download a simple web page, and test if your unique code is in the output.
If not, kill the apache with -9 and then do a restart. And send a mail to the root (which is a forwarded mail address to the roots of the company/server/project).
It's even simplier:
#!/bin/bash
export DISPLAY=:0
process=processname
makerun="/usr/bin/processname"
if ! pgrep $process > /dev/null
then
$makerun &
fi
You have to remember though to make sure processname is unique.
One can install minutely monitoring cronjob like this:
crontab -l > crontab;echo -e '* * * * * export DISPLAY=":0.0" && for
app in "eiskaltdcpp-qt" "transmission-gtk" "nicotine";do ps aux|grep
-v grep|grep "$app";done||"$app" &' >> crontab;crontab crontab
disadvantage is that the app names you enter have to be found in ps aux|grep "appname" output and at same time being able to be launched using that name: "appname" &
also you can use the pm2 library.
sudo apt-get pm2
And if its a node app can install.
Sudo npm install pm2 -g
them can run the service.
linux service:
sudo pm2 start [service_name]
npm service app:
pm2 start index.js

Resources