In my usecase, I am trying to write an upstart script for grafana service. Here is my content,
#grafana upstart script
description "start and stop grafana server"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn
respawn limit 5 30
console output
exec service grafana-server stop
But this is not woking all the time. Do I need remove any pid after stopping the service?
Also is this the right way to create upstart conf with service. Because I tried to do the same for nginx service. But it fails because we need to remove the pid after stopping the service.
So in general how can we write upstart script for services?
I am using amazon linux [ec2]. Could someone help me with this?
You can just look at any script in the folder /etc/init.d and change the instructions according your your needs. The you can use the systemctl (or the appropiate command for your distribution) to make it run on the desired init levels
Related
I need running a (bash)Script after each start from my EC2.
The machine stops 90% a day - after wake up - a script should run.
I tried to push it in the USER-DATA - but this only runs on init.
After this I followed up here: https://aws.amazon.com/de/premiumsupport/knowledge-center/execute-user-data-ec2/
but this didn't also work - because to stop a machine and start a machine seems to be no reboot
I also implement a simple output in the rc.local but also: nothing happens.
Is there a way?
So we talk to switch from this Instance-State
to this:
You could use the oneshot feature of systemd
Write scripts for mystart.sh and mystop.sh, chmod/chown them
/etc/systemd/system/mystart.service. Note that we must specify RemainAfterExit=true so that systemd considers the service as active after the setup action is successfully finished.
[Unit]
Description=mystart
After=network.target
[Service]
Type=oneshot
ExecStart=/usr/local/bin/mystart.sh
RemainAfterExit=true
ExecStop=/usr/local/bin/mystop.sh
StandardOutput=journal
[Install]
WantedBy=multi-user.target
Reload the systemd (systemctl reload) and try stopping and starting to test it
Cloud-Init (which runs the User Data script on first boot) can also run scripts:
On every boot
On the next boot only
So, simply install your script in this directory:
/var/lib/cloud/scripts/per-boot/
Each time the instance is booted (started), the script will run. This is a great way to trigger a batch process on the instance.
See also: Auto-Stop EC2 instances when they finish a task - DEV Community
I have a systemd service for my spring boot application connected to consul server, behind haproxy. consul provides consul-template to automatically update the service location in haproxy configuration file via consul-template command.
consul-template takes a template file and writes to the final haproxy configuration file and then reload the haproxy.
Now, consul-template process needs to run in background always along with my application, so that as the application comes up, it can detect new application startup and update its location in the configuration file.
Here is my systemd service file for this.
[Unit]
Description=myservice
Requires=network-online.target
After=network-online.target
[Service]
Type=forking
PIDFile=/home/dragon/myservice/run/myservice.pid
ExecStart=/home/dragon/myservice/bin/myservice-script start
ExecReload=/home/dragon/myservice/bin/myservice-script reload
ExecStop=/home/dragon/myservice/bin/myservice-script stop
ExecStartPost=consul-template -template '/etc/haproxy/haproxy.cfg.template:/etc/haproxy/haproxy.cfg:sudo systemctl reload haproxy'
User=dragon
[Install]
WantedBy=multi-user.target
Now, when I start systemctl start myservice, my application starts and the call to consul-template also works, but consul-template process doesn't go in background. I have to press Ctl+C and then systemctl comes back and I have both my application and consul-template process running.
Is there way to run the consul-template process in background specified in ExecStartPost?
I was trying to add & at the end of the ExecStartPost command, but then consul-template complains that it is an additional invalid argument and it fails.
I was also trying to make the command as /bin/sh -c "consul-template command here...", but then this also doesn't work. Even nohup in this command wasn't working.
Any help is really appreciated.
A workaround would be to have a bash file as your entrypoint, add all you need in there, then it will all magically work
I was trying to accomplish the same task. I wanted to fire off some HTTP requests to Tomcat once the service had started, so that I could warmup our servers ahead of the first user request.
I went through a lot of trial and error with using trying to use ExecStartPost to fire off an async process, but actually worked. By calling a shell script, I could trigger off background processes, but from my testing Systemd appears to kill the process thread when ExecStartPost finishes, so any child processes end up getting killed too. I tried various combinations of using &, setsid, nohup, etc, even some Perl to try and trigger off the an executable in it's own thread, but as soon as the shell script exite from ExecStartPost any processes running where killed. It's possible there's some solution that would work using ExecStartPost, but I couldn't find it.
However, what did work is creating a new service (like #divinedragon mentions) which piggy backs off the service I wanted to monitor (in this case Tomcat).
Since it took me a little research to get something working the way I wanted, I wanted to share my solution in case it helps someone.
The first step is to create a new service (e.g. /usr/lib/systemd/system/tomcat-service-listener.service):
[Unit]
Description=Tomcat start/stop event listener
# make sure to stop the service when Tomcat stops
BindsTo=tomcat.service
# waits for both Nginx & Tomcat to be started before this service is started
After=nginx.service tomcat.service
[Service]
Type=oneshot
ExecStart=/path/to/your/script.sh start
ExecStop=/path/to/your/script.sh stop
RemainAfterExit=yes
TimeoutStartSec=300
[Install]
# When the service is enabled, forces this service to start when Tomcat is started
WantedBy=tomcat.service
Some notes on what is happening here:
The BindsTo make sure the service gets stopped when Tomcat is stopped. This triggers the ExecStop command.
The After make sure that on server reboot, this service does not start until both Nginx & Tomcat have started.
The WantedBy will create the wants symlink for Tomcat (when the service is enabled), which will force Tomcat to start this service any time it's restarted.
The RemainAfterExit=yes is necessary for the ExecStop to work. If you only care about triggering something when you're service is started and don't care about when the service is stopped, you can set this to no and remove the ExecStop line.
Make the TimeoutStartSec long enough for whatever task you plan on running.
To get this service working, you then need to do the following:
# make the service executable
chmod 664 /usr/lib/systemd/system/tomcat-service-listener.service
# make Systemd aware of the new service
systemctl daemon-reload
# register the service so it's started/stopped with Tomcat
systemctl enable tomcat-service-listener.service
Now all you need script to trigger off the logic you want. In my case, I wanted to warmup some servers once Tomcat started so my /path/to/your/script.sh looks something like:
#!/bin/sh
SCRIPT_MODE="$1"
LOGFILE=/var/logs/myscript.log
log_message() {
local MESSAGE="$1"
echo "$(date '+%Y-%m-%d %H:%M:%S') $MESSAGE" >> "$LOGFILE"
return 0
}
warmup_server() {
local SERVER_ADDRESS="$1"
local SERVER_DESCRIPTION="$2"
log_message "Warming up $SERVER_DESCRIPTION..."
# we want to track the time it took to warm up the server
local START_TIME=$(date +%s)
# server restarts can take a while for all services to start, so we must retry long enough for all relevant services to start
HTTP_STATUS=$(curl --insecure --location --silent --show-error --fail --retry 60 --retry-delay 2 --retry-max-time 240 --output /dev/null --write-out "%{http_code}" '$SERVER_ADDRESS')
# we want to track the time it took to warm up the server
local TOTAL_STARTUP_TIME=$(($(date +%s)-$START_TIME))
log_message "$SERVER_DESCRIPTION started in $TOTAL_STARTUP_TIME seconds... (Status: $HTTP_STATUS)"
return 0
}
# monitor when Tomcat has stopped
if [ "$SCRIPT_MODE" == "stop" ]; then
log_message "Tomcat listener shutting down..."
exit 0
elif [ "$SCRIPT_MODE" == "start" ]; then
log_message "Tomcat listener started..."
fi
# servers to warm up
warmup_server 'https://127.0.0.1' 'Localhost #1'
warmup_server 'https://127.0.0.2' 'Localhost #2'
This seems to be working exactly as I want. The service starts up when the server is reboot and starting/stopping/restarting Tomcat fires off the expected events. Since it's independent of the Tomcat service, I can restart this warmup script if needed. It also doesn't delay the Tomcat startup time, since it is its own service, therefore running asynchronously like I wanted.
I am using upstart to start my golang application. I have my application folder structure like this,
web-app/
/app
main.go
I built the application as below,
$cd /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app/
$go build ./...
It generated the binary app
And placed the web-app.conf in /etc/init/ folder. Here is the web-app.conf content,
#Web app upstart script
description "start and stop web app"
start on (net-device-up
and local-filesystems
and runlevel [2345])
stop on runlevel [016]
respawn
respawn limit 5 30
console output
script
chdir /home/ec2-user/go/src/github.com/dineshappavoo/web-app/app
exec ./app
end script
When I tried sudo initctl list, it lists the process as stop/waiting. And I tried to start the process
$sudo initctl start web-app
It shows the process as start/running. But it is not started.
I checked the /var/log/messages logs. It shows,
init: web-app main process (18740) terminated with status 127
I couldn't start the process. I think there is some issue with the chdir. I tried different options for past two days. And I am fairly new to upstart but no luck. Could someone help me with this?
OP eventually solved after fixing a few issues. See comments, notably:
upstart might not recognise environment variables
Amazon Linux Image currently uses an old version of init (upstart 0.6.5), lacking newer features such as console log & nested script tags
status 127 can occur if exec can't find the binary
status 1 can occur if binary runs but fails
substituting a simple program in an upstart script can help diagnose errors
I have my executable file in service folder /var/www/project/service/project.exe.
Currently I'm running on runlevel 2. I used this link to change my default runlevel to 5.
Since this will be deployed on the cloud ,I want that it should start the service(project.exe) in runlevel 5 automatically, rather than manually.
How this can be achieved?
You could create an Upstart job. Write /etc/init/project.conf file:
start on runlevel 5
exec /var/www/project/service/project.exe
You probably want to specify other options such as description, stop on, respawn, setuid.
I have recently completed the Wiki web development tutorial (http://golang.org/doc/articles/wiki/). I had tons of fun and I would like to experiment more with the net/http package.
However, I noticed that when I run the wiki from a console, the wiki takes over the console. If I close the console terminal or stop the process with CTRL+Z then the server stops.
How can I get the server to run in the background? I think the term for that is running in a daemon.
I'm running this on Ubuntu 12.04. Thanks for any help.
Simple / Usable things first
If you want a start script without much effort (i.e. dealing with the process, just having it managed by the system), you could create a systemd service. See Greg's answer for a detailled description on how to do that.
Afterwards you can start the service with
systemctl start myserver
Previously I would have recommended trying xinetd or something similar for finer granuarlity regarding resource and permission management but systemd already covers that.
Using the shell
You could start your process like this:
nohup ./myexecutable &
The & tells the shell to start the command in the background, keeping it in the job list.
On some shells, the job is killed if the parent shell exits using the HANGUP signal.
To prevent this, you can launch your command using the nohup command, which discards the HANGUP signal.
However, this does not work, if the called process reconnects the HANGUP signal.
To be really sure, you need to remove the process from the shell's joblist.
For two well known shells this can be achieved as follows:
bash:
./myexecutable &
disown <pid>
zsh:
./myexecutable &!
Killing your background job
Normally, the shell prints the PID of the process, which then can be killed using the kill command, to stop the server. If your shell does not print the PID, you can get it using
echo $!
directly after execution. This prints the PID of the forked process.
You could use Supervisord to manage your process.
Ubuntu? Use upstart.
Create a file in /etc/init for your job, named your-service-name.conf
start on net-device-up
exec /path/to/file --option
You can use start your-service-name, as well as: stop, restart, status
This will configure your service using systemd, not a comprehensive tutorial but rather a quick jump-start of how this can be set up.
Content of your app.service file
[Unit]
Description=deploy-webhook service
After=network.target
[Service]
ExecStart=/usr/bin/go webhook.go
WorkingDirectory=/etc/deploy-webhook
User=app-svc
Group=app-svc
Restart=always
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=deploy-webhook-service
PrivateTmp=true
Environment=APP_PARAM_1=ParamA
Environment=APP_PARAM_2=ParamB
[Install]
WantedBy=multi-user.target
Starting the Service
sudo systemctl start deploy-webhook.service
Service Status
sudo systemctl status deploy-webhook.service
Logs
journalctl -u deploy-webhook -e
After you press ctrl+z (putting the current task to sleep) you can run the command bg in the terminal (stands for background) to let the latest task continue running in the background.
When you need to, run fg to get back to the task.
To get the same result, you can add to your command & at the end to start it in the background.
To add to Greg's answer:
To run the Go App as a service you need to create a new service unit file.
However, the App needs to know where Go is installed. The easiest way to lookup that location is by running this command:
which go
which gives you an output like this:
/usr/local/go/bin/go
With this piece of information, you can create the systemd service file. Create a file named providus-app.service in the /etc/systemd/system/ using the command below:
sudo touch /etc/systemd/system/providus-app.service
Next open the newly created file:
sudo nano /etc/systemd/system/providus-app.service
Paste the following configuration into your service file:
[Unit]
Description=Providus App Service
After=network.target
[Service]
Type=forking
User=deploy
Group=deploy
ExecStart=/usr/local/go/bin/go run main.go
WorkingDirectory=/home/deploy/providus-app
Restart=always
RestartSec=10
KillSignal=SIGINT
SyslogIdentifier=providus-app-service
PrivateTmp=true
[Install]
WantedBy=multi-user.target
When you are finished, save and close the file.
Next, reload the systemd daemon so that it knows about our service file:
sudo systemctl daemon-reload
Start the Providus App service by typing:
sudo systemctl restart providus-app
Double-check that it started without errors by typing:
sudo systemctl status providus-app
And then enable the Providus App service file so that Providus App automatically starts at boot, that is, it can start on its own whenever the server restarts:
sudo systemctl enable providus-app
This creates a multi-user.target symlink in /etc/systemd/system/multi-user.target.wants/providus-app.service for the /etc/systemd/system/providus-app.service file that you created.
To check logs:
sudo journalctl -u providus-app