I have a service on Centos7 that runs a script in /usr/local/sbin/restarthelp2.sh and outputs a tunnel check by checking the status of network connection. The output of this ends up in /var/log/messages and makes the file huge. I already have the output being sent to its own log file, how do I keep the output results from the script/service out of the "messages" file?
[Unit]
Description=CHECK the wlan
[Service]
Type=simple
ExecStart=/usr/local/sbin/restarthelp2.sh
[Install]
WantedBy=default.target
Code for the script mentioned above:
#!/bin/bash
while true;
do
status=$(</sys/class/net/wlan0/operstate)
tunstate=$(</sys/class/net/tun0/carrier)
now=$(date)
if [ $status == up ] && [ $tunstate -eq 1 ];
then
echo "everything was good at $now, tunnel status was $tunstate" >> /var/log/wlancheck.log
echo "tunnel status is UP"
sleep 10
fi
done
You can add to your [Service] section of the Unit the line
StandardOutput=null
so that this output is not logged to the journal, and from there to syslog.
For other values see man systemd.exec.
If you are using rsyslogd you can filter messages much later, just before they get put in /var/log/messages. Remove the above Unit line to get back to normal logging. Look for a file like /etc/rsyslog.conf and a line like
*.info;... /var/log/messages
Add in front of this line a filter that compares a property with what you want to suppress, and use the action stop, for example one of:
if $programname startswith "restarthelp" then stop
if $msg contains 'tunnel status is UP' then stop
There is extensive rsyslog documentation, but it is hard to follow as there are many old formats that are still supported, so you must be careful not to mix them up.
If you also change the Unit StandardOutput=null to StandardOutput=syslog, you will no longer get the messages logged in the systemd journal, and they will go straight to rsyslogd. I don't know if this will provide you with the status information you wanted though.
Related
I want to trigger a desktop notification pop-up when a service is executed with systemd on my Linux desktop. The main reason why I am doing this is that I want to learn how to work with systemd timers and services by creating my own scheduled jobs and I would like to pop-up a desktop notification, when a service/job is executed, just to know that something is happening.
I have created a basic example to do that:
notifysystemd.sh:
#!/bin/bash
# Variable to hold path to systemd job logs
SYSTEMD_LOG_DIR='/home/jay/scheduledJobLogs/systemDJobLogs'
SYSTEMD_JOB_NAME='NotifySystemD'
CURRENT_MONTH=$(date '+%b')
# Send notification to desktop
notify-send 'You can automate and schedule anything with systemd today!'
# Write down in the log
CURRENT_TIME=$(date '+%Y-%m-%d-%H:%M')
LOG_RECORD="${CURRENT_TIME} SystemD notification job executed."
# Create a directory for systemd jobs logging, if it doesn't already exist. And don't error if it does exist
mkdir -p $SYSTEMD_LOG_DIR/$SYSTEMD_JOB_NAME
# Write the log record!
echo $LOG_RECORD >> $SYSTEMD_LOG_DIR/$SYSTEMD_JOB_NAME/$CURRENT_MONTH.txt
with this service file:
notifysystemd.service:
[Unit]
Description=A basic service to send a desktop notification using the systemd scheduler
Wants=notifysystemd.timer
[Service]
Type=forking
ExecStart=/home/jay/systemDJobs/notifysystemd.sh
Environment="DISPLAY=:0" "DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$(id -u)/bus" "XAUTHORITY=/home/jay/.Xauthority"
[Install]
WantedBy=default.target
and this timer file:
notifysystemd.timer:
[Unit]
Description=Send a notification three minutes after PC start
RefuseManualStart=false # Allow manual starts
RefuseManualStop=false # Allow manual stops
[Timer]
#Execute job if it missed a run due to machine being off
Persistent=true
OnBootSec=180
#File describing job to execute
Unit=notifysystemd.service
[Install]
WantedBy=timers.target
The service is executed correctly with the correct delay (I can see that in the log created), but I am getting no desktop notification.
I have looked into several questions already asked on this forum:
systemd service not executing notify-send
notify-send command doesn't launch the notification through systemd service
Which suggest specifying environment variables in either the .service file or in the shell script.
I have tried all of them and none led to a notification appearing.
I have done the same with cronie, where was sufficient to specify the DISPLAY and DBUS_SESSION_BUS_ADDRESS environement variables the same way as I did in the notifysystemd.service file.
Lastly, if there is a better way how to achieve the same result, but which revolves around usage of systemd, I am opened to optimal, or more ergonomic solutions.
In systemd instead of notify-send put the output in a file.
Write a bash script and start it on desktop login (so this script can notify-send).
In the script write an endless loop.
Use inotifywait in the loop to monitor the output file for modification.
(This will suspend your script and won't eat cpu)
After inotify, read the content of the file and use notify-send.
I want to run a game server inside my Ubuntu machine. I want to run it in the background and write the live output of that process inside a log file. I tried using nohup and running the game server using "&" at the end but I couldn't make it work the way I wanted.
Then I started reading about named pipes and actually gave it a go. I made a simple script that in theory should work. But, of course I am missing something.
First, I made a pipe using the mkfifo command.
mkfifo testpipe
Then I created a small script:
#!/bin/bash
./mta-server64 > pipe &
pid=$!
echo $pid // so I know the pid of the process
cat < pipe > log.txt &
(Note: I wrote this code from memory.)
The code works only when there is an error and the process stops. It actually records the game console error. But when the game server is running I get no output in the log file.
I want to read the output (stdout and stderr if I am not mistaken) of a process running in background and record it those inside a log file.
I also thought about using screen as it logs everything inside a file but I would prefer not using it if there is a better solution.
EDIT:
First of all: thank you for the interest you had in helping me. In the same way, I have to apologize for only giving scarce details about what I intend to do with this small project and for my limited understanding of stdout and stderr.
Let's go to the first base.
I want to run a game server named Multi Theft Auto (https://multitheftauto.com/). This is GTA San Andreas but multiplayer.
I can easily run this game server in my Ubuntu server by calling the executable ./mta-server-64. After calling it the game server console appears:
[|] MTA: San Andreas :: 0/32 players :: 196 resources :: 125 fps (25)
MTA:BLUE Server for MTA:SA
==================================================================
= Multi Theft Auto: San Andreas v1.5.6 [64 bit]
==================================================================
= Server name : Default MTA Server
= Server IP address: auto
= Server port : 22884
=
= Log file : /root/mta/mods/deathmatch/logs/server.log
= Maximum players : 32
= HTTP port : 22564
= Voice Chat : Disabled
= Bandwidth saving : Medium
==================================================================
[09:49:07] Resource 'mapmanager' requests some acl rights. Use the command 'aclrequest list mapmanager'
[09:49:07] Resources: 196 loaded, 0 failed
[09:49:07] Starting resources...
[09:49:07] Server minclientversion is now 1.5.6-9.16588.0
[09:49:07] INFO: MAPMANAGER: Some important ACL permissions are missing. To ensure the correct functioning of Mapmanager, please write: aclrequest allow mapmanager all
[09:49:07] Gamemode 'play' started.
[09:49:07] Authorized serial account protection is enabled for the ACL group(s): `Admin` See http://mtasa.com/authserial
[09:49:07] WARNING: <owner_email_address> not set
[09:49:07] Server started and is ready to accept connections!
[09:49:07] To stop the server, type 'shutdown' or press Ctrl-C
[09:49:07] Type 'help' for a list of commands.
[09:49:07] Querying MTA master server... success! (Auto detected IP:xxx.xxx.xxx.xxx)
I am using the following script to run the process in the background and (try to) get the live output from:
#!/bin/bash
newport=$(shuf -i 22003-22900 -n 1)
newip=$(shuf -i 22003-22900 -n 1)
rm -rf ~/server/*
cp -r /home/user*/ftp/server/mtaserver/serverfiles/* ~/server
sed -i "s/<httpport>[0-9][0-9][0-9][0-9][0-9]<\/httpport>/<httpport>$newport<\/httpport>/g" ~/server/mods/deathmatch/mtaserver.conf
sed -i "s/<serverport>[0-9][0-9][0-9][0-9][0-9]<\/serverport>/<serverport>$newip<\/serverport>/g" ~/server/mods/deathmatch/mtaserver.conf
~/server/mta-server64 2>&1 | tee -a outfile &
mta_pid=$!
echo $mta_pid
sleep 6
pkill $mta_pid
(Note: Because of some technical problems I had to add the first few lines of script which automatically replace the game files with new ones and also replace the existing ports with random ones.)
This script starts the server and tries to log the output of the process. The process is automatically killed after few seconds so there is only one instance of the game server at any given time.
THE ISSUE:
This script only logs the output if there is an error. I still cannot get the live output of the process when it is still running. Maybe this is an issue with the game server but truly believe there should be a way to make it work the way I intend.
I believe you want to use tee command to split the pipe output to log file.
I suggest you read this article and these answers 1 2.
Usually this is enough nohup somecommand > somecommand.log 2>&1 & then, tail -F somecommand.log to follow the logs.
After 2 days I finally figured out a way to make it work (the way I intended to work, without taking in consideration any major security/performance risks).
Reading the comments made me realize I was attacking the wrong point. The stdout of the game server is buffered, thus making it impossible to log it into a log file using the methods I tried when I posted my question At least this is what I came to understand).
I did some research on how to run the application without having the stdout buffered: https://serverfault.com/questions/294218/is-there-a-way-to-redirect-output-to-a-file-without-buffering-on-unix-linux
My code now:
stdbuf -o0 ~/server/mta-server64 >> pipe &
cat < pipe | tee -a outfile &
After creating the named pipe it executes the game server inside that pipe and then appends the stdout into the log file.
The stdbug -o0 command disables the stdout buffering (as noted in the link above).
This works for me and I cannot guarantee it will work for anybody else. I am still not aware if disabling the buffering is a safe approach to my issue but for now it is what I need.
I have bash script which do something, (for example:)
[program:long_script]
command=/usr/local/bin/long.sh
autostart=true
autorestart=true
stderr_logfile=/var/log/long.err.log
stdout_logfile=/var/log/long.out.log
and it is bounded to supervisor.
I want to add if check in this script to determine is it executed by:
supervisor> start long_script
or
supervisor> restart long_script
I want something like that:
if [ executed by start command ]
then
echo "start"
else
echo "restart"
fi
but i don't know what should be in if clause.
Is it possible to determine this?
If not, how to achieve different behaviour of script for start and restart commands?
Please help.
Within the code there is no current difference between a restart and a stop/start. Restart within the supervisorctl calls:
self.do_stop(arg)
self.do_start(arg)
There is no status within the app for "restart" though there is some discussion of allowing different signals. The supervisor is already able to send different signals to the process. (allowing more control over reload/restart has been a long standing "gap")
This means you have at least two options but the key to making this work is that the process needs to record some state at shutdown
Option 1. The easiest option would be to use the supervisorctl signal <singal> <process> instead of calling supervisorctl restart <process> and record somewhere what signal was sent so that on startup you can read back the last signal.
Option 2. However a more interesting solution is to not expect any upstream changes ie continue to allow restart to be used and distinguish between stop, crash and restart
In this case, the only information that will be different between a start and a restart is that a restart should have a much shorter time between the shutdown of the old process and the start of the new process. So if, on shutdown, a timestamp is recorded, then on startup, the difference between now and the last shutdown will distinguish between a start and a restart
To do this, Ive got a definition like yours but with stopsignal defined:
[program:long_script]
command=/usr/local/bin/long.sh
autostart=true
autorestart=true
stderr_logfile=/var/log/long.err.log
stdout_logfile=/var/log/long.out.log
stopsignal=SIGUSR1
By making the stop from supervisord a specific signal, you can tell the difference between a crash and a normal stop event, and not interfere with normal kill or interrupt signals
Then as the very first line in the bash script, i set a trap for this singal:
trap "mkdir -p /var/run/long/; date +%s > /var/run/long/last.stop; exit 0" SIGUSR1
This means the date as epoch will be recorded in the file /var/run/long/last.stop everytime we are sent a stop from supervisord
Then as the immediate next lines in the script, calculate the difference between the last stop and now
stopdiff=0
if [ -e /var/run/long/last.stop ]; then
curtime=$(date +%s)
stoptime=$(cat /var/run/long/last.stop | grep "[0-9]*")
if [ -n "${stoptime}" ]; then
stopdiff=$[ ${curtime} - ${stoptime} ]
fi
else
stopdiff=9999
fi
stopdiff will now contain the difference in seconds between the stop and start or 9999 if the stop file didnt exist.
This can then be used to decide what to do:
if [ ${stopdiff} -gt 2 ]; then
echo "Start detected (${stopdiff} sec difference)"
elif [ ${stopdiff} -ge 0 ]; then
echo "Restart detected (${stopdiff} sec difference)"
else
echo "Error detected (${stopdiff} sec difference)"
fi
You'll have to make some choices about how long it actually takes to get from sending a stop to the script actually starting: here, ive allowed only 2 seconds and anything greater is considered a "start". If the shutdown of the script needs to happen in a specific way, you'll need a bit more complexity in the trap statement (rather than just exit 0
Since a crash shouldnt record any timestamp to the stop file, you should be able to tell that a startup is occurring because of a crash if you also regularly recorded somewhere a running timestamp.
I understand your problem. But I don't know about supervisor. Please check whether this idea works.
Instantiate a global string variable and put values to the variable before you enter the supervisor commands. Here I am making your each start and restart commands as two bash programs.
program : supervisor_start.sh
#!/bin/bash
echo "Starting.."
supervisor> start long_script
supervisor_started_command="start" # This is the one
echo "Started.."
program : supervisor_restart.sh
#!/bin/bash
echo "ReStarting.."
supervisor> restart long_script
supervisor_started_command="restart" # This is the one
echo "ReStarted.."
Now you can see what is in "supervisor_started_command" variable :)
#!/bin/bash
if [ $supervisor_started_command == "start" ]
then
echo "start"
elif [ $supervisor_started_command == "restart" ]
echo "restart"
fi
Well, I don't know this idea works for you or not..
I'm on Debian and I have a systemd service that calls a bash script.
The script contains an infinite while loop, as I need it to check something every X seconds infinitely.
The systemd service crashes once it hits the "while true; do" line.
The script runs fine if I execute it manually.
Why doesn't systemd like it? What do I do?
Here are the service and the script. As I've indicated, an echo statement before the "while true; do" prints. The echo statement after the "while true; do" line does not print.
/etc/systemd/system/stream.service:
[Service]
WorkingDirectory=/home/pi/
ExecStart=/home/pi/joi_main.sh
Restart=no
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=stream_service
User=pi
Group=pi
Environment=NODE_ENV=production
[Install]
WantedBy=multi-user.target
/home/pi/joi_main.sh:
#!/bin/bash -e
today=`/bin/date '+%Y_%m_%d__%H_%M_%S'`
exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2
#Wait 120s for system to finish booting
sleep 120
#Initial config
export AUDIODEV=mic_mono
export AUDIODRIVER=alsa
sudo sysctl fs.pipe-max-size=1048576
echo "This line prints"
# Check if video buffer is full every minute. if full, the stream needs to restart
while true; do
echo "This line doesn't"
if grep "100% full" /home/pi/video_buffer_usage.txt; then
echo "Buffer is full!"
# Kill existing processes
pkill -f “raspivid|rec|buffer|ffmpeg”
# Wait 10s
sleep 10
./joi_stream.sh &
fi
sleep 60
done
Journalctl seems completely unhelpful, but here it is. No errors. Why is "session closed"?
Mar 31 02:13:41 raspberrypi sudo[1369]: pi : TTY=unknown ; PWD=/home/pi ; USER=root ; COMMAND=/sbin/sysctl fs.pipe-max-size=1048576
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 31 02:13:41 raspberrypi sudo[1369]: pam_unix(sudo:session): session closed for user root
(Please don't tell me to start yet another systemd service for just this while loop. I want it to be part of this main script because it needs to run after everything else, and if I turn off the main service I don't want the while loop running either, so maintaining two systemd services would only add troube.)
The contents of ./joi_stream.sh were not shared, but here's a problem I see with your systemd solution. It doesn't directly explain your behavior, but may be related:
In your systemd configuration, you redirect both STDOUT and STDERR to syslog, but in your script, you redirect STDERR (file descriptor "2") to a file, and redirect STDOUT (file descriptor "1') to STDERR.
exec 2> "/home/pi/stream_logs/$today.$RANDOM.log"
exec 1>&2
If your ./joi_stream.sh expected your redirection of these file descriptors to another file to work, it may not. If the file is just for logging, I would get rid of these lines and let the systemd journal handle that-- it will tag the logs with your unit you can review your logs specifically:
journalctl -u your-unit-name.service
Also, in systemd, you wouldn't normally put in a sleep to wait until the systemd has booted. Instead, you would use a .timer unit.
The .timer file would instruct to run the main logic every minute, so the "while" loop would not be required. The timer unit would contain directives like:
# Run for the first time 2 minutes after boot
# and every minute after that
OnBootSec=120
OnUnitActiveSec=60
It would be timer unit which is enabled to start on boot. Timer files can be super-simple. Just create a .timer file in /etc/systemd/system and give it the same name as the service file you want it to activate:
[Unit]
Description=Runs my service every minute
[Timer]
# Run for the first time 2 minutes after boot
# and every minute after that
OnBootSec=120
OnUnitActiveSec=60
[Install]
WantedBy=timers.target
To start and test your timer immediately, run:
sudo systemctl start my-service.timer
You can review the status of timers with:
sudo systemctl list-timers
The systemd solution is more robust than the rc.local solution. If your rc.local solution dies for any reason, it will not restart. However, if your script dies will run under systemd, the timer will still run it again a minute later.
FYI, everything works if I call /home/pi/joi_main.sh from /etc/rc.local instead of using a systemd service. I'll use rc.local and kill the service.
Lets say I have a service (like rsyslog) that when I stop it, I want to log the reasoning behind someone shutting it down. Expressing this simply would be
systemctl start rsyslog
systemctl stop rsyslog
(Begin the prompt as to why a user is doing this after shutting down rsyslog)
#!/bin/bash
echo "you are shutting down the syslog service. Please state your name and reason."
read -p "[name?]" name && read -p "[reason?]" reason
logger $name:$reason
Modifying the Unit Files (located in /usr/lib/systemd/system/rsyslog.service), to include an ExecStop of the path to the script, I am able to run the above script. I know that the script is working as checking the log messages shows a :, the nonvariable portion that was passed to logger.
I need to be able to have this script operate like said shell script the instant someone attempts to shutdown the logging service. This means that echo commands are shown on the terminal and variables can be recorded using the read command.
This may be similar to this persons question, but I can not understand it.
Thanks to Mark Stosberg for sharing information about the systemd-ask-password command that takes user input during a systemctl call.
For those unaware, the systemd-ask-password command is a password prompt that's available for all machines sponsoring the systemd service. A unique feature of this command is the fact it can allow for user input during a systemctl/service call. Knowing this, one can prompt for as much data as they like which integrates perfectly into standard bash scripts, allowing for the interaction that may or may not be needed during a call.
Here is an example script:
#!/bin/bash
date=$(/bin/date)
echo "Rsyslog has been turned off at $date. Below is the name and reason"
name=`systemd-ask-password --echo --no-tty "name:"`
reason=`systemd-ask-password --echo --no-tty "reason for shutting down rsyslog:"`
echo LOG:$name:$reason:ENDLOG >>/var/log/messages
You must make sure that when initializing any of your services you make changes to the unit files located in /usr/lib/systemd/system/ with an ExecStart, ExecStop, and so forth under the [Service] tag to equal the path to your scripts. You can find what other options you can here as well as some syntax, and tie in with the unit file as needed.