How to debug AlertManager? - prometheus-alertmanager

I have configured Alertmanager to send a mail everytime an alert is triggered. However for an unknow reason I'm not receiving any mail.
How can I debug this? Is there a log file stored somewhere?

How have you started the Alertmanager? The tool should show the warnings/errors in the terminal. You can start the Alertmanager redirecting the output to a log file like in the following example:
ALERTMANAGER-INSTALL-PATH/alertmanager >> ALERTMANAGER-LOG-PATH/alertmanager.log 2>&1 &
If you're running the Alertmanager inside a Docker container try to use the Docker logs.

Related

Docker container STDOUT not showing in Docker Logs

I am trying to configure my php errors to output to docker logs. All documentation I have read indicates that docker logs are tied to the containers stdout and stderr which come from /proc/self/fd/1 and /proc/self/fd/2. I created a symlink from my php error log file /var/log/php_errors.log to /proc/self/fd/1 with command:
ln -sf /proc/self/fd/1 /var/log/php_errors.log
After linking the error log I have tested its functionality by running this php script:
<?php
error_log("This is a custom error message constructed to test the php error logging functionality of this site.\n");
?>
The output echos the error message to the console so I can see that php error logging is now redirected to stdout in the container, but when I run docker logs -f <containername> I never see the error message in the logs. Also echoing from inside the container doesn't show in the logs either which is confusing because my understanding is the echo command is stdout.
Further reading informed me that docker logs will only show output from pid 1 which could be the issue. If this is the case how can I correctly configure my php error logging to show in docker logs outside the container.
Also I have checked that I am using the default json-file docker driver, and have tried this both on my local environment and a web server.

Logstash: how to reset everything

I'm new to Logstash and Elasticsearch.
After some tests I want to restart and make a new index.
However, I'm getting this:
root#imageoffice:/etc/logstash/conf.d# sudo -Hu root /usr/share/logstash/bin/logstash --path.settings /etc/logstash/
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-04-22T17:00:04,910][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2019-04-22T17:00:04,965][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
How can I simple reset everything and run the sudo -Hu root /usr/share/logstash/bin/logstash --path.settings /etc/logstash/ command succesfull?
The error message tells you that there's already a logstash running on your machine. If you don't want the old one running, shut it down or kill it.
Typically, logstash is managed through a startup script (systemd, etc).
Removed the .lock file from bin/data and the error was gone.

How to redirect logs to /dev/null when app is running using pm2?

I start a nodejs app using pm2, the log out-file config setting is default.
Now, i find the log is too huge, so i need to redirect the log to /dev/null without restarting the process and without using pm2 -logrotate.
Is there any way around this issue?
You will have to restart the process if you want a fresh redirection. If restarting is okay, then it could be like:
pm2 'command' > /dev/null
If not, then you can possibly write a very basic shell script and schedule it via cron. It would just clean your logs (pm2 flush), or compress and dump (whichever you prefer).

ubuntu - disable logging to syslog from a specific process/from the rsyslog process

I have an ubuntu machine, and a server app running on it.
I also have rsyslog running which collects logs from the server app and write it to my own logs.
The issue I am having is that, it seems that the messages that are sent to rsyslog from the server app, are also written in var/logs/syslog.
How can I disable this? I don't want any of the server-app related messages that are sent to rsyslog to appear in var/logs/syslog.
Add a & stop; at the next line following the command that writes to your logfile.

node-apn logs is not getting stored in file

I need to debug the node-apn (some pushes are getting lost), so i need to analyze node-apn logs. Therefore, i want to store node-apn logs in a file.
What i have tried
I have enabled the node-apn logs and they are appearing on my console. Now, i am running following commands to start the server but i could not see node-apn logs in the file. I can see application logs (generated by winston) in file.
sudo node app.js test >> /home/gaurav/temp.txt
sudo node app.js test | tee /home/gaurav/temp.txt
Can anybody suggest how to achieve this.
The logs are probably being sent to stderr instead of stdout, so try this:
sudo node app.js test >> /home/gaurav/temp.txt 2>&1
sudo node app.js test 2>&1 | tee /home/gaurav/temp.txt
The 2>&1 redirects everything being sent to stderr to stdout.

Resources