I have an ubuntu machine, and a server app running on it.
I also have rsyslog running which collects logs from the server app and write it to my own logs.
The issue I am having is that, it seems that the messages that are sent to rsyslog from the server app, are also written in var/logs/syslog.
How can I disable this? I don't want any of the server-app related messages that are sent to rsyslog to appear in var/logs/syslog.
Add a & stop; at the next line following the command that writes to your logfile.
Related
I have a website with Nginx installed as a reserve proxy for an ExpressJS server (proxies to port 3001). This uses Node and ReactJS for my frontend application.
This is simply a testing website currently, and isn't known or used by any users. I have this installed on a Digital Ocean Droplet with Ubuntu.
Every morning when I wake up, I load my website and see 502 Bad Gateway. The problem is, I don't know how to find out how this happened. I have PM2 installed which should automatically restart my ExpressJS server but it hasn't done so, and when I run pm2 list, my application is still showing online:
When I run pm2 logs, I get the following error (I am running this as an Administrator):
So I'll run pm2 restart all to restart the app, but then I don't see any crash information. However on this occasion when taking this screenshot, there were a couple of unusual requests. /robots.txt, /sitemap.xml and /.well-known/security.txt, but nothing indicating a crash:
When I look at my Nginx error.log file, all I can see is the following:
There is, however, something obscure within my access.log ([09/Oct/2018:06:33:19 +0000]) but I have no idea what this means:
If I run curl localhost:3001 whilst the server is offline, I will receive a connection error message. This works fine after I run pm2 restart all.
I'm completely stuck with this and even the smallest bit of help would be appreciated greatly, even if it's just to tell me I'm barking up the wrong tree completely and need to look elsewhere - thank you.
I think you should check this github thread, it seems like it could help you.
Basically, after few hours, a Nodejs server stop functioning, and the poor nginx can not forward its requests, as the service listening to the forward port is dead. So it triggers a 502 error.
It was all due to a memory leak, that leads to a massive garbage collection, then to the server to crash. Check your memory consumption, you could have some surprises. And try to debug your app code, a piece (dependency) at the time.
Updated answer:
So, i will add another branch to my question as it seems it has not helped you so far.
You could try to get rid of pm2, and use systemd to manage your app life cycle.
Create a service file
sudo vim /lib/systemd/system/appname.service
this is a simple file i used myself for a random ExpressJS app:
[Unit]
Description=YourApp Site Server
[Service]
ExecStart=/home/user/appname/index.js
Restart=always
Environment=PATH=/usr/bin:/usr/local/bin
Environment=NODE_ENV=production
WorkingDirectory=/home/user/appname
[Install]
WantedBy=multi-user.target
Note that it will try to restart if it fails somehow Restart=always
Manage it with systemd
Register the new service with:
sudo systemctl daemon-reload
Now start your app from systemd with:
sudo systemctl start appname
from now on you should be able to manage your app life cycle with the usual systemd commands.
You could add stdout and stderr to syslog to understand what you app is doing
StandardOutput=syslog
StandardError=syslog
Hope it helps more
You cannot say when exactly NodeJS will crash, or will do big GC, or will stun for other reason.
Easiest way to cover such issues is to do health check and restart an app. This should not be an issue when working with cluster.
Please look at health check module implementation, you may try to use it, or write some simple shell script to do the check
So I am putting my app on a server and I want to see some console logs generated from my nodejs app. Before putting it on the server I saw the output of my console log on my terminal, but where do I find the output for these on Ubuntu? I'm running version 16.04 on Ubuntu
Lifted straight from #SlovyanskiyYehor comment two years ago, but I missed his answer as a comment. Also answered here: Make pm2 log to console.
If you are using pm2 to start your node.js application, you can use:
pm2 logs
or for just a single app running in pm2:
pm2 logs yourAppNameHere
If you are using some bash script which is runed by cron for example,
you can specify file where you need to output all logs of your app
It will looks like that:
myApp > /path/to/logs/myAppLogs.txt
This command will run application myApp and put all logs into
"/path/to/logs/myAppLogs.txt"
Also you can put this string into cron directly without some bash scripts
Of course you can see all console logs by user here ~/.bash_history
but it`s not always useful
If you used pm2, you can check log in this folder:
/root/.pm2/log
I am launching a new instance through AWS and I am using Puppet as the configuration management tool. I am using ELK Stack for the analytic purposes. So I am configuring Puppet in such a way that whenever I'm launching a new server, it will automatically install Logstash. So everything is setup properly now, I can see the Logstash service running, but the problem is that the logs are not getting shipped to ElasticSearch.
And there is more to it.
If I run the following command manually, I can see the logs getting shipped to elasticsearch.
/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf
I installed and configured kibana in my local machine and I am getting event log from 3 different machine.
As log forwarder I am using nxlog which is lightweight so nice to install on production without affecting performance.
One thing I need is NSSM - the Non-Sucking Service Manager from http://nssm.cc/ which is blocked by MacAfee. I need it to install all component as window service.
Can anybody suggest Other alternative to install kibana - logstash as window service ?
Update - I try to user apache common daemon to install service but service failed to start saying error code 5 which I guess Access denied.
I am using following command to install
prunsrv //IS//logstash-shipper --DisplayName="logstash-shipper " --Install="C:\Jitendra\ELK\Apache_Commons_Daemon-1.0.15-windows\prunsrv.exe" --StartMode=exe --StopMode=exe --StartImage="C:\Jitendra\ELK\logstash\logstash-1.5.4\bin\logstash.bat agent -f logstash.conf" --LogPath="C:\Jitendra\ELK\logstash\logstash-1.5.4\bin" --PidFile="C:\Jitendra\ELK\logstash\logstash-1.5.4\bin\shipper.pid" --StopPath="C:\Jitendra\ELK\logstash\logstash-1.5.4\bin" --StopImage="shipper-stop.bat"
Can anyone used this component ?
Try this
"C:\Windows\Microsoft.NET\Framework\v4.0.30319\installutil.exe" "c:\"(path to bat files)
or
Allow exception in MacAfee for NSSM.exe
I have a linux server that contains many websites under vhosts, once I deleted an alt site the apache2 coud not be restarted indecating this message in the error log:
#eror log
(2)No such file or directory: apache2: could not open error log file /var/www/vhosts/deleted_Site/logs/error.log.
Unable to open logs
How can I stop that so I can start my apache2
You should modify your apache configuration, for instance in conf.d/, all configuration related to your old website, especially for error logs (look for access logs and errors logs directives specification, somewhere in your conf files, apache2.conf, httpd.conf, sites-enabled, mods or what/wherever it is).
To resolve this error, you need to stop other process using same port. In some of cases it is nginx. you can stop it
sudo /etc/init.d/nginx stop
Turn off any servers using the same port. For example to turn off nginx try this:
sudo systemctl stop nginx