Forever is truncating logs - node.js

I've been using forever for a long time but recently it started to behave a little bit weird: everything is ok except for the logs.
I used to run forever start /path/to/app.js and everything was fine. Even for huge logs (1-2 gigs).
But I currently have an app, a very busy web app and the log is being truncated every 3-4 hours. The size is not that big, 80-120 megs actually.
After realising this I decided to try (unsuccessfully though) starting forever with options: forever --append -o /path/to/out.log -e /path/to/error.log start /path/to/app.js but the problem persists.
I really don't know what to do.
Any thoughts??
Thanks!!
Versions:
node v0.12.7
npm 2.11.3
forever v0.15.1
3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux
UPDATE1:
BTW: I have plenty of hard disk an memory available
UPDATE2:
I found a related Issue (https://github.com/foreverjs/forever/issues/106#issuecomment-116933382) and beging testing with the following command: forever -a -l >(logger -t fileteTrackchile) start /path/to/app.js So far so good, but it will store log information in /var/log/user.log, /var/log/syslog and /var/log/messages. It's the same information so it would be better to save it only once.
I'll leave it running for a couple of days and see if it works or not.
UPDATE3 (FINAL):
The problem had nothing to do with forever, nor winston. I didn't realise that the files were so big that the log viewer was only showing part of it. The confusing part is that the first line (past) was truncated by the viewer (OSX Console) what led me to think that the file itself was truncated.

Regarding your first approach, unexpected log truncation is often caused by log rotation. However, you didn't mention rotating your logs.
Your second approach is to send the logs to a syslog daemon using the logger client.
Your logging is then ending up in three logs due your syslog configuration.
Try this solution. Create a file named /etc/rsyslog.d/10-fileteTrackchile.conf.
In the file, add these lines:
$template JustMsg,"%msg:2:10000%\n"
if $programname == 'fileteTrackchile' then /var/log/fileteTrackchile.log;JustMsg
if $programname == 'fileteTrackchile' then stop
Then:
service rsyslog stop
service rsyslog start
Because the file name starts with 10, it will get run before other configuration.
The logic instructs the logging for your app be logged to a particular file, unmodified, and then stop further processing by rsyslog, so the logging doesn't end up in the three other files you mentioned.
You may also wish to create /etc/logrotate.d/fileteTrackchile with contents like this:
/var/log/fileteTrackchile.log
{
rotate 7
daily
missingok
notifempty
compress
sharedscripts
postrotate
service rsyslog reload >/dev/null 2>&1 || true
endscript
}
See man logrotate for the details of those log rotation options.

The problem had nothing to do with forever, nor winston. I didn't realise that the files were so big that the log viewer was only showing part of it. The confusing part is that the first line (past) was truncated by the viewer (OSX Console) what led me to think that the file itself was truncated.

Related

Stop emacs warning at startup

For some reason emacs started to display a warning at whenever my computer starts up. The warning looks like this: https://imgur.com/a/fWz1Nu1. The text-form of the error is:
Warning (initialization): An error occurred while loading '/home/username/.emacs':
error: No such directory found via CDPATH environment variable
To ensure normal operation, you should investigate and remove the cause of the error in your initialization file. Start Emacs with the '--debug-init' option to view a complete error backtrace.
It seems, from the error, emacs is having trouble finding the ~/.emacs config file.
This warning is incorrect, however, since after this emacs normally works fine for me without any error, after this error message at start-up. It is however a mild inconvenience.
If I had to guess why it's happening I would say it might be somehow related to the way my fstab has been set up. It's slightly different from the norm but nothing too unusual. The setup is like this (in human language):
/ is mounted on a normal partition.
/home is mounted on a LUK encrypted partition. (One thing to note is that I have always had my /home encrypted but emacs only started to play up recently)
I also have two external drives that are auto-mounted to run my media server via my workstation (although I don't think this is related to the issue at hand)
Is there anyway that I can prevent emacs from displaying this in the future? I don't have emacs in my System 'Autostart' so I'm not sure why it keeps popping up at every boot.
UPDATE
I decided to look into the log from my last reboot:
$ journalctl -b | grep emacs
..and saw this:
Jul 21 14:30:46 my-PC-name systemd[3000]: Started apps-emacs-6284d6112ffa4ee29646bcb4b3b00b0f.scope.
Jul 21 14:30:54 my-PC-name systemd[3000]: apps-emacs-6284d6112ffa4ee29646bcb4b3b00b0f.scope: Succeeded.
Why would emacs starts itself when it's disabled as per below?
systemctl --user status emacs
● emacs.service - Emacs text editor
Loaded: loaded (/usr/lib/systemd/user/emacs.service; disabled; vendor preset>
Active: inactive (dead)
Docs: info:emacs
man:emacs(1)
https://gnu.org/software/emacs/
My emacs.service file doesn't look different from the default:
$ cat /usr/lib/systemd/user/emacs.service
[Unit]
Description=Emacs text editor
Documentation=info:emacs man:emacs(1) https://gnu.org/software/emacs/
[Service]
Type=simple
ExecStart=/usr/bin/emacs --fg-daemon
ExecStop=/usr/bin/emacsclient --eval "(kill-emacs)"
Environment=SSH_AUTH_SOCK=%t/keyring/ssh
Restart=on-failure
[Install]
WantedBy=default.target

Prevent forwarding of systemd service logs to syslog without affecting other service logs sent to syslog

My computer runs serveral java processes as systemd services.The systemd logs get accumulated in the syslog eventually leading to low disk space . How to re-direct the logs started by systemd services towards /dev/null so that it does not accumulate in syslog.The machine is constantly running out of disk space due to this issue.However , I need to be able to use journalctl to view the systemd service logs. The possible solutions I found were :
1.To modify configurations in /etc/systemd/journald.conf by setting
'ForwardToSyslog=no'
2.Adding StandardOutput=null within the systemd service file itself
However the first solution completely stopped all the logs sent to syslog and solution 2 did not work.I wish to stop forwarding only the log messages from systemd services.
The second option with StandardOutput=null should work. I think what you need is to redirect also STDERR to /dev/null, by adding StandardError=null.
Summarize - in your *.service file should be two lines:
[Service]
StandardOutput=null
StandardError=null
Refer SYSTEMD.EXEC(5) man page for more details.
Similarly on a Redhat 7.4 box, running systemd-219-42.el7_4.1.x86_64, I was unable to turn off log redirection with the ForwardToSyslog=no setting in /etc/systemd/journald.conf.
I instead had success with setting MaxLevelSyslog=warning which removed all the INFO and DEBUG level messages that were being forwarded to rsyslog.
You can configure rsyslog to ignore logs from specific application:
# cat /etc/rsyslog.d/mydaemon.conf
if $programname == 'mydaemon' then {
stop
}
This will result in:
You will see systemd-generated messages about daemon being started/stopped/reloaded/etc within journal AND syslog.
You will see daemon's-generated messages ONLY in journal (and custom log file written directly by application, if any).
Alternatively, you can use the journalctl command to limit the size of the journal. You can limit the size of the journal to a certain size, by adding the following line to /etc/systemd/journald.conf:
SystemMaxUse=10M
This will limit your journal to 10 MB.
You can still use "syslog" to redirect logs from the service to /dev/null. In your system file try:
[service]
SyslogIdentifier=<service-name>
StandardOutput=null
SyslogFacility=local7
Meanwhile, check that the syslog deamon is configured to receive logs with local7 facility. In rsyslog configuration file make sure that:
local7.* /dev/null
journalctl should still be working.
check out the systemd manual for more information
If your service is logging to a file, the messages will not be appended to systemd syslog.
For example:
[Service]
StandardOutput=append:/var/log/python-script-stdout.log
StandardError=append:/var/log/python-script-stderr.log
will forward all messages (stdout and stderr) only to their respective filepath.
If you want to truncate the logs on every service startup, replace append: with file:.

logrotate is not working

I did configuration inside logrotate.d for tomcat catalina.out and fixed the size to 200M below are my config file:
/opt/apache-tomcat-7.0.55/logs/catalina.out {
copytruncate
size 200M
rotate 20
compress
missingok
daily
}
but unfortunately it does not run even the log cross 900 MiB. I have to manually run the command logrotate -f tomcat. The manual execution works so I believe my configuration is right.
DO I need to reboot the box or anything need to be done to get this file executed.
I even tried by removing daily from the file but nothing changed.
This is first time I am configuring the logrotate.
Sounds like its not setup to run in the cron. http://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/ provides a good overview
Sorry to say I'm not sure what else it could be other than possibly file permission issues.

Error When Starting OmniEvents

I am attempting to install REDHAWK v1.8.2 on a fresh install of CentOS 6.4 32 bit, but I am unable to get omniNames and omniEvents to start.
sudo /sbin/service omniEvents stop
Stopping CORBA event service: omniEvents
sudo /sbin/service omniNames stop
Stopping omniNames [ OK ]
sudo /sbin/service omniNames start
Starting omniNames [ OK ]
sudo /sbin/service omniEvents start
Starting CORBA event service on port 11169: omniEvents: [25848]: Warning - failed to resolve initial reference 'NameService'. Exception: TRANSIENT
omniEvents.
I tried to verify if omniNames was really running by calling the naming client, but got an error (see below), so it seems omniNames is not successfully starting.
nameclt list
Caught a TRANSIENT exception when trying to validate the type of the
NamingContext. Is the naming service running?
As part of the debugging process, I tried to kill the omniNames process and start it a different way (see below).
sudo killall omniNames
omniNames -start
Wed Nov 13 21:08:08 2013:
Starting omniNames for the first time.
Error: cannot create initial log file '/var/omninames/omninames-orion.log':
No such file or directory
You can set the environment variable OMNINAMES_LOGDIR to specify the
directory where the log files are kept.
I'm not sure why omniNames can't create the log file, because I verified that /var/omninames folder actually exists and even starting omniNames as root yields the same error. Regardless, I set the log directory to my desktop to circumvent the error (see below).
export OMNINAMES_LOGDIR=/home/$USER/Desktop/logs
mkdir -p /home/$USER/Desktop/logs
omniNames -start
Wed Nov 13 21:09:17 2013:
Starting omniNames for the first time.
Wrote initial log file.
Read log file successfully.
Root context is IOR:010000002b00000049444c3a6f6d672e6f72672f436f734e616d696e672f4e616d696e67436f6e746578744578743a312e30000001000000000000005c000000010102000a00000031302e322e382e333500f90a0b0000004e616d6553657276696365000200000000000000080000000100000000545441010000001c00000001000000010001000100000001000105090101000100000009010100
Checkpointing Phase 1: Prepare.
Checkpointing Phase 2: Commit.
Checkpointing completed.
Even though it looks like omniNames successfully started, when I open another terminal window and call the naming client, I get the same error as before (see below).
nameclt list
Caught a TRANSIENT exception when trying to validate the type of the
NamingContext. Is the naming service running?
The only modification I made in the /etc/omniORB.cfg file is to add the lines for InitRef (see below).
InitRef = NameService=corbaname::localhost
InitRef = EventService=corbaloc::localhost:1169/omniEvents
Also, I am not connected to the internet so my version of CentOS has not been updated from the base version, except for the boost libraries as recommended in Appendix J of the manual (http://sourceforge.net/projects/redhawksdr/files/redhawk-doc/1.9.0/REDHAWK_Manual_v1.9.0.pdf/download).
Looks like the issue is in your configuration. You've got the wrong port in your configuration file. It should be port 11169 however you've listed port 1169.
See: http://redhawksdr.github.io/Documentation/mainch2.html#x4-120002.6 for details.
A few other observations and tricks regarding omniOrb in case this was not the issue.
Sometimes omninames/omnievents can get into a bad state. The fix is to delete the log files created by omniNames and omniEvents and restart the services. They are located:
/var/lib/omniEvents/*
/var/omniNames/*
You'll need to be root to delete those files. I always forget where they are located and often do a "locate omni | grep -i log" to remind myself but you must do this as root since they are not visible to standard users.
While it should not matter, I've personally found that using 127.0.0.1 is more reliable than localhost. For some reason, using localhost within a VM in the configuration file has caused me problems in the past. Consider using 127.0.0.1 instead of localhost. This is what the current version of the Redhawk Manual recommends as well.
You mentioned you are using Redhawk v1.8.2. As an FYI, the latest REDHAWK version in the 1.8 series is currently v1.8.5 and 1.9.0 was also recently released.
Hopefully this gets you up and running!

Tomcat 6 log4j - linux - safely remove catalina.out

Adding log4j [1] in tomcat 6.0.x forces tomcat to produce logs in "catalina" file. However, the default catalina.out is still produced and populated with logs. So, questions:
Is it safe to delete catalina.out file (while server running)?
If yes, could this deletion be added to tomcat startup script? If yes, could anyone please point out the file and the required script?
Is it possible for tomcat to stop createing the catalina.out, since it is not nessesary anymore?
http://tomcat.apache.org/tomcat-6.0-doc/logging.html
Thanks in advance people!
Tomcat redirects its stdout and stderr to catalina.out. So direct out/err writes and log4j ConsoleAppender messages will go to catalina.out. See catalina.sh file for details. To disable it completely you can redirect stdout and stderr to /dev/null setting CATALINA_OUT environment variable:
export CATALINA_OUT=/dev/null
But I recommend to disable ConsoleAppender instead to reduce catalina.out size and monitor it periodically looking for error messages, that may go to stdout bypassing log4j.
If the catalina.out is deleted after tomcat is stopped, it will create a new catalina.out once tomcat starts again and it is totally safe.
But if you remove the catalina.out while tomcat is running, it will keep on logging to catalina.out which is removed already (reference of the file is hold by the tomcat) hence the space will not be released. So you will need to restart the tomcat sever to release the space. It is not recommended.
You can try following to disable writing to catalina.out :
Locate and Edit File: {CATALINA_BASE}/bin/catalina.sh
Locate "CATALINA_OUT" and replace the path with "/dev/null":
CATALINA_OUT="$CATALINA_BASE"/logs/catalina.out // Original Location
CATALINA_OUT=/dev/null // replace path to /dev/null
Is there a reason you'd want to delete the catalina.out file? It seems like it might lead to potentially missing important event messages. Perhaps consider just setting
org.apache.catalina.level=INFO
Otherwise I can't think of a reason that it would negatively impact the functionality of the container if you deleted it. On a *nix install its still writing to the file descriptor of a file that is diconnected from an inode(so otherwise unreachable) and on windows it won't let you delete it because the container will have a file lock.
logging docs - a quick rtfm and it looks like you should just be able to remove the handlers from logging.properties to discontinue producing this file

Resources