I am working on a very old Nodejs application which creates a new child process using forever-monitor. The logs of this child process are taken care by forever-monitor only. This is how the configuration looks like:
var child = new (forever.Monitor)(__dirname + '/../lib/childprocess.js', {
max: 3,
silent: true,
options: [program.port],
'errFile': __dirname + '/../childprocess_error.log',
'outFile': __dirname + '/../childprocess_output.log'
}
);
Everything is working fine in this setup. The new requirement is to rotate these logs every 12 hours. That is every 12 hours a new file will be created which will have all the content of this file childprocess_output.log and should be stored in some other directory. The new log file will obviously have the timestamp appended at the end of the name (eg: childprocess_output_1239484034.log).
And the original file childprocess_output.log should be reset, that is all its content should be deleted and it should start logging from fresh.
I am trying to understand which npm library should I used for this purpose. I googled a bit and found a few of the npm libraries which matches my requirement, but the number of downloads for these libraries was really small, so I doubt the reliability of those libraries.
Which library NodeJs developers use for log rotation?
Also, my last resort would be to use the Linux tool Logrotate if I couldn't find any appropriate library in Node. I am avoiding using Logroate because I want my application to handle the scenario and not depend on the instance configuration.
you can use :
fs (the file system library) handled with methods like statSync and renameSync coupled with try-catches block-codes.
Related
I'm trying to use Tail (https://www.npmjs.com/package/tail) to export Minecraft server log data to Discord (The discord bot part works, so I have excluded it from here).
If I say something in the game and then check "latest.log", it has been changed accordingly. However, using this script, the bot only sees a change if I open "latest.log" in notepad, it doesn't work otherwise. The bot will recognize changes as long as "latest.log" is open in the background, which is an annoyance but not too big of a deal.
However, my friend is the one who I was making this for, and for him Tail only updates the moment he opens "latest.log". Which means he would need to keep opening up that file for Tail to see it, instead of just letting it run in the background.
Tail = require('tail').Tail;
var fileToTail = "C:/Users/user/Downloads/logs/latest.log";
tail = new Tail(fileToTail);
tail.on("line", function(data) {
//Working code that sends data
});
tail.on("error", function(error) {
console.log('ERROR: ', error);
});
What could be causing the discrepancy between the two of us, and what can I do so that the bot can see the file changes without the user opening the file? Thanks in advance!
If you are using chokidar, you should pay attention to whether you are using fs.watch Vs. fs.watchFile. If using fs.watch you not successfully catch changes (which might be what you are experiencing).
See below from official docs for chokidar options:
usePolling (default: false). Whether to use fs.watchFile (backed by
polling), or fs.watch. If polling leads to high CPU utilization,
consider setting this to false. It is typically necessary to set this
to true to successfully watch files over a network, and it may be
necessary to successfully watch files in other non-standard
situations. Setting to true explicitly on MacOS overrides the
useFsEvents default. You may also set the CHOKIDAR_USEPOLLING env
variable to true (1) or false (0) in order to override this option.
Is there a way to make a limited require in wich files require()'d by VM programs are also run inside the VM?
Is it safe to pass a function to the script wich reads a file and then it returns it a VM instance of the code.
My situation is that I need a sandboxed file that is kind-of trusted to start up the enviroment by requiring all the other files. This should not be done by passing them in context because those files have to be required when the file wants, since those required files might change.
The required files are not really trusted, so normal require() is dangerous. As only way code has to mess up is that.
So: init.js calls files/core1.js, files/core2.js and files/whatever_it_wants.js
What I use:
var sandbox = {
fs: vfs,
write: console.log,
files: f,
exit: process.exit
}
var context = new vm.createContext(sandbox);
var script = new vm.Script(fs.readFileSync("./e/master.js"),{timeout: 10000});
script.runInContext(context);
master.js needs to load files that are not very trustable, having files (contains all files available) and it's virtual filesystem for it to do its FS operations, and write/exit to do what it needs.
I am using the Bunyan module for NodeJS logging. When I try using the rotating-file type, it makes my app crash every time and outputs this error:
Error: ENOENT, rename 'logs/info.log.3'
However, it never happens at the same time so I can't find any logic...
This is how I instanciate my logger:
var log = Bunyan.createLogger(config.log.config);
log.info('App started, ' + process.env.NODE_ENV);
And here is my config.json:
{
"name" : "app",
"streams" : [
{
"type" : "rotating-file",
"period": "5000ms", //Low period is for testing purposes
"count" : 12,
"level" : "info",
"path" : "logs/info.log"
},
{
"type" : "rotating-file",
"period": "5000ms",
"count" : 12,
"level" : "error",
"path" : "logs/error.log"
},
{
"type" : "rotating-file",
"period": "5000ms",
"count" : 12,
"level" : "trace",
"path" : "logs/trace.log"
}
]
}
Can anyone advise how to fix my issue? Thanks in advance.
What I have just done (last night actually) to get around this problem of a master + workers contending over a Bunyan rotating-file is to have the workers write "raw" log records to a stream-like object I created called a WorkerStream. The write method of the WorkerStream simply calls process.send to use IPC to deliver the log record to the master. The master uses a different logger config that points to a rotating-file. The master uses the code shown below to listen for log records from its workers and write them to the log file. So far it appears to be working perfectly.
cluster.on('online', function (worker) {
// New worker has come online.
worker.on('message', function (msg) {
/* Watch for log records from this worker and write them
to the real rotating log file.
*/
if (msg.level) {
log._emit(msg);
}
});
});
ln is your friend.
Existing logging libraries have rotation problem with cluster module. Why doesn't ln have this issue?
Both bunyan and log4js rename the log file on rotation. The disaster happens on file renaming under cluster environment because of double files renaming.
bunyan suggests using the process id as a part of the filename to tackle this issue. However, this will generate too many files.
log4js provides a multiprocess appender and lets master log everything. However, this must have the bottleneck issue.
To solve this, I just use fs.createWriteStream(name, {"flags": "a"}) to create a formatted log file at the beginning instead of fs.rename at the end. I tested this approach with millisecond rotation under cluster environment and no disasters occurred.
I have experienced the same issue without using clustering. I believe the problem is being caused by old files sitting in the log directory. While the main logger can open and append to existing files, the file rotation logic uses rename, which files when it steps on an existing file. (e.g. an existing info.log.3 file).
I'm still digging into the source to figure out what needs to be changed to recover from the rolling error.
One additional thought as I review the source. If you have multiple Bunyan log instances that use the same log file (in my case, a common error.log), the rename calls could be happening nearly concurrently from the OS level (asynchronous and separate calls from a Node.js perspective, but concurrently from the OS perspective).
It's sadly not possible to use multiple rotating file streams against the same file.
If you're in the same process, you must use a single logger object - make sure you're not creating multiple of them.
If you're working across processes, you must log to different files. Unfortunately there's nothing yet that has the IPC in place to allow different rotators to coordinate amongst themselves.
I have a plugin rotating file stream that detects if you try to create 2 rotators against the same file in the a single process and throws an error.
It can't help in the case of multiple processes tho.
bunyan-rotating-file-stream
From my experience, it happens sometimes when the logs directory (or whatever you named it) does not exist.
If you are running through this error in a automation pipeline, for example, you may be ignoring all the files in logs and committing it empty, then it is not created when the repository is cloned by the pipeline.
Simply make sure that logs is created by placing a .gitkeep file inside it (or any other trick).
This may be the case of many of you who come across this question.
I am working on more of a security dashboard, it watches for changes in files in the entire home directory with hundreds of sites (all Joomla, so a lot of files).
In order to keep on top of potential security issues we want to watch for file changes in an efficient way without creating unnecessary CPU/Memory overhead. We want to watch it at a faster interval but I know its more of a balancing act when you do want to keep it from using more cpu then a side process should.
I have tried to use "watch" with the following code, running in the home directory:
var watch, fs;
watch = require('watch');
fs = require('fs');
watch.createMonitor(__dirname,{interval:500,filter:function(file,stat){
if(file.indexOf('index.php')!=-1){
return true;
}else{
return false;
}
}},function(monitor){
monitor.filter(function(file){
console.log(file);
})
monitor.on('created',function(file,stat){
console.log(file + ' new');
});
monitor.on('changed',function(file,stat){
console.log(file + ' changed');
});
monitor.on('removed',function(file,stat){
console.log(file + ' deleted');
});
});
However this spikes the CPU to over 100% of a single core (sometimes 2) out of 8. Memory also takes up about 20% of 8gb pretty quickly as well. This is all just to create the watch event on all the files, so its before it can actually detect any file changes.
I know the issue with this is it goes through each file individually, and only does not track it if you filter that sort of file. Typically all I need to watch is the index.php in every directory, down to a point that it could be consistent (with some exceptions).
Is there a module already built to do this? Or is this something new? All modules I find assume its a smaller directory (like watching LESS or something) So not built for this sort of application at all.
Any ideas? I know this code will need to be scrapped as there is no way I can see to stop the CPU overhead.
Do not use package 'watch', just use fs.watch(...)
package 'watch':
consistent APIs
very slow because implement mostly in node, look source to see how it work
souce code: https://github.com/mikeal/watch/blob/master/main.js
fs.watch(..)
non-consistent APIs, not all OSs are supported.
very fast because it reused OS features
document: http://nodejs.org/docs/latest/api/fs.html#fs_fs_watch_filename_options_listener
I am running play on multiple machines in our datacenter. We loadbalance the hell out of everything. On each play node/VM I'm using Apache and an init.d/play script to start and stop the play service.
The problem is that our play websites are hosted on shared network storage. This makes deployment really nice, you deploy to one place and the website is updated on all 100 machines. Each machine has a mapped folder "/z/www/PlayApp1" where the play app lives.
The issue is that when the service starts or stops the server.pid file is being written to that network location where the apps files live.
The problem is that as I bring up 100 nodes, the 100th node will override the PID file with it's pid and now that pid file only represents the correct process ID for 1 out of 100 nodes.
So how do I get play to store the pid file locally and not with the app files on the network share? I'll need each server's PID file to reflect that machines actual process.
We are using CentOS (Linux)
Thanks in advance
Josh
According to https://github.com/playframework/play/pull/43 it looks like there is a --pid_file command line option; it might only work with paths under the application root so you might have to make directories for each distinct host (which could possibly be symlinks)
I have 0 experience with Play so hopefully this is helpful information.
I don't even think it should run a second copy, based on the current source code. The main function is:
public static void main(String[] args) throws Exception {
File root = new File(System.getProperty("application.path"));
if (System.getProperty("precompiled", "false").equals("true")) {
Play.usePrecompiled = true;
}
if (System.getProperty("writepid", "false").equals("true")) {
writePID(root);
}
:
blah blah blah
}
and writePID is:
private static void writePID(File root) {
String pid = ManagementFactory.getRuntimeMXBean().getName().split("#")[0];
File pidfile = new File(root, PID_FILE);
if (pidfile.exists()) {
throw new RuntimeException("The " + PID_FILE + " already exists. Is the server already running?");
}
IO.write(pid.getBytes(), pidfile);
}
meaning it should throw an exception when you try to run multiple copies using the same application.path.
So either you're not using the version I'm looking at or you're discussing something else.
It seems to me it would be a simple matter to change that one line above:
File root = new File(System.getProperty("application.path"));
to use a different property for the PID file storage, one that's not on the shared drive.
Although you'd need to be careful, root is also passed to Play.int so you should investigate the impact of changing it.
This is, after all, one of the great advantages of open source software, inasmuch as you can fix the "bugs" yourself.
For what it's worth, I'm not a big fan of the method you've chosen for deployment. Yes, it simplifies deployment but upgrading your servers is an all-or-nothing thing which will cause you grief if you accidentally install some dodgy software.
I much prefer staged deployments so I can shut down non-performing nodes as needed.
Change your init script to write the pid to /tmp or somewhere else machine-local.
If that is hard, a symlink might work.