I'm using node.js with morgan as logger like this:
// create a rotating write stream
var accessLogStream = require("stream-file-archive")({
path: "logs/app-%Y-%m-%d.log", // Write logs rotated by the day
symlink: "logs/current.log", // Maintain a symlink called current.log
compress: true // Gzip old log files
});
app.use(logger('combined', {stream: accessLogStream}));
and I wanted to know how to limit the max file size of the access.log.
Thanks
Related
My application is simple and I want to avoid using a logging library like Winston. I need to log the output to both the console, and to file. I found a few tutorials on how to do this using a child process, such as this, but I can't find anything that leverages the main process stdio, like process.stdout and process.stdin
The key to solving this was recognizing that process.stdio is a writable stream whereas a child process's stdio using the child_process module is a readable stream (thanks to this article). Therefore I needed to create both a readable and writable file stream, and pipe the readable stream out to process.stdio. You could probably simplify this even further with a duplex stream, but for noobs like myself, this is a straightforward and easy to read approach.
const { Console } = require("console")
, process = require("process")
, path = require("path")
, fs = require('fs');
// Define the file paths to log to
const outputFilePath = path.join(__dirname, './stdout.log');
const errorFilePath = path.join(__dirname, './stderr.log');
// Create the empty files synchronously to guarantee it exists prior to stream creation.
// Change flag to 'w' to overwrite rather than append.
fs.closeSync(fs.openSync(outputFilePath, 'a+'));
fs.closeSync(fs.openSync(errorFilePath, 'a+'));
// Create a writable file stream for both stdout and stderr
const fileWriterOut = fs.createWriteStream(outputFilePath);
const fileWriterErr = fs.createWriteStream(errorFilePath);
// Create a new Console object using the file writers
const Logger = new Console({ stdout: fileWriterOut, stderr: fileWriterErr });
// Create readable file streams for process.stdio to consume
const fileReaderOut = fs.createReadStream(path.join(__dirname, './stdout.log'));
const fileReaderErr = fs.createReadStream(path.join(__dirname, './stderr.log'));
// Pipe out the file reader into process stdio
fileReaderOut.pipe(process.stdout);
fileReaderErr.pipe(process.stderr);
// Test the new logger
Logger.log("Logger initialized");
// Export
module.exports = Logger;
I'm new to Node.js and Express.
I want to use log4js but not certain about in which file I should configure my logger.
Is there a conventional file for the initialization? If not, where should I create a new configuration file?
Thanks :)
Answer (based on #jfriend00 answer)
In logger.js
'use strict';
var log4js = require('log4js');
log4js.configure({
"appenders": [...]
});
var logger = log4js.getLogger("structuredLogger");
module.exports = logger
In client.js
var logger = require('../../../../../config/logger.js');
logger.info('My message');
This module will allow me to:
Easily configure the the log4js
Easily replace the log4js with another package.
One common option for a module that needs to get initialized once is to create your own container module that does the initialization. Then, every other module that wants to use the logging can just load your container module and, if not already initialized, your container module will initialize the logging.
// mylog.js
// initialization code will only be called the first time the module is loaded
// after that, the module is cached by the `require()` infrastructure
var log4js = require('log4js');
log4js.configure({
appenders: [
{ type: 'console' },
{ type: 'file', filename: 'logs/cheese.log', category: 'cheese' }
]
});
module.exports = log4js;
Then, every module that wishes to use the common configuration logging can just do this near the top of the module:
var log4js = require('./mylog.js');
I can not find in my app directory where is 'test.log' file?
below code is in server.js
var winston = require('winston'), mylogger = new (winston.Logger)({
transports: [
new (winston.transports.Console) (),
new (winston.transports.File) ({filename: 'test.log'})
]
});
mylogger.log('Hello world');
my app directory:
/
app/
config/
public/
server.js
Interesting question. Looking at the source code for the file transport, it appears that the directory, if not specified, is derived from the filename parameter itself. It seems to suggest that you either use an absolute path or relative path with a directory explicitly specified.
This line is where it figures out the absolute path.
var fullname = path.join(self.dirname, target);
self.dirname is setup here:
this.dirname = options.dirname || path.dirname(options.filename);
so the question is that if options.filename does not include a directory, what does path.dirname return?
I do not actually know, but there are two possibilities I would suspect:
Current working directory of the process
The file system root, because if path.dirname takes what is left of the last /, then it is undefined and undefined + '/test.log' is '/test.log'
There are two steps you can take:
Check the current directory and file system root to see which it is. (in other words, test the theories)
Specify the directory explicitly (probably a good idea anyway)
https://github.com/flatiron/winston/blob/master/lib/winston/transports/file.js
I wrote a test file like this:
var winston = require('winston');
var logger = new (winston.Logger)({
transports: [
new (winston.transports.Console)(),
new (winston.transports.File)({ filename: 'somefile.log' })
]
});
logger.log('info', 'Hello distributed log files!');
logger.info('Hello again distributed logs');
process.exit();
and I found that if I deleted the last line, the log file would be created and logged correctly.
I used process.exit() to break my program as I just wanted to test the logger, and it looked like it somehow broke the process on log file as well.
Check this issue: https://github.com/winstonjs/winston/issues/228
In my web analytics, I am logging the data in plain text file. I want to rotate the log on a daily basis because its logging too much data. Currently I am using bunyan to rotate the logs.
Problem I am facing
It is rotating the file correctly, but rotated log file are in the name log.0, log.1, etc. I want the file name to be log.05-08-2013, log.04-08-2013
I can't edit the source of the bunyanpackage because we are installing the modules using package.json via NPM.
So my question is - Is there any other log rotation in Node.js that meets my requirement?
Winston does support log rotation using a date in the file name. Take a look at this pull request which adds the feature and was merged four months ago. Unfortunately the documentation isn't listed on the site, but there is another pull request pending to fix that. Based on that documentation, and the tests for the log rotation features, you should be able to just add it as a new Transport to enable the log rotation functionality. Something like the following:
winston.add(winston.transports.DailyRotateFile, {
filename: './logs/my.log',
datePattern: '.dd-MM-yyyy'
});
If you also want to add logrotate (e.g. remove logs that are older than a week) in addition to saving logs by date, you can add the following code:
var fs = require('fs');
var path = require("path");
var CronJob = require('cron').CronJob;
var _ = require("lodash");
var logger = require("./logger");
var job = new CronJob('00 00 00 * *', function(){
// Runs every day
// at 00:00:00 AM.
fs.readdir(path.join("/var", "log", "ironbeast"), function(err, files){
if(err){
logger.error("error reading log files");
} else{
var currentTime = new Date();
var weekFromNow = currentTime -
(new Date().getTime() - (7 * 24 * 60 * 60 * 1000));
_(files).forEach(function(file){
var fileDate = file.split(".")[2]; // get the date from the file name
if(fileDate){
fileDate = fileDate.replace(/-/g,"/");
var fileTime = new Date(fileDate);
if((currentTime - fileTime) > weekFromNow){
console.log("delete fIle",file);
fs.unlink(path.join("/var", "log", "ironbeast", file),
function (err) {
if (err) {
logger.error(err);
}
logger.info("deleted log file: " + file);
});
}
}
});
}
});
}, function () {
// This function is executed when the job stops
console.log("finished logrotate");
},
true, /* Start the job right now */
'Asia/Jerusalem' /* Time zone of this job. */
);
where my logger file is:
var path = require("path");
var winston = require('winston');
var logger = new winston.Logger({
transports: [
new winston.transports.DailyRotateFile({
name: 'file#info',
level: 'info',
filename: path.join("/var", "log", "MY-APP-LOGS", "main.log"),
datePattern: '.MM--dd-yyyy'
}),
new winston.transports.DailyRotateFile({
name: 'file#error',
level: 'error',
filename: path.join("/var", "log", "MY-APP-LOGS", "error.log"),
datePattern: '.MM--dd-yyyy',
handleExceptions: true
})
]});
module.exports = logger;
There's the logrotator module for log rotation that you can use regardless of the logging mechanism.
You can specify the format option to format the date format (or any other format for that matter)
var logrotate = require('logrotator');
// use the global rotator
var rotator = logrotate.rotator;
// or create a new instance
// var rotator = logrotate.create();
// check file rotation every 5 minutes, and rotate the file if its size exceeds 10 mb.
// keep only 3 rotated files and compress (gzip) them.
rotator.register('/var/log/myfile.log', {
schedule: '5m',
size: '10m',
compress: true,
count: 3,
format: function(index) {
var d = new Date();
return d.getDate()+"-"+d.getMonth()+"-"+d.getFullYear();
}
});
mongodb
winston itself does not support log rotation. My bad.
mongodb has a log rotation use case. Then you can export the logs to file names per your requirement.
winston also has a mongodb transport but I don't think it supports log rotation out of the box judging from its API.
This may be an overkill though.
forking bunyan
You can fork bunyan and add your repo's url in package.json.
This is the easiest solution if you're fine with freezing bunyan's feature or maintaining your own code.
As it is an open source project, you can even add your feature to it and submit a pull request to help improve bunyan.
I am using Winston logging with my Node.js app and have defined a file transport. Throughout my code, I log using either logger.error, logger.warn, or logger.info.
My question is, how do I specify the log level? Is there a config file and value that I can set so that only the appropriate log messages are logged? For example, I'd like the log level to be "info" in my development environment but "error" in production.
If you are using the default logger, you can adjust the log levels like this:
const winston = require('winston');
// ...
winston.level = 'debug';
will set the log level to 'debug'. (Tested with winston 0.7.3, default logger is still around in 3.2.1).
However, the documentation recommends creating a new logger with the appropriate log levels and then using that logger:
const myLogger = winston.createLogger({
level: 'debug'
});
myLogger.debug('hello world');
If you are already using the default logger in your code base this may require you to replace all usages with this new logger that you are using:
const winston = require('winston');
// default logger
winston.log('debug', 'default logger being used');
// custom logger
myLogger.log('debug', 'custom logger being used');
Looks like there is a level option in the options passed covered here
From that doc:
var logger = new (winston.Logger)({
transports: [
new (winston.transports.Console)({ level: 'error' }),
new (winston.transports.File)({ filename: 'somefile.log' })
]
});
Now, those examples show passing level in the option object to the console transport. When you use a file transport, I believe you would pass an options object that not only contains the filepath but also the level.
That should lead to something like:
var logger = new (winston.Logger)({
transports: [
new (winston.transports.File)({ filename: 'somefile.log', level: 'error' })
]
});
Per that doc, note also that as of 2.0, it exposes a setLevel method to change at runtime. Look in the Using Log Levels section of that doc.
There are 6 default levels in winston: silly=0(lowest), debug=1, verbose=2, info=3, warn=4, error=5(highest)
While creating the logger transports, you can specify the log level like:
new (winston.transports.File)({ filename: 'somefile.log', level: 'warn' })
Above code will set log level to warn, which means silly, verbose and info will not be output to somefile.log, while warn, debug and error will.
You can also define your own levels:
var myCustomLevels = {
levels: {
foo: 0,
bar: 1,
baz: 2,
foobar: 3
}
};
var customLevelLogger = new (winston.Logger)({ levels: myCustomLevels.levels });
customLevelLogger.foobar('some foobar level-ed message');
Note that it's better to always include the 6 predefined levels in your own custom levels, in case somewhere used the predefined levels.
You can change the logging level in runtime by modifying the level property of the appropriate transport:
var log = new (winston.Logger)({
transports: [
new (winston.transports.Console)({ level : 'silly' })
]
});
...
// Only messages with level 'info' or higher will be logged after this.
log.transports.Console.level = 'info';
I guess, it works similarly for file but I haven't tried that.
If you want to change the log level on the fly. Like for when you need to trace production issue for short amount of time; then revert to error log level. You can use a dynamic logger provided you can expose a service on the web https://github.com/yannvr/Winston-dynamic-loglevel
apart from this you can cleanly achieve this by imlplementing runtime-node-refresh follow this link for more.