I wrote a Lambda function to send Cloudfront logs into Elasticsearch.
The worklow is as follow:
1. Cloudfront send logs (compressed into .gz format) into S3
2. The Bucket send a notification which is caught by the Lambda function
3. The Lambda is triggered. Decompress the logs and send them into Elasticsearch.
I use for this s3-to-logstore combined with winston-parser.
The Lambda is indeed triggered, but only one part of the logs is sent to Elasticsearch, because the Lambda function times out (I set the timeout to the max: 5min).
I suspect decompressing the .gz logs take some time, but it's at most 30 KB which is not much and should not take long.
I was inspired by this example, and here is my function:
var s3ToLogstore = require('s3-to-logstore');
var winston = require('winston');
require('winston-elasticsearch');
var elasticsearch = require('elasticsearch');
var client = new elasticsearch.Client({
host: process.env.ES_HOST,
log: 'trace'
});
var transport = new winston.transports.Elasticsearch({
indexPrefix: process.env.ES_INDEXPREFIX,
client: client
});
var options = {
format: process.env.FORMAT,
transport: transport,
reformatter: function(data){
data.environment = process.env.STAGE;
data.origin = process.env.FORMAT;
return data;
}
};
exports.handler = s3ToLogstore(options);
The cloudwatch logs are totally fine, and there is no error in them. The Lambda just times out, and I can't figure out why.
Any help would be greatly appreciated.
Most likely the elasticsearch client keeps a connection open, so the lambda never stops. Try setting keepAlive property to false.
var client = new elasticsearch.Client({
host: process.env.ES_HOST,
log: 'trace',
keepAlive: false
});
Refer to keepAlive and related properties.
https://www.elastic.co/guide/en/elasticsearch/client/javascript-api/current/configuration.html
I have been using #google-cloud/logging-winston module for logging in appengine flexible environment.
But recently I noticed no logs are printing under 'winston_log' logName, its only printing in stout, following is my configuration:
const logLevel = 'info'
const transports = [new Console({level: logLevel, colorize: isDevMode(), timestamp: isDevMode()})]
transports.push(LoggingWinston({level: logLevel}))
// application logger
const logger = new winston.Logger({ transports: transports })
logger.error('error test') // no logLevel
logger.info('app listening on port 8080') // no logLevel
logger.debug('debug test') // no logLevel
Nothing is printing under 'winsont_log' logName, all logs are printed only in console like below:
Any help?
Currently, you are using console transport that is why you are getting logs inside the console window. Instead, use File transport to write logs in the log file.
I'm writing a rest api with NodeJS and express and I'm using express-winston to log accesses and erros. But I want to separate de log daily. Like in this post
I'm trying to do so with winston.transports.DailyRotateFile. A piece of code below.
api.use(expressWinston.logger({
transports: [
new winston.transports.DailyRotateFile({
name: 'file',
datePattern: '.yyyy-MM-ddTHH',
filename: path.join(__dirname, "log-access", "log_file.log")
})
]
}));
Then I receive the error: winston.transports.DailyRotateFile is not a function
I guess I have to install another pack, since reading winston's doc I found that you can write custom transports.
Would you have information on which package I would have to install? I found some that doesn't match or was discontinued.
Thanks for any help
I had to do this so it would work:
var winston = require('winston'), expressWinston = require('express-winston');
winston.transports.DailyRotateFile = require('winston-daily-rotate-file');
I already had the right package but it wouldn't work, until I wrote the lines above.
You do not need to assign:
var winston = require('winston'), expressWinston = require('express-winston');
winston.transports.DailyRotateFile = require('winston-daily-rotate-file');
You just need to require it:
const winston = require('winston');
require('winston-daily-rotate-file');
What you're looking for is this module.
Just follow the documentation, and you're good to go.
I am using bunyan for logging on my node.js API which is hosted in aws. When I load test the API, it falls over after 4000 file descriptors. We found that the open file descriptors on the app increase with each request. The code to configure the bunyan logger is given below
var defaultLogFileName = "/var/log/test_api.log",
defaultLogLevel = "debug";
var logger = bunyan.createLogger({
name: 'resumes',
streams: [{
path: process.env.LogFileName || defaultLogFileName,
level: process.env.LogLevel || defaultLogLevel
}]
});
app.on('after', function(req) {
logger.info("response = ", res);
}
We are running this API in a docker container hosted on ECS and using restify.
Has the bunyan logger been implemented correctly? Is there anything that I have missed?
Is there any library which will help me to handle logging in my Node.Js application? All I want to do is, I want to write all logs into a File and also I need an options like rolling out the file after certain size or date.
I have incorporated log4js im trying to keep all the configuration details in one file and use only the methods in other application files for ease of maintenance. But it doesnt work as expected. Here is what I'm trying to do
var log4js = require('log4js');
log4js.clearAppenders()
log4js.loadAppender('file');
log4js.addAppender(log4js.appenders.file('test.log'), 'test');
var logger = log4js.getLogger('test');
logger.setLevel('ERROR');
var traceLogger = function (message) {
logger.trace('message');
};
var errorLogger = function (message) {
logger.trace(message);
};
exports.trace = traceLogger;
exports.error = errorLogger;
I have included this file in other files and tried
log.error ("Hello Error Message");
But it is not working. Is there anything wrong in this ?
Winston is a pretty good logging library. You can write logs out to a file using it.
Code would look something like:
var winston = require('winston');
var logger = new (winston.Logger)({
transports: [
new (winston.transports.Console)({ json: false, timestamp: true }),
new winston.transports.File({ filename: __dirname + '/debug.log', json: false })
],
exceptionHandlers: [
new (winston.transports.Console)({ json: false, timestamp: true }),
new winston.transports.File({ filename: __dirname + '/exceptions.log', json: false })
],
exitOnError: false
});
module.exports = logger;
You can then use this like:
var logger = require('./log');
logger.info('log to file');
Scribe.JS Lightweight Logger
I have looked through many loggers, and I wasn't able to find a lightweight solution - so I decided to make a simple solution that is posted on github.
Saves the file which are organized by user, date, and level
Gives you a pretty output (we all love that)
Easy-to-use HTML interface
I hope this helps you out.
Online Demo
http://bluejamesbond.github.io/Scribe.js/
Secure Web Access to Logs
Prints Pretty Text to Console Too!
Web Access
Github
https://github.com/bluejamesbond/Scribe.js
Log4js is one of the most popular logging library for nodejs application.
It supports many cool features:
Coloured console logging
Replacement of node's console.log functions (optional)
File appender, with log rolling based on file size
SMTP, GELF, hook.io, Loggly appender
Multiprocess appender (useful when you've got worker processes)
A logger for connect/express servers
Configurable log message layout/patterns
Different log levels for different log categories (make some parts
of your app log as DEBUG, others only ERRORS, etc.)
Example:
Installation: npm install log4js
Configuration (./config/log4js.json):
{"appenders": [
{
"type": "console",
"layout": {
"type": "pattern",
"pattern": "%m"
},
"category": "app"
},{
"category": "test-file-appender",
"type": "file",
"filename": "log_file.log",
"maxLogSize": 10240,
"backups": 3,
"layout": {
"type": "pattern",
"pattern": "%d{dd/MM hh:mm} %-5p %m"
}
}
],
"replaceConsole": true }
Usage:
var log4js = require( "log4js" );
log4js.configure( "./config/log4js.json" );
var logger = log4js.getLogger( "test-file-appender" );
// log4js.getLogger("app") will return logger that prints log to the console
logger.debug("Hello log4js");// store log in file
You can also use npmlog by issacs, recommended in
https://npmjs.org/doc/coding-style.html.
You can find this module here
https://github.com/isaacs/npmlog
The "logger.setLevel('ERROR');" is causing the problem. I do not understand why, but when I set it to anything other than "ALL", nothing gets printed in the file. I poked around a little bit and modified your code. It is working fine for me. I created two files.
logger.js
var log4js = require('log4js');
log4js.clearAppenders()
log4js.loadAppender('file');
log4js.addAppender(log4js.appenders.file('test.log'), 'test');
var logger = log4js.getLogger('test');
logger.setLevel('ERROR');
var getLogger = function() {
return logger;
};
exports.logger = getLogger();
logger.test.js
var logger = require('./logger.js')
var log = logger.logger;
log.error("ERROR message");
log.trace("TRACE message");
When I run "node logger.test.js", I see only "ERROR message" in test.log file. If I change the level to "TRACE" then both lines are printed on test.log.
Winston is strong choice for most of the developers. I have been using winston for long. Recently I used winston with with papertrail which takes the application logging to next level.
Here is a nice screenshot from their site.
How its useful
you can manage logs from different systems at one place. this can be very useful when you have two backend communicating and can see logs from both at on place.
Logs are live. you can see realtime logs of your production server.
Powerful search and filter
you can create alerts to send you email if it encounters specific text in log.
and you can find more http://help.papertrailapp.com/kb/how-it-works/event-viewer/
A simple configuration using winston,winston-express and winston-papertrail node modules.
import winston from 'winston';
import expressWinston from 'express-winston';
//
// Requiring `winston-papertrail` will expose
// `winston.transports.Papertrail`
//
require('winston-papertrail').Papertrail;
// create winston transport for Papertrail
var winstonPapertrail = new winston.transports.Papertrail({
host: 'logsX.papertrailapp.com',
port: XXXXX
});
app.use(expressWinston.logger({
transports: [winstonPapertrail],
meta: true, // optional: control whether you want to log the meta data about the request (default to true)
msg: "HTTP {{req.method}} {{req.url}}", // optional: customize the default logging message. E.g. "{{res.statusCode}} {{req.method}} {{res.responseTime}}ms {{req.url}}"
expressFormat: true, // Use the default Express/morgan request formatting. Enabling this will override any msg if true. Will only output colors with colorize set to true
colorize: true, // Color the text and status code, using the Express/morgan color palette (text: gray, status: default green, 3XX cyan, 4XX yellow, 5XX red).
ignoreRoute: function (req, res) { return false; } // optional: allows to skip some log messages based on request and/or response
}));
A 'nodejslogger' module can be used for simple logging. It has three levels of logging (INFO, ERROR, DEBUG)
var logger = require('nodejslogger')
logger.init({"file":"output-file", "mode":"DIE"})
D : Debug, I : Info, E : Error
logger.debug("Debug logs")
logger.info("Info logs")
logger.error("Error logs")
The module can be accessed at : https://www.npmjs.com/package/nodejslogger
Observe that errorLogger is a wrapper around logger.trace. But the level of logger is ERROR so logger.trace will not log its message to logger's appenders.
The fix is to change logger.trace to logger.error in the body of errorLogger.
Each answer is 5 6 years old, so bit outdated or depreciated. Let's talk in 2020.
simple-node-logger is simple multi-level logger for console, file, and rolling file appenders. Features include:
levels: trace, debug, info, warn, error and fatal levels (plus all and off)
flexible appender/formatters with default to HH:mm:ss.SSS LEVEL message
add appenders to send output to console, file, rolling file, etc
change log levels on the fly
domain and category columns
overridable format methods in base appender
stats that track counts of all log statements including warn, error, etc
You can easily use it in any nodejs web application:
// create a stdout console logger
const log = require('simple-node-logger').createSimpleLogger();
or
// create a stdout and file logger
const log = require('simple-node-logger').createSimpleLogger('project.log');
or
// create a custom timestamp format for log statements
const SimpleNodeLogger = require('simple-node-logger'),
opts = {
logFilePath:'mylogfile.log',
timestampFormat:'YYYY-MM-DD HH:mm:ss.SSS'
},
log = SimpleNodeLogger.createSimpleLogger( opts );
or
// create a file only file logger
const log = require('simple-node-logger').createSimpleFileLogger('project.log');
or
// create a rolling file logger based on date/time that fires process events
const opts = {
errorEventName:'error',
logDirectory:'/mylogfiles', // NOTE: folder must exist and be writable...
fileNamePattern:'roll-<DATE>.log',
dateFormat:'YYYY.MM.DD'
};
const log = require('simple-node-logger').createRollingFileLogger( opts );
Messages can be logged by
log.info('this is logged info message')
log.warn('this is logged warn message')//etc..
PLUS POINT: It can send logs to console or socket. You can also append to log levels.
This is the most effective and easy way to handle logs functionality.
Here is lightweight module for logging data with full stack trace
#grdon/logger
const logger = require('#grdon/logger')({
defaultLogDirectory : __dirname + "/logs",
})
// ...
logger(someParams, 'logfile.txt')
logger(anotherParams, 'anotherLogFile.log')