I am developing an express project which will have multiple modules/services in it. The folder structure looks mostly like this:
-- app.js
-- payment_service
-- routes.js
-- index.js
-- models
-- model_1.js
-- model_2.js
APIs in index.js are the only exposed APIs and they work as a gateway for all requests coming for this module/service.
Most of the services can throw operational error under many circumstances, so manual intervention may needed to fix things. So I need to:
Log errors properly with proper context so that some person/script can do the needful.
Figure out the reason of failure.
There will be dedicated teams owning each service. So I should be able to differentiate between error logs for each service so that it can be aggregated and forwarded to concerned person.
I decided to go with ELK stash so that I can generate reports by script.
The main problem that I am facing is that I can't maintain correlation between logs. For example; If a request comes and it travels through five functions and each function logs something then I can't relate those logs.
One way is to create a child logger for each request and pass it to all the functions but that seems to be extra overhead passing logger instance to all the functions.
Another option is to use something like verror and do the logging only at entry point of the service/module so that the whole context can be contained in the log. This approach looks ok for logging errors, however it can't help with info and debug logs - they help me a lot in development and testing phase.
For the sake of differentiating between error logs, I am going to create
A dedicated logger for each service with log level error.
An application wide generic logger for info and debug purpose.
Is this the correct approach?
What will be the best way so that I can achieve all the requirements in simplest way?
I'd recommend you use a logger and you don't need anything too complex. For example:
npm install 12factor-log
Then create a file in your root folder near app.js (or in a /lib folder is where I'd place libraries)
logger.js
const Log = require('12factor-log');
module.exports = (params) => {
return new Log(params);
}
Then in your modules, import your logger and pass in the module name when you instantiate it so you can track where statements come from...
model_1.js
var log = require('./logger')({name: 'model_1'});
// ...
log.info("Something happened here");
// ...
try {
// ...
catch (error) {
const message = `Error doing x, y, z with value ${val}`;
log.error(message);
throw new Error(message);
}
Then handle error gracefully at your controller -> view layer for user-friendly experience.
Your logs would print something like this:
{"ts":"2018-04-27T16:37:24.914Z","msg":"Something happened here","name":"model_1","type":"info","level":3,"hostname":"localhost","pid":18}
As far as correlation of logs, if you see in the output above it includes the hostname of the machine it's running on, and also the name of the module and severity level. You can import this JSON into Logstash and load into Elasticsearch and it will store JSON for easy search and indexing.
See: https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html
Logging is complex and many people have worked on it. I would suggest not doing so yourself.
So, not following my own advice, I created my own logging package:
https://www.npmjs.com/package/woveon-logger
npm install woveon-logger
This prints file and line numbers of errors and messages, has logging levels and aspect-oriented logging, and can dump a stack trace in one call. It even has color coding options. If you get stuck and need some feature in logging, let me know.
let logger1 = new Logger('log1', {level : 'info', debug : true, showname : true};
let logger2 = new Logger('log2', {level : 'verbose', debug : true, showname : true};
...
log1.info('Here is a log message, that is on line 23.');
log1.verbose('Does not show');
log2.verbose('Shows because log2 is verbose logging');
log2.setAspect('IO', true);
log2.aspect('IO', 'Showing aspect IO logging, for logs for IO related operations');
[2018-06-10T10:43:20.692Z] [INFO--] [log1 ] [path/to/myfile:23] Here is a log message, that is on line 23.
[2018-06-10T10:43:20.792Z] [VERBOS] [log2 ] [path/to/myfile:25] Shows because log2 is verbose logging
[2018-06-10T10:43:20.892Z] [IO----] [log2 ] [path/to/myfile:27] Showing aspect IO logging, for logs for IO related operations
Also, some other features like:
log1.throwError('Logs this as both a line of logging, and throws the error with the same message');
log1.printStack('Prints this label next to the stack trace.');
Hope it helps!
You can use grackle_tracking library https://www.getgrackle.com/analytics_and_tracking
It logs errors & traffic to your db.
Related
This is probably a duplicated question, but I couldn't find anything.
Since I'm learning NodeJS, I think that I'm not using the right words to search, so it's hard to find an answer.
Here is the situation:
I'm currently following an online course about NodeJS and coding an API.
In the current step we are using Winston library to log errors. The instructor, have configured on Index,js, which is the entry point of the app, like this:
File: index.js
const winston = require('winston');
const errorHandler = require(./middleware/error.js);
//(...) some other imports
app.use(errorHandler);
winston.add(winston.transports.File,{filename:'logFile.log'});
And in other module we've created in the course to handle errors, he requires winston and simply call to log the error. Something like this:
File: error.js
const winston = require('winston');
function errorHandler(err,req,res,next){
winston.error(err.message,err);
res.status(500).send("something failed");
}
module.exports = errorHandler;
After doing a test, the error is correctly written to the file, and my question is: How it works? How a setting made on the 'required version' of winston at index.js is visible from the other required version at error.js?
From index.js we are importing error.js too, so i can imagine somehow this two modules are sharing this winston object, but again, I don't understand how or where is it shared.
Again, please excuseme if I'm not using the right terms to refer anything here, I'll accept any advice.
Thanks.
When a module is loaded in node.js, it is cached by the require() sub-system. So, when you then require() it again, that means you'll get the exact same module as the previous one.
So ... if you initialized the module after you first loaded it and the module stores some state that represents that intialization, then subsequent use of that module will be using the same (already initialized) module.
And in other module we've created in the course to handle errors, he requires winston and simply call to log the error.
It gets the same instance of the winston module that was already initialized/configured previously.
After doing a test, the error is correctly written to the file, and my question is: How it works? How a setting made on the 'required version' of winston at index.js is visible from the other required version at error.js?
Module caching as describe above. There's only one winston module that all are sharing so if it's initialized/configured in one place, all will use that configuration.
I am looking for a logging solution for my node.js app that would allow me to set the logging level via file/folder selectors.
For example I would like to be able to set the logging for all files in /app/schema to 'info'. And all the rest to 'error'.
Exemplary configuration:
{
"*":"error",
"/app/schema":"info" //<--Regex expression would be great too.
}
I constantly comment/uncomment/remove logging statements when I need to examine something. I would rather do that via a configuration change and leave the logging files intact. A global debugging level just creates too way to much noise and volume (which matters when storing logs).
Is there something like this? Apache log4j is similar, you can set logging level on package level.
If I get your question right, I'd suggest Bunyan, it lets you configure as many log streams and log levels as you like, i.e. from the docs:
var log = bunyan.createLogger({
name: 'myapp',
streams: [
{
level: 'info',
stream: process.stdout // log INFO and above to stdout
},
{
level: 'error',
path: '/var/tmp/myapp-error.log' // log ERROR and above to a file
}
]
});
You'll have to set up the logic for each path with each log-level.
I am trying to understand how to build my error handling system for my api.
Let's say I have a the following line in a controller method :
var age = json.info.age;
with
json = {"id":1, "name":"John", info": {"age":27, "sex":"m"}}
Let's say that the object doesn't contain an info field, I'll get the following error TypeError: Cannot read property 'info' of undefined and my server will crash.
Is there a way to make a higher level abstraction and catch all the potential errors that I could have? Or should I have a try/catch system for each of the methods of my controllers?
BEWARE OF THE CODE BELOW, IT WILL BITE YOU WHENEVER IT CAN!
Don't use the code snippet below if you do not understand its
implications, please read the whole answer.
You can use the node way for uncaught errors. Add this in your config/bootstrap.js
Updated the snippet below to add what was said in the comments, also added a warning about using a global to respond to the user.
process.on('uncaughtException', function (err) {
// Handle your errors here
// global.__current__ is added via middleware
// Be aware that this is a bad practice,
// global.__current__ being a global, can change
// without advice, so you might end responding with
// serverError() to a different request than the one
// that originated the error if this one happened async
global.__current__.res.serverError();
})
Now, can doesn't mean should. It really depends on your needs, but do not try to catch BUGS in your code, try to catch at a controller level the issues that might not happen every time but are somehow expected, like a third-party service that responded with empty data, you should handle that in your controller. The uncaughtException is mainly for logging purposes, its better to let your app crash if there is a bug. Or you can do something more complicated (that might be better IMHO), which is to stop receiving requests, respond to the error 500 (or a custom one) to user that requested the faulty endpoint, and try to complete the other requests that do not relate to that controller, then log and shutdown the server. You will need several instances of sails running to avoid zero downtime, but that is material for another question. What you asked is how to get uncaught exceptions at a higher lvl than the controllers.
I suggest you read the node guide for error handling
Also read about domains, even thought they are deprecated you can use them, but you would have to deal with them per controller action, since sails does not provide any help with that.
I hope it helps.
You can check this way if you want to:
if (object != null && object.response != null && object.response.docs != null){
//Do your stuff here with your document
}
I don't really get what is your "object" variable in the first place, so i don't know if you can check it at a different level, is it a sails parameter to your controller ?
So that's how I did it, thanks to Zagen's answer.
module.exports.bootstrap = function(cb) {
process.on('uncaughtException', function (err) {
//Handle your errors here
logger.fatal(err);
global.__current__.res.serverError();
})
cb();
};
I send a generic error 500 to the user if any uncaught exception is thrown, and I log the error to the fatal level. On that way, my server is still accessible 24/7 and I can monitor the logs at another level and trigger an alarm on a fatal error. I can then fix the exception that was thrown.
Is there a way by which I can turn on/off certain logs, based on its type/levels.
For eg:
I have defined 3 levels: ALL, WARNING, CRITICAL
And I have my Log class, where I will set this. Say I set Level: 'ALL'
So this will log everything, wherever I have logged messages.
Now, when I set Level: 'WARNING'
This will only log messages, which are of warning type.
Can I do this with Bunyan ?
Or any other module?
Please help !!
One work around would be to use Bunyan's DTrace facilities.Keep the log level higher and if you need to inspect low level log like debug you can run Dtrace command
Examples
Trace all log messages coming from any Bunyan module on the system
dtrace -x strsize=4k -qn 'bunyan*:::log-*{printf("%d: %s: %s", pid, probefunc, copyinstr(arg0))}'
Trace all log messages coming from the "wuzzle" component:
dtrace -x strsize=4k -qn 'bunyan*:::log-*/strstr(this->str =
copyinstr(arg0), "\"component\":\"wuzzle\"") != NULL/{printf("%s",
this->str)}'
you need to manually install the "dtrace-provider" lib separately via npm install dtrace-provider
Check out the documentation here
try using winston module for logging .this is good for logging and has log rotation and other features
I am using the Bunyan module for NodeJS logging. When I try using the rotating-file type, it makes my app crash every time and outputs this error:
Error: ENOENT, rename 'logs/info.log.3'
However, it never happens at the same time so I can't find any logic...
This is how I instanciate my logger:
var log = Bunyan.createLogger(config.log.config);
log.info('App started, ' + process.env.NODE_ENV);
And here is my config.json:
{
"name" : "app",
"streams" : [
{
"type" : "rotating-file",
"period": "5000ms", //Low period is for testing purposes
"count" : 12,
"level" : "info",
"path" : "logs/info.log"
},
{
"type" : "rotating-file",
"period": "5000ms",
"count" : 12,
"level" : "error",
"path" : "logs/error.log"
},
{
"type" : "rotating-file",
"period": "5000ms",
"count" : 12,
"level" : "trace",
"path" : "logs/trace.log"
}
]
}
Can anyone advise how to fix my issue? Thanks in advance.
What I have just done (last night actually) to get around this problem of a master + workers contending over a Bunyan rotating-file is to have the workers write "raw" log records to a stream-like object I created called a WorkerStream. The write method of the WorkerStream simply calls process.send to use IPC to deliver the log record to the master. The master uses a different logger config that points to a rotating-file. The master uses the code shown below to listen for log records from its workers and write them to the log file. So far it appears to be working perfectly.
cluster.on('online', function (worker) {
// New worker has come online.
worker.on('message', function (msg) {
/* Watch for log records from this worker and write them
to the real rotating log file.
*/
if (msg.level) {
log._emit(msg);
}
});
});
ln is your friend.
Existing logging libraries have rotation problem with cluster module. Why doesn't ln have this issue?
Both bunyan and log4js rename the log file on rotation. The disaster happens on file renaming under cluster environment because of double files renaming.
bunyan suggests using the process id as a part of the filename to tackle this issue. However, this will generate too many files.
log4js provides a multiprocess appender and lets master log everything. However, this must have the bottleneck issue.
To solve this, I just use fs.createWriteStream(name, {"flags": "a"}) to create a formatted log file at the beginning instead of fs.rename at the end. I tested this approach with millisecond rotation under cluster environment and no disasters occurred.
I have experienced the same issue without using clustering. I believe the problem is being caused by old files sitting in the log directory. While the main logger can open and append to existing files, the file rotation logic uses rename, which files when it steps on an existing file. (e.g. an existing info.log.3 file).
I'm still digging into the source to figure out what needs to be changed to recover from the rolling error.
One additional thought as I review the source. If you have multiple Bunyan log instances that use the same log file (in my case, a common error.log), the rename calls could be happening nearly concurrently from the OS level (asynchronous and separate calls from a Node.js perspective, but concurrently from the OS perspective).
It's sadly not possible to use multiple rotating file streams against the same file.
If you're in the same process, you must use a single logger object - make sure you're not creating multiple of them.
If you're working across processes, you must log to different files. Unfortunately there's nothing yet that has the IPC in place to allow different rotators to coordinate amongst themselves.
I have a plugin rotating file stream that detects if you try to create 2 rotators against the same file in the a single process and throws an error.
It can't help in the case of multiple processes tho.
bunyan-rotating-file-stream
From my experience, it happens sometimes when the logs directory (or whatever you named it) does not exist.
If you are running through this error in a automation pipeline, for example, you may be ignoring all the files in logs and committing it empty, then it is not created when the repository is cloned by the pipeline.
Simply make sure that logs is created by placing a .gitkeep file inside it (or any other trick).
This may be the case of many of you who come across this question.