I have built WPF and Electron desktop applications where I'm currently logging custom usage and error logs. They are sent to an Azure Function which in turn talks to an Azure MSSQL database with a stored procedure to add log messages.
I feel this is not a very optimal solution. The function sometimes takes a long time to spin up, and the database potentially feels a bit overkill for logging (maybe it's OK). And I don't know if it's safe to add the link to the logging function in my applications even though it's very restrictive in what it accepts. You need to send the logs in a specific structure with some mandatory fields. But this feels more like "security by obscurity".
Isn't there a better way to implement a centralized logging service for my different applications? I want to be able to add more applications in the future as well (all will send the same structured logs). Are there any features in Azure to handle custom logging?
I am looking for system or library for node.js, that can log information about client access on every remote server and automatically gather that information on central log server for later analysis. Remote server will have write only access, while central server will accumulate a lot of data to read.
I hope there is solution using distributed [NoSQL] database, like MongoDB.
However I have not found how to set it up.
For example I hope that cleaning old data can be initiated on central log server (when data has been processed) and entries on old dates can be removed on remote server with little overhead.
Currently we have logging into files and Hadoop system for log analysis.
But I think we need to accumulate data in database.
Winston, currently the best logging framework for node.js, has option to log into MongoDB or CouchDB.
Scribe could be what you're looking for. There are node packages too
I have never checked it out so I'd be interested in reading your thoughts in the comments if you investigate it and find it good/bad, easy/hard to setup, etc.
MongoDB or any other distributed databases will not solve problem.
In-house project must be created.
Some features of MongoDB for consideration:
Capped Collections are actually way to loose data. I may be good for short history.
I have developed an application XPages that work very well in a Browser (Firefox ) and in every page the browser load max 150Kb of content (html, image, js, css...etc...)
When I have deploy the application to my remote user that directly access to server with XPiNC mode the speed are very very poor!
With a tool I sniffed the traffic and I see that for every GET there are 10Mbytes of data transfered (seem to transfer XML source and other code that is compiled on the fly...)
The application inside Notes Client is not useable so...and my customer has disappointed for this feature (is not possible use in local and replicate)
I have 8.5.3FP2 (client and server) with PRELOAD option setting.... without any change of this.
Have someone any suggest for me? Is this a BUG ?
It is true that remote applications (NSFs residing on a non-local server) are slower than local client replicas or remote apps run in a web browser. This is due to the fact that a lot more network transactions are generated when running in this mode. There are various things that can be done however to remedy the problem.
First however we need to identify the cause of the problem - you are seeing a 10MB transfer for each GET request, which is very large and will obviously negatively impact performance. One or more of the XPages in your application may be using the computeWithForm feature? If an XPages document data source "computes" a Notes form (typically to execute pre-existing application logic) then the form must be copied across the net to be computed in the local client. However all children of the form will also be hauled over - subforms, shared fields etc, and this can result in large net transactions like those you are seeing.
Often the computeWithForm feature is used as a development convenience and as long as the size of the form is small then the performance impact can be negligible. However, if the aggregate form is large, then it may be worth your while replacing the computeWithForm usage with separate XPages SSJS application logic.
Before going further we would need verify that this is in fact the issue - there could be other issues. Typically this manifests only on pages that open/edit documents - so you can maybe try turning computeWithForm off in a test environment and see if there is a difference.
XPiNC is a little special. When you open a server based NSF, all the program code needs to be downloaded to the client to be executed in the server container of the Notes client. The reasonable way to use an XPiNC with data in the server is to split the application. Have one NSF that contains all the program logic (all XPages and other code) and the other with forms, views and documents.
Replicate the application NSF locally and access only the data on the server. This should give you much better performance. You could have a configuration setting to compute the data NSF, so disconnected users could use a local replica of the data.
Let us know how it goes.
P.S.: There are some more tuning ideas...
I want to log some information about my visitors. Is it better to use the IIS generated log or to create my own in an SQL 2008 db.
I know I should probably provide more information about my specific scenario, but I'd like just generally, pros and cons of either proposal.
You can add additional information to the IIS logs from ASP.NET using HttpResponse.AppendToLog, additionally you could use the Advanced Logging Module to create your own logs with custom filters and custom data including data from Performance Counters, and more.
It all depends on what information you want to analyse.
If you're doing aggregations and rollups then you'd want to pull this data into a database for analysis. Pulling your data into a database will give you access to indexes and better querying tools.
If you're doing infrequent one-off simple queries then LogParser might be sufficient for your needs. However you'll be constantly scanning unindexed flat files looking for data which is I/O intensive.
But as you say, without knowing more about your specific scenario it's hard to say what would be best.
Okay, here's the scenario. I have a utility that processes tons of records, and enters information to the Database accordingly.
It works on these records in multi-threaded batches. Each such batch writes to the same log file for creating a workflow trace for each record. Potentially, we could be making close to a million log writes in a day.
Should this log be made into a database residing on another server? Considerations:
The obvious disadvantage of multiple threads writing to the same log file is that the log messages are shuffled amongst each other. In the database, they can be grouped by batch id.
Performance - which would slow down the batch processing more? writing to a local file or sending log data to a database on another server on the same network. Theoretically, the log file is faster, but is there a gotcha here?
Are there any optimizations that can be done on either approach?
Thanks.
The interesting question, should you decide to log to the database, is where do you log database connection errors?
If I'm logging to a database, I always have a secondary log location (file, event log, etc) in case there are communication errors. It really does make it easier to diagnose issues later on.
One thing that comes to mind is that you could have each thread writing to its own log file and then do a daily batch run to combine them.
If you are logging to database you probably need to do some tuning and optimization, especially if the DB will be across the network. At the least you will need to be reusing the DB connections.
Furthermore, do you have any specific needs to have the log in database? If all you need is a "grep " then I don't think you gain much by logging into database.
I second the other answers here, depends on what you are doing with the data.
We have two scenarios here:
The majority of the logging is to a DB since admin users for the products we build need to be able to view them in their nice little app with all the bells and whistles.
We log all of our diagnostics and debug info to file. We have no need for really "prettifying" it and TBH, we don't even often need it, so we just log and archive for the most part.
I would say if the user is doing anything with it, then log to DB, if its for you, then a file will probably suffice.
Not sure if it helps, but there's also a utility called Microsoft LogParser that you can supposedly use to parse text-based log files and use them as if they were a database. From the website:
Log parser is a powerful, versatile
tool that provides universal query
access to text-based data such as log
files, XML files and CSV files, as
well as key data sources on the
Windows® operating system such as the
Event Log, the Registry, the file
system, and Active Directory®. You
tell Log Parser what information you
need and how you want it processed.
The results of your query can be
custom-formatted in text based output,
or they can be persisted to more
specialty targets like SQL, SYSLOG, or
a chart. Most software is designed to
accomplish a limited number of
specific tasks. Log Parser is
different... the number of ways it can
be used is limited only by the needs
and imagination of the user. The
world is your database with Log
Parser.
I haven't used the program myself, but it seems quite interesting!
Or how about logging to a queue? That way you can switch out pollers whenever you like to log to different things. It makes things like rolling over and archiving log files very easy. It's also nice because you can add pollers that log to different things, for example:
a poller that looks for error messages and posts them to your FogBugz account
a poller that looks for access violations ('x tried to access /foo/y/bar.html') to a 'hacking attempts' file
etc.
Database - since you mentioned multiple threads. Synchronization as well as filtered retrieval are my reasons for my answer.
See if you have a performance problem before deciding to switch to files
"Knuth: Premature optimization is the root of all evil" I didn't get any further in that book... :)
There are ways you can work around the limitations of file logging.
You can always start each log entry with a thread id of some kind, and grep out the individual thread ids. Or a different log file for each thread.
I've logged to database in the past, in a separate thread at a lower priority. I must say, queryability is very valuable when you're trying to figure out what went wrong.
How about logging to database-file, say a SQLite database? I think it can handle multi-threaded writes - although that may also have its own performance overheads.
I think it depends greatly on what you are doing with the log files afterwards.
Of the two operations writing to the log file will be faster - especially as you are suggesting writing to a database on another server.
However if you are then trying to process and search the log files on a regular basis then the best place to do this would be a database.
If you use a logging framework like log4net they often provide simple config file based ways of redirecting input to file or database.
I like Gaius' answer. Put all the log statements in a threadsafe queue and then process them from there. For DB you could batch them up, say 100 log statements in one batch and for file you could just stream them into the file as they come into the queue.
File or Db? As many others say; it depends on what you need the log file for.