Currently I'm trying out ElasticSearch as a logging solution to pump out ETW events to.
I've followed this tutorial (https://learn.microsoft.com/en-us/azure/service-fabric/service-fabric-diagnostic-how-to-use-elasticsearch), and this is working great for my own custom ActorEventSource logs, but I haven't found a way to log the Actor Runtime events (ActorMethodStart, ActorMethodStop... etc) using the "in-process" trace capturing.
1) Is this possible using the in-process trace capturing?
I'm also considering using the "out-of-process" trace capturing, which to me seems like the preferable way of doing things in our situation, as we already have WAD setup which includes all of the Actor Runtime events already. Not to mention the potential performance impact / other side-effects of running the ElasticSearchListener inside of our Actor Services.
2) I'm not quite sure how to implement this.The https://github.com/Azure/azure-diagnostics-tools/tree/master/ES-MultiNode project doesn't seem to include Logstash, so i'm assuming I would need a template such as this one: https://github.com/Azure/azure-diagnostics-tools/tree/master/ELK-Semantic-Logging/ELK/AzureRM/elk-simple-on-ubuntu, otherwise I would need to modify the ES-MultiNode project to install Logstash as well? Just trying to get an idea if I'm going down the right path with regards to this.
If there's any other suggestions in terms of logging, I'd love to hear them!
Related
I want to capture my logs in the program, which include INFO, DEBUG, WARN, EXCEPTION to the database. without using any django libraries. Please have a look
Your question is quite unclear.
If you want to aggregate your logs in a database, you have to define a new Logging handler, give a look to StreamHandler.
Note : I would suggest Sentry as it has free plan and allow you to get all errors in one application
I am trying to understand the use of application insights for capturing the module logs and considering appinsights as a potential option.
I am keen on understanding how would the appinsights work considering there would be multiple devices each running the same modules where modules are configured to send log data to appinsights. The type of data I want to capture are container logs which are currently being sent to stderr/stdout streams.I am expecting this to work on windows devices , hence the logspout project may not be useful here (https://github.com/veyalla/logspout-loganalytics) but i want to do something similar.
I am trying to figure out a design where module logs from multiple edge devices can be captured using appinsights. It would be immensely useful for me to know if appinisghts is really suited for the problem I am trying to solve and how can it be used for multiple devices.
I'm not aware of a good/secure solution for Windows containers that does a continuous push of module logs to log analytics.
Since the built-in log pull via edgeAgent is experimental, we might change the API or make some modifications but we're unlikely to pull the feature entirely without an equivalent alternative.
Context
I want to develop an automated script for broker (IIB9/10) resource monitoring, capturing information about broker running status, message flows deployed, jvm usage, number of threads running, etc.
The initial thought is to have a report generated using scripts and then displayed over a browser.
Question
Can this be entirely done using only Ant scripts (i am not sure as have not explored iterative processing in Ant in detail) or a combination of Ant and batch/shell scripts is the best bet?
I know Web user interface in IIB10 does most of it but i want to add some features.
I suggest you to take a look at message flow statistics and accounting:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/ac19100_.htm?lang=en
This is a feature of IIB by which it is capable of emitting resource statistics. The statistics are published to a topic in a well defined XML format. I would try solving your requirement by writing an application to read these messages and use the data in them to generate your graphs or other reports.
There is a support pack, IS03 which can give you an idea of such an application.
This will not cover everything you mentioned, for example monitoring what flows are deployed cannot be achieved like this, but it gives a comprehensive view of the load and performance of your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj10440_.htm?lang=en
And there is a resource statistics feature as well for monitoring resources used by your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj43310_.htm?lang=en
To get everything you will need a variety of tools I think. You can use Resource Stats and Accounting / Stats as suggested by Attila to get JVM and thread usage. The Broker publishes updates to a topic so you can create a simple subscriber to grab that info.
For deploy related info, stop / start state and so forth I would be looking at building simple Integration API or REST API applications to call from ant.
You can find documentation for these API's here:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/be43410_.htm?lang=en
and here:
http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSMKHH_10.0.0/com.ibm.etools.mft.restapi.doc/index.html
I'm currently using Microsoft Network Monitor to parse thru debug event traces. It is not a bad tool, but not very good either. Do you know some better solutions?
These are readers for exploring custom ETW traces:
SvcPerf - End-to-End ETW trace viewer for manifest based traces
LINQPad + Tx (LINQ for Logs and traces) driver - Simple reader that allows you to query ETW traces
PerfView - multitool that allows you to do amost everything with ETW, but not particularly user-friendly
PerfView http://www.microsoft.com/download/en/details.aspx?id=28567
If you're after giving graphic visualization of traces for the sake of performance analysis, you may use the following:
1. Windows Reliability and Performance Monitor which is an MMC snap-in and is easy to use for basic analysis (locally, from the server)
2. xperf, which is a stand-alone tool from the Windows Performance Tools.
Xperf itself is a command-line tool for captures and processing traces and Xperfview allows creating graphs and tables from the captured data. Look at this blog post for an overview.
3. Visual Studio 2010 profiler contains a "Concurrency Visualizer" which is actually a nice tool to collect and visualize ETW traces, specifically tailored around analysis of thread contention issues (but can also be used to analyze network traces, I think). See this blog post on using the tool and also you may use the underlying tools directly: VSPerfCmd and VSPerfReport.
I like to use Log Parser [link] to parse through the logs for the events that I am most interested in. I love the SQL-like query structure.
I'm wondering if anyone has any experience using log4net in a multi-threaded environment like asp.net. We are currently using log4net and I want to make sure we won't run into any issues.
We run log4net (and log4cxx) in highly multi-threaded environments without issue. You will want to be careful how you configure them though.
The issue with log4net that Jeff describes pertains to the use of a certain appender. We stick with simple log file appenders on the whole to reduce the impact of logging on the operation of the code. Writing a line to a file is pretty minimal, kicking off another database transaction is very heavy.