I need to find some open source log management system for central log. The log format is unstructured on multiple host and I need to collect and send these log on central log system. The best way is to use some kind of "tail" these logs and send to central log system.
Do you know any solution that can tail some file and send to remote central log system?
Take a look at the ELK-Stack or Graylog both satisfy your requirements. In both solutions you can use logstash or an other tool like filebeat or the graylog collector sidecar for log shipping. A pricey solution would be Splunk.
I personally recommend Graylog, because it has a lot of open source features like authentication and authorization for example out of the box which ELK does not have. There you have to pay for those features.
Related
I need some help on finding a way to manage my log information.
I have 20 windows servers build with application on glassfish which generate logs everyday, so to manage these log in case i need to find something specific from all my servers im trying to group all these data on a single server (windows or linux) and filter them according to my specs.
Best regard Egis
It's too broad question but a common solution it's ELK Stack
elasticsearch - to store the data
logstash - to process the data, installing it on servers that generate log to send they to elasticsearch server
kibana - visualize the data
An article explaining the stack solution
https://www.guru99.com/elk-stack-tutorial.html
I am trying to understand the use of application insights for capturing the module logs and considering appinsights as a potential option.
I am keen on understanding how would the appinsights work considering there would be multiple devices each running the same modules where modules are configured to send log data to appinsights. The type of data I want to capture are container logs which are currently being sent to stderr/stdout streams.I am expecting this to work on windows devices , hence the logspout project may not be useful here (https://github.com/veyalla/logspout-loganalytics) but i want to do something similar.
I am trying to figure out a design where module logs from multiple edge devices can be captured using appinsights. It would be immensely useful for me to know if appinisghts is really suited for the problem I am trying to solve and how can it be used for multiple devices.
I'm not aware of a good/secure solution for Windows containers that does a continuous push of module logs to log analytics.
Since the built-in log pull via edgeAgent is experimental, we might change the API or make some modifications but we're unlikely to pull the feature entirely without an equivalent alternative.
I am playing with the ELK module and other "beats". I realized there are cool default dashboards for Metricbeats and Heartbeat. But I couldn't find anything about logstash.
So I was wondering: Is there an example of a dashboard for Logstash in Kibana?
Logstash actually doesn't have any dashboard with it. It doest work as Heartbeat or Metricbeats on one task.
Logstash just powerful instrument to capture and modify data on the fly. It has many different plug-ins for it and can be used regardless elastic for example to capture data, parce it, create fields from raw data and send it to back-end which could be elastic, hive, sql or just e-mail.
So it doesn't, but you can create your own dashboard from data which coming from logstash
I have an application that is generating 3 kind of log files
Transaction log
Server log
Fatal log
and I want to analyse the performance of my server using appdynamics so what kind of data my logs should be generating to generate analytics for server health, performance, throughput, server utilization?
That's the beauty of APM is you don't need to deal with logging to get performance data. APM tools instrument the applications themselves regardless of what the code does (logging, generating metrics, etc). AppDynamics can collect log data, and provide similar capabilities to what a log analytics tool can do, but it's of less value than the transaction tracing and instrumentation you get. Your application has to be built in a supported language (Java, .NET, PHP, Python, C++, Node.js) and if it's web or mobile based you can also collect client side performance data and unify between both frontend and backend. If you have questions just reach out and I can answer them for you. Good luck!
You basically need the AppDynamics Controller and a AppDynamics Machine-Agent which will be installed on the machine to monitor. In the Machine-Agent configuration you set the URI of the controller and the agent starts to report machine metrics to the controller. Then you can configure alarms, see metrics, create dashboards, etc. I can deliver you more information if you want, but as Jonah Kowall said, take a look at the documentation as well AppDynamics Machine Agent Doc
Context
I want to develop an automated script for broker (IIB9/10) resource monitoring, capturing information about broker running status, message flows deployed, jvm usage, number of threads running, etc.
The initial thought is to have a report generated using scripts and then displayed over a browser.
Question
Can this be entirely done using only Ant scripts (i am not sure as have not explored iterative processing in Ant in detail) or a combination of Ant and batch/shell scripts is the best bet?
I know Web user interface in IIB10 does most of it but i want to add some features.
I suggest you to take a look at message flow statistics and accounting:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/ac19100_.htm?lang=en
This is a feature of IIB by which it is capable of emitting resource statistics. The statistics are published to a topic in a well defined XML format. I would try solving your requirement by writing an application to read these messages and use the data in them to generate your graphs or other reports.
There is a support pack, IS03 which can give you an idea of such an application.
This will not cover everything you mentioned, for example monitoring what flows are deployed cannot be achieved like this, but it gives a comprehensive view of the load and performance of your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj10440_.htm?lang=en
And there is a resource statistics feature as well for monitoring resources used by your applications:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_9.0.0/com.ibm.etools.mft.doc/bj43310_.htm?lang=en
To get everything you will need a variety of tools I think. You can use Resource Stats and Accounting / Stats as suggested by Attila to get JVM and thread usage. The Broker publishes updates to a topic so you can create a simple subscriber to grab that info.
For deploy related info, stop / start state and so forth I would be looking at building simple Integration API or REST API applications to call from ant.
You can find documentation for these API's here:
http://www-01.ibm.com/support/knowledgecenter/SSMKHH_10.0.0/com.ibm.etools.mft.doc/be43410_.htm?lang=en
and here:
http://www-01.ibm.com/support/knowledgecenter/api/content/nl/en-us/SSMKHH_10.0.0/com.ibm.etools.mft.restapi.doc/index.html