Please advise:
I don’t have any experience with the ELC stack and I’m slowly starting to figure it out, and the first thing I need to do is filter the messages from the Cisco asa by message ID, because I need logs with a certain code, but everyone comes and very quickly takes up all the free space.
I tried to do this using pipeline, highlighting the message code from each log and adding a processor with the drop function, but it honestly does not work very well, because. there are many such logs.
I have a Cisco asa module installed in filebeat
As I understand it, I definitely need a logstash, because now filebeat sends logs directly to elastic, how can I apply filtering there and how can I do it? And do I need logstash at all, maybe there is a simpler solution?
Thanks in advance for your reply.
Related
I'd like to get the following system: once an event occurs in Cloud Foundry, it is loaded to elasticsearch. Using logstash would be fine, but I explored its input plugin and couldn't find anything that I could use. What is the best solution for this scenario? At the moment I can think of writing a script that would continuously pull the data using CF api and load it to elasticsearch. Is there a better way of doing it?
I can think of two solutions:
Create a "drain" (e.g., via the drain CLI) for the app you
would like to see events for and drain it to your ELK deployment.
This should forward each event (formatted as rfc 5425 syslog) to
elastic search.
If you are using the Loggregator Firehose to write data into elastic
search (e.g., via firehose-to-syslog) then you will get events
(as log messages). This has the downside of everything ends up in
your ELK deployment.
Does someone knows how to scale Logstash? What do I mean by scaling?
Let's say I have one instance of Logstash producing messages from jdbc to Kafka and lets say that node where Logstash service is installed and instance of him is running fall down suddenly.
How to continue with the same process, something like master-slave config? Is there any way to acomplish that or I need to set some work around by myself?
That depends on how you are sending your messages to logstash. Are you using filebeat? I know it has an option to configure more than 1 logstash instance to send to. Here is the documentation for it.
In our application we want to read logs from 2 different servers i.e apache tomcat and Jboss and want to monitor the logs. I have tried to search online to how to configure it but not albe to understand clearly about how can i implement it in graylog. Please help. Thank you.
You can send logs from an arbitrary number of applications and systems to Graylog (even on the same input).
Simply configure your applications and systems to send logs to Graylog and create an appropriate input for them.
See http://docs.graylog.org/en/2.1/pages/sending_data.html for some hints.
Hope you were able to send your logs to graylog server. Centralized logging using graylog will help newbies to get started with graylog and the article explains use cases like integrating apache, nginx, mysql slow query logs to graylog. This covers various ways like sending logs via syslog, graylog apache module, filebeat etc. which most articles miss out explaining in detail.
I am working on a solution for centralized log file aggregation from our CentOs 6.x servers. After installing Elasticsearch/Logstash/Kibana (ELK) stack I came across an Rsyslog omelasticsearch plugin which can send messages from Rsyslog to Elasticsearch in logstash format and started asking myself why I need Logstash.
Logstash has a lot of different input plugins including the one accepting Rsyslog messages. Is there a reason why I would use Logstash for my use case where I need to gather the content of logs files from multiple servers? Also, is there a benefit of sending messages from Rsyslog to Logstash instead of sending them directly to Elasticsearch?
I would use Logstash in the middle if there's something I need from it that rsyslog doesn't have. For example, getting GeoIP from an IP address.
If, on the other hand, I would need to get syslog or file contents indexed in Elasticsearch, I'd use rsyslog directly. It can do buffering (disk+memory), filtering, you can choose how the document will look like (you can put the textual severity instead of the number, for example), and it can parse unstructured data. But the main advantage is performance, on which rsyslog is focused on. Here's a presentation with some numbers (and tips and tricks) on Logstash, rsyslog and Elasticsearch:
http://blog.sematext.com/2015/05/18/tuning-elasticsearch-indexing-pipeline-for-logs/
I would recommend logstash. That would be easier to setup, more examples and they are tested to fit together.
Also, there are some benefits, in logstash you can filter and modify your logs.
You can extend logs with useful data: server name, timestamp, ...
Cast types, string to int, etc. (useful for correct Elastic index)
Filter out logs by some rules
Moreover, you can setup batch size to optimize saving to elastic.
Another feature, if something went wrong and there are crazy amount of logs per second that elastic can not process, you can setup logstash that it would save some queue of events or drop events that can not be saved.
If you go straight from the server to elasticsearch, you can get the basic documents in (assuming the source is json, etc). For me, the power of logstash is to add value to the logs by applying business logic to modify and extend the logs.
Here's one example: syslog provides a priority level (0-7). I don't want to have a pie chart where the values are 0-7, so I make a new field that contains the pretty names ("emerg", "debug", etc) that can be used for display.
Just one example...
Neither are a viable option if you really want to rely on the system to operate under load and be highly available.
We found that using rsyslog to send to a centralized location, archive it using redis of kafka and then using logstash to do its magic and ship to Elasticsearch is the best option.
Read our blog about it here - http://logz.io/blog/deploy-elk-production/
(Disclaimer - I am the VP product for logz.io and we offer ELK as a service)
I understand that a similar question was asked here:
How can I configure logtash to use "stdout" as input?, but I wanted to generalize the question a little.
What I have currently configured is the basic ELK stack(elasticSearch, logstash, kibana) on a Linux VM and I want to feed a bit of standard output into logstash in order to remove log files from the file system, thus not consuming disk space.
So for example, how would I take the std-out from say, Kibana, and feed that directly into Logstash?
Any pointers in the right direction would be appreciated.
This is a bad idea. If you connect, say, Kibana's stdout to Logstash's stdin, if you restart Kibana you'll have to restart Logstash and vice versa. Also, if Logstash's pipeline gets clogged (e.g. because a downstream host is unavailable) the pipe's buffer will eventually fill up and the process writing its logs to stdout will block (if its logging is synchronous; otherwise you'll only see log messages dropped on the floor).
I suggest you use disk-based logging but rotate the files aggressively.