I'd like to get the following system: once an event occurs in Cloud Foundry, it is loaded to elasticsearch. Using logstash would be fine, but I explored its input plugin and couldn't find anything that I could use. What is the best solution for this scenario? At the moment I can think of writing a script that would continuously pull the data using CF api and load it to elasticsearch. Is there a better way of doing it?
I can think of two solutions:
Create a "drain" (e.g., via the drain CLI) for the app you
would like to see events for and drain it to your ELK deployment.
This should forward each event (formatted as rfc 5425 syslog) to
elastic search.
If you are using the Loggregator Firehose to write data into elastic
search (e.g., via firehose-to-syslog) then you will get events
(as log messages). This has the downside of everything ends up in
your ELK deployment.
Related
Hello i have filebeat which is collecting logs and it is connected with logstash.
My idea is to show logs from logstash to Grafana.
Is there any option to send logstash logs directly to prometheus or grafana?
In my solution i dont want to use elasticsearch. I found some logstash exporter but that is for status of logstash not for logs.
Grafana is a visualization tool that reads the data from a data source, you will need to store your logs in one of the supported data sources, prometheus and elasticsearch are just two of the supported data sources.
To send your logs from Logstash to Prometheus you would need an output plugin, but there isn't an official plugin for it, it seems that a third party plugin exists, but it is currently in beta and maybe it still do not have all the features that you want.
Grafana, by itself, doesn't store any data (besides users/dashboards, etc.). Storing the raw logs in Prometheus is not recommended. Prometheus doesn't handle well high cardinality labels, and each different log line would generate a new value for the label. And this is assuming that you transform your log line into a set of labels and send that to Prometheus (again, don't do this).
That being said, you might want to give it a try to Loki. This a new~ish system that (as described by its authors) "Like Prometheus but for your logs". They even support a query language LogQL that is a subset of PromQL, and you can even extract metrics from the logs while storing the logline. Ingestion is usually through Promtail but Loki has an HTTP endpoint that can be used to push the logs.
The data model from Loki is quite similar: a set of labels, a timestamp, and a logline. Grafana ships with out of the box support for Loki, and it is improving with each release.
I'm working on node application and my main goal is to maintain the logs (error, info) of the backend part in logstash so that I could do some analysis of which API is breaking and why. I'm new to logstash and I read some basics of the logstash and elastic stacks. I want to achieve the following -
Integrate logstash to maintain the logs.
Read the logs to analysis the breaking changes.
I don't want to integrate the elastic search and kibana. I tried winston-logstash but it's not working and this library source code is not maintainable either. If anyone knows how to implement the above thing in nodejs application, Please let me know.
If your nodejs app runs as a docker container, you can use the gelf logging driver and then just log to console/stdout in nodejs and it will get routed to logstash.
Keep in mind Logstash is really just for transformation/enrichment/filtering/etc. you still probably want to output the log events (from Logstash) to an underlying logging solution e.g. elasticsearch.
I'm trying to get a few metrics from a Cassandra node that has a Cassandra Exporter running on it (https://github.com/criteo/cassandra_exporter/). I don't want to go into the details, but using Prometheus is not an option at this time.
I'd like to access the data with HTTP requests or something similar. With a simple HTTP Get I can access all the cached information, but I would like to do more sophisticated operations on this, such as filtering for certain messages. Is there a way to do this? I could not find any information on this. Or do I have to get the entire log and then do filtering operations on my local machine?
I'm using the jmx-exporter tag because cassandra-exporter used to be a fork of it and I couldn't find a more fitting tag.
I would suggest to use telegraf + jolokia.
It is easy to setup and it will expose the metrics via HTTP.
I wrote a post about it (in my case I saved the result into InfluxDb and used it in Grafana), it might be useful:
cassandra-performance-monitoring-by-using-jolokia-agent-telegraf-influxdb-and-grafana
Using Prometheus exporters without the Prometheus server itself is a perfectly valid approach if you don't care about historical data and just want to get an immediate snapshot of metrics (state of the system) or make a recording of some short period manually.
One of the instruments you might look at is Metricat application (https://metricat.dev/), it allows you to have filters by metrics and make recordings of how metrics change in time during period of your interest.
I am working on a solution for centralized log file aggregation from our CentOs 6.x servers. After installing Elasticsearch/Logstash/Kibana (ELK) stack I came across an Rsyslog omelasticsearch plugin which can send messages from Rsyslog to Elasticsearch in logstash format and started asking myself why I need Logstash.
Logstash has a lot of different input plugins including the one accepting Rsyslog messages. Is there a reason why I would use Logstash for my use case where I need to gather the content of logs files from multiple servers? Also, is there a benefit of sending messages from Rsyslog to Logstash instead of sending them directly to Elasticsearch?
I would use Logstash in the middle if there's something I need from it that rsyslog doesn't have. For example, getting GeoIP from an IP address.
If, on the other hand, I would need to get syslog or file contents indexed in Elasticsearch, I'd use rsyslog directly. It can do buffering (disk+memory), filtering, you can choose how the document will look like (you can put the textual severity instead of the number, for example), and it can parse unstructured data. But the main advantage is performance, on which rsyslog is focused on. Here's a presentation with some numbers (and tips and tricks) on Logstash, rsyslog and Elasticsearch:
http://blog.sematext.com/2015/05/18/tuning-elasticsearch-indexing-pipeline-for-logs/
I would recommend logstash. That would be easier to setup, more examples and they are tested to fit together.
Also, there are some benefits, in logstash you can filter and modify your logs.
You can extend logs with useful data: server name, timestamp, ...
Cast types, string to int, etc. (useful for correct Elastic index)
Filter out logs by some rules
Moreover, you can setup batch size to optimize saving to elastic.
Another feature, if something went wrong and there are crazy amount of logs per second that elastic can not process, you can setup logstash that it would save some queue of events or drop events that can not be saved.
If you go straight from the server to elasticsearch, you can get the basic documents in (assuming the source is json, etc). For me, the power of logstash is to add value to the logs by applying business logic to modify and extend the logs.
Here's one example: syslog provides a priority level (0-7). I don't want to have a pie chart where the values are 0-7, so I make a new field that contains the pretty names ("emerg", "debug", etc) that can be used for display.
Just one example...
Neither are a viable option if you really want to rely on the system to operate under load and be highly available.
We found that using rsyslog to send to a centralized location, archive it using redis of kafka and then using logstash to do its magic and ship to Elasticsearch is the best option.
Read our blog about it here - http://logz.io/blog/deploy-elk-production/
(Disclaimer - I am the VP product for logz.io and we offer ELK as a service)
I want to store logs of applications like uWSGI ("/var/log/uwsgi/uwsgi.log") on a device that can be accessed from
multiple instances and can save their logs to that particular device under their own instance name dir.
So does AWS provides any solution to do that....
There are a number of approaches you can take here. If you want to have an experience that is like writing directly to the filesystem, then you could look at using something like s3fs to mount a common S3 bucket to each of your instances. This would give you more or less a real-time log merge though honestly I would be concerned over the performance of such a set up in a high volume application.
You could process the logs at some regular interval to push the data to some common store. This would not be real time, but would likely be a pretty simple solution. The problem here is that it may be difficult to interleave your log entries from different servers if you need to have them arranged in time order.
Personally, I set up a Graylog server for each instance cluster I have, to which I log all my access logs, error logs, etc. It is UDP based, so it is fire and forget from the application servers' standpoint. It provides nice search/querying tools as well. Personally I like this approach as it removes log management from the application servers altogether.
Two options that I've used:
Use syslog (or Syslog-NG) to log to a centralized location. We do this to ship our AWS log data offsite to our datacenter. Syslog-NG is more reliable than plain ole' Syslog and allows us to use MongoDB as a backing store.
Use logrotate to push your logs to S3. It's not real-time like the Syslog solution, but it's a lot easier to set up and manage, especially if you have a lot of instances and aren't using a VPC
Loggly and Splunk Storm are also two interesting SaaS products intended to solve this problem.