I'm currently in front of a structure problem with logstash.
I have a syslog-ng client sending logs from different files through the network to an ELK stack.
I noticed that Logstash is mixing logs, especially multiline with adding exception lines to non error logs from others files. So, i guess the trouble is that my logs are treated without any differentiation from their file origins. I found 2 ways to avoid that, but they are not optimal in my opinion:
Instead of using syslog-ng, use FileBeat and add a tag for each file which identify their origins. Then, parse my log with Logstash depending of this tag. The problem is that using syslog-ng as client is a requirement, and it will bother me if I have to change it
Change my syslog-ng sources to send each log files to a different port on ELK. I found that a little dirty and can be embarrassing with a great quantity of log files
What do you think about that ? Did I miss a better solution?
Is there a way to add a tag just like filebeat in syslog-ng?
Related
I want to be able to track logstash logs in case of an error.
I want to be able to monitor issues when logstash tries to send events to output destination. I've checked the monitor API but it doesn't fulfills my requirements.
Is it a good practice to use file input plugin and set the path to its own log?
I am new to elk stack. Let me explain what i am trying to do. I have a application that is running separately for different users i.e. 5 different users will have 5 independent instance of the same application. I am using filebeats to send the logs from the application to logstash where it will be processed first before being sent to the elasticsearch. What i want is to write the application which enables the users to view the logs of theirs instance of application only. Now what i tried to do is creating the logstash pipeline for each the user with different port which will process the log and send it to elasticsearch with the different index name.
Can you suggest me if this is the best practice or i am doing it wrong? Is there a more better way to do it without having separate pipeline for individual users with separate port? I think the way I am doing it is wrong and it will be harder for me to manage when the instances will grow in numbers.
Thank You
I would suggest if there's no skinning , validation and enrichment involved then skip logstash altogether. You can straight away pass filebeat logs to ES. Now there are two ways from here. Filebeat can additionally send a parameter (any fixed string) along with the scanned message to ES or you can store the meta (like ip) source which filebeat will send along with message. This string can then be used to identify the source of the log message and then on kibana you can configure to show dashboard based on that fixed string / user / meta. This simply the process and avoid unnecessary hops.
I am using rc.local to start my node script on start with:
node .> "/log_file_$(date +"%H:%M:%S_%m_%d_%Y").txt"
It works fine - but now once the log grows in size - I need to create a new log on a server every 12/24 hours; without restarting the server.
Is there any simple way to change the node app output destination?
I would prefer not to use any library for that, because I need to log all the messages including errors, warns, not only console.log.
Thanks for your help.
There are a number of options, I'll offer two:
1. Stackdriver
Stream your logs to Stackdriver, which is part of Google Cloud, and don't store them on your server at all. In your node.js application, you can can setup Winston and use the Winston transport for Stackdriver. Then you can analyze and query them there, and don't need to worry about storage running out.
2. logrotate
If you want to deal with this manually, you can configure logrotate. It will gzip older logs so that they consume less disk space. This is a sort of older, "pre-cloud" way of doing things.
Currently I have a ELK stack in which logs are shipped by filebeat and after some filters in logstash, it is forwarded to ES. As there are a lot of servers and logs, a huge logs are coming to logstash, but I have configured the filter to only process a very specific type of log message. Which it is doing fine, but the logs which are not even matching are logged in logstash.log file. As I mentioned earlier that huge logs are coming, the size of logstash.log file is soon reaching to a high value and there is space issue coming up. How to configure the logstash so that I only log the processed logs, and not all.
You could use logrotate to automatically rotate on either a daily basis or once it hits a certain threshold. You could then set the number of rotations to be 1 or 2. This would allow you time to see what is going to the file in case you need to troubleshoot, but purge before it creates space contention.
Currently we are working on forwarding logs to 2 different logstash servers. We cannot figure out a way with which we can install logstash-forwarder on a single machine. Is it possible with logstash-forwarder forwarding logs to multiple logstash ??
Else how can we do it with filebeat ??
In the LSF config, you can specify a list of hosts, but it will pick one at random and only switch to another in case of failure.
FB has the same system, but it allows you to also load balance across the list of hosts.
AFAIK, neither allows you to send events to multiple logstash instances.
Logstash, on the other hand, will send events to all of its outputs, so you could have FB send to a single LS, and have that LS output to your other LS instances. Note that if one output is unavailable, the system will block.