How to prevent logs getting loaded multiple times from logstash - logstash

I a newbie in ELk. I want to load the logs only once from logstash, again if run the config file the logs should not be read. And in the input plugin if i dont give start_position="beginning" the log will not load once also. and if I give start_position="beginning", each time I run conf file, the logs keeps on loading. Also tried with start_position="beginning", ignore_older=>0 this time logs are not loading not even once.

Related

Logstash listens to its own logs

I want to be able to track logstash logs in case of an error.
I want to be able to monitor issues when logstash tries to send events to output destination. I've checked the monitor API but it doesn't fulfills my requirements.
Is it a good practice to use file input plugin and set the path to its own log?

Change the log destination for node.js running on GCE

I am using rc.local to start my node script on start with:
node .> "/log_file_$(date +"%H:%M:%S_%m_%d_%Y").txt"
It works fine - but now once the log grows in size - I need to create a new log on a server every 12/24 hours; without restarting the server.
Is there any simple way to change the node app output destination?
I would prefer not to use any library for that, because I need to log all the messages including errors, warns, not only console.log.
Thanks for your help.
There are a number of options, I'll offer two:
1. Stackdriver
Stream your logs to Stackdriver, which is part of Google Cloud, and don't store them on your server at all. In your node.js application, you can can setup Winston and use the Winston transport for Stackdriver. Then you can analyze and query them there, and don't need to worry about storage running out.
2. logrotate
If you want to deal with this manually, you can configure logrotate. It will gzip older logs so that they consume less disk space. This is a sort of older, "pre-cloud" way of doing things.

Solution to bypass Logstash mixing logs

I'm currently in front of a structure problem with logstash.
I have a syslog-ng client sending logs from different files through the network to an ELK stack.
I noticed that Logstash is mixing logs, especially multiline with adding exception lines to non error logs from others files. So, i guess the trouble is that my logs are treated without any differentiation from their file origins. I found 2 ways to avoid that, but they are not optimal in my opinion:
Instead of using syslog-ng, use FileBeat and add a tag for each file which identify their origins. Then, parse my log with Logstash depending of this tag. The problem is that using syslog-ng as client is a requirement, and it will bother me if I have to change it
Change my syslog-ng sources to send each log files to a different port on ELK. I found that a little dirty and can be embarrassing with a great quantity of log files
What do you think about that ? Did I miss a better solution?
Is there a way to add a tag just like filebeat in syslog-ng?

What's the best way stream logs to CloudWatch Logs for a Spark Structured Streaming application?

The easiest solution I can think of is to attach an appender for CloudWatch Logs to Log4J (e.g., https://github.com/kdgregory/log4j-aws-appenders). The problem is this will not capture YARN logs, so if YARN failed to start the application altogether, nothing would reach CloudWatch about this failure.
Another option is to forward all spark-submit output (stdin and stdout) to a file and use CloudWatch Logs agent (installed on master) to stream everything. These will be simple text though, so I'll need to process the logs and extract date, level etc.
I'm running my application on AWS EMR. S3 logs are not an option as these are essentially archived logs and not real time.

Logstash should log only grok parsed messages

Currently I have a ELK stack in which logs are shipped by filebeat and after some filters in logstash, it is forwarded to ES. As there are a lot of servers and logs, a huge logs are coming to logstash, but I have configured the filter to only process a very specific type of log message. Which it is doing fine, but the logs which are not even matching are logged in logstash.log file. As I mentioned earlier that huge logs are coming, the size of logstash.log file is soon reaching to a high value and there is space issue coming up. How to configure the logstash so that I only log the processed logs, and not all.
You could use logrotate to automatically rotate on either a daily basis or once it hits a certain threshold. You could then set the number of rotations to be 1 or 2. This would allow you time to see what is going to the file in case you need to troubleshoot, but purge before it creates space contention.

Resources