I have been trying (and failing) to get logstash working with log4j SocketAppender.
Here is my log4j.properties:
log4j.appender.tcp=org.apache.log4j.net.SocketAppender
log4j.appender.tcp.Port=4560
log4j.appender.tcp.RemoteHost=localhost
log4j.appender.tcp.ReconnectionDelay=10000
log4j.appender.tcp.Application=playground
logstash.conf
input {
log4j {
}
}
output {
stdout {}
}
I have set logstash logging level to TRACE and sent a log message to it. Afterward I found the following lines in the console log of logstash:
11:44:35.778 [Ruby-0-Thread-21: c:/_work/issues/log4j_socketappender/logstash-5.0.1/vendor/bundle/jruby/1.9/gems/logstash-input-log4j-3.0.3-java/lib/logstash/inputs/log4j.rb:155] DEBUG logstash.inputs.log4j - Accepted connection {:client=>"127.0.0.1:61823", :server=>"0.0.0.0:4560"}
11:44:35.795 [Ruby-0-Thread-21: c:/_work/issues/log4j_socketappender/logstash-5.0.1/vendor/bundle/jruby/1.9/gems/logstash-input-log4j-3.0.3-java/lib/logstash/inputs/log4j.rb:155] DEBUG logstash.inputs.log4j - Closing connection {:client=>"127.0.0.1:61823", :exception=>#<IOError: org.apache.log4j.spi.LoggingEvent; class invalid for deserialization>}
Any help is much appreciated.
It's a bug. There is a workaround:
Locate file logstash-core/lib/jars.rb in the Logstash installation directory, comment out the below line:
require_jar('org.apache.logging.log4j', 'log4j-1.2-api', '2.6.2')
Discussed in here: GitHub Issue
Related
I am trying to run logstash on windows.
using the following command -
logstash.bat -f D:\confignfig.txt --path.settings D:\Software\Elasticsearch\logstash-5.5.0\config\ -l LogstashLogs
But I get this error -
ERROR StatusLogger No log4j2 configuration file found. Using default
configuration: logging only errors to the console. WARNING: Could not
find logstash.yml which is typically located in $LS_HOME/config or
/etc/logstash. You can specify the path using --path.settings.
Continuing using the defaults An unexpected error occurred! :error =>
bad URI(is not URI?):
file:///D:\Software\Elasticsearch\logstash-5.5.0\config" -
LogstashLogs"/log4j2.properties, :backtrace =>
["D:/Software/Elasticsearch/logstash-5.5.0/vendor/jruby/lib/ruby/1.9/uri/common.rb:176:in
split",
"D:/Software/Elasticsearch/logstash-5.5.0/vendor/jruby/lib/ruby/1.9/uri/common.rb:210:in
parse",
"D:/Software/Elasticsearch/logstash-5.5.0/vendor/jruby/lib/ruby/1.9/uri/common.rb:747:in
parse",
"D:/Software/Elasticsearch/logstash-5.5.0/vendor/jruby/lib/ruby/1.9/uri/common.rb:994:in
URI",
"D:/Software/Elasticsearch/logstash-5.5.0/logstash-core/lib/logstash/logging/logger.rb:77:in
initialize", "org/jruby/ext/thread/Mutex.java:149:in synchronize'",
"D:/Software/Elasticsearch/logstash-5.5.0/logstash-core/lib/logstash/logging/logger.rb:75:in
initialize",
"D:/Software/Elasticsearch/logstash-5.5.0/logstash-core/lib/logstash/runner.rb:229:in
execute",
"D:/Software/Elasticsearch/logstash-5.5.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in
run",
"D:/Software/Elasticsearch/logstash-5.5.0/logstash-core/lib/logstash/runner.rb:209:in
run",
"D:/Software/Elasticsearch/logstash-5.5.0/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in
run",
"D:\Software\Elasticsearch\logstash-5.5.0\lib\bootstrap\environment.rb:71:in
(root)"]
Any idea what is going here? Any suggestion will be much appreciated.
This seems to be a known issue and is expected to get fixed in next logstash release , 5.6.0.
Pls refer: https://discuss.elastic.co/t/logstash-5-5-0-error-statuslogger-no-log4j2-configuration-file-found/92921
https://github.com/elastic/logstash/issues/7526
Running logstash from root folder fixed this. I don't know if this is known issue or not but if you run logstash from bin folder you might get 'ERROR StatusLogger No log4j2 configuration file found'
This is just for future reference since I solved it myself.
When I switched from logstash 2.x to 5.x, I was dealing with this warning (when I was runnig my logstash on this path D:\somepath\logstash-5.0.1):
Could not find log4j2 configuration at path /somepath/logstash-5.0.1/config/log4j2.properties. Using default config which logs to console
After some searching on internet and digging in ruby code (in the extracted logstash) I have found out this:
necessary to use path.settings (as mentioned many times) correctly
use correctly file or directory as URL path.
Finally I run my logstash as:
logstash.bat --path.settings=file://D:/somepath/logstash-5.0.1/config
I am trying to suppress the message
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
when i run my Spark app. I've redirected the INFO messages successfully, however this message keeps on showing up. Any ideas would be greatly appreciated.
Even simpler you just cd SPARK_HOME/conf then mv log4j.properties.template log4j.properties then open log4j.properties and change all INFO to ERROR. Here SPARK_HOME is the root directory of your spark installation.
Some may be using hdfs as their Spark storage backend and will find the logging messages are actually generated by hdfs. To alter this, go to the HADOOP_HOME/etc/hadoop/log4j.properties file. Simply change hadoop.root.logger=INFO,console to hadoop.root.logger=ERROR,console. Once again HADOOP_HOME is the root of your hadoop installation for me this was /usr/local/hadoop.
Okay, So I've figured out a way to do this. So basically, I had my own log4j.xml initially, that was being used, and hence we were seeing this property. Once I had my own "log4j.properties" file, this message went away.
If you put a log4j.properties file under both the main/resources and the test/resources this also occurs. In this case, deleting the file from the test/resources and using only the file from the main/resources fixes the issue.
None of the answers above did work for me using SBT. Turns out you need to explicitly define an appender in your log4j.properties, such as:
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.Target=System.out
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%d{HH:mm:ss} %-5p %c{1}:%L - %m%n
log4j.rootLogger=WARN, stdout
log4j.logger.org.apache.spark=WARN, stdout
log4j.logger.com.yourcompany=INFO, stdout
Put this in your resources directory and Bob's your uncle!
I would like to parse a directory of logs files with logstash.
When the logs are formatted like this :
server-20140604.log
server-20140603.log
server-20140602.log
There is no problem, I am using globs like this :
input {
file {
path=>["D:/*.log"]
}
}
But my logs are formatted like this :
server.log
server.log.1
server.log.2
client.log
client.log.1
client.log.2
So I would like to know how to tell to logstash to parse in the folder all the files starting with "server" expression in their names. I really need to do it like that, because I have other files in the folder (i.e client logs) that I don't want to parse but also cannot remove from the folder.
With this configuration I can only parse all the log files start with prefix server.
input {
file {
path => ["D:/server*"]
}
}
output {
stdout {
codec => rubydebug
}
}
I think the possible problem you have meet is the start_position config. It means that where does logstash start to read the logs. Please refer to here. Remember this option only modifies first contact situations where a file is new and not seen before. If a file has already been seen before, this option has no effect.
When you stop logstash, logstash will save a .sincedb* in your home directory. Next time you start it, logstash will start read the file according to .sindb*. If you do not input new logs to server.log, logstash will never parse the old logs.
What you can try to do is delete all the .sincedb before you start logstash and add start_posistion to your config. In your comment you have say if you overwrite the server.log logstash can parse the file from beginning, it is because logstash detect it as a new file and the .sincedb* do not save any information about this file. So logstash will parse it! You can try to find out your .sincedb and try to delete it.
I have been very frustrated about this.
I am trying to do the following:
Log all application related logs in application.log that are INFO or above
Understand what controls the configuration for catalina.out
Log only WARN to catalina.out
I am running my server on ubuntu and I have the default configuration for tomcat which includes a conf directory with a logging.properties. I renamed this file to l.p so it wouldn't conflict. (Not sure if this is a good idea)
In my config file, I have:
def catalinaBase = System.properties.getProperty('catalina.base')
if (!catalinaBase) catalinaBase = '.' // just in case
def logDirectory = "${catalinaBase}${File.separator}logs"
println "Log Directory: ${logDirectory}"
log4j = {
appenders {
rollingFile name: 'applog', file: "${logDirectory}${File.separator}application.log", layout: pattern(conversionPattern: '%d{dd-MM-yyyy HH:mm:ss,SSS} %5p %c{1} - %m%n'), maxFileSize: 1024
}
error 'org.codehaus.groovy.grails.web.servlet', // controllers
'org.codehaus.groovy.grails.web.pages', // GSP
'org.codehaus.groovy.grails.web.sitemesh', // layouts
'org.codehaus.groovy.grails.web.mapping.filter', // URL mapping
'org.codehaus.groovy.grails.web.mapping', // URL mapping
'org.codehaus.groovy.grails.commons', // core / classloading
'org.codehaus.groovy.grails.plugins', // plugins
'org.codehaus.groovy.grails.orm.hibernate', // hibernate integration
'org.springframework',
'org.hibernate',
'net.sf.ehcache.hibernate'
warn 'org.mortbay.log'
info applog: 'grails.app'
root {
info 'applog'
}
}
As a result, I am getting three logs:
catalina.2011-01-17.log catalina.out localhost.2011-01-17.log
The catalina.out has the following output:
Log Directory: /var/lib/tomcat6/logs
log4j:WARN No appenders could be found for logger (org.apache.commons.beanutils.PropertyUtils).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
log4j:ERROR WARNING: Exception occured configuring log4j logging: Cannot invoke org.apache.log4j.FileAppender.setFile - argument type mismatch
I do NOT see the application.log in the log file directory. Any help would be appreciated I am really frustrated about this.
One more thing, in windows everything come out to the console and the application.log is created in the .grails\1.3.5\projects\<appnmae>\tomcat directory
Your problem is obviously that there's a type mismatch. Specifically, log4j is expecting a String when you're giving it a GString. Try replacing:
"${logDirectory}${File.separator}application.log"
With this:
"${logDirectory}${File.separator}application.log".toString()
EDIT: Please read this BUG