I am Working on Logstash input Jdbc with different databases, Primarily I am trying for cassandra and I am using DBSchema driver, the driver is properly working with my java jdbc code, but when coming to integrate with Logstash it was failing to connect, I am trying this from many days, i did n't find any exact solution in forum, i tried all the things, but not worked, with maria db was also i tried it is also failing.
Here I am providing Logs while starting Logstash
Error: com.dbschema.CassandraJdbcDriver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
Exception: LogStash::ConfigurationError
Stack: /opt/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
Logstash conf file: (Cassandra)
#Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
input {
jdbc {
clean_run => true
jdbc_connection_string => "jdbc:cassandra://localhost:9042/cloud"
jdbc_user => "cassandra"
jdbc_password => "cassandra"
jdbc_driver_library => "/usr/share/logstash/logstash-core/lib/jars/cassandrajdbc1.2.jar"
jdbc_validate_connection => true
jdbc_driver_class => "com.dbschema.CassandraJdbcDriver"
statement => "SELECT * FROM cloud.event_history_all"
}
}
output {
elasticsearch { hosts => ["localhost:9200"]
index => "log_cassandra" }
stdout { codec => rubydebug }
}
Cassandra version - 3.11.5
elk versions - 7.1.1
java version - 11.0.5
Thanks in advance
Related
I have configured JMS input in logstash to subscribe to JMS topic messages and push messages to elastic search.
input {
jms {
id => "my_first_jms"
yaml_file => "D:\softwares\logstash-6.4.0\config\jms-amq.yml"
yaml_section => "dev"
use_jms_timestamp => true
pub_sub => true
destination => "mytopic"
# threads => 50
}
}
filter {
json{
source => "message"
}
}
output {
stdout { codec => json }
elasticsearch {
hosts => ['http://localhost:9401']
index => "jmsindex"
}
}
System specs:
RAM: 16 GB
Type: 64 bit
Processor: Intel i5-4570T CPU # 2.9 GHz
This is extremely slow. Like 1 message every 3-4 minutes. How should I debug to figure out what is missing?
Note: Before this, I was doing this with #JMSListener in java and that could process 200-300 records per sec easily.
Currently i am processing gzip files in logstash using file input plugin. its consuming very high memory and keeps on restarting even after giving a high heap size. As of now on an avg we are processing 50 files per min and the planning to process 1000's of file per min. With 100 files the RAM requirement touches 10Gb. What is the best way to tune this config or is there a better way to process such a huge volume of data in logstash.
is it advisable to write a processing engine in nodejs or any other languages.
Below is the logstash conf.
input {
file {
id => "client-files"
mode => "read"
path => [ "/usr/share/logstash/plugins/message/*.gz" ]
codec => "json"
file_completed_action => log_and_delete
file_completed_log_path => "/usr/share/logstash/logs/processed.log"
}
}
filter {
ruby {
code => 'monitor_name = event.get("path").split("/").last.split("_").first
event.set("monitorName", monitor_name )
split_field = []
event.get(monitor_name).each do |x|
split_field << Hash[event.get("Datapoints").zip(x)]
end
event.set("split_field",split_field)'
}
split {
field => "split_field"
}
ruby {
code => "event.get('split_field').each {|k,v| event.set(k,v)}"
remove_field => ["split_field","Datapoints","%{monitorName}"]
}
}
I am using Logstash 2.2.4.
My current configuration
input {
file {
path => "/data/events/*/*.txt"
start_position => "beginning"
codec => "json"
ignore_older => 0
max_open_files => 46000
}
}
filter {
if [type] not in ["A", "B", "C"] {
drop {}
}
}
output {
HTTP {
http_method => "post"
workers => 3
url => "http://xxx.amazonaws.com/event"
}
}
In an input folder, I have about 25000 static (never updatable) txt files.
I configured --pipeline-workers to 16. In described configuration LS process running 1255 threads and opens about 2,560,685 file descriptors.
After some investigation, I found that LS keeping open files descriptor for all the files in the input folder and HTTP output traffic became very slow.
My question is why LS does not close file descriptor of already processed (transferred) files or implementing kind of input files pagination?
Maybe someone meets the same problem? If yes, please share your solution.
Thanks.
I've got a problem with running logstash. My conf looks like this:
input {
udp {
port => 1514
type => docker
}
}
filter {
grok {
match => {
"message" => "<%{NUMBER}>%{DATA}(?:\s+)%{DATA:hostname}(?:\s+)%{DATA:imageName}(?:\s+)%{DATA:containerName}(?:\s*\[%{NUMBER}\]:) (\s+(?<logDate>%{YEAR}-%{MONTHNUM}-%{MONTHDAY}\s+%{HOUR}:%{MINUTE}:%{SECOND}) %{LOGLEVEL:logLevel}(?:\s*);* %{DATA:logAdditionalInfo};*)?%{GREEDYDATA:logMsg}"
}
keep_empty_captures => false
remove_field => ["message"]
}
}
output {
if [type] == "gelf" {
elasticsearch {
index => "phpteam-%{+YYYY.MM.dd}"
}
} else {
elasticsearch { }
}
}
The configuration is correct, but after running it /var/log/logstash/logstash.log shows the following output:
{:timestamp=>"2016-06-22T11:43:03.105000+0200", :message=>"SIGTERM
received. Shutting down the pipeline.", :level=>:warn}
{:timestamp=>"2016-06-22T11:43:03.532000+0200", :message=>"UDP
listener died", :exception=>#,
:backtrace=>["org/jruby/RubyIO.java:3682:in select'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:77:in
udp_listener'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-2.0.3/lib/logstash/inputs/udp.rb:50:in
run'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:206:in
inputworker'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.1-java/lib/logstash/pipeline.rb:199:in
`start_input'"], :level=>:warn}
The only thing I found to workaround this error is to edit those .rb files, but sadly I have no idea how to do it. Could you help me somehow?
Thanks in advance.
I found solution that is not perfect, but works, so maybe it will help somebody.
After installing the whole instance on new server everything works fine.
Everything crashed after upgrade'ing logstash/elasticsearch/kibana, so maybe there's something wrong with configuration files, but I couldn't figure out which ones.
I have ganglia set up on a cluster of servers, all of which have gmond, one of which has gmetad, and one which has log stash and elasticsearch. I’d like to use Logstash’s ganglia input plugin to collect data directly from the monitoring daemons, but I’ve been unsuccessful so far. My logstash logs always show:
{:timestamp=>"2015-07-14T14:33:25.192000+0000", :message=>"ganglia udp listener died", :address=>"10.1.10.178:8664", :exception=>#, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:160:in bind'", "/opt/logstash/lib/logstash/inputs/ganglia.rb:61:inudp_listener'", "/opt/logstash/lib/logstash/inputs/ganglia.rb:39:in run'", "/opt/logstash/lib/logstash/pipeline.rb:163:ininputworker'", "/opt/logstash/lib/logstash/pipeline.rb:157:in `start_input'"], :level=>:warn}
Here's the input config I've been testing with:
input {
ganglia {
host => "10.1.10.178" #ip of logstash node
port => 8666
type => "ganglia_test"
}
}
and I have this in gmond.conf on one of the gmond nodes
udp_send_channel {
host = 10.1.10.178 #logstash node
port = 8666
bind_hostname = yes
}
I've found this problem too. It looks like there's a bug in the Ganglia listener since about version 1.2 (I know it used to work in 1.1..)
I managed to work around the problem by adding an explicit 'UDP' listener. This seems to satisfy logstash and allows the Ganglia listener to keep running.
e.g.
input {
udp {
port => "1112"
type => "dummy"
}
ganglia {
port => "8666"
type => "ganglia"
}
}