I'm trying to sync some tables from MariaDB to ElasticSearch using LogStash.
I'm on a Debian Buster (10) Server
$ java -version
openjdk version "11.0.4" 2019-07-16
OpenJDK Runtime Environment (build 11.0.4+11-post-Debian-1deb10u1)
OpenJDK 64-Bit Server VM (build 11.0.4+11-post-Debian-1deb10u1, mixed mode, sharing)
$ mariadb --version
mariadb Ver 15.1 Distrib 10.3.15-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
$ /usr/share/logstash/bin/logstash --version
logstash 7.2.0
I tried different connectors :
$ ls -l /usr/share/java/
mariadb-java-client.jar
$ ls -l /etc/logstash/connectors/
mariadb-java-client-2.1.2.jar
mariadb-java-client-2.2.6.jar
mariadb-java-client-2.3.0.jar
mariadb-java-client-2.4.2.jar
mysql-connector-java-8.0.17.jar
Using "org.mariadb.jdbc.Driver" for mariadb connectors and "com.mysql.cj.jdbc.Driver" for mysql connector
$ cat /etc/logstash/conf.d/db-fr-bank.conf
input {
jdbc {
jdbc_connection_string => "jdbc:mariadb://localhost:3306/db_fr"
jdbc_user => "logstash"
jdbc_password => "<password>"
jdbc_driver_library => "/usr/share/java/mariadb-java-client.jar"
jdbc_driver_class => "org.mariadb.jdbc.Driver"
statement => "SELECT * FROM bank"
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "fr-bank"
}
}
But, instead of syncing, i keep getting :
$ /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/db-fr-bank.conf
...
[ERROR] 2019-07-29 02:08:17.563 [[main]<jdbc] jdbc - Failed to load /usr/share/java/mariadb-java-client.jar {:exception=>#<TypeError: failed to coerce jdk.internal.loader.ClassLoaders$AppClassLoader to java.net.URLClassLoader>}
[ERROR] 2019-07-29 02:08:17.598 [[main]<jdbc] javapipeline - A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::Jdbc jdbc_user=>"logstash", jdbc_password=><password>, statement=>"SELECT * FROM bank", jdbc_driver_library=>"/usr/share/java/mariadb-java-client.jar", jdbc_connection_string=>"jdbc:mariadb://localhost:3306/db_fr", id=>"38a6d112755a5e87278761cf5f41b7e509212d1d02837a03672df2face00943a", jdbc_driver_class=>"org.mariadb.jdbc.Driver", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_f3094292-7482-4b73-95c4-7f78da4da911", enable_metric=>true, charset=>"UTF-8">, jdbc_paging_enabled=>false, jdbc_page_size=>100000, jdbc_validate_connection=>false, jdbc_validation_timeout=>3600, jdbc_pool_timeout=>5, sql_log_level=>"info", connection_retry_attempts=>1, connection_retry_attempts_wait_time=>0.5, parameters=>{"sql_last_value"=>1970-01-01 00:00:00 UTC}, last_run_metadata_path=>"/root/.logstash_jdbc_last_run", use_column_value=>false, tracking_column_type=>"numeric", clean_run=>false, record_last_run=>true, lowercase_column_names=>true>
Error: org.mariadb.jdbc.Driver not loaded. Are you sure you've included the correct jdbc driver in :jdbc_driver_library?
Exception: LogStash::ConfigurationError
Stack: /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:163:in `open_jdbc_connection'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/plugin_mixins/jdbc/jdbc.rb:221:in `execute_statement'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:277:in `execute_query'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-jdbc-4.3.13/lib/logstash/inputs/jdbc.rb:263:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:309:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:302:in `block in start_input'
...
Same issue here.
I use the workaround from here with this and it works.
ie:
Copy the driver file to {logstash install dir}/logstash-core/lib/jars/ directory. These jars get added to the correct JDK classpath as logstash is started via java.
And
Change the jdbc_driver_library value in the logstash conf to "". i.e.: jdbc_driver_library => "" as well, otherwise the code still tries to load the jar separately
Related
I have a problem about running defined configuration file in logstash though its command.
Here is my conf file shown below.
input {
file {
type => "userslog"
path => "C:\Users\aaa\Downloads\logstash-8.1.0\users-ms.log"
}
file {
type => "albumslog"
path => "C:\Users\aaa\Downloads\logstash-8.1.0\albums-ms.log"
}
}
output {
if[type]=="userslog"{
elasticsearch {
hosts => ["localhost:9200"]
index => "userslog-%{+YYYY.MM.dd}"
}
} else if[type]=="albumslog"{
elasticsearch {
hosts => ["localhost:9200"]
index => "albumslog-%{+YYYY.MM.dd}"
}
}
stdout {codec => rubydebug}
}
Here is the result shown below.
C:\Users\aaa\Desktop\logstash-8.1.0\bin>logstash logstash-simple.conf
"Using bundled JDK: ."
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[FATAL] 2022-03-17 12:03:32.267 [main] Logstash - Logstash stopped processing because of an error: (NameError) missing class name (`org.apache.http.─▒mpl.client.StandardHttpRequestRetryHandler')
org.jruby.exceptions.NameError: (NameError) missing class name (`org.apache.http.─▒mpl.client.StandardHttpRequestRetryHandler')
When changed 'Java::OrgApacheHttpImplClient::StandardHttpRequestRetryHandler' to 'Java::OrgApacheHttp.impl.client::StandardHttpRequestRetryHandler', it didn't work.
How can I fix it?
input {
jdbc {
jdbc_connection_string => “jdbc:postgresql://localhost:5432/FraudInsuranceBizapp”
jdbc_user => “postgres”
jdbc_password => “postgres”
jdbc_driver_class => “org.postgresql.Driver”
statement => “SELECT * from \”jhi_user\””
}
}
output {
elasticsearch {
hosts => [“http://localhost:9200"]
index => “jhi_user”
document_id => “users_%{id}”
doc_as_upsert => true
#user => “es_user”
#password => “es_password”
}
}
And when i run logstash from the command line .\bin\logstash.bat -f
D:\MESCN\softwares\logstash-7.13.2\config\jhi-user.conf getting
the error below
Using JAVA_HOME defined java: "D:\MESCN\softwares\logstash-7.13.2\jdk"
WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future
release.
Sending Logstash logs to D:/MESCN/softwares/logstash-7.13.2/logs which is now configured via log4j2.properties
[2021-06-24T10:11:35,655][INFO ][logstash.runner ] Log4j configuration path used is:
D:\MESCN\softwares\logstash-7.13.2\config\log4j2.properties
[2021-06-24T10:11:35,689][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.13.2",
"jruby.version"=>"jruby 9.2.16.0 (2.5.7) 2021-03-03 f82228dc32 OpenJDK
64-Bit Server VM 11.0.11+9 on 11.0.11+9 +indy +jit [mswin32-x86_64]"}
[2021-06-24T10:11:35,989][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file
because modules or command line options are specified
[2021-06-24T10:11:38,460][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-06-24T10:11:38,776][ERROR][logstash.agent ] Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of
[ \\t\\r\\n], \"#\", [A-Za-z0-9_-], '\"', \"'\", [A-Za-z_], \"-\",
[0-9], \"[\", \"{\" at line 3, column 32 (byte 51) after input {\r\n
jdbc {\r\n jdbc_connection_string => ",
:backtrace=>["D:/MESCN/softwares/logstash-7.13.2/logstash-core/lib/logstash/compiler.rb:32:in
`compile_imperative'",
"org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'",
"org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'",
"D:/MESCN/softwares/logstash-7.13.2/logstash-core/lib/logstash/java_pipeline.rb:47:in
`initialize'",
"D:/MESCN/softwares/logstash-7.13.2/logstash-core/lib/logstash/pipeline_action/create.rb:52:in
`execute'",
"D:/MESCN/softwares/logstash-7.13.2/logstash-core/lib/logstash/agent.rb:389:in
`block in converge_state'"]}
[2021-06-24T10:11:39,047][INFO ][logstash.runner ] Logstash shut down.
[2021-06-24T10:11:39,071][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747)
~[jruby-complete-9.2.16.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710)
~[jruby-complete-9.2.16.0.jar:?]
at D_3a_.MESCN.softwares.logstash_minus_7_dot_13_dot_2.lib.bootstrap.environment.<main>(D:\MESCN\softwares\logstash-7.13.2\lib\bootstrap\environment.rb:89)
~[?:?]
jdbc_connection_string => “jdbc:postgresql://localhost:5432/FraudInsuranceBizapp”
logstash will not accept curly quotes. You need to use regular double quotes. The same applies to all of the options in the input and output.
Currently trying to populate the employee index with the below settings:
CONF
input {
jdbc {
jdbc_driver_library => "~/sqljdbc_6.2/enu/mssql-jdbc-6.2.1.jre8.jar"
jdbc_driver_class => "com.microsoft.sqlserver.jdbc.SQLServerDriver"
jdbc_connection_string => "jdbc:sqlserver://SERVER;user=USER;password=PASSWORD"
jdbc_user => "DB_USER"
jdbc_password => "DB_PASSWORD"
jdbc_validate_connection => true
jdbc_validation_timeout => -1
statement => "SELECT * FROM [dbo].Employee ORDER BY ID"
type => "employee"
}
}
filter {
}
output {
}
NOTE: filter and output sections of the conf file is purposely blank
LINUX COMMAND
sudo /usr/share/logstash/bin/logstash -f /home/ubuntu/Employee-pipeline.conf --path.settings /etc/logstash/ --path.data /var/lib/logstash_new
RESULT
Looks like logstash does not know or don't have access to ~/sqljdbc...*.jar
I also confirmed that the mssql-jdbc-6.2.1.jre8.jar exists
However, when I changed the path to /home/ubuntu/sqljdbc_6.2/enu/mssql-jdbc-6.2.1.jre8.jar, it runs successfully.
So ~/ is the same as /home/ubuntu
This started to occur after upgrading our Elastic Stack from v5.5 to v5.6. Also, note that this does not occur if we run the same conf file with the logstash service.
I have a problem with my logstash that can't send log to elaticsearch.
With following details
Logstash version : 1.5.1
Elasticsearch version : 1.6.0
jvm on both servers version : 1.8.0
Linux 3.10.0-229.7.2.el7.x86_64 #1 SMP Tue Jun 23 22:06:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
Azure Openlogic 7.1
Here is my logstash.err file
INFO: [ls1] failed to send join request to master
[[es1][e8A0li5pRfeMklozmDXgkQ][elastic][inet[/x.x.x.x:9300]]], reason
[RemoteTransportException[[es1][inet[/x.x.x.x:9300]]
[internal:discovery/zen/join]]; nested:
ConnectTransportException[[ls1][inet[/x.x.x.x:9300]]
connect_timeout[30s]]; nested: ConnectTimeoutException[connection
timed out: /x.x.x.x:9300]; ]
My logstash configuration output
output {
elasticsearch {
host => "x.x.x.x"
bind_port => 9300
index => "syslog"
cluster => "test-cluster"
node_name => 'ls1'
}
stdout {
codec => rubydebug
}
}
Here is my elasticsearch.yml configuration file in elasticsearch server
cluster.name: test-cluster
node.name: "es1"
network.bind_host: 0.0.0.0
network.publish_host: <my_elasticsearch_public_ip>
transport.tcp.port: 9300
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["my_logstash_public_ip:9300"]
Here is my elasticsearch.yml file in logstash server (/var/lib/logstash)
network.publish_host: my_logstash_public_ip
discovery.zen.ping.multicast.enabled: false
I've allowed port 9300 on both servers.
You need to include protocol attribute in your logstash configuration. find the below updated code.
output {
elasticsearch {
host => "x.x.x.x"
protocol => "http"
bind_port => 9300
index => "syslog"
cluster => "test-cluster"
node_name => 'ls1'
}
I use Microsoft Azure VM, Now I can solve this problem by create VPN connection via azure virtual machines.
I have ELK installed on a VM in my laptop.Elasticsearch is up and running.
./bin/logstash -f logstash-filter.conf gives me the below error
Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.
I tried setting up JAVA_HOME and $ PATH, still the issue is persistent. Am I missing something?
which java
/usr/bin/java
java -version
java version "1.7.0_79"
OpenJDK Runtime Environment (IcedTea 2.5.5) (7u79-2.5.5-0ubuntu0.14.04.2)
OpenJDK 64-Bit Server VM (build 24.79-b02, mixed mode)
echo $JAVA_HOME
/usr/local/java/jdk1.8.0_45
echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/divija/bin:/usr/local/java/jdk1.8.0_45/bin
logstash-filter.conf
input { stdin { } }
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { host => localhost
index=>"myindex"
}
stdout { codec => rubydebug }`enter code here`
}
I had to
export JAVACMD=`which java`
to make this work.
I know, it's an old thread, but I was also having the same problem and was doing a very silly thing.
I had updated my $JAVA_HOME var in /etc/environment but was not reloading the file because of which it was not taking effect, and running source /etc/environment solved my problem.