I have recently taken over running a Logstash system which runs on debian 9. The previous owner had installed an older version of Logstash and has left incomplete documentation on the project. I have successfully configured Logstash 7.2 locally on windows 10 and have tried to transfer this across to the Debian system replacing the necessary paths etc. I'm comming up against the following error and despite hours searching for a clue I'm left scratching my head. Any pointers would be appreciated!
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/home/user/logstash/logstash-7.2.0/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.FileDescriptor.fd
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /home/user/logstash/logstash-7.2.0/ which is now configured via log4j2.properties
[2020-07-21T08:04:35,773][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-07-21T08:04:35,781][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.2.0"}
[2020-07-21T08:04:37,165][INFO ][logstash.outputs.jdbc ] JDBC - Starting up
[2020-07-21T08:04:37,195][INFO ][com.zaxxer.hikari.HikariDataSource] HikariPool-1 - Starting...
[2020-07-21T08:04:45,302][INFO ][com.zaxxer.hikari.HikariDataSource] HikariPool-1 - Start completed.
[2020-07-21T08:04:45,404][ERROR][logstash.javapipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<ArgumentError: invalid byte sequence in UTF-8>, :backtrace=>["org/jruby/RubyRegexp.java:1113:in `=~'", "org/jruby/RubyString.java:1664:in `=~'", "/home/user/logstash/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:72:in `block in add_patterns_from_file'", "org/jruby/RubyIO.java:3329:in `each'", "/home/user/logstash/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:70:in `add_patterns_from_file'", "/home/user/logstash/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:403:in `block in add_patterns_from_files'", "org/jruby/RubyArray.java:1792:in `each'", "/home/user/logstash/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:399:in `add_patterns_from_files'", "/home/user/logstash/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:279:in `block in register'", "org/jruby/RubyArray.java:1792:in `each'", "/home/user/logstash/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in `block in register'", "org/jruby/RubyHash.java:1419:in `each'", "/home/user/logstash/logstash-7.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:270:in `register'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "/home/user/logstash/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:192:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/home/user/logstash/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `register_plugins'", "/home/user/logstash/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:463:in `maybe_setup_out_plugins'", "/home/user/logstash/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:204:in `start_workers'", "/home/user/logstash/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:146:in `run'", "/home/user/logstash/logstash-7.2.0/logstash-core/lib/logstash/java_pipeline.rb:105:in `block in start'"], :thread=>"#<Thread:0x1bda40f7 run>"}
[2020-07-21T08:04:45,422][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2020-07-21T08:04:45,553][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-07-21T08:04:50,602][INFO ][logstash.runner ] Logstash shut down.
solved by adding ":ISO-8859-1:UTF-8" to grok-pure.rb:72
file = File.new(path, "r:ISO-8859-1:UTF-8")
I later noticed that the file encoding of the patterns file was set to text/plain; charset=us-ascii through the command "file -bi file_name". Setting this to UTF8 may also have had an impact.
the issue is because you should not have any other file under patterns_dir except grok patterns. I was having some rpm in that folder that caused the issue
Related
We've upgraded our ELK stack from 6.4 version to 6.8.20 version. Elasticsearch and Kibana were upgraded without any issues. But we observed that Logstash service is restarting continuously after the upgradation.
Elastic: 6.8.20
Kibana:6.8.20
Logstash: 6.8.20
We found the below error logs in Logstash.
[ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Beats codec=><LogStash::Codecs::JSON id=>\"json :error=>"cannot link Java class io.netty.handler.ssl.OpenSsl, probable missing dependency: io.netty.internal.tcnative.SSLContext.setCipherSuite(JLjava/lang/String;Z)Z", :thread=>"#<Thread:0x4768d63d run>"}
[2021-11-25T09:49:30,071][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<NameError: cannot link Java class io.netty.handler.ssl.OpenSsl, probable missing dependency: io.netty.internal.tcnative.SSLContext.setCipherSuite(JLjava/lang/String;Z)Z>, :backtrace=>["org/jruby/javasupport/JavaClass.java:292:in `for_name'", "org/jruby/javasupport/JavaClass.java:288:in `for_name'", "org/jruby/javasupport/JavaUtilities.java:38:in `get_proxy_class'", "uri:classloader:/jruby/java/core_ext/object.rb:44:in `block in java_import'", "org/jruby/RubyArray.java:2577:in `map'", "uri:classloader:/jruby/java/core_ext/object.rb:37:in `java_import'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-5.1.9-java/lib/logstash/inputs/beats.rb:146:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:413:in `start_inputs'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:311:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176:in `block in start'"], :thread=>"#<Thread:0x4367d63d run>"}
[2021-11-25T09:49:30,091][ERROR][logstash.agent
[ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"<LogStash::Inputs::Beats codec=><LogStash::Codecs::JSON id=>\"json :error=>"cannot link Java class io.netty.handler.ssl.OpenSsl, probable missing dependency: io.netty.internal.tcnative.SSLContext.setCipherSuite(JLjava/lang/String;Z)Z", :thread=>"#<Thread:0x4768d63d run>"}
We were unable to resolve this issue while performing upgrade ELK upgrade logstash from 6.4 --> 6.8 --> 7.15 . To fix this issue we upgraded our Logstash directly to 7.15 ( all the other components were at 6.8 elastic, kibana, filebeat etc) .
7.15 (or 7.12) version comes with its own JDK and removes necessity to have jdk installed separately. Also the codecs written in JRUBY works perfectly here.
I wanted to update my GitLab CE installed on a EC2 instance following the recommendation. I went from 13.0.x to 13.9.2 -> 13.12.12 -> 14.0.11 -> 14.1.6 as the documentation recommend.
Every time I check if the server was responding. I just had to restart the process when I jump to 14.X.
But when I went to 14.1.6 I got an error on the update :
[...]
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.4p191 (2021-07-07 revision a21a3b7d23) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Recipe: gitlab::gitlab-rails
* execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Recipe: gitaly::enable
* runit_service[gitaly] action restart (up to date)
* runit_service[gitaly] action hup
- send hup to runit_service[gitaly]
Running handlers:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"events", :column_name=>"id", :job_arguments=>[["id"], ["id_convert_to_bigint"]]}
Finalize it manualy by running
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
For more information, check the documentation
https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1109:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210622045705_finalize_events_bigint_conversion.rb:11:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migrations/lock_retry_mixin.rb:31:in `ddl_transaction'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrating ==================
-- add_column(:plan_limits, :pages_file_entries, :integer, {:default=>200000, :null=>false})
-> 0.0021s
== 20210611082822 AddPagesFileEntriesToPlanLimits: migrated (0.0022s) =========
== 20210622045705 FinalizeEventsBigintConversion: migrating ===================
STDERR:
---- End output of "bash" "/tmp/chef-script20211028-68712-eysjor" ----
Ran "bash" "/tmp/chef-script20211028-68712-eysjor" returned 1
Running handlers complete
Chef Infra Client failed. 27 resources updated in 51 seconds
===
There was an error running gitlab-ctl reconfigure. Please check the output above for more
details.
===
dpkg: error processing package gitlab-ce (--configure):
installed gitlab-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
gitlab-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
So as the output told me I tried :
sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,'[["id"]\, ["id_convert_to_bigint"]]']
and got
rake aborted!
JSON::ParserError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:87:in `rescue in adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:82:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Oj::ParseError: unexpected character () at line 1, column 1 [parse.c:714]
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:85:in `adapter_load'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/json.rb:25:in `parse'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:background_migrations:finalize
(See full trace by running task with --trace)
Any idea ?
After upgrading from 13 to 14.0 gitlab will commence several database migrations in the background (see yourgitlabserver.example.com/admin/background_migrations).
You have to wait for these to complete before advancing any further.
If you already did, there are several ways to fix this (see documentation).
I had the same error in this situation. To find out what's going on I've added following code into the
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/background_migrations.rake:17
puts "JSON: '#{args[:job_arguments]}'"
to have the code looks like this:
puts "JSON: '#{args[:job_arguments]}'"
Gitlab::Database::BackgroundMigration::BatchedMigrationRunner.finalize(
args[:job_class_name],
args[:table_name],
args[:column_name],
Gitlab::Json.parse(args[:job_arguments])
)
And I noticed that command args[:job_arguments] returns invalid JSON string.
So finally for me (zsh) worked to execute the following command:
sudo gitlab-rake 'gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,events,id,[["id"]\, ["id_convert_to_bigint"]]]'
The difference is in single quotes.
I'm new to the elastic stack and im trying to set it up with RabbitMQ using this guide(but in .NET):
https://piotrminkowski.com/2017/02/03/how-to-ship-logs-with-logstash-elasticsearch-and-rabbitmq/
When I startup Logstash I get the errors
[2020-11-14T09:51:50,997][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [0-9], [ \\t\\r\\n], \"#\", \"}\" at line 2, column 16 (byte 35) after input { rabbitmq {\nhost => 192.168", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2020-11-14T09:51:51,296][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-11-14T09:51:56,179][INFO ][logstash.runner ] Logstash shut down.
[2020-11-14T09:51:56,209][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
I don't know what is wrong but I can see that the nhost is "192.168" which probably isnt right, my ip is 192.168.0.29
I'm thankfull for any help
The host option for a rabbitmq input takes a string. A string should be surrounded by double (or single) quotes.
The configuration compiler is quite forgiving, and in many places will accept a "bareword" in place of a string, so it would accept localhost, but you cannot have punctuation in a "bareword", so example.com would result in an error. Likewise, once it sees the periods in the IP address it throws an exception.
Try
host => "192.168.0.29"
I am not able to get any output on the command prompt screen
E:\kibana\logstash-7.1.1\logstash-7.1.1>bin\logstash -f E:\kibana\logstash-7.1.1\logstash-7.1.1\config\pipeline.conf --config.reload.automatic
Java HotSpot(TM) 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.runtime.encoding.EncodingService (file:/E:/kibana/logstash-7.1.1/logstash-7.1.1/logstash-core/lib/jars/jruby-complete-9.2.7.0.jar) to field java.io.Console.cs
WARNING: Please consider reporting this to the maintainers of org.jruby.runtime.encoding.EncodingService
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Sending Logstash logs to E:/kibana/logstash-7.1.1/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-06-14T12:33:19,407][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-06-14T12:33:19,427][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-06-14T12:33:22,210][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, :thread=>"#<Thread:0x6177c4b4 run>"}
[2019-06-14T12:33:23,035][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"E:/kibana/logstash-7.1.1/logstash-7.1.1/data/plugins/inputs/file/.sincedb_039f8a57349afd1e3fb106bf0e1c330b", :path=>["/E/kibana/logstash-7.1.1/logstash-7.1.1/data/event-data/apache_access.log"]}
[2019-06-14T12:33:23,119][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-06-14T12:33:23,189][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-06-14T12:33:23,198][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-06-14T12:33:23,479][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
I am only getting this and not the output. What could be going wrong?
I did follow the description of the official elastic.co documentation when I was configuring the ELK stack. As per the https://www.elastic.co/guide/en/logstash/master/configuration-file-structure.html#_escape_sequences page shows how to set Logstash to enable escape characters, I just uncommented the line config.support_escapes: true in my logstash.yml.
When I start my ELK stack in my Docker container,I get the following error:
An unexpected error occurred! {:error=># ArgumentError: Setting "config.support_escapes" hasn't been registered>,
:backtrace=>["/opt/logstash/logstash-core/lib/logstash/settings.rb:32:in `get_setting'",
"/opt/logstash/logstash-core/lib/logstash/settings.rb:64:in `set_value'",
"/opt/logstash/logstash-core/lib/logstash/settings.rb:83:in `merge'",
"org/jruby/RubyHash.java:1342:in `each'",
"/opt/logstash/logstash-core/lib/logstash/settings.rb:83:in `merge'",
"/opt/logstash/logstash-core/lib/logstash/settings.rb:135:in `validate_all'",
"/opt/logstash/logstash-core/lib/logstash/runner.rb:244:in `execute'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'",
"/opt/logstash/logstash-core/lib/logstash/runner.rb:209:in `run'",
"/opt/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'",
"/opt/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
What did I miss here?