logstash opensearch oss output plugin integration with opensearch - logstash

Question

I am trying to do this logstash and opensearch setup. Here are the details of the docker-compose file that i am using.
opensearch and dasboard service docker-compose file.
version: '3'
services:
opensearch-node1:
image: opensearchproject/opensearch:2.3.0
#image: opensearchproject/opensearch:latest
container_name: opensearch-node1
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node1
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
hard: 65536
volumes:
- opensearch-data1:/usr/share/opensearch/data
ports:
- 9200:9200
- 9600:9600 # required for Performance Analyzer
networks:
- opensearch-net
opensearch-node2:
#image: opensearchproject/opensearch:latest
image: opensearchproject/opensearch:2.3.0
container_name: opensearch-node2
environment:
- cluster.name=opensearch-cluster
- node.name=opensearch-node2
- discovery.seed_hosts=opensearch-node1,opensearch-node2
- cluster.initial_cluster_manager_nodes=opensearch-node1,opensearch-node2
- bootstrap.memory_lock=true
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- opensearch-data2:/usr/share/opensearch/data
networks:
- opensearch-net
opensearch-dashboards:
image: opensearchproject/opensearch-dashboards:2.3.0
#image: opensearchproject/opensearch-dashboards:latest
container_name: opensearch-dashboards
ports:
- 5601:5601
expose:
- "5601"
environment:
OPENSEARCH_HOSTS: '["https://opensearch-node1:9200","https://opensearch-node2:9200"]' # must be a string with no spaces when specified as an environment variable
networks:
- opensearch-net
volumes:
opensearch-data1:
opensearch-data2:
networks:
opensearch-net:
logstash oss
version: '2.1'
services:
logstash:
#image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.3
image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
ports:
- "5044:5044"
volumes:
- $PWD/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
networks:
- opensearch-net
networks:
opensearch-net:
and here is the logstash.conf file.
input {
stdin { }
}
filter {
}
output {
opensearch {
hosts => ["https://opensearch_fqdn:9200/"]
index => "testindexing"
user => "admin"
password => "admin"
ssl => true
ssl_certificate_verification => false
}
}
I am expecting that the testingindexing would be be created with the data that is sent to the stdin terminal. But i get the below error.
Error
logstash_1 | [2022-10-25T07:44:39,946][ERROR][logstash.outputs.opensearch][main] Failed to install template {:message=>"Failed to load default template for OpenSearch v2 with ECS disabled; caused by: #<ArgumentError: Template file '/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/templates/ecs-disabled/2x.json' could not be found>", :exception=>RuntimeError, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:33:in `load_default_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch/template_manager.rb:21:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:412:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:247:in `finish_register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/outputs/opensearch.rb:224:in `block in register'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-opensearch-1.2.0-java/lib/logstash/plugin_mixins/opensearch/common.rb:83:in `block in after_successful_connection'"]}
As per the below issue details - opensearch-project/opensearch-devops#85, i tried to use the opensearchproject/logstash-oss-with-opensearch-output-plugin:8.4.0 for the logstash oss output plugin but the logstash terminates without much information.
logstash_1 | [2022-10-25T12:03:11,405][INFO ][logstash.outputs.opensearch][main] New OpenSearch output {:class=>"LogStash::Outputs::OpenSearch", :hosts=>["https://opensearch_fqdn:9200/"]}
logstash_1 | [2022-10-25T12:03:11,435][WARN ][logstash.outputs.opensearch][main] ** WARNING ** Detected UNSAFE options in opensearch output configuration!
logstash_1 | ** WARNING ** You have enabled encryption but DISABLED certificate verification.
logstash_1 | ** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
logstash_1 | [2022-10-25T12:03:11,691][INFO ][logstash.outputs.opensearch][main] OpenSearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://admin:xxxxxx#opensearch_fqdn:9200/]}}
logstash_1 | [2022-10-25T12:03:11,903][WARN ][logstash.outputs.opensearch][main] Restored connection to OpenSearch instance {:url=>"https://admin:xxxxxx#opensearch_fqdn:9200/"}
logstash_1 | [2022-10-25T12:03:11,956][INFO ][logstash.outputs.opensearch][main] Cluster version determined (2.3.0) {:version=>2}
logstash_1 | [2022-10-25T12:03:12,039][INFO ][logstash.outputs.opensearch][main] Using a default mapping template {:version=>2, :ecs_compatibility=>:v8}
logstash_1 | [2022-10-25T12:03:12,058][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x50914657 run>"}
logstash_1 | [2022-10-25T12:03:12,571][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.51}
logstash_1 | [2022-10-25T12:03:12,622][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1 | [2022-10-25T12:03:12,692][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1 | [2022-10-25T12:03:13,008][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
logstash_1 | [2022-10-25T12:03:13,262][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
logstash_1 | [2022-10-25T12:03:13,360][INFO ][logstash.runner ] Logstash shut down.
Any suggestion what i am missing here.