Elasticsearch cluster isn't shown up - linux

Hi I installed Elasticsearch 6.6 with Ansible playbook over a cluster with 3 nodes.
All nodes are on the same port.
When I run the query:
curl -u es_admin:<pass> -X GET 'https://<hostname1>:9201/_nodes/process?pretty' -k
I see only one node in the cluster:
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "new_cluster",
"nodes" : {
"Qlqcbgs_QmWXpglNVoOApQ" : {
"name" : "node1",
"transport_address" : "<IP_address>:9301",
"host" : "<hostname1>",
"ip" : "<IP_address>",
"version" : "6.6.0",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "<build_hash_number>",
"roles" : [
"master",
"data",
"ingest"
],
"attributes" : {
"ml.machine_memory" : "16653647872",
"xpack.installed" : "true",
"ml.max_open_jobs" : "20",
"ml.enabled" : "true"
},
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 11674,
"mlockall" : false
}
}
}
}
I get the same output for each node separately:
curl -u es_admin:<pass> -X GET 'https://<hostname2>:9201/_nodes/process?pretty' -k
curl -u es_admin:<pass> -X GET 'https://<hostname3>:9201/_nodes/process?pretty' -k
Under elasticsearch.template.yml I do see the other nodes. For example if I go to node1 I see the other two:
discovery.zen.ping.unicast.hosts:
- <hostname2>:9301
- <hostname3>:9301
here is elasticsearch.yml:
node.name: node1
network.host: <hostname>
http.port: 9201
transport.tcp.port: 9301
node.master: true
node.data: true
node.ingest: true
search.remote.connect: true
#################################### Paths ####################################
# Path to directory containing configuration (this file and logging.yml):
path.data: /var/lib/elasticsearch/node1
path.logs: /var/log/elasticsearch/node1
discovery.zen.ping.unicast.hosts:
- <hostname2>:9301
- <hostname3>:9301
xpack.license.self_generated.type: trial
node.ml: true
xpack.ml.enabled: true
xpack.security.audit.enabled: true
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true
xpack.ssl.keystore.path: **path**
xpack.ssl.keystore.password: *passwd*
xpack.ssl.truststore.path: **path**
xpack.ssl.truststore.password: *passwd*
What should be done in order to see all the nodes under the same cluster?

In 6.X you also need to set discovery.zen.minimum_master_nodes to say to your nodes what is the minimum number of master nodes required to form a cluster.
Since you didn't set it, each of your nodes think they are the master node and they won't join any cluster.
Set it to discovery.zen.minimum_master_nodes: 2 in each elasticsearch.yml file and restart your nodes.

I think all discovery.zen.ping.unicast.hosts must be the same in all node.
discovery.zen.ping.unicast.hosts:
- <hostname1>:9301
- <hostname2>:9301
- <hostname3>:9301
please try this or just:
discovery.zen.ping.unicast.hosts: ["hostname1:9301"]

Related

Failing to index csv file based data in opendistro elasticsearch

I am trying to index sample csv based data into opendistro elasticsearch but failing to create the index. Could you please let me what i am missing here.
csv file to index
[admin#fedser32 logstashoss-docker]$ cat /tmp/student.csv
"aaa","bbb",27,"Day Street"
"xxx","yyy",33,"Web Street"
"sss","mmm",29,"Adam Street"
logstash.conf
[admin#fedser32 logstashoss-docker]$ cat logstash.conf
input {
file {
path => "/tmp/student.csv"
start_position => "beginning"
}
}
filter {
csv {
columns => ["firstname", "lastname", "age", "address"]
}
}
output {
elasticsearch {
hosts => ["https://fedser32.stack.com:9200"]
index => "sampledata"
ssl => true
ssl_certificate_verification => false
user => "admin"
password => "admin#1234"
}
}
My Opendistro cluster is listening on 9200 as shown below.
[admin#fedser32 logstashoss-docker]$ curl -X GET -u admin:admin#1234 -k https://fedser32.stack.com:9200
{
"name" : "odfe-node1",
"cluster_name" : "odfe-cluster",
"cluster_uuid" : "5GOEtg12S6qM5eaBkmzUXg",
"version" : {
"number" : "7.10.0",
"build_flavor" : "oss",
"build_type" : "tar",
"build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
"build_date" : "2020-11-09T21:30:33.964949Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
As per the logs it does indicate it is able to find the csv file as shown below.
logstash_1 | [2022-03-03T12:11:44,716][INFO ][logstash.outputs.elasticsearch][main] Index Lifecycle Management is set to 'auto', but will be disabled - Index Lifecycle management is not installed on your Elasticsearch cluster
logstash_1 | [2022-03-03T12:11:44,716][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"#timestamp"=>{"type"=>"date"}, "#version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
logstash_1 | [2022-03-03T12:11:44,725][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/pipeline/logstash.conf"], :thread=>"#<Thread:0x5c537d14 run>"}
logstash_1 | [2022-03-03T12:11:45,439][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>0.71}
logstash_1 | [2022-03-03T12:11:45,676][INFO ][logstash.inputs.file ][main] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_20d37e3ca625c7debb90eb1c70f994d6", :path=>["/tmp/student.csv"]}
logstash_1 | [2022-03-03T12:11:45,697][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1 | [2022-03-03T12:11:45,738][INFO ][filewatch.observingtail ][main][2f140d63e9cab8ddc711daddee17a77865645a8de3d2be55737aa0da8790511c] START, creating Discoverer, Watch with file and sincedb collections
logstash_1 | [2022-03-03T12:11:45,761][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1 | [2022-03-03T12:11:45,921][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Could you check the access right for /tmp/student.csv file? it must be readable by user logstash.
check with this command:
#ls -l /tmp
Other way, if you have already indexed the file path, you have to clean up the sincedb
The thing that i was missing is i had to volume mount my CSV file into the logstash container as shown below after which i was able to index my csv data.
[admin#fedser opensearch-logstash-docker]$ cat docker-compose.yml
version: '2.1'
services:
logstash:
image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
ports:
- "5044:5044"
volumes:
- $PWD/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
- $PWD/student.csv:/tmp/student.csv

How to setup replica sets to docker containers using custom MongoDB Configuration file

I Need Your help,
I Created three docker MongoDB containers using a custom sample config file, then I need to implement replica sets to these containers, but I can't implement, and I can't access other containers IP and Port
db.yaml
storage:
dbPath: /data/db
journal:
enabled: true
replication:
replSetName: "my_replicaSet"
net:
bindIp: 127.0.0.1
port: 26017
db1.yaml
storage:
dbPath: /data/db
journal:
enabled: true
replication:
replSetName: "my_replicaSet"
net:
bindIp: 127.0.0.1
port: 28017
db2.yaml
storage:
dbPath: /data/db
journal:
enabled: true
replication:
replSetName: "my_replicaSet"
net:
bindIp: 127.0.0.1
port: 29017
First created three docker containers using below command
Container Name:DB
docker run --name DB -v /home/mahesh/Documents/Trishula/cortana/database:/etc/mongo --net my-mongo-cluster -d mongo --config /etc/mongo/db.yaml
Container Name:DB1
docker run --name DB -v /home/mahesh/Documents/Trishula/cortana/database:/etc/mongo --net my-mongo-cluster -d mongo --config /etc/mongo/db1.yaml
Container Name:DB2
docker run --name DB -v /home/mahesh/Documents/Trishula/cortana/database:/etc/mongo --net my-mongo-cluster -d mongo --config /etc/mongo/db2.yaml
then open a docker container DB shell with mongo --port 26017
initiated Replica sets with rs.initiate()
then add another docker container as a member to that shell by defining rs.add("DB1"), here DB1 is the name of another container, I got the error message like this
my_replicaSet:PRIMARY> rs.add("DB1")
{
"operationTime" : Timestamp(1597812494, 1),
"ok" : 0,
"errmsg" : "Either all host names in a replica set configuration must be localhost references, or none must be; found 1 out of 2",
"code" : 103,
"codeName" : "NewReplicaSetConfigurationIncompatible",
"$clusterTime" : {
"clusterTime" : Timestamp(1597812494, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
I also tried with given the container port and IP address which is defined in YAML file,
my_replicaSet:PRIMARY> rs.add("127.0.0.1:28017")
{
"operationTime" : Timestamp(1597812984, 1),
"ok" : 0,
"errmsg" : "Quorum check failed because not enough voting nodes responded; required 2 but only the following 1 voting nodes responded: 127.0.0.1:26017; the following nodes did not respond affirmatively: 127.0.0.1:28017 failed with Error connecting to 127.0.0.1:28017 :: caused by :: Connection refused",
"code" : 74,
"codeName" : "NodeNotFound",
"$clusterTime" : {
"clusterTime" : Timestamp(1597812984, 1),
"signature" : {
"hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
"keyId" : NumberLong(0)
}
}
}
I Had Implemented replica sets with command lines, but I can't implement replica sets with custom MongoDB configuration YAML file with docker MongoDB containers, Please help, I have been working on this for the past one week...
Note: I didn't use docker-compose YAML file...
If you are trying to setup mongodb replicaset locally using docker, refer https://medium.com/#simone.pezzano/quick-docker-and-mongodb-replica-set-on-your-computer-5c2470012a41

Logstash stopping {:plugin=>"LogStash::Inputs::Http"}

I'm trying to run Logstash in an EC2 Ubuntu instance,
but when I run:
logstash-5.2.0/bin/logstash -f logstash.conf --debug
I get:
Starting puma
Trying to start WebServer {:port=>9600}
start
Trying to start WebServer {:port=>9601}
[api-service] start
Successfully started Logstash API endpoint {:port=>9601}
PeriodicPoller: Stopping
stopping pipeline {:id=>"main"}
Closing inputs
stopping {:plugin=>"LogStash::Inputs::Http"}
Closed inputs
This is logstash.conf
input
{
http
{
host => "127.0.0.1"
port => 31311
}
}
output
{
elasticsearch
{
hosts => ["localhost:9200"]
}
stdout
{
codec => rubydebug
}
}
When I run
curl 'http://localhost:9200/?pretty'
I get:
{
"name" : "QrRfI_U",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "LLdvAaAsQSCULTfl_b4xIA",
"version" : {
"number" : "5.2.0",
"build_hash" : "24e05b9",
"build_date" : "2017-01-24T19:52:35.800Z",
"build_snapshot" : false,
"lucene_version" : "6.4.0"
},
"tagline" : "You Know, for Search"
}
So elasticsearch is running fine.
What if you have your hosts as:
hosts => "localhost"
And make sure that the http port which you've mentioned above is not bound to any other process.
If that's not the case just to make sure, run plugin list and check whether http-input plugin does exist.

ELK not passing metadata from filebeat into logstash

Installed an ELK server via: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7
It seems to work except for the filebeat connection; filebeat does not appear to be forwarding anything or at least I can't find anything in the logs to indicate anything is happening.
My filebeat configuration is as follows:
filebeat:
prospectors:
-
paths:
- /var/log/*.log
- /var/log/messages
- /var/log/secure
encoding: utf-8
input_type: log
timeout: 30s
idle_timeout: 30s
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["my_elk_fqdn:5044"]
bulk_max_size: 1024
compression_level: 3
worker: 1
tls:
certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
shipper:
logging:
to_files: true
files:
path: /var/log/filebeat
name: filebeat.log
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7
level: debug
The log file output I keep getting from filebeat is just not very helpful:
2016-07-14T17:32:21-04:00 DBG Start next scan
2016-07-14T17:32:31-04:00 DBG Start next scan
2016-07-14T17:32:41-04:00 DBG Start next scan
2016-07-14T17:32:46-04:00 DBG Flushing spooler because of timeout. Events flushed: 0
2016-07-14T17:32:51-04:00 DBG Start next scan
Is there anything wrong with my configuration file?
When I test on the ELK server to see if I am getting anything:
[root#my_elk_server ~]# curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}
}
Oh and my logstash configuration for filebeats:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
UPDATE: It is not filebeat. Somewhat relieved that messages are indeed being passed but still have an issue I can't track:
Discovered it wasn't filebeat that was causing the issue. It appears that the configuration file in logstash to send to elasticsearch is not properly labeling the index (and the type) to make it searchable as shown in the question. Instead of putting filebeat in the index name it gives a result like this:
"_index" : "%{[#metadata][beat]}-2016.07.14",
The elasticsearch output put in the file turned out to be incorrect in the
output {
elasticsearch {
hosts => "my_elk_fqdn:9200"
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
Apparently this #metadata is not being passed in correctly. Has anyone been able to get the _index and _type fields to populate correctly???
This might be a bug with filebeat??
https://github.com/logstash-plugins/logstash-input-beats/issues/6

Could not connect to mongod from mongo shell when auth enabled (on ubuntu)

System: ubuntu 14.04
mongodb 3.0.3 tar ball is downloaded from mongodb download center
connected to mongodb without auth, then from mongo shell, created a user for 'test' db. following is the command.
db.createUser({user: "user1",
pwd: "test123",
roles: [ { role: "readWrite", db: "test" }
]})
Verified that user details in admin db. Following is the command & result:
> db.system.users.findOne({user:'user1'})
{
"_id" : "testdb.user1",
"user" : "user1",
"db" : "testdb",
"credentials" : {
"SCRAM-SHA-1" : {
"iterationCount" : 10000,
"salt" : "kNfOd1vs+QT+ueH7SI6Vzw==",
"storedKey" : "JCesIKSW1pb74ddo2Y19rEO1GVY=",
"serverKey" : "d87Sb1htoD5K8zecAy73JPZyHdc="
}
},
"roles" : [
{
"role" : "readWrite",
"db" : "test"
}
]
}
Now exit from the mongo shell, killed the mongod.
Started the mongodb with auth, following is the command.
$ ./mongod --auth
Connected to mongo shell as usual, see the below:
$ ./mongo
MongoDB shell version: 3.0.3
connecting to: test
> show collections
2016-05-11T22:33:46.302+0530 E QUERY Error: listCollections failed: {
"ok" : 0,
"errmsg" : "not authorized on test to execute command { listCollections: 1.0 }",
"code" : 13
}
at Error (<anonymous>)
at DB._getCollectionInfosCommand (src/mongo/shell/db.js:646:15)
at DB.getCollectionInfos (src/mongo/shell/db.js:658:20)
at DB.getCollectionNames (src/mongo/shell/db.js:669:17)
at shellHelper.show (src/mongo/shell/utils.js:625:12)
at shellHelper (src/mongo/shell/utils.js:524:36)
at (shellhelp2):1:1 at src/mongo/shell/db.js:646
> db.auth({user:'user1', pwd:'test123'})
1
> use test
switched to db test
> db.collone.insert({name:'firstcollection'})
WriteResult({ "nInserted" : 1 })
> show collections
collone
system.indexes
> db.collone.find()
{ "_id" : ObjectId("5733669fb7d44cd444ebf028"), "name" : "firstcollection" }
> exit
bye
When i tried to do the authentication while starting the mongo shell, getting authentication failed error. See below:
$ ./mongo test -u 'user1' -p 'test123' --authenticationDatabase 'admin'
MongoDB shell version: 3.0.3
connecting to: test
2016-05-11T22:37:21.559+0530 E QUERY Error: 18 Authentication failed.
at DB._authOrThrow (src/mongo/shell/db.js:1266:32)
at (auth):6:8
at (auth):7:2 at src/mongo/shell/db.js:1266
exception: login failed
All this is just a POC that i'm trying to do.
Once it's success, my target is to connect from mongoose client(from Node.js app) to mongod.
The following command from a stackoverflow post can help me to set up connection from mongoose to mongod with auth.

Resources