filebeat send messages to certain index - linux

Question

I have an installed pair elasticsearch - logstash - kibana, 2 clients: ELKclient1 and ELKclient2. Filebeat is installed on clients. I need that both clients write logs in separate index, ELKclient1 in index test-%{+YYYY.MM.dd, ELKclient2 in index test2-%{+YYYY.MM.dd (sending nginx access logs). For some reason logs from clients are written in both indexes, eg, from client ELKclient2 logs are written in both indexes test-%{+YYYY.MM.dd and test2-%{+YYYY.MM.dd (attachment 1 and attachement 2). Do you have any clue why its happening?
#config filebeat on client2
filebeat.inputs:
- type: log
enabled: true
paths:
- /var/log/nginx/access.log
fields:
type: nginx_access
fields_under_root: true
scan_frequency: 5s
registry_file: /var/lib/filebeat/registry
output:
logstash:
hosts: ["ip-address_logstash:5044"]
index: "test2-%{+YYYY.MM.dd}"
bulk_max_size: 1024
shipper:
logging:
to_syslog: false
to_files: true
level: info
files:
path: /var/log/filebeat
name: filebeat.log
#config logstash output
output {
elasticsearch {
hosts => "localhost:9200"
index => "test-%{+YYYY.MM.dd}"
}
#stdout { codec => rubydebug }
elasticsearch {
hosts => "localhost:9200"
index => "test2-%{+YYYY.MM.dd}"
}
#stdout { codec => rubydebug }
}

Answer 1

In order to make both clients write logs in a separate index, Take the workflow idea in the below picture, You need to add a tag to differentiate the logs coming from different servers.
Considering your requirement in your question one of the ways is to put the following code in the output section of your logstash config file.
output {
if [beat][hostname] == "ELKclient1"
elasticsearch {
hosts => "localhost:9200"
index => "test-%{+YYYY.MM.dd}"
}
else if [beat][hostname] == "ELKclient2"
elasticsearch {
hosts => "localhost:9200"
index => "test2-%{+YYYY.MM.dd}"
}
else
stdout {
codec => rubydebug
}
}