Customize log output in Kibana - logstash

Question

Finally, I got working ELK stack to get some logs from a remote server. However, I would like to customize the output of the logs. Is there a way to remove some fields which I am highlighting in yellow:
I tried to remove them from _source including remove_field in the logstash.conf:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/..."
ssl_key => "/..logstash.key"
}
}
filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
}
remove_field => [ "tags", "prospector.type", "host.architecture", "host.containerized", "host.id", "host.os.platform", "host.os.family" ]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
}
}
Do you know how can I get rid of the yellow fields in _source for the logs coming from filebeat?
Update of logstash.conf based on Leandro comments:
input {
beats {
port => 5044
ssl => true
ssl_certificate => ".../logstash.crt"
ssl_key => ".../logstash.key"
}
}
filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
}
remove_field => [ "tags","[prospector][type]","[host][architecture]", "[host][containerized]", "[host][id]", "[host][os][platform]", "[host][os][family]", "[beat][hostname]", "[beat][name]", "[beat][version], "[offset]", "[input][type]", "[meta][cloud][provider]", "[meta][cloud][machine_type]", "[meta][cloud][instance_id]"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
}
}
In logs:
019-02-27T17:03:41.637-0800 DEBUG [input] file/states.go:68 New state added for /logs/api.log
2019-02-27T17:03:41.637-0800 DEBUG [registrar] registrar/registrar.go:315 Registrar state updates processed. Count: 1
2019-02-27T17:03:41.637-0800 DEBUG [registrar] registrar/registrar.go:400 Write registry file: /filebeat/registry
2019-02-27T17:03:41.637-0800 INFO log/harvester.go:255 Harvester started for file: /logs/api.log
2019-02-27T17:03:41.647-0800 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"#timestamp": "2019-02-28T01:03:41.647Z",
"#metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.6.0"
},
"log": {
"file": {
"path": "/logs/api.log"
}
},
"input": {
"type": "log"
},
"host": {
"name": "tomcat",
"os": {
"family": "redhat",
"name": "CentOS Linux",
"codename": "Core",
"platform": "centos",
"version": "7 (Core)"
},
"id": "6aaed308aa5a419f880c5e45eea65414",
"containerized": true,
"architecture": "x86_64"
},
"meta": {
"cloud": {
"region": "CanadaCentral",
"provider": "az",
"instance_id": "6452bcf4-7f5d-4fc3-9f8e-5ea57f00724b",
"instance_name": "tomcat",
"machine_type": "Standard_D8s_v3"
}
},
"message": "2018-09-14 20:23:37 INFO ContextLoader:272 - Root WebApplicationContext: initialization started",
"source": "/logs/api.log",
"offset": 0,
"prospector": {
"type": "log"
},
"beat": {
"hostname": "tomcat",
"version": "6.6.0",
"name": "tomcat"
}
}
Thanks

Answer 1

Some of those fields are nested fields, the way to access them in a Logstash filter is using the [field][subfield] notation.
Your remove_field shoud be something like this:
remove_field => ["tags","[host][architecture]","[meta][cloud][provider]"]
But I don't think you can remove the #version field.
UPDATE:
Using the event example from your Filebeat log I simulated a pipeline and got a _grokparsefailure, to remove the fields even when the grok fails you need to use the remove_field inside a mutate filter:
filter {
grok {
your grok
}
mutate {
remove_field => ["[prospector]","[host][architecture]", "[host][containerized]", "[host][id]", "[host][os][platform]", "[host][os][family]", "[beat]", "[offset]", "[input]", "[meta]"]
}
}
Don't remove the tags field until you have fixed your groks.
The logstash output on that example is:
{
"source": "/logs/api.log",
"tags": [
"_grokparsefailure"
],
"#timestamp": "2019-02-28T01:03:41.647Z",
"message": "2018-09-14 20:23:37 INFO ContextLoader:272 - Root WebApplicationContext: initialization started",
"log": {
"file": {
"path": "/logs/api.log"
}
},
"#version": "1",
"host": {
"os": {
"codename": "Core",
"version": "7 (Core)",
"name": "CentOS Linux"
},
"name": "tomcat"
}
}