Export ntopng log to logstash - logstash

Question

I know ntopng can put direct to elasticsearch but my boss want use logtash as layer to transfer log to elasticsearch.
I'm try many time but failed.
ntopng log like:
{"index": {"_type": "ntopng", "_index": "ntopng-2016.08.23"}}
{ "#timestamp": "2016-08-23T01:49:41.0Z", "type": "ntopng", "IN_SRC_MAC": "04:2A:E2:0D:62:FB", "OUT_DST_MAC": "00:16:3E:8D:B7:E4", "IPV4_SRC_ADDR": "14.152.84.14", "IPV4_DST_ADDR": "xxx.xxx.xxx", "L4_SRC_PORT": 34599, "L4_DST_PORT": 53, "PROTOCOL": 17, "L7_PROTO": 5, "L7_PROTO_NAME": "DNS", "IN_PKTS": 15, "IN_BYTES": 1185, "OUT_PKTS": 15, "OUT_BYTES": 22710, "FIRST_SWITCHED": 1471916981, "LAST_SWITCHED": 1471916981, "SRC_IP_COUNTRY": "CN", "SRC_IP_LOCATION": [ 113.250000, 23.116699 ], "DST_IP_COUNTRY": "VN", "DST_IP_LOCATION": [ 105.849998, 21.033300 ], "NTOPNG_INSTANCE_NAME": "ubuntu", "INTERFACE": "ens192", "DNS_QUERY": "cpsc.gov", "PASS_VERDICT": true }
Logstash config:
input {
tcp {
port => 5000
codec => json
}
}
filter{
json{
source => "message"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout{ codec => rubydebug }
}
Thanks

Answer 1

Since the ntopng logs are already in the bulk format expected by Elasticsearch you don't need to use the elasticsearch output but you can use the http output directly like this. No need to have Logstash parse JSON, simply forward the raw bulk commands to ES.
There's one catch, though: we need to add a newline character after the second line otherwise ES will reject the bulk call. We can achieve this with a mutate/update filter that adds a verbatim newline character after the message. Try it out, this will work.
input {
tcp {
port => 5000
codec => multiline {
pattern => "_index"
what => "next"
}
}
}
filter{
mutate {
update => {"message" => "%{message}
"}
}
}
output {
http {
http_method => "post"
url => "http://localhost:9200/_bulk"
format => "message"
message => "%{message}"
}
}