Logstash - parse array of JSON - logstash

Question

I'm trying to parse SendGrid Webhook events using Logstash. The issue is that Logstash's output is not an array of JSON, but only JSON. Square brackets are missing.
The reason I'm doing this is GeoIP location and UserAgent parsing for analytics.
I am posting to 127.0.0.1:3000, then I want to forward the output to 127.0.0.1:8080.
8080 is just a basic Express server which prints requests/responses and it sends final data to ElasticSearch.
This is the input:
[
{
"email": "email#domain.com",
"event": "click",
"ip": "8.8.8.8",
"sg_event_id": "WS1wXXhERnefBsqEt5FSFA",
"sg_message_id": "mk4Msf8nQvycsZIAHQPOrw.filter0321p1iad2-30191-5E686C57-5D.0",
"timestamp": 1596484698,
"url": "http://10.0.0.6/ServiceCenter/view",
"url_offset": { "index": 1, "type": "html" },
"useragent": "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko"
}
]
This is the output:
{
sg_event_id: 'WS1wXXhERnefBsqEt5FSFA',
event: 'click',
email: 'email#domain.com',
sg_message_id: 'mk4Msf8nQvycsZIAHQPOrw.filter0321p1iad2-30191-5E686C57-5D.0',
timestamp: 1596484698,
useragent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko',
ip: '8.8.8.8',
url: 'http://10.0.0.6/ServiceCenter/view',
url_offset: { index: 1, type: 'html' }
}
This is my config:
input {
http {
host => "127.0.0.1"
port => "8080"
}
}
filter {
mutate {
remove_field => [ "#version", "#timestamp", "headers", "host" ]
}
}
output {
http {
http_method => "post"
url => "http://127.0.0.1:3000"
}
stdout {
codec => rubydebug
}
}

Answer 1

So finally, I've found an workaround to do it using the json_encode filter.
Used ruby code to store keys and values to #DATA[oldJSON]
Used json_encode plugin on #DATA[oldJSON] and save the results to #DATA[newJSON]
Important: Set http output format => message and content_type => "application/json; charset=UTF-8". Default is text/plain and we don't want this.
Set message value to '[ %{[#DATA][newJSON]} ]'
Config:
input {
http {
host => "127.0.0.1"
port => "8080"
}
}
filter {
mutate {
remove_field => [
"#version",
"#timestamp",
"headers",
"host"
]
}
ruby {
code => '
event.to_hash.each { |k,v|
event.set("[#DATA][oldJSON][#{k}]", v)
}
'
}
json_encode {
source => "[#DATA][oldJSON]"
target => "[#DATA][newJSON]"
}
}
output {
http {
http_method => "post"
url => "http://127.0.0.1:3000"
format => message
content_type => "application/json; charset=UTF-8"
message => '[ %{[#DATA][newJSON]} ]'
}
}
Output:
[
{
ip: '8.8.8.8',
sg_message_id: 'mk4Msf8nQvycsZIAHQPOrw.filter0321p1iad2-30191-5E686C57-5D.0',
url_offset: { type: 'html', index: 1 },
sg_event_id: 'WS1wXXhERnefBsqEt5FSFA',
email: 'email#domain.com',
event: 'click',
url: 'http://10.0.0.6/ServiceCenter/view',
timestamp: 1596484698,
useragent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko'
}
]
Maybe somebody will find this useful.