Json transformation in Logstash - logstash

Question

I want to transform a json document from one structure to another.
I've already tried to solve it with multiple workarounds but somehow I don't get my expected output.
I have a json log file as following:
{
"consumer": {
"created_at": 1566912618154,
"username": "dummyapp",
"id": "07912445-ca35-464b-8596-b2ace5e60481"
},
"service": {
"created_at": 1567173649,
"connect_timeout": 60000,
"protocol": "https",
"read_timeout": 60000,
"port": 9090,
"updated_at": 1567173649,
"retries": 1,
"write_timeout": 60000
},
"request": {
"querystring": {},
"size": "361",
"headers": {
"cache-control": "no-cache",
"content-length": "0",
"postman-token": "fb9e10e4-2f66-4126-beec-d5c1f7c06bf7",
"user-agent": "PostmanRuntime/7.15.0",
"accept": "/",
"accept-encoding": "gzip, deflate",
"connection": "keep-alive"
},
"method": "POST"
}
}
I want to transform it to something like this:
{
"Title":"Sample"
"consumer": {
"created_at": 1566912618154,
"username": "dummyapp"
},
"service": {
"created_at": 1567173649,
"connect_timeout": 60000
},
"request": {
"querystring": {},
"headers": {
"user-agent": "PostmanRuntime\/7.15.0"
}
"method": "POST"
}
}
This is my pipeline configuration:
input {
file {
path => "Path_to_Log\test.log"
start_position => "beginning"
type => "json"
codec => "json"
}
}
filter {
mutate {
add_field => {
"Title" => "Sample"
}
add_field => {
"consumer" => "%{[consumer]}"
}
...further changes
}
}
output {
file {
path => "Output_Path\Output.log"
}
}
This is the response I get:
{
"#timestamp":"2019-09-10T09:20:38.569Z",
"Title":"Sample",
"#version":"1",
"consumer":[
[
"created_at",
1566912618154
],
[
"username",
"dummyapp"
],
"{\"created_at\":1566912618154,\"username\":\"dummyappapp\"}"
],
"type":"json"
..some data
}
Why am I getting an output like above?
How can I tranform json documents to fulfill the requirements I described above?
How can I set a tag from the source to the target child element?

Answer 1

So why don't you just remove the fields that you are not interested in?
filter {
mutate {
add_field => {
"Title" => "Sample"
}
remove_field => [
"[consumer][id]",
"[service][protocol]",
# and so on for the service element
"[request][size]",
"[request][headers][cache-control]",
# and so on for the request.headers element
]
# THIS IS OBSOLETE NOW!
#add_field => {
# "consumer" => "%{[consumer]}"
#}
...further changes
}
}
Please refer to this guide for further information on how to access event data and fields (especially nested ones).