How to process http posted files in logstash - line by line? - logstash

Question

I successfully configured logstash to process csv files from the file system and put them into Elastic for further analysis.
But our ELK is heavily separated from the original source of the csv files, so I thought about sending the csv files via http to logstash instead of using a file system.
The issue is that if I use input "http" the whole file is taken and processed as one big bunch. The csv filter only recognized the first line. As mentioned, the same file works via "file" input.
logstash config is like this:
input {
# http {
# host => "localhost"
# port => 8080
# }
file {
path => "/media/sample_files/debit_201606.csv"
type => "items"
start_position => "beginning"
}
}
filter {
csv {
columns => ["Created", "Direction", "Member", "Point Value", "Type", "Sub Type"]
separator => " "
convert => { "Point Value" => "integer" }
}
date {
match => [ "Created", "YYYY-MM-dd HH:mm:ss" ]
timezone => "UTC"
}
}
output {
# elasticsearch {
# action => "index"
# hosts => ["localhost"]
# index => "logstash-%{+YYYY.MM.dd}"
# workers => 1
# }
stdout {
codec => rubydebug
}
}
My goal is to pass the csv via curl. So switching to the commented part of the input area above, and then use curl to pass the files:
curl http://localhost:8080/ -T /media/samples/debit_201606.csv
What do I need to do to achieve that logstash is processing the csv line by line?

Answer 1

I tried this and I think what you need to do is to split your input. Here's how you do that:
My configuration:
input {
http {
port => 8787
}
}
filter {
split {}
csv {}
}
output {
stdout { codec => rubydebug }
}
And for my test I created a csv file looking like this:
artur#pandaadb:~/tmp/logstash$ cat test.csv
a,b,c
d,e,f
g,h,i
And now for the test:
artur#pandaadb:~/dev/logstash/conf3$ curl localhost:8787 -T ~/tmp/logstash/test.csv
Outputs:
{
"message" => "a,b,c",
"#version" => "1",
"#timestamp" => "2016-08-01T15:27:17.477Z",
"host" => "127.0.0.1",
"headers" => {
"request_method" => "PUT",
"request_path" => "/test.csv",
"request_uri" => "/test.csv",
"http_version" => "HTTP/1.1",
"http_host" => "localhost:8787",
"http_user_agent" => "curl/7.47.0",
"http_accept" => "*/*",
"content_length" => "18",
"http_expect" => "100-continue"
},
"column1" => "a",
"column2" => "b",
"column3" => "c"
}
{
"message" => "d,e,f",
"#version" => "1",
"#timestamp" => "2016-08-01T15:27:17.477Z",
"host" => "127.0.0.1",
"headers" => {
"request_method" => "PUT",
"request_path" => "/test.csv",
"request_uri" => "/test.csv",
"http_version" => "HTTP/1.1",
"http_host" => "localhost:8787",
"http_user_agent" => "curl/7.47.0",
"http_accept" => "*/*",
"content_length" => "18",
"http_expect" => "100-continue"
},
"column1" => "d",
"column2" => "e",
"column3" => "f"
}
{
"message" => "g,h,i",
"#version" => "1",
"#timestamp" => "2016-08-01T15:27:17.477Z",
"host" => "127.0.0.1",
"headers" => {
"request_method" => "PUT",
"request_path" => "/test.csv",
"request_uri" => "/test.csv",
"http_version" => "HTTP/1.1",
"http_host" => "localhost:8787",
"http_user_agent" => "curl/7.47.0",
"http_accept" => "*/*",
"content_length" => "18",
"http_expect" => "100-continue"
},
"column1" => "g",
"column2" => "h",
"column3" => "i"
}
What the split filter does is:
It takes your input message (which is one String including the new-lines) and splits it by the configured value (which by default is a new-line). Then it cancels the original event and re-submits the split events to logstash. It is important that you execute the split before you execute the csv filter.
I hope that answers your question!
Artur