grok parser failure - for django logs - logstash

Question

This is 1 of my log entries,
INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E
This is my pattern ,
DJANGOTIMESTAMP %{YEAR}-%{MONTHNUM}-%{MONTHDAY}%{SPACE}%{HOUR}:%{MINUTE}:%{SECOND}
This is my logstash conf file,
input {
beats {
port => "5043"
}
}
filter {
if [type] in ["django"] {
grok {
patterns_dir => ["/opt/logstash/patterns"]
match => [ "message" , "%{LOGLEVEL:level}%{SPACE}%{DJANGOTIMESTAMP:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:uid}%{SPACE}%{INT:django-id}%{SPACE}%{GREEDYDATA:action}" ]
}
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "%{type}_indexer"
}
}
IN elasticsearch output, the fields are not made,
luvpreet#DHARI-Inspiron-3542:/usr/bin$ curl -XGET 'localhost:9200/django_indexer/_search?pretty=true&q=*:*'
{
"_index" : "django_indexer",
"_type" : "django",
"_id" : "AVwu8tE7j-Kh6vl1kUdf",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2017-05-22T06:55:52.819Z",
"offset" : 144,
"#version" : "1",
"beat" : {
"hostname" : "DHARI-Inspiron-3542",
"name" : "DHARI-Inspiron-3542",
"version" : "5.4.0"
},
"input_type" : "log",
"host" : "DHARI-Inspiron-3542",
"source" : "/var/log/django/a.log",
"message" : "INFO 2017-05-16 06:33:08,673 views 40152 139731056719616 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E",
"type" : "django",
"tags" : [
"beats_input_codec_plain_applied"
]
}
It is not saying that parser has failed, but why are the fields not being made ?
What am I lacking ?

Answer 1

Try with this grok pattern:
%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:id}%{SPACE}%{INT:number}%{SPACE}%{URI:action}
Input
INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E
Output
number 139643033982720
timestamp 2017-05-16·17:24:11
id 14463
port
pid 690
origin views
action https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E
loglevel INFO
You can then remove the field port with a mutate in you filter plugin
mutate {
remove_field => ["port"]
}
UPDATE
Ok, I tried your configuration on with my logstash.
This is what I did:
1- Configure filebeat:
filebeat.prospectors:
- paths:
- /etc/filebeat/FilebeatInputTest.txt
document_type: django
output.logstash:
hosts: ["127.0.0.1:5044"]
2- Configure logstash
input {
beats {
port => "5044"
}
}
filter {
if [type] == "django" {
grok {
match => [ "message" , "%{LOGLEVEL:loglevel}%{SPACE}%{TIMESTAMP_ISO8601:timestamp},%{INT:pid}%{SPACE}%{WORD:origin}%{SPACE}%{INT:id}%{SPACE}%{INT:number}%{SPACE}%{GREEDYDATA:action}" ]
}
mutate {
remove_field => ["#timestamp", "beat","input_type","offset","source","#version","host","tags","message"]
}
}
}
output {
elasticsearch {
hosts => [ "xx.xx.xx.xx:9200" ]
index => "%{type}_indexer"
user => "xxxx"
password => "xxxx"
}
}
You can remove user and password if your elasticsearch is not secured.
Input (content of /etc/filebeat/FilebeatInputTest.txt)
INFO 2017-05-16 17:24:11,690 views 14463 139643033982720 https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E
Output (In elasticsearch)
{
"_index" : "django_indexer",
"_type" : "django",
"_id" : "AVwhFe30JYGYNG_7C7YI",
"_score" : 1.0,
"_source" : {
"origin" : "views",
"pid" : "690",
"type" : "django",
"number" : "139643033982720",
"loglevel" : "INFO",
"action" : "https://play.google.com/store/apps/details?id=com.VoDrive&referrer=referral_code%3DP5E",
"id" : "14463",
"timestamp" : "2017-05-16 17:24:11"
}
}
Hope this helps.