How to parse stream of logs aggregated from multiple files with logstash? - gitlab

Question

I have logs from GitLab installed on Kubernetes. Amongst other pods, there is Sidekiq which has a very peculiar structure of logs - it gathers multiple files that all then go into the stdout (see example at the end or official documentation). I want to gather all these logs by Filebeat, send them to Logstash and process them in a sane way (parse JSONs, get important data from line logs, etc. Also, I would like to add info about the original file) and send the output to elasticsearch.
However, I am struggling with how to do that - as a newbie regarding Logstash I am not sure how it works under the hood - and so far, I was able to come up only with grok that matches line with the file name.
From one perspective it should be relatively easy - I just need to use some sort of a state to mark which file is being processed in the log stream but in the first place I am not sure if Filebeat somehow passes information about the stream to Logstash (important to distinguish from which pod logs came) and secondly whether Logstash allows this state-based processing of log stream.
Is it possible to parse these logs and add the original filename as a field this state-based way? Could you possibly point me in the right direction?
filter {
grok {
match => {"message" => "\*\*\* %{PATH:file} \*\*\*"}
}
if [file] == "/var/log/gitlab/production_json.log" {
json {
match => { ... }
}
}
else if [file] == "/var/log/gitlab/application_json.log" {
grok {
match => { ... }
}
}
}
Please notice that even for each file, there might be multiple types of logs (/var/log/gitlab/sidekiq_exporter.log)
*** /var/log/gitlab/application.log ***
2020-11-18T10:08:28.568Z: Cannot obtain an exclusive lease for Namespace::AggregationSchedule. There must be another instance already in execution.
*** /var/log/gitlab/application_json.log ***
{"severity":"ERROR","time":"2020-11-18T10:08:28.568Z","correlation_id":"BsVuSTdkM45","message":"Cannot obtain an exclusive lease for Namespace::AggregationSchedule. There must be another instance already in execution."}
*** /var/log/gitlab/sidekiq_exporter.log ***
[2020-11-18T10:08:32.076+0000] 10.103.149.75 - - [18/Nov/2020:10:08:32 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:08:42.076+0000] 10.103.149.75 - - [18/Nov/2020:10:08:42 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:08:43.771+0000] 10.103.149.75 - - [18/Nov/2020:10:08:43 UTC] "GET /liveness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:08:52.076+0000] 10.103.149.75 - - [18/Nov/2020:10:08:52 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:09:02.076+0000] 10.103.149.75 - - [18/Nov/2020:10:09:02 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:09:12.076+0000] 10.103.149.75 - - [18/Nov/2020:10:09:12 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:09:22.076+0000] 10.103.149.75 - - [18/Nov/2020:10:09:22 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:09:32.076+0000] 10.103.149.75 - - [18/Nov/2020:10:09:32 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:09:42.076+0000] 10.103.149.75 - - [18/Nov/2020:10:09:42 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:09:43.771+0000] 10.103.149.75 - - [18/Nov/2020:10:09:43 UTC] "GET /liveness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:09:52.076+0000] 10.103.149.75 - - [18/Nov/2020:10:09:52 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:10:02.076+0000] 10.103.149.75 - - [18/Nov/2020:10:10:02 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:10:12.076+0000] 10.103.149.75 - - [18/Nov/2020:10:10:12 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
2020-11-18T10:10:15.783Z 10 TID-oslmgxbxm PagesDomainSslRenewalCronWorker JID-e4891c8d6d57d73f401da697 INFO: start
2020-11-18T10:10:15.807Z 10 TID-oslmgxbxm PagesDomainSslRenewalCronWorker JID-e4891c8d6d57d73f401da697 INFO: done: 0.024 sec
[2020-11-18T10:10:22.076+0000] 10.103.149.75 - - [18/Nov/2020:10:10:22 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:10:32.076+0000] 10.103.149.75 - - [18/Nov/2020:10:10:32 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:10:42.076+0000] 10.103.149.75 - - [18/Nov/2020:10:10:42 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:10:43.771+0000] 10.103.149.75 - - [18/Nov/2020:10:10:43 UTC] "GET /liveness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
*** /var/log/gitlab/application_json.log ***
{"severity":"ERROR","time":"2020-11-18T10:49:11.565Z","correlation_id":"H9wDObekY74","message":"Cannot obtain an exclusive lease for Ci::PipelineProcessing::AtomicProcessingService. There must be another instance already in execution."}
*** /var/log/gitlab/application.log ***
2020-11-18T10:49:11.564Z: Cannot obtain an exclusive lease for Ci::PipelineProcessing::AtomicProcessingService. There must be another instance already in execution.
2020-11-18T10:49:11.828Z 10 TID-gn2cjsz0a ProjectServiceWorker JID-ccb9b5b0f74ced684e15af75 INFO: done: 0.275 sec
2020-11-18T10:49:11.835Z 10 TID-gn2dwudy2 Namespaces::ScheduleAggregationWorker JID-7db9fe9200701bbc7dc7360c INFO: start
2020-11-18T10:49:11.844Z 10 TID-gn2dwudy2 Namespaces::ScheduleAggregationWorker JID-7db9fe9200701bbc7dc7360c INFO: done: 0.009 sec
2020-11-18T10:49:11.888Z 10 TID-oslmgxbxm ArchiveTraceWorker JID-999cc768143b644d051cfe82 INFO: done: 0.21 sec
*** /var/log/gitlab/sidekiq_exporter.log ***
[2020-11-18T10:49:12.076+0000] 10.103.149.75 - - [18/Nov/2020:10:49:12 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:49:22.076+0000] 10.103.149.75 - - [18/Nov/2020:10:49:22 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:49:32.076+0000] 10.103.149.75 - - [18/Nov/2020:10:49:32 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
[2020-11-18T10:49:42.076+0000] 10.103.149.75 - - [18/Nov/2020:10:49:42 UTC] "GET /readiness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"
2020-11-18T10:49:43.216Z 10 TID-gn2cjsz0a Namespaces::RootStatisticsWorker JID-c277b38f3daa09648934d99f INFO: start
2020-11-18T10:49:43.243Z 10 TID-gn2cjsz0a Namespaces::RootStatisticsWorker JID-c277b38f3daa09648934d99f INFO: done: 0.027 sec
[2020-11-18T10:49:43.771+0000] 10.103.149.75 - - [18/Nov/2020:10:49:43 UTC] "GET /liveness HTTP/1.1" 200 15 "-" "kube-probe/1.17+"

Answer 1

You can give all the logs path in filebeat.yml for filebeat to read the logs and send it to logstash.
Example filebeat.yml for gitlab:
###################### Filebeat Configuration Example #########################
#=========================== Filebeat inputs =============================
filebeat.inputs:
-
paths:
- /var/log/gitlab/gitlab-rails/application_json.log
fields:
- type: gitlab-application-json
fields_under_root: true
encoding: utf-8
-
paths:
- /var/log/gitlab/sidekiq_exporter.log
fields:
- type: gitlab-sidekiq-exporter
fields_under_root: true
encoding: utf-8
-
paths:
- /var/log/gitlab/gitlab-rails/api_json.log
fields:
- type: gitlab-api-json
fields_under_root: true
encoding: utf-8
-
paths:
- /var/log/gitlab/gitlab-rails/application.log
fields:
- type: gitlab-application
fields_under_root: true
encoding: utf-8
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["10.127.55.155:5066"]
#================================ Processors =====================================
# Configure processors to enhance or manipulate events generated by the beat.
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
Now, in logstash, you can create different grok pattern to filter these logs.
Here is a sample logstash.yml,
input {
beats {
port => "5066"
}
}
filter {
if [type] == "gitlab-sidekiq-exporter" {
grok {
match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[(?<timestamp>%{MONTHDAY}/%{MONTH}/%{YEAR}\:%{TIME}) %{TZ:timezone}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}" }
overwrite => [ "message" ]
}
}
filter {
mutate {
remove_tag => [
"_grokparsefailure"
]
}
}
output {
#filtered logs are getting indexed in elasticsearch
elasticsearch {
hosts => ["10.127.55.155:9200"]
user => elastic
password => elastic
action => "index"
index => "gitlab"
}
stdout { codec => rubydebug } #filtered logs can be seen as console output as well, you can comment this out as well, this is for debugging purpose only
}
Note: The beat input port in logstash.yml should be same, as given in output.logstash in filebeat.yml
You can append the logstash.yml for filtering out application_json.log and application.log similar to that of sidekiq_exporter.log
For creating and validating grok pattern to filter the logs, you can use online Grok Debugger.
Here, I have used the Grok Debugger to create a pattern for filtering sidekiq_exporter.log
Pattern: %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}