fluentd -> logstash (net.logstash.logback.encoder.LogstashEncoder) - logstash

Question

Does anyone have example fluentd & logstash config to make this scenario work?
My Java containers use Logback, with log output encoded into json by the logback LogstashEncoder.
Containers run in OpenShift which enforces use of "fluentd" for log forwarding.
We have fluentd pointing to a stand-alone ELK using its "fluentForward" capability.
On the logstash side we have nominated an input codec of fluent, and a json filter sourcing json from the message field.
We observe errors with text such as: "reason"=>"Can't get text on a START_OBJECT at 1:1712"
I will happily share config here if that is helpful, but seeing some known-to-be-working example would also help.
Kubernetes (OpenShift) ClusterLogForwarder
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
spec:
inputs:
- application:
namespaces:
- unix
name: my-app-logs
outputs:
- name: fluentd-server-insecure
type: fluentdForward
url: 'tcp://logstash.domain.com:24224'
pipelines:
- inputRefs:
- application
- audit
name: forward-to-fluentd-insecure
outputRefs:
- fluentd-server-insecure
Logstash Config
input {
tcp {
host => "10.110.250.127"
port => 24224
codec => fluent
type => "rsyslog"
}
}
filter {
json {
source => "message"
skip_on_invalid_json => true
}
}
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => [ "elasticsearch.domain.com:9200" ]
}
}
}
Error
Can't get text on a START_OBJECT at 1:1712

Answer 1

First things first: you don't have to use the built-in logging functionality with OpenShift, but could instead deploy the Elastic Cloud Operator and then use a beat if that suites your needs a little better.
However, if you want to stick to cluster-logging-operator, stick close to the documentation:
input {
tcp {
codec => fluent
port => 4000
}
}
You are not receiving rsyslog type logs in this scenario, so don't specify that.
Additionally, if you're not doing any bigger processing in your logstash pipeline, which is what it looks like, I'd recommend for you to forward directly to the Elasticsearch instance:
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
- name: elasticsearch-insecure
type: "elasticsearch"
url: http://elasticsearch.insecure.com:9200
One last remark: make sure to always check if the versions you use are compatible. There is a lot of change ongoing with OpenShift clusterlogging and the forwarder so things might change. Red Hat provides a table where you can look up the compatiblity