Our solution requires that some of the data created or modified in CouchDB is logged to Elastic Search using LogStash. So far we have been able to configure LogStash to connect to CouchDb via the changes feed however we have not been able to apply a CouchDB filter to the changes feed. How can we configure LogStash to specify and apply a CouchDB filter to the _changes feed and only log records that pass the filter based on a GET parameter?
LogStash Configuration:
input {
couchdb_changes {
db => "database-members"
host => "192.168.0.18"
sequence_path => "/root/.couchdb_seq_database-members"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
document_id => "%{[#metadata][_id]}"
index => "logstash-database-members-%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}
unfortunately the logstash plugin does not support filters yet.
if you know a bit of ruby its easy to add though and will roughly go here:
https://github.com/logstash-plugins/logstash-input-couchdb_changes/blob/master/lib/logstash/inputs/couchdb_changes.rb#L115
you would need to add the filter option to the logstash module and then make sure the request uri that the module creates is according to this doc
http://docs.couchdb.org/en/2.0.0/api/database/changes.html
Related
Does anyone have example fluentd & logstash config to make this scenario work?
My Java containers use Logback, with log output encoded into json by the logback LogstashEncoder.
Containers run in OpenShift which enforces use of "fluentd" for log forwarding.
We have fluentd pointing to a stand-alone ELK using its "fluentForward" capability.
On the logstash side we have nominated an input codec of fluent, and a json filter sourcing json from the message field.
We observe errors with text such as: "reason"=>"Can't get text on a START_OBJECT at 1:1712"
I will happily share config here if that is helpful, but seeing some known-to-be-working example would also help.
Kubernetes (OpenShift) ClusterLogForwarder
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
spec:
inputs:
- application:
namespaces:
- unix
name: my-app-logs
outputs:
- name: fluentd-server-insecure
type: fluentdForward
url: 'tcp://logstash.domain.com:24224'
pipelines:
- inputRefs:
- application
- audit
name: forward-to-fluentd-insecure
outputRefs:
- fluentd-server-insecure
Logstash Config
input {
tcp {
host => "10.110.250.127"
port => 24224
codec => fluent
type => "rsyslog"
}
}
filter {
json {
source => "message"
skip_on_invalid_json => true
}
}
output {
if [type] == "rsyslog" {
elasticsearch {
hosts => [ "elasticsearch.domain.com:9200" ]
}
}
}
Error
Can't get text on a START_OBJECT at 1:1712
First things first: you don't have to use the built-in logging functionality with OpenShift, but could instead deploy the Elastic Cloud Operator and then use a beat if that suites your needs a little better.
However, if you want to stick to cluster-logging-operator, stick close to the documentation:
input {
tcp {
codec => fluent
port => 4000
}
}
You are not receiving rsyslog type logs in this scenario, so don't specify that.
Additionally, if you're not doing any bigger processing in your logstash pipeline, which is what it looks like, I'd recommend for you to forward directly to the Elasticsearch instance:
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
name: instance
namespace: openshift-logging
spec:
outputs:
- name: elasticsearch-insecure
type: "elasticsearch"
url: http://elasticsearch.insecure.com:9200
One last remark: make sure to always check if the versions you use are compatible. There is a lot of change ongoing with OpenShift clusterlogging and the forwarder so things might change. Red Hat provides a table where you can look up the compatiblity
I want to log error from my node.js server to another server. I'm thinking of using elasticsearch, logstash and kibana. I want to know how to setup ELK with my node server.
I had exactly this use case in my older organization. A basic tutorial to startup with Beats + ELK - https://www.elastic.co/guide/en/beats/libbeat/current/getting-started.html
So basically this is how it works - Your nodejs app will log in the files (you can use bunyan for this) in different formats like error/warning/info etc. Filebeat will tail these log files and send messages to logstash server. Logstash input.conf will have some input filters (in your case it will be error filters). When any log message passes these filters then logstash will forward it to some endpoint as decided in output.conf file.
Here is what we did -
Initial architecture - Install filebeat (earlier we used logstash forwarder) client to tail the logs on nodejs server and forward it to logstash machine. Logstash will do some processing on input logs and send them to ES cluster (can be on same machine as Logstash). Kibana is just a visualization on this ES.
Final Architecture - Initial setup was cool for small traffic but we realized that logstash can be single point of failure and may result in log loss when traffic increased. So we integrated Kafka along with Logstash so that system scales smoothly. Here is an article - https://www.elastic.co/blog/logstash-kafka-intro
Hope this helps!
It is possible to use logstash without agents running to collect logs from the application.
Logstash has input plugins (https://www.elastic.co/guide/en/logstash/current/input-plugins.html). This can be configured in the pipeline. One basic setup is to configure the TCP (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-tcp.html) or UDP (https://www.elastic.co/guide/en/logstash/current/plugins-inputs-udp.html)input plugin. Logstash can listen on the port configured in the plugin.
Then the application can send the log directly to the logstash server. The pipeline can then transform and forward to ES.
By configuring Logstash pipeline to be durable, data loss can be avoided. This approach is better when the application servers are ephemeral ( as in containers).
For nodejs, https://www.npmjs.com/package/winston-logstash is a package which is quite active. This gist https://gist.github.com/jgoodall/6323951 provides a good example for the overall approach in other languages.
This is the sample (minimal) TCP input plugin configuration
input {
tcp {
'port' => '9563'
}
}
You can install Logstash in the NodeJS Server, and then create a configuration file that accepts input (location of the log file(s)) and output to your Elastic Server host.
Below is the sample configuration file (custom.conf) which has to created in your logstash directory.
input {
file {
path => "/path to log"
start_position => beginning
}
}
output {
stdout { }
elasticsearch{
type => "stdin-type"
embedded => false
host => "192.168.0.23"
port => "9300"
cluster => "logstash-cluster"
node_name => "logstash"
}
}
Execute the logstash
logstash -f custom.conf
Reference: https://www.elastic.co/guide/en/logstash/current/config-examples.html
If you are planning to customize a NodeJS application for sending error logs then you can install some ELKStack Nodjs wrapper and post error log within your application. ELKStack Nodjs wrapper - https://www.npmjs.com/package/elksdk
I want to control flow of logs from logstash forwarder client, as for now it reads the entire log file from beginning, which is not required in the project.
I want that the logs before Nov 10, 2015 should not be forwarded to the logstash Server. Is there a way we can do it.
you could simply drop the events that are too old in your logstash indexer config by using the drop filter:
if [somevalue] < X {
drop { }
}
Check the docs at: https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html
Presently I have my logs and logstash running on the same machine, so I read my logs placed on my local machine with this config(using pull model)
input {
file {
path => "/home/Desktop/Logstash-Input/**/*_log"
start_position => "beginning"
}
}
Now, we have logstash running on a different machine and want to read the logs remote mechine.
Is there a way to set the ip in file input of config file?
EDIT:
I manage to do this with logstash-forwarder which is a push model(log shipper/logstash-forwarder will ship log to logstash index server) but still i am looking for a pull model without shipper, where logstash index server will go and contact directly to remote host.
Take a look to FileBeat: https://www.elastic.co/products/beats/filebeat
It´s not a pull model but it seems a better choice than logstash-forwarder.
It monitors log files and forwards them to Logstash or Elasticsearh. It keeps also the state of log files and guarantees that events will be delivered at least one time (depends on log rotation speed). It's really easy to configure:
Input configuration:
input_type: log
paths:
- /opt/app/logs
Output configuration
output.logstash:
hosts: ["remote_host:5044"]
index: filebeat_logs
In the logstash side you must install and configure the Beats input plugin:
input {
beats {
port => 5044
}
}
Logstash doesn't contain any magic to read files from other computer's file systems (and that's probably a good thing). You'll either have to mount the remote file system that contains the logs you're interested in or you have to install a log shipper (like e.g. Logstash) on the remote machine and configure it to send the data to your current Logstash instance (or an intermediate broker like Redis, RabbitMQ, or Kafka).
You could also use the syslog daemon (that's probably already installed on the machine) to ship logs via the syslog protocol, but keep in mind that there's no guarantee of the maximum allowed length of each message.
You can add the remote system IP in the path and access the logs from Remote machine.
input {
file {
path => "\\IP address/home/Desktop/Logstash-Input/**/*_log"
start_position => "beginning"
}}
We have the following setup.
1 central logstash server (behind that we have an elasticsearch
cluster based on two nodes)
1 central zabbix server
10 Servers with logstash-forwarder
On our logstash server we are getting syslogs apache/nginx access and error logs from 10 mentioned servers trough logstash-forwarder.
Since we want to see the amount of error logs per server per minute in a nice graph in our zabbix system we are using the metrics plugin (http://logstash.net/docs/1.4.2/contrib-plugins)
Here is the PROBLEM:
we are currently not able the get the logs with the correct tags from the plugin to send them to zabbix.
logstash-forwarder confing and logstash server conf see link
https://db.tt/4cn8DWi2
if anyone has an idea, how we can get rid of this problem, we would be very thankful.
It looks like you are messing up top level sections in your config files, check the logstash config language.
Each file should be something like this;
# section input
input {
# multiple plugin defintions *within* the input section
file {}
file {}
}
# section filter
filter {
grok{}
grok{}
grok{}
}
# section output
output {
elasticsearch{}
stoud{}
}
Your config looks like this:
input {
file {}
}
input {
file {}
}
output {
elasticsearch{}
}
output {
stoud{}
}