This simple config file has an error. I've tried everything I can think of to fix it. I've simplified this example. Ultimately I'd like to use multiple input files and send them to different ports on the output.
input {
file {
path => "/home/user/log/*"
type => "test1"
start_position => "beginning"
}
}
output {
syslog {
host => "server.com"
if [type] == "test1" {
port => 5555
}
severity => "informational"
facility => "syslogd"
}
}
The error is:
Error: Expected one of #, => at line 14, column 8 (byte 168) after output {
syslog {
host => "server.com"
if
logstash is running on ubuntu:
apt --installed list|grep logstash
logstash/stable,now 1.4.3-1-7e387fb all [installed]
I found the problem. I needed to move "syslog" like this:
output {
if [type] == "test1" {
syslog {
host => "server.com"
port => 5555
severity => "informational"
facility => "syslogd"
}
}
}
Related
I have a simple setup for capturing logs though HTTP and TCP.
I've created 2 conf files at /etc/logstash/conf.d/ (see below) but logs sent though HTTP are also being passed through the TCP pipeline and vise versa. For example when I send a log through TCP it ends up both in http-logger-* index and in tcp-logger-*.. it makes no sense to me :(
http_logger.conf
input {
http {
port => 9884
}
}
filter {
grok {
match => ["[headers][request_path]", "\/(?<component>[\w-]*)(?:\/)?(?<env>[\w-]*)(?:\/)?"]
}
}
output {
amazon_es {
hosts => ['XXXXX']
region => 'us-west-2'
aws_access_key_id => 'XXXXX'
aws_secret_access_key => 'XXXXX'
index => 'http-logger-%{+YYYY.MM.dd}'
}
stdout { codec => rubydebug }
}
tcp_logger.conf
input {
tcp {
port => 9885
codec => json
}
}
filter {
}
output {
amazon_es {
hosts => ['XXXXX']
region => 'us-west-2'
aws_access_key_id => 'XXXXX'
aws_secret_access_key => 'XXXXX'
index => 'tcp-logger-%{+YYYY.MM.dd}'
}
stdout { codec => rubydebug }
}
Any ideas on what am I missing?
Thank you
The Input, filter and Output configuration even when split across a different file the logstash while processing it will process it as a single big configuration as if all the input, filter and output is specified in a single file.
So said that the event coming into logstash will pass through all the output and filter plugin configured, in your case, each event picked up by the TCP and HTTP input plugin will pass through filter plugin and output plugin configured in both http_logger.conf and tcp_logger.conf, that's the reason you are seeing events stashed in both http-logger-* and tcp-logger-* index
So in order to fix this, we can specify a unique type field for events picked by both tcp and http input plugins and then apply the filter and output plugin selectively using the type set in the input plugin as shown below
http_logger.conf
input {
http {
port => 9884
type => "http_log"
}
}
filter {
if [type] == "http_log"
{
grok {
match => ["[headers][request_path]", "\/(?<component>[\w-]*)(?:\/)?(?<env>[\w-]*)(?:\/)?"]
}
}
}
output {
if ([type] == "http_log")
{
amazon_es {
hosts => ['XXXXX']
region => 'us-west-2'
aws_access_key_id => 'XXXXX'
aws_secret_access_key => 'XXXXX'
index => 'http-logger-%{+YYYY.MM.dd}'
}
}
stdout { codec => rubydebug }
}
tcp_logger.conf
input {
tcp {
port => 9885
codec => json
type => "tcp_log"
}
}
output {
if ([type] == "tcp_log")
{
amazon_es {
hosts => ['XXXXX']
region => 'us-west-2'
aws_access_key_id => 'XXXXX'
aws_secret_access_key => 'XXXXX'
index => 'tcp-logger-%{+YYYY.MM.dd}'
}
}
stdout { codec => rubydebug }
}
The explanation provided by #Ram is spot on however there is a cleaner way of solving the issue: enter pipelines.yml.
By default it looks like this:
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
basically it loads and combines all *.conf files - in my case I had two.
To solve the issue just separate the pipelines like so:
- pipeline.id: httplogger
path.config: "/etc/logstash/conf.d/http_logger.conf"
- pipeline.id: tcplogger
path.config: "/etc/logstash/conf.d/tcp_logger.conf"
The pipelines are now running separately :)
P.S. Don't forget to reload logstash after any changes here
I have problem to send data from *.log file to logstash. This is filebeat configuration:
filebeat.prospectors:
- type: log
enabled: true
paths:
- /home/centos/logs/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
output.logstash:
hosts: "10.206.81.234:5044"
This is logstash configuration:
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d/*.conf
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.url: ["10.206.81.236:9200", "10.206.81.242:9200", "10.206.81.243:9200"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: logstash
queue.type: persisted
queue.checkpoint.writes: 10
And this is my pipeline in /etc/logstash/conf.d/test.conf
input {
beats {
port => "5044"
}
file{
path => "/home/centos/logs/mylogs.log"
tags => "mylog"
}
file{
path => "/home/centos/logs/syslog.log"
tags => "syslog"
}
}
filter {
}
output {
if [tag] == "mylog" {
elasticsearch {
hosts => [ "10.206.81.246:9200", "10.206.81.236:9200", "10.206.81.243:9200" ]
user => "Test"
password => "123456"
index => "mylog-%{+YYYY.MM.dd}"
}
}
if [tag] == "syslog" {
elasticsearch {
hosts => [ "10.206.81.246:9200", "10.206.81.236:9200", "10.206.81.243:9200" ]
user => "Test"
password => "123456"
index => "syslog-%{+YYYY.MM.dd}"
}
}
}
I tried to have two separate outputs for mylog and syslog. At first, it works like this: everything was passed to mylog-%{+YYYY.MM.dd} index even files from syslog. So I tried change second if statement to else if. It did not work so I changed it back. Now, my filebeat are not able to send data to logstash and I am receiving this errors:
2018/01/20 15:02:10.959887 async.go:235: ERR Failed to publish events caused by: EOF
2018/01/20 15:02:10.964361 async.go:235: ERR Failed to publish events caused by: client is not connected
2018/01/20 15:02:11.964028 output.go:92: ERR Failed to publish events: client is not connected
My second test was change my pipeline like this:
input {
beats {
port => "5044"
}
file{
path => "/home/centos/logs/mylogs.log"
}
}
filter {
grok{
match => { "message" => "%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }
}
}
output {
elasticsearch {
hosts => [ "10.206.81.246:9200", "10.206.81.236:9200", "10.206.81.243:9200" ]
user => "Test"
password => "123456"
index => "mylog-%{+YYYY.MM.dd}"
}
}
If I add some lines to mylog.log file, filebeat will print the same ERR files but it is passed to logstash and I can see it in Kibana. Could anybody explain me why does it not work? What does those errors means?
I am using filebeat an logstash version 6.1.
Sorry if I make any mistake in english.
In the output section, you are using "tag" (note: is in singular) which doesn't exists. But changing this to "tags" will not work either because the field tags is an array and you will be comparing it to a string, so you should get the first item instead of getting the whole array and then compare. Try this:
if [tags[0]] == "mylog" { ......
Data missed a lot in logstash version 5.0,
is it a serous bug ,when a config the config file so many times ,it useless,data lost happen again and agin, how to use logstash to collect log event property ?
any reply will thankness
Logstash is all about reading logs from specific location and based on you interested information you can create index in elastic search or other output also possible.
Example of logstash conf
input {
file {
# PLEASE SET APPROPRIATE PATH WHERE LOG FILE AVAILABLE
#type => "java"
type => "json-log"
path => "d:/vox/logs/logs/vox.json"
start_position => "beginning"
codec => json
}
}
filter {
if [type] == "json-log" {
grok {
match => { "message" => "UserName:%{JAVALOGMESSAGE:UserName} -DL_JobID:%{JAVALOGMESSAGE:DL_JobID} -DL_EntityID:%{JAVALOGMESSAGE:DL_EntityID} -BatchesPerJob:%{JAVALOGMESSAGE:BatchesPerJob} -RecordsInInputFile:%{JAVALOGMESSAGE:RecordsInInputFile} -TimeTakenToProcess:%{JAVALOGMESSAGE:TimeTakenToProcess} -DocsUpdatedInSOLR:%{JAVALOGMESSAGE:DocsUpdatedInSOLR} -Failed:%{JAVALOGMESSAGE:Failed} -RecordsSavedInDSE:%{JAVALOGMESSAGE:RecordsSavedInDSE} -FileLoadStartTime:%{JAVALOGMESSAGE:FileLoadStartTime} -FileLoadEndTime:%{JAVALOGMESSAGE:FileLoadEndTime}" }
add_field => ["STATS_TYPE", "FILE_LOADED"]
}
}
}
filter {
mutate {
# here converting data type
convert => { "FileLoadStartTime" => "integer" }
convert => { "RecordsInInputFile" => "integer" }
}
}
output {
elasticsearch {
# PLEASE CONFIGURE ES IP AND PORT WHERE LOG DOCs HAS TO PUSH
document_type => "json-log"
hosts => ["localhost:9200"]
# action => "index"
# host => "localhost"
index => "locallogstashdx_new"
# workers => 1
}
stdout { codec => rubydebug }
#stdout { debug => true }
}
To know more you can go throw many available websites like
https://www.elastic.co/guide/en/logstash/current/first-event.html
I am using logstash for the first time and trying to setup a simple pipeline for just printing the nginx logs. Below is my config file
input {
file {
path => "/var/log/nginx/*access*"
}
}
output {
stdout { codec => rubydebug }
}
I have saved the file as /opt/logstash/nginx_simple.conf
And trying to execute the following command
sudo /opt/logstash/bin/logstash -f /opt/logstash/nginx_simple.conf
However the only output I can see is:
Logstash startup completed
Logstash shutdown completed
The file is not empty for sure. As per my understanding I should be seeing the output on my console. What am I doing wrong ?
Make sure that the character encoding of your logfile is UTF-8. If it is not, try to change it and restart the Logstash.
Please try this code as your Logstash configuration, in order to setup a simple pipeline for just printing the nginx logs.
input {
file {
path => "/var/log/nginx/*.log"
type => "nginx"
start_position => "beginning"
sincedb_path=> "/dev/null"
}
}
filter {
if [type] == "nginx" {
grok {
patterns_dir => "/home/krishna/Downloads/logstash-2.1.0/pattern"
match => {
"message" => "%{NGINX_LOGPATTERN:data}"
}
}
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
}
stdout { codec => rubydebug }
}
I have just put up an ELK stack, but I am having trouble regarding the logstash configuration in /etc/logstash/conf.d I have two input sources being forwarded from one linux server, which has a logstash forwarder installed on it with the "files" looking like:
{
"paths": ["/var/log/syslog","/var/log/auth.log"],
"fields": { "type": "syslog" }
},
{
"paths": ["/var/log/osquery/osqueryd.results.log"],
"fields": { "type": "osquery_json" }
}
As you can see, one input is an osquery output (json formatted), and the other is syslog. My current config for logstash is osquery.conf:
input {
lumberjack {
port => 5003
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
codec => "json"
}
}
filter {
if [type] == "osquery_json" {
date {
match => [ "unixTime", "UNIX" ]
}
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
Which works fine for the one input source, but I do not know how to add my other syslog input source to the same config, as the "codec" field is in the input -- I can't change it to syslog...
I am also planning on adding another input source in a windows log format that is not being forwarded by a logstash forwarder. Is there anyway to structure this differently?
It's probably better to just remove the codec from your input if you are going to be handling different codecs on the same input:
input {
lumberjack {
port => 5003
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "osquery_json" {
json {
source => "field_name_the_json_encoded_data_is_stored_in"
}
date {
match => [ "unixTime", "UNIX" ]
}
}
if [type] == "syslog" {
}
}
output {
elasticsearch { host => localhost }
stdout { codec => rubydebug }
}
Then you just need to decide what you want to do with your syslog messages.
I would suggest also splitting your config into multiple files. I tend to to use 01-filename.conf - 10-filename.conf for inputs, 11-29 as filters and anything above that for outputs. These files will be loaded in to logstash in the order they are printed in an ls.