I'm new in this ELK stuff. I've been trying to create visualizations using this stack, but I'm not able to use fields such as verb, response, request, etc, I'm only able to select a few available fields:
However, in the Discover section I'm perfectly able to work with those fields. Here is a sample of one of my query results:
(I'm using Kibana 4.4.2, filebeat forwarding to logstash 2.2.3)
{
"_index": "filebeat-2016.04.12",
"_type": "apache_log",
"_id": "AVQMoRFwO5HM5nz1lmXf",
"_score": null,
"_source": {
"message": "187.142.15.173 - - [12/Apr/2016:16:39:23 -0600] \"GET /v1.0/person/297312123/client/1132347/profile HTTP/1.1\" 200 2051 \"-\" \"Android CEX 2.2.0\"",
"#version": "1",
"#timestamp": "2016-04-12T22:39:27.064Z",
"beat": {
"hostname": "myhost",
"name": "myhost"
},
"count": 1,
"fields": null,
"input_type": "log",
"offset": 30034512,
"source": "/var/log/httpd/access_log",
"type": "apache_log",
"host": "myhost",
"tags": [
"beats_input_codec_plain_applied"
],
"clientip": "187.142.15.173",
"ident": "-",
"auth": "-",
"timestamp": "12/Apr/2016:16:39:23 -0600",
"verb": "GET",
"request": "/v1.0/person/297312123/client/1132347/profile",
"httpversion": "1.1",
"response": "200",
"bytes": "2051",
"referrer": "\"-\"",
"agent": "\"Android CEX 2.2.0\"",
},
"fields": {
"#timestamp": [
1460500767064
]
},
"sort": [
1460500767064
]
}
What could posibly be wrong with this?
Here is my config file:
filter {
if [type] == "syslog" {
grok {
match => { "message" =>
"%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
}
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
if [type] == "apache_log" {
grok {
# match => [ "message", "%{COMBINEDAPACHELOG}" ]
# match => { "message" => "%{COMBINEDAPACHELOG}" }
# add_field => [ "received_at", "%{#timestamp}" ]
# add_field => [ "received_from", "%{host}" ]
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
#syslog_pri { }
#date {
# match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
#}
}
}
Thanks in advance!
My first thought would be the kibana field cache. Go to Settings->Indexes, select your index, and click the orange Reload button.
Related
Logstash is receiving a docs JSON object which contains various types of docs.
{
"docs": [
{
"_id": "project:A",
"_rev": "project:1",
"name": "secret",
"children": ["item:A"]
},
{
"_id": "item:A",
"_rev": "item:1",
"name": "secret"
}
]
}
I want each doc with an _id starting with project to include matching children. The end result should be:
{
"docs": [
{
"_id": "project:A",
"_rev": "project:1",
"name": "secret",
"children": [{
"_id": "item:A",
"_rev": "item:1",
"name": "secret"
}]
},
]
}
How can I achieve this?
Here is my conf file. I aven't been able to figure out how to solve this:
input {
file {
path => ["/home/logstash/logstash-testdata.json"]
sincedb_path => "/dev/null"
start_position => "beginning"
}
}
filter {
json {
source => "message"
}
// ... ???
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout {
codec => rubydebug
}
}
Finally, I got working ELK stack to get some logs from a remote server. However, I would like to customize the output of the logs. Is there a way to remove some fields which I am highlighting in yellow:
I tried to remove them from _source including remove_field in the logstash.conf:
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/..."
ssl_key => "/..logstash.key"
}
}
filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
}
remove_field => [ "tags", "prospector.type", "host.architecture", "host.containerized", "host.id", "host.os.platform", "host.os.family" ]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
}
}
Do you know how can I get rid of the yellow fields in _source for the logs coming from filebeat?
Update of logstash.conf based on Leandro comments:
input {
beats {
port => 5044
ssl => true
ssl_certificate => ".../logstash.crt"
ssl_key => ".../logstash.key"
}
}
filter {
grok {
match => {
"message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}"
}
remove_field => [ "tags","[prospector][type]","[host][architecture]", "[host][containerized]", "[host][id]", "[host][os][platform]", "[host][os][family]", "[beat][hostname]", "[beat][name]", "[beat][version], "[offset]", "[input][type]", "[meta][cloud][provider]", "[meta][cloud][machine_type]", "[meta][cloud][instance_id]"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
}
}
In logs:
019-02-27T17:03:41.637-0800 DEBUG [input] file/states.go:68 New state added for /logs/api.log
2019-02-27T17:03:41.637-0800 DEBUG [registrar] registrar/registrar.go:315 Registrar state updates processed. Count: 1
2019-02-27T17:03:41.637-0800 DEBUG [registrar] registrar/registrar.go:400 Write registry file: /filebeat/registry
2019-02-27T17:03:41.637-0800 INFO log/harvester.go:255 Harvester started for file: /logs/api.log
2019-02-27T17:03:41.647-0800 DEBUG [publish] pipeline/processor.go:308 Publish event: {
"#timestamp": "2019-02-28T01:03:41.647Z",
"#metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.6.0"
},
"log": {
"file": {
"path": "/logs/api.log"
}
},
"input": {
"type": "log"
},
"host": {
"name": "tomcat",
"os": {
"family": "redhat",
"name": "CentOS Linux",
"codename": "Core",
"platform": "centos",
"version": "7 (Core)"
},
"id": "6aaed308aa5a419f880c5e45eea65414",
"containerized": true,
"architecture": "x86_64"
},
"meta": {
"cloud": {
"region": "CanadaCentral",
"provider": "az",
"instance_id": "6452bcf4-7f5d-4fc3-9f8e-5ea57f00724b",
"instance_name": "tomcat",
"machine_type": "Standard_D8s_v3"
}
},
"message": "2018-09-14 20:23:37 INFO ContextLoader:272 - Root WebApplicationContext: initialization started",
"source": "/logs/api.log",
"offset": 0,
"prospector": {
"type": "log"
},
"beat": {
"hostname": "tomcat",
"version": "6.6.0",
"name": "tomcat"
}
}
Thanks
Some of those fields are nested fields, the way to access them in a Logstash filter is using the [field][subfield] notation.
Your remove_field shoud be something like this:
remove_field => ["tags","[host][architecture]","[meta][cloud][provider]"]
But I don't think you can remove the #version field.
UPDATE:
Using the event example from your Filebeat log I simulated a pipeline and got a _grokparsefailure, to remove the fields even when the grok fails you need to use the remove_field inside a mutate filter:
filter {
grok {
your grok
}
mutate {
remove_field => ["[prospector]","[host][architecture]", "[host][containerized]", "[host][id]", "[host][os][platform]", "[host][os][family]", "[beat]", "[offset]", "[input]", "[meta]"]
}
}
Don't remove the tags field until you have fixed your groks.
The logstash output on that example is:
{
"source": "/logs/api.log",
"tags": [
"_grokparsefailure"
],
"#timestamp": "2019-02-28T01:03:41.647Z",
"message": "2018-09-14 20:23:37 INFO ContextLoader:272 - Root WebApplicationContext: initialization started",
"log": {
"file": {
"path": "/logs/api.log"
}
},
"#version": "1",
"host": {
"os": {
"codename": "Core",
"version": "7 (Core)",
"name": "CentOS Linux"
},
"name": "tomcat"
}
}
Hi i am trying to setup ELK server for log management . My logstash service is running fine .I am receiving logs from other machine but logstash pipeline is not able to send data to elastic search
When i look at the logstash .log file it shows this error
I am not able to identify the error in my configuration file
:message=>"Error: Expected one of #, input, filter, output at line 34, column 1 (byte 855) after ", :level=>:error}
my logstash/conf.d
input {
beats {
port => 5044
ssl => true
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[#metadata][type]}"
}
}
{
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": "*"
}
}
],
"properties": {
"#timestamp": {
"type": "date"
},
"message": {
"type": "string",
"index": "analyzed"
},
"offset": {
"type": "long",
"doc_values": "true"
},
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "filebeat-*"
}
This error happens when there are files in /etc/logstash/conf.d directory which logstash can not parse. Remove them and see if this helps. In my case, I had the same error when reports.xml file was presented in conf.d directory.
geoip {
source => "remoteip"
target => "geoip"
database => "/etc/logstash/mmcity6.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
date {
locale => "en"
match => ["timestamp", "dd/MMM/YYYY:HH:mm:ss +0530" ]
target => "#timestamp"
}
kv {
field_split => "&?"
}
}
FORWARDEDipV6 [CLIENTIPV6] HOSTNAME [25/Mar/2016:19:47:13 +0530] HIT "GET URL HTTP/1.1" 200 5 "-" "Apache-HttpClient/UNAVAILABLE (java 1.4)" 0.000 -
"forwarded_ip": "ipv6",
"remoteip": "IPV6",
"loghost": "HOSTN,E",
"timestamp": "25/Mar/2016:19:47:13 +0530",
"cache": "HIT",
"httpmethod": "GET",
"request": "URL",
"httpversion": "1.1",
"response": "200",
"bytes": 5,
"agent": "\"Apache-HttpClient/UNAVAILABLE (java 1.4)\"",
"request_time": 0
BUT I AM NOT GETTING GEO LOCATION
The Support has just come in Elastic STack 5 ..
Logstash filter by tags for different websites
Issue: I have multiple websites inside a single IIS Server.. I want to add a "Tag" for each of the log files i am sending towards logstash
This is my logstash forwarder config
Each log file represents a different website.. so i want to add tags for each of these logs and be able to filter by this particular tag.
"logs\svr05\ex*",
{
"network": {
"servers": [ "logsvr1.logs.local:5000", "logsvr2.logs.local:5000" ],
"timeout": 15,
"ssl ca": "logstash-forwarder-new.crt"
},
"files": [
{
"paths": [
"logs\\svr08\\ex*",
"logs\\svr05\\ex*",
"logs\\svr04\\ex*",
"logs\\svr03\\ex*"
],
"fields": { "type": "iis" },
"dead time": "24h"
}
]
}
This is my IIS config for logstash..
filter {
if [type] == "iis" {
if [message] =~ "^#" {
drop {}
}
grok {
break_on_match => false
match => [
"message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:s-sitename} %{IPORHOST:s-ip} %{URIPROTO:cs-method} %{URIPATH:cs-uri-stem} (?:%{NOTSPACE:cs_query}|-) %{NUMBER:src_port} %{NOTSPACE:cs_username} %{IP:clientip} %{NOTSPACE:useragent} %{NUMBER:sc-substatus} %{NUMBER:sc_win32_status} %{NUMBER:sc-bytes} %{NUMBER:cs-bytes} %{NUMBER:timetaken}"
]
}
date {
locale => "en"
match => [ "log_timestamp", "YYYY-MM-dd HH:mm:ss" ]
target => "#timestamp"
timezone => "Indian/Maldives"
}
useragent {
source=> "useragent"
prefix=> "browser"
}
geoip {
source => "clientip"
target => "geoip"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
add_field => [ "src_ip", "%{clientip}" ]
convert => [ "[geoip][coordinates]", "float" ]
replace => [ "#source_host", "%{clientip}" ]
replace => [ "#message", "%{message}" ]
rename => [ "cs_method", "method" ]
rename => [ "cs_stem", "request" ]
rename => [ "useragent", "agent" ]
rename => [ "cs_username", "username" ]
rename => [ "sc_status", "response" ]
rename => [ "timetaken", "time_request" ]
}
}
}
filter
{
if [type] == "iis" {
mutate {
remove_field => [ "clientip", "host", "hostname", "logtime" ]
}
}
}
Suppose I want to send logs different apps
app1.egov.mv
app2.egov.mv
how can i add tags for these different IIS applications? and filter them in the discovery module to make graphs for specific websites using the tag? :|
regards,
Ismail
You already know how to add the type field so just use the same method to add another field containing the name of the host:
{
...,
"files": [
{
"paths": [
"logs\\svr08\\ex*",
"logs\\svr05\\ex*",
"logs\\svr04\\ex*",
"logs\\svr03\\ex*"
],
"fields": {
"type": "iis",
"virtualhost": "app1.egov.mv"
},
"dead time": "24h"
}
]
}
Obviously, if your different logfile patterns are for different servers you'll have to split your configuration:
{
...,
"files": [
{
"paths": [
"logs\\svr08\\ex*"
],
"fields": {
"type": "iis",
"virtualhost": "app1.egov.mv"
},
"dead time": "24h"
},
{
"paths": [
"logs\\svr05\\ex*"
],
"fields": {
"type": "iis",
"virtualhost": "app2.egov.mv"
},
"dead time": "24h"
},
...
]
}
Another option (that I prefer) is to have the web server itself include the hostname in each log entry.