Analyzing apache access logs with elasticsearch Watcher - access-log

Question

I am using the ELK Stack to analyze logs and I need to analyze and detect anomalies of apache access logs. What can I analyze with apache access logs and how should I give the conditions with curl -XPUT to Watcher?

Answer 1

If you haven't found it already, there's a decent tutorial at https://www.elastic.co/guide/en/watcher/watcher-1.0/watch-log-data.html. It provides a basic example of creating a log watch.
You can analyze/watch anything that you can query in Elasticsearch. It's just a matter of formatting the query with the correct JSON syntax. The guide for crafting the conditions is at https://www.elastic.co/guide/en/watcher/watcher-1.0/condition.html.
You'll also want to look at https://www.elastic.co/guide/en/watcher/watcher-1.0/actions.html to get an idea of the possible actions Watcher can take when a query meets a condition.
As far as the post to Watcher, each watch is essentially a JSON object. Because they can get pretty elaborate, I have found that it's best to create a file for each watch you want to create, and post them like this:
curl -XPUT http://my_elasticsearch:9200/_watcher/watch/my_watch_name -d #/path/to/my_watch_name.json
my_watch_name.json should have these basic elements (as described in the first link above):
{
"trigger" : { ... },
"input" : { ... },
"condition" : { ... },
"actions" : { ... }
}
The actions section is going to be specific to your use case, but here's a basic example of the other sections that I'm using successfully:
{
"trigger" : {
"schedule" : { "interval" : "5m" }
},
"input" : {
"search" : {
"request" : {
"indices" : [ "logstash" ],
"body" : {
"query" : {
"filtered" : {
"query" : {
"match" : { "message" : "error" }
},
"filter" : {
"range" : { "#timestamp" : { "gte" : "now-5m" } }
}
}
}
}
}
}
},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 } }
},
"actions" : {
...
}
}