How to parse the custom logs - logstash

I am new to logstash , can someone help me on grok filter to parse the data from multiple newline characters in the same log
2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()
url: GET::/system/data/connect/service
response: 200
elapsed: 10 ms

1.Using Grok
http://grokdebug.herokuapp.com/
[First Input Box] INPUT
2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()
response: 200
elapsed: 10 ms
[Second Input Box] Grok Parse ==>%{UPTONEWLINE:Part1}%{UPTONEWLINE:Part2}
Check Add custom patterns and add the following line
UPTONEWLINE (?:(.+?)(\n))
OUTPUT
{
"Part1": [
[
"2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()\n"
]
],
"Part2": [
[
"response: 200\n"
]
]
}
2.Without using Grok filter - Logstash configuration file
INPUT
2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()\nresponse: 200\nelapsed: 10 ms
Logstash Config File
input {
http {
port => 5043
response_headers => {
"Access-Control-Allow-Origin" => "*"
"Content-Type" => "text/plain"
"Access-Control-Allow-Headers" => "Origin, X-Requested-With, Content-Type,
Accept"
}
}
}
filter {
mutate {
split => ['message','\n']
add_field => {
"Part1" => "%{[message][0]}"
"Part2" => "%{[message][1]}"
"Part3" => "%{[message][2]}"
}
}
}
output {
stdout {
codec => rubydebug
}
}
OUTPUT
{
"host"=>"0:0:0:0:0:0:0:1",
"#version"=>"1",
"message"=>[
[0]"2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exe c-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()",
[1]"response: 200",
[2]"elapsed: 10 ms"
],
"Part1"=>"2018-10-08 13:38:34,280 [https-openssl-apr-0:0:0:0:0:0:0:0-8443-exec-424] INFO Rq:144839 ControllerInterceptor - afterCompletion()",
"Part2"=>"response: 200",
"Part3"=>"elapsed: 10 ms",
"#timestamp"=>2018-10-09T05: 27: 41.695Z
}

Related

Grok pattern for creating separate section in Kibana dashboard

I having been trying since long time to extract and mark data from my customized log using logstash, but not getting anywhere, I having a customized haproxy log like below:
Feb 22 21:17:32 ap haproxy[1235]: 10.172.80.45:32071 10.31.33.34:44541 10.31.33.34:32772 13.127.229.72:443 [22/Feb/2020:21:17:32.006] this_machine~ backend_test-tui/test-tui_32772 40/0/5/1/836 200 701381 - - ---- 0/0/0/0/0 0/0 {testtui.net} {cache_hit} "GET /ob/720/output00007.ts HTTP/1.1"
I want to extract and mark specific content in kibana dashboard from log, like:
from "40/0/5/1/836" section i want to mark the only the last section digit (836) as "response_time"
"701381" as "response_bytes"
"/ob/720/output00007.ts" as "content_url"
And want to use the timestamp in the log file and not the default one
I have created a grok filter using https://grokdebug.herokuapp.com/ but whenever i apply it i m seeing "_grokparsefailure" message and the kibana dashboard stops getting populated
Below is the logstash debug log
{
"#version" => "1",
"message" => "Mar 8 13:53:59 ap haproxy[22158]: 10.172.80.45:30835 10.31.33.34:57886 10.31.33.34:32771 43.252.91.147:443 [08/Mar/2020:13:53:59.827] this_machine~ backend_noida/noida_32771 55/0/1/0/145 200 2146931 - - ---- 0/0/0/0/0 0/0 {testalef1.adcontentamtsolutions.} {cache_hit} \"GET /felaapp/virtual_videos/og/1080/output00006.ts HTTP/1.1\"",
"#timestamp" => 2020-03-08T10:24:07.348Z,
"path" => "/home/alef/haproxy.log",
"host" => "com1",
"tags" => [
[0] "_grokparsefailure"
]
}
Below is the Filter which i have created
%{MONTH:[Month]} %{MONTHDAY:[date]} %{TIME:[time]} %{WORD:[source]} %{WORD:[app]}\[%{DATA:[class]}\]: %{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]} %{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]} \[%{HAPROXYDATE:[accept_date]}\] %{NOTSPACE:[frontend_name]}~ %{NOTSPACE:[backend_name]} %{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]} %{NUMBER:[http_status_code]} %{INT:[response_bytes]} - - ---- %{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]} %{DATA:[cache_status]} %{DATA:[domain_name]} %{NOTSPACE:[content]} HTTP/%{NUMBER:[http_version]}
Below is my logstash conf file:
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{MONTH:[Month]} %{MONTHDAY:[date]} %{TIME:[time]} %{WORD:[source]} %{WORD:[app]}\[%{DATA:[class]}\]: %{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]} %{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]} %{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]} \[%{HAPROXYDATE:[accept_date]}\] %{NOTSPACE:[frontend_name]}~ %{NOTSPACE:[backend_name]} %{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]} %{NUMBER:[http_status_code]} %{INT:[response_bytes]} - - ---- %{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]} %{DATA:[cache_status]} %{DATA:[domain_name]} %{NOTSPACE:[content]} HTTP/%{NUMBER:[http_version]} " }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
}
Using the below filter resolved my issue had to do debugging in the logstash itself to get proper filter:
input { beats {
port => 5044 } }
filter { grok {
match => { "message" => "%{MONTH:month} %{MONTHDAY:date} %{TIME:time} %{WORD:[source]} %{WORD:[app]}[%{DATA:[class]}]:
%{IPORHOST:[UE_IP]}:%{NUMBER:[UE_Port]}
%{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Source_Port]}
%{IPORHOST:[NATTED_IP]}:%{NUMBER:[NATTED_Destination_Port]}
%{IPORHOST:[WAN_IP]}:%{NUMBER:[WAN_Port]}
[%{HAPROXYDATE:[accept_date]}] %{NOTSPACE:[frontend_name]}~
%{NOTSPACE:[backend_name]}
%{NOTSPACE:[ty_name]}/%{NUMBER:[response_time]:int}
%{NUMBER:[http_status_code]} %{NUMBER:[response_bytes]:int} - - ----
%{NOTSPACE:[df]} %{NOTSPACE:[df]} %{DATA:[domain_name]}
%{DATA:[cache_status]} %{DATA:[domain_name]} %{URIPATHPARAM:[content]}
HTTP/%{NUMBER:[http_version]}" }
add_tag => [ "response_time", "response_time" ]
} date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }
output { elasticsearch { hosts => ["localhost:9200"] }
stdout {
codec => rubydebug
} }

Parsing json using logstash (ELK stack)

I have created a simple json like below
[
{
"Name": "vishnu",
"ID": 1
},
{
"Name": "vishnu",
"ID": 1
}
]
I am holding this values in file named simple.txt . Then i used file beat to listen the file and send the new updates to port 5043,on other side i started the log-stash service which listen to this port in order to parse and pass the json to elastic search.
log-stash is not processing the json values,it hangs in the middle.
logstash
input {
beats {
port => 5043
host => "0.0.0.0"
client_inactivity_timeout => 3600
}
}
filter {
json {
source => "message"
}
}
output {
stdout { codec => rubydebug }
}
filebeat config:
filebeat.prospectors:
- input_type: log
paths:
- filepath
output.logstash:
hosts: ["localhost:5043"]
Logstash output
**
Sending Logstash's logs to D:/elasticdb/logstash-5.6.3/logstash-5.6.3/logs which is now configured via log4j2.properties
[2017-10-31T19:01:17,574][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"D:/elasticdb/logstash-5.6.3/logstash-5.6.3/modules/fb_apache/configuration"}
[2017-10-31T19:01:17,578][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"D:/elasticdb/logstash-5.6.3/logstash-5.6.3/modules/netflow/configuration"}
[2017-10-31T19:01:18,301][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-10-31T19:01:18,388][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2017-10-31T19:01:18,573][INFO ][logstash.pipeline ] Pipeline main started
[2017-10-31T19:01:18,591][INFO ][org.logstash.beats.Server] Starting server on port: 5043
[2017-10-31T19:01:18,697][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
**
Every time when i am running log-stash using command
logstash -f logstash.conf
And since there is no processing of json i am stopping that service by pressing ctrl + c .
Please help me in finding the solution.Thanks in advance.
finally i got ended up with config like this.It works for me.
input
{
file
{
codec => multiline
{
pattern => '^\{'
negate => true
what => previous
}
path => "D:\elasticdb\logstash-tutorial.log\Test.txt"
start_position => "beginning"
sincedb_path => "D:\elasticdb\logstash-tutorial.log\null"
exclude => "*.gz"
}
}
filter {
json {
source => "message"
remove_field => ["path","#timestamp","#version","host","message"]
}
}
output {
elasticsearch { hosts => ["localhost"]
index => "logs"
"document_type" => "json_from_logstash_attempt3"
}
stdout{}
}
Json format:
{"name":"sachin","ID":"1","TS":1351146569}
{"name":"sachin","ID":"1","TS":1351146569}
{"name":"sachin","ID":"1","TS":1351146569}

How to send only error logs via logstash shipper

I am using Logstash to output JSON message to an API. I am reading logs from a log file. My configurations are working fine and it is also sending all the messages to the API.
Following is the sample log file:
Log File:
TID: [-1234] [] [2016-06-07 12:52:59,862] INFO {org.apache.synapse.core.axis2.ProxyService} - Successfully created the Axis2 service for Proxy service : TestServiceHttp {org.apache.synapse.core.axis2.ProxyService}
TID: [-1234] [] [2016-06-07 12:59:04,893] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/TestServiceHttp.TestServiceHttpHttpSoap12Endpoint********* Sending Message to the Queue*****WSAction: urn:mediate********* Sending Message to the Queue*****SOAPAction: urn:mediate********* Sending Message to the Queue*****MessageID: urn:uuid:d1bbe24a-2ce3-497f-8224-d260b0632506********* Sending Message to the Queue*****Direction: request********* Sending Message to the Queue*****Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><name> Omer</name></soapenv:Body></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-06-07 12:59:04,925] INFO {org.apache.synapse.core.axis2.TimeoutHandler} - This engine will expire all callbacks after : 120 seconds, irrespective of the timeout action, after the specified or optional timeout {org.apache.synapse.core.axis2.TimeoutHandler}
TID: [-1234] [] [2016-06-07 12:59:04,933] ERROR {org.apache.axis2.description.ClientUtils} - The system cannot infer the transport information from the jms:/Customer.01.Request.Queue.01?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&java.naming.provider.url=tcp://localhost:61616&transport.jms.DestinationType=queue URL. {org.apache.axis2.description.ClientUtils}
TID: [-1234] [] [2016-06-07 12:59:04,949] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Unexpected error during sending message out {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: The system cannot infer the transport information from the jms:/Customer.01.Request.Queue.01?transport.jms.ConnectionFactoryJNDIName=QueueConnectionFactory&java.naming.factory.initial=org.apache.activemq.jndi.ActiveMQInitialContextFactory&java.naming.provider.url=tcp://localhost:61616&transport.jms.DestinationType=queue URL.
at org.apache.axis2.description.ClientUtils.inferOutTransport(ClientUtils.java:81)
at org.apache.axis2.client.OperationClient.prepareMessageContext(OperationClient.java:288)
at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
TID: [-1234] [] [2016-06-07 12:59:05,009] INFO {org.apache.synapse.mediators.builtin.LogMediator} - To: /services/TestServiceHttp.TestServiceHttpHttpSoap12Endpoint, WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:d1bbe24a-2ce3-497f-8224-d260b0632506, Direction: request, MESSAGE = Executing default 'fault' sequence, ERROR_CODE = 0, ERROR_MESSAGE = Unexpected error during sending message out, Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"><soapenv:Body><name> Omer</name></soapenv:Body></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-06-07 13:00:04,890] INFO {org.apache.axis2.transport.http.HTTPSender} - Unable to sendViaPost to url[http://Omer-PC:8280/services/TestServiceHttp.TestServiceHttpHttpSoap12Endpoint] {org.apache.axis2.transport.http.HTTPSender}
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:78)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:106)
ent.ServiceClient.sendReceive(ServiceClient.java:530)
at org.apache.jsp.admin.jsp.WSRequestXSSproxy_005fajaxprocessor_jsp._jspService(WSRequestXSSproxy_005fajaxprocessor_jsp.java:294)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:155)
at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
TID: [-1234] [] [2016-06-07 13:01:40,447] INFO {org.wso2.carbon.core.init.CarbonServerManager} - Shutdown hook triggered.... {org.wso2.carbon.core.init.CarbonServerManager}
TID: [-1234] [] [2016-06-07 13:01:40,464] INFO {org.wso2.carbon.core.init.CarbonServerManager} - Gracefully shutting down WSO2 Enterprise Service Bus... {org.wso2.carbon.core.init.CarbonServerManager}
TID: [-1234] [] [2016-06-07 13:01:40,477] INFO {org.wso2.carbon.core.ServerManagement} - Starting to switch to maintenance mode... {org.wso2.carbon.core.ServerManagement}
TID: [-1234] [] [2016-06-07 13:01:40,481] INFO {org.apache.axis2.transport.jms.JMSListener} - JMS Listener Shutdown {org.apache.axis2.transport.jms.JMSListener}
Following is my configuration file:
Configuration File:
input {
stdin {}
file {
path => "C:\WSO2Environment\wso2esb-4.9.0\repository\logs\wso2carbon.log"
type => "wso2"
start_position => "beginning"
codec => multiline {
pattern => "(^\s*at .+)|^(?!TID).*$"
negate => false
what => "previous"
}
}
}
filter {
if [type] == "wso2" {
grok {
match => [ "message", "TID:%{SPACE}\[%{INT:SourceSystemId}\]%{SPACE}\[%{DATA:ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:TimeStamp}\]%{SPACE}%{LOGLEVEL:MessageType}%{SPACE}{%{JAVACLASS:MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:Message}" ]
add_tag => [ "grokked" ]
}
mutate {
gsub => [
"TimeStamp", "\s", "T",
"TimeStamp", ",", "."
]
}
}
if !( "_grokparsefailure" in [tags] ) {
grok{
match => [ "message", "%{GREEDYDATA:StackTrace}" ]
add_tag => [ "grokked" ]
}
date {
match => [ "timestamp", "yyyy MMM dd HH:mm:ss:SSS" ]
target => "TimeStamp"
timezone => "UTC"
}
}
if ( "multiline" in [tags] ) {
grok {
match => [ "message", "%{GREEDYDATA:StackTrace}" ]
add_tag => [ "multiline" ]
tag_on_failure => [ "multiline" ]
}
date {
match => [ "timestamp", "yyyy MMM dd HH:mm:ss:SSS" ]
target => "TimeStamp"
}
}
}
output {
stdout { }
http {
url => "http://localhost:8086/messages"
http_method => "post"
format => "json"
mapping => ["TimeStamp","%{TimeStamp}","MessageType","%{MessageType}","MessageTitle","%{MessageTitle}","Message","%{log_EventMessage}","SourceSystemId","%{SourceSystemId}","StackTrace","%{log_StackTrace}"]
}
}
Problem Statement:
The configuration file is working correctly and sending all the log entries to the API, but I only want to send error logs to the API. So, I want to place a check on "MessageType" in which I am getting the Log Level that If it's value is "ERROR" only then it should send messages through to the API otherwise logstash should discard the message.
In your logstash configuration in the filter section you can use add tag based on your if condition. And in the output add if statement that checks if the tag error is present it will send otherwise it ignores.
After the following if statement:
if [type] == "wso2" {
grok {
match => [ "message", "TID:%{SPACE}\[%{INT:SourceSystemId}\]%{SPACE}\[%{DATA:ProcessName}\]%{SPACE}\[%{TIMESTAMP_ISO8601:TimeStamp}\]%{SPACE}%{LOGLEVEL:MessageType}%{SPACE}{%{JAVACLASS:MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:Message}" ]
add_tag => [ "grokked" ]
}
mutate {
gsub => [
"TimeStamp", "\s", "T",
"TimeStamp", ",", "."
]
}
}
Add the following statement in your filter:
if "grokked" in [tags] {
grok {
match => ["MessageType", "ERROR"]
add_tag => [ "loglevelerror" ]
}
}
Then in your output make following changes:
output {
if "loglevelerror" in [tags] {
stdout { }
http {
url => "http://localhost:8086/messages"
http_method => "post"
format => "json"
mapping => ["TimeStamp","%{TimeStamp}","MessageType","%{MessageType}","MessageTitle","%{MessageTitle}","Message","%{log_EventMessage}","SourceSystemId","%{SourceSystemId}","StackTrace","%{log_StackTrace}"]
}
}
}
I tested it out on my machine using stdout. It works fine. Hope it helps!

Logstash Simple File Input Configuration

I am new to Logstash, and I have been trying to make a simple .conf file to read logs from sample Log file. I have tried everything from making sincedb_path to $HOME/.sincedb to setting the start_path to "Beginning", but I can't seem to get the data to be read even to stdout. The following is a sample line from my sample log:
10.209.12.40 - - [06/Aug/2014:22:59:18 +0000] "GET /robots.txt HTTP/1.1" 200 220 "-" "Example-Prg/1.0"
www.example.com 10.209.11.40 - - [06/Aug/2014:23:05:15 +0000] "GET /robots.txt HTTP/1.1" 200 220 "-" "Example-Prog/1.0"
www.example.com 10.209.11.40 - - [06/Aug/2014:23:10:21 +0000] "GET /File/location-path HTTP/1.1" 404 25493 "http://blog.example.com/link-1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_2) AppleWebKit/537.36(KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36"
The following is the .conf file that I am using:
input
{
stdin
{
}
file
{
# type => "access"
path => ["/home/user/Desktop/path1/www.example.com-access_log_20140806.log"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter
{
# if [type] == "access"
# {
grok
{
break_on_match => false
match => {
"message" => '%{IP:sourceIP} %{DATA:User_Id} %{DATA:User_Auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:HTTP_Command} %{DATA:HTTP_Path} %{DATA:HTTP_Version}\" %{NUMBER:HTTP_Code} %{NUMBER:Bytes} \"%{DATA:Host}\" \"%{DATA:Agent}\"'
}
match => {
"message" => '%{HOST:WebPage} %{IP:sourceIP} %{DATA:User_Id} %{DATA:User_Auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:HTTP_Command} %{DATA:HTTP_Path} %{DATA:HTTP_Version}\" %{NUMBER:HTTP_Code} %{NUMBER:Bytes} \"%{DATA:Host}\" \"%{DATA:Agent}\"'
}
}
# }
}
output
{
stdout
{
codec => rubydebug
}
}
I am running it through stdout to get check whether I am getting an output or not. I am getting the following output:
{
"message" => "",
"#version" => "1",
"#timestamp" => "2015-07-02T20:48:55.453Z",
"host" => "monil-Inspiron-3543",
"tags" => [
[0] "_grokparsefailure"
]
}
I have spent a good number of hours trying to figure out what is wrong. Please tell me where I am going wrong.
Thanks in Advance.
EDIT: It was an error in the file name.

Logstash Grok filter for uwsgi logs

I'm a new user to ELK stack. I'm using UWSGI as my server. I need to parse my uwsgi logs using Grok and then analyze them.
Here is the format of my logs:-
[pid: 7731|app: 0|req: 357299/357299] ClientIP () {26 vars in 511 bytes} [Sun Mar 1 07:47:32 2015] GET /?file_name=123&start=0&end=30&device_id=abcd&verif_id=xyzsghg => generated 28 bytes in 1 msecs (HTTP/1.0 200) 2 headers in 79 bytes (1 switches on core 0)
I used this link to generate my filter, but it didn't parse much of the information.
The filter generated by the above link is
%{SYSLOG5424SD} %{IP} () {26 vars in 511 bytes} %{SYSLOG5424SD} GET %{URIPATHPARAM} => generated 28 bytes in 1 msecs (HTTP%{URIPATHPARAM} 200) 2 headers in 79 bytes (1 switches on core 0)
Here is my logstash-conf file.
input { stdin { } }
filter {
grok {
match => { "message" => "%{SYSLOG5424SD} %{IP} () {26 vars in 511 bytes} %{SYSLOG5424SD} GET %{URIPATHPARAM} => generated 28 bytes in 1 msecs (HTTP%{URIPATHPARAM} 200) 2 headers in 79 bytes (1 switches on core 0)" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
output {
stdout { codec => rubydebug }
}
Upon running logstash with this conf file, I get an error message saying:-
{
"message" => "[pid: 7731|app: 0|req: 357299/357299] ClientIP () {26 vars in 511 bytes} [Sun Mar 1 07:47:32 2015] GET /?file_name=123&start=0&end=30&device_id=abcd&verif_id=xyzsghg => generated 28 bytes in 1 msecs (HTTP/1.0 200) 2 headers in 79 bytes (1 switches on core 0)",
"#version" => "1",
"#timestamp" => "2015-03-01T07:57:02.291Z",
"host" => "cube26-Inspiron-3542",
"tags" => [
[0] "_grokparsefailure"
]
}
The date has been properly formatted. How do I extract other information from my logs, such as my query parameters(filename, start,end, deviceid etc) and ClientIP , Response code etc.
Also, is there any built-in UWSGI log parser which can be used, such as the one built for apache and syslog?
EDIT
I wrote this on my own, but it throws the same error:
%{SYSLOG5424SD} %{IP:client_ip} () {%{NUMBER:vars} vars in %{NUMBER:bytes} bytes} %{SYSLOGTIMESTAMP:date} %{WORD:method} %{URIPATHPARAM:request} => generated %{NUMBER:generated_bytes} bytes in {NUMBER:secs} msecs (HTTP/1.0 %{NUMBER:response_code}) %{NUMBER:headers} headers in %{NUMBER:header_bytes} (1 switches on core 0)
EDIT 2
I'm finally able to crack it myself. The GROK filter for the above log will be:
\[pid: %{NUMBER:pid}\|app: %{NUMBER:app}\|req: %{NUMBER:req_num1}/%{NUMBER:req_num2}\] %{IP:client_ip} \(\) \{%{NUMBER:vars} vars in %{NUMBER:bytes} bytes\} %{SYSLOG5424SD} %{WORD:method} /\?file_name\=%{NUMBER:file_name}\&start\=%{NUMBER:start}\&end\=%{NUMBER:end} \=\> generated %{NUMBER:generated_bytes} bytes in %{NUMBER:secs} msecs \(HTTP/1.0 %{NUMBER:response_code}\) %{NUMBER:headers} headers in %{NUMBER:header_bytes}
But my questions still remain:
is there any default uwsgi log filter in grop??**
I've been applying different matches for different query parameters. Is there anything in grok that fetches the different query parameters by itself??
I found the solution for extracting the query parameters:-
Here is my final configuration:-
For log line
[pid: 7731|app: 0|req: 426435/426435] clientIP () {28 vars in 594 bytes} [Mon Mar 2 06:43:08 2015] GET /?file_name=wqvqwv&start=0&end=30&device_id=asdvqw&verif_id=qwevqwr&lang=English&country=in => generated 11018 bytes in 25 msecs (HTTP/1.0 200) 2 headers in 82 bytes (1 switches on core 0)
the configuration is
input { stdin { } }
filter {
grok {
match => { "message" => "\[pid: %{NUMBER}\|app: %{NUMBER}\|req: %{NUMBER}/%{NUMBER}\] %{IP} \(\) \{%{NUMBER} vars in %{NUMBER} bytes\} %{SYSLOG5424SD:DATE} %{WORD} %{URIPATHPARAM} \=\> generated %{NUMBER} bytes in %{NUMBER} msecs \(HTTP/1.0 %{NUMBER}\) %{NUMBER} headers in %{NUMBER}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
kv {
field_split => "&? "
include_keys => [ "file_name", "device_id", "lang", "country"]
}
}
output {
stdout { codec => rubydebug }
elasticsearch { host => localhost }
}
I found your solution did't support HTTP/1.1. I fixed it and also add variables name. Ref
Here's my grok config:
grok {
match => { "message" => "\[pid: %{NUMBER:pid}\|app: %{NUMBER:id}\|req: %{NUMBER:currentReq}/%{NUMBER:totalReq}\] %{IP:remoteAddr} \(%{WORD:remoteUser}?\) \{%{NUMBER:CGIVar} vars in %{NUMBER:CGISize} bytes\} %{SYSLOG5424SD:timestamp} %{WORD:method} %{URIPATHPARAM:uri} \=\> generated %{NUMBER:resSize} bytes in %{NUMBER:resTime} msecs \(HTTP/%{NUMBER:httpVer} %{NUMBER:status}\) %{NUMBER:headers} headers in %{NUMBER:headersSize} bytes %{GREEDYDATA:coreInfo}" }
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}

Resources