Exception while parsing logfile in logstash - logstash-grok

Question

I am trying to parse Selenium Log file using logstash , but i am getting exception while parsing.
grok pattern used :
grok {
match => { "message" => "%{LOGLEVEL:severity} *%{GREEDYDATA:msg}" }
}
and sample log file is :
INFO Parental Page Loaded and Lock Content displayed
INFO Received PIN:1234
INFO The UI element mtfileTV.ParentalControls.Enabled [Enabled On] is displayed
INFO Parental Switch is already Enabled
ERROR Exception occoured while finding the given element name "mtfileTV.ParentalControls.Disabled" type "NAME" description " Parental lock is disabled text status" Value "Disabled Off"
ERROR An element could not be located on the page using the given search parameters. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 382 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: 'unknown', revision: 'b526bd5', time: '2017-03-07 11:11:07 -0800'
System info: host: 'IN5033-01', ip: '192.168.86.127', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_151'
Driver info: io.appium.java_client.windows.WindowsDriver
Capabilities [{app=mtfileTVLLC.mtfileTV_vgszm6stshdqy!App, platformName=Windows, platform=ANY}]
Session ID: 9D664F38-2D99-4751-A535-60DF0466A08F
*** Element info: {Using=name, value=Disabled Off}
ERROR Exception occoured while finding the given element name "mtfileTV.ParentalControls.Disabled" type "NAME" description " Parental lock is disabled text status" Value "Disabled Off" The specified element mtfileTV.ParentalControls.Disabled Parental lock is disabled text status does not exist
ERROR The UI element mtfileTV.ParentalControls.Disabled is not displayed
INFO Parental control is already Enabled.Process ahead
INFO The UI element mtfileTV.ParentalControls.Status.RNC17TVMA.Locked [R, NC-17, TV-MA Locked] is displayed
INFO The UI element mtfileTV.ParentalControls.Status.RNC17TVMA.Locked [R, NC-17, TV-MA Locked] is displayed
INFO The text "R, NC-17, TV-MA Locked" got from UI element with name mtfileTV.ParentalControls.Status.RNC17TVMA.Locked [R, NC-17, TV-MA Locked]
INFO The UI element mtfileTV.ParentalControls.Status.RNC17TVMA.Locked [R, NC-17, TV-MA Locked] is displayed
INFO The text "R, NC-17, TV-MA Locked" got from UI element with name mtfileTV.ParentalControls.Status.RNC17TVMA.Locked [R, NC-17, TV-MA Locked]
INFO Cur status of the Requested Rating :R, NC-17, TV-MA Locked: is = R, NC-17, TV-MA Locked
INFO R, NC-17, TV-MA Locked is already locked.Do Nothing.
ERROR Exception occoured while finding the given element name "mtfileTV.ParentalControls.Disabled" type "NAME" description " Parental lock is disabled text status" Value "Disabled Off"
ERROR An element could not be located on the page using the given search parameters. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 438 milliseconds
For documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html
Build info: version: 'unknown', revision: 'b526bd5', time: '2017-03-07 11:11:07 -0800'
System info: host: 'IN5033-01', ip: '192.168.86.127', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_151'
Driver info: io.appium.java_client.windows.WindowsDriver
Capabilities [{app=mtfileTVLLC.mtfileTV_vgszm6stshdqy!App, platformName=Windows, platform=ANY}]
Please suggest me suitable grok pattern

Answer 1

The problem is that you have messages consisting of multiple lines. Therefore, you need to use the multiline codec to join all lines which do not start with a LOGLEVEL.
We can write a sample configuration (call it selenium.conf):
input {
stdin {
codec => multiline {
pattern => "^%{LOGLEVEL} "
negate => true
what => "previous"
}
}
}
filter {
grok {
match => { "message" => "%{LOGLEVEL:severity} %{GREEDYDATA:lines}" }
remove_field => [ "message" ]
}
}
output {
stdout {
codec => rubydebug
}
}
If we assume that the sample log file you provided is stored in selenium.log, we can do
$ logstash -f selenium.conf < selenium.log
and see that it works :)
Sample output follows (note the multiline tag in the first entry):
{
"host" => "d125q-PC",
"#timestamp" => 2018-07-04T11:10:28.141Z,
"lines" => "An element could not be located on the page using the given search parameters. (WARNING: The server did not provide any stacktrace information)\nCommand duration or timeout: 382 milliseconds\nFor documentation on this error, please visit: http://seleniumhq.org/exceptions/no_such_element.html\nBuild info: version: 'unknown', revision: 'b526bd5', time: '2017-03-07 11:11:07 -0800'\nSystem info: host: 'IN5033-01', ip: '192.168.86.127', os.name: 'Windows 10', os.arch: 'amd64', os.version: '10.0', java.version: '1.8.0_151'\nDriver info: io.appium.java_client.windows.WindowsDriver\nCapabilities [{app=mtfileTVLLC.mtfileTV_vgszm6stshdqy!App, platformName=Windows, platform=ANY}]\nSession ID: 9D664F38-2D99-4751-A535-60DF0466A08F\n*** Element info: {Using=name, value=Disabled Off}",
"tags" => [
[0] "multiline"
],
"#version" => "1",
"severity" => "ERROR"
}
{
"host" => "d125q-PC",
"#timestamp" => 2018-07-04T11:10:28.142Z,
"lines" => "The UI element mtfileTV.ParentalControls.Status.RNC17TVMA.Locked [R, NC-17, TV-MA Locked] is displayed",
"#version" => "1",
"severity" => "INFO"
}
{
"host" => "d125q-PC",
"#timestamp" => 2018-07-04T11:10:28.144Z,
"lines" => "The text \"R, NC-17, TV-MA Locked\" got from UI element with name mtfileTV.ParentalControls.Status.RNC17TVMA.Locked [R, NC-17, TV-MA Locked]",
"#version" => "1",
"severity" => "INFO"
}