Laravel Parsing log with elk (elasticsearch, logstash, kibana) - logstash

Question

I have configured ELK successfully for Laravel app, But we are facing issue with Laravel log. I have configured logstash template with below code. but I am receiving Break line in Kibana. I have tried two different configuration code as per below details.
20-laravel.conf
input {
stdin{
codec => multiline {
pattern => "^\["
what => "previous"
negate => true
}
}
}
filter {
grok {
match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: %{DATA:message}" }
}
}
output {
elasticsearch {
document_type => "logs"
hosts => ["127.0.0.1"]
index => "laravel_logs"
}
}
filter {
# Laravel log files
if [type] == "laravel" {
grok {
match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] %{DATA:env}\.%{DATA:severity}: %{DATA:message} \[" }
}
}
}
laravel sample log is :
[2017-09-13 16:19:28] production.ERROR: Symfony\Component\Debug\Exception\FatalThrowableError: Parse error: syntax error, unexpected identifier (T_STRING), expecting ',' or ')' in /var/www/app/Http/Controllers/BrandsController.php:57
Stack trace:
#0 /var/www/vendor/composer/ClassLoader.php(322):
Composer\Autoload\includeFile('/var/www/vendor...')
#1 [internal function]: Composer\Autoload\ClassLoader-
>loadClass('App\\Http\\Contro...')
#2 [internal function]: spl_autoload_call('App\\Http\\Contro...')
So my main issue is we are reciveing this log in kibana in single line. for example above log code is a divided in different line message and we can't figure out that which line message is from which error?
Kibana log output for single laravel log is displayed in below image.kibana log-output

Answer 1

An easy alternative is to use Laralog.
With Laralog it is possible to Laravel logs directly to Elastic Search without install all the full Logstash stack, so it is suitable for small and container environments.
Example of usage:
laralog https://elasticsearch:9200 --input=laravel.log
Laralog will parse and send the logs automatically.

Answer 2

You should create a new provider to setup monolog properly, try the following setup:
class LogstashProvider extends ServiceProvider
{
public function boot(): void
{
$stream = storage_path('logs/laravel.log');
$name = env('APP_NAME');
$formatter = new LogstashFormatter($name, null, null, 'ctxt_', LogstashFormatter::V1);
$streamHandler = new StreamHandler($stream, Logger::DEBUG, false);
$streamHandler->setFormatter($formatter);
Log::getMonolog()->pushHandler(
$streamHandler
);
}
}
You also should configure your logstash to parse json instead