How can I monitor and detect errors when connecting kafka to logstash.
Say for example my kafka broker is down and no connection is established between kafka and logstash.
Is there a way in to get the monitor the connection status between logstash and kafka?
I can query logstash logs (but I don't think it is the appropriate way) and I tried to use logstash monitoring API (for example localhost:9600/_node/stats/pipelines?pretty) but no api gives me the connection status is off
Thank you in advance
If you have an elastic agent or a metricbeat agent installed on the Kafka node, you can configure the agent to monitor them using their Kafka specific module.
elastic-agent kafka module
metricbeat kafka module
For getting the connection status from logstash as you mentioned, you can also configure your logstash config to grab the status from the log message.
Sample document in elasticsearch :
{
"_index": "topicname",
"_type": "_doc",
"_id": "ulF8uH0BK9MbBSR7DPEw",
"_version": 1,
"_score": null,
"fields": {
"#timestamp": [
"2022-05-09T10:27:56.956Z"
],
"#version": [
"1"
],
"#version.keyword": [
"1"
],
"message": [
"{\"requestMethod\":\"GET\",\"headers\":{\"content-type\":\"application/json\",\"user-agent\":\"PostmanRuntime/7.XX.XX\",\"accept\":\"*/*\",\"postman-token\":\"11224442345223\",\"host\":\"localhost:2300\",\"accept-encoding\":\"gzip, deflate, br\",\"connection\":\"keep-alive\",\"content-length\":\"44\"},\"body\":{\"category\":\"CAT\",\"noise\":\"purr\"},\"query\":{},\"requestUrl\":\"http://localhost:2300/kafka\",\"protocol\":\"HTTP/1.1\",\"remoteIp\":\"1\",\"requestSize\":302,\"userAgent\":\"PostmanRuntime/7.XX.X\",\"statusCode\":200,\"response\":{\"success\":true,\"message\":\"Kafka Details are added\",\"data\":{\"kafkaData\":{\"_id\":\"12gvsddwqbwrfteacr313rcet5\",\"category\":\"DOG\",\"noise\":\"bark\",\"__v\":0},\"postData\":{\"category\":\"DOG\",\"noise\":\"bark\"}}},\"latency\":{\"seconds\":0,\"nanos\":61000000},\"responseSize\":193}"
]
} }
Below configuration can be added to fetch the status:
input {
kafka {
topics => ["topicname"]
bootstrap_servers => "11.11.11.11:1111"
}
}
filter{
mutate { add_field => { "StatusCode" => "%{[message][0][status]}" } }
}
output {
elasticsearch {
hosts => ["11.11.11.12:9200"]
index => "topic-name-index"
}
}
Related
Im trying to parse a message from my network devices which send messages in format similar to
<30>Feb 14 11:33:59 wireless: ath0 Sending auth to xx:xx:xx:xx:xx:xx. Status: The request has been declined due to MAC ACL (52).\n
<190>Feb 14 11:01:29 CCR00 user admin logged out from xx.xx.xx.xx via winbox
<134>2023 Feb 14 11:00:33 ZTE command-log:An alarm 36609 level notification occurred at 11:00:33 02/14/2023 CET sent by MCP GponRm notify: <gpon-onu_1/1/1:1> SubType:1 Pos:1 ONU Uni lan los. restore\n on \n
using this logstash.conf file
input {
beats {
port => 5044
}
tcp {
port => 50000
}
udp {
port => 50000
}
}
## Add your filters / logstash plugins configuration here
filter {
grok {
match => {
"message" => "^(?:<%{POSINT:syslog_pri}>)?%{GREEDYDATA:message_payload}"
}
}
syslog_pri {
}
mutate {
remove_field => [ "#version" , "message" ]
}
}
output {
stdout {}
elasticsearch {
hosts => "elasticsearch:9200"
user => "logstash_internal"
password => "${LOGSTASH_INTERNAL_PASSWORD}"
}
}
which results in this output
{
"#timestamp": [
"2023-02-14T10:38:59.228Z"
],
"data_stream.dataset": [
"generic"
],
"data_stream.namespace": [
"default"
],
"data_stream.type": [
"logs"
],
"event.original": [
"<14> Feb 14 11:38:59 UBNT BOXSERV[boxs Req]: boxs.c(691) 55381193 %% Error 17 occurred reading thermal sensor 2 data\n\u0000"
],
"host.ip": [
"10.125.132.10"
],
"log.syslog.facility.code": [
1
],
"log.syslog.facility.name": [
"user-level"
],
"log.syslog.severity.code": [
5
],
"log.syslog.severity.name": [
"notice"
],
"message_payload": [
" Feb 14 11:38:59 UBNT[boxs Req]: boxs.c(691) 55381193 %% Error 17 occurred reading thermal sensor 2 data\n\u0000"
],
"syslog_pri": [
"14"
],
"_id": "UzmBT4YBAZPdbqc4m_IB",
"_index": ".ds-logs-generic-default-2023.02.04-000001",
"_score": null
}
which is mostly satisfactory, but i would expect the
log.syslog.facility.name
and
log.syslog.severity.name
fields to be processed by the
syslog_pri
filter
with imput of
<14>
to result into
secur/auth
and
Alert
recpectively,
but i keep getting the default user-level notice for all my messages, no matter what the part of the syslog message contains
anyone could advise and maybe fix my .conf syntax, if its wrong?
thank you very much!
i have logstash configured properly to receive logs and send them to elastics, but the grok/syslog_pri doesnt yield expected results
The fact that the syslog_pri filter is setting [log][syslog][facility][code] shows that it has ECS compatibility enabled. As a result, if you do not set the syslog_pri_field_name option on the syslog_pri filter, it will try to parse [log][syslog][priority]. If that field does not exist then it will parse the default value of 13, which is user-level/notice.
thank you for the answer, i have adjusted the code by the given advice
filter {
grok {
match => { "message" => "^(?:<%{POSINT:syslog_code}>)?%{GREEDYDATA:message_payload}"
} }
syslog_pri { syslog_pri_field_name => "syslog_code"
}
mutate { remove_field => [ "#version" , "message" ] } }
and now it behaves as intended
"event" => {
"original" => "<30>Feb 15 18:41:04 dnsmasq-dhcp[960]: DHCPACK(eth0) 10.0.0.165 xx:xx:xx:xx:xx CZ\n"
},
"#timestamp" => 2023-02-15T17:41:04.977038615Z,
"message_payload" => "Feb 15 18:41:04 dnsmasq-dhcp[960]: DHCPACK(eth0) 10.0.0.165 xx:xx:xx:xx:xx CZ\n",
"log" => {
"syslog" => {
"severity" => {
"code" => 6,
"name" => "informational"
},
"facility" => {
"code" => 3,
"name" => "daemon"
}
}
},
"syslog_code" => "30",
"host" => {
"ip" => "xx.xx.xx.xx"
} }
i will adjust the message a bit to fit my needs,
but that is out of the scope of this question
thank you very much!
I have a Node.js server application with Express. I would like to log its activity into ElasticSearch and visualize the logs using Kibana.
What would be the right way to do that?
Should I write a log file of json lines and read it with Logstash?
I'd recommend log4js. It has a range of useful appenders, and logstash is one of them. It works over UDP.
Here is an example taken from the log4js site:
var log4js = require('../lib/log4js');
/*
Sample logstash config:
udp {
codec => json
port => 10001
queue_size => 2
workers => 2
type => myAppType
}
*/
log4js.configure({
"appenders": [
{
type: "console",
category: "myLogger"
},
{
"host": "127.0.0.1",
"port": 10001,
"type": "logstashUDP",
"logType": "myAppType", // Optional, defaults to 'category'
"fields": { // Optional, will be added to the 'fields' object in logstash
"field1": "value1",
"field2": "value2"
},
"layout": {
"type": "pattern",
"pattern": "%m"
},
"category": "myLogger"
}
]
});
var logger = log4js.getLogger("myLogger");
logger.info("Test log message %s", "arg1", "arg2");
I'm a Logstash newbie and I've looked at numerous examples of grok patterns and I'm still struggling to acheive my goal which is to parse the following JSON formatted log event.
{
"#fields": {
"level": "DEBUG",
"mdc": {},
"file": "SearchServiceImpl.java",
"class": "com.blah.blah.service.impl.SearchServiceImpl",
"line_number": "767",
"method": "getUserSavedSearches"
},
"#timestamp": "2015-04-24T12:30:37.953+01:00",
"#message": "username: admin sessionid: 56cR73aBpuIBzRgIElzLUtJJ method_name: getUserSavedSearches",
"#source_host": "Kens-MacBook.local"
}
In particular I'd like to extract the session id and username.
I'm also hoping I can be pointed to detailed documentation explaining how to use Grok. (I've read the available docs on logstash etc).
Any help will be appreciated
First, your log format is in JSON. So, in your config you can use json codec to read the log. Then, use GROK to parse the username & session id.
input {
stdin{
codec => json
}
}
filter {
grok {
match => [
"#message", "username: %{USERNAME:username} sessionid: %{NOTSPACE:sessionId} method_name: %{WORD:method_name}"
]
}
}
output {
stdout { codec => rubydebug }
}
For more detailed document, you can use this site to help you try your grok pattern and this site for the grok pattern that can use.
Here is the sample output:
{
"#fields" => {
"level" => "DEBUG",
"mdc" => {},
"file" => "SearchServiceImpl.java",
"class" => "com.blah.blah.service.impl.SearchServiceImpl",
"line_number" => "767",
"method" => "getUserSavedSearches"
},
"#timestamp" => "2015-04-24T19:30:37.953+08:00",
"#message" => "username: admin sessionid: 56cR73aBpuIBzRgIElzLUtJJ method_name: getUserSavedSearches",
"#source_host" => "Kens-MacBook.local",
"#version" => "1",
"host" => "BEN_LIM",
"username" => "admin",
"sessionId" => "56cR73aBpuIBzRgIElzLUtJJ",
"method_name" => "getUserSavedSearches"
}
I am doing centralized logging using logstash. I am using logstash-forwarder on the shipper node and ELK stack on the collector node.The issue is that i want logstash to parse the file from the beginning which is present on the shipper node. The config file logstash-forwarder.conf on the shipper has following configuration :
{
"network": {
"servers": [ "XXX.XX.XX.XXX:5000" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
},
"files": [
{
"paths": [
"/apps/newlogs.txt"
],
"fields": { "type": "syslog" }
}
]
}
And the collector configuration is :
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logdate}\s%{LOGLEVEL:level}\s-\s%{WORD:USE_CASE}\s:\s%{WORD:STEP_DETAIL}\s:\s\[%{WORD:XXX}\]\s:\s(?<XXX>([^\s]+))\s:\s%{GREEDYDATA:MESSAGE_DETAILS}" }
add_field => [ "received_at", "%{#timestamp}" ]
add_field => [ "received_from", "%{host}" ]
add_tag => [ "level:%{level}" ]
add_tag => [ "USE_CASE:%{USE_CASE}" ]
}
}
}
output {
elasticsearch { host => localhost}
stdout { codec => rubydebug }
}
I want that the file should be parsed from the begging and not for every event generated,Which we do easily in the logstash.conf by specifying start_position => beginning but i am unable to find a straightforward way in logstash-forwarder as the file will be present on the shipper side.
Thanks.
As far as I'm aware, the default behaviour for logstash-forwarder is to start from the beginning of a file - so the shipper should already be reading from the start as intended.
You haven't said what you've tried doing to diagnose the problem. If you haven't already done so, I would temporarily bypass the collector to confirm that the shipper is working as expected and rule out potential issues with the certificates.
I am not able to get specific lines from logs file /var/log/messages. I am using logstash-forwarder in client-server and logstash, elasticsearch and kibana in log-server. I tried to install grep filter but it gives me some error so I try to implement below with grok. My original post is here . I found this but m quite unsatisfied.
Following is the configuration for logstash-forwarder file-name: logstash-forwarder in client-server
{
"network": {
"servers": [ "logstashserver-ip:5000" ],
"timeout": 15,
"ssl ca": "xxx.crt"
},
"files": [
{
"paths": [
"/var/log/messages"
],
"fields": { "type": "syslog" }
}
]
}
and following is the logstash configuration in logstashserver
file-name:input.conf
input {
lumberjack {
port => 5000
type => "logs"
ssl_certificate => "xxx.crt"
ssl_key => "xxx.key"
}
}
file-name:filter.conf
filter {
grok {
match => ["message", "\[%{WORD:messagetype}\]: %{GREEDYDATA}"]
}
}
file-name:output.conf
output {
elasticsearch { host => "logstashserver-ip" }
if [messagetype] == "ERROR" {
stdout {
codec => "rubydebug"
}
}
}
Is there anything wrong?
Not sure if you're still having this problem, but I'd look at dropping the messages you don't want. On my server, I get syslog severity levels which include syslog_severity_code as defined at http://en.wikipedia.org/wiki/Syslog#Severity_levels.
If you're getting them in your indices, try something like
filter {
if [type] == 'syslog' and [syslog_severity_code] > 5 {
drop { }
}
}