Im running logstash inside kubernetes, i have another pods is generating random logs constantly. Grafana is able to detect the loki driver and labels, but the namespace, pod, containers name is not showing in grafana.
if u see the grafana only host, path and type are listed, it is not picking like containers, pods, namespaces.
logstash conf
input {
file {
id => "varlog"
path => "/var/log/containers/"
type => "var log"
start_position => "beginning"
}
}
filter {
if [kubernetes] {
mutate {
add_field => {
"container_name" => "%{[kubernetes][container][name]}"
"namespace" => "%{[kubernetes][namespace]}"
"pod" => "%{[kubernetes][pod][name]}"
}
replace => { "host" => "%{[kubernetes][node][name]}"}
}
}
mutate {
remove_field => ["tags"]
}
}
output {
stdout { codec => rubydebug}
loki {
url => "http://loki-loki-distributed-distributor.loki-benchmark.svc.cluster.local:3100/loki/api/v1/push"
}
}
Related
I am trying to config a filebeat with logstash. At the moment I managed to successfully config filebeat with logstash and I am running into same issues when creating multiple conf files in the logstash.
So currently I have one filebeats input which is something like :
input {
beats {
port => 5044
}
}
filter {
}
output {
if [#metadata][pipeline] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "systemsyslogs"
pipeline => "%{[#metadata][pipeline]}"
}}
else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "systemsyslogs"
}
}}
And a file Logstash config which is like :
input {
file {
path => "/var/log/foldername/number.log"
start_position => "beginning"
}
}
filter {
grok {
match => { "message" => "%{WORD:username} %{INT:number} %{TIMESTAMP_ISO8601:timestamp}" }
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "numberlogtest"
}
}
The grok filter is working as I successfully managed to create 2 index patterns in kibana and view the data correctly.
The problem is that when I am running logstash with both configs applied, logstash is fetching the data from number.log multiple times and logstash plain logs are getting lots of warning, therefore using a lot of computing resources and CPU is going over 80% ( this is an oracle instance ). If I remove the file config from logstash the system is running properly.
I managed to run logstash with each one of these config files applied individually, but not both at once.
I already added an exception in the filebeats config :
exclude_files:
- /var/log/foldername/*.log
Logstash plain logs when running both config files:
[2023-02-15T12:42:41,077][WARN ][logstash.outputs.elasticsearch][main][39aca10fa204f31879ff2b20d5b917784a083f91c2eda205baefa6e05c748820] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"numberlogtest", :routing=>nil}, {"service"=>{"type"=>"system"}
"caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:607"}}}}}
I already added an exception in the filebeat config :
exclude_files:
- /var/log/foldername/*.log
Fixed by creating a single logstash config with both inputs :
input {
beats {
port => 5044
}
file {
path => "**path**"
start_position => "beginning"
}
}
filter {
if [path] == "**path**" {
grok {
match => { "message" => "%{WORD:username} %{INT:number} %{TIMESTAMP_ISO8601:timestamp}" }
}
}
}
output {
if [#metadata][pipeline] {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "index1"
pipeline => "%{[#metadata][pipeline]}"
}
} else {
if [path] == "**path**" {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "index2"
}
} else {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "index1"
}
}
}
}
I'm using following configuration file for Logstash to create multiple indices, but they are not visible in Kibana. Logs are parsed, but index is not created. What do I need to change for this to work?
input {
stdin{
type => "stdin-type"
}
file{
tags => ["prod"]
type => ["json"]
path => ["C:/Users/DELL/Downloads/log/prod/*.log"]
}
file{
tags => ["dev"]
type => ["json"]
path => ["C:/Users/DELL/Downloads/log/test/*.log"]
}
}
output {
stdout {
codec => rubydebug
}
if "prod" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => ["prod-log"]
}
}
if "dev" in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => ["dev-log"]
}
}
}
I have a syslog server and the ELK stack on the same server. I have a directory for each syslog source.
I'm trying to parse syslog files with Logstash, and I'd like to keep the ip adress or the hostname of the syslog source in the "host" field. At the moment I have the 0.0.0.0 source after Logstash parsing.
My logstash.conf :
input {
file {
path => ["path/to/file.log"]
start_position => "beginning"
type => "linux-syslog"
ignore_older => 0
}
}
filter {
if [type] == "linux-syslog" {
grok {
match => {"message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
}
}
}
output {
elasticsearch {
hosts => ["#IP_Elastic:Port_Elastic"]
}
stdout { codec => rubydebug }
}
you can overwrite your host with your ip variable once you have parsed it. Consider this example:
Pipeline main started
{"ip":"1.2.3.4"}
{
"message" => "{\"ip\":\"1.2.3.4\"}",
"#version" => "1",
"#timestamp" => "2016-08-10T13:36:18.875Z",
"host" => "pandaadb",
"ip" => "1.2.3.4",
"#host" => "1.2.3.4"
}
I am parsing json to get the IP. Then I write the IP field into the host.
The filter:
filter {
# this parses the ip json
json {
source => "message"
}
mutate {
add_field => { "#host" => "%{ip}" }
}
}
replace %{ip} with whatever field contains your ip address.
Cheers,
Artur
I'm trying a "simple" logstash configuration and want to ouput on a file to check. So I took the conf from https://www.elastic.co/guide/en/logstash/current/plugins-outputs-file.html and put it in my conf:
input {
file {
exclude => ['*.gz']
path => ['/var/log/*.log']
type => 'system logs'
}
syslog {
port => 5000
}
}
output {
elasticsearch {
hosts => ['elasticsearch']
}
file {
path => "/config/logstash_out.log"
codec => {
line {
format => "message: %{message}"
}
}
}
stdout {}
}
but when I launch it (sudo docker run -it --rm --name logstash -p 514:5000 --link elasticsearch:elasticsearch -v "$PWD":/config logstash logstash -f /config/logstash.conf), I've got a complaint from logstash:
fetched an invalid config
{:config=>"input {
file {
exclude => ['*.gz']
path => ['/var/log/*.log']
type => 'system logs'
}
syslog {
port => 5000
}
}
output {
elasticsearch {
hosts => ['elasticsearch']
}
file {
path => \"/config/logstash_out.log\"
codec => {
line {
format => \"message: %{message}\"
}
}
}
stdout {}
}"
, :reason=>"Expected one of #, => at line 20, column 13 (byte 507)
after output { elasticsearch {\n hosts => ['elasticsearch']\n }
\n\n file {\n path => \"/config/logstash_out.log\"\n
codec => { \n line ", :level=>:error}
(I've reformatted a bit so it's more readable)
Any ideas why? I'seen logstash output to file and ignores codec but the proposed solution is marked as DEPRECATED so I would like to avoid
Thanks!
You have the wrong format just like the tutorial.
Here is the pull request.
It isn't
codec => {
line {
format => \"message: %{message}\"
}
}
but it is
codec =>
line {
format => "message: %{message}"
}
You don't need to add quirly brackets around line.
Here is your config correctly.
input {
file {
exclude => ['*.gz']
path => ['/var/log/*.log']
type => 'system logs'
}
syslog {
port => 5000
}
}
output {
elasticsearch {
hosts => ['elasticsearch']
}
file {
path => "/config/logstash_out.log"
codec =>
line {
format => "message: %{message}"
}
}
stdout {}
}
We are storing our data types in each of its database in couchdb. What sort of format will the config file have to import data from multiple databases? Or do I need to have multiple config files for importing data from each database to an index. Will appreciate any help.
Thanks.
We use a single config file for multiple databases.
It's not perfect, but functional for now.
Currently looks like:
input {
couchdb_changes {
sequence_path => "db1.seq"
db => "db1"
host => "xxx.xxx.xxx.xxx"
username => "xxx"
password => "xxx"
add_field => {
"organization" => "db1"
}
}
couchdb_changes {
sequence_path => "db2.seq"
db => "db2"
host => "xxx.xxx.xxx.xxx"
username => "xxx"
password => "xxx"
add_field => {
"organization" => "db2"
}
}
}
filter {
mutate {
remove_field => [ "_attachments" ]
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
cluster => "cluster0"
host => ["xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx", "xxx.xxx.xxx.xxx"]
protocol => "http"
index => "%{[organization]}"
document_id => "%{[#metadata][_id]}"
}
}