I'm having issues sending logs from a file to a destination using logstash - logstash

I'm very new to logstash configuration. Below is my logstash pipeline configuration that I'm using to send a log file to Coralogix without success.
input {
file {
path => "/etc/logstash/conf.d/CSE-Candidate-Exercise-LOG.log"
type => "log" # a type to identify those logs (will need this later)
start_position => "beginning"
sincedb_path => "NUL"
}
}
output {
http {
url => "https://api.coralogix.com/logs/rest/singles "
http_method => "post"
headers => ["private_key", "xxxxxxxx-cccc-7777-ffff-000000000000"]
format => "json_batch"
codec => "line"
mapping => {
"applicationName" => "%{[#metadata][application]}"
"subsystemName" => "%{[#metadata][subsystem]}"
"computerName" => "%{host}"
"text" => "%{[#metadata][event]}"
}
http_compression => true
automatic_retries => 5
retry_non_idempotent => true
connect_timeout => 30
keepalive => false
}
}
The input should be ok but I think my issue is with the output. I'm not sure what "codec" plugin to use, do I need the "format" field?
Each time I run logstash to send the log in the file specified in the path, all I get is the below log snippet:
[INFO ] 2022-08-02 13:25:02.659 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2022-08-02 13:25:03.588 [Converge PipelineAction::Create<main>] Reflections - Reflections took 135 ms to scan 1 urls, producing 124 keys and 408 values
[INFO ] 2022-08-02 13:25:04.158 [Converge PipelineAction::Create<main>] javapipeline - Pipeline `main` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[INFO ] 2022-08-02 13:25:04.278 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash-config.conf"], :thread=>"#<Thread:0x5a5e32c1 run>"}
[INFO ] 2022-08-02 13:25:05.154 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.87}
[INFO ] 2022-08-02 13:25:05.235 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-08-02 13:25:05.316 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2022-08-02 13:25:05.344 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
I see no log in Coralogix platform. I simply want to send the raw log in the file without any filtering or manipulations to the destination. Any help would be appreciated.

Related

How can logstash be executed? (error occured)

How can logstash be executed? (error occured)
run logstash on AWS Linux.
but error occurred while executing.
my Linux(Ubuntu Version: 20.04)
elasticsearch installed.(execute successfully)
kibana installed.(execute successfully)
logstash installed. (execute error occured)
my .conf file code
input {
jdbc {
clean_run => true
jdbc_driver_library => "/usr/share/java/mysql-connector-java-8.0.23.jar"
jdbc_driver_class => "com.mysql.jdbc.Driver"
jdbc_connection_string => "jdbc:mysql://AWSLINK:3306/schema_name?useSSL=false&user=root&password=1234"
jdbc_user => "root"
jdbc_password => "1234"
schedule => "* * * * *"
statement => "select * from schema_name"
}
}
output {
elasticsearch {
hosts => 52.188.20.167:9200"
index => "AWS_DB_0514"
}
stdout {
codec => rubydebug
}
}
I execute logstash in linux(command)
./logstash -f test.conf --path.settings /etc/logstash/
I execute above code but error occured.(attempt)
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2021-05-14T08:37:16,025][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2021-05-14T08:37:16,039][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.12.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.10+9 on 11.0.10+9 +indy +jit [linux-x86_64]"}
[2021-05-14T08:37:16,466][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2021-05-14T08:37:17,524][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-05-14T08:37:18,048][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [0-9], [ \\t\\r\\n], \"#\", \"}\" at line 16, column 24 (byte 608) after output {\n elasticsearch {\n hosts => 52.188", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}
[2021-05-14T08:37:18,165][INFO ][logstash.runner ] Logstash shut down.
[2021-05-14T08:37:18,177][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.13.0.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.13.0.jar:?]
at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]
There is an error on line 16.
incorrect code
hosts => 52.188.20.167:9200"
correct code
hosts => "52.188.20.167:9200"

Grokparsefailure and type problems in logstash configuration file

I have several problems with my configuration file. My goal is to parse three types of logs (for the moment). Here they are :
[29/05/2020 07:41:51.354] - ih912865 - 10.107.119.121 - 93 - Transaction 7635 COMPLETED 318 ms wait time 3183 ms
[29/05/2020 10:30:01.318] - Process status database sync - us1salx08167.corpnet2.com:8400(#52279) (load 0 grace period 5 minutes) : current date 2020/02/02 21:30:01 update date 2020/02/02 21:29:58 old state OK new state OK
31730 31626 464 10980020 52:25 /plw/modules/bin/Lx86_64/opx2-intranet.exe -I /plw/modules/bin/Lx86_64/opx2-intranet.dxl -H /plw/modules/bin/Lx86_64 -L /plw/PLW_PROD/modules/preload-intranet.ini -- plw-sysconsole -port 8400 -logdir /plw/PLW_PROD/httpdocs/admin/log/ -slaves 2
Two of these logs can be in slave files named intranet-2020-06-25-8401.log or intranet-2020-06-25-8400.log the last one is in a master file named intranet-2020-06-25-8402.log
For my tests I simplified the architecture of my log files, so I have a Log-test folder in which I put a slave file and a master file.
In these files I only put the corresponding logs and a different log to be able to see how to manage this case.
Here is the content of a "slave" :
[29/05/2020 07:41:51.354] - ih912865 - 10.107.199.125 - 93 - Transaction 7635 COMPLETED 318 ms wait time 3183 ms
[29/05/2020 10:30:01.318] - Process status database sync - us1salx08167.corpnet2.com:8400(#52279) (load 0 grace period 5 minutes) : current date 2020/02/02 21:30:01 update date 2020/02/02 21:29:58 old state OK new state OK
[29/05/2020 13:49:20.635] - Main process - Transaction SYSTEM 105238-12 SQL done 1 ms
Here is the content of a "master" :
31730 31626 464 10980020 52:25 /plw/modules/bin/Lx86_64/opx2-intranet.exe -I /plw/modules/bin/Lx86_64/opx2-intranet.dxl -H /plw/modules/bin/Lx86_64 -L /plw/PLW_PROD/modules/preload-intranet.ini -- plw-sysconsole -port 8400 -logdir /plw/PLW_PROD/httpdocs/admin/log/ -slaves 2
[26/06/2020 21:38:01.386] - Main process - Starting HTTP service on port 8402 (socket #<MULTIVALENT stream socket waiting for connection at */8402 # #x1022d2ddbb2>)
Now that you have a better understanding of my environment and my purpose, here's the problem. When I launch my logstash configuration, I retrieve my data in kibana. But kibana shows me that each log has been treated as coming from a slave file while I also have a log coming from a master file which doesn't have the same processing.
For a better understanding here is my configuration file :
input {
file {
path => "/home/mathis/Documents/**/intranet*.log"
exclude =>"*8402.log"
sincedb_path => '/dev/null'
start_position => beginning
type => "slave"
}
file {
path => "/home/mathis/Documents/**/intranet*8402.log"
sincedb_path => '/dev/null'
type => "master"
}
}
filter {
if [type] == "slave" {
grok {
match => { "message" => ["\[%{DATESTAMP:eventtime}\] \- %{USERNAME:user} \- %{IPV4:clientip} \- %{NUMBER} \- %{WORD} %{NUMBER:exectime} %{WORD} %{NUMBER:time} %{GREEDYDATA:data} %{NUMBER:waittime}","\[%{DATESTAMP:eventtime}\] \- Process status database sync \- %{WORD}\.%{WORD}\.%{WORD}\:%{NUMBER:slavenumb}\(\#%{NUMBER}\) \(load %{NUMBER:nbutilisateur} grace period 5 minutes\) %{GREEDYDATA}"] }
remove_field => "message"
}
date {
match => [ "eventtime", "dd/MM/YYYY HH:mm:ss.SSS" ]
target => "#timestamp"
}
}
if [type] == "master" {
grok {
match => {"message" => ["%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}%{NUMBER}%{SPACE}(?<starttime>((?!<[0-9])%{HOUR}:)?%{MINUTE}(?::%{SECOND})(?![0-9]))"]}
remove_field => "message"
}
date {
match => [ "starttime", "HH:mm:ss","mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => "127.0.0.1:9200"
index => "logstash-local3-%{+YYYY.MM.dd}"
}
}
And now this is what kibana shows me:
As you can see, the type field is slave for all logs but we can also observe that the logs of the slave file "intranet-2020-06-25-8401.log" are correctly parsed and that the line of added log that does not interest me has the field tags _grokparsefailure (the middle line in the picture).
The other problem is that the other logs (the first two lines on the image) are from a slave file (which is not true) according to kibana, so I guess they are processed in my first grok which would explain why they also have the _grokparsefailure tags field.
So I guess there are several errors in my input and filter part. I've been searching for a long time and doing a lot of testing, could you help me fix my config file please?

Logstash: Nothing displayed on console (Mac)

I am trying to set up a very simple logstash config
input {
file {
path => "/path/to/my/log/file"
start_position => "beginning"
ignore_older => 0
}
}
filter {
}
output {
stdout {
codec => rubydebug
}
}
and here is how i start logstash
[logstash-7.1.1]$ bin/logstash -r -f log.conf
but here is all i see on the console
Sending Logstash logs to path/to/logstash-7.1.1/logs which is now configured via log4j2.properties
[2019-05-28T13:22:57,294][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-05-28T13:22:57,313][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2019-05-28T13:23:02,904][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x7ad3cf30 run>"}
[2019-05-28T13:23:03,254][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"path/to/logstash-7.1.1/data/plugins/inputs/file/.sincedb_8164b23a475b43f1b0c9aba125f7f5cf", :path=>["/path/to/my/log/file"]}
[2019-05-28T13:23:03,284][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"main"}
[2019-05-28T13:23:03,355][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-05-28T13:23:03,360][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-05-28T13:23:03,703][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
i can see that
No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"path/to/logstash-7.1.1/data/plugins/inputs/file/.sincedb_8164b23a475b43f1b0c9aba125f7f5cf", :path=>["/path/to/my/log/file"]}
so the path seems correct. Also, my log file is not empty.
What am i doing wrong? Why cant I see the content of my log file on the console?
input {
file {
path => "/salaries.csv"
start_position => "beginning"
type => "data"
}
}
filter {
csv{
separator => ","
}
}
output {
stdout {
codec => rubydebug
}
}
This link may helpful to you

Parsing json using logstash (ELK stack)

I have created a simple json like below
[
{
"Name": "vishnu",
"ID": 1
},
{
"Name": "vishnu",
"ID": 1
}
]
I am holding this values in file named simple.txt . Then i used file beat to listen the file and send the new updates to port 5043,on other side i started the log-stash service which listen to this port in order to parse and pass the json to elastic search.
log-stash is not processing the json values,it hangs in the middle.
logstash
input {
beats {
port => 5043
host => "0.0.0.0"
client_inactivity_timeout => 3600
}
}
filter {
json {
source => "message"
}
}
output {
stdout { codec => rubydebug }
}
filebeat config:
filebeat.prospectors:
- input_type: log
paths:
- filepath
output.logstash:
hosts: ["localhost:5043"]
Logstash output
**
Sending Logstash's logs to D:/elasticdb/logstash-5.6.3/logstash-5.6.3/logs which is now configured via log4j2.properties
[2017-10-31T19:01:17,574][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"D:/elasticdb/logstash-5.6.3/logstash-5.6.3/modules/fb_apache/configuration"}
[2017-10-31T19:01:17,578][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"D:/elasticdb/logstash-5.6.3/logstash-5.6.3/modules/netflow/configuration"}
[2017-10-31T19:01:18,301][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-10-31T19:01:18,388][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2017-10-31T19:01:18,573][INFO ][logstash.pipeline ] Pipeline main started
[2017-10-31T19:01:18,591][INFO ][org.logstash.beats.Server] Starting server on port: 5043
[2017-10-31T19:01:18,697][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
**
Every time when i am running log-stash using command
logstash -f logstash.conf
And since there is no processing of json i am stopping that service by pressing ctrl + c .
Please help me in finding the solution.Thanks in advance.
finally i got ended up with config like this.It works for me.
input
{
file
{
codec => multiline
{
pattern => '^\{'
negate => true
what => previous
}
path => "D:\elasticdb\logstash-tutorial.log\Test.txt"
start_position => "beginning"
sincedb_path => "D:\elasticdb\logstash-tutorial.log\null"
exclude => "*.gz"
}
}
filter {
json {
source => "message"
remove_field => ["path","#timestamp","#version","host","message"]
}
}
output {
elasticsearch { hosts => ["localhost"]
index => "logs"
"document_type" => "json_from_logstash_attempt3"
}
stdout{}
}
Json format:
{"name":"sachin","ID":"1","TS":1351146569}
{"name":"sachin","ID":"1","TS":1351146569}
{"name":"sachin","ID":"1","TS":1351146569}

Get JSON from file

Logstash 5.2.1
I can't read JSON documents from a local file using Logstash. There are no documents in the stdout.
I run Logstash like this:
./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --config.reload.automatic
Logstash config:
input {
file {
path => "/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"
codec => json {}
start_position => "beginning"
}
}
output {
stdout {
codec => rubydebug
}
}
Also, I tried with charset:
...
codec => json {
charset => "UTF-8"
}
...
Also, I tried with/without json codec in the input and with filter:
...
filter {
json {
source => "message"
}
}
...
Logstash console after start:
[2017-02-28T11:37:29,947][WARN ][logstash.agent ] fetched new config for pipeline. upgrading.. {:pipeline=>"main", :config=>"input {\n file {\n path => \"/home/trex/Development/Shipping_Data_To_ES/shakespeare.json\"\n codec => json {\n charset => \"UTF-8\"\n }\n start_position => \"beginning\"\n }\n}\n#filter {\n# json {\n# source => \"message\"\n# }\n#}\noutput {\n stdout {\n codec => rubydebug\n }\n}\n\n"}
[2017-02-28T11:37:29,951][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-02-28T11:37:30,434][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-02-28T11:37:30,446][INFO ][logstash.pipeline ] Pipeline main started
^C[2017-02-28T11:40:55,039][WARN ][logstash.runner ] SIGINT received. Shutting down the agent.
[2017-02-28T11:40:55,049][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
^C[2017-02-28T11:40:55,475][FATAL][logstash.runner ] SIGINT received. Terminating immediately..
The signal INT is in use by the JVM and will not work correctly on this platform
[trex#Latitude-E5510 Shipping_Data_To_ES]$ ./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --config.test_and_exit
^C[trex#Latitude-E5510 Shipping_Data_To_ES]$ ./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --confireload.automatic
^C[trex#Latitude-E5510 Shipping_Data_To_ES]$ ./logstash-5.2.1/bin/logstash -f logstash-5.2.1/config/shakespeare.conf --config.reload.aumatic
Sending Logstash's logs to /home/trex/Development/Shipping_Data_To_ES/logstash-5.2.1/logs which is now configured via log4j2.properties
[2017-02-28T11:45:48,752][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
[2017-02-28T11:45:48,785][INFO ][logstash.pipeline ] Pipeline main started
[2017-02-28T11:45:48,875][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Why Logstash doesn't put my JSON documents in stdout?
Did you try including the file type within your file input:
input {
file {
path => "/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"
type => "json" <-- add this
//codec => json {} <-- for the moment i'll comment this
start_position => "beginning"
}
}
And then have your filter as such:
filter{
json{
source => "message"
}
}
OR if you're going with the codec plugin make sure to have the synopsis as such within your input:
codec => "json"
OR you might want to try out json_lines plugin as well. Hope this thread comes in handy.
It appears that sincedb_path is important to read JSON files. I was able to import the JSON only after adding this option. It is needed to maintain the current position in the file to be able to resume from that position in case the import is interrupted. I don't need any position tracking, so I just set this to /dev/null and it works.
The basic working Logstash configuration:
input {
file {
path => ["/home/trex/Development/Shipping_Data_To_ES/shakespeare.json"]
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output {
stdout {
codec => json_lines
}
elasticsearch {
hosts => ["localhost:9200"]
index => "shakespeare"
}
}

Resources