I want use Filebeat with logstash.But the logstash.conf is wrong.
logstash.conf:
```
input {
beats {
port => "5044"
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
```
It reponse this:
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 3, column 1 (byte 76) after ", :backtrace=>["/opt/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/opt/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline.rb:171:in initialize'", "/opt/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:335:in block in converge_state'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:332:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:319:in converge_state'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/opt/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/opt/logstash/logstash-core/lib/logstash/runner.rb:343:inblock in execute'", "/opt/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
The beats plugin is wrong. The port should be a number.
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html
Also you have no filter plugin, maybe that will be necessary too:
input {
beats {
port => 5044
}
}
filter{}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
}
}
Related
Hi Im trying to set up a log analysis with Filebeat and Logstash.
Below are the changes i made in
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\elasticsearch-5.4.3\elasticsearch-5.4.3\logs\elasticsearch.log
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
And here is my logstash configuration file.
input {
beats {
port => 5044
}
}
filter {
grok {
match => { "message" => "%{plugins}" }
}
date {
match => [ "timestamp" , "yyyy-MM-DD:HH:mm:ss" ]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
}
While running the above, i see the below error:
[2019-10-22T06:07:32,915][ERROR][logstash.javapipeline ] Pipeline aborted due
to error {:pipeline_id=>"main", :exception=>#<Grok::PatternError: pattern %{plu
gins} not defined>, :backtrace=>["D:/logstash-7.1.0/logstash-7.1.0/vendor/bundle
/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:123:in `block in compile'", "
org/jruby/RubyKernel.java:1425:in `loop'", "D:/logstash-7.1.0/logstash-7.1.0/ven
dor/bundle/jruby/2.5.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:93:in `compile'", "
D:/logstash-7.1.0/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-
grok-4.0.4/lib/logstash/filters/grok.rb:281:in `block in register'", "org/jruby/
RubyArray.java:1792:in `each'", "D:/logstash-7.1.0/logstash-7.1.0/vendor/bundle/
jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in
`block in register'", "org/jruby/RubyHash.java:1419:in `each'", "D:/logstash-7.1
.0/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/logstash-filter-grok-4.0.4/lib/
logstash/filters/grok.rb:270:in `register'", "org/logstash/config/ir/compiler/Ab
stractFilterDelegatorExt.java:56:in `register'", "D:/logstash-7.1.0/logstash-7.1
.0/logstash-core/lib/logstash/java_pipeline.rb:191:in `block in register_plugins
'", "org/jruby/RubyArray.java:1792:in `each'", "D:/logstash-7.1.0/logstash-7.1.0
/logstash-core/lib/logstash/java_pipeline.rb:190:in `register_plugins'", "D:/log
stash-7.1.0/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:446:in `m
aybe_setup_out_plugins'", "D:/logstash-7.1.0/logstash-7.1.0/logstash-core/lib/lo
gstash/java_pipeline.rb:203:in `start_workers'", "D:/logstash-7.1.0/logstash-7.1
.0/logstash-core/lib/logstash/java_pipeline.rb:145:in `run'", "D:/logstash-7.1.0
/logstash-7.1.0/logstash-core/lib/logstash/java_pipeline.rb:104:in `block in sta
rt'"], :thread=>"#<Thread:0x15997940 run>"}
[2019-10-22T06:07:32,970][ERROR][logstash.agent ] Failed to execute ac
tion {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message
=>"Could not execute action: PipelineAction::Create<main>, action_result: false"
, :backtrace=>nil}
Im rather new to this intergration, im not sure of where i should look into.
Please help me.
The problem looks to be with
grok {
match => { "message" => "%{plugins}" }
}
What is %{plugins} here? It is NOT a pre-defined grok pattern. The list of grok patterns can be found here.
Also, the syntax for a grok pattern from documentation is %{SYNTAX:SEMANTIC}. You could do something like
grok {
match => { "message", "%{GREEDYDATA:plugins}" }
}
Try giving data type of the "%{plugins}".
filter {
grok {
match => { "message" => "%{WORD:plugins}" }
}
}
You can find data types from here
If this not working try removing date filter and try again.
Apparently these kind of errors can happen because of some regexp syntax error deep into a config file. That's just crack.
I am new to logstash.When I am trying to put a if statement in logstash config file it gives me error
if statement used is:
if {await} > 10
{ mutate {add_field => {"RULE_DATA" => "Value is above threshold"}
add_field => {"ACTUAL_DATA" => "%{await}"}
}
}
the error faced is given below:
[ERROR] 2018-07-20 16:52:21.327 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, => at line 18, column 10 (byte 729) after filter{\n grok {\n patterns_dir => [\"./patterns\"]\n match => { \"message\" => [\"%{TIME:time}%{SPACE}%{USERNAME:device}%{SPACE}%{USERNAME:tps}%{SPACE}%{SYSLOGPROG:rd_sec/s}%{SPACE}%{SYSLOGPROG:wr_sec/s}%{SPACE}%{SYSLOGPROG:avgrq-sz}%{SPACE}%{SYSLOGPROG:avgqu-sz}%{SPACE}%{NUMBER:await}%{SPACE}%{SYSLOGPROG:svctm}%{SPACE}%{SYSLOGPROG:%util}\"]\n }\n overwrite => [\"message\"]\n } \n if \"_grokparsefailure\" in [tags] {\n drop { }\n }\nif {await", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
Please suggest what has caused this error.
You have a syntax error. If you have a field as name it await. Like output of grok parse etc.
use the below
if [await] > 10
{
mutate {
add_field => {"RULE_DATA" => "Value is above threshold"}
add_field => {"ACTUAL_DATA" => "%{await}"}
}
}
Logstash conditional's expression enclosed in [] not {}, have a look at the following example from conditional documentation,
filter {
if [action] == "login" {
mutate { remove_field => "secret" }
}
}
I am a newbie in logstash and i just want to make a basic http get to a simple api and display result in console
my conf file named "api.conf" contain :
input {
http {
url => 'https://jsonplaceholder.typicode.com/albums'
}
}
output {
stdout { codec => rubydebug }
}
and i launch it from logstash folder i have just dowloaded and have not changed thanks a windos cmd command:
C:\Users\username\Desktop\logstash-6.2.2>.\bin\logstash.bat -f .\api.conf
it returning n error in console:
Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Something is
wrong with your configuration.",
:backtrace=>["C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/config/mixin.rb:89:in
config_init'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/inputs/base.rb:62:in
initialize'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/plugins/plugin_factory.rb:89:in
plugin'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:112:in
plugin'", "(eval):8:in <eval>'", "org/jruby/RubyKernel.java:994:in
eval'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:84:in
initialize'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/pipeline.rb:169:in
initialize'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/pipeline_action/create.rb:40:in
execute'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:315:in
block in converge_state'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in
with_pipelines'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:312:in
block in converge_state'", "org/jruby/RubyArray.java:1734:in
each'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:299:in
converge_state'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:166:in
block in converge_state_and_update'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:141:in
with_pipelines'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:164:in
converge_state_and_update'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/agent.rb:90:in
execute'",
"C:/Users/username/Desktop/logstash-6.2.2/logstash-core/lib/logstash/runner.rb:348:in
block in execute'",
"C:/Users/username/Desktop/logstash-6.2.2/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in
block in initialize'"]}
Do you know what I am dig wrong and how make it working?
For pulling web rest api, the correct input plug-in is http_poller
input {
http_poller {
urls => {
test1 => "https://jsonplaceholder.typicode.com/albums"
}
request_timeout => 60
# Supports "cron", "every", "at" and "in" schedules by rufus scheduler
schedule => { cron => "* * * * * UTC"}
codec => "json"
# A hash of request metadata info (timing, response headers, etc.) will be sent here
metadata_target => "http_poller_metadata"
}
}
output {
stdout {
codec => rubydebug
}
}
I got this exception in logstash log when I run it.
[2018-01-14T15:42:00,912]
[ERROR][logstash.outputs.elasticsearch]
Unknown setting 'host' for elasticsearch
[2018-01-14T15:42:00,921][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/
pipeline_id:main, :exception=>"LogStash::ConfigurationError",
:message=>"Something is wrong with your configuration.",
:backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config
/mixin.rb:89:in config_init
"/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:63:in
initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:3:in
initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:25:in
initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/plugins/plugin_factory.rb:86:in
plugin'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:114:in
plugin'", "(eval):87:in <eval>'","org/jruby/RubyKernel.java:994:in
eval'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:86:in
initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:171:in
initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in
execute'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:335:inblock
in converge_state'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in
with_pipelines'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:332:inblock
in converge_state'", "org/jruby/RubyArray.java:1734:in each'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:319:in
converge_state'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in block
in converge_state_and_update'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in
with_pipelines'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in
converge_state_and_update'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in
execute'",
"/usr/share/logstash/logstash-core/lib/logstash/runner.rb:343:in
block in execute'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in
block in initialize'"]}
It is my configure:
input{
lumberjack {
port => 5044
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}
filter{
if[type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:sysylog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => ["received_at", "%{#timestamp}" ]
add_field => ["received_from", "%{host}" ]
}
syslog_pri {}
date {
match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output{
elasticsearch { host =>localhost }
stdout { codec => rubydebug }
}
How can I solve it . thank you.
I use latest version of ELK
If you check your output elasticsearch plugin, it has host parameter.
It needs a hosts parameter and a string array.
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-hosts
My logstash->elastic plugin looks like this:
elasticsearch{
hosts=>["localhost:9200"]
index=>"logstash-%{+YYYY.MM.dd}"
}
You might need the index parameter set too.
I want to send a +8Go csv file to my ES server form my machine.
I use Logstash to send the file with this conf :
input {
file {
path => "/Users/karnag/Downloads/siren201703.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
#Date,Open,High,Low,Close,Volume (BTC),Volume (Currency),Weighted Price
columns => ["SIREN", "NIC", "L1_NORMALISEE", "L2_NORMALISEE", "L3_NORMALISEE", "L4_NORMALISEE", "L5_NORMALISEE", "L6_NORMALISEE", "L7_NORMALISEE", "L1_DECLAREE", "L2_DECLAREE", "L3_DECLAREE", "L4_DECLAREE", "L5_DECLAREE", "L6_DECLAREE", "L7_DECLAREE", "NUMVOIE", "INDREP", "TYPVOIE", "LIBVOIE", "CODPOS", "CEDEX", "RPET", "LIBREG", "DEPET", "ARRONET", "CTONET", "COMET", "LIBCOM", "DU", "TU", "UU", "EPCI", "TCD", "ZEMET", "SIEGE", "ENSEIGNE", "IND_PUBLIPO", "DIFFCOM", "AMINTRET", "NATETAB", "LIBNATETAB", "APET700", "LIBAPET", "DAPET", "TEFET", "LIBTEFET", "EFETCENT", "DEFET", "ORIGINE", "DCRET", "DDEBACT", "ACTIVNAT", "LIEUACT", "ACTISURF", "SAISONAT", "MODET", "PRODET", "PRODPART", "AUXILT", "NOMEN_LONG", "SIGLE", "NOM", "PRENOM", "CIVILITE", "RNA", "NICSIEGE", "RPEN", "DEPCOMEN", "ADR_MAIL", "NJ", "LIBNJ", "APEN700", "LIBAPEN", "DAPEN", "APRM", "ESS", "DATEESS", "TEFEN", "LIBTEFEN", "EFENCENT", "DEFEN", "CATEGORIE", "DCREN", "AMINTREN", "MONOACT", "MODEN", "PRODEN", "ESAANN", "TCA", "ESAAPEN", "ESASEC1N", "ESASEC2N", "ESASEC3N", "ESASEC4N", "VMAJ", "VMAJ1", "VMAJ2", "VMAJ3", "DATEMAJ"]
}
}
output {
elasticsearch {
hosts => "http://192.168.10.19:8080/"
index => "siren"
}
stdout {}
}
And I got this error:
[2017-03-15T10:23:04,628][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "" hasn't been registered>, :backtrace=>["/Users/karnag/Documents/Epitech/ElasticStack/Logstash/logstash-core/lib/logstash/settings.rb:29:in `get_setting'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/logstash-core/lib/logstash/settings.rb:61:in `set_value'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/logstash-core/lib/logstash/settings.rb:80:in `merge'", "org/jruby/RubyHash.java:1342:in `each'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/logstash-core/lib/logstash/settings.rb:80:in `merge'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/logstash-core/lib/logstash/settings.rb:115:in `validate_all'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/logstash-core/lib/logstash/runner.rb:210:in `execute'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/logstash-core/lib/logstash/runner.rb:183:in `run'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/Users/karnag/Documents/Epitech/ElasticStack/Logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}
I can't find where is the typo in my conf file (clearly there is something wrong here).
Thanks.