How to communicate logstash pipelines - logstash

I want in one instance there has to be two logstash pipelines running,one of them's output will be the other one's input.
I have read below documentations,
https://www.elastic.co/guide/en/logstash/current/ls-to-ls.html
https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#pipeline-to-pipeline-overview
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-lumberjack.html
I'm confused about which approach I should follow.
The thing I want is below :
The first logstash :
input {
# someplugins and codecs are here
}
filter {
# some operations here
}
output {
elasticsearch {
...
}
file {
...
}
logstash {
}
}
Second one like below :
input {
logstash {
}
}
filter {
#some operations are here
}
output {
elasticsearch {
}
}
I know there is no plugin that name is logstash. I put that name for explaining the situations.
So, for this purpose what should i follow ? Should i need message queue (kafka,redis) or lumberjack protocol or should i need beats for this purpose or is there any better alternative ?
Can someone answer with basic pipeline for this purpose ?
Thanks for answering

The below one is the easiest way that the solves question.
- pipeline.id: first_pipeline
config.string:
input { stdin { } }
output { pipeline { send_to => [commonOut] } }
- pipeline.id: second_pipeline
config.string: |
input { pipeline { address => commonOut } }
filter {
}
output { stdout { codec => rubydebug } }

Related

How to use mutate filter plugin in output

I have a logstash configuration. With that configuration logstash do some operation in filter and send outputs. Everything works well.
I want one more elasticsearch output in same configuration file. I mean after parsing the logs, logstash
send results to one index after that removing some fields and send them to another index. My aim is in below example.
output {
elasticsearch {
..index1..
}
mutate {
remove_field ...
}
mutate {
add_field ..
}
elasticsearch {
... index2 ...
}
}
But i can not work with mutate plugin in output ? How can i achieve this ?
Thanks for answering
mutate is a filter plugin, so it will only work inside the filter block.
What you can do maybe is use the clone filter to duplicate your event, apply different filters to the original and cloned event and then deliver the two copies to different outputs accordingly.
It could look roughly like this
filter {
clone {
clones => ["cloned"]
}
# apply here filters that should be applied to both the primary and secondary event
if [type] == "cloned" {
# apply here filters that should be applied to the secondary event
}
}
output {
if [type] == "cloned" {
elasticsearch {
... index2 ...
}
} else {
elasticsearch {
... index1 ...
}
}
}
See https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/5

How to check for an index with a date in logstash output

I have an output like so:
output {
if [target_index] == "mystream-%{+YYYY.MM.dd}"{
kinesis {
stream_name => "mystream"
region => "us-east-1"
}
}
I'd like to filter by the index pattern mystream-{date}. For some reason this conditional is not working. I'm not sure what the problem is here. Any help would be greatly appreciated.
Values compared in conditionals are not sprintf'd. You would have to use mutate to add a value that gets sprintf'd, and then compare to that. For example
filter { mutate { add_field => { "[#metadata][streamName]" => "mystream-%{+YYYY.MM.dd}" } } }
output {
if [target_index] == [#metadata][streamName] { ...

How to output logs to different Kafka topics under different keywords?

Here is my Logstash input log file:
{"timestamp":"2019-10-18T16:37:53.137068+0800","flow_id":1072393101531249,"in_iface":"eno1","event_type":"event A"}
{"timestamp":"2019-10-18T16:37:53.137069+0800","flow_id":1072393101531249,"in_iface":"eno1","event_type":"event B"}
Now I want to output these logs to different kafka topics(topic.A and topic.B) based on different event_type.
Here is my current Logstash configuration:
input {
file {
path => "/data/*.json"
}
}
filter {
}
output {
kafka {
codec => plain {
format => "%{message}"
}
topic_id => "topic.A"
...
}
}
How can I do what I want in logstash filter and output?

Not able to create csv output from LogStash

Maybe it is me, but how come that when I use the CSV Output from LogStash it does not output in a csv format? I am using nothing special (as seen in the configuration). Can someone tell me what I am doing wrong?
input
{
stdin {
type => "stdin-type"
}
}
filter
{
mutate { add_field => { "test" => "testme" } }
mutate { add_field => { "[#metadata][test]" => "Hello" } }
mutate { add_field => { "[#metadata][test2]" => "world" } }
}
output {
# .\bin\logstash-plugin.bat install logstash-output-csv
csv {
fields => ["test", "[#metadata][test]"]
path => "./TestLogs.csv"
}
stdout { codec => rubydebug { metadata => true } }
}
It actually create an output. If I type something (Ex.: test me) in the console (stdin) it creates the file and all. But the CSV file contains the following:
2016-11-25T11:49:40.338Z MyPcName test me
And I am expecting the following:
testme,Hello
Note: I am using LogStash 5 (latest version at the moment).
This is a Logstash 5.x issue. For now, I'm using the script below:
output {
file {
path => "/app/logstash/test.csv"
message_pattern => (grok pattern)
}

How to distinct grok filter for similar logs

I have this to kind of logs for dhcpack:
Jun 30 06:34:18 HOSTNAME dhcpd: DHCPACK to IP (MAC) via eth2
Jun 30 06:34:28 HOSTNAME dhcpd: DHCPACK on IP to MAC via eth2
How can I use grok, to use two different matches?
I have these two matches for dhcpack, but just use the first:
((%{SYSLOGTIMESTAMP:timestamp})\s*(%{HOSTNAME:hostname})\sdhcpd\S+\s(%{WORD:dhcp_action})?.[for|on]
(%{IPV4:dhcp_client_ip})?.[from|to]
(%{COMMONMAC:dhcp_client_mac})?.*via (%{USERNAME:interface}))
((%{SYSLOGTIMESTAMP:timestamp})\s*(%{HOSTNAME:hostname})\sdhcpd\S+\s(%{WORD:dhcp_action})?.*[to]
(%{IPV4:dhcp_client_ip})?.*via (%{USERNAME:interface}))
Someone can help?
I would suggest pulling the common stuff (up to the colon) off first and then processing the more specific stuff with more specific patterns. Some details here.
As shown in the doc, grok{} can take multiple patterns:
filter {
grok { match => { "message" => [
"Duration: %{NUMBER:duration}",
"Speed: %{NUMBER:speed}"
] } }
}
By default, it will stop processing after the first match, but that's configurable.
EDIT:
Based on your other comments, you can also branch based on conditionals:
if [myField] == "someValue" {
grok {
...
}
}
else {
grok {
...
}
}
In this case, you're running a comparison ("==") or regexp ("=~") to see if you should run a regexp (grok{}). Depending on the full business logic, this seems like a waste.
I solve the problem with this:
filter
{
grok
{
match => ["message", "(dhcpd\S+\s*(%{WORD:dhcp_action_test}))"]
}
if "DHCPINFORM" in [message]
{
grok
{
match => ["message","((%{SYSLOGTIMESTAMP:timestamp})\s* (%{HOSTNAME:hostname})\sdhcpd\S+\s(%{WORD:dhcp_action})?.[from] (%{IPV4:dhcp_client_ip}))"]
}
}
else if "DHCPDISCOVER" in [message]
{
grok
{
match => ["message","((%{SYSLOGTIMESTAMP:timestamp})\s(%{HOSTNAME:hostname})\sdhcpd\S+\s(%{WORD:dhcp_action})?.*[from] (%{COMMONMAC:dhcp_client_mac})"]
}
}
else
{
drop {}
}
}
I want do something like:
In ((%{SYSLOGTIMESTAMP:timestamp})\s*(%{HOSTNAME:hostname})\sdhcpd\S+\s(%{WORD:dhcp_action})?.[for|on] (%{IPV4:dhcp_client_ip})?.[from|to] (%{COMMONMAC:dhcp_client_mac})?.*via (%{USERNAME:interface}))
get just dhcp_action and use if statement, like:
if (mCursor != null && mCursor.moveToFirst()) {
......
} else {
......
}
It's possible?

Resources