at logstash conf filter , I assigned multiple email id to variable(variable name = targetmailid) and output field , I am trying to use that variable. However, it is assiging multiple email id's to CC field in output. Please suggest.
filter {
kv {
field_split => ","
value_split=>":"
}
if [ref_type] =~ /tag/ {
ruby {
code => "tag= event['ref']
targetmailid = 'testuser1#mail.com,testuser2#mail.com,testuser3#mail.com'
}
}
}
output {
if "tagcreate" in [tags] {
email {
body => "test messgage"
from => "admin#emil.com"
to => "admin2#email.com"
cc => "targetmailid"
subject => "test mail"
options => {
smtpIporHost => "smtp"
port => 25
}
}
}
You need to use the sprintf format %{...} like this:
email {
body => "test messgage"
from => "admin#emil.com"
to => "admin2#email.com"
cc => "%{targetmailid}" <--- modify this
subject => "test mail"
options => {
smtpIporHost => "smtp"
port => 25
}
}
UPDATE
Also make sure to modify the following part:
if [ref_type] =~ /tag/ {
ruby {
code => "event['targetmailid'] = 'testuser1#mail.com,testuser2#mail.com,testuser3#mail.com'"
}
} else {
mutate {
add_field => { "targetmailid" => ""}
}
}
Related
I am using Logstash to update by query existing Elasticsearch documents with an additional field that contains aggregate values extracted from Potgresql table.
I use elastichsearch output to load one index using document_id and http output to update another index that have different document_id but receving errors:
[2023-02-08T17:58:12,086][ERROR][logstash.outputs.http ][main][b64f19821b11ee0df1bd165920785876cd6c5fab079e27d39bb7ee19a3d642a4] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/medico/_update_by_query", :event=>#LogStash::Event:0x19a14c08}
This is my pipeline configuration:
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "jdbc:postgresql://handel:5432/mydb"
statement_filepath => "D:\ProgrammiUnsupported\logstash-7.15.2\config\nota_sede.sql"
}
}
filter {
aggregate {
task_id => "%{idCso}"
code => "
map['idCso'] = event.get('idCso')
map['noteSede'] ||= []
map['noteSede'] << {
'id' => event.get('idNota'),
'tipo' => event.get('tipoNota'),
'descrizione' => event.get('descrizione'),
'data' => event.get('data'),
'dataInizio' => event.get('dataInizio'),
'dataFine' => event.get('dataFine')
}
event.cancel()"
push_previous_map_as_event => true
timeout => 60
timeout_tags => ['_aggregatetimeout']
}
}
}
output {
stdout { codec => rubydebug { metadata => true } }
# this works
elasticsearch {
hosts => "https://localhost:9200"
document_id => "STRUTTURA_%{idCso}"
index => "struttura"
action => "update"
user => "user"
password => "password"
ssl => true
cacert => "/usr/share/logstash/config/ca.crt"
}
http {
url => "http://localhost:9200/medico/_update_by_query"
user => "elastic"
password => "changeme"
http_method => "post"
format => "message"
content_type => "application/json"
message => '{
"query":{
"term":{
"idCso":"%{idCso}"
}
},
"script":{
"source":"ctx._source.noteSede=params.noteSede",
"lang":"painless",
"params":{
"noteSede":"%{noteSede}"
}
}
}
}'
}
}
The stdout output show me the sended docs to output like this:
{
"query" => {
"term" => {
"idCso" => "859119"
}
},
"script" => {
"source" => "ctx._source.noteSede=params.noteSede",
"lang" => "painless",
"params" => {
"noteSede" => "{dataFine=null, dataInizio=2020-02-13, descrizione=?, tipo=DB, id=6390644, data=2020-02-13 12:26:58.409},{dataFine=null, dataInizio=2020-02-13, descrizione=?, tipo=DE, id=6390645, data=2020-02-13 12:26:58.41}"
}
}
}
}
How could I set noteSede array field into message to _update_by_query ?
I am new to Logstash manipulations and I have no idea how to do the below.
I have a sample data as below:
Column:Type
Incident Response P3
Incident Resolution L1.5 P2
...
I want to extract the word 'Response' and 'Resolution' into a new column 'SLA type'
Im looking for something very alike to the below SQL statement:
case when Type like '%Resolution%' then Resolution
when Type like '%Response%' then Response
end as SLA_Type
How do i manipulate this in Logstash?
Below is my conf. I'm using an API input.
input {
http_poller {
urls => {
snowinc => {
url => "https://service-now.com"
user => "your_user"
password => "yourpassword"
headers => {Accept => "application/json"}
}
}
request_timeout => 60
metadata_target => "http_poller_metadata"
schedule => { cron => "* * * * * UTC"}
codec => "json"
}
}
filter
{
json {source => "result" }
split{ field => ["result"] }
date {
match => ["[result][sys_created_on]","yyyy-MM-dd HH:mm:ss"]
target => "sys_created_on"
}
}
output {
elasticsearch {
hosts => ["yourelastuicIP"]
index => "incidentsnow"
action=>update
document_id => "%{[result][number]}"
doc_as_upsert =>true
}
stdout { codec => rubydebug }
}
The output for the API json url looks like the below:
{"result":[
{
"made_sla":"true",
"Type":"incident resolution p3",
"sys_updated_on":"2019-12-23 05:00:00",
"number":"INC0010275",
"category":"Network"} ,
{
"made_sla":"true",
"Type":"incident resolution l1.5 p4",
"sys_updated_on":"2019-12-24 07:00:00",
"number":"INC0010567",
"category":"DB"}]}
You can use the following filter block in your pipeline to add a new field if a word is present in another field.
if "response" in [Type] {
mutate {
add_field => { "SLA_Type" => "Response" }
}
}
if "resolution" in [Type] {
mutate {
add_field => { "SLA_Type" => "Resolution" }
}
}
If the word response is present in the field Type a new field named SLA_Type with the value Response will be added to your document, the same in will happen with resolution.
I have fields like A_Latitude, A_Longitude, B_Latitude and B_Longitude. I would like to make use of this data and create Maps in Kibana. The problem is data is getting into elasticsearch, but the gejson columns created in Logstash filter not gettin recognized and data is not being fed into geo_point1 and geo_point2.
Hence, first created a geo_point mapping in Kibana dev tools as follows,
PUT cc-test
{
"mappings": {
"properties": {
"geo_point1":{
"type": "geo_point"
},
"geo_point2":{
"type": "geo_point"
}
}
}
}
I have configured my logstash config file the following way,
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "some string"
# The user we wish to execute our statement as
jdbc_user => "User"
jdbc_password => "Password"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/apps/ELK/logstash/driver/ngdbc-2.4.56.jar"
jdbc_driver_class => "com.sap.db.jdbc.Driver"
# our query
#jdbc_validate_connection => true
#schedule => "* * * * *"
#record_last_run => true
# last_run_metadata_path => "login.txt"
statement => "SELECT
inputdata.A_LATITUDE, inpudata.A_LONGITUDE, inputdata.B_LATITUDE,
inputdata.B_LONGITUDE, outputdata.BANDWIDTH, inputdata.SEQUENCEID,
inputdata.REQUESTTIMESTAMP
FROM inputdata, outputdata
WHERE
inputdata.SEQUENCEID = outputdata.SEQUENCEID
AND inputdata.REQUEST_TIMESTAMP >= '2019-01-01 00:00:00'
AND inputdata.SEQUENCEID IS NOT NULL
AND inputdata.SEQUENCEID NOT IN ('N/A')
ORDER BY inputdata.SEQUENCEID DESC "
# jdbc_paging_enabled => "true"
# jdbc_page_size => "10000"
}
}
filter {
mutate {
convert => { "A_LONGITUDE" => "float" }
convert => { "A_LATITUDE" => "float" }
convert => { "B_LONGITUDE" => "float" }
convert => { "B_LATITUDE" => "float" }
}
mutate {
rename => {
"A_LONGITUDE" => "[geo_point1][lon]"
"A_LATITUDE" => "[geo_point1][lat]"
}
}
mutate {
rename => {
"B_LONGITUDE" => "[geo_point2][lon]"
"B_LATITUDE" => "[geo_point2][lat]"
}
}
}
output {
elasticsearch {
hosts => ["http://some server"]
index => "cc-test"
#document_type => "system_logs"
user => "Username"
password => "Password"
}
stdout { codec => rubydebug }
}
Don't understand what is wrong with the Filter part and why data is not getting into the columns geo_point1 and geo_point2!!
Somebody please help :pray::pray::pray:
I am absolutely new to Logstash and I am trying to parse my multiline logentries, that are in the following format
<log level="INFO" time="Wed May 03 08:25:03 CEST 2017" timel="1493792703368" host="host">
<msg><![CDATA[Method=GET URL=http://localhost (Vers=[Version], Param1=[param1], Param2=[param1]) Result(Content-Length=[22222], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:1098/CSI:-/Me:1/Total:1099]]>
</msg>
</log>
Do you know how to implement the filter in logstash config to be able to index the following fields in elasticsearch
time, host, Vers, Param1, Param2, TISP
Thank you very much
OK, I found out how to do it. This is my pipeline.conf file and it works
input {
beats {
port => 5044
}
}
filter {
xml {
store_xml => false
source => "message"
xpath => [
"/log/#level", "level",
"/log/#time", "time",
"/log/#timel", "unixtime",
"/log/#host", "host_org",
"/log/#msg", "msg",
"/log/msg/text()","msg_txt"
]
}
grok {
break_on_match => false
match => ["msg_txt", "Param1=\[(?<param1>-?\w+)\]"]
match => ["msg_txt", "Param2=\[(?<param2>-?\w+)\]"]
match => ["msg_txt", "Vers=\[(?<vers>-?\d+\.\d+)\]"]
match => ["msg_txt", "TISP:(?<tisp>-?\d+)"]
match => [unixtime, "(?<customTime>-?\d+)"]
}
if "_grokparsefailure" in [tags] {
drop { }
}
mutate {
convert => { "tisp" => "integer" }
}
date {
match => [ "customTime", "UNIX_MS"]
target => "#timestamp"
}
if "_dateparsefailure" in [tags] {
drop { }
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => user
password => passwd
}
}
We can send email notification to a particular email address but I want to send email to different mailing address based on some pattern in logs.
For example say I have three users with email address
userOne#something.com receives mail id log contains [userOneModule]
userTwo#something.com receives mail id log contains [userTwoModule]
userThree#something.com receives mail id log contains [userThreeModule
Logstash version used is 1.3.3
Any help if this is possible in logstash or any workaround to achieve something like this.
This is my configuration, Although both 'Security' and 'Portal' matches but email is sent to only one.
When I keep only one kind of logs say Security Logs or Portal Logs it works but when I keep both the logs it only sends email to one of it.
output {
if [module] == "Security"{
email {
to => "userOne#somemail.com"
from => "dummy2161#somemail.com"
match =>["%{message}","severity,ERROR"]
subject => "Error Occured"
body => "%{message}"
via => "smtp"
options => {
starttls => "true"
smtpIporHost => "smtp.gmail.com"
port => "587"
userName => "dummy2161#somemail.com"
password => "*******"
authenticationType => "plain"
}
}
}
if [module] == "Portal"{
email {
to => "userTwo#somemail.com"
from => "dummy2161#gmail.com"
match =>["%{message}","severity,ERROR"]
subject => "Error Occured"
body => "%{message}"
via => "smtp"
options => {
starttls => "true"
smtpIporHost => "smtp.gmail.com"
port => "587"
userName => "dummy2161#somemail
password => "*****"
authenticationType => "plain"
}
}
}
}
Thanks
You can either store the recipient email address in a field (using conditionals or grok filters to assign the value) and refer to that field in the email output's to parameter, or you can wrap multiple email outputs in conditionals.
Using a field for storing the address:
filter {
# If the module name is the same as the recipient address's local part
mutate {
add_field => { "recipient" => "%{modulename}#example.com" }
}
# Otherwise you might have to use conditionals.
if [modulename] == "something" {
mutate {
add_field => { "recipient" => "someuser#example.com" }
}
} else {
mutate {
add_field => { "recipient" => "otheruser#example.com" }
}
}
}
output {
email {
to => "%{recipient}"
...
}
}
Wrapping outputs in conditionals:
output {
if [modulename] == "something" {
email {
to => "someuser#example.com"
...
}
} else {
email {
to => "otheruser#example.com"
...
}
}
}