Hello I have this configuration for a logstash running on my computer :
input {
exec {
command => "powershell -executionpolicy unrestricted -f scripts/windows/process.ps1 command logstash"
interval => 30
type => "process_data"
codec => line
tags => [ logstash" ]
}
}
output
{
if "sometype-logs" in [tags] {
elasticsearch {
action => "index"
doc_as_upsert => true
index => "sometype-logs-%{+YYYY.MM.dd}"
hosts => "locahost:9200"
template_overwrite => true
}
} else {
elasticsearch {
action => "index"
doc_as_upsert => true
index => "%{type}"
hosts => "localhost:9200"
template_overwrite => true
}
}
When displaying indexes I have :
Why is index name is "%type" and not "process_data" ?
Probably just something about syntax. To used some field of the data, you must use this syntax
%{[somefield]}
(see example on this documentation page)
So, in your case, try this :
"%{[type]}"
in place of
"%{type}"
Related
I am using Logstash to update by query existing Elasticsearch documents with an additional field that contains aggregate values extracted from Potgresql table.
I use elastichsearch output to load one index using document_id and http output to update another index that have different document_id but receving errors:
[2023-02-08T17:58:12,086][ERROR][logstash.outputs.http ][main][b64f19821b11ee0df1bd165920785876cd6c5fab079e27d39bb7ee19a3d642a4] [HTTP Output Failure] Encountered non-2xx HTTP code 400 {:response_code=>400, :url=>"http://localhost:9200/medico/_update_by_query", :event=>#LogStash::Event:0x19a14c08}
This is my pipeline configuration:
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "jdbc:postgresql://handel:5432/mydb"
statement_filepath => "D:\ProgrammiUnsupported\logstash-7.15.2\config\nota_sede.sql"
}
}
filter {
aggregate {
task_id => "%{idCso}"
code => "
map['idCso'] = event.get('idCso')
map['noteSede'] ||= []
map['noteSede'] << {
'id' => event.get('idNota'),
'tipo' => event.get('tipoNota'),
'descrizione' => event.get('descrizione'),
'data' => event.get('data'),
'dataInizio' => event.get('dataInizio'),
'dataFine' => event.get('dataFine')
}
event.cancel()"
push_previous_map_as_event => true
timeout => 60
timeout_tags => ['_aggregatetimeout']
}
}
}
output {
stdout { codec => rubydebug { metadata => true } }
# this works
elasticsearch {
hosts => "https://localhost:9200"
document_id => "STRUTTURA_%{idCso}"
index => "struttura"
action => "update"
user => "user"
password => "password"
ssl => true
cacert => "/usr/share/logstash/config/ca.crt"
}
http {
url => "http://localhost:9200/medico/_update_by_query"
user => "elastic"
password => "changeme"
http_method => "post"
format => "message"
content_type => "application/json"
message => '{
"query":{
"term":{
"idCso":"%{idCso}"
}
},
"script":{
"source":"ctx._source.noteSede=params.noteSede",
"lang":"painless",
"params":{
"noteSede":"%{noteSede}"
}
}
}
}'
}
}
The stdout output show me the sended docs to output like this:
{
"query" => {
"term" => {
"idCso" => "859119"
}
},
"script" => {
"source" => "ctx._source.noteSede=params.noteSede",
"lang" => "painless",
"params" => {
"noteSede" => "{dataFine=null, dataInizio=2020-02-13, descrizione=?, tipo=DB, id=6390644, data=2020-02-13 12:26:58.409},{dataFine=null, dataInizio=2020-02-13, descrizione=?, tipo=DE, id=6390645, data=2020-02-13 12:26:58.41}"
}
}
}
}
How could I set noteSede array field into message to _update_by_query ?
I have fields like A_Latitude, A_Longitude, B_Latitude and B_Longitude. I would like to make use of this data and create Maps in Kibana. The problem is data is getting into elasticsearch, but the gejson columns created in Logstash filter not gettin recognized and data is not being fed into geo_point1 and geo_point2.
Hence, first created a geo_point mapping in Kibana dev tools as follows,
PUT cc-test
{
"mappings": {
"properties": {
"geo_point1":{
"type": "geo_point"
},
"geo_point2":{
"type": "geo_point"
}
}
}
}
I have configured my logstash config file the following way,
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "some string"
# The user we wish to execute our statement as
jdbc_user => "User"
jdbc_password => "Password"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/apps/ELK/logstash/driver/ngdbc-2.4.56.jar"
jdbc_driver_class => "com.sap.db.jdbc.Driver"
# our query
#jdbc_validate_connection => true
#schedule => "* * * * *"
#record_last_run => true
# last_run_metadata_path => "login.txt"
statement => "SELECT
inputdata.A_LATITUDE, inpudata.A_LONGITUDE, inputdata.B_LATITUDE,
inputdata.B_LONGITUDE, outputdata.BANDWIDTH, inputdata.SEQUENCEID,
inputdata.REQUESTTIMESTAMP
FROM inputdata, outputdata
WHERE
inputdata.SEQUENCEID = outputdata.SEQUENCEID
AND inputdata.REQUEST_TIMESTAMP >= '2019-01-01 00:00:00'
AND inputdata.SEQUENCEID IS NOT NULL
AND inputdata.SEQUENCEID NOT IN ('N/A')
ORDER BY inputdata.SEQUENCEID DESC "
# jdbc_paging_enabled => "true"
# jdbc_page_size => "10000"
}
}
filter {
mutate {
convert => { "A_LONGITUDE" => "float" }
convert => { "A_LATITUDE" => "float" }
convert => { "B_LONGITUDE" => "float" }
convert => { "B_LATITUDE" => "float" }
}
mutate {
rename => {
"A_LONGITUDE" => "[geo_point1][lon]"
"A_LATITUDE" => "[geo_point1][lat]"
}
}
mutate {
rename => {
"B_LONGITUDE" => "[geo_point2][lon]"
"B_LATITUDE" => "[geo_point2][lat]"
}
}
}
output {
elasticsearch {
hosts => ["http://some server"]
index => "cc-test"
#document_type => "system_logs"
user => "Username"
password => "Password"
}
stdout { codec => rubydebug }
}
Don't understand what is wrong with the Filter part and why data is not getting into the columns geo_point1 and geo_point2!!
Somebody please help :pray::pray::pray:
I am currently using logstash, elasticsearch and kibana 6.3.0
My log are generated at a unique id path : /tmp/USER_DATA/FactoryContainer/images/(my unique id)/oar/oar_image_job(my unique id).stdout
What I want to do is to match this unique id and to create a field with this id.
I m a bit novice to logstash filter but I don't know why it doesn't want to use my uid and always return me %{uid} in my field or this Failed to execute action error.
my filter :
input {
file {
path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
start_position => "beginning"
add_field => { "data_source" => "oar-image-job" }
}
}
filter {
grok {
match => ["path","%{UNIXPATH}%{NUMBER:uid}%{UNIXPATH}"]
}
mutate {
add_field => [ "unique_id" => "%{uid}" ]
}
}
output {
if [data_source] == "oar-image-job" {
elasticsearch {
index => "oar-image-job-%{+YYYY.MM.dd}"
hosts => ["localhost:9200"]
}
}
}
the data_source field is to avoid this issue: When you put multiple config files in a directory for Logstash to use, they will all be concatenated
in the grok debugger %{UNIXPATH}%{NUMBER:uid}%{UNIXPATH} my path return me the good value
link to the solution : https://discuss.elastic.co/t/cant-create-a-field-with-a-variable-from-a-grok-match-regex/142613/7?u=thesmartmonkey
the correct filter :
input {
file {
path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
start_position => "beginning"
add_field => { "data_source" => "oar-image-job" }
}
}
filter {
grok {
match => { "path" => [ "/tmp/USER_DATA/FactoryContainer/images/%{DATA:unique_id}/oar/oar_image_job%{DATA}.stdout" ] }
}
}
output {
if [data_source] == "oar-image-job" {
elasticsearch {
index => "oar-image-job-%{+YYYY.MM.dd}"
hosts => ["localhost:9200"]
}
}
}
Here is an exmple of event message:
{
"timestamp":"2016-03-29T22:35:44.770750-0400",
"flow_id":45385792,
"in_iface":"eth1",
"event_type":"alert",
"src_ip":"3.3.3.8",
"src_port":21,
"dest_ip":"2.2.2.2",
"dest_port":52934,
"proto":"TCP",
"alert":{
"action":"allowed",
"gid":1,
"signature_id":4027,
"rev":0,
"signature":"FTP Successful Login",
"category":"",
"severity":3
},
"payload":"MjU3ICIvaG9tZS9uZXd1c2VyIg0K",
"payload_printable":"257 newuser",
"stream":0,
"packet":"AFBWo0NoAFBWoxZWCABFAABJKDpAAEAGCGcDAwMIAgICAgAVzsbd4MhqOBOjfoAYAOMYcwAAAQEIChHN4EQHnwugMjU3ICIvaG9tZS9uZXd1c2VyIg0K"
}
And I'd like to be able to identify the string "newuser" (comes always after the number "257") and to create another field named user, and to add the "newuser" string into it.
My Logstash config file is the following:
input
beats
port => 5044
codec => json
type => "SuricataIDPS"
output
elasticsearch
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
#document_type => "%{[#metadata][type]}"
How can I extract the "newuser" string and to add a new field with that value?
You have to define filter stanza in your config with grok filter, see below:
input {
beats {
port => 5044
codec => json
type => "SuricataIDPS"
}
}
filter {
grok {
match => ["payload_printable", "257 (?<user>.+)"]
tag_on_failure => []
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
sniffing => true
manage_template => false
index => "%{[#metadata][beat]}-%{+YYYY.MM.dd}"
#document_type => "%{[#metadata][type]}"
}
}
I'm getting some errors attempting to do a data import in logstash. I'm seeing it for every "geo" field that I have. Here are some of my config files
input {
jdbc {
jdbc_driver_library => "c:\binaries\driver\ojdbc6.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:#random:1521/random"
jdbc_user => "user"
jdbc_password => "password"
statement => "select a.*, myfunc() as geo from foo a"
type => "sometype"
}
}
filter{
if [type] == "sometype" {
mutate {
rename => { "sometype_id" => "id" }
remove_field => ["gdo_geometry"]
add_field => [ "display", "%{id}" ]
}
# parses string to json
json{
source => "geo"
target => "geometry"
}
}
}
output {
if [type] == "sometype" {
elasticsearch {
hosts => ["myesbox:80"]
document_id => "%{id}"
index => "sjw"
}
}
}
Here is a second.
input {
jdbc {
jdbc_driver_library => "c:\binaries\driver\ojdbc6.jar"
jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
jdbc_connection_string => "jdbc:oracle:thin:#random:1521/random"
jdbc_user => "user"
jdbc_password => "password"
statement => "select a.*, myfunc() as geo from foo2 a"
type => "sometype2"
}
}
filter{
if [type] == "sometype2" {
mutate {
rename => { "sometype2_id" => "id" }
remove_field => ["gdo_geometry"]
add_field => [ "display", "%{id}" ]
}
# parses string to json
json{
source => "geo"
target => "geometry"
}
}
}
output {
if [type] == "sometype2" {
elasticsearch {
hosts => ["myesbox:80"]
document_id => "%{id}"
index => "sjw"
}
}
}
And here is the error message (repeated once for each record in my database tables).
{:timestamp=>"2016-01-05T13:33:18.258000-0800", :message=>"Trouble parsing json", :source=>"geo", :raw=>nil, :exception=>java.lang.ClassCastException: org.jruby.RubyNil cannot be cast to org.jruby.RubyIO, :level=>:warn}
Now interestingly, the field DOES seem to import successfully. I can see the data populated as expected. But I don't know why this warning is being generated. I'm running the logstash as
logstash -f /my/logstash/dir
Also interesting to note is that if I modify the first config file given and changed the source json filter name to "geom" instead of "geo" -- this warning would no longer occur. It seems to only occur when I have multiple config files with the same field/json filter combinations. So if I then added a third config file and it had a "geo" field being parsed by the json filter -- the issue occurs again -- though I would still not see any warning messages for the first config file -- only the second and third.
The issue here actually turned out to be a bug with the 2.0 version of logstash. I'm not sure what exactly the problem was, but upgrading to 2.1 resolved the issue for me.