logstash.conf:
elasticsearch{
hosts => ["172.31.29.xxx:9200"]
index => "redis-%{+YYYYMMdd}"
}
I used YYYYMMdd to define index ,I want to change the index timezone,How can I do it?
thx
You can use the logstash "date" filter plugin to change log event timezone.
Here is an example of a logstash.conf file I have put together to show you how and where the date filter goes:
input {
file {
...
type => "typeName"
}
}
filter {
if [type] == "typeName" {
grok {
match => {"message" => "^%{TIMESTAMP_ISO8601:event_timestamp}..."}
}
}
date {
match => [ "event_timestamp", "YYYY-MM-dd HH:mm:ss,SSS" ]
timezone => "Etc/GMT"
locale => "en"
}
}
output {
if [type] == "typeName" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "indexName"
}
}
}
you can look at the available timezone values here.
Related
I am setting up Logstash to ingest Airflow logs. The following config is giving me the output I need:
input {
file {
path => "/my_path/logs/**/*.log"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
if [path] =~ /\/my_path\/logs\/containers\/.*/ or [path] =~ /\/my_path\/logs\/scheduler\/.*/ {
drop{}
}
else {
grok {
"match" => [ "message", "\[%{TIMESTAMP_ISO8601:log_task_execution_datetime}\]%{SPACE}\{%{DATA:log_file_line}\}%{SPACE}%{WORD:log_level}%{SPACE}-%{SPACE}%{GREEDYDATA:log_message}" ]
"remove_field" => [ "message" ]
}
date {
"match" => [ "log_task_execution_datetime", "ISO8601" ]
"target" => "log_task_execution_datetime"
"timezone" => "UTC"
}
dissect {
"mapping" => { "path" => "/my_path/logs/%{dag_id}/%{task_id}/%{dag_execution_datetime}/%{try_number}.%{}" }
"add_field" => { "log_id_template" => "{%{dag_id}}-{%{task_id}}-{%{dag_execution_datetime}}-{%{try_number}}" }
}
}
}
output {
stdout {codec => rubydebug{metadata => true}}
}
But I do not like having to specify the path "/my_path/logs/" multiple times.
In my input section, I tried to use:
add_field => { "[#metadata][base_path]" => "/my_path/logs/" }
and then, in the filter section:
if [path] =~ /[#metadata][base_path].*/ or [path] =~ /[#metadata][base_path].*/ {
drop{}
}
...
dissect {
"mapping" => { "path" => "[#metadata][base_path]%{dag_id}/%{task_id}/%{dag_execution_datetime}/%{try_number}.%{}" }
But it doesn't seem to work for the regex in the filter or in the dissect mapping. I get a similar issue when trying to use an environment variable as described here.
I have the - maybe naïve - notion that I should be able to use one variable for all references to the base path. Is there a way?
Using an environment variable in a conditional is not supported. There has been a github issue requesting it as an enhancement open since 2016. The workaround is to use mutate+add_field to add a field to [#metadata] then test that.
"mapping" => { "path" => "${[#metadata][base_path]}%{dag_id}/%{task_id} ...
should work. The terms in a conditional are not sprintf'd, so you cannot use %{}, but you can do a substring match. If FOO is set to /home/user/dir then
mutate { add_field => { "[#metadata][base_path]" => "${FOO}" } }
mutate { add_field => { "[path]" => "/home/user/dir/file" } }
if [#metadata][base_path] in [path] { mutate { add_field => { "matched" => true } } }
results in the [matched] field getting added. I do not know of a way to anchor the string match, so if FOO were set to /dir/ then that would also match.
I am currently using logstash, elasticsearch and kibana 6.3.0
My log are generated at a unique id path : /tmp/USER_DATA/FactoryContainer/images/(my unique id)/oar/oar_image_job(my unique id).stdout
What I want to do is to match this unique id and to create a field with this id.
I m a bit novice to logstash filter but I don't know why it doesn't want to use my uid and always return me %{uid} in my field or this Failed to execute action error.
my filter :
input {
file {
path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
start_position => "beginning"
add_field => { "data_source" => "oar-image-job" }
}
}
filter {
grok {
match => ["path","%{UNIXPATH}%{NUMBER:uid}%{UNIXPATH}"]
}
mutate {
add_field => [ "unique_id" => "%{uid}" ]
}
}
output {
if [data_source] == "oar-image-job" {
elasticsearch {
index => "oar-image-job-%{+YYYY.MM.dd}"
hosts => ["localhost:9200"]
}
}
}
the data_source field is to avoid this issue: When you put multiple config files in a directory for Logstash to use, they will all be concatenated
in the grok debugger %{UNIXPATH}%{NUMBER:uid}%{UNIXPATH} my path return me the good value
link to the solution : https://discuss.elastic.co/t/cant-create-a-field-with-a-variable-from-a-grok-match-regex/142613/7?u=thesmartmonkey
the correct filter :
input {
file {
path => "/tmp/USER_DATA/FactoryContainer/images/*/oar/oar_image_job*.stdout"
start_position => "beginning"
add_field => { "data_source" => "oar-image-job" }
}
}
filter {
grok {
match => { "path" => [ "/tmp/USER_DATA/FactoryContainer/images/%{DATA:unique_id}/oar/oar_image_job%{DATA}.stdout" ] }
}
}
output {
if [data_source] == "oar-image-job" {
elasticsearch {
index => "oar-image-job-%{+YYYY.MM.dd}"
hosts => ["localhost:9200"]
}
}
}
I am absolutely new to Logstash and I am trying to parse my multiline logentries, that are in the following format
<log level="INFO" time="Wed May 03 08:25:03 CEST 2017" timel="1493792703368" host="host">
<msg><![CDATA[Method=GET URL=http://localhost (Vers=[Version], Param1=[param1], Param2=[param1]) Result(Content-Length=[22222], Content-Type=[text/xml; charset=utf-8]) Status=200 Times=TISP:1098/CSI:-/Me:1/Total:1099]]>
</msg>
</log>
Do you know how to implement the filter in logstash config to be able to index the following fields in elasticsearch
time, host, Vers, Param1, Param2, TISP
Thank you very much
OK, I found out how to do it. This is my pipeline.conf file and it works
input {
beats {
port => 5044
}
}
filter {
xml {
store_xml => false
source => "message"
xpath => [
"/log/#level", "level",
"/log/#time", "time",
"/log/#timel", "unixtime",
"/log/#host", "host_org",
"/log/#msg", "msg",
"/log/msg/text()","msg_txt"
]
}
grok {
break_on_match => false
match => ["msg_txt", "Param1=\[(?<param1>-?\w+)\]"]
match => ["msg_txt", "Param2=\[(?<param2>-?\w+)\]"]
match => ["msg_txt", "Vers=\[(?<vers>-?\d+\.\d+)\]"]
match => ["msg_txt", "TISP:(?<tisp>-?\d+)"]
match => [unixtime, "(?<customTime>-?\d+)"]
}
if "_grokparsefailure" in [tags] {
drop { }
}
mutate {
convert => { "tisp" => "integer" }
}
date {
match => [ "customTime", "UNIX_MS"]
target => "#timestamp"
}
if "_dateparsefailure" in [tags] {
drop { }
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
user => user
password => passwd
}
}
I am parsing json log file in Logstash. There is a field named #person.name. I tried to rename this field name before sending it to elasticsearch. I also tried to remove the field but I couldn't remove or delete that field because of that my data not getting indexed in Elasticsearch.
Error recorded in elasticsearch
MapperParsingException[Field name [#person.name] cannot contain '.']
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:276)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parse(ObjectMapper.java:196)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:308)
at org.elasticsearch.index.mapper.object.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:221)
at org.elasticsearch.index.mapper.object.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:138)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:119)
at org.elasticsearch.index.mapper.DocumentMapperParser.parse(DocumentMapperParser.java:100)
at org.elasticsearch.index.mapper.MapperService.parse(MapperService.java:435)
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.applyRequest(MetaDataMappingService.java:257)
at org.elasticsearch.cluster.metadata.MetaDataMappingService$PutMappingExecutor.execute(MetaDataMappingService.java:230) at org.elasticsearch.cluster.service.InternalClusterService.runTasksForExecutor(InternalClusterService.java:458)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:762)
My Logstash config
input {
beats {
port => 11153
}
}
filter
{
if [type] == "person_get" {
##Parsing JSON input to JSON Filter..
json {
source => "message"
}
mutate{
rename => { "#person.name" => "#person-name" }
remove_field => [ "#person.name"]
}
fingerprint {
source => ["ResponseTimestamp"]
target => "fingerprint"
key => "78787878"
method => "SHA1"
concatenate_sources => true
}
}
}
output{
if [type] == "person_get" {
elasticsearch {
index => "logstash-person_v1"
hosts => ["xxx.xxx.xx:9200"]
document_id => "%{fingerprint}" # !!! prevent duplication
}
stdout {
codec => rubydebug
}
} }
Is there a way of having the filename of the file being read by logstash as the index name for the output into ElasticSearch?
I am using the following config for logstash.
input{
file{
path => "/logstashInput/*"
}
}
output{
elasticsearch{
index => "FromfileX"
}
}
I would like to be able to put a file e.g. log-from-20.10.2016.log and have it indexed into the index log-from-20.10.2016. Does the logstash input plugin "file" produce any variables for use in the filter or output?
Yes, you can use the path field for that and grok it to extract the filename into the index field
input {
file {
path => "/logstashInput/*"
}
}
filter {
grok {
match => ["path", "(?<index>log-from-\d{2}\.\d{2}\.\d{4})\.log$" ]
}
}
output{
elasticsearch {
index => "%{index}"
}
}
input {
file {
path => "/home/ubuntu/data/gunicorn.log"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => "%{USERNAME:u1} %{USERNAME:u2} \[%{HTTPDATE:http_date}\] \"%{DATA:http_verb} %{URIPATHPARAM:api} %{DATA:http_version}\" %{NUMBER:status_code} %{NUMBER:byte} \"%{DATA:external_api}\" \"%{GREEDYDATA:android_client}\""
remove_field => ["message"]
}
}
date {
match => ["http_date", "dd/MMM/yyyy:HH:mm:ss +ssss"]
}
ruby {
code => "event.set('index_name',event.get('path').split('/')[-1].gsub('.log',''))"
}
}
output {
elasticsearch {
hosts => ["0.0.0.0:9200"]
index => "%{index_name}-%{+yyyy-MM-dd}"
user => "*********************"
password => "*****************"
}
stdout { codec => rubydebug }
}