Maybe it is me, but how come that when I use the CSV Output from LogStash it does not output in a csv format? I am using nothing special (as seen in the configuration). Can someone tell me what I am doing wrong?
input
{
stdin {
type => "stdin-type"
}
}
filter
{
mutate { add_field => { "test" => "testme" } }
mutate { add_field => { "[#metadata][test]" => "Hello" } }
mutate { add_field => { "[#metadata][test2]" => "world" } }
}
output {
# .\bin\logstash-plugin.bat install logstash-output-csv
csv {
fields => ["test", "[#metadata][test]"]
path => "./TestLogs.csv"
}
stdout { codec => rubydebug { metadata => true } }
}
It actually create an output. If I type something (Ex.: test me) in the console (stdin) it creates the file and all. But the CSV file contains the following:
2016-11-25T11:49:40.338Z MyPcName test me
And I am expecting the following:
testme,Hello
Note: I am using LogStash 5 (latest version at the moment).
This is a Logstash 5.x issue. For now, I'm using the script below:
output {
file {
path => "/app/logstash/test.csv"
message_pattern => (grok pattern)
}
Related
I want to process a text file locally and the output I want to save in a file as log/text. This is my code but this does not work.
input {
file {
path => "C:/Users/USERNAME/Documents/Projects/test_data.txt"
start_position => beginning
}
}
filter {
}
output {
stdout { codec => rubydebug }
file {
path => "C:/Users/USERNAME/Documents/Projects/out.log"
}
}
In the terminal I have noticed
incedb_path set, generating one based on the "path" setting
{:sincedb_path=>"C:/Users/..
I've tried to parse it using the json, json_lines and even the multiline input plugin, yet to no avail. The multiline works well on my local machine but doesn't seem to work on my s3 and ec2 instance.
How would I write the grok filter to parse this?
This is what my JSON file looks like
{
"sourceId":"94:54:93:3B:81:6F1",
"machineId":"c1VR21A0GoCBgU6EMJ78d3CL",
"columnsCSV":"timestamp,state,0001,0002,0003,0004",
"tenantId":"iugcp",
"valuesCSV":"1557920277890,1,98.66,0.07,0.1,0.17 ",
"timestamp":"2019-05-15T11:37:57.890Z"
}
This is my config -
input {
file{
codec => multiline
{
pattern => '^\{'
negate => true
what => previous
}
path => "/home/*myusername*/Desktop/data/*.json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
mutate
{
replace => [ "message", "%{message}}" ]
gsub => [ 'message','\n','']
}
if [message] =~ /^{.*}$/
{
json { source => message }
}
}
//Output tag is correct, haven't included it here
The results I get is just the json file present in the "message" field.
What I wanted is for every json tag, there should be a separate field in the document.
Logstash v2.4.1.
I'm sending JSON formatted logs to my Logstash server via UDP packet. The logs look something similar to this.
{
"key1":"value1",
"key2":"value2",
"msg":"2017-03-02 INFO [com.company.app] Hello world"
}
This is my output filter
output {
stdout {
codec => rubydebug
}
file {
path => "/var/log/trm/debug.log"
codec => line { format => "%{msg}" }
}
}
The rubydebug output codec shows the log like this
{
"message" => {\"key1\":\"value1\", "key2\":\"value2\", \"msg\":\"2017-03-02 INFO [com.company.app] Hello world\"
}
and the file output filter also shows the JSON log correctly, like this
{"key1":"value1", "key2":"value2", "msg":"2017-03-02 INFO [com.company.app] Hello world"}
When I use the JSON code in the input filter, I get _jsonparsefailures from Logstash on "some" logs, even though different online JSON parsers parse the JSON correctly, meaning my logs are in a valid JSON format.
input {
udp => {
port => 5555
codec => json
}
}
Therefore, I'm trying to use the json filter instead, like this
filter {
json => {
source => "message"
}
}
Using the json filter, how can I extract the "key1", "key2", and the "msg" fields in the "message?"
I tried this to no avail, that is, I don't see the "key1" field in my rubydebug output.
filter {
json => {
source => "message"
add_field => {
"key1" => "%{[message][key1]}"
}
}
}
I would suggest you to start with one of the two configuration below (I use the multiline codec to concatenate the input into a json, because otherwise logstash will read line by line, and one line of a json is not a valid json), then either filter the json, or use the json codec, and then output it to wherever it is needed. You will still have some configuration to do, but I believe it might help you to get started:
input{
file {
path => "/an/absolute/path/tt2.json" #It really has to be absolute!
start_position => beginning
sincedb_path => "/another/absolute/path" #Not mandatory, just for ease of testing
codec => multiline{
pattern => "\n"
what => "next"
}
}
}
filter{
json {
source => "multiline"
}
}
output {
file {
path => "data/log/trm/debug.log"
}
stdout{codec => json}
}
Second possibility:
input{
file {
path => "/an/absolute/path/tt2.json" #It really has to be absolute!
start_position => beginning
sincedb_path => "/another/absolute/path" #Not mandatory, just for ease of testing
codec => multiline{
pattern => "\n"
what => "next"
}
codec => json{}
}
}
output {
file {
path => "data/log/trm/debug.log"
}
stdout{codec => json}
}
Edit With the udp input I guess it should be (not tested):
input {
udp => {
port => 5555
codec => multiline{ #not tested this part
pattern => "^}"
what => "previous"
}
codec => json{}
}
}
I'm building out logstash and would like to build functionality to anonymize fields as specified in the message.
Given the message below, the field fta is a list of fields to anonymize. I would like to just use %{fta} and pass it through to the anonymize filter, but that doesn't seem to work.
{ "containsPII":"True", "fta":["f1","f2"], "f1":"test", "f2":"5551212" }
My config is as follows
input {
stdin { codec => json }
}
filter {
if [containsPII] {
anonymize {
algorithm => "SHA1"
key => "123456789"
fields => %{fta}
}
}
}
output {
stdout {
codec => rubydebug
}
}
The output is
{
"containsPII" => "True",
"fta" => [
[0] "f1",
[1] "f2"
],
"f1" => "test",
"f2" => "5551212",
"#version" => "1",
"#timestamp" => "2016-07-13T22:07:04.036Z",
"host" => "..."
}
Does anyone have any thoughts? I have tried several permutations at this point with no luck.
Thanks,
-D
EDIT:
After posting in the Elastic forums, I found out that this is not possible using base logstash functionality. I will try using the ruby filter instead. So, to ammend my question, How do I call another filter from within the ruby filter? I tried the following with no luck and honestly can't even figure out where to look. I'm very new to ruby.
filter {
if [containsPII] {
ruby {
code => "event['fta'].each { |item| event[item] = LogStash::Filters::Anonymize.execute(event[item],'12345','SHA1') }"
add_tag => ["Rubyrun"]
}
}
}
You can execute the filters from ruby script. Steps will be:
Create the required filter instance in the init block of inline ruby script.
For every event call the filter method of the filter instance.
Following is the example for above problem statement. It will replace my_ip field in event with its SHA1.
Same can be achieved using ruby script file.
Following is the sample config file.
input { stdin { codec => json_lines } }
filter {
ruby {
init => "
require 'logstash/filters/anonymize'
# Create instance of filter with applicable parameters
#anonymize = LogStash::Filters::Anonymize.new({'algorithm' => 'SHA1',
'key' => '123456789',
'fields' => ['my_ip']})
# Make sure to call register
#anonymize.register
"
code => "
# Invoke the filter
#anonymize.filter(event)
"
}
}
output { stdout { codec => rubydebug {metadata => true} } }
Well, I wasn't able to figure out how to call another filter from within a ruby filter, but I did get to the functional goal.
filter {
if [fta] {
ruby {
init => "require 'openssl'"
code => "event['fta'].each { |item| event[item] = OpenSSL::HMAC.hexdigest(OpenSSL::Digest::SHA256.new, '123456789', event[item] ) }"
}
}
}
If the field FTA exists, it will SHA2 encode each of the fields listed in that array.
I want to put a value into part of a nested hash, but name that part depending on upstream filters. This is to refactor and reduce overall code size as currently each of the 20+ incoming event types have their own section like this with 18 lines in the logstash file (but currently the %{detail_part} bit is hard-coded).
# Working code
filter {
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][OtherThing][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][OtherThing][MoreDetail]" => "false"
}
}
}
}
Above is hard-coded to "OtherThing". Below has a variable, but doesn't work.
# Non-Working code
filter {
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][%{detail_part}][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][%{detail_part}][MoreDetail]" => "false"
}
}
}
}
In the above (non-Working code), detail_part is set in an upstream filter to a string value like "OtherThing". This currently compiles and runs, but no XML is output from it, so I don't think anything is set into the hash as a result of these statements.
I know it can be done with embedded Ruby code, but I would like a way that is as simple as possible. The output of this process is going to XML so I am constrained to use this kind of nested hash.
Is this possible with Logstash?
It turns out that yes, Logstash supports this, just the syntax was wrong. So here is the fix:
filter {
# This field could be set conditionally in an if statement ...
mutate { add_field => { "[detail_part]" => "Result" } }
if [result] == 0 {
# Success
mutate {
add_field => {
"[Thing][ThingDetail][%{[detail_part]}][MoreDetail]" => "true"
}
}
}
else {
# Failed
mutate {
add_field => {
"[Thing][ThingDetail][%{[detail_part]}][MoreDetail]" => "false"
}
}
}
}
I just couldn't find any non-trivial examples that did things like this.