logstash - output single event into multiple line output file - logstash

I have a jdbc input with a select statement. each row in the restult set has 3 columns. c1, c2, c3. the event emitted has the following structure:
{"c1":"v1", "c2":"v2", "c3":"v3", "file_name":"tmp.csv"}
I want to output the values in a file in the following manner:
output file:
v1
v2
v3
this is the output configuration:
file {
path => "/tmp/%{file_name}"
codec => plain { format => "%{c1}\n%{c2}\n%{c3}" }
write_behavior => "overwrite"
flush_interval => 0
}
but what is generated is
outputfile:
v1\nv2\nv3
is the plain codec plugin not the one i need? is there any other codec plugin for the output file plugin that i can use? or is the only option i have is to write my own plugin?
Thanks!

A bit late to the party, but maybe this helps others. Although it looks funky, you should be able to get away with simply hitting Enter within the format string (using the line codec).
file {
path => "/tmp/%{file_name}"
codec => line {
format => "%{c1}
%{c2}
%{c3}"
}
write_behavior => "overwrite"
flush_interval => 0
}
Not the prettiest approach, but it works. Not sure if there is a better way.

what you are looking for is the line codec plugin: https://www.elastic.co/guide/en/logstash/current/plugins-codecs-line.html

Related

How to force gem that converts all bins to 93k multibins to output 93k native bins?

My need is to get good old fashioned 93k native bad bins defined in my testflow. My ruby file compiles but looks like the gem is converting all bins to multibins. Is there a way to force this from my ruby file instead of hacking the gem files? If yes, going ahead with this, I couldn't find how to specify hardbin description and softbin description in origen. That is something I would like to add in the ruby code instead of on ATE.
Also on a side note, I am trying to force the output file name to something i want. Like in the sample code below i want the output file to be test.tf. The gem is adding some string and an underscore in front of "test". I don't need that either.
sample code:
Flow.create interface: 'MyTester::Interface', params: :room, unique_test_names: nil, flow_name:
:test, file_name: :test, insertion: :prb do
test_info1 = {"key_1" =>
[{:testname => "t1",
:sbin => 100,
:patternname => "p1"}],
"key_2" =>
[{:testname => "t2",
:sbin => 200,
:patternname => "t3"}]
}
testnum = 100000
test_info1.each do |key,val|
puts key
val.each do |info|
tname, sb, pname = info.values_at(:testname, :sbin, :patternname)
puts "#{tname} : #{sb} : #{pname}"
test_suites.add("#{tname}", pattern: "#{pname}", tim_spec_set: 1, timset: 1, lev_equ_set: 1,
lev_spec_set: 10, levset: 1, test_method: test_methods.ac_tml.ac_test.functional_test)
testnum = testnum+100
test :"#{tname}", bin: 10, softbin: "#{sb}", tnum: testnum
end
end
end

How use regex in logstash input file

I am trying add multiple logs files to my logstash for load all index data in kibana (according to my regular expression https://regex101.com/r/njG6Qq/2).
This is my /etc/logstash/conf.d/apache-01.conf
It appears not work because the index be dont shows in in kibana
input {
file {
path => "/var/lib/jenkins/workspace/GetLogs/(.+\.)?themaindomain\.com-ssl_log-.+[0-9]{4}$"
type => "apache_access"
sincedb_path => ["/var/lib/logstash/"]
start_position => "beginning"
}
}
Example of my logs files in contain in /var/lib/jenkins/workspace/GetLogs/ folder like my regex https://regex101.com/r/njG6Qq/2
somesubdomain.themaindomain.com-Nov-2018
somesubdomain.themaindomain.com-Oct-2018
somesubdomain.themaindomain.com-Sep-2018
somesubdomain.themaindomain.com-Sep-2018.gz.1
somesubdomain.themaindomain.com-ssl_log-Jan-2018
somesubdomain.themaindomain.com-ssl_log-Jan-2018.gz.1
somesubdomain.themaindomain.com-ssl_log-Nov-2018
somesubdomain.themaindomain.com-ssl_log-Oct-2018
somesubdomain.themaindomain.com-ssl_log-Sep-2018
somesubdomain.themaindomain.com-ssl_log-Sep-2018.gz.1
ftp.themaindomain.com-ftp_log-Mar-2018
ftp.themaindomain.com-ftp_log-Mar-2018.gz.1
ftp.themaindomain.com-ftp_log-Oct-2018
ftp.themaindomain.com-ftp_log-Sep-2018
merged.txt
somesubdomain.themaindomain.com-Oct-2018
somesubdomain.themaindomain.com-Sep-2018
somesubdomain.themaindomain.com-Sep-2018.gz.1
somesubdomain.themaindomain.com-ssl_log-Oct-2018
somesubdomain.themaindomain.com-ssl_log-Sep-2018
somesubdomain.themaindomain.com-ssl_log-Sep-2018.gz.1
OTHERsubdomain.themaindomain.com-Sep-2018
OTHERsubdomain.themaindomain.com-Sep-2018.gz.1
OTHERsubdomain.themaindomain.com-ssl_log-Sep-2018
OTHERsubdomain.themaindomain.com-ssl_log-Sep-2018.gz.1
somesubdomain.themaindomain.com-Nov-2018
somesubdomain.themaindomain.com-Oct-2018
somesubdomain.themaindomain.com-Sep-2018
somesubdomain.themaindomain.com-Sep-2018.gz.1
somesubdomain.themaindomain.com-ssl_log-Nov-2018
somesubdomain.themaindomain.com-ssl_log-Oct-2018
somesubdomain.themaindomain.com-ssl_log-Sep-2018
somesubdomain.themaindomain.com-ssl_log-Sep-2018.gz.1
OTHERsubdomain.themaindomain.com-Jun-2018
OTHERsubdomain.themaindomain.com-Jun-2018.gz.1
OTHERsubdomain.themaindomain.com-May-2018
OTHERsubdomain.themaindomain.com-May-2018.gz.1
OTHERsubdomain.themaindomain.com-ssl_log-Jun-2018
OTHERsubdomain.themaindomain.com-ssl_log-Jun-2018.gz.1
OTHERsubdomain.themaindomain.com-ssl_log-May-2018
OTHERsubdomain.themaindomain.com-ssl_log-May-2018.gz.1
somesubdomain.themaindomain.com-Nov-2018
somesubdomain.themaindomain.com-Oct-2018
somesubdomain.themaindomain.com-Sep-2018
how should I add the regex sentence to the logstash configure file?
Can someone explain me?
Thank you very much
You can only use filename patterns in path. In your case use /var/lib/jenkins/workspace/GetLogs/*.themaindomain.com-ssl_log-???-????
Documentation shows only filename patterns can be used - https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-path
For more information on file patterns see the below link
https://www.ibm.com/support/knowledgecenter/en/SSMKHH_10.0.0/com.ibm.etools.mft.doc/ac55200_.htm

how to deal with multine messages in log files using logstash

I am using beats for file path in logstash and I want to merge multiline messages into a single event, I am using codec in my config file but it is not working, I am getting an error as below
Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of
, { at line 8, column 8 (byte 169) after # The # character at the beginning of a li e indicates a comment. Use\n# cooments to describe
your configuration.\ninput {\n beats{\n\tport => \"5044\"\n
}\n \n codec ",

Get mounted disk space on Temp in Linux machine

I am new to perl world. I written one perl script for calculating free disk space. But whenever output generates, it gives me different number than what actually shows using df -h command.
So my requirement is i want to show specific mounted free disk space. E.g I want to show only /boot "Use%" figure and it should match with df -h command figure.
Please find my script for reference as follows by clicking link named Actual Script.
Actual Script
The df function from Filesys::Df
module returns a reference to a hash (perldoc perlreftut) with fs info fields
Example:
$VAR1 = {
user_bavail => '170614.21875',
user_blocks => '179796.8203125',
user_fused => 408762,
used => '9182.6015625',
fused => 408762,
bavail => '170614.21875',
user_used => '9182.6015625',
su_bavail => '180077.20703125',
ffree => 11863876,
fper => 3,
user_favail => 11863876,
favail => 11863876,
user_files => 12272638,
blocks => '189259.80859375',
su_favail => 11863876,
files => 12272638,
per => 5,
su_blocks => '189259.80859375',
bfree => '180077.20703125',
su_files => 12272638
};
So you free space is
my $ref = df($dir, 1);
print $ref->{bavail} . " bytes\n";

can anyone tell me what the encoding of this string is? Its meant to be base64

cpxSR2bnPUihaNxIFFA8Sc+8gUnWuJxJi8ywSW5ju0npWrFJHW2MSZAeMklcZ71IjrBySF2ci0gdecRI0vD/SM4ZF0m1ZSJJBY8bSZJl/0intaxIlQJBSPdY3EdBLM9Hp4wLSOK8Nki8L1pIoglxSAvNbkjHg0VIDlv7R6B2Y0elCqVGFWuVRgagAkdxHTdHELxRR9i2VkdyEUlHU84kRzTS2kalKFxG
This is a string from an XML file from my mass spectrometer. I am trying to write a program to load two such files, subtract one set of values from another, and write the results to a new file. According to the specification file for the .mzML format, the encoding of the numerical data is alleged to be base64. I can't convert this data string to anything legible using any of the many online base64 converter or using NotepaD++ and the MIME toolkit's base64 converter.
The string, in the context of the results file, looks like this:
<binaryDataArray encodedLength="224">
<cvParam cvRef="MS" accession="MS:1000515" name="intensity array" unitAccession="MS:1000131" unitName="number of counts" unitCvRef="MS"/>
<cvParam cvRef="MS" accession="MS:1000521" name="32-bit float" />
<cvParam cvRef="MS" accession="MS:1000576" name="no compression" />
<binary>cpxSR2bnPUihaNxIFFA8Sc+8gUnWuJxJi8ywSW5ju0npWrFJHW2MSZAeMklcZ71IjrBySF2ci0gdecRI0vD/SM4ZF0m1ZSJJBY8bSZJl/0intaxIlQJBSPdY3EdBLM9Hp4wLSOK8Nki8L1pIoglxSAvNbkjHg0VIDlv7R6B2Y0elCqVGFWuVRgagAkdxHTdHELxRR9i2VkdyEUlHU84kRzTS2kalKFxG</binary>
I can't proceed until I can work out what format this encoding is meant to be!
Thanks in advance for any replies.
You can use this trivial program to convert it to plaintext:
#include <stdio.h>
int main(void)
{
float f;
while (fread(&f, 1, 4, stdin) == 4)
printf("%f\n", f);
}
I compiled this to "floatdecode" and used this command:
echo "cpxSR2bnPUihaNxIFFA8Sc+8gUnWuJxJi8ywSW5ju0npWrFJHW2MSZAeMklcZ71IjrBySF2ci0gdecRI0vD/SM4ZF0m1ZSJJBY8bSZJl/0intaxIlQJBSPdY3EdBLM9Hp4wLSOK8Nki8L1pIoglxSAvNbkjHg0VIDlv7R6B2Y0elCqVGFWuVRgagAkdxHTdHELxRR9i2VkdyEUlHU84kRzTS2kalKFxG" | base64 -d | ./floatdecode
Output is:
53916.445312
194461.593750
451397.031250
771329.250000
1062809.875000
1283866.750000
1448337.375000
1535085.750000
1452893.125000
1150371.625000
729577.000000
387898.875000
248514.218750
285922.906250
402376.906250
524166.562500
618908.875000
665179.312500
637168.312500
523052.562500
353709.218750
197642.328125
112817.929688
106072.507812
142898.609375
187123.531250
223422.937500
246822.531250
244532.171875
202255.109375
128694.109375
58230.625000
21125.322266
19125.541016
33440.023438
46877.441406
53692.062500
54966.843750
51473.445312
42190.324219
28009.101562
14090.161133
Yet another Java Base64 decode with options to uncompress should you need it.
Vendor spec indicated "32-bit float" = IEEE-754 and specified little-endian.
Schmidt's converter shows the bit pattern for IEEE-754.
One more Notepad++ step to look at the hex codes:
Notepad++ TextFX plugin (after the Base64 decode you already did)
select the text
TextFX > TextFX Convert > Convert text to Hex-32
lets you look at the hex codes:
"000000000 72 9C 52 47 66 E7 3D 48- ... 6E 63 BB 49 |rœRGfç=H¡hÜHP
Little-endian: 47529C72 converts (via Schmidt) as shown above by David.
You can access such data from mzML files in Python through pymzML, a python interface to mzML files.
http://pymzml.github.com/

Resources