I am trying to use logstash's KV plugin. I have following log format:
time taken for transfer for all files in seconds=23 transfer start time= 201708030959 transfer end time = 201708030959
My .conf file has following KV plugin:
filter {
kv {
value_split => "="
}
}
When I run logstash, it parses complete log file line by line excluding the one having "=". I need seconds, start time and end time to be separated as key value pairs. Please suggest.
Related
I have an application that generates a txt file with thousands of lines. I have to delete some lines manually by going through the file (using vim). However, I might need to generate the same file again if a change in format is required. That will make me go through the file again to delete the same lines.
The solution to avoid deleting manually repeatedly is that vim somehow logs the line number when I delete a line. I can then use some script to remove those lines. Is it possible to get this behavior in vim?
Otherwise, is there any other editor to get this behavior? There are many lines I have to delete and it's not feasible for me to log each line number manually.
As suggested by phd and wxz, I was able to use git-diff of the file to extract the deleted lines by using node package gitdiff-parser for parsing the diff.
const gitDiffParser = require('gitdiff-parser')
const { exec } = require("child_process");
let p = new Promise( (res,rej) => {
exec("git diff -U0 file.txt", (error, stdout) => {
res(stdout)
});
});
p.then(s=>{
diff = gitDiffParser.parse(s);
diff[0].hunks.forEach(element => {
console.log(`start: ${element.oldStart}, end: ${element.oldStart + element.oldLines - 1}`)
});
})
Another solution or say hack was to append line number in each line of the file and extract the undeleted line numbers after removing the required lines.
I am fairly new to puppet, I am trying to pass a TimeOut value in 'httpd.conf' file using the below code snippet in my puppet 'config.pp'
replace { 'Httpd Timeout':
file => 'httpd.conf',
pattern => 'TimeOut*',
replacement => 'TimeOut 900',
}
But instead of replacing the code just appends the value "900" to the line like below
"TimeOut 900 300 "
How I can modify the code that the line will be like below
TimeOut 900
I am using Puppet community version 3.7
I am working with ANSI 835 plain text files and am looking to capture all data in segments which start with “BPR” and end with “TRN” including those markers. A given file is a single line; within that line the segment can, but not always, repeats. I am running the process on multiple files at a time and ideally I would be able to record the file name in which the segment(s) occur.
Here is what I have so far, based on an answer to another question:
#!/bin/sed -nf
/BPR.*TRN/ {
s/.*\(BPR.*TRN\).*/\1/p
d
}
/from/ {
: next
N
/BPR/ {
s/^[^\n]*\(BPR.*TRN\)[^n]*/\1/p
d
}
$! b next
}
I run all files I have through this and write the results to a file which looks like this:
BPR*I*393.46*C*ACH*CCP*01*011900445*DA*0000009046*1066033492**01*071923909*DA*72
34692932*20150120~TRN
BPR*I*1611.07*C*ACH*CCP*01*031100209*DA*0000009108*1066033492**01*071923909*DA*7
234692932*20150122~TRN
BPR*I*1415.25*C*CHK************20150108~TRN
BPR*H*0*C*NON************20150113~TRN
BPR*I*127.13*C*CHK************20150114~TRN
BPR*I*22431.28*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*72346
92932*20150112~TRN
BPR*I*182.62*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*7234692
932*20150115~TRN
Ideally each line would be prepended with the file name like this:
IDI.Aetna.011415.64539531.rmt:BPR*I*393.46*C*ACH*CCP*01*011900445*DA*0000009046*1066033492**01*071923909*DA*72
34692932*20150120~TRN
IDI.BCBSIL.010915.6434438.rmt:BPR*I*1611.07*C*ACH*CCP*01*031100209*DA*0000009108*1066033492**01*071923909*DA*7
234692932*20150122~TRN
IDI.CIGNA.010215.64058847.rmt:BPR*I*1415.25*C*CHK************20150108~TRN
IDI.GLDRULE.011715.646719.rmt:BPR*H*0*C*NON************20150113~TRN
IDI.MCREIN.011915.6471442.rmt:BPR*I*127.13*C*CHK************20150114~TRN
IDI.UHC.011915.64714417.rmt:BPR*I*22431.28*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*72346
92932*20150112~TRN
IDI.UHC.011915.64714417.rmt:BPR*I*182.62*C*ACH*CCP*01*071000152*DA*99643*1361236610**01*071923909*DA*7234692
932*20150115~TRN
The last two lines would be an example of a file where the segment pattern repeats.
Again, prepending each line with the file name is ideal. What I really need is to be able to process a given single-line file which has the “BPR…TRN” segment repeating and write all segments in that file to my output file.
Try with awk:
awk '
/BPR/ { sub(".*BPR","BPR") }
/TRN/ { sub("TRN.*","TRN") }
/BPR/,/TRN/ { print FILENAME ":" $0 }
' *.rmt
I know there's no built in "line count" functionality while processing files through logstash (for various, understandable and documented reasons). But - there should be a mechanism, within any given logstash instance - to have an monotonically increasing variable / count for every parsed line.
I don't want to go the metrics route since it's a continuous polling mechanism (every n-seconds). Alternatives include pre-processing of log files which given my particular use case - is unacceptable.
Again, let me reiterate - I need the ability to generate/read a monotonically increasing variable that I can store during in a logstash filter.
Thoughts?
here's nothing built into logstash to do it.
You can build a filter to do it pretty easily
Just drop something like this into lib/logstash/filters/seq.rb
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "set"
#
# This filter will adds a sequence number to a log entry
#
# The config looks like this:
#
# filter {
# seq {
# field => "seq"
# }
# }
#
# The `field` is the field you want added to the event.
class LogStash::Filters::Seq < LogStash::Filters::Base
config_name "seq"
milestone 1
config :field, :validate => :string, :required => false, :default => "seq"
public
def register
# Nothing
end # def register
public
def initialize(config = {})
super
#threadsafe = false
# This filter needs to keep state.
#seq=1
end # def initialize
public
def filter(event)
return unless filter?(event)
event[#field] = #seq
#seq = #seq + 1
filter_matched(event)
end # def filter
end # class LogStash::Filters::Seq
This will start at 1 every time Logstash is restarted, but for most situations, this would be ok. If you need something that is persistent across restarts, you need to do a bit more work to persist it somewhere
For anyone finding this in 2018+: logstash now has a ruby filter that makes this much simpler. Put the following in a file somewhere:
# encoding: utf-8
def register(params)
#seq = 1
end
def filter(event)
event.set("seq", #seq)
#seq += 1
return [event]
end
And then configure it like this in your logstash.conf (substitute in the filename you used):
ruby {
path => "/usr/local/lib/logstash/seq.rb"
}
It would be pretty easy to make the field name configurable from logstash.conf, but I'll leave that as an exercise for the reader.
I suspect this isn't thread-safe, so I'm running only a single logstash worker.
this is another choice to slove the problem,this work for me,thanks to the answer from the previous person about thread safe. i use seq field to sort my desc
this is my configure
logstash.conf
filter {
ruby {
code => 'event.set("seq", Time.now.strftime("%N").to_i)'
}
}
logstash.yml
pipeline.batch.size: 200
pipeline.batch.delay: 60
pipeline.workers: 1
pipeline.output.workers: 1
I'm reading up on Log4perl and want to try and use it for simple log management of my Perl scripts running on a Linux box. I've also read up on newsyslog and logrotate but want to use Log4perl if at all possible.
I'm trying to configure the /etc/log4perl.conf file so that it:
Defines a widget logger (INFO level) that will write all output to /opt/myapp/logs/myapp-<datetime>.log, where <datetime> is a date/time formatted string like 2012-12-20
This myapp-<datetime>.log file needs to be rotated daily (preferably at midnight), where the old file is deleted, and a new file is created with <datetime> + 1. For instance, myapp-2012-12-20.log would be replaced with myapp-2012-12-21.log, etc.
Here's my best attempt which I believe is close, but is still missing some configuration:
#####/etc/log4perl.conf############################################################
log4perl.logger.widget = INFO, MyAppLogAppender
log4perl.appender.MyAppLogAppender = Log::Log4perl::Appender::File
log4perl.appender.MyAppLogAppender.filename = /opt/myapp/logs/myapp-???.log
log4perl.appender.MyAppLogAppender.layout = Log::Log4perl::Layout::SimpleLayout
###################################################################################
How do I configure log4perl.appender.MyAppLogAppender to rotate once a day, delete the old file, and create a new one with a correct timestamp? Thanks in advance.
Here's an example of a Log::Log4perl configuration file, defining a daily rollover at midnight (date pattern yyyy-MM-dd), keeping a maximum of 5 saved logfiles around, at WARN level, and dumping everything to screen:
log4perl.logger = TRACE, Screen, Logfile
log4perl.appender.Logfile = Log::Dispatch::FileRotate
log4perl.appender.Logfile.Threshold = WARN
log4perl.appender.Logfile.filename = test.log
log4perl.appender.Logfile.max = 5
log4perl.appender.Logfile.DatePattern = yyyy-MM-dd
log4perl.appender.Logfile.TZ = PST
log4perl.appender.Logfile.layout = Log::Log4perl::Layout::PatternLayout
log4perl.appender.Logfile.layout.ConversionPattern = %d %m %n
log4perl.appender.Screen = Log::Log4perl::Appender::Screen
log4perl.appender.Screen.stderr = 0
log4perl.appender.Screen.utf8 = 1
log4perl.appender.Screen.layout = Log::Log4perl::Layout::PatternLayout::Multiline
log4perl.appender.Screen.layout.ConversionPattern = [%p] %m %n
(reference: https://metacpan.org/module/Log::Log4perl::FAQ#How-can-I-roll-over-my-logfiles-automatically-at-midnight-)
There is a gotcha in Log::Dispatch::FileRotate, if your daily job is run later in the day (say 23:00) and takes 2h (so ends at about 01:00), the log rotation will never happens with a "day" pattern like :
log4perl.appender.Logfile.DatePattern = yyyy-MM-dd
A simple workaround is to use an "hourly" pattern like this :
log4perl.appender.Logfile.DatePattern = yyyy-MM-dd-HH
So when the next run starts at 23:00, the log file get rotated as more than one hour has passed.