splunk extraction from the log entries - python-3.x

Need to extract payload data from logs entries and extract the PlatformVersion and PlatformClient values. Need in python code.
"tracking~2015~526F3D98","2015:1302",164,1,"2022-02-07 11:10:08.744 INFO [threadPoolTaskExecutorTransformed5 - ?] saving event to log =core-server-event-tracking-api, payload={""PlatformVersion"":""6.34.36 - 4.18.6"",""PlatformClient"":""html""},53
"tracking~2015~526F3D98","2015:130",164423,1,"2022-02-07 11:10:08.744 INFO [threadPoolTaskExecutorTransformed5 - ?] saving event to log =core-server-event-tracking-api, payload={""PlatformVersion"":""6.34.37 - 4.18.7"",""PlatformClient"":""xml""},54

Not sure how Python and Splunk are relating here - but this is just a matter of doing some field extractions.
Something like this should do it:
index=ndx sourcetype=srctp
| field field=_raw "PlatformVersion\W+(?<platform_version>[^\"]+)"
| rex field=_raw "PlatformClient\W+(?<platform_client>[^\"]+)"

Related

How to push complex legacy logs into logstash?

I'd like to use ELK to analyze and visualize our GxP Logs, created by our stoneold LIMS system.
At least the system runs on SLES but the whole logging structure is some kind of a mess.
I try to give you an impression:
Main_Dir
| Log Dir
| Large number of sub dirs with a lot of files in them of which some may be of interest later
| Archive Dir
| [some dirs which I'm not interested in]
| gpYYMM <-- subdirs created automatically each month: YY = Year ; MM = Month
| gpDD.log <-- log file created automatically each day.
| [more dirs which I'm not interested in]
Important: Each medical examination, that I need to track, is completely logged in the gpDD.log file that represents the date of the order entry. The duration of the complete examination varies between minutes (if no material is available), several hours or days (e.g. 48h for a Covid-19 examination) or even several weeks for a microbiological sample. Example: All information about a Covid-19 sample, that reached us on December 30th is logged in ../gp2012/gp30.log even if the examination was made on January 4th and the validation / creation of report was finished on January 5th.
Could you please provide me some guidance of the right beat to use ( I guess either logbeat or filebeat) and how to implement the log transfer?
Logstash file input:
input {
file {
path => "/Main Dir/Archive Dir/gp*/gp*.log"
}
}
Filebeat input:
- type: log
paths:
- /Main Dir/Archive Dir/gp*/gp*.log
In both cases the path is possible, however if you need further processing of the lines, I would suggest using at least Logstash as a passthrough (using beats input if you do not want to install Logstash on the source itself, which can be understood)

what are the usual problems that we face with sincedb in logstash

I am using ELK stack, so using file input plugin in logstash i am working on it
at first i used file*.txt to match with file pattern
later i used masterfile.txt as a single file which has the data of all matching patterns
and now i am going back to file*.txt , but here i see the problem- I am seeing the data on kibana which is the date after the file*.txt is replaced with masterfile.txt but not the history,
I feel like i must understand the behavior of sincedb logstash here
also a possible solution to get the history data
Logstash stores information about the position of the last byte read in the file that contains the logs with sincedb_path. During the execution, Logstash starts reading the input file from the mentioned position.
Take into account 'start_position' and the name of the index ( Logstash -> output) if you want to create a new index with different logs.
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-sincedb_path

YARN resource manager message losgstash

I am trying to generate pie chart based on the final status of the job from RM logs. Since "RESULT=" string is in the "message" of the log, we are not able to extract it.
After doing some research we came to know that we need to write a grok pattern to break this string and extract "RESULT=".
Here is the string that I want to break, I want to extract user and result from this.
message:2017-02-28 21:24:44,223 INFO resourcemanager.RMAuditLogger (RMAuditLogger.java:logSuccess(191)) - USER=test1 OPERATION=Application Finished - Succeeded TARGET=RMAppManager RESULT=SUCCESS APPID=application_1486072728057_33195

colorful console output by unix pipe filter

while examining the console output and logging messages of different software it is sometimes difficult to keep the overview. It would be much easier to make the output colorful and highlight the text phrases which are currently important.
Is there a program for Linux/UNIX shell which could be used as a filter by utilizing unix pipes to make the console output colorful according to predefined patterns and colors?
p.ex.
pattern definition:
INFO=green
WARN=yellow
ERROR=red
\d+=lightgreen
to highlight the severity of the message and also numbers.
usage:
$ chatty_software | color_filter
11:41:21.000 [green:INFO] runtime.busevents - SensorA state updated to [lightgreen:17]
11:41:21.004 [green:INFO] runtime.busevents - SensorB state updated to [lightgreen:20]
original output:
11:41:21.000 INFO runtime.busevents - SensorA state updated to 17
11:41:21.004 INFO runtime.busevents - SensorB state updated to 20
we use a sed script along these lines:
s/.* error .*/^[[31m&^[[0m/
t done
s/.* warning .*/^[[33m&^[[0m/
t done
:done
and invoke it by
sed -f log_color.sed
I guess you could do something similar?

How to check logs in this case ( One log file for all the dates )

How to check logs in this case .
Actually from our Application , we are contacting some third party Service (Application )
The way they provided their log file is different . (Only one log file no matter what the date is )
For instance , this looks this way
11:29:32,862 - INFO main:http-8082-2 <ServicePlugger> <gurthu>DE</gurthu>
11:29:32,862 - INFO main:http-8082-2 <ServicePlugger> <enni>0</enni>
11:29:32,862 - INFO main:http-8082-2 <ServicePlugger> <konadate>0</konadate>
11:29:32,862 - INFO main:http-8082-2 <ServicePlugger> <costentha>0</costentha>
Now my question is how to check log files ??
Whether the starting lines indicate time (11:29:32,862) ??
Thank you .
I am not sure if I understand your question correctly. Can you not use regular expression to match the string?

Resources