Is there any way to get fields from following message event using grok pattern
fields i have to parse in following and store in elastic as below
timestamp => NET-Log-Time
device_ip => NAS-IP-Address
message => message
Message =>
" Tue Nov 22 15:20:33 2011
NET-Log-Time = "Tue Nov 22 15:20:31 2011\n"
Acct-Status-Type = Start
NET-Session-Ingress-CallId = 0
NET-Session-Egress-CallId = 65535
NET-Session-Generic-Id = 14
Acct-Multi-Session-Id = "132200403114"
NET-Ingress-Signaling-Group = 19992
NET-Egress-Signaling-Group = 0
NET-Ingress-Channel-Number = 23
NET-Egress-Channel-Number = 0
NET-Call-Origin = 1
NET-Calling-Number = "1029"
NET-Called-Number = "2006"
NET-Calling-Name = "System Test"
NET-Ingress-Channel-Id = "1:1:23"
NET-Setup-Time = "1322004031"
Acct-Session-Id = "0"
NET-Firmware-Version = "2.0.0v107"
NET-Local-Time-Zone = "PST8PDT"
NET-Gw-Id = "f5a4fdc8f970ae35addf36"
NET-Time-And-Day = "1322004031"
NAS-Port = 1813
Acct-Delay-Time = 0
NAS-IP-Address = 134.56.227.230
Acct-Unique-Session-Id = "df05da217dbe16e4"
Timestamp = 1322004033
Request-Authenticator = Verified"
Using %{GREEDYDATA:messageBody} i could extract
{
"messageBody": "Mon Jun 14 18:08:54 2021"
}
I'm doing with radius too. I have to merge all these lines first,
you can do it with multiline either on the input codec / logstash filter(terminated). multiline-codec
This one is perfect example for you
input {
stdin {
codec => multiline {
pattern => "^\s"
what => "previous"
}
}
}
pattern => "^\s"
This match the beginning space of every line
what => "previous"
This is the action, what will you do if the line match the pattern. In this example will be appended to the previous line.
With your log sample I think the pattern would be ^\s{2,} since you only want to merge those who have beginning 2 or more space.
Those appended to the other message have separator \n
Related
I would like to know is there any way to set Header with 60 char in SALV? Since there is a CHAR limit of 40 when I try using SET_LONG_TEXT method. I am using this SALV to both display and download it as an excel too.
My Prod Code looks like this below.
TRY.
" Call SALV Factory Method
cl_salv_table=>factory(
EXPORTING
list_display = abap_false
IMPORTING
r_salv_table = DATA(lo_salv)
CHANGING
t_table = <fs_global_factor_data> ).
DATA(lo_columns) = lo_salv->get_columns( ).
lo_columns->set_optimize( ).
DATA(lo_col_ref) = lo_columns->get( ).
" Get the Field Description in the table
lr_strucdesc ?= cl_abap_typedescr=>describe_by_data( ls_struct ).
/stl/cl_gf_validations=>at_gf_table_struct = lr_strucdesc->get_ddic_field_list( ).
Here, I am replacing the long text with my own texts with a length of 60 Char from the AT_FIELD_HEADER table.
" Remove Short and Medium texts and Replace long texts from AT_FIELD_HEADER Table
LOOP AT lo_col_ref ASSIGNING FIELD-SYMBOL(<fs_col_ref>).
<fs_col_ref>-r_column->set_short_text( space ).
<fs_col_ref>-r_column->set_medium_text( space ).
<fs_col_ref>-r_column->set_fixed_header_text( lc_char_l ) .
<fs_col_ref>-r_column->set_long_text( SWITCH #( <fs_col_ref>-columnname+0(5) WHEN /stl/cl_gf_validations=>ac_desc_field_prefix THEN /stl/cl_gf_validations=>ac_fd_description
ELSE CONV #( LET line = VALUE /stl/cl_gf_validations=>ty_field_header( /stl/cl_gf_validations=>at_field_header[ fieldname = <fs_col_ref>-columnname ] OPTIONAL )
IN COND #( WHEN line-title IS NOT INITIAL THEN line-title ELSE line-description )
) ) ).
ENDLOOP.
DATA(lv_xml_type) = if_salv_bs_xml=>c_type_xlsx.
DATA(lv_data_in_xstring) = lo_salv->to_xml( xml_type = lv_xml_type ).
CATCH cx_root.
MESSAGE e033(/stl/bw_gf_msg).
ENDTRY.
" Convert XSTRING TO BIN
CALL FUNCTION 'SCMS_XSTRING_TO_BINARY'
EXPORTING
buffer = lv_data_in_xstring
IMPORTING
output_length = lv_file_size
TABLES
binary_tab = lt_data_in_bin.
Here, GUI_DOWNLOAD is used to download it as an excel file.
" Download the Excel
cl_gui_frontend_services=>gui_download(
EXPORTING
bin_filesize = lv_file_size
filename = lv_fullpath
filetype = lc_filetype_bin
CHANGING
data_tab = lt_data_in_bin
EXCEPTIONS
file_write_error = 1
no_batch = 2
gui_refuse_filetransfer = 3
invalid_type = 4
no_authority = 5
unknown_error = 6
header_not_allowed = 7
separator_not_allowed = 8
filesize_not_allowed = 9
header_too_long = 10
dp_error_create = 11
dp_error_send = 12
dp_error_write = 13
unknown_dp_error = 14
access_denied = 15
dp_out_of_memory = 16
disk_full = 17
dp_timeout = 18
file_not_found = 19
dataprovider_exception = 20
control_flush_error = 21
not_supported_by_gui = 22
error_no_gui = 23
OTHERS = 24
).
IF sy-subrc <> 0.
MESSAGE ID sy-msgid TYPE sy-msgty NUMBER sy-msgno
WITH sy-msgv1 sy-msgv2 sy-msgv3 sy-msgv4.
ENDIF.
Also, I am using the same piece of code to display the SALV where header gets truncated to 40 char.
I'm processing through Telegram history (txt file) reading it line by line when I when there is a pattern
Free_Trade_Calls__AltSignals:* / * (*)
example
Free_Trade_Calls__AltSignals:GAS / BTC (binance)
then store "following lines" in a separate variables each (and also store that that traiding pair RDN/ BTC and exchange bittrex) in separate variables) until it hits timestamp (today's date), then search for the pattern again
but when I see pattern
Free_Trade_Calls__AltSignals:TARGET
proceed to next line
whole example
Free_Trade_Calls__AltSignals:IOC/ BTC (bittrex)
BUY : 0.00164
SELL :
TARGET 1 : 0.00180
TARGET 2 : 0.00205
TARGET 3 : 0.00240
STOP LOS : 0.000120
2018-04-19 15:46:57 Free_Trade_Calls__AltSignals:TARGET
I've got some incomplete pseudocode but it is mostly BS anyway - quite new to parsing text aynway here is
let now = new Date();
// var diff = (now.getTime() - date.getTime()) / 60000; // diff in min
const fs = require('fs'),
path = require('path'),
filePath = path.join(__dirname, 'chats.txt'),
// myString = "Free_Trade_Calls__AltSignals";
buy,
sell = [3],
sl;
let lines = [];
const buf = fs.readFileSync(filePath)
lines [] = buf.toString().split('\n'){
for (line < lines.length){
if (!err) {
if(line.indexOf("Trade_Calls__AltSignals: * / * (*)") > -1) {
currency = line.indexOf(2);
exchange = line.indexOf(4);
nextline();
while (line.indexOf($new.Date) < - 1){ // no containing Date
if (line.indexOf(buy) > -1 ){
line = buy
}
if (line.indexOf(buy) > -1 ){
}
}else{
nextline();
}
}
}
}
}
I have a 3-year data in a string tableformat.txt. Three of its lines are given below:
12-13 Jan -10.5
14-15 Jan -9.992
15-16 Jan -8
How to change the 3rd column (-10.5, -9.992 and -8) of string to be (-10.500, -9.992 and -8.000) of number?
I have made the following script:
clear all; clc;
filename='tableformat.txt';
fid = fopen(filename);
N = 3;
for i = [1:N]
line = fgetl(fid)
a = line(10:12);
na = str2num(a);
ma(i) = na;
end
ma
which gives:
ma = -1 -9 -8
When I did this change: a = line(10:15);, I got:
Error message: Index exceeds matrix dimensions.
This will work for you.
clear all;
clc;
filename='tableformat.txt';
filename2='tableformat2.txt';
fid = fopen(filename);
fid2 = fopen(filename2,'w');
formatSpec = '%s %s %6.4f\n';
N = 3;
for row = [1:N]
line = fgetl(fid);
a = strsplit(line,' ');
a{3}=cellfun(#str2num,a(3));
fprintf(fid2, formatSpec,a{1,:});
end
fclose(fid);
fclose(fid2);
Not sure how to ask this question, thus I don't know how to search for it on google or SO. Let me just show you the given data. By the way, this is just an Awk exercise, its not homework. Been trying to solve this off and on for 2 days now. Below is an example;
Mon Sep 15 12:17:46 1997
User-Name = "wynng"
NAS-Identifier = 207.238.228.11
NAS-Port = 20104
Acct-Status-Type = Start
Acct-Delay-Time = 0
Acct-Session-Id = "239736724"
Acct-Authentic = RADIUS
Client-Port-DNIS = "3571800"
Framed-Protocol = PPP
Framed-Address = 207.238.228.57
Mon Sep 15 12:19:40 1997
User-Name = "wynng"
NAS-Identifier = 207.238.228.11
NAS-Port = 20104
Acct-Status-Type = Stop
Acct-Delay-Time = 0
Acct-Session-Id = "239736724"
Acct-Authentic = RADIUS
Acct-Session-Time = 115
Acct-Input-Octets = 3915
Acct-Output-Octets = 3315
Acct-Input-Packets = 83
Acct-Output-Packets = 66
Ascend-Disconnect-Cause = 45
Ascend-Connect-Progress = 60
Ascend-Data-Rate = 28800
Ascend-PreSession-Time = 40
Ascend-Pre-Input-Octets = 395
Ascend-Pre-Output-Octets = 347
Ascend-Pre-Input-Packets = 10
Ascend-Pre-Output-Packets = 11
Ascend-First-Dest = 207.238.228.255
Client-Port-DNIS = "3571800"
Framed-Protocol = PPP
Framed-Address = 207.238.228.57
So the log file contains the above data for various users. I specifically pasted this to show that this user had a login, Acct-Status-Type = Start, and a logoff, Acct-Status-Type = Stop. This counts as one session. Thus I need to generate the following output.
User: "wynng"
Number of Sessions: 1
Total Connect Time: 115
Input Bandwidth Usage: 83
Output Bandwidth Usage: 66
The problem I have is keeping the info somehow attached to the user. Each entry in the log file has the same information when the session is in Stop so I cant just regex
/Acct-Input-Packets/{inPackets =$3}
/Acct-Output-Packets/{outPackets = $3}
Each iteration through the data will overwrite the past values. What I want to do is if I find a User-Name entry and this entry has a Stop, then I want to record for that user, the input/output packet values. This is where I get stumped.
For the session values I was thinking of saving the User-Names in an array and then in the END{} count the duplicates and divide by 2 those that are greater than 2 if even. If odd then divide by two then floor it.
I don't necessarily want the answer but maybe some hints/guidance or perhaps a simple example on which I could expand on.
You can check each line for :
a date pattern : /\w+\s\w+\s[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s[0-9]{4}/
user name value : /User-Name\s+=\s+\"\w+\"/
status value : /Acct-Status-Type\s+=\s+\w+/
input packet value : /Acct-Input-Packets\s+=\s[0-9]+/
output packet value : /Acct-Output-Packets\s+=\s[0-9]+/
an empty line : /^$/
Once you have defined what you are looking for (above pattern), it's just a matter of conditions and storing all those data in some array.
In the following example, I store each value type above in a dedicated array for each type with a count index that is incremented when an empty line /^$/ is detected :
awk 'BEGIN{
count = 1;
i = 1;
}{
if ($0 ~ /\w+\s\w+\s[0-9]{2}\s[0-9]{2}:[0-9]{2}:[0-9]{2}\s[0-9]{4}/){
match($0, /\w+\s(\w+)\s([0-9]{2})\s([0-9]{2}):([0-9]{2}):([0-9]{2})\s([0-9]{4})/, n);
match("JanFebMarAprMayJunJulAugSepOctNovDec",n[1])
n[1] = sprintf("%02d",(RSTART+2)/3);
arr[count]=mktime(n[6] " " n[1] " " n[2] " " n[3] " " n[4] " " n[5]);
order[i]=count;
i++;
}
else if ($0 ~ /User-Name\s+=\s+\"\w+\"/){
match($0, /User-Name\s+=\s+\"(\w+)\"/, n);
name[count]=n[1];
}
else if ($0 ~ /Acct-Status-Type\s+=\s+\w+/){
match($0, /Acct-Status-Type\s+=\s+(\w+)/, n);
status[count]=n[1];
}
else if ($0 ~ /^$/){
count++;
}
else if ($0 ~ /Acct-Input-Packets\s+=\s[0-9]+/){
match($0, /Acct-Input-Packets\s+=\s([0-9]+)/, n);
input[count]=n[1];
}
else if ($0 ~ /Acct-Output-Packets\s+=\s[0-9]+/){
match($0, /Acct-Output-Packets\s+=\s([0-9]+)/, n);
output[count]=n[1];
}
}
END{
for (i = 1; i <= length(order); i++) {
val = name[order[i]];
if (length(user[val]) == 0) {
valueStart = "0";
if (status[order[i]] == "Start"){
valueStart = arr[order[i]];
}
user[val]= valueStart "|0|0|0|0";
}
else {
split(user[val], nameArr, "|");
if (status[order[i]]=="Stop"){
nameArr[2]++;
nameArr[3]+=arr[order[i]]-nameArr[1]
}
else if (status[order[i]] == "Start"){
# store date start
nameArr[1] = arr[order[i]];
}
nameArr[4]+=input[order[i]];
nameArr[5]+=output[order[i]];
user[val]= nameArr[1] "|" nameArr[2] "|" nameArr[3] "|" nameArr[4] "|" nameArr[5];
}
}
for (usr in user) {
split(user[usr], usrArr, "|");
print "User: " usr;
print "Number of Sessions: " usrArr[2];
print "Total Connect Time: " usrArr[3];
print "Input Bandwidth Usage: " usrArr[4];
print "Output Bandwidth Usage: " usrArr[5];
print "------------------------";
}
}' test.txt
The values are extracted with match function like :
match($0, /User-Name\s+=\s+\"(\w+)\"/, n);
For the date, we have to parse the month string part, I've used the solution in this post to extract like :
match($0, /\w+\s(\w+)\s([0-9]{2})\s([0-9]{2}):([0-9]{2}):([0-9]{2})\s([0-9]{4})/, n);
match("JanFebMarAprMayJunJulAugSepOctNovDec",n[1])
n[1] = sprintf("%02d",(RSTART+2)/3);
All the processing of the collected values is done in the END clause where we have to group the values, I create a user array with the username as key and as value a concatenation of all your different type delimited by | :
[startDate] "|" [sessionNum] "|" [connectionTime] "|" [inputUsage] "|" [outputUsage]
With this data input (your data extended), it gives :
User: TOTO
Number of Sessions: 1
Total Connect Time: 114
Input Bandwidth Usage: 83
Output Bandwidth Usage: 66
------------------------
User: wynng
Number of Sessions: 2
Total Connect Time: 228
Input Bandwidth Usage: 166
Output Bandwidth Usage: 132
------------------------
Here is my sphinx search configuration (sphinxsearch_0.9.9-6_amd64):
index FULL
{
charset_type = utf-8
source = FULL
path = /var/sphinx/data/Full
docinfo = extern
mlock = 0
min_stemming_len = 1
min_prefix_len = 1
min_word_len = 1
html_strip = 1
index_exact_words = 1
}
searchd
{
listen = 192.168.2.3
log = /var/log/sphinxsearch/searchd.log
query_log = /var/log/sphinxsearch/query.log
read_timeout = 3
client_timeout = 60
max_children = 30
pid_file = /var/run/searchd.pid
max_matches = 1000
seamless_rotate = 1
preopen_indexes = 0
unlink_old = 1
mva_updates_pool = 1M
max_packet_size = 8M
max_filters = 256
max_filter_values = 4096
}
I use php as client
$sphinx_client->SetServer('localhost', 9312);
$sphinx_client->SetConnectTimeout(1);
$sphinx_client->SetArrayResult(true);
$sphinx_client->setRankingMode(SPH_RANK_WORDCOUNT);
$sphinx_client->SetMatchMode(SPH_MATCH_EXTENDED2);
if ($mode == 'all') {
$sphinx_client->SetSortMode(SPH_SORT_RELEVANCE, 'category');
} else {
$sphinx_client->setFilter('category', array($this->_filter_category), FALSE);
}
$sphinx_client->SetLimits(0, $this->_limit);
$results = $sphinx_client->Query('"^'.$query.'$"', 'FULL');
for example i have those names in index :
1. Alex
2. Alen
3. George
4. A
5. G
::: When i try to search for simple 1 char string "A" i get Alen / Alex / A and so on.
How can i search based on string length so i can display them in right order like :
A / Alen / Alex ...
I also get "WARNING: index 'FULL': no morphology, index_exact_words=1 has no effect, ignoring"
Best Regards
use an ordinal field ( str2ordinal ) , do your normal search , but modify sort mode : switch to extended mode and use a combination like $sphinx_client->SetSortMode(SPH_SORT_EXTENDED, '#weight desc , myordinal asc');