I am using Telegraf to get logs information from specific logs with Hexa data.
I am using tail but i still to get the same error : Invalid data format: Grok.
My log look like this :
18/08/2022 21:04:23 01 41 7B 00 04 14 00 00 00 FD AB
and a configuration for tail in telegraf :
[[inputs.tail]]
files = ["/mnt/cle/*a.*.log"]
from_beginning = true
max_undelivered_lines = 300
character_encoding = "utf-8"
data_format = "Grok"
grok_patterns = ['%{DATE_EU:date} %{TIME:time} %{WORD:my1id} %{WORD:my2id} %{BASE16NUM:01hexa} %{BASE16NUM:02hexa} %{BASE16NUM:03hexa} %{BASE16NUM:04hexa} %{BASE16NUM:05hexa} %{BASE16NUM:06hexa} %{BASE16NUM:07hexa} %{BASE16NUM:08hexa} %{BASE16NUM:09hexa} %{BASE16NUM:10hexa} %{BASE16NUM:11hexa}']
I try also for grok_patterns :
grok_patterns = ['%{TIMESTAMP_ISO8601:timestamp:ts-"2006/01/02 15:04:05"} %{WORD:MRIid} %{WORD:OPUid} %{WORD:01hexa} %{WORD:02hexa} %{WORD:03hexa} %{WORD:04hexa} %{WORD:05hexa} %{WORD:06hexa} %{WORD:07hexa} %{WORD:08hexa} %{WORD:09hexa} %{WORD:10hexa} %{WORD:11hexa}']
Also, i would like to convert my hexa data to decimal and apply a conversion formula.
And to complicate things, i vould like to join two pattern before converting data.
I have used this link for the grok_patterns : Grok input data format
I found a solution and below my correction :
grok_patterns = ['%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"} %{WORD:MATid} %{WORD:OPUid}']
grok_custom_patterns = '''
'''
With this log :
2022-08-19 17:21:18 MAT01 OPU30
2022-08-19 17:21:19 MAT01 OPU30
However, I'm still looking to convert my hexa data to decimal and apply a conversion formula. And to complicate things, i vould like to join two pattern before converting data.
Related
Hi guys I have a MongoDb schema where one of the columns type is buffer, here I want to save a string then read it correctly. How can I do that ?
For example when I insert data there with
var info = Buffer.from(string);
then read it with
Buffer.from(info, 'binary').toString('utf8');
I am getting a string where /n is visible. How to do this correctly ?
(Assuming that you meant \n)
Did you trim the input string to remove a trailing \n? E.g. in node.js REPL:
> var info = Buffer.from("string\n");
undefined
> Buffer.from(info, 'binary').toString('utf8');
'string\n'
vs.
> var string = "string\n";
undefined
> info = Buffer.from(string.trim());
<Buffer 73 74 72 69 6e 67>
> Buffer.from(info, 'binary').toString('utf8');
'string'
I cannot find documentation anywhere that will tell me what this message means.
it looks like this in Wireshark.
00 00 00 03 14 03 01
I realize it is a 3 byte message, it is an extended message, ie type 20, but I don't know what 03 01 represent.
The scenario is that I send an 'Interested' message to the peer to unchoke my client, the peer then responds with the above message, followed by the 'Unchoke' message.
It is a extension message with ID = 3 and 01 is message data.
What ID = 3 means in this case, is defined by the previously extended message handshake (ID = 0) your client has sent.
A educated guess is that the message you see means: upload_only = 1. ('Extension for Partial Seeds' - BEP21)
Addendum:
uTorrent and most other clients implementation of upload_only differs from the 'out of date' specification explained here; alus = Greg Hazel
It's defined as a extension message in the extension handshake were the 1 byte message data means: 0x00 = false or < anything else> = true.
This can be verified by using Wireshark.
I have a text files, which is text copied from a subtitle file, that looks like this:
1
00 : 00 : 02 , 240 --> 00 : 00 : 04 , 240
(tadashi) <watashi no namae wa kanzaki jika.
2
00 : 00 : 04 , 240 --> 00 : 00 : 06 , 240
makikomare te shimatta watashi wa
tsuini?
...
it goes on for some ~300 more chunks like this.
How would I make it look like this, without doing it manually :) :
1
00:00:02,240 --> 00:00:04,240
(tadashi) <watashi no namae wa kanzaki jika.
2
00:00:04,240 --> 00:00:06,240
makikomare te shimatta watashi wa
tsuini?
...
Basically, I would like to remove all spaces on lines that are starting with the number zero, except those spaces that are before and after the "arrow"
I am on OSX Yosemite but, if the only solution would be on some other os, I'd be glad to hear it regardless
Since no one has answered you yet, here is a solution in python. You need to replace source and target filenames with what is appropriate for you.
#!/usr/bin/python
import re # this is the regex library
f = open('source.txt', 'rt') # this is the name of your source file
fnew = open('target.txt', 'wt') # this is the name of your target file
for line in f:
new = re.sub(r'(\d\d) ([:|,]) (\d\d)', "\\1\\2\\3", line)
fnew.write(new)
f.close()
fnew.close()
I'm trying to use the md5 digest of an attachment I put on the CouchDB, but I can't understand what format it uses.
{
"_id":"ef467479af422db0c388fa00b3000d40",
"_rev":"3-6d1015e7d25103180817136eefa9f942",
"_attachments":{
"foo":{
"content_type":"application/octet-stream",
"revpos":2,
"digest":"md5-yDbs1scfYdqqLpxyFb1gFw==",
"length":1952913,"stub":true }
}
}
That md5 is not hexadecimal but still it is ASCII, how do I use it?
The part of the digest after the md5- prefix looks like it's in Base-64 format.
If parsing in Javascript, the atob function can turn it back into binary data.
Assuming the above is correct then the hexadecimal equivalent is:
c8 36 ec d6 c7 1f 61 da aa 2e 9c 72 15 bd 60 17
For anyone looking to work with the digest format used by couchdb using nodejs you can turn the base64 encoded digest into a "normal" hex string by removing the "md5-" prefix and then do:
new Buffer('yDbs1scfYdqqLpxyFb1gFw==', 'base64').toString('hex')
To go the other way and create the digest string from a hex value:
new Buffer('c836ecd6c71f61daaa2e9c7215bd6017', 'hex').toString('base64')
I'm trying to implement an ISO8589 message to a financial institution. They however, have a Web Service that I call and then I load the ISO8589 payload into an appropriate field of the WCF service.
I have created an ISO8589 message this way:
var isoMessage = new OpenIso8583.Net.Iso8583();
isoMessage.MessageType = Iso8583.MsgType._0100_AUTH_REQ;
isoMessage.TransactionAmount = (long) 123.00;
isoMessage[Iso8583.Bit._002_PAN] = "4111111111111111";
// More after this.
I can't seem to figure out how I can convert the isoMessage into an ASCII human readable format so I can pass it through to the web service.
Anyone have any idea how this can be done with this library? Or am I using this library the wrong way?
Thanks.
UPDATED:
I have figured out how to do this doing:
var asciiFormatter = new AsciiFormatter();
var asciiValue = asciiFormatter.GetString(isoMessage.ToMsg());
However, Now I am trying to take the isoMessage and pass the entire thing as hex string easily using OpenIso8583.Net, as follows:
var isoMessage = new OpenIso8583Net.Iso8583();
isoMessage.MessageType = Iso8583.MsgType._0800_NWRK_MNG_REQ;
isoMessage[Iso8583.Bit._003_PROC_CODE] = "000000";
isoMessage[Iso8583.Bit._011_SYS_TRACE_AUDIT_NUM] = "000001";
isoMessage[Iso8583.Bit._041_CARD_ACCEPTOR_TERMINAL_ID] = "29110001";
I know this is tricky, because some fields are BCD, AlpahNumeric, Numeric, etc. however, this should be realively easy (or I would think) using OpenIso8583.Net? The result I'd like to get is:
Msg Bitmap (3, 11, 41) ProcCode Audit Terminal ID
----- ----------------------- -------- -------- -----------------------
08 00 20 20 00 00 00 80 00 00 00 00 00 00 00 01 32 39 31 31 30 30 30 31
Any help would be greatly appreciated!
Essentially, you need to extend Iso8583 which you initialise with your own Template In the Template, you can set the formatters for each field so that BCD and binary packing is not used. Have a look at the source code for Iso8583 as to how it works.