All
I'm using logstash to ship logs from the remote server.
The message i got is a hash type like this:
[2014-12-06 23:59:57] 112.254.70.37 <AUDIO> {"type":"Stat", "eid":4800316, "mid":"87192133091532", "ccid":3228662, "ver":102, "ip":"114.113.200.227", "port":9081, "jitter":"0 0 0 0 0 ", "break":"0 0 0 0 0 ", "interrupt":"0 0 0 0 0 ", "tcp_rtt":"40 40 45 50 50 ", "udp_rtt":"31 33 35 40 35 ", "all_pkts":"107180 107193 107249 107323 107358 ", "lost":"0 0 0 0 0 ", "delay":"40.78", "pull":"3 3 3 3 3 "}
Then how can I write the grok part, I search the doc everywhere, but i still don't konw how...
thx!
First, you have to parse out your json data by grok filter. Then, use json filter to parse all the hashmap value. With this config I can parse your log and create all the field:value. Hope this can help you.
input {
stdin{
}
}
filter {
grok {
match => [ "message" , "\[%{TIMESTAMP_ISO8601:datatime}\] %{IP:ip} <%{WORD:level}> %{GREEDYDATA:data}"]
}
json {
source => "data"
}
}
output {
stdout{
codec => rubydebug
}
}
Related
I have a column like this:
business_time_left
3 Hours 24 Minutes
59 Minutes
4 Days 23 Hours 58 Minutes
0 Seconds
1 Hour
and so on..
What I want to do in Logstash is to convert this entirely into hours.
So mu value should entirety convert to something like
business_time_left
3.24
0.59
119.58
0
1
Is this possible?
My config file:
http_poller {
urls => {
snowinc => {
url => "https://service-now.com"
user => "your_user"
password => "yourpassword"
headers => {Accept => "application/json"}
}
}
request_timeout => 60
metadata_target => "http_poller_metadata"
schedule => { cron => "* * * * * UTC"}
codec => "json"
}
}
filter
{
json {source => "result" }
split{ field => ["result"] }
}
output {
elasticsearch {
hosts => ["yourelastuicIP"]
index => "inc"
action=>update
document_id => "%{[result][number]}"
doc_as_upsert =>true
}
stdout { codec => rubydebug }
}
Sample Json input data, when the url is hit.
{"result":[
{
"made_sla":"true",
"Type":"incident resolution p3",
"sys_updated_on":"2019-12-23 05:00:00"
"business_time_left":" 59 Minutes"} ,
{
"made_sla":"true",
"Type":"incident resolution l1.5 p4",
"sys_updated_on":"2019-12-24 07:00:00"
"business_time_left":"3 Hours 24 Minutes"}]}
Thanks in advance!
Q: Is this possible?
A: Yes.
Assuming your json- and split-filters are working correctly and the field business_time_left holds a single value like you showed (e.g. 4 Days 23 Hours 58 Minutes) I personally would do the following:
First, make sure that your data is in a kind of pattern meaning, you standardize the "quantity-descriptions". This means that the minutes are always labeled as "Minutes" not Mins, min or whatever.
Nextup, you can parse the field with the grok-filter like so:
filter{
grok{
match => { "business_time_left" => "(%{INT:calc.days}\s+Days)?%{SPACE}?(%{INT:calc.hours}\s+Hours)?%{SPACE}?(%{INT:calc.minutes}\s+Minutes)?%{SPACE}?(%{INT:calc.seconds}\s+Seconds)?%{SPACE}?" }
}
}
This will extract all available values into the desired fields, e.g. calc.days. The ? character prevents that grok fails if e.g. there are no seconds. You can test the pattern on this site.
With the data extracted, you can implement a ruby filter to aggregate the numeric values like so (untested though):
ruby{
code => '
days = event.get("calc.days")
hours = event.get("calc.hours")
minutes = event.get("calc.minutes")
sum = 0
if days
days_numeric = days.to_i
days_as_hours = days_numeric * 24
sum += days_as_hours
end
if hours
sum += hours.to_i
end
if minutes
sum += (minutes.to_i / 100)
end
# seconds and so on ...
event.set("business_time_left_as_hours", sum)
'
}
So basically you check if the values are present and add them to a sum with your custom logic.
event.set("business_time_left_as_hours", sum) will set the result as a new field to the document.
These code snippets are not intended to be working out of the box they are just hints. So please check the documentations about the ruby filter, ruby coding in general and so on.
I hope I could help you.
I'm trying to get pattern grok on my log file data
this is the message log
116.50.181.5 - - [18/May/2019:09:05:32 +0000] "SHARP56" 50 245 "INFO: System componement ready for use" 23 "A4" "/user/admistrator/68768.pdf" "INFO: No ERROR TO SHOW"
I've tried this grok pattern but it didn't works
%{IP:client} %{HTTPDATE:timestamp}\] %{WORD:name} %{NUMBER:X1} %{NUMBER:x2} %{WORD:msg} %{NUMBER:X3} %{WORD:format} %{WORD:path} %{WORD:label}
the output file that I want should look like this
{
client = 116.50.181.5
timeStamp = 18/May/2019:09:05:32 +0000
name = SHARP56
x1 = 50
x2 = 245
msg =INFO
format = A4
type = pdf
label = INFO: No ERROR TO SHOW
}
any suggestion ?
you can use the following:
%{IP:client} - - \[%{HTTPDATE:timestamp}\] \"%{DATA:name}\" %{NUMBER:X1} %{NUMBER:x2} \"%{GREEDYDATA:msg}\" %{NUMBER:X3} \"%{WORD:format}\" \"%{DATA:path}\" \"%{GREEDYDATA:label}\"
After requesting a document via the DocuSign api and writing it to the file system it appears blank after opening it. The docs say it returns a "PDF File" and the response body is returned as below.
const doc =
await rp.get(`${apiBaseUrl}/${BASE_URI_SUFFIX}/accounts/${accountId}/envelopes/${envelopeId}/documents/${document.documentId}`,
{auth: { bearer: token }}
);
fs.writeFile(document.name, new Buffer(doc, "binary"), function(err) {
if (err) throw err;
console.log('Saved!');
});
Response body:
{
"documents": [
{
"name": "Name of doc.docx",
"content": "%PDF-1.5\n%\ufffd\ufffd\ufffd\ufffd\n%Writing objects...\n4 0 obj\n<<\n/Type /Page\n/Resources 5 0 R\n/Parent 3 0 R\n/MediaBox [0 0 612 792 ]\n/Contents [6 0 R 7 0 R 8 0 R 9 0 R 10 0 R ]\n/Group <<\n/Type /Group\n/S /Transparency\n/CS /DeviceRGB\n>>\n/Tabs /S\n/StructParents 0\n>>\nendobj\n5 0 obj\n<<\n/Font <<\n/F1 11 0 R\n/F2 12 0 R\n/F3 13 0 R\n>>\n/ExtGState <<\n/GS7 14 0 R\n/GS8 15 0 R\n>>\n/ProcSet [/PDF /Text ...
}
]}
Screenshot of document:
The EnvelopeDocuments::get API method returns the PDF itself, not an object as you are showing.
For a working example of the method, see example 7, part of the Node.js set of examples.
Added
Also, the fs.writeFile call supports writing from a string source. I'd try:
fs.writeFile(document.name, doc, {encoding: "binary"},
function(err) {
if (err) throw err;
console.log('Saved!');
});
Incorrect encoding
Your question shows the pdf's content as a string with the control characters encoded as unicode strings:
"%PDF-1.5\n%\ufffd\ufffd\ufffd\ufffd\n%Writing objects...
but this is not correct. The beginning of a PDF file includes binary characters that are not displayable except in a hex editor. This is what you should see at the top of a PDF:
Note the 10th character. It is hex c4. In your string, the equivalent character has been encoded as \ufffd (it is ok that they aren't the same character, they are two different PDFs). The fact that the character has been encoded is your problem.
Solutions
Convince the requests library and the fs.WriteFile methods to not encode the data. Or to decode it as needed. See this solution for the requests library.
Or use the DocuSign Node.js SDK as I show in the example code referenced above.
I am writing a puppet defined type as follows:
1 #--------------------------------------------------#
2 #-------------------WindowsLog---------------------#
3 #--------------------------------------------------#
4 # Type to set up a windows log #
5 #--------------------------------------------------#
6
7 define windows_log($size = '25MB', $overflowAction = 'OverwriteAsNeeded', $logName = $title)
8 {
9
10 #Microsoft is stupid. Get-WinEvent has different names for logmode than limit-eventlog.
11 #The following selector (basuically a ternary operator) should fix that
12 $overflowWinEventName = $overflowAction ? {
13 OverwriteAsNeeded => "Circular",
14 OverwriteOlder => "AutoBackup",
15 DoNotOverwrite => "Retain",
16 default => undef,
17 }
18
19 if($overflowWinEventName == undef)
20 {
21 fail("${$overflowAction} is not a valid overflow action")
22 }
23 else{
24 exec { "Set maximum log size for ${logName}":
25 provider => powershell,
26 command => "Limit-EventLog -LogName ${logName} -MaximumSize ${size} -OverflowAction ${overflowAction}",
27 unless => "\$log = Get-WinEvent -ListLog ${logName}; if(\$log.MaximumSizeInBytes -eq ${size} -and \$log.LogMode -eq '${overflowWinEventName}'){exit 0}else{exit 1}",
28 }
29 }
30 }
However the method 'fail' does not have the effect I want, and none of the methods listed at http://docs.puppetlabs.com/references/latest/function.html seem to be right either.
Basically I am trying to get puppet to throw an error for this specific resource only, stop applying it, and then continue applying everything else. Fail throws a parser error which kills everything, and the other methods (warn, error, etc) seem to have no effect on the agent.
Any help would be greatly appreciated! I may have just stupidly overlooked something.
Your construct is basically sound. Defined resources cannot really 'fail' like native resources, but using your if/else construct, it will only do any work if there is no error.
Use fail() only if you detect an error that should make the whole catalog invalid. To just send a message to the agent, use a notify resource instead.
notify {
"FATAL - ${overflowAction} is not a valid overflow action":
loglevel => 'err',
withpath => true; # <- include the fully qualified resource name
}
I'm using Node.js with Express.js and Redis. I'm recording the uptime of a site component by incrementing a redis key. I want to update the uptimerecord:tracker key once the current uptime > the current uptime record but somehow it's not updating it and evaluating uptimeTracker > uptimeRecordTracker with false even though it's true.
Is there anything I'm missing?
Thanks!
db.get("uptime:tracker", function(err, uptimeTracker) {
db.get("uptimerecord:tracker", function(err, uptimeRecordTracker) {
console.log("[Stats] uptimeTracker: " + uptimeTracker)
console.log("[Stats] uptimeRecordTracker: " + uptimeRecordTracker)
console.log("[Stats] Compare: " + (uptimeTracker > uptimeRecordTracker))
if(uptimeTracker > uptimeRecordTracker) {
console.log("[Stats] Tracker Records updated")
db.set('uptimerecord:tracker', uptimeTracker)
}
});
});
The console output:
[Stats] uptimeTracker: 213
[Stats] uptimeRecordTracker: 99
[Stats] Compare: false
It looks like you're comparing strings instead of integers, in fact:
"213" > "99" == false
while
213 > 99 == true
Try converting them to integers before doing the comparison:
parseInt(uptimeTracker) > parseInt(uptimeRecordTracker)