01-10-2022 14:05:11.584 INFO - Destination IP:0:0:0:0:0:0:0:1 | Source System IP:0:0:0:0:0:0:0:1 | BrowserName:Chrome | BrowserVersion:105 | requestURI:/dashboard | Feature name:Dashboard | Application:null | SubFeature name:Line Check | UserId:tushar | ApiCalled:/ruambot/api/getGraph() | ApiStatus:Success | Login Time:01-10-2022 13:46:42
how to handle the word sub-feature name in greedy data
with grok filter split message into two parts and then apply kv filter for second part of the message
filter {
grok {
match => ["message", "%{GREEDYDATA:message_part1}\-%{GREEDYDATA:message_part2}"]
}
kv {
source => "message_part2"
field_split => "|"
value_split => ":"
}
}
Related
21-09-2022 15:37:52.781 INFO - Destination IP:0:0:0:0:0:0:0:1 | Source System IP:0:0:0:0:0:0:0:8 | BrowserName:Edge(Chromium) | BrowserVersion:105 | requestURI:/home | Feature name:RUAMBOT | UserId:rohan | ApiCalled:/ruambot/api/getAllApplicationsByUserID() | ApiStatus:Success | Login Time:21-09-2022 15:37:51
Here is the grok pattern for the mentioned log line:
%{DATESTAMP:timestamp} %{LOGLEVEL:loglevel} \- %{DATA:data}\:%{IP:destination_ip} \| %{DATA:data}\:%{IP:source_ip} \| %{DATA:data}\:%{DATA:browser_name} \| %{DATA:data}\:%{INT:browser_version} \| %{DATA:data}\:%{URIPATH:uri} \| %{DATA:data}\:%{DATA:feature_name}\| %{DATA:data}\:%{USERNAME:userid} \| %{DATA:data}\:%{URIPATH:api_call} \| %{DATA:data}\:%{DATA:api_status} \| %{DATA:data}\:%{DATESTAMP:login_time}
I have used the Grok Debugger to created the grok pattern.
Also, you can make use of the Drop Filter of the logstash to drop the field data generated [see the output screenshot below] after parsing your logs using the above GROK pattern.
Output:
I'm running Cilium inside an Azure Kubernetes Cluster and want to parse the cilium log messages in the Azure Log Analytics. The log messages have a format like
key1=value1 key2=value2 key3="if the value contains spaces, it's wrapped in quotation marks"
For example:
level=info msg="Identity of endpoint changed" containerID=a4566a3e5f datapathPolicyRevision=0
I couldn't find a matching parse_xxx method in the docs (e.g. https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/parsecsvfunction ). Is there a possibility to write a custom function to parse this kind of log messages?
Not a fun format to parse... But this should work:
let LogLine = "level=info msg=\"Identity of endpoint changed\" containerID=a4566a3e5f datapathPolicyRevision=0";
print LogLine
| extend KeyValuePairs = array_concat(
extract_all("([a-zA-Z_]+)=([a-zA-Z0-9_]+)", LogLine),
extract_all("([a-zA-Z_]+)=\"([a-zA-Z0-9_ ]+)\"", LogLine))
| mv-apply KeyValuePairs on
(
extend p = pack(tostring(KeyValuePairs[0]), tostring(KeyValuePairs[1]))
| summarize dict=make_bag(p)
)
The output will be:
| print_0 | dict |
|--------------------|-----------------------------------------|
| level=info msg=... | { |
| | "level": "info", |
| | "containerID": "a4566a3e5f", |
| | "datapathPolicyRevision": "0", |
| | "msg": "Identity of endpoint changed" |
| | } |
|--------------------|-----------------------------------------|
With the help of Slavik N, I came with a query that works for me:
let containerIds = KubePodInventory
| where Namespace startswith "cilium"
| distinct ContainerID
| summarize make_set(ContainerID);
ContainerLog
| where ContainerID in (containerIds)
| extend KeyValuePairs = array_concat(
extract_all("([a-zA-Z0-9_-]+)=([^ \"]+)", LogEntry),
extract_all("([a-zA-Z0-9_]+)=\"([^\"]+)\"", LogEntry))
| mv-apply KeyValuePairs on
(
extend p = pack(tostring(KeyValuePairs[0]), tostring(KeyValuePairs[1]))
| summarize JSONKeyValuePairs=parse_json(make_bag(p))
)
| project TimeGenerated, Level=JSONKeyValuePairs.level, Message=JSONKeyValuePairs.msg, PodName=JSONKeyValuePairs.k8sPodName, Reason=JSONKeyValuePairs.reason, Controller=JSONKeyValuePairs.controller, ContainerID=JSONKeyValuePairs.containerID, Labels=JSONKeyValuePairs.labels, Raw=LogEntry
I am using express-validator in my project
my json from the client is
{"name": "john doe"}
my express validation code is
[check('name', 'invalid name').isAlpha()]
why this code is returning invalid name while this is a string.
Also I tried isString() but it is also not working it is working in the same style as isAlpha().
Error json response to the client is
{
"errors": [
{
"value": "john doe",
"msg": "invalid name",
"param": "name",
"location": "body"
}
]
}
does isAlpha() function consider only one word as a string
How can I fix this
There is an option of .isAlpha you can use to ignore white spaces:
check('name', 'invalid name').isAlpha('en-US', {ignore: ' '})
The first parameter 'en-US' is AlphaLocale. For example I use 'es-ES' to validate Spanish special characters. You can use one of these to validate other languages: 'ar' | 'ar-AE' | 'ar-BH' | 'ar-DZ' | 'ar-EG' | 'ar-IQ' | 'ar-JO' | 'ar-KW' | 'ar-LB' | 'ar-LY' | 'ar-MA' | 'ar-QA' | 'ar-QM' | 'ar-SA' | 'ar-SD' | 'ar-SY' | 'ar-TN' | 'ar-YE' | 'az-AZ' | 'bg-BG' | 'cs-CZ' | 'da-DK' | 'de-DE' | 'el-GR' | 'en-AU' | 'en-GB' | 'en-HK' | 'en-IN' | 'en-NZ' | 'en-US' | 'en-ZA' | 'en-ZM' | 'es-ES' | 'fa-AF' | 'fa-IR' | 'fr-FR' | 'he' | 'hu-HU' | 'id-ID' | 'it-IT' | 'ku-IQ' | 'nb-NO' | 'nl-NL' | 'nn-NO' | 'pl-PL' | 'pt-BR' | 'pt-PT' | 'ru-RU' | 'sk-SK' | 'sl-SI' | 'sr-RS' | 'sr-RS#latin' | 'sv-SE' | 'th-TH' | 'tr-TR' | 'uk-UA' | 'vi-VN'.
The second parameter is the object IsAlphaOptions. It only contains an optional parameter 'ignore', and it can have the value of a string, string[] or RegExp.
So you can also ignore white spaces with the RegExp \s.
.isAlpha('en-US', {ignore: '\s'})
I got the answer. I used custom validation method. It resolved my issue.
[check('name').custom((value,{req})=>{
if(isNaN(value)){
return true;
}else{
throw new Error('invalid name')
}
})]
To check, using express-validator, a string contains only letters and spaces you can use a regular expression
check('name').custom((value) => {
return value.match(/^[A-Za-z ]+$/);
})
"john doe" consisting white space " ". Due to this white-space isAlpha() throwing error. isAlpha allows only a-zA-Z.
Hopefully Im not late to the party.
With class-validator#0.13.2, we can use
#Matches(/^[a-zA-Z0-9 -]*$/)
Just tweak the regex to satisfy your needs. In my case, I want to use #IsAlphanumeric() but with spaces and hyphen/dash
Simply replace isAlpha() or isAlphaNumeric()
with
isAlphanumericWithSpace()/ isAlphaWithSpace().
I have this pattern and I want the grok filter for this:
24 May 2016 23:04:03,003 [] [] [] INFO [listenerContainer-35] com.newworld.mmp.orderlist.NewDataUtil - | 1464048002998 | 201605233157123319 | Account | 67578625
09896 | DHW | 2016-05-23T23:59:56.621Z | 2016-05-24T00:00:02.676Z | STARTED PROCESSING
I wrote the pattern but it is incomplete:
%{MONTHDAY} %{MONTH} 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{DATA:junk} %{DATA:junk} %{DATA:junk} %{LOGLEVEL:level} %{DATA:junk1} %{JAVACLASS:class}
The %{POSINT:mynewint} or %{NUMBER:mynewint} for the 1464048002998 is not working.
Like %{MONTHDAY} %{MONTH} 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) %{DATA:junk} %{DATA:junk} %{DATA:junk} %{LOGLEVEL:level} %{DATA:junk1} %{JAVACLASS:class}- | %{POSINT:mynewint}
I need help on this and the complete grok expression?
Your Log line:
24 May 2016 23:04:03,003 [] [] [] INFO [listenerContainer-35] com.newworld.mmp.orderlist.NewDataUtil - | 1464048002998 | 201605233157123319 | Account | 6757862509896 | DHW | 2016-05-23T23:59:56.621Z | 2016-05-24T00:00:02.676Z | STARTED PROCESSING
SAMPLE GROK PATTERN that matches your log record:
%{MONTHDAY:MonthDay} %{MONTH:Month} %{YEAR:Year} %{TIME:Time} \[] \[] \[] %{LOGLEVEL:LogLevel} %{NOTSPACE:ListenerContainer} %{JAVACLASS:JavaClass} - \| %{NUMBER:Number1} \| %{NUMBER:Number2} \| %{WORD:Account} \| %{NUMBER:Number3} \| %{WORD:DHW} \| %{TIMESTAMP_ISO8601:Timestamp1} \| %{TIMESTAMP_ISO8601:Timestamp2} \| %{JAVALOGMESSAGE:LogMessage}
This will give output fields as follows:
MonthDay = 24
Month = May
Year = 2016
Time = 23:04:03,003
LogLevel = INFO
ListenerContainer = [listenerContainer-35]
JavaClass = com.newworld.mmp.orderlist.NewDataUtil
Number1 = 1464048002998
Number2 = 201605233157123319
Account
Number3 = 6757862509896
DHW
Timestamp1 = 2016-05-23T23:59:56.621Z
Timestamp2 = 2016-05-24T00:00:02.676Z
LogMessage = STARTED PROCESSING
You can try your own grok filters, create parse and test on the following site: http://grokconstructor.appspot.com/do/construction
Hi I am trying to write express a set of requirements in gherkin syntax, but it requires a good deal of repetition. I saw here that I can use placeholders which would be perfect for my task, however some of the data in my Given and in my then are collections. How would I go about representing collections in the examples?
Given a collection of spaces <spaces>
And a <request> to allocate space
When I allocate the request
Then I should have <allocated_spaces>
Examples:
| spaces | request | allocated_spaces |
| ? | ? | ? |
A bit hacky, but you can delimit a string:
Given a collection of spaces <spaces>
And a <request> to allocate space
When I allocate the request
Then I should have <allocated_spaces>
Examples:
| spaces | request | allocated_spaces |
| a,b,c | ? | ? |
Given(/^a collection of spaces (.*?)$/) do |arg1|
collection = arg1.split(",") #=> ["a","b","c"]
end
You can use Data Tables. I never try to have param in data table before, but in theory it should work.
Given a collection of spaces:
| space1 |
| space2 |
| <space_param> |
And a <request> to allocate space
When I allocate the request
Then I should have <allocated_spaces>
Examples:
| space_param | request | allocated_spaces |
| ? | ? | ? |
The given data table would be an instance of Cucumber::Ast::Table, checkout the rubydoc for its API.
Here's an example, again using split, but without the regex:
Scenario Outline: To ensure proper allocation
Given a collection of spaces <spaces>
And a <request> to allocate space
When I allocate the request
Then I should have <allocated_spaces>
Examples:
| spaces | request | allocated_spaces |
| "s1, s2, s3" | 2 | 2 |
| "s1, s2, s3" | 3 | 3 |
| "s1, s2" | 3 | 2 |
I use cucumber-js so this is what the code may look like:
Given('a collection of spaces {stringInDoubleQuotes}', function (spaces, callback) {
// Write code here that turns the phrase above into concrete actions
this.availableSpaces = spaces.split(", ");
callback();
});
Given('a {int} to allocate space', function (numToAllocate, callback) {
this.numToAllocate = numToAllocate;
// Write code here that turns the phrase above into concrete actions
callback();
});
When('I allocate the request', function (callback) {
console.log("availableSpaces:", this.availableSpaces);
console.log("numToAllocate:", this.numToAllocate);
// Write code here that turns the phrase above into concrete actions
callback();
});
Then('I should have {int}', function (int, callback) {
// Write code here that turns the phrase above into concrete actions
callback(null, 'pending');
});