I need help in constructing a grok pattern for the following fields:
Here are some of the data we can get from it my log
timestamp: 2022-02-02T10:37:09.721-05:00
httpMethod: POST
httpResource: /graphql
httpResponse: 200 (200 is successful, 4xx, or 5xx is error)
how can I represent that in a grok pattern please?
I have read and tried some documenation on grok patterns to no avail.
If all your logs have the same structure, you can use something similar to this:
timestamp: %{TIMESTAMP_ISO8601:DateTime} httpMethod: %{WORD:Method} httpResource: /%{WORD:Resource} httpResponse: %{NUMBER:Response} %{GREEDYDATA:Message}
You can use the follow URL to test:
https://grokdebug.herokuapp.com/
Related
I've got data that looks like this:
CK/YZfB6XUmSOSM3IJqM2Q; Response code: 404. Elapsed: 0ms. Request: GET /marketing
4kk/TiKjYU2JY0L2N14QLg; Response code: 200. Elapsed: 10ms. Request: GET /api/monitor
vhXVsw4sBk69qv7dGE8JYw; Response code: 404. Elapsed: 0ms. Request: GET /graph-statistics
4kk/TiKjYW2JY0L2N14QLg; Response code: 200. Elapsed: 10ms. Request: GET /api/monitor
I'm trying to query/filter it out so I only remain with the 4xx responses.
I've read the documentation regarding wildcards, so I'd expect at least one of the following queries to work (this is as written in Kibana):
message: "Response code: 4??"
message: 4??.
message: "Response code: 4*"
Here's how these look in JSON, in case escaping matters (I've enumerated all the 3 queries in the same JSON for brevity):
"filter" : [],
"query" : {
"query_string" : {
"query" : "message: \"Response code: 4??\"",
"query" : "message: 4??.",
"query" : "message: \"Response code: 4*\"",
"analyze_wildcard" : true
}
},
I've had no luck so far and I'm running out of ideas...
Based on your question, the text appears to be indexed in the field message.
If you want your query to return only 4XX responses, then try the below query.
message: (Response AND code AND 4??)
This query is essentially asking lucene to get records that have the words response, code and 4xx in them.
I tested against the following case, where your record might contain 400 as part of Elapsed time.
......... Response code:200 Elapsed:404ms.......
But the query works fine and doesn't return these results as the 404 is having ms as suffix. So, this doesn't match with your search for 4??.
Also, you might have to check for how the field is indexed in your collection. Is it stored as text or string?
In your code example, you have not escaped : which is a special character in lucene.
Note: this query checks for presence of these keywords in text but nor necessarily in the given order
Reference.find(
{$and: [
{'url': url},
{"text" : {$regex : text, '$options' : 'i'}}
]})
Whenever I test out text with 'Hello World', I get back documents that have that text exactly. However, if I were to do 'Hello Worl', I get nothing back, even with the same url. I've tried numerous implementations of ReGex, and so far none have worked. I'm using Mongo 3.2.11 if that matters.
Try with the following string '.*content.*' or regex format /.*content.*/, which are the widely supported regex formats.
Also, see Checking if a field contains a string
My question is related to this
findAndModify Error in mongodb - nodejs - error code 17287
But the solution hasn't worked (i tried specify the order but i get the same error) I think it might be something along the lines of the index I am using example instead of _id (_id is a field in this collection I just don't want to search by _id in this case) not sure at all...
The Error:
{ [MongoError: exception: nextSafe(): { $err: "Can't canonicalize query: BadValue bad sort specification", code: 17287 }]
name: 'MongoError',
message: 'exception: nextSafe(): { $err: "Can\'t canonicalize query: BadValue bad sort specification", code: 17287 }',
errmsg: 'exception: nextSafe(): { $err: "Can\'t canonicalize query: BadValue bad sort specification", code: 17287 }',
code: 13106,
ok: 0 }
This is my code, it should find and modify (and return using new:true) the first instance of a document that has the field example equal to minus one.
db.collection('documents').findAndModify({example:-1},{$set:{example:0}},{new:true},function(err,result){
console.dir(err||result);
});
But all it does is error like it hates my face!
Here is a document:
{'_id':0,'example':-1}
My id field is custom numerical Where I ensure the _id is always unique (For my purposes I cannot change the _id to standard mongodb way.)
It was painful to get this to work as the node docs end abruptly and then you are left guessing on how the code structure should be on what would/could of just been a standard structure but isn't! I have wasted a day thanks to the devs undocumented methods.
Here is the working code
db.collection('documents').findAndModify({'example':{$eq:-1}},[['example',1]],{$set:{'example':-1}},{'new':true},function(err,result){
console.dir(err||result);
});
I am not sure I really like writing code like this ether. I might look for another node module for this as the query above looks disgusting!
Would be greate if I could find something that looks as simple as this:
db.collection('documents').findModifyReturn('find:example==-1;order:asc;set:example=0;',function(e,r){});
Try using the $eq operator.
db.collection('documents').findAndModify({example: {$eq: -1}}, {'example', 'ascending'},{$set:{example:0}},{new:true},function(err,result){
console.dir(err||result);
});
I'm having a profoundly hard time getting this elasticsearch query to cooperate. I'm currently trying to use a Bool Query to get results for both exact and analyzed/fulltext searches.
Seems I cannot, for what ever reason, use match filters inside a bool query:
{
"bool":{
"should":[
"match":{"mainInfo.states": "Wisconsin"},
"match":{"mainInfo.cities": "WI"}
]
}
}
This throws a parser error, telling me that match is not a query type.
I need the ability to have fulltext searches inside of a bool query that also has term filters. So essentially I'd like to have a query somewhat like this:
{
"bool":{
"must":[
"term":{"mainInfo.clientId":"123456"}
],
"should":[
"match":{"mainInfo.states": "Wisconsin"},
"match":{"mainInfo.cities": "WI"}
]
}
}
Where I can have 1 or 2 terms which must exists, and several fulltext searches where 2 of them should exist.
I'm also using a function_score query to return random, page-able results using a random_seed.
My problem is that I cannot get fulltext queries to run inside of my bool query. Seems that only term filters work. Also when I put the match queries out of the bool query, it seems that either the match query works, or the bool query works, but never both.
I cannot seem to form a query that will match several term queries, and several match queries. The term must exist, where as the match should exist(so at least matching one of the filters).
I'm by no means an elasticsearch expert, and the docs seem pretty vague when you're looking for information on more complicated queries.
If anyone could lend an example as how to query in such a manner, it would be greatly appreciated. However, please don't just link me to the docs for Bool Queries, or any other ElasticSearch docs. I assure you I've read them thoroughly, and I've tried a number of different ways to execute this query, but none seem to have the expected behavior.
I'm using ElasticSearch 1.3.2
Seems I cannot, for what ever reason, use match filters inside a bool query:
The syntax of your query is not correct, if you have multiple clauses each clause should be wrapped in their own object:
{
"query":{
"custom_filters_score":{
"query":{
"nested":{
"path":"mainInfo",
"query":{
"bool":{
"should":[
{
"match":{
"mainInfo.states":"WI"
}
},
{
"match":{
"mainInfo.cities":"Wisconsin"
}
}
]
}
}
}
},
"filters":[{
"term":{
"mainInfo.clientId":"123456"
},
"script":"yourscriptgoeshere"
}]
}
}
}
Im trying to put together an API that would consume json data, and I'm having problems preserving variable types.
caveat: I'm using Node.js
Given the data:
{
id: "string",
data: [
{input: [0,1,0], output: [1,1]},
{input: [1,0,0], output: [1,0]}
]
}
When i make a jquery ajax post request with that data it is transfered as:
data[0][input][] 0
data[0][input][] 1
data[0][input][] 0
data[0][output][] 1
data[0][output][] 1
data[1][input][] 1
data[1][input][] 0
data[1][input][] 0
data[1][output][] 1
data[1][output][] 0
id string
or: id=string&data%5B0%5D%5Binput%5D%5B%5D=0&data%5B0%5D%5Binput%5D%5B%5D=1&data%5B0%5D%5Binput%5D%5B%5D=0&data%5B0%5D%5Boutput%5D%5B%5D=1&data%5B0%5D%5Boutput%5D%5B%5D=1&data%5B1%5D%5Binput%5D%5B%5D=1&data%5B1%5D%5Binput%5D%5B%5D=0&data%5B1%5D%5Binput%5D%5B%5D=0&data%5B1%5D%5Boutput%5D%5B%5D=1&data%5B1%5D%5Boutput%5D%5B%5D=0
What I need to be able to do is decode the data on the server side but all the values of my arrays in data are being transformed to strings.
Is there a technique for preserving the type of a value ie: "string" or "int" etc?
I'm currently using the qs module on npm to parse my POST request body.
JQuery serializes any object passed in the data option of an AJAX request into application/x-www-form-urlencoded format (just like a query string). What you want is to JSON.stringify() your object and pass the resulting string as data, along with setting the Content-Type as the previous answer mentions.
Remember that the dataType option specifies how jQuery should deal with the response to the request, not how to encode the body of the request (I've gotten burned by this one).
Since it is a JSON server, set Content-Type: application/json in the POST request, so you'll be parsing an object rather than a string. The object will retain data types. Use req.is('application/json') on the server to ensure that the request is of type 'application/json'.
Here is a working solution, hope it helps.
$.ajax({
type: 'POST',
data: JSON.stringify(data),
contentType: 'application/json',
url: 'http://localhost:3000/endpoint',
success: function(data) {
//TODO: do something with data
}
});