Nested query params with Tesla - get

This is the URL I'm trying to hit:
/example/?fields=*&filter[platform][eq]=111&order=date:asc&filter[date][gt]=1500619813000&expand=yes
My code:
get("/release_dates",
query: [
fields: "*",
order: "date:desc",
expand: "games",
filter: %{
date: %{gt: unix_now},
version_parent: %{not_exists: 1}
}
]
)
I'm trying to perform a Tesla GET request that has those filter[date][gt]=123123123123 type query param.
Appreciate the help!

If I understand you correctly, you want to generate a URI with a filter for "greater than" timestamps which are variable.
According to your inital example, this can be done like this:
Tesla.get("/example/",
fields: "*",
filter: [
platform: [
eq: 111
]
],
order: "date:asc",
filter: [
date: [
gt: unix_now
]
],
expand: "yes"
)
Note that /example is a relative reference and can only be resolved with a base URI. Better provide a complete URI.
If you want to experiment with the URI generator in an iex console, you can use iex -S mix in your project directory and then use the following functions:
Tesla.build_url("/example/",
fields: "*",
filter: [
platform: [
eq: 111
]
],
order: "date:asc",
filter: [
date: [
gt: 123123123123
]
],
expand: "yes"
) |> URI.decode()

Related

How do you find a quoted string with specific word in a log message using grok pattern

I have a log message from my server with the format below:
{"host":"a.b.com","source_type":"ABCD"}
I have this grok pattern so far but it accepts any word in double quotation.
\A%{QUOTEDSTRING}:%{PROG}
how can I change "QUOTEDSTRING" that only check for "host"?
"host" is not at the beginning of the message all the time and it can be found in the middle of message as well.
Thanks for your help.
Since the question specified that "host" can be anywhere between in the log, you can use the following:
\{(\"%{GREEDYDATA:data_before}\",)?(\"host\":\"%{DATA:host_value}\")?(,\"%{GREEDYDATA:data_after}\")?\}
Explanation :
data_before stores the optional data before host type entry is found. You can separate it more as per your need
host : this stores the host value
data_after stores the optional data after host type entry is found. You can seaprate it more as per your need
Example :
{"host":"a.b.com","source_type":"ABCD"}
Output :
{
"data_before": [
[
null
]
],
"host_value": [
[
"a.b.com"
]
],
"data_after": [
[
"source_type":"ABCD"
]
]
}
{"host":"a.b.com"}
Output :
{
"data_before": [
[
null
]
],
"host_value": [
[
"a.b.com"
]
],
"data_after": [
[
null
]
]
}
{"source_type":"ABCD","host":"a.b.com","data_type":"ABCD"}
Output :
{
"data_before": [
[
"source_type":"ABCD"
]
],
"host_value": [
[
"a.b.com"
]
],
"data_after": [
[
"data_type":"ABCD"
]
]
}
Tip : Use the following resources to tune and test your logging patterns :
Grok Debugger
Grok Patterns Full List

Logstash filter for ip

Need logstash filter for client ip , 12.34.56.78:1234
I need to filter the client Ip , only I require 12.34.56.78 not the things after :.
Try this:
GROK pattern:
%{IP:ip}:%{GREEDYDATA:others}
OUTPUT:
{
"ip": [
[
"12.34.56.78"
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
"12.34.56.78"
]
],
"others": [
[
"1234"
]
]
}
This should work (I haven't tested it):
mutate {
gsub => ["ip_field_name", ":\d+", ""]
}
The :\d+ will capture the : and all following digits and the mutate#gsub option will replace this with an empty string.

logstash grok, parse a line with json filter

I am using ELK(elastic search, kibana, logstash, filebeat) to collect logs. I have a log file with following lines, every line has a json, my target is to using Logstash Grok to take out of key/value pair in the json and forward it to elastic search.
2018-03-28 13:23:01 charge:{"oldbalance":5000,"managefee":0,"afterbalance":"5001","cardid":"123456789","txamt":1}
2018-03-28 13:23:01 manage:{"cuurentValue":5000,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}
I am using Grok Debugger to make regex pattern and see the result. My current regex is:
%{TIMESTAMP_ISO8601} %{SPACE} %{WORD:$:data}:{%{QUOTEDSTRING:key1}:%{BASE10NUM:value1}[,}]%{QUOTEDSTRING:key2}:%{BASE10NUM:value2}[,}]%{QUOTEDSTRING:key3}:%{QUOTEDSTRING:value3}[,}]%{QUOTEDSTRING:key4}:%{QUOTEDSTRING:value4}[,}]%{QUOTEDSTRING:key5}:%{BASE10NUM:value5}[,}]
As one could see it is hard coded since the keys in json in real log could be any word, the value could be integer, double or string, what's more, the length of the keys varies. so my solution is not acceptable. My solution result is shown as follows, just for reference. I am using Grok patterns.
My question is that trying to extract keys in json is wise or not since elastic search use json also? Second, if I try to take keys/values out of json, are there correct,concise Grok patterns?
current result of Grok patterns give following output when parsing first line in above lines.
{
"TIMESTAMP_ISO8601": [
[
"2018-03-28 13:23:01"
]
],
"YEAR": [
[
"2018"
]
],
"MONTHNUM": [
[
"03"
]
],
"MONTHDAY": [
[
"28"
]
],
"HOUR": [
[
"13",
null
]
],
"MINUTE": [
[
"23",
null
]
],
"SECOND": [
[
"01"
]
],
"ISO8601_TIMEZONE": [
[
null
]
],
"SPACE": [
[
""
]
],
"WORD": [
[
"charge"
]
],
"key1": [
[
""oldbalance""
]
],
"value1": [
[
"5000"
]
],
"key2": [
[
""managefee""
]
],
"value2": [
[
"0"
]
],
"key3": [
[
""afterbalance""
]
],
"value3": [
[
""5001""
]
],
"key4": [
[
""cardid""
]
],
"value4": [
[
""123456789""
]
],
"key5": [
[
""txamt""
]
],
"value5": [
[
"1"
]
]
}
second edit
Is it possible to use Json filter of Logstash? but in my case Json is part of line/event, not whole event is Json.
===========================================================
Third edition
I do not see updated solution functions well to parse json. My regex is as follows:
filter {
grok {
match => {
"message" => [
"%{TIMESTAMP_ISO8601}%{SPACE}%{GREEDYDATA:json_data}"
]
}
}
}
filter {
json{
source => "json_data"
target => "parsed_json"
}
}
It does not have key:value pair, instead it is msg+json string. The parsed json is not parsed.
Testing data is as below:
2018-03-28 13:23:01 manage:{"cuurentValue":5000,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}
2018-03-28 13:23:03 payment:{"cuurentValue":5001,"reload":0,"newbalance":"5002","posid":"987654321","something":"new3","additionalFields":2}
2018-03-28 13:24:07 management:{"cuurentValue":5002,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}
[2018-06-04T15:01:30,017][WARN ][logstash.filters.json ] Error parsing json {:source=>"json_data", :raw=>"manage:{\"cuurentValue\":5000,\"payment\":0,\"newbalance\":\"5001\",\"posid\":\"123456789\",\"something\":\"new2\",\"additionalFields\":1}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'manage': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"manage:{"cuurentValue":5000,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}"; line: 1, column: 8]>}
[2018-06-04T15:01:30,017][WARN ][logstash.filters.json ] Error parsing json {:source=>"json_data", :raw=>"payment:{\"cuurentValue\":5001,\"reload\":0,\"newbalance\":\"5002\",\"posid\":\"987654321\",\"something\":\"new3\",\"additionalFields\":2}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'payment': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"payment:{"cuurentValue":5001,"reload":0,"newbalance":"5002","posid":"987654321","something":"new3","additionalFields":2}"; line: 1, column: 9]>}
[2018-06-04T15:01:34,986][WARN ][logstash.filters.json ] Error parsing json {:source=>"json_data", :raw=>"management:{\"cuurentValue\":5002,\"payment\":0,\"newbalance\":\"5001\",\"posid\":\"123456789\",\"something\":\"new2\",\"additionalFields\":1}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'management': was expecting ('true', 'false' or 'null')
at [Source: (byte[])"management:{"cuurentValue":5002,"payment":0,"newbalance":"5001","posid":"123456789","something":"new2","additionalFields":1}"; line: 1, column: 12]>}
Please check the result:
You can use GREEDYDATA to assign entire block of json to a separate field like this,
%{TIMESTAMP_ISO8601}%{SPACE}%{GREEDYDATA:json_data}
This will create a separate file for your json data,
{
"TIMESTAMP_ISO8601": [
[
"2018-03-28 13:23:01"
]
],
"json_data": [
[
"charge:{"oldbalance":5000,"managefee":0,"afterbalance":"5001","cardid":"123456789","txamt":1}"
]
]
}
Then apply a json filter on json_data field as follows,
json{
source => "json_data"
target => "parsed_json"
}

grok parser (unexpected null in %{IPORHOST:syslog_server}) HAproxy

Following log:
Jul 25 07:45:12 tst-proxy202 haproxy[1104]: 10.64.111.222:36635 [25/Jul/2016:07:45:12.479] promocloud~ promocloud/tst-service-proxy203 32/0/1/27/60 200 664 - - ---- 0/0/0/0/0 0/0 {} {} "POST /RTI HTTP/1.1"
Is parsed with ${HAPROXYHTTP} grok pattern
%{SYSLOGTIMESTAMP:syslog_timestamp} %{IPORHOST:syslog_server} %{SYSLOGPROG}: %{IP:client_ip}:%{INT:client_port} \[%{HAPROXYDATE:accept_date}\] %{NOTSPACE:frontend_name} %{NOTSPACE:backend_name}/%{NOTSPACE:server_name} %{INT:time_request}/%{INT:time_queue}/%{INT:time_backend_connect}/%{INT:time_backend_response}/%{NOTSPACE:time_duration} %{INT:http_status_code} %{NOTSPACE:bytes_read} %{DATA:captured_request_cookie} %{DATA:captured_response_cookie} %{NOTSPACE:termination_state} %{INT:actconn}/%{INT:feconn}/%{INT:beconn}/%{INT:srvconn}/%{NOTSPACE:retries} %{INT:srv_queue}/%{INT:backend_queue} (\{%{HAPROXYCAPTUREDREQUESTHEADERS}\})?( )?(\{%{HAPROXYCAPTUREDRESPONSEHEADERS}\})?( )?"(<BADREQ>|(%{WORD:http_verb} (%{URIPROTO:http_proto}://)?(?:%{USER:http_user}(?::[^#]*)?#)?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"
This works well, up to some unexpected null in the syslog_server in a HOSTNAME section
"syslog_server": [
[
"tst-proxy202"
]
],
"HOSTNAME": [
[
"tst-proxy202",
null <<<<<<<<<
]
],
"IP": [
[
null,
null
]
],
"IPV6": [
[
null,
null,
null
]
],
"IPV4": [
[
null,
"10.64.111.222",
null
]
],
I did parse this with https://grokdebug.herokuapp.com/
and the patterns IPORHOST, and the IPORHOST
https://grokdebug.herokuapp.com/patterns#
works well against the hostname
tst-proxy202
%{IPORHOST:syslog_server}
{
"syslog_server": [
[
"tst-proxy202"
]
],
"HOSTNAME": [
[
"tst-proxy202"
]
],
"IP": [
[
null
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
null
]
]
}
Any idea what might be the problem?
If I understood you correctly you are trying to get rid of that null value. Well, the null value occurs because of the last part of the HAPROXYHTTP pattern (where it says ?(?:%{URIHOST:http_host})?(?:%{URIPATHPARAM:http_request})?( HTTP/%{NUMBER:http_version})?))?"). It somehow adds an empty HOSTNAME. Luckily, this is not a serious problem and here is why:
The default options of the grok filter include named_captures_only => true (docs) and keep_empty_captures => false (docs). Try these two options in the grok debugger and your output looks pretty clean. In logstash you don't have to change anything.
If logstash misinterprets your hostname try to retrieve it from the grok values yourself (e.g. use the mutate filter):
filter {
mutate {
replace => { "HOSTNAME" => "%{syslog_server}" }
}
}
Please let me know if you have further problems.

Separate output values from a single grok query?

I've been capturing web logs using logstash, and specifically I'm trying to capture web URLs, but also split them up.
If I take an example log entry URL:
"GET https://www.stackoverflow.com:443/some/link/here.html HTTP/1.1"
I use this grok pattern:
\"(?:%{NOTSPACE:http_method}|-)(?:%{SPACE}http://)?(?:%{SPACE}https://)?(%{NOTSPACE:http_site}:)?(?:%{NUMBER:http_site_port:int})?(?:%{GREEDYDATA:http_site_url})? (?:%{WORD:http_type|-}/)?(?:%{NOTSPACE:http_version:float})?(?:%{SPACE})?\"
I get this:
{
"http_method": [
[
"GET"
]
],
"SPACE": [
[
" ",
null,
""
]
],
"http_site": [
[
"www.stackoverflow.com"
]
],
"BASE10NUM": [
[
"443"
]
],
"http_site_url": [
[
"/some/link/here.html"
]
],
"http_type": [
[
"HTTP"
]
]
}
The trouble is, I'm trying to ALSO capture the entire URL:
https://www.stackoverflow.com:443/some/link/here.html
So in total, I'm seeking 4 separate outputs:
http_site_complete https://www.stackoverflow.com:443/some/link/here.html
http_site www.stackoverflow.com
http_site_port 443
http_site_url /some/link/here.html
Is there some way to do this?
First, look at the built-in patterns for dealing with URLs. Putting something like URIHOST in your pattern will be easier to read and maintain that a bunch od WORDs or NOTSPACEs.
Second, once you have lots of little fields, you can always use logstash's filters to manipulate them. You could use:
mutate {
add_field => { "http_site_complete", "%{http_site}:%{http_site_port}%{http_site_url}" }
}
}
Or you could get fancy with your regexp and use a named group:
(?<total>%{WORD:wordOne} %{WORD:wordTwo} %{WORD:wordThree})
which would individually capture three fields and make one more field from the whole string.

Resources