We are loading access logs data into elasticsearch using logstash.log file data look like below.
2020-12-14 05:19:27.441 10.20.20.198 - narayana.sathya [14/Dec/2020:05:19:27 +0000] "GET /zoomdata/api/groups/5c9349a029a3fa0700a243ae HTTP/1.1" 200 5552 "https://sidcpdata.abc.com:8443/zoomdata/visualization/5abb7a37498e961613d64bea+5ea7ce37ed982daaa8019c75" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.60" 315
Could anybody help me to get GROK pattern for above file , i have written below GROK patten in logstash configuration file but getting error.
grok {
match => [ "message", "%{DATESTAMP_12H:timestamp} %{NUMBER:ip} %{WORD:user} %{DATESTAMP_12H:timestamp}
%{WORD:api_details} %{NUMBER:responce_type} %{NUMBER:type}
%{WORD:dashbaord} %{GREEDYDATA:daemon_message}" ]
}
Try this pattern :
%{TIMESTAMP_ISO8601:Time1}\s%{IPV4:IP}\s-\s%{NOTSPACE:UserName}\s\[%{NOTSPACE:TIME2}.*?\"%{WORD:APIMethod}\s%{URIPATH:API}\s%{NOTSPACE:Protocol}\"\s%{NUMBER:ResponseCode}\s%{NUMBER:PORT}\s\"%{URI:URL}%{GREEDYDATA:daemon_message}"
Related
I am trying to get the data from a site using requests using this simple code (running on Google Colab):
import requests, json
def GetAllStocks():
url = 'https://iboard.ssi.com.vn/dchart/api/1.1/defaultAllStocks'
res = requests.get(url)
return json.loads(res.text)
This worked well until this morning and I could not figure out why it is returning "TooManyRedirects: Exceeded 30 redirects." error now.
I can still get the data just by browsing the url directly from Google Chrome in Incognito mode so I donot think this is because of the Cookies. I tried passing the whole headers but still it does not work. I tried passing 'allow_redirects=False' and the returned status_code is 302.
I am not sure if there is anything I could try as this is so strange to me.
Any guidance is much appreciated. Thank you very much!
You need to send user-agent header to mimic a regular browser behaviour.
import requests, json, random
def GetAllStocks():
user_agents = [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0",
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:77.0) Gecko/20190101 Firefox/77.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:77.0) Gecko/20100101 Firefox/77.0",
]
headers = {
"User-Agent": random.choice(user_agents),
"Accept": "application/json",
}
url = "https://iboard.ssi.com.vn/dchart/api/1.1/defaultAllStocks"
res = requests.get(url, headers=headers)
return json.loads(res.text)
data = GetAllStocks()
print(data)
views.py:
def showLoginPage(request):
if request.method == "POST":
try:
body_unicode = request.body.decode('utf-8')
if 'csrfmiddlewaretoken' not in body_unicode:
body = json.loads(body_unicode)
user_obj = AuthenticateUser()
user_obj.validate_user(body)
c={}
c.update(csrf(request))
return redirect('http://abchostname/mainPage/')
# return redirect('/mainPage') This is another url which i want to redirect after
# successful login
except Exception as exe:
print("Inside Exception : ",exe)
raise
else:
print("Inside else {}".format(request.method))
return render(request, 'login.html')
#login_required(login_url="/login/")
def showMainPage(request):
return render(request, 'mainPage.html')
I want to redirect after a successful login, I see the login is getting successful and it is hitting by backend correctly as well.
[07/Jul/2020:06:59:29 +0000] "GET /login/ HTTP/1.1" 200 2082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
[07/Jul/2020:06:59:36 +0000] "POST /login/ HTTP/1.1" 200 2081 "http://abchostname/login/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
[07/Jul/2020:06:59:36 +0000] "POST /login/ HTTP/1.1" 302 306 "http://abchostname/login/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
In the third option 302 status code is shown which means it is redirecting. I need some help on this.
To perform a redirect, its better to give named urls and acccess url with the name,
return redirect('main-page')
redirect() will try to use its given arguments to reverse a URL.
path('/main-page/', showMainPage, name='main-page')
Even if giving the url directly, dont give the full url, give a relative url like:
return redirect('/mainPage/')
Read More: https://realpython.com/django-redirects/#:~:text=Django%20Redirects%3A%20A%20Super%20Simple%20Example,-In%20Django%2C%20you&text=Just%20call%20redirect()%20with,then%20return%20from%20your%20view.&text=Assuming%20this%20is%20the%20main,to%20%2Fredirect%2Dsuccess%2F%20.
Iam trying to create a grok logstash filter for my log4js log.
The code in my nodejs app is as follows:
var httpLogFormat = ':remote-addr - - [:date] ":method :url ' + 'HTTP/:http-version" :status :res[content-length] ' + '":referrer" ":user-agent" :response-time';
log4js.loadAppender('file');
log4js.addAppender(log4js.appenders.file('logs/access.log'), 'access');
var logger = log4js.getLogger('access');
app.use(log4js.connectLogger(logger, { level: 'auto', format: httpLogFormat }));
This results in the following log message:
[2017-01-31 08:54:32.491] [WARN] access - 192.1.1.10 - - [Tue, 31 Jan 2017 07:54:32 GMT] "GET /api/test HTTP/1.0" 304 undefined "https://localhost.com/test" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36" 111
My current grok filter looks like this (UPDATED):
grok {
match => { "message" => "\[%{HTTPDATE:timestamp}\] \[%{WORD:loglevel}\] %{WORD:logtype} - %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \"%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})\" %{NUMBER:response} - \"%{DATA:rawrequest}\" \"%{QS:agent}\""}
}
There is some parsing errors, and i suspect it is due to the [] but i'am unsure.
http://grokconstructor.appspot.com/ fails with:
NOT MATCHED. The longest regex prefix matching the beginning of this line is as follows:
prefix "
before match: [2017-01-31 08:54:32.491] [WARN] access - 192.1.1.10 - - [Tue, 31 Jan 2017 07:54:32 GMT]
after match: GET /api/test HTTP/1.0" 304 undefined "https://test.localhost.com/test" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.75 Safari/537.36" 111
I've updated the grok to work for your example. I think you were misusing a few of the types (QS for example you don't need to have the "'s around it):
\[%{GREEDYDATA:timestamp}\]\ \[%{WORD:loglevel}\]\ %{WORD:logtype}\ -\ %{IPORHOST:clientip}\ %{USER:ident}\ %{USER:auth}\ \[%{GREEDYDATA}\]\ \"%{WORD:verb}\ %{NOTSPACE:request}(?: HTTP\/%{NUMBER:httpversion}|)\"\ %{NUMBER:response}\ %{WORD}\ \"%{DATA:rawrequest}\"\ %{QS:agent}\ %{INT:time_taken}
Check the docs for other words you can use.
Your parsing issues are probably down to literal use of the [ and ] characters as they are used in regex's, they need to be escaped as in my example.
I'm new to grok in logstash and I have to parse the following log pattern.
Jul 26 09:46:37 abc-lb1 2016-07-26 09:46:37.245 +0200 abc-lb1 WF WARN UNRECOGNIZED_COOKIE 188.200.126.234 50011 10.50.51.25 443 global GLOBAL LOG NONE [Cookie\="_ga" Service-created\="769 days back" Reason\="No valid encrypted pair"] GET example.com/search.action?searchText\=EH-5H&token\=--0----EH-5H-- TLSv1.2 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" 188.200.126.234 50011 "-" https://example.com/my-account/login
I need to know How to avoid a set of strings in GROK
In the above logs, repeated time-stamps could be seen, I need to know, how to avoid the strings like:
Jul 26 09:46:37 abc-lb1
Suppose you need only two fields that is 2016-07-26 09:46:37.245 and https://example.com/my-account/login then your grok filter should be as follows:
grok{ match => {"message" => "%{TIMESTAMP_ISO8601:time} %{GREEDYDATA} %{URI:url}"} }
You will get the following output:
{
"time": [
[
"2016-07-26 09:46:37.245"
]
],
"url": [
[
"https://example.com/my-account/login"
]
]
}
Here you are avoiding the first few fields in your log line by directly starting off with 2016-07-26 09:46:37.245 and you are avoiding everything in between by not naming %{GREEDYDATA}. If you name %{GREEDYDATA} as %{GREEDYDATA:data} then you will the output as follows:
{
"time": [
[
"2016-07-26 09:46:37.245"
]
],
"data": [
[
"+0200 abc-lb1 WF WARN UNRECOGNIZED_COOKIE 188.200.126.234 50011 10.50.51.25 443 global GLOBAL LOG NONE [Cookie\\="_ga" Service-created\\="769 days back" Reason\\="No valid encrypted pair"] GET example.com/search.action?searchText\\=EH-5H&token\\=--0----EH-5H-- TLSv1.2 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" 188.200.126.234 50011 "-""
]
],
"url": [
[
"https://example.com/my-account/login"
]
]
}
Now you can apply the same steps to whichever fields you want to avoid.
you can debug the results here
I have some troubles filtering my logs using logstash because my pattern only works in the debugger
FilterString
127.0.0.1 - - [06/Jan/2016:15:43:41 +0000] "GET /index.php/banner/ajax/load/?sections=&_=1452095017076 HTTP/1.1" 200 74 "https://magento2-dev.argento.io/index.php/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36" "-"
My Pattern
%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}
When I use this in http://grokdebug.herokuapp.com/ everything is just fine, but it breaks when I use it in my config.
in logstash conf
filter {
grok {
match => [ 'message', '%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent}' ]
}
}
I strongly belive this is related to quotes or slashes.
I tried with single quotes, but no luck here.
thanks for helping me out