hopefully someone can help, i've been trying to set up nginx with virtual hosts but the default host always overrides any of the others that i specify.
here is my config file found in /etc/nginx/sites-enabled
server {
listen 80;
server_name sub.example.com;
return 404;
}
server {
listen 80 default;
server_name *.example.com;
return 501;
}
and the access.log shows
*.*.*.* - - [30/Oct/2013:04:09:11 +0400] "GET / HTTP/1.1" 501 582 "http://sub.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36"
*.*.*.* - - [30/Oct/2013:04:09:14 +0400] "GET / HTTP/1.1" 501 582 "http://www.example.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.101 Safari/537.36"
thanks in advance for any help / ideas.
Ant
Related
I am trying to get the data from a site using requests using this simple code (running on Google Colab):
import requests, json
def GetAllStocks():
url = 'https://iboard.ssi.com.vn/dchart/api/1.1/defaultAllStocks'
res = requests.get(url)
return json.loads(res.text)
This worked well until this morning and I could not figure out why it is returning "TooManyRedirects: Exceeded 30 redirects." error now.
I can still get the data just by browsing the url directly from Google Chrome in Incognito mode so I donot think this is because of the Cookies. I tried passing the whole headers but still it does not work. I tried passing 'allow_redirects=False' and the returned status_code is 302.
I am not sure if there is anything I could try as this is so strange to me.
Any guidance is much appreciated. Thank you very much!
You need to send user-agent header to mimic a regular browser behaviour.
import requests, json, random
def GetAllStocks():
user_agents = [
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:101.0) Gecko/20100101 Firefox/101.0",
"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:77.0) Gecko/20190101 Firefox/77.0",
"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:77.0) Gecko/20100101 Firefox/77.0",
]
headers = {
"User-Agent": random.choice(user_agents),
"Accept": "application/json",
}
url = "https://iboard.ssi.com.vn/dchart/api/1.1/defaultAllStocks"
res = requests.get(url, headers=headers)
return json.loads(res.text)
data = GetAllStocks()
print(data)
I am trying to use Node.js to read phoenix channels using npm package phoenix-channels. Phoenix channels are multiplexed on top of websockets. I'm using an NGINX proxy in front of my phoenix webserver, so for NGINX, it's just a websocket.
Phoenix channels work fine going to a web page, as you can see here (you'll see data coming through in the web page).
It also works fine from nodejs on my internal network:
test_chan.js (with explicit IP and port):
const { Socket } = require('phoenix-channels')
let socket = new Socket("https://192.168.1.113:4445/socket")
socket.connect()
// Now that you are connected, you can join channels with a topic:
let channel = socket.channel("room:lobby", {})
channel.on("new_msg", payload => {
console.log(`${payload.body}`);
});
channel.join()
.receive("ok", resp => { console.log("Joined successfully", resp) })
.receive("error", resp => { console.log("Unable to join", resp) })
However if I replace the explicity IP:PORT address with the domain name, and run it from externally, it doesn't work (the only difference here from the script above is the URL):
test_chan.js (through domain name, and via my NGINX proxy):
const { Socket } = require('phoenix-channels')
let socket = new Socket("https://suprabonds.com/socket")
socket.connect()
// Now that you are connected, you can join channels with a topic:
let channel = socket.channel("room:lobby", {})
channel.on("new_msg", payload => {
console.log(`${payload.body}`);
});
channel.join()
.receive("ok", resp => { console.log("Joined successfully", resp) })
.receive("error", resp => { console.log("Unable to join", resp) })
So the suprabonds.com websockets works fine in a browser through the proxy, but doesn't work as a nodejs script.
Here is my nginx conf for suprabonds.com:
sites-enabled relevant server section:
server {
server_name suprabonds.com www.suprabonds.com;
location / {
proxy_pass http://localhost:4445;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/suprabonds.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/suprabonds.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
Any idea as to what I'm doing wrong?
EDIT
Here are the /var/log/nginx/access.log latest entries:
86.143.74.170 - - [22/Apr/2021:15:52:56 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
86.143.74.170 - - [22/Apr/2021:15:52:58 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
86.143.74.170 - - [22/Apr/2021:15:53:03 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
86.143.74.170 - - [22/Apr/2021:15:53:08 +0000] "GET /phoenix/live_reload/socket/websocket?vsn=2.0.0 HTTP/1.1" 101 143 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
86.143.74.170 - - [22/Apr/2021:15:53:08 +0000] "GET /socket/websocket?token=undefined&vsn=2.0.0 HTTP/1.1" 101 113098 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
86.143.74.170 - - [22/Apr/2021:15:53:19 +0000] "GET /phoenix/live_reload/socket/websocket?vsn=2.0.0 HTTP/1.1" 101 79 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
86.143.74.170 - - [22/Apr/2021:15:53:19 +0000] "GET /socket/websocket?token=undefined&vsn=2.0.0 HTTP/1.1" 101 27025 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
86.143.74.170 - - [22/Apr/2021:15:53:20 +0000] "GET /socket/websocket?token=undefined&vsn=2.0.0 HTTP/1.1" 101 479 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
86.143.74.170 - - [22/Apr/2021:15:53:20 +0000] "GET /phoenix/live_reload/socket/websocket?vsn=2.0.0 HTTP/1.1" 101 79 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) Gecko/20100101 Firefox/87.0"
86.143.74.170 - - [22/Apr/2021:15:53:23 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
86.143.74.170 - - [22/Apr/2021:15:53:24 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
86.143.74.170 - - [22/Apr/2021:15:53:26 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
86.143.74.170 - - [22/Apr/2021:15:53:31 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
86.143.74.170 - - [22/Apr/2021:15:53:41 +0000] "GET /socket/websocket?vsn=1.0.0 HTTP/1.1" 301 178 "-" "-"
The firefox ones are the ones that work fine. The others (with 301 178 in them) are the ones from the non-working Node.js script (that is, the one using the domain name). The error.log file in the same location is empty.
Please note that I'm also using noip dynamic dns.
If you change your URL to:
let socket = new Socket("wss://suprabonds.com/socket/websocket?token=undefined")
or
let socket = new Socket("wss://suprabonds.com/socket/websocket?vsn=1.0.0")
It will connect
We are loading access logs data into elasticsearch using logstash.log file data look like below.
2020-12-14 05:19:27.441 10.20.20.198 - narayana.sathya [14/Dec/2020:05:19:27 +0000] "GET /zoomdata/api/groups/5c9349a029a3fa0700a243ae HTTP/1.1" 200 5552 "https://sidcpdata.abc.com:8443/zoomdata/visualization/5abb7a37498e961613d64bea+5ea7ce37ed982daaa8019c75" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 Edg/87.0.664.60" 315
Could anybody help me to get GROK pattern for above file , i have written below GROK patten in logstash configuration file but getting error.
grok {
match => [ "message", "%{DATESTAMP_12H:timestamp} %{NUMBER:ip} %{WORD:user} %{DATESTAMP_12H:timestamp}
%{WORD:api_details} %{NUMBER:responce_type} %{NUMBER:type}
%{WORD:dashbaord} %{GREEDYDATA:daemon_message}" ]
}
Try this pattern :
%{TIMESTAMP_ISO8601:Time1}\s%{IPV4:IP}\s-\s%{NOTSPACE:UserName}\s\[%{NOTSPACE:TIME2}.*?\"%{WORD:APIMethod}\s%{URIPATH:API}\s%{NOTSPACE:Protocol}\"\s%{NUMBER:ResponseCode}\s%{NUMBER:PORT}\s\"%{URI:URL}%{GREEDYDATA:daemon_message}"
views.py:
def showLoginPage(request):
if request.method == "POST":
try:
body_unicode = request.body.decode('utf-8')
if 'csrfmiddlewaretoken' not in body_unicode:
body = json.loads(body_unicode)
user_obj = AuthenticateUser()
user_obj.validate_user(body)
c={}
c.update(csrf(request))
return redirect('http://abchostname/mainPage/')
# return redirect('/mainPage') This is another url which i want to redirect after
# successful login
except Exception as exe:
print("Inside Exception : ",exe)
raise
else:
print("Inside else {}".format(request.method))
return render(request, 'login.html')
#login_required(login_url="/login/")
def showMainPage(request):
return render(request, 'mainPage.html')
I want to redirect after a successful login, I see the login is getting successful and it is hitting by backend correctly as well.
[07/Jul/2020:06:59:29 +0000] "GET /login/ HTTP/1.1" 200 2082 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
[07/Jul/2020:06:59:36 +0000] "POST /login/ HTTP/1.1" 200 2081 "http://abchostname/login/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
[07/Jul/2020:06:59:36 +0000] "POST /login/ HTTP/1.1" 302 306 "http://abchostname/login/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.18362"
In the third option 302 status code is shown which means it is redirecting. I need some help on this.
To perform a redirect, its better to give named urls and acccess url with the name,
return redirect('main-page')
redirect() will try to use its given arguments to reverse a URL.
path('/main-page/', showMainPage, name='main-page')
Even if giving the url directly, dont give the full url, give a relative url like:
return redirect('/mainPage/')
Read More: https://realpython.com/django-redirects/#:~:text=Django%20Redirects%3A%20A%20Super%20Simple%20Example,-In%20Django%2C%20you&text=Just%20call%20redirect()%20with,then%20return%20from%20your%20view.&text=Assuming%20this%20is%20the%20main,to%20%2Fredirect%2Dsuccess%2F%20.
I need to collect the user access log in the application, mainly the name and version of the browser he is using. However, morgan is bringing many details that I don't need, can you help me?
Currently:
Firefox
::1 - OPTIONS - /signin - 204 - 0 - 0.126 ms http://localhost:8080/auth - Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0 -
Chrome
::1 - POST - /signin - 200 - 545 - 106.758 ms http://localhost:8080/auth - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 -
::1 - OPTIONS - /signin - 204 - 0 - 0.163 ms http://localhost:8080/auth - Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36 -
Expected:
Firefox
::1 - OPTIONS - /signin - 204 - 0 - 0.126 ms http://localhost:8080/auth - Firefox/76.0 -
Chrome
::1 - POST - /signin - 200 - 545 - 106.758 ms http://localhost:8080/auth -
Chrome/83.0.4103.61 -
My code:
app.use((req,res,next) => {
const logger = morgan(function (tokens, req, res) {
return [
tokens['remote-addr'](req, res), '-',
tokens.method(req, res), '-',
tokens.url(req, res), '-',
tokens.status(req, res), '-',
tokens.res(req, res, 'content-length'), '-',
tokens['response-time'](req, res), 'ms',
tokens.referrer(req, res), '-',
tokens['user-agent'](req, res), '-',
].join(' ')
})
logger(req,res,next)
})
var result = accessLogEntry.replace(/(?<=auth\s-\s).*(?=(?:Firefox|Chrome)\/[\d\.]+)/g, "");
This takes care of most of your problem. In your 2nd chrome example it will leave both the Chrome and Safari versions listed. You'll need to determine which one is the correct browser. This will trim out all the garbage in the middle of the log entry.
Demo
I'm use a component called:
app.use(require('express-useragent').express())
const navigator = req.useragent.browser