Is there a way for Passport to check if request came from mobile or web app when doing authentication? Because if request came from the web I want to return a view otherwise return a json payload.
This is my opinion,you can check user-agent in the request header ,its look like this(came from windows):
user-agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36
and this is came from my iPhone
User-Agent:Mozilla/5.0 (iPhone; CPU iPhone OS 10_3 like Mac OS X) AppleWebKit/602.1.50 (KHTML, like Gecko) CriOS/56.0.2924.75 Mobile/14E5239e Safari/602.1
and this is Android
User-Agent:Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Mobile Safari/537.36
so you can figer it out from user-agent,which request came from mobile or pc
If you have two different clients expecting different results, then you should explicitly send different requests, not try to guess which response is wanted from some header that isn't necessarily reliable. Plus, there's nothing keeping a mobile device from also accessing the web interface. You can either vary the path or vary a query string.
So, from web, you might use /login and from mobile, you might use /login-json or some different path that indicates you want json.
Or from web, you might use /login and from mobile, you might use /login?type=json.
I would NOT recommend using the user-agent header to detect the intent of the request. Instead, specify the intent directly in the request.
Related
In the past few years I've sometimes ran into websites that don't work in Firefox on Linux, and I'm trying to understand why so I can notify the owners with more than just a vague “it doesn't work”.
Now this happens of course. While most web developers do test in Firefox, not many will have tested their products in Firefox on Linux, and some really don't care. Some only target Chrome/Webkit and don't bother with Firefox at all. That is not what this question is about though.
There is something here that makes me suspect that there is an underlying cause that is repeated on seemingly unrelated websites, and I suspect some repeated bit of configuration code or web content serving library or application that does this. Something is fishy.
The problem
The websites affected return only a plain HTML message with a 403 HTTP status code for any resource requested; it looks like this:
Forbidden
You don't have permission to access / on this server.
These websites do work when:
The operating system is not a Linux distribution
or
The browser is not Firefox
Example websites
While I normally wouldn't include a link to someone else's website, in this case I do because it is the website of a doctors office. These websites should be available to any patient at all times for anything short of a imminently life threatening emergency (in which case the national emergency number should be called of course) to provide contact information in times of need.
This website displays the symptoms described above: https://www.huisartsenpraktijkdehaan.nl/
There are more websites though, but the pattern is always the same.
The user-agent string
Trying to figure out what is actually causing this seems simple enough though. If I change the user-agent string to that of Chrome, it works.
So my tentative conclusion is that this is purely a user-agent driven bug/feature.
Some further testing yields this:
These work
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36
Foo
Mozilla/5.0 (X11; Ubuntu; x86_64; rv:85.0) Gecko/20100101 Firefox/85.0
Mozilla/5.0 (X11; Ubuntu; inux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0
Mozilla/5.0 (X11; Ubuntu; Linu x86_64; rv:85.0) Gecko/20100101 Firefox/85.0
Mozilla/5.0 (X11; Xubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0
X11; Ubuntu; Linu
X11;Ubuntu;Linux
11; Ubuntu; Linux
These do not work
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:85.0) Gecko/20100101 Firefox/85.0
X11; Ubuntu; Linux
x11; ubuntu; linux
Hypothesis
Having the literal string X11; Ubuntu; Linux (case insensitive but including spaces and semi-colons as-is) in the user-agent HTTP header of your request triggers the broken behaviour.
The conundrum
I could, of course, reach out the owners of these websites (and eventually I will), but there is a catch. They likely won't use Firefox on Linux (because you would notice your own website being broke), and if they pass on the message to whoever maintains or built the website, the response may very well be “well it works for you, and it works for me, that user must have some weird virus-ridden computer and an ancient browser with a Bonzy Buddy toolbar”, or something similar.
So I want some more ammunition, and preferably a cause I can explain to anyone with a website like this. Even better would be to find out why this happens, and fix it at the source.
So what is happening here? Some Apache of Nginx module/config/plugin written by someone who really hates people who use Firefox on Linux? Some weird bug repeated on multiple sites?
Does anyone recognize this peculiar website behaviour?
I saw the link to this post in forwarded emails.
ErrorDocument 503 "Your connection was refused"
SetEnvIfNoCase User-Agent "X11; Ubuntu; Linux" bad_user
Deny from env=bad_user
Was set in the .htaccess to block "WP-login bots", since I use Ubuntu myself this was rather easy to replicate.
When I type url of a site to browser's address bar, browser sends a request to get the resource by the url. But when I go to different web sites (google.com, amazon.com, etc.), requests which initialize the page, have different headers for different sites.
Where browser gets the set of request's headers to load the page if browser has only information about URL of this resource at the first initialization?
for example when I go to google.com browser sends such request headers:
:authority: www.google.com
:method: GET
:path: /
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9,ru-RU;q=0.8,ru;q=0.7
cache-control: max-age=0
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: same-origin
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
For amazon.com, the request's headers are different:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,ru-RU;q=0.8,ru;q=0.7
Connection: keep-alive
Host: amazon.com
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36
When you type in a URL into the address the bar this needs to be translated to an HTTP request.
So typing www.google.com means you need to GET the default page (/) from that server. That's basically all covered in the first 4 lines in the first request.
The browser also knows what types of format it can accept. Mostly we deliver HTML back so text/html is certainly in there, but we also accept other formats - including the completely generic */* btw! :-)
Requests are often compressed (with either gzip, deflate or the newer brotli (br) format) so the browser tells the server which of those it supports in the accept-encoding header.
When you installed your browser you also set a default language so we can tell the server that. Some servers will return different content based on this.
Then there are some security headers (I won't go into these as quite complicated).
Finally we have the user-agent header. this is basically where the browser tells the server whether it's Chrome, or Firefox or whatever. But for historical reasons it's much longer than just "Chrome".
So basically the request headers are things the browser sends to the server to give it more information about the browser and it's capabilities. For a request that's just typed into the browser the request headers will basically be the same no matter what the URL is. For additional requests made by the page - e.g. by JavaScript code they may be different if it adds more headers.
As to the differences between the two example requests you gave:
Google uses HTTP/2 (or QUIC if using Chrome but for now that's basically HTTP/2 as far as this question is concerned). you can see this if you add the option Protocol column to developer tools.
HTTP/2 has a couple of changes from HTTP/1, namely:
HTTP Header Names are lower cased. Technically in HTTP/1 they are case insensitive, but by convention many tools like browser used title case (capitalising first letter of each word).
The request (e.g. GET / HTTP/1.1) is converted to pseudo headers beginning with a colon (:method: GET, :path: /...etc.).
Host is basically :authority in HTTP/2.
:scheme is basically new in HTTP/2 as previously it wasn't explicitly part of the HTTP request, and handled at a connection level.
Connection is defunct in HTTP/2. Even in HTTP/1.1 it defaulted to keep-alive so above header was not necessary but lots of browsers and other clients sent it for historical reasons.
I think that explains all the differences.
So how does the browser know whether to use HTTP/2 or HTTP/1.1? Which already has an answer on Stack Overflow, but basically it's decided when the HTTPS session is established if the server advises it can support HTTP/2 and the browser wants to use it.
I buy web traffic from several sources (including the major names in the industry) and recently got reports from advertisers that there's quite a bit of "invalid" traffic. They won't share which filter they use so I can block it on my end. I tested all the navigator properties, resolution, window size, modernizr features, etc, and the bad traffic seems to be spoofing everything.
After some testing, I found that using this code:
document.addEventListener('click', function() {
window.open('/save?' + navigator.userAgent ,'_blank');
});
In some cases, the saved user agent is different from the one saved on the top window. Meaning, a visit hits a page, in that page the user agent could be something like:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML,
like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/18.17763
Then that page uses window.open() to open a new window and reads the user agent again, and it will read something like this:
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
HeadlessChrome/72.0.3617.0 Safari/537.36
I tried all the usual methods, window.chrome, webdriver, permissions, plugins, fonts, reading those vars in an iframe, etc, they pass all the tests, the only thing that works is the window.open, but I obviously can't open a popup to filter the traffic.
Is there any way to detect this type of traffic?
I am writing a web scraper that I am trying to proxy, but can't quite figure out how to do it in Elixir.
I am using Hound running on top of a headless ChromeDriver. I purchased some proxy IPs through https://luminati.io and they offer both a chrome extension and a user/password base proxy server.
The webscraper actions comprise of a GenServer that represent a user scraping the web. There is no front end of the app, it accepts commands that are sent to it through a bot I built on Telegram, so when a user sends the login command for instance it triggers the login function of the GS.
At that point the GenServer will change the ChromeDriver session using Hound.change_session_to/2 and then log the user in.
This works great, but now I want to send every request through the proxy server via username and password. When changing the session with Hound, it allows the chromeOptions to be set as well.
ua = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
change_session_to(String.to_atom(account.username), %{browserName: "chrome", chromeOptions: %{"args" => ["--user-agent=#{ua}", "--proxy-server=http://user:password#proxy.luminati.io:22225"]}})
navigate_to "https://www.website.com/"
Another thing that I have tried doing is loading luminati's ChromeExtension that I would be able to use to proxy the traffic through, but I can't get the extension to load for each session. I downloaded the packed CRM chrome extension and placed it within my priv folder. When the session loads it seems to load the User Agent just fine, but the extension never starts. When I am trying to load the extension I am not running headless.
ua = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
priv_dir = :code.priv_dir(:boost_buddy)
change_session_to(String.to_atom(account.username), %{browserName: "chrome",
chromeOptions: %{"extensions" => ['#{priv_dir}/luminati/3.2_1'], "args" => ["-
-user-agent=#{ua}", "--proxy-server=http://user:password#proxy.luminati.io:22225"]}})
navigate_to "https://www.website.com/"
Does anyone have experience using chrome driver with Elixir? With Ruby and Java setting up the extension is typically no problem.
https://github.com/GoogleChrome/puppeteer/issues/659
-1 because this was the top result for googling "chrome headless extension"
Regarding sending each request through the proxy, I think you either need to interface with the chrome driver yourself (hijacking hound) or skip hound and use either chrome directly or through a selenium grid.
I think the issue stems from the fact that hound will initiate one single chrome instance, where the proxy settings will be defined. Further requests are done using that proxy.
So in order to achieve multiple proxy connections for different sessions you either need a way to set them through navigational steps (visiting a proxy website that then serves as a hard proxy) or use different browser instances altogether (I might be wrong though and perhaps there's an easier way of proxying the requests)
Can we use Mobile Tools module in Drupal6 with Varnish?
I doubt varnish will cache the index page and will not allow redirection to mobile version of the page.
Any work arround?
You want to make your server return different responses based on the used device/browser. This means your pages 'vary' based on the used User-Agent http request header, and in theory you should instruct any http proxy/cache in between to only use a cached version if the User-Agent string is the same by adding a http response header:
Vary: User-Agent
However, because browsers like Internet Explorer (unlike Chrome) use many slightly different User-Agent headers, this will completely kill your cache hit ratio. You need a smarter cache to understand that Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; WOW64; Trident/6.0) for your purposes is equal to Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.0; Trident/4.0; InfoPath.1; SV1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 3.0.04506.30), or any other user-agent string used by a desktop browser.
There are two options for you to solve this with Varnish:
1: Do mobile user-agent detection yourself in varnish logic, the same way mobile tools does it. E.g.:
vcl_recv {
if (req.http.user-agent ~ 'ipad|ipod|iphone|android|mini opera|blackberry|up.browser|up.link|mmp|symbian|smartphone|midp|wap|vodafone|o2|pocket|kindle|mobile|pda|psp|treo') {
hash += "mobile"
}
}
2: Or, always set a session cookie mobile=true or mobile=false after you've seen the first request, and only serve cached pages for requests with this cookie.
And after googling a bit, you should read: http://fangel.github.com/mobile-detection-varnish-drupal/