https website cannot be properly loaded without proxy - .htaccess

I have a website that has in the homepage a forums (created with Xenforo). I have recently put HTTPS with Let's encrypt (I have enabled it server-side with cPanel). The website worked fine with HTTP.
Now that I have HTTPS I am having issues because some people can access the website as always but others cannot. The ones who can't open my website have to use a proxy and then the website loads.
This error has started happening after I have edited a line in my .htaccess file:
# Mod_security can interfere with uploading of content such as attachments. If you
# cannot attach files, remove the "#" from the lines below.
<IfModule mod_security.c>
SecFilterEngine Off
SecFilterScanPOST Off
</IfModule>
ErrorDocument 401 default
ErrorDocument 403 default
ErrorDocument 404 default
ErrorDocument 405 default
ErrorDocument 406 default
ErrorDocument 500 default
ErrorDocument 501 default
ErrorDocument 503 default
<IfModule mod_rewrite.c>
RewriteEngine On
# I HAVE ADDED THESE 2 NEW LINES
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://forums.example.com/$1 [R,L]
# If you are having problems with the rewrite rules, remove the "#" from the
# line that begins "RewriteBase" below. You will also have to change the path
# of the rewrite to reflect the path to your XenForo installation.
#RewriteBase /xenforo
# This line may be needed to enable WebDAV editing with PHP as a CGI.
#RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -l [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule ^.*$ - [NC,L]
RewriteRule ^(data/|js/|styles/|install/|favicon\.ico|crossdomain\.xml|robots\.txt) - [NC,L]
RewriteRule ^.*$ /index.php [NC,L]
</IfModule>
I have added these 2 lines:
RewriteCond %{SERVER_PORT} 80
RewriteRule ^(.*)$ https://forums.example.com/$1 [R,L]
And now I have this weird issue: some people can access my website, others cannot and they have to use a proxy!
I have added those rules because I need to redirect all http to https so http://forums.example.com/ must become https://forums.example.com/. I have never had this issue before. Any idea?

First
What's the specific error message that these people are getting?
When I was building Greenlock I encountered a similar problem and it turned out to be that that certificate was not being loaded correctly, so I'm assuming that this is a TLS "privacy error" from the browsers, not a DNS or HTTP issue.
Next
I'm not familiar with cPanel, but I am very familiar with the ACME standard and clients.
Greenlock, certbot, and many other Let's Encrypt clients use a convention of naming the certificate files like this:
privkey.pem
cert.pem
chain.pem
fullchain.pem (cert.pem + chain.pem)
Some also have bundle.pem (fullchain.pem + privkey.pem).
Many web servers call for CRT and KEY in their documentation. Intuitively you might think that CRT would be cert.pem and KEY is privkey.pem.
This is usually incorrect.
CRT is fullchain.pem
If your site is configured to use cert.pem as the CRT instead of fullchain.pem you will have the issue you describe.
The reason is that anyone who has visited any site that properly uses the same intermediate authority as you will see the page as intended - the necessary chain.pem will already exist in the browser's cache.
However, anyone with a browser that doesn't have that missing piece already in a cache will get a security error.
Why would it work through a proxy?
It depends on the type of "proxy" - because that can mean different things to different people.
My guess is that the proxy is being used for more sites than the person's browser is (particularly a lot of small hobbyist sites that are using that same chain) and perhaps the proxy is actually downloading the site, decrypting it, and then relaying it, or perhaps the proxy is somehow supplementing the certificate chain with its own cache.
Possible Solution for Browser Privacy Error
Your problem may be completely different from the problem that I've had. It may be coincidence that the symptoms sound so similar.
I don't want to lead you down a rabbit hole that won't get you anywhere, but I think that checking your settings to make sure that you're using fullchain.pem and not cert.pem is an important first step.
Possible Solution for .htaccess Redirects
The issue with the redirects sounds coincidental to me. I doubt it's related.
Most likely once your site forced https, more visitors with the browsers that didn't have Let's Encrypt intermediate certificates in their caches suddenly started to notice the problem because they were now affected.
However, if you can back out those changes and confirm that HTTPS (SSL-enabled) works for those users, then I'd suggest that instead of doing redirects that you try adding headers that will do the same:
Upgrade-Insecure-Requests: 1
Strict-Transport-Security: max-age=86400

Related

500 error for html site on shared hosting, is my .htaccess syntax wrong?

First time here. I'm actually an Industrial & Systems Engineer from Mexico, trying to become a self-taught Web Developer/Graphic Designer. I'M SORRY if my english is flawed & for being such a noob, especially if my question was already asked by someone else, or whatever. (I did search questions, though)
I just finished (out of a bought html template) the 1º stage of the website (& everything design wise) of a new company (which is my client): https://www.starhauss.com/
The site works fine... considering what was requested of me, considering it's the 1º stage, it has minor flaws... I think.
ANYWAY...
I need to do (& tried) several things with .htaccess for my client, ASAP:
Redirect to custom 404
Declare Spanish language (the site will soon have an english version on /en/ directory)
Block sensible Server info
Disable directory listings
Remove www from urls
Remove html file extensions from urls
Block hidden files from appearing
Block "risky" files from appearing
Force https on the urls
Check & correct simple spelling errors on urls
I researched the Apache guides & everything, from different sources & I came up with my own .htaccess file, but I'm not sure if the syntax is wrong, or if I need to contact the respective Hosting Support Guys so they allow my file to be executed, because when it's uploaded (RIGHT NOW IT'S NOT UPLOADED) with CyberDuck to the root directory encoding text via US-ASCII with 755 file permissions, accessing my site results in a 500 Internal Server Error. I don't know what modules are on by default on the server of the shared hosting company, but of what I understand, they do work with the Apache Platform.
My .htaccess code is the next
# redirect not found
ErrorDocument 404 /404.html
# declare language for multilingual sites, adding a .htaccess file for each language subdirectory
DefaultLanguage es
# prevent server from outputing sensible information
ServerSignature Off
# disable directory listings, only when on
Options -Indexes
# requirement to change urls & such
RewriteEngine On
# use in subdirectories if rewrite rules are not working properly
# RewriteBase /
# mod_rewrite.c wrapper works mainly with WordPress & such
# <IfModule mod_rewrite.c>
# remove www from urls
RewriteCond %{HTTPS} !=on
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ http://%1%{REQUEST_URI} [R=301,L]
# remove html file extension
RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^\.]+)$ $1.html [NC, L]
# block hidden files
RewriteCond %{SCRIPT_FILENAME} -d [OR]
RewriteCond %{SCRIPT_FILENAME} -f
RewriteRule "(^|/)\." - [F]
# </IfModule>
# block risky files
<FilesMatch "(^#.*#|\.(bak|config|dist|fla|inc|ini|log|psd|sh|sql|sw[op])|~)$">
Order allow,deny
Deny from all
Satisfy All
</FilesMatch>
# force https
<IfModule mod_headers.c>
Header set Strict-Transport-Security max-age=16070400;
</IfModule>
# check & correct url spelling errors, useful for SEO
<IfModule mod_speling.c>
CheckSeplling On
</IfModule>
So, what do you think guys? Should it work? Is something wrong with the code syntax, the order or whatever? Is something missing Server Side for the file to work & if so, how can I know, if I have no access to the Httpd.Conf? Should I get in contact with the shared hosting support for them to allow my file?
RewriteRule ^([^\.]+)$ $1.html [NC, L]
You have an erroneous space in the RewriteRule flags argument. This will result in a 500 Internal Server Error and if you look at your server's error.log you will see an error reported like "RewriteRule: bad flag delimiters".
Whenever you get a 500 error (which is just a generic server response), you need to check the server's error log for the details of that error.
In this case, [NC, L] should be [NC,L]. Although the NC flag is superfluous here, so [L] is sufficient.
<IfModule mod_speling.c>
CheckSeplling On
</IfModule>
You've spelt CheckSpelling wrong. (This would also trigger a 500 error response.)
Force https on the urls
# remove www from urls
RewriteCond %{HTTPS} !=on
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ http://%1%{REQUEST_URI} [R=301,L]
You're not actually "forcing HTTPS" anywhere.
The above only removes www on HTTP only. Why are you specifically checking that the request is not HTTPS in the above www to non-www redirect and redirecting to HTTP, not HTTPS?
You could change the above to the following in order to remove www on HTTP and HTTPS (canonicalising HTTPS at the same time):
# Remove www (redirect to HTTPS)
RewriteCond %{HTTP_HOST} ^www\.(.+)$ [NC]
RewriteRule ^ https://%1%{REQUEST_URI} [R=301,L]
And add an additional rule to force HTTPS (however, see note below about HSTS*1):
# Force HTTPS
RewriteCond %{HTTPS} !=on
RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
# force https
<IfModule mod_headers.c>
Header set Strict-Transport-Security max-age=16070400;
</IfModule>
This doesn't strictly "force https".
This "HSTS header" instructs the browser to always request HTTPS, once the user has visited the site over HTTPS. It does nothing if the user has only visited the site over HTTP (hence the requirement to first redirect/force the user to HTTPS - see above). However, this is not complete - you need to redirect on the same host first (*1ie. reverse the two redirects above) and also set the header on the 301 redirect that removes www over HTTPS (this directive does not). And ideally you should avoid sending the header over plain HTTP (although that does not strictly matter, as the browser will simply ignore it).
I would avoid setting HSTS initially, until your site is working correctly over HTTPS. You should consider HSTS as a one-way trip, it is problematic to backtrack on this (max-age=16070400 will persist this for 6+ months for anyone who visits over HTTPS).
See the following related question on the CodeReview stack regarding HSTS implementation: https://codereview.stackexchange.com/questions/250805/hsts-recommendations-in-htaccess
Order allow,deny
Deny from all
Satisfy All
These are Apache 2.2 directives and are formerly deprecated on Apache 2.4. I assume you are using Apache 2.4 (Apache 2.2 was EOL almost 2 years ago). You should use the Apache 2.4 directives instead:
Require all denied

Rewriting .htaccess to remove sub-directory from URL

I want to mask a folder in URL, so instead of www.mysite.com /employers/university-of-worcester/profile.html there would be www.mysite.com /university-of-worcester/profile.html , meaning that sub-directory "employers" is hidden.
There will be many folders created inside "employers" folder.
This is a matter of modifying .htaccess, I have tried a lot of solutions that I have found on stack.
The latest line of code I tried to add is:
RewriteRule ^employers/(.*)$ /$1 [L]
My .htaccess looks like this now:
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
RewriteRule ^employers/(.*)$ /$1 [L]
</IfModule>
If I go to URL www.mysite.com/employers/ this will redirect to www.mysite.com/ (the homepage), which I am happy about, but if I try www.mysite.com/university-of-worcester/profile.html I get "The page can’t be found.", but the home profile.html is definitly inside "university-of-worcester" folder.
I am using wordpress.
This probably is what you are looking for:
RewriteEngine on
RewriteRule ^/employers/?(.*)$ /$1 [R=301,QSA]
RewriteCond %{REQUEST_URI} !^/employers/
RewriteRule ^/?(.*)$ /employers/$1 [END,QSA]
This will redirect direct requests to that folder and internally rewrite requests to that folder.
It is a good idea to start out with a 302 temporary redirection and only change that to a 301 permanent redirection later, once you are certain everything is correctly set up. That prevents caching issues while trying things out...
In case you receive an internal server error (http status 500) using the rule above then chances are that you operate a very old version of the apache http server. You will see a definite hint to an unsupported [END] flag in your http servers error log file in that case. You can either try to upgrade or use the older [L] flag, it probably will work the same in this situation, though that depends a bit on your setup.
This rule will work likewise in the http servers host configuration or inside a dynamic configuration file (".htaccess" file). Obviously the rewriting module needs to be loaded inside the http server and enabled in the http host. In case you use a dynamic configuration file you need to take care that it's interpretation is enabled at all in the host configuration and that it is located in the host's DOCUMENT_ROOT folder.
And a general remark: you should always prefer to place such rules in the http servers host configuration instead of using dynamic configuration files (".htaccess"). Those dynamic configuration files add complexity, are often a cause of unexpected behavior, hard to debug and they really slow down the http server. They are only provided as a last option for situations where you do not have access to the real http servers host configuration (read: really cheap service providers) or for applications insisting on writing their own rules (which is an obvious security nightmare).

Some users reporting site not secure

I have a client who just installed an SSL certificate. I added this to the .htaccess file to force users to redirect to https and force them to www:
# Redirect bare domain to www and HTTPS
RewriteCond %{HTTP_HOST} !^www\.
RewriteRule (.*) https://www.twentyteninc.com/$1 [R=301,L]
# Redirect HTTP to HTTPS
RewriteCond %{HTTPS} off
RewriteRule (.*) https://%{HTTP_HOST}/$1 [R=301,L]
I also addressed all the mixed content warnings. Now, everything redirects to https and I get the beautiful lock symbol in all browsers I test with.
Unfortunately, though, some users are still seeing the "Not Secure" messages:
If I run a Qualys SSL Test, it comes back with an A score:
What could be causing this? Why does everything look secure for me but not for some?
My problem was that the WordPress theme the client is using was pulling the favicon over HTTP instead of HTTPS and for whatever reason, my browser (and all the other browsers I tested with) was simply choosing not to load it at all.
This caused me never to get a mixed content warning.
After staring at the screenshot above for long enough, I noticed that the favicon was loading and that it wasn't for me which tipped me off to go digging through the theme for it.

htaccess doesn't allow my domain

I created a htaccess file and put it in public_html directory(root of my server):
<IfModule mod_rewrite.c>
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http://mdpcomics.ir/.*$ [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ https://blogvault.net/wp-content/uploads/2014/12/no-hotlinking.png [NC,R,L]
</ifModule>
it blocks hotlinking and shows another image. but it doesn't allow my own domain too.
i.e. this link should not show the image:
http://dl.mdpcomics.ir/logo.png
but this link should show it. I mean I want it to show the image by using this link:
http://mdpcomics.ir/?imagename=logo
but it won't allow that too.
I already searched everywhere and I tried so many suggestions but all of them worked as follow:
everyone can see images
no one can see images including myself O.o
my server os is linux and my panel is directadmin
Edit:
I realized that my host has a fake or invalid ip: 178.63.56.20320
I got that ip by php code:
echo $_SERVER['REMOTE_ADDR'];
Maybe this is your problem: In your RewriteRule it says https://, but your RewriteCond is testing for http://. You can make it test for both by simply adding s?:
RewriteCond %{HTTP_REFERER} !^https?://mdpcomics.ir/.*$ [NC]
Further things to be aware of:
clients won't send the Referer in a non-HTTPS request if the request originated from an HTTPS location
some clients never send the Referer header or set it to something totally different in order to protect their privacy
To catch these cases I would add another condition:
# Assume client sent "real" Referer if it begins with http(s)://
RewriteCond %{HTTP_REFERER} ^https?:// [NC]
The downside is, images can be hotlinked from foreign HTTPS locations if your site is running normal HTTP.

https to http redirection when ssl is not installed

We have a site which was https enabled. After a site revamp we have removed the certificate and https://www.foo.com is not http://www.foo.com.
There are many links in other sites, which link to th old https site, which we have no control. Is there something we can do in out side to redirect these links to home page atleast?
Will url rewriting work in this case?
There were some broken link which we fixed with a custom 404 page and tracking the links.
Does this belong to serverfault?
You're pretty much out of luck - you can do URL rewriting as #Josh says, but before the browser even gets that far, most of them will give the user a big warning message telling them the SSL certificate isn't valid, which will put off most of the visitors.
I'd recommend buying an SSL certificate - they're not so expensive - then doing the rewrite.
using mod_rewrite
# forces everything to non-secure if secure (http)
RewriteCond %{SERVER_PORT} =443
RewriteRule ^(.*)$ http://%{SERVER_NAME}/$1 [R,L]
Josh
The solution for IIS may be ISAPI_Rewrite 3. Here's the .htaccess:
RewriteBase /
RewriteCond %{HTTPS} on
RewriteRule ^(.*)$ http://%{HTTP_HOST}/$1 [R=301,L]

Resources