I am getting error 500 in my logs when using the below Rewrite Rules to block by User Agent and Referral URL. I have tried removing the referral url block but still getting error 500 for the user agent block. The traffic that is not block work properly. Just getting 500 for bots and refurl that are blocked. Any idea why is it giving me 500 instead of 403 in the log?
# Block by REFURL
RewriteEngine on
RewriteCond %{HTTP_REFERER} sample\.com [NC]
RewriteRule .* - [F]
# Block by User Agent
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} bot1 [NC,OR]
RewriteCond %{HTTP_USER_AGENT} bot2 [NC]
RewriteRule .* - [F]
# BLOCK BLANK USER AGENTS
RewriteCond %{HTTP_USER_AGENT} ^-?$
RewriteRule ^ - [F]
Here is sample of one of the log. My server use cpanel so I pull this from the raw access log. I only get error log when an IP has been block. It doesn't show anything for the 500 errirs.
xx.xx.xxx.xxx - - [02/Sept/2014:21:54:25 -0400] "GET / HTTP/1.0" 500 - "-" "Mozilla/5.0 (compatible; BadBot/5.0; +http://badbot.com/robot/)"
Related
In htaccess, how can i block every visitor except those who come from a specific domain
i tried this but without any success :
# serve everyone from specific-domain or specific-user-agent
RewriteCond %{HTTP_REFERER} ^https?://www.specific-domain.com
RewriteRule ^ - [L]
# everybody else receives a forbidden
RewriteRule ^ - [F]
ErrorDocument 403 /forbidden.html
Update : i had certain success with below code BUT it broked my webpage certainly because of the following parameters that overrride or disturbe appearance. if someone has a clue how to order it the good way ?
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^https://authorizedreferer.com
RewriteRule ^ - [L]
RewriteRule ^ https://unprotected.mydomain.com/ [R,L]
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>
# serve everyone from specific-domain or specific-user-agent
RewriteCond %{HTTP_REFERER} ^https?://www.specific-domain.com
RewriteRule ^ - [L]
# everybody else receives a forbidden
RewriteRule ^ - [F]
This will indeed allow requests that link from specific-domain.com (ie. this domain is the HTTP Referer) and block everything else. However, it will also block all requests for your static resources, that originate from your site, where your domain is the Referer. So, you need to also allow requests from your domain.
You should also probably allow an empty Referer header. ie. direct requests, when a user types the URL into their browser address bar. Also note that the Referer header can be suppressed in other ways depending on the referrer-policy as set by the originating website. The user themselves can also override the Referer header, so relying on the Referer header is not reliable.
Try the following:
# Serve everyone from specific-domain (and internal requests)
RewriteCond %{HTTP_REFERER} ^https?://www\.your-domain\.com/ [OR]
RewriteCond %{HTTP_REFERER} ^https?://www\.specific-domain\.com/
RewriteRule ^ - [L]
# everybody else receives a forbidden
RewriteRule ^ - [F]
And to allow an empty Referer, include an additional condition:
# Serve everyone from specific-domain (and internal requests and empty referer)
RewriteCond %{HTTP_REFERER} ^$ [OR]
RewriteCond %{HTTP_REFERER} ^https?://www\.your-domain\.com/ [OR]
RewriteCond %{HTTP_REFERER} ^https?://www\.specific-domain\.com/
RewriteRule ^ - [L]
Note that you are currently allowing http or https in the Referer. If this is always https then be specific and remove the ? (optional quantifier). ie. ^https://www\.specific-domain\.com/. And remember to backslash escape the literal dots.
I have this very simple rule in my .htaccess. Basically, i want to redirect all users except me during maintenance. For some reason, it is also redirecting me to maintenance.php.
I don't have any other .htaccess.
The .htaccess is found on My document root: /Users/myname/Sites
# redirect to maintenance page
RewriteCond %{REMOTE_ADDR} !^127\.0\.0\.1
RewriteCond %{REQUEST_URI} !/maintenance.php$ [NC]
RewriteRule ^(.*)$ /maintenance.php [R=302,L]
I've search and search here and tried pretty much everything for the last 3 days with no success. I cant find what it's wrong with this very simple piece of code? Anyone has any ideas?
If it helps, my access_log says something like:
::1 - - [07/Mar/2021:13:47:36 -0500] "GET /maintenance.php HTTP/1.1" 200 2956
You are redirected your self on /meintanence.php as logfile indicates. Please test this to avoid local request redirecting:
# redirect to maintenance page
RewriteCond %{REMOTE_ADDR} !^::1
RewriteCond %{REQUEST_URI} !/maintenance.php$ [NC]
RewriteRule ^(.*)$ /maintenance.php [R=302,L]
::1 is the loopback address in IPv6, as the IPv4 version is 127.0.0.1.
I have a problem with my website opening pages that should show 404 error. It seems that only the first part of the URL is taken into account.
Here's an example
example.com/page.php - (opens - correct)
example.com/asdasd.php - (error 404 - correct)
example.com/asdasd.php/asdasd - (error 404 - correct)
example.com/page.php/asdasd - (opens - incorrect)
example.com/page/ - (opens - incorrect)
example.com/page/asdasd (opens - incorrect)
I would like to get an Error 404 and if I feel that the URL is important also be able to Redirect it. Now I can't even Redirect URLs that for example look like this "example.com/page.php/asdasd"
The closest answer that I found was this:
Options +FollowSymLinks -MultiViews
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=302,L]
RewriteCond %{THE_REQUEST} ^(?:GET|HEAD)\s([^.]+)\.php [NC]
RewriteRule ^ %1 [R,L,NC]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^([^.]+?)/?$ $1.php [L]
But the problem was that I got something like this:
Input URL
example.com/page.php/asdasd
Output URL
example.com/page
I need it for the .php extention to stay. And if possible instead of "Redirecting" to the example.com/page.php, I'd like to get an Error 404
This is what I currently have in the htaccess file:
ErrorDocument 404 /404.php
ErrorDocument 401 /404.php
ErrorDocument 403 /404.php
Options +FollowSymLinks -MultiViews
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)/$ /$1 [L,R=301]
RewriteCond %{HTTP_HOST} ^example\.com [NC]
RewriteRule ^(.*) http://www.example.com/$1 [L,R=301]
RewriteRule ^index\.php$ / [R=301,L]
Sorry if this is a duplicate but I don't even know what I should look for.
example.com/page.php/asdasd - (opens - incorrect)
This is due to a feature called Path Info - if Apache manages to match the first part of the requested URL to an existing file, then it will serve that file, and pass the rest of the path along via environment variables (so that a script could in theory make use of this information.)
The AcceptPathInfo directive makes it possible to disable this behavior.
https://httpd.apache.org/docs/2.4/mod/core.html#acceptpathinfo
I need to block all requests to my site if the referrer is equal to, for example, siteb.com UNLESS the URL being requested is equal to mysite.com/pagex/?random-variable
So if the referrer is siteb.com and the URL being requested is anything other than mysite.com/pagex/?random-variable, my .htaccess file should block the request, otherwise it should be allowed through. If the referrer is not siteb.com, nothing should be done.
I've gotten as far as this:
RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} siteb\.com [NC]
RewriteRule .* - [F]
However that just blocks ALL requests when the referrer is siteb.com. I need to allow requests to mysite.com/pagex/?random-variable when the referrer is siteb.com
You can add one more condition to your rule to allow /pagex/?random-variable:
RewriteEngine on
# Options +FollowSymlinks
RewriteCond %{HTTP_REFERER} siteb\.com [NC]
RewriteCond %{THE_REQUEST} !/pagex?random-variable [NC]
RewriteRule ^ - [F]
A stranger bot (GbPlugin) is codifying the urls of the images and causing error 404.
I tried to block the bot without success with this in the bottom of my .htaccess, but it didn't work.
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_USER_AGENT} ^$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^GbPlugin [NC]
RewriteRule .* - [F,L]
The log this below.
201.26.16.9 - - [10/Sep/2011:00:06:05 -0300] "GET /wp%2Dcontent/themes/my_theme%2Dpremium/scripts/timthumb.php%3Fsrc%3Dhttp%3A%2F%2Fwww.example.com%2Fwp%2Dcontent%2Fuploads%2F2011%2F08%2Fmy_image_name.jpg%26w%3D100%26h%3D65%26zc%3D1%26q%3D100 HTTP/1.1" 404 1047 "-" "GbPlugin"
Sorry for my language mistakes
Here's what you can put in your .htacces file
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
SetEnvIfNoCase Referer "^$" bad_user
SetEnvIfNoCase User-Agent "^GbPlugin" bad_user
SetEnvIfNoCase User-Agent "^Wget" bad_user
SetEnvIfNoCase User-Agent "^EmailSiphon" bad_user
SetEnvIfNoCase User-Agent "^EmailWolf" bad_user
SetEnvIfNoCase User-Agent "^libwww-perl" bad_user
Deny from env=bad_user
This will return:
HTTP request sent, awaiting response... 403 Forbidden
2011-09-10 11:15:48 ERROR 403: Forbidden.
May I recommend this method:
Put this is .htaccess in root of your site.
ErrorDocument 503 "Your connection was refused"
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^(Mozilla.*537.36|Mozilla.*UCBrowser\/9.3.1.344)$ [NC]
RewriteRule .* - [R=503,L]
Where
^(Mozilla.*537.36|Mozilla.*UCBrowser\/9.3.1.344)$
are the two useragents I wanted to block in this example case.
You can use regex so a useragent like
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0
could be
Mozilla.*Firefox\/40.0
^means match from beginning and $ to the end so you could block just one useragent with:
ErrorDocument 503 "Your connection was refused"
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*Firefox\/40.0$ [NC]
RewriteRule .* - [R=503,L]
Or add several using the | character to separate them inside ( and ) like in the first example.
RewriteCond %{HTTP_USER_AGENT} ^(Mozilla.*537.36|Mozilla.*UCBrowser\/9.3.1.344)$ [NC]
You can test it by putting your useragent in the code and then try to access the site. http://whatsmyuseragent.com/
To block empty referers, you can use the following Rule :
RewriteEngine on
RewriteCond %{HTTP_REFERER} ^$
RewriteRule ^ - [F,L]
This will forbid all requests to your site if HTTP_REFERER value is empty ^$ .
To block user agents, you can use
RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} opera|firebox|foo|bar [NC]
RewriteRule ^ - [F,L]
This will forbid all requests to your site if HTTP_USER_AGENT matches the Condition pattern.