This morning, a lot of my website where tagged "this site may be compromised" by Google in it's result. Sites that are under my supervision on my own VPS server. I'ved run a deep scan on it and nothing unsual. I'ved look for suspicious htaccess and for javascript injection and nothing wrong so far.
Yesterday, I put an htaccess file to my web root to insure no sql, javascript, base64 and any other suspicious hacking solution might attack my server.
So I do suspect that Google add "this site may be compromised" since I add this protection to all my web sites.
there is the content of this htaccess :
RewriteEngine On
RewriteCond %{REQUEST_URI} !^/robots.txt
RewriteCond %{REQUEST_URI} !^/sitemap.xml
RewriteCond %{HTTP_USER_AGENT} ^-?$ [OR]
RewriteCond %{HTTP_USER_AGENT} ^[bcdfghjklmnpqrstvwxz\ ]{8,}|^[0-9a-z]{15,}|^[0-9A-Za-z]{19,}|^[A-Za-z]{3,}\ [a-z]{4,}\ [a-z]{4,} [OR]
RewriteCond %{HTTP_USER_AGENT} ^<sc|<\?|^adwords|#nonymouse|Advanced\ Email\ Extractor|almaden|anonymous|Art-Online|autoemailspider|blogsearchbot-martin|CherryPicker|compatible\ \;|Crescent\ Internet\ ToolPack|Digger|DirectUpdate|Download\ Accelerator|^eCatch|echo\ extense|EmailCollector|EmailWolf|Extractor|flashget|frontpage|Go!Zilla|grub\ crawler|HTTPConnect|httplib|HttpProxy|HTTP\ agent|HTTrack|^ia_archive|IDBot|id-search|Indy\ Library|^Internet\ Explorer|^IPiumBot|Jakarta\ Commons|^Kapere|Microsoft\ Data|Microsoft\ URL|^minibot\(NaverRobot\)|^Moozilla|^Mozilla$|^MSIE|MJ12bot|Movable\ Type|NICErsPRO|^NPBot|Nutch|Nutscrape/|^Offline\ Explorer|^Offline\ Navigator|OmniExplorer|^Program\ Shareware|psycheclone|PussyCat|PycURL|python|QuepasaCreep|SiteMapper|Star\ Downloader|sucker|SurveyBot|Teleport\ Pro|Telesoft|TrackBack|Turing|TurnitinBot|^user|^User-Agent:\ |^User\ Agent:\ |vobsub|webbandit|WebCapture|webcollage|WebCopier|WebDAV|WebEmailExtractor|WebReaper|WEBsaver|WebStripper|WebZIP|widows|Wysigot|Zeus|Zeus.*Webster [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^curl|^Fetch\ API\ Request|GT\:\:WWW|^HTTP\:\:Lite|httplib|^Java/1.|^Java\ 1.|^LWP|libWeb|libwww|^PEAR|PECL\:\:HTTP|PHPCrawl|python|Rsync|Snoopy|^URI\:\:Fetch|WebDAV|^Wget [NC]
RewriteRule (.*) - [F]
RewriteCond %{REQUEST_METHOD} (GET|POST) [NC]
RewriteCond %{QUERY_STRING} ^(.*)(%3C|<)/?script(.*)$ [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)(%3D|=)?javascript(%3A|:)(.*)$ [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)document\.location\.href(.*)$ [OR]
RewriteCond %{QUERY_STRING} ^(.*)(%3D|=)http(%3A|:)(/|%2F){2}(.*)$ [NC,OR]
RewriteCond %{QUERY_STRING} ^(.*)base64_encode(.*)$ [OR]
RewriteCond %{QUERY_STRING} ^(.*)GLOBALS(=|[|%[0-9A-Z]{0,2})(.*)$ [OR]
RewriteCond %{QUERY_STRING} ^(.*)_REQUEST(=|[|%[0-9A-Z]{0,2})(.*)$ [OR]
RewriteCond %{QUERY_STRING} ^(.*)(SELECT(%20|\+)|UNION(%20|\+)ALL|INSERT(%20|\+)|DELETE(%20|\+)|CHAR\(|UPDATE(%20|\+)|REPLACE(%20|\+)|LIMIT(%20|\+))(.*)$ [NC]
RewriteRule (.*) - [F]
There is a lot of keyword within this file regarding hacking terminology ... is there any way that Google might look into the htaccess file ?
Should I block google with a robots.txt for this htaccess only or could/should I add a line of code directly into the htaccess to block Google for scanning this file... ?
What do you think ?
If .htaccess is visible from outside, then you have a serious problem. That file should never be visible by anybody accessing the site through http. Blocking it in robots.txt would just prevent well-behaved bots from looking at it. But bots that ignore robots.txt would still have access.
If you suspect that your .htaccess is the cause of the problem, you need to make sure that it can't be served. That's the default on Apache, but if you were mucking around with permissions I suppose you could have exposed it. If you did, you need to fix that.
I think you need to look somewhere else for the cause of Google's "this site may be compromised" message. A Google (or Bing) search on [this site may be compromised] reveals lots of information about why that warning might appear.
Related
I am attempting to block hotlinked images from a specific site and re-route to an externally loaded/hosted image somewhere else.
I made some edits to my .htaccess file
Buy it also seems to block my OWN site/domain from my own images.. (even though I believe I am allowing it?)
I cant seem to get things to work with JUST blocking the external site...(without blocking my own site from my own images?)
I've tried so many lines... I cant make heads or tails on what is the issue that is blocking images from my own site.
Here is is my latest attempt
RewriteEngine On
#RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://myfakesite.net.*$ [NC]
RewriteCond %{HTTP_REFERER} !^http://www.myfakesite.net.*$ [NC]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?external-site\.com(/.*)*$ [NC,OR]
RewriteCond %{HTTP_REFERER} ^https://(www\.)?external-site\.com(/.*)*$ [NC,OR]
RewriteRule .*\.(gif|jpg|jpeg|bmp|png)$ https://path-to-externally-hosted-image.jpg [R,NC]
*I erroneously though this would be much easier.. LOL
How can I block external-site.com and allow everything from mysite.net?
I blocked some sites in the file but they keep comming on my server and asking for images that slow down my server how i can add a rule for them so they will see big red sign STOP HOTLINKING
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} ^http://(www\.)?somesite\.pl [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?somesite\.pl [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?somesite\.pl [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(www\.)?sklep.somesite\.eu [NC]
RewriteRule \.(gif|jpe?g|js|css)$ - [F,NC,L]
Mate try the below ,
Following code will only allow the mentioned domain "alloweddomain.com" and block others from hot linking
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?alloweddomain.com [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]
The above code will allow "Blank Referrers"
What is Blank Referrers ?
Some visitors uses a personal firewall or antivirus program, that deletes the page referrer information sent by the web browser. Hotlink protection is based on this information. So if you choose not to allow blank referrers, you will block these users. You will also prevent people from directly accessing an image by typing in the URL in their browser.
Suppose if you don't want to allow "Blank Referrers" then use the following code mate
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?alloweddomain.com [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ - [NC,F,L]
Also if suppose you want to display a image like "STOP HOTLINKING" then use the below method
RewriteEngine on
RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http(s)?://(www\.)?alloweddomain.com [NC]
RewriteRule \.(jpg|jpeg|png|gif)$ mydomain.com/img/stop_hotlink.jpg [NC,R,L]
Above code will allow "Blank referrers" . To not allow, follow as mentioned in previous step again.
Note that :
In case of displaying image for "STOP HOTLINK" make sure the image is not hotlink protected or your server can go into an endless loop.
alloweddomain.com - The domain that you want to allow for hotlink
mydomain.com/img/stop_hotlink.jpg - URL for the "STOP HOTLINK" image
Update : [ Block Specific Domains ]
To stop hotlinking from specific outside domains only, such as blockurl1.com, blockurl2.com and blockurl3.com, but allow any other web site to hotlink images:
RewriteEngine On
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?blockurl1\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?blockurl2\.com/ [NC,OR]
RewriteCond %{HTTP_REFERER} ^http://(.+\.)?blockurl3\.com/ [NC]
RewriteCond %{REQUEST_URI} !blocked\.gif$ [NC]
RewriteRule .*\.(jpe?g|gif|bmp|png)$ http://example.com/blocked.gif [L]
You can add as many different domains as needed. Each RewriteCond line should end with the [NC,OR] code. NC means to ignore upper and lower case. OR means "Or Next", as in, match this domain or the next line that follows. The last domain listed omits the OR code since you want to stop matching domains after the last RewriteCond line.
The last line contains the URL "http://example.com/blocked.gif" which contains the image that will be displayed when the condition occurs.
You can display a 403 Forbidden error code instead of an image. Replace the last line of the previous examples with this line:
RewriteRule .*\.(jpe?g|gif|bmp|png)$ - [F]
Hope this helped you mate!
I just put up a new website. New domain, new everything. I set up Google Analytics and my first ever visitor is this:
forum.topic60670121.darodar.com (i do not linking it, it is spam)
I know lot of people experience traffic from several Russian spam sites those redirect to some other site. I read a lot of about it. I can't ban them even from htaccess like this:
## SITE REFERRER BANNING
RewriteEngine on
RewriteCond %{HTTP_REFERER} semalt.com [NC,OR]
RewriteCond %{HTTP_REFERER} Darodar.com [NC,OR]
RewriteCond %{HTTP_REFERER} Priceg.com [NC,OR]
RewriteCond %{HTTP_REFERER} 7makemoneyonline.com [NC,OR]
RewriteCond %{HTTP_REFERER} Buttons-for-website.com [NC,OR]
RewriteCond %{HTTP_REFERER} Ilovevitaly.com [NC,OR]
RewriteCond %{HTTP_REFERER} Blackhatworth.com [NC,OR]
RewriteCond %{HTTP_REFERER} Econom.co [NC,OR]
RewriteCond %{HTTP_REFERER} Iskalko.ru [NC,OR]
RewriteCond %{HTTP_REFERER} Lomb.co [NC,OR]
RewriteCond %{HTTP_REFERER} Lombia.co [NC,OR]
RewriteCond %{HTTP_REFERER} hulfingtonpost.com [NC,OR]
RewriteCond %{HTTP_REFERER} cenoval.ru [NC,OR]
RewriteCond %{HTTP_REFERER} o-o-6-o-o.com [NC,OR]
RewriteCond %{HTTP_REFERER} humanorightswatch.org [NC,OR]
RewriteCond %{HTTP_REFERER} bestwebsitesawards.com [NC]
RewriteRule .* - [F]
deny from 78.110.60.230
deny from 217.23.11.15
deny from 217.23.7.180
deny from 217.23.8.124
It just doesn't work. And I given up on this at another website. But the question is:
How it is possible they visit a site that is so new even I barely know about it?
Why you get the traffic at all
This blog post might be an actual explanation for your question: The author of the blog post finds evidence that the spammers harvest Google Analytics UA codes to artificially send page hits. The link in your GA reports should then contain a referral identifier.
The author suspects that UA codes may also be auto-generated.
Other possible causes for it happening so quickly
Have you entered the domain name into a webservice tool to test something? E.g. DNS setup, SEO tools, or similar? They might have been breached and do not know about it. I experience such weird traffic on my domains, too and have begun to ignore them.
How to get rid of the weird traffic in your reports
You can also setup your reports in GA to exclude those from the numbers being reported (unless you are also actively targeting the area where the weird traffic originates). There is a great blog post explaining how to filter bad traffic from GA. The blog post also states darodar.com as their traffic source.
How it is possible they visit a site that is so new even I barely know about it?
They don't visit your site. They send data directly to Google Analytics and leverage the fact that by picking property IDs randomly, there is a significant probability to hit existing Web properties.
For more background information and a discussion about different approaches to eliminate referrer spam:
http://veithen.github.io/2015/01/21/referrer-spam.html
at the end of my .htaccess i redirect all url that i consider faulty.
this works perfect but i need an exception for my own pc
for my firefox browser i want the server to react as every other request
for my chrome browsers i want full access to all files on the server
i do it this way and it works:
RewriteCond %{REQUEST_URI} !=/index.php
RewriteCond %{REQUEST_URI} !=/retpic.php
RewriteCond %{REQUEST_URI} !^/pcs/.*$
RewriteCond %{REMOTE_HOST} ^11\.11\.1\.11$
RewriteCond %{HTTP_USER_AGENT} Firefox
RewriteRule . index.php [L]
RewriteCond %{REQUEST_URI} !=/index.php
RewriteCond %{REQUEST_URI} !=/retpic.php
RewriteCond %{REQUEST_URI} !^/pcs/.*$
RewriteCond %{REMOTE_HOST} !^11\.11\.11\.11$
RewriteRule . index.php [L]
but i end up doubling the whole code... is there a more elegant solution like combining the following with an AND statement? (i found something about an OR statement but not about AND)
RewriteCond %{REMOTE_HOST} ^11\.11\.1\.11$
RewriteCond %{HTTP_USER_AGENT} Firefox
edit: added more explanation on how the code works:
RewriteCond %{REMOTE_HOST} !^11\.11\.11\.11$
this is from the second part it excludes my ip from this rule so i can access all files on the server, this is important since i want to be able to access my cms
RewriteCond %{REMOTE_HOST} ^11\.11\.1\.11$
RewriteCond %{HTTP_USER_AGENT} Firefox
this parts includes my firefox browser when i am home so i can see if the website works with all restrictions in place. why do i have this rule: i was working on my site and restructuring some parts and it kept on working for me but when i was at a friends place i noticed it did not so i needed something to be able to check this at home.
You nee do apply little bit of boolean algebra to combine these rules into one.
Here is you can do it for you:
RewriteCond %{REMOTE_ADDR} !=11.11.2.11 [OR]
RewriteCond %{HTTP_USER_AGENT} Firefox
RewriteRule !(index\.php|retpic\.php|pcs/) index.php [L,NC]
How do I prevent spriders crawling pages that start with mydomain.com/abc...
For example mydomain.com/abcSGGSHS or mydomain.com/abc6bNNha
I think I need to add some sort of regular expression to the web root's .htaccess, right?
With mod_rewrite enabled, you can do the following
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^YourBadSpiderName [OR]
RewriteCond %{HTTP_USER_AGENT} ^AotherBadSpider
RewriteCond %{REQUEST_URI} ^abc
RewriteRule ^$ http://mydomain.com/404.html [NC,L]
You'll have to update the spider names accordingly. If a bot changes his user agent, let's say to 'Mozilla/Firefox', you're out of luck..