I need to use fail2ban due to many attack attempts on my server, I also have filters that I had to activate/create to block attack attempts.
But now I'm pretty sure that some google ip ends up in the jail of my fail2ban...
I added some ip in the ignoreip directive in the jail.local file, but they are only the ones that I managed to identify as real google ip in my access.log (I also have many fake google)
It would be nice to be able to give a list of ip to ignore to fail2ban, but google does not release its ip list, google says: https://support.google.com/webmasters/answer/80553?hl=en
So the question is: is it possible to do a reverse dns to understand if an ip belongs to google and tell fail2ban to ignore it?
Can it be done via fail2ban? Do you need any external script? Could it be too heavy, long and tiring for the server?
yes, you can identify google bots using reverse IP lookup.
all crawler bots will end with xxxxxx.google.com or xxxxxxx.googlebot.com
for e.g. crawl-203-208-60-1.googlebot.com
but it is not possible to identify in fail2ban, but you can whitelist the IP address once you know if its a Googlebot.
there are many ways to perform for reverse IP look.
you can use Python, Ruby or bash to find out. check the following article.
http://searchsignals.com/tutorials/reverse-dns-lookup/
there are websites that can find you reverse IP lookup.
https://dnschecker.org/reverse-dns.php
http://reverseip.domaintools.com/
if you can code in python, you easily dump reverse IP data in a file from a list of IP addresses.
Google does have a page about verifying GoogleBot addresses by doing a reverse-lookup on the IP address and verifying that it comes from a specific hostname (you'd then get the IP of that host, to double-check it comes back to the appropriate source IP).
There are also DNS TXT records that specify IP ranges for SPF (emails), Google Compute Cloud, and the wider Google IP addresses that can be used (many of which would be in use by GCP user's VMs and other services).
dig #8.8.8.8 +short TXT _spf.google.com
dig #8.8.8.8 +short TXT _cloud-netblocks.google.com
dig #8.8.8.8 +short TXT _cloud-netblocks.googleusercontent.com
The first query will return something like this:
"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"
And you would then parse it to get the IP address ranges, or do a sub-query on the include:_netblocks.google.com etc to get other sets.
The information these records are not fixed, and can regularly change. (AWS publishes a .JSON file with several updates per week, for example).
I'm working on a system to automatically detect 'lying user-agents', with these, and some other techniques.
Related
For eg. we wanted to search google.com (let us think that its an alias name), then we will lookup in DNS and get its canonical name which further helps to get the IP address. Why cant we just get IP address from alias name as it would also be unique.
It is not always guaranteed that alias name resolves to same IP address. And, there is a very good reason for it. Lets say person A is browsing google.com from country A. Google has it servers all over the world (for efficiency purposes). It is beneficial if person A requests are directed towards google servers in country A than towards some other distant location. Here where CNAME records comes into the picture. CNAME records are configured in such a way that google.com resolves to servers which are specific to country A. And another case where you get different IP for same alias name is when you fetch MX records (mail server records), for the same domain you can have different servers managing mails and web traffic.
The design of URL is for convenience. The convenience is that when we want to change the server IP, we don't need to tell all the users the new ip of the website. In other words, what we have done in server will make no change to users. That is the core thought in server design.
I am not able to access one of my website from office which is hosted in Microsoft azure. When i have tried to lookup for dns lookup.
I is not showing me the correct ip address or showing some thing like mydomainname.kkph.com.
While i can access the same website from other devices around the world.
I have also checked with Godaddy support team but didn't find any issue at their end.
First, you could ensure that you have a correct IP address binds with your website and have a mapping A or CNAME record in your DNS domain provider.
For this issue, you could check the followings:
DNS cache on the local office machine. Use ipconfig/flushdns to clear it.
DNS Suffix Search List. You could check if you have a DNS suffix automatically appended to your hostname when you query. You could check it via 'ipconfig/all' or find the box Append parent suffixes of the primary DNS suffix and DNS Suffix Search List under the Advanced TCP/IP Settings---DNS. Also, to avoid using the search list, always use a Fully Qualified Domain Name (that is, add the trailing dot to the name) when you run nslookup fqdn. Read here.
Try to nslookup on another machine from the office to find if there is something wrong with a specific machine.
If no luck, please show the result of nslookup for further help. I would like to suggest to capture the netmon traces on your local environment.
My website suddenly stopped working.
When I search for the domain name in WHOIS websites it is showing the correct server ip address and correct DNS IP address.
I can reach the website by its IP address but somehow when I am trying the domain name in browser its not working and its showing "This site can’t be reached"!
There is no error in my server log.
I tried different browsers and different systems and it is same issue.
I am really confused. Even when I am sending GET requests with Postman to my domain, it not reachable but sending request to IP is working!
whois and DNS resolution are two separate things and one does not imply anything for the other, so in short, except in very specific cases, if you have a DNS resolution problem you should use DNS troubleshooting tools, not the whois and especially not web-based whois (the only relevant whois is the registry one).
Now you are giving so few details that noone can really help.
Among the possible ideas to check and probable problems:
you forgot to renew the domain, your registrar put it on hold or worse deleted it (that you can see in whois)
you did a change in the DNS resolution and now it does not work anymore, use online troubleshooting tools like Zonemaster or DNSViz; alternatively your registrar and/or webhosting company should be able to help (since you are neither giving here the domain name nor details about the troubleshooting you do: for DNS problems, the browser is not the first tool to use, look instead at dig).
in appear that the problem was DNS on our local system. we changed it to 8.8.8.8 and then we could access to our domain!
it's usually because you use an addon domain, not the main domain for hosting orders that are set up on cpanel whm
I want to query Spamhaus's SBL using a domain name. I know this is possible to do because this form (Find SBL Listings by ISP Domain Name) does it and SpamAssassin does it, but I can only seem to get it to work with IP addresses. I took a quick look at the SpamAssassin code, but it has been so generalized that I could probably spend a couple hours tracking down the code that actually does something. Right now I can successfully query SBL for IP addresses like this:
#returns 127.0.0.2, so 208.73.210.0 is on the blacklist
dig +short 0.210.73.208.sbl.spamhaus.org
#returns nothing, so 72.14.225.72 isn't on the blacklist
dig +short 72.225.14.72.sbl.spamhaus.org
Querying with domain names seems to have something to do with DNS TXT records, but I don't know the right hostname to lookup. When I try something like
dig oversee.net.sbl.spamhaus.org TXT
I don't get any useful information back, but if you search with the form you find that oversee.net is associated with 208.73.210.0 which was reported as spamming on 30-Jul-2009 21:17 GMT.
Domains are in the "Domain Block List", not the SBL. Use dbl.spamhaus.org as the domain suffix.
The particular search you linked to is based on the ISP's domain name, and I don't believe it uses the same DNSBL interface.
I was wondering if it's possible to dynamically add subdomains that point to dynamic IP addresses, and how I would go about doing that? In other words, "how is dyndns/no-ip implemented" :-)? (The part I don't get is adding/changing the DNS entries... I understand how the client sends a packet every few minutes -___-). I can tell all my users to just use DynDNS/No-IP, of course, but having it integrated with the application would be much cooler.
Thanks,
Robert
To be able to directly update/control where a domain/subdomain resolves to, you must have your own name server. When you register a domain under a TLD (for example, .com), that TLD has a nameserver. Anytime a client needs to look up the IP to something.com, they ask the .com nameserver where to find the nameserver for something. That nameserver in turn returns data about the domain or subdomain.
When you register a domain at a place like GoDaddy or Network Solutions, and you use their online tools to point your various subdomains to IP addresses, you are creating entries on their nameserver. When a client requests your domain, the root nameserver tells them to check with GoDaddy's nameserver. If you look through the configuration options of your registrar, you'll generally find a place to specify your own nameserver instead of entering domain IPs. Setting that will tell the chain of nameservers to defer resolution of your subdomains to that nameserver. Obviously at that point, having direct control over the mechanism of name-address resolution, you can do whatever you like.
Here's one list of open-source name servers. There are many others, ranging from free OSS to custom, proprietary and very expensive. Technically you could also write your own, as BIND is a public, standard format.
As you've partially said, the way DynDNS and other dynamic IP services work is that they update their server's DNS records based on a heartbeat from a client every few minutes.
The trick is that they use extremely short TTL times so that caches for the record expire very quickly and need to re-query the DynDNS server (which makes dynamic IP changes propagate quickly).
If you wanted to implement this, either find a DNS host that offers an API, or programatically update the DNS on your own server with a short TTL.