I'm trying to create an automated bulk domain name availability lookup service. I'm wondering if there are guidelines available as to how many lookups are allowed from a single ip address in a set period of time?
Edit
To add further clarification I am automating using multiple calls to the command terminal whois command, not through any particular website.
Whois bulk lookup limitations are usually enforced on the source IP seen by the whois server in the query. The solution is to bounce around as many IPs as possible by using proxies, server farms, web based whois forms, etc.
Each registry/Registrar has its own throttle rule, as an example: godaddy only allows 100 or less queries from any IP.
Related
I am curious on how to setup multiple load-balancers (with different IP addresses) with a specific domain.
I understand that it is possible to setup multiple A-records in a DNS to all of my load-balancers, but I can understand that this is not ideal.
DNS' doesn't do any kind of is-alive checks, so if a load-balancer dies, the DNS will still send users to this address, right?
So how do you connect a domain/DNS with multiple load-balancers, while preventing a dead load-balancer from getting requests...
I read something about anycast, but is this the only solution?
I am just curious about how this issue is normally handled.
Thanks.
You have multiple solutions.
On a pure DNS level you can publish your records with a low TTL (say 5 minutes), and have your monitoring systems change the content of the zone by removing the dead record when detected. This does not provide immediate fail-over but can be often good enough.
It does not involve too complicated systems.
Also, some DNS servers allow some "programmed part", with a dynamic backend that can compute records based on some external parameters, like doing live checks and replying only with the live records.
Anycast is another solution indeed, and has then no relationship with the DNS anymore (although the DNS itself can be "anycasted" but then it is to resolve its possible failover needs, not the ones of your application).
Basically your multiple systems, on various places in the world, are advertised with the same IP address. So the DNS has only one record.
With the "magic" of BGP, each instance announcing a given IP address will collect all the nearby traffic, so you get load-balancing for free in fact. And you need some specific tooling so that, as soon as some local instance is dead (or in maintenance mode for example), you stop announcing its IP address there, so that all other networks in the world, again because of BGP, learn that to reach "something" behing that IP they need to go somewhere else, to another instance of yours announcing this IP.
This is far more complicated to setup as you need a proven BGP setup (and making errors in BGP can have even greater consequences than in DNS), and multiple instances located in different datacentres, and possibly multiple AS numbers, depending on how you want to do your anycast done. This clearly needs skilled professional in BGP routing where the first solution with only DNS (in the first case of just changing a static zonefile) is reachable by any enthousiastic amateur.
So the answer also slightly depend on the network locations of your load-balancers.
I've discovered a couple Node.js modules that allow for checking DNS records for domains, but none that allow for changing and setting DNS records to update domains.
Is what I'm asking even possible? I'm just looking for a step in the right direction.
I am using DNS Made Easy for my DNS servers, if that helps.
I think it really depends on the DNS server you're using. There are some standardized ways to update DNS servers - see https://en.wikipedia.org/wiki/Dynamic_DNS, or if you're using a service like afraid.org, then they have a REST-ish/HTTP endpoint you can call to update DNS records. There's no reason you couldn't initiate that call from node.js. That said - if you're trying to do real DDNS, you might have to implement a more tricky protocol to get it to happen. Some DNS servers just read text files from the disk, so you could also build something that updates those files and the domain serial number and reloads the server when an update occurs. TL;DR: it's possible, but you might have to implement some tricky stuff to get it to work.
We have a website which provides services for people based in particular city.
We want to scale and provide it for more cities but we want to remain separated IT within the city realm: one webhost, cloud service , database etc for one location. It does not only enables us to scale individually (some cities are bigger than other several times) but most significantly it improves our code-base and db queries to not use city's predicates - despite the fact it is more expensive in general.
At the same time we do not want to use subdomain. User can switch city through dropdown and request should go to appropriate VM without url being changed so the routing should work seamlessly.
Based on Azure documentation we are still not sure what solution would meet our needs, Traffic Manager, Load balancer or custom redirects.
How you accomplish this is ultimately up to you, but from an Azure-specific perspective, the only multi-region built-in load-balancing service is Traffic Manager. This operates in one of three routing modes:
Primary/failover
Round-robin
Closest (based on latency, not physical distance)
For any other type of routing (such as letting the user choose location, per your question), you'd need to implement this on your own or via 3rd-party service (and how to accomplish that would be a matter of opinion/debate/discussion, which is off-topic for StackOverflow).
Since you're looking to have a separate DB, cloud-role and webhost per city, I do not see how you can get away from doing subdomains.
Do you not want subdomains because of SEO? If so, it'd be easier to find another way to solve SEO problem.
But whatever Traffic Manager or other DNS based routing solution you use, it'll be splitting users by where they come FROM and not where they're going TO.
The destination problem is solved thru separate sub-domains
Could you please cite references/books for more details about Dynamic DNS? I've already tried Wikipedia, IEEE papers and RFCs for all those people rolling their eyes reading this. So please, any inputs are welcome. I need help implementing it in a project and would love to know more about it. Thanks.
Dynamic DNS is a concept of updating DNS records on-the-fly, as opposed to normal (static) DNS where change in a DNS records required manual intervention.
Dynamic DNS means that you have some DNS server, and you may programmatically update records on it. This can be achieved in different ways:
RFC 2136 dynamic DNS. It's an extension to good ol' DNS protocol which allows not obtaining DNS records, but updating them. Most DNS servers today (for example BIND9 and PowerDNS) support this protocol. Documentation sources: RFC 2136 defines the protocol. nsupdate is the command line tool which supports this protocol, read man nsupdate. For details on how to configure BIND9 for dynamic updates, refer to the BIND9 ARM. Libraries exist for most languages which allow Dynamic DNS updates using this protocol. For PHP for example, it's Net_DNS2. It's not well documented, but sites have nice examples which easily allowed me to use it.
Some DNS servers (especially PowerDNS) can read their DNS records from database back-end. Thus it makes possible to write new DNS records into normal SQL database, and server just takes them from there. Documentation sources: If you choose this way I very suggest to use PowerDNS, look for documentation on PowerDNS site.
If updates are not frequent, it's also possible to update text zone files on the DNS server and then request server to re-read the updated zone files. Though this is probably not a convenient way. All major DNS servers support same zone file format, I find DNS for Rocket Scientists excellent.
Now, there's a completely different side of dynamic DNS is a dynamic DNS services like to no-ip.com, my own net-me.net and many others. They all expose some HTTP-based API (usually very simple) to update DNS records, and often provide a GUI client software which actually updates them. Quick overview on the update protocol, the client and the whole process you can get here. As no standard exists, every provider uses his own variation of the protocol, usually they all look quite similar. (All these Dynamic DNS providers internally use some sort of 1.2.3. described above.)
The last but not least - there's a great Oreilly book - DNS and BIND 5th edition which covers all possible aspects of DNS.
I need to come up with a way to limit users' access to a page to IP addresses within North Carolina. This has to do with usage rights for some MP3 audio. Are there companies that provide this service or is there a list of IP address ranges for a geographical location? This is on a Unix, JBoss/Seam environment. Thanks.
Despite the fact that is not a good idea (users can use a proxy and bypass this restriction), you can find many services googling for it: http://www.google.com/search?client=ubuntu&channel=fs&q=ip+geolocation&ie=utf-8&oe=utf-8
As far as I know, http://www.maxmind.com is one of the best availableand is used in sites like Sourceforge.
You should implement this restrictions on your application not your server via .htaccess files.
I doubt such a list is available. You can check the register information from the IP address you receive which resolves to the ISP of the user (not very accurate).
But there is always a way to bypass this using proxy services like proxy.org (or google proxy)
use geoip by http://www.maxmind.com/
they have both free and subscription-based services (more precise)
You can try ip2location, and from that list try extracting specific providers in North Carolina. When you check the IP, you can query that list to get the provider and check if that provider in that area or not.
Hope this helps.