To PROGRAMMATICALLY verify if a domain exists I do the following:
DNS Query it and see if it resolves. If it does, it's obviously registered. So no need for step 2. If it doesn't, it might STILL be registered. So a whois check is required.
Backtrack from whois.iana.org and see if the designated whois server knows the domain or not.
Well, whois is not really meant for bulk checking. Not to mention that the RFC has only 4 pages and there's no clear specifications as to the format or even the encoding of the data. So you pretty much have to train the parser for each specific answer format (server).
Is there a way to circumvent the whois query and check (as close to the metal as possible) if the domain is registered in another (publicly available) standardized (preferably free or affordable) way? And not by downloading the TLD zone file or using third-party APIs (as they have a bad habit of snatching domains that you check before you get to register them). :)
I know registrars have their own protocol but I'm not sure if it's open to public use.
There isn't really any good way to do this accurately without looking at zone files or checking directly with the registry, unfortunately.
Registrars typically use a protocol like EPP to talk to a registry, check name availability and place orders. It's unlikely that anyone other than an accredited registrar would be permitted to use this protocol, but it may be worth checking with the registry that manages the TLDs you are interested in, e.g. Verisign.
I'd (personally) be wary of relying too much on DNS queries or WHOIS lookups to ascertain whether a particular domain exists, as both can produce inaccurate results from time to time. For example, certain TLDs have name servers configured for any unregistered domain name (they often direct you to the registry's website). The Vietnamese registry is one example of this. WHOIS lookups can fail for any number of reasons, so lack of a record is not concrete evidence of the domain's availability.
Related
I have been searching this one for weeks, so far I found that You can use the whois queries to check if a domain is registered and/or do a DNS record check. But while playing with the whois cli tool today I found this
NOTE: FAILURE TO LOCATE A RECORD IN THE WHOIS DATABASE IS NOT
INDICATIVE OF THE AVAILABILITY OF A DOMAIN NAME.
from whois example.com. Now I want to know if there's a reliable way of knowing if a domain is registered or not, and is it possible for someone who is not an accredited registrar to do so.
Theoretically, whois gives you complete and correct information about the status of a domain.
Practically, it's up to the person registering the domain, the registrar they dealt with and the registry for the TLD in question to make sure that whois actually contains complete and correct information. All these parties have to be willing and able to provide it, which in the real world is not always the case. Note that this is not a technical problem, it is a human (and sometimes legal) problem. There is no 100% reliable way, for the simple reason that people are never 100% reliable.
If you're interested in names under a specific TLD, you may get better answers through that TLD's registry's website than through whois. It shouldn't be that way, but it's not rare that it is.
Not sure if I want to just do a kind of brute-force thing, or a dictionary attack (though this would require a Romanised dictionary of every word in the target language), but I want to scan area websites (area meaning in my country) and I don't want to just whitelist everything. Is there a better way to do this?
If you have a legitimate need for a list of all zones in a TLD, contact the registry for that TLD and ask. How willing they will be to help you varies enormously, so without knowing which TLD you're thinking of it's impossible to guess how viable this way is.
If the TLD uses DNSSEC with NSEC, you can walk the zone by following the NSEC chain. This is the best way if you can't get a file to download, and the fact that a zone is using NSEC is implicit permission to do so.
If you can't do either of the above, you're down to guessing. You're also working against the wishes of the registry, so be prepared to have your server get blocked from even talking to the TLD's name servers. There are about 6.26*10^98 possible names directly under each TLD, so you'll need to send quite a few queries.
Also note that "web servers in my country" is not a very well defined concept. Does that mean all sites with domain names in your country's ccTLD? All sites hosted on servers that are physically in your country? Sites intended for people in your country, no matter their domain name or hosting location?
This is something where I get confused..
Say I acquired a domain name blabla.ge (ge is for Georgia) and hosting my files with US based hosting company. What are the downsides if any and is there an option to change the DNS server?
Cheers!
Agreed, there is no real downside. The tld is really not that important to basic usage. Yes root servers factor in here but really nothing that will impact your daily activities and you don't really need to worry.
For the nameservers, you can change these to any servers you wish and have access to manage the records. Location isn't important other than basic routing and response time. Nameservers generally should be on diverse networks and diverse locations per Best Practices. I have nameservers available in multiple countries and there's nothing wrong with that. If you are using the nameservers provided by your registrar, you likely have the diversity I mentioned, although they may be located in a single country (which is fine).
I have multiple domains registered with tlds such as .nl, .im, .com.de, etc. Some of these point to US-only nameservers, some use nameservers in multiple countries and a couple use the nameservers provided by my registrar (who I purchased the domain from).
From there, my A records point to servers in diverse locations.. Primarily the US and Netherlands. This set up works great, performance is adequate and there are no major downsides to doing it this way. You can change your nameservers for the .ge domain to use US servers or you can leave them overseas and use A records to point to your server(s) in the US. You can debate which method would be "best" given a situation but neither method is "wrong."
So in short, no major downside to doing this at all. And yes, changing your DNS server (nameserver) is always an option. Hope this helps.
Could you please cite references/books for more details about Dynamic DNS? I've already tried Wikipedia, IEEE papers and RFCs for all those people rolling their eyes reading this. So please, any inputs are welcome. I need help implementing it in a project and would love to know more about it. Thanks.
Dynamic DNS is a concept of updating DNS records on-the-fly, as opposed to normal (static) DNS where change in a DNS records required manual intervention.
Dynamic DNS means that you have some DNS server, and you may programmatically update records on it. This can be achieved in different ways:
RFC 2136 dynamic DNS. It's an extension to good ol' DNS protocol which allows not obtaining DNS records, but updating them. Most DNS servers today (for example BIND9 and PowerDNS) support this protocol. Documentation sources: RFC 2136 defines the protocol. nsupdate is the command line tool which supports this protocol, read man nsupdate. For details on how to configure BIND9 for dynamic updates, refer to the BIND9 ARM. Libraries exist for most languages which allow Dynamic DNS updates using this protocol. For PHP for example, it's Net_DNS2. It's not well documented, but sites have nice examples which easily allowed me to use it.
Some DNS servers (especially PowerDNS) can read their DNS records from database back-end. Thus it makes possible to write new DNS records into normal SQL database, and server just takes them from there. Documentation sources: If you choose this way I very suggest to use PowerDNS, look for documentation on PowerDNS site.
If updates are not frequent, it's also possible to update text zone files on the DNS server and then request server to re-read the updated zone files. Though this is probably not a convenient way. All major DNS servers support same zone file format, I find DNS for Rocket Scientists excellent.
Now, there's a completely different side of dynamic DNS is a dynamic DNS services like to no-ip.com, my own net-me.net and many others. They all expose some HTTP-based API (usually very simple) to update DNS records, and often provide a GUI client software which actually updates them. Quick overview on the update protocol, the client and the whole process you can get here. As no standard exists, every provider uses his own variation of the protocol, usually they all look quite similar. (All these Dynamic DNS providers internally use some sort of 1.2.3. described above.)
The last but not least - there's a great Oreilly book - DNS and BIND 5th edition which covers all possible aspects of DNS.
I need to determine whether a domain entered by a user is a standard domain e.g example.co.uk or just the TLD e.g co.uk.
Is there a way I can do this e.g. by querying nameservers using nslookup or dig commands?
Just for background, I'm building a tool which works with subdomains e.g. sub.domain.example.co.uk and need to be able to spilt each part of the subdomain into subdomain, domain and TLD parts.
Thanks,
Tom
The "problem" you are trying to solve isn't even well-defined.
There is no way to get this information from DNS itself. DNS makes no distinction between what you call a subdomain, a domain, and a TLD.
Browser makers and other interested parties have apparently not found any solution to this short of building and maintaining a list manually. And even that's an incomplete solution. For example, I know that in Canada you can register a domain as <your-label>.ca, <your-label>.<province>.ca, or <your-label>.<municipality>.<province>.ca, but the current version of Mozilla's list only accounts for the first two possibilities. (Listing all municipalities would be too burdensome anyway).
More importantly, the boundary between what is a "public" domain and a "private" domain isn't a technical one. You're not supposed to be able to register a domain under ac.jp unless you are a university (or similar) in Japan. You're not supposed to be able to register a domain under u-tokyo.ac.jp unless you are a department inside 東大 (or similar). Those two restrictions aren't fundamentally different on a technical level, yet one of those domains is considered "public" and the other one is not. It's a difference of politics/law.
Furthermore, if the public/private domain distinction is being used for security purposes (as it is used for example in web browsers to disallow supercookies), who says that different departments inside one university don't distrust each other just as much as different universities do? There's a well known piece of advice that applies: you shouldn't attempt to solve a political problem with a technical solution.