I am trying to extract all domain names out of COM and NAME dns zone file. Those zone files contain all dns entries and there seem to be lack of information about structure of zone files.
Do all domain registered has NS entries? Even those which are not actively used? Which record/records should I use to extract domain names.
Zone files are very large and sorting them would be stupid idea. So if I can use one DNS record type to extract domain name than it would be easier.
I found this python script(I dont know python) on GitHub which uses only NS entries. Is it correct logically?
Someone with experience please comment.
The format of the DNS zone file is defined in RFC 1035 (section 5) and RFC 1034 (section 3.6.1). You can find many details on Wikipedia: https://en.wikipedia.org/wiki/Zone_file
It contains only the published domain names that is those having at least one nameserver and not being under clientHold or serverHold statuses (see http://www.icann.org/epp#clientHold and http://www.icann.org/epp#serverHold), which means in short it is NOT all domain names registered.
.COM zone file is huge indeed. In any case, you need to match on NS records lines and deduplicate domain names. There are multiple strategies to do that, depending on your constraints.
Note that many providers on line already do this work for you and can provide directly the domain names if this is all you are interested in. Some may also provide differential content, one day from the previous.
Related
I am writing an application to query the DNS SRV record to find out an internal service for a domain obtained from the email address. Is it correct to do the following.
Lets say the email domain is test.example.com
Query SRV record _service._tcp.test.example.com
No SRV record is returned
Now query SRV record _service._tcp.example.com
A record is returned. Hence use this record to connect
Is the above approach right? Assuming its not, are there any RFCs or standards that prevents an application from doing it?
Is the above approach right?
No, it is not. You should not "climb" to the root.
There is nothing explicitly telling you not to do that in RFCs and you will even find some specifications telling you to climb to the root, see CAA specifications (but they had to be changed over the year because of some unclarity exactly around the part about climbing to the root).
Most of the time, such climbing creates more problems than solution, and it all come from "finding the administrative boundaries" which looks far more simple than what it is really.
If we go back to you example, you say, use _service._tcp.test.example.com and then _service._tcp.example.com and then I suppose you stay there, because you "obviously" know that you shouldn't go to _service._tcp.com as next step, because you "know" that example.com and com are not under the same administrative boundaries, so you shouldn't cross that limit.
Ok, yes, in that specific example (and TLD) things seem simple. But imagine an arbitrary name, let us say www.admin.santé.gouv.fr, how do you know where to stop climbing?
It is a difficult problem in all generality. Attempts were made to solve it (see IETF DBOUND working group) and failed. You have only basically two venues if you need to pursue: either find delegations (zone cuts) by DNS calls (not all delegations are new administrative boundaries, but a change of administration should mean a delegation; and obviously there is not necessarily a delegation at each dot, so you can not find all of this by just looking at the string, you need to do live DNS queries) OR using Mozilla Public Suffix List, which has a lot of drawbacks.
This is all basically a rehash of what you can read in "§4. Zone Boundaries are Invisible to Applications" of RFC5507, quoting the core part here:
The false assumption has lead to an approach called "tree climbing",
where a query that does not receive a positive response (either the
requested RRSet was missing or the name did not exist) is retried by
repeatedly stripping off the leftmost label (climbing towards the
root) until the root domain is reached. Sometimes these proposals
try to avoid the query for the root or the TLD level, but still this
approach has severe drawbacks:
[..]
o For reasons similar to those outlined in RFC 1535 [RFC1535],
querying for information in a domain outside the control of the
intended entity may lead to incorrect results and may also put
security at risk. Finding the exact policy boundary is impossible
without an explicit marker, which does not exist at present. At
best, software can detect zone boundaries (e.g., by looking for
SOA Resource Records), but some TLD registries register names
starting at the second level (e.g., CO.UK), and there are various
other "registry" types at second, third, or other level domains
that cannot be identified as such without policy knowledge
external to the DNS.
Note indeed also the example given for MX because a naive view you apply the same algorithm there, but as the RFC says:
To restate, the zone boundary is purely a boundary that exists in the
DNS for administrative purposes, and applications should be careful
not to draw unwarranted conclusions from zone boundaries. A
different way of stating this is that the DNS does not support
inheritance, e.g., an MX RRSet for a TLD will not be valid for any
subdomain of that particular TLD.
There are various examples of people having tried to climb to the root... and creating a lot of problems:
in the past, Microsoft and wpad.dat: https://news.softpedia.com/news/wpad-protocol-bug-puts-windows-users-at-risk-504443.shtml
more recently, Microsoft again about email autodiscover: https://www.zdnet.com/article/design-flaw-in-microsoft-autodiscover-abused-to-leak-windows-domain-credentials/
So, in short, without a solid understanding of DNS, please do not create anything "climbing" to the root. Do note that RFC2782 about SRV gives "Usage Rules" without a case of climbing to the root.
You are not explaining fully why you are thinking about this. I suggest you have a look at the newest HTTPS/SVCB DNS records (RFCs not published yet, but RR type codepoint assigned by IANA already, and in use by Apple, Cloudflare and Google already), as they may provide similar features set as SRV but may be more relevant for your use case.
I am planning to build a service based on subdomain like wordpress or tumblr.
I want to know what is the maximum number of subdomains a domain can have.
Well. The absolute theoretical maximum is 2^504, but that assumes no limitations on the octets making up the names. If you want the names limited to ASCII letters and digits, the answer is 111444219848545291112918149658401217019177846881717006276548100629318214534968256903948922840416256 (that is, 36^63).
In another sense the answer is "Far, far more than you will ever need".
This would be more dependant on the DNS server than on the standard. BIND allows for a maximum of 16'777'216 objects per zone file while Microsoft DNS is reported to be stable up to 20'000 objects per zone. This does not mean however that you will be able to max out the DNS object limits as your average website owner is going to want to have a meaningful object name for their sites sub domain, additionally the maximum character count for fully qualified domain names is 255 characters with no individual segment (between dots) being longer than 63 characters as per the DNS specifications.
Effectively what this means is that while there are restrictions and limitations the practical answer is that you are unlikely to encounter limitations due to DNS specifications in any reasonable timeframe.
I'm trying to write a script where part of its functionality depends on if a user-provided domain is a zone apex (example.com) or not (www.example.com). What is a robust way to determine if I'm dealing with a zone apex or a subdomain? I'm hoping for a purely pattern-based approach but that seems tricky (example: xx.yy.co is not a root domain but xx.co.uk is).
Are there any tried and true approaches to determine if a zone is a root domain or not?
The Public Suffix List indicates lists of top level and second level domains under which one can register a domain name. If a name has exactly one more level beyond its matching entry on this list, then it's what you are looking for.
(Note that "subdomains" as you call them can be DNS zones in their own right and have independent nameservers from the parent zone. These can generally be detected by the presence of an SOA record for that fully qualified name, and nameservers for that name in the parent zone.)
Yes, I had to write a script that performed this recently.
Run a non-recursive query (i.e. dig +norecurse) against the authoritative nameserver for the entity you're examining. Use a query type of SOA. Do not use a recursive server, the behavior becomes much less predictable. If it's a server that mixes auth and recursive answers, make sure you're checking for the AA (authoritative answer) flag on the reply.
If the response code is NOERROR, examine the leftmost component of the returned ANSWER section (if present). Otherwise, check the AUTHORITY section. One of the two will be present. The upshot of preferring the ANSWER is that this ensures your result is a SOA record instead of a NS record. It keeps the type of your result consistent, which can be useful if you're writing something against a resolver library.
If the response code is NXDOMAIN, examine the leftmost component of the returned AUTHORITY section. Obviously this won't be the apex, but this will tell you what the apex is.
Anything other than those response codes indicates that the server does not consider itself authoritative.
The result will be the apex. Your request is not the apex if your result is less specific, and it is the apex if they're identical.
www.example.com (assuming it isn't a subdomain, e.g., there are no foo.www.example.com entries) will not have a DNS SOA RR. However, example.com may have other subdomains e.g, xyz.example.com which contains foo.xyz.example.com and bar.xyz.example.com so I don't know if this helps you.
Walk the name backwards component by component checking for NS records.
Example: www.example.com
Does www.example.com. have a NS record? No.
Does example.com. have a NS record? Yes.
Does com. have a NS record? Yes.
Make your determination accordingly, based on whatever definition you use for "is a zone apex" (it's not 100% clear to me from your question.)
I'm writing a Regex to validate email. The only one thing confuse me is:
Is it possible to have single character for top level domain name? (e.g.: lockevn.c)
Background: I knew top level domain name can be from 2 characters to anything (.uk, .us to .canon, .museum). I read some documents but I can't figure out does it allow 1 character or not.
It is technically possible, however, there are no single character tlds that have been accepted into the root (as of the moment) so the answer is:
Yes, it is possible to have single character for top level domain name, however, there are currently no single character TLDs in the root.
You can see the list of TLDs that are currently in the root at this URL:
http://data.iana.org/TLD/tlds-alpha-by-domain.txt
RFC-952 shows what a "name" is, this includes what is valid as a top level domain:
A "name" (Net, Host, Gateway, or Domain name) is a text string up
to 24 characters drawn from the alphabet (A-Z), digits (0-9), minus
sign (-), and period (.).
Additionally, the grammar from RFC-952 shows:
<name> ::= <let>[*[<let-or-digit-or-hyphen>]<let-or-digit>]
RFC-1123 section 2.1 specifically allowed single letter domains & subdomains, changing the initial grammar of RFC-952 from starting with just a letter to being more relaxed, so now you are allowed to have single letter top level domains that are a number:
2.1 Host Names and Numbers
The syntax of a legal Internet host name was specified in RFC-952.
One aspect of host name syntax is hereby changed: the
restriction on the first character is relaxed to allow either a
letter or a digit. Host software MUST support this more liberal
syntax.
EDIT: As per #mr.spuratic's comment, RFC-3696 section 2 tightened the rules for top level domains, stating:
There is an additional rule that essentially requires
that top-level domain names not be all-numeric.
This means that:
a. is a valid top level domain
1. is not a valid top level domain
A very unscientific test of this shows that if I add "a" into my hosts file pointing to my local machine, going to http://a in my address bar does show my Apache welcome page.
I'm not sure about the internet standard, but in practice, no.
See,
http://www.norid.no/domenenavnbaser/domreg.html
and,
http://sqa.fyicenter.com/Online_Test_Tools/Domain_Name_Format_Validator.php
You should DEFINITELY allow 1-character domains since some registries allow them not by accident (and I speak of quite big registries like UK, Germany, Poland, Ireland too - so important contributors to the Internet community, not oney exotic small exceptions). Since I also plan using such domains, that definitely work also with all e-mail services I used, letters AND numbers, I really would give the hint to allow this, else your script might need later correction.
Also some of the biggest internet companies use such domains - one of the most famous examples is Twitters t.co for shortening. Other companies I know of who have such domains are Facebook, Google, PayPal, Deutsche Telekom. But the list is longer and also some bigger investors hold them as assets.
By the way as proof there is a website trading this kind of domains online if You search for "1 letter domain names" :)
I need a script to find out what is the lowest available domain name with a give TLD (say .com, .info, or .net).
For example, 1000423.com is free but 1000.com is taken.
Probably my spammiest question so far.
NOTE
I mean "lowest" domain name numerically (i.e. 1.com, 2.com, 3.com, ..., n.com, n+1.com, ...) and not shortest as in String.length.
In your web-capable language of choice:
Ask the user for a top-level domain name.
i <- 0.
Send out an HTTP GET to a registrar to see if "i.(tld)" is taken.
If it's not taken, notify the user and quit.
i <- i + 1.
Go to step 2.
You may need to add a loop delay to avoid the registrar thinking that you're trying to do a DOS attack.
Unfortunately, as far as I know, there's no central repository saying that a certain domain name is or isn't for sale. You'll have to look up a domain name and see if it's owned by any of the major domain name vendors (GoDaddy, etc.)