Defining two sub domains of my domain as nameservers of another domain - dns

Question

Suppose that I own example.com that is served by my own DNS server and I can create every records that I want.
Now imagine that one of my friends get a new domain called new-domain.com and I want to help him manage his domain with his own DNS server.
So in my dns system for example.com, I create two A records as:
my.ns1.example.com -> some.ip.addr
and
my.ns2.example.com -> some.ip.addr
(some.ip.addr is the ip address of his DNS server)
and ask him to set my.ns1.example.com and my.ns2.example.com as name servers for his domain.
But he cannot set them because it gets invalid nameserver error!
Its my understanding that because example.com is working properly in DNS system and thus my.ns1.example.com and my.ns2.example.com are resolved to the IP address properly, so nothing can prevent them to be used as nameservers.
I searched around and found that some people say the nameservers should be registered. I understand registering when we have to ask for setting glue records, but for this case I have no idea why would we need to register those name.
To be more specific with real life example, why would jobs.ns.cloudflare.com is a valid nameserver but www.cloudflare.com is not?

Answer 1

I asked the same question on serverfault.com with this link
There, I quote important part of the answer here,
From a pure DNS perspective, an authoritative nameserver (such as those for com) should not perform any kind of recursion to learn the IP address of the nameservers that are defined in your example.com zone. Instead, the registry permits registrars to add glue records to the com domain, and those registrars can provide a user interface so that the owners of the domains that these custom nameservers live in can do so. (example: Namecheap - How do I register personal nameservers for my domain?)
(To address the elephant in the room...no, these glue records are not strictly required. But policies are policies, and if the registrar interface requires the registry level glue to be present, you have little choice in the matter.)
While the answer does not answer my updated part of the question, I picked it as the answer and decided to ask another question.

Answer 2

The problem does not lie in the names: my.ns1.example.com and my.ns2.example.com are fine.
The registry, and sometimes even the registrar, normally perform a few checks before approving a nameserver change. If your nameservers are rejected as invalid they are most likely not yet correctly configured for your friend's domain. I mean, the servers at my.ns1.example.com and my.ns2.example.com do not contain the minimum required records for new-domain.com.
That said, the registrar support team should be able to provide more details: if it's them who reject the change they should let you know what part of the automatic tests fails and even provide the test output so you can see by yourself. On the other hand, if they just pass the change to the registry (your friend should see a "operation pending at registry level" notice in his control panel for some time) they could do the extra effort of helping you out by providing hints based on their experience with that particular TLD. That is, if your friend didn't grab a promo offer in the 0.99$-5.99$ a year range for the domain: if he pays them something in the 20$-50$ a year range then he should expect and demand a proper, helpful support. I use one of the cheapest registrars and if my nameserver change gets rejected I still get a full report:
Dear customer,
The registry did not accept the nameservers you tried assigning to
new-domain.com because they did not pass the registry tests. Please
check the report we got from the registry below, fix the errors
and try assigning the nameservers again.
Nameservers Resolvable Test: ERROR
my.ns1.example.com. ERROR Unresolvable host my.ns1.example.com.
my.ns2.example.com. ERROR Unresolvable host my.ns2.example.com.
my.ns3.example.com. OK
my.ns4.example.com. OK
SOAQueryAnswerTest: ERROR
my.ns1.example.com. ERROR java.net.SocketTimeoutException
my.ns2.example.com. ERROR java.net.SocketTimeoutException
my.ns3.example.com. OK
my.ns4.example.com. OK
... ... ...
Update: The OP posted an update saying that as soon as the nameservers were registered with the registry, they were accepted in his friend's control panel. It appears that particular registrar checks for glue records and rejects the nameservers if they have none. This is an unnecessary check because glue records are only needed if the nameservers are within the same domain they serve, as explained in these questions. Registrars usually explain this very clearly or at least mention this above the nameserver change form:
Please note that in most cases the ip address is not required and will actually be ignored. It is only necessary if the nameservers you are entering are sub-domains of the selected domain (also called custom nameservers or vanity nameservers).
We can conclude that the friend's registrar performs an unnecessary blocking test and does not respond to user inquiries in a helpful matter. Since the OP has the following need (citation from his updated post on serverfault):
I need to be able to create dynamic nameservers programmatically and ask my users to enter their specific nameservers for their domains in their registrars.
I warmly recommend he does some research looking for a decent and reasonably priced registrar he can point his customers/friends to in case they have any issues with their current ones.