How can I download all domain WHOIS data? - dns

I'm writing some software that analyzes registered domain names and looks for trends. I'm experimenting with some machine learning to help predict what domain names will be purchased in the future based on what types of domains are being registered.
I've been looking around searching for a way to download "all" of the registered domains that exist, but I haven't been able to find a way to do so.
It's easy for me to query individual domain names using the whois command line tool, for example:
$ whois google.com
Domain Name: GOOGLE.COM
Registry Domain ID: 2138514_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2018-02-21T18:36:40Z
Creation Date: 1997-09-15T04:00:00Z
Registry Expiry Date: 2020-09-14T04:00:00Z
Registrar: MarkMonitor Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: abusecomplaints#markmonitor.com
Registrar Abuse Contact Phone: +1.2083895740
Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited
Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited
Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited
Name Server: NS1.GOOGLE.COM
Name Server: NS2.GOOGLE.COM
Name Server: NS3.GOOGLE.COM
Name Server: NS4.GOOGLE.COM
DNSSEC: unsigned
URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/
>>> Last update of whois database: 2018-03-20T03:16:59Z <<<
For more information on Whois status codes, please visit https://icann.org/epp
NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar. Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.
TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability. VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.
The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.
Domain Name: google.com
Registry Domain ID: 2138514_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2018-02-21T10:45:07-0800
Creation Date: 1997-09-15T00:00:00-0700
Registrar Registration Expiration Date: 2020-09-13T21:00:00-0700
Registrar: MarkMonitor, Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: abusecomplaints#markmonitor.com
Registrar Abuse Contact Phone: +1.2083895740
Domain Status: clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)
Domain Status: clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)
Domain Status: clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)
Domain Status: serverUpdateProhibited (https://www.icann.org/epp#serverUpdateProhibited)
Domain Status: serverTransferProhibited (https://www.icann.org/epp#serverTransferProhibited)
Domain Status: serverDeleteProhibited (https://www.icann.org/epp#serverDeleteProhibited)
Registry Registrant ID:
Registrant Name: Domain Administrator
Registrant Organization: Google LLC
Registrant Street: 1600 Amphitheatre Parkway,
Registrant City: Mountain View
Registrant State/Province: CA
Registrant Postal Code: 94043
Registrant Country: US
Registrant Phone: +1.6502530000
Registrant Phone Ext:
Registrant Fax: +1.6502530001
Registrant Fax Ext:
Registrant Email: dns-admin#google.com
Registry Admin ID:
Admin Name: Domain Administrator
Admin Organization: Google LLC
Admin Street: 1600 Amphitheatre Parkway,
Admin City: Mountain View
Admin State/Province: CA
Admin Postal Code: 94043
Admin Country: US
Admin Phone: +1.6502530000
Admin Phone Ext:
Admin Fax: +1.6502530001
Admin Fax Ext:
Admin Email: dns-admin#google.com
Registry Tech ID:
Tech Name: Domain Administrator
Tech Organization: Google LLC
Tech Street: 1600 Amphitheatre Parkway,
Tech City: Mountain View
Tech State/Province: CA
Tech Postal Code: 94043
Tech Country: US
Tech Phone: +1.6502530000
Tech Phone Ext:
Tech Fax: +1.6502530001
Tech Fax Ext:
Tech Email: dns-admin#google.com
Name Server: ns1.google.com
Name Server: ns4.google.com
Name Server: ns2.google.com
Name Server: ns3.google.com
DNSSEC: unsigned
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/
>>> Last update of WHOIS database: 2018-03-19T20:13:36-0700 <<<
The Data in MarkMonitor.com's WHOIS database is provided by MarkMonitor.com for
information purposes, and to assist persons in obtaining information about or
related to a domain name registration record. MarkMonitor.com does not guarantee
its accuracy. By submitting a WHOIS query, you agree that you will use this Data
only for lawful purposes and that, under no circumstances will you use this Data to:
(1) allow, enable, or otherwise support the transmission of mass unsolicited,
commercial advertising or solicitations via e-mail (spam); or
(2) enable high volume, automated, electronic processes that apply to
MarkMonitor.com (or its systems).
MarkMonitor.com reserves the right to modify these terms at any time.
By submitting this query, you agree to abide by this policy.
MarkMonitor is the Global Leader in Online Brand Protection.
MarkMonitor Domain Management(TM)
MarkMonitor Brand Protection(TM)
MarkMonitor AntiPiracy(TM)
MarkMonitor AntiFraud(TM)
Professional and Managed Services
Visit MarkMonitor at http://www.markmonitor.com
Contact us at +1.8007459229
In Europe, at +44.02032062220
For more information on Whois status codes, please visit
https://www.icann.org/resources/pages/epp-status-codes-2014-06-16-en
--
The WHOIS data contains everything I need, but I can't find a way to download the WHOIS data for all currently registered domains.
Is there some way for me to get this data? I feel like it must be publicly available somewhere since the whois CLI tool can so easily query the info.
What am I missing here?

TL;DR: You can not (download all "whois" data).
(side preliminary note: "whois data", while often used is kind of incorrect. You use the whois protocol with a whois client to query a whois server at a registry, and more specifically here a domain name registry, that stores contact data about domain names it sponsors. For the same reason there is no "whois database".)
Now for the long sad story:
It is not possible for many obvious technical and non technical reasons. And you are deeply mistaken if you think the whois CLI command is simple (see my other answer here: https://unix.stackexchange.com/a/407030/211833 for details on that point)
First your question makes no sense for all TLDs at once. You have at least to separate ccTLDs from gTLDs.
1) ccTLDs
ccTLDs have often stricter rules about privacy on personal data and this ought to be even stricter with ongoing European regulations such as GDPR.
Basically some of them already forbid to have access to the complete list of domain names (which is often refered as the "zonefile") which has no personal data, so there is no way you will get access to all the content and the personal data.
You may try to approach some and ask if there is anything possible like for research studies, but I doubt you will be successful and you will need to deal with each ccTLD registry separately as they each deal with their own content (all data on the domain names in the TLD they manage)
2) gTLDs
For them, the situation is quite different.
First, since things are by default more liberal (no protection of personal data), you will see that many registrars/companies provide proxy/privacy services which means that even in a whois query output you will not see much useful data.
But still due to GDPR and assimilated, things are changing. Do a whois on godaddy.com for example and watch all these stars for contact names and emails, and hence the need to go to a website.
However registrars and registries are under contract with ICANN. Which means they both have some requirements, and they are uniform.
First, all registries are mandated to give access to their zonefiles. It is often done throught the CZDA, for which you can find details on ICANN website. Note that it is in fact the list of all domain names publishes, not exactly the list of all domain names registered as you can register a domain name and not put it visible on the DNS.
As for the contact data, that is the rest of the information visible in whois, there are other points not wellknown.
See the registrar agreement at https://www.icann.org/resources/pages/approved-with-specs-2013-09-17-en and specially section 3.3.6 that provide bulk acces to registrar "whois" data. Note how it is tied to some money (USD$10 000) and comes with various limitations on what you can do with it.
Remember that you would need to do it per registrar, so in the gTLDs world that is more than 1000 of them.
There is no equivalent provisions in the registry agreements for public bulk access (see https://newgtlds.icann.org/sites/default/files/agreements/agreement-approved-31jul17-en.html).
Things are complicated because as up today and for some months yet, .COM/.NET remains a thin registry that is one without the contact data stored at registry level, only at registrars.
Also all the above will change in the coming months/years because of the new regulations and also because RDAP, a new protocol, is slated to replace whois at one point. RDAP will allow far greater level of granularity on the access given and the amount of data returned.
Of course, in all cases above, nothing technically forbids anyone to just do regular whois queries and store the results locally. As you can see in a whois output your use of the data is constrained by various limits and bulk querying whois servers always expose you to the risk of being blacklisted or at least heavily rate limited.
Note that for the input (which names to query the whois server for), it is easy to start with zonefiles, even cross TLDs (if site.example exists you can try also site.test even if you do not have .test zonefile), or search engines queries, or dictionaries, etc.
Multiple companies do that and provide tools to search their data, like to do reverse queries and things like that. Maybe some could deliver you bulk results, but certainly not for free.

You can get the Whois record for millions of the registered domains (Active & Inactive) by using WhoisFreaks Database. https://whoisfreaks.com/. It provides well-parsed Whois domain information such as domain registration details, domain registrar details, registrant details, administrative contact, technical contact, server names, domain status, registry data, etc.
WhoisFreaks Database is One of the biggest whois databases with whois
information for 1000+ TLDs (gTLDs & ccTLDs), 422M+ tracked domains,
555M+ whois records.
Well parsed and normalized whois data records which are available in
CSV file format easy to read and integrate into any business system.
Accurate and updated, the WhoisFreaks database is updated on the
daily basis.
WhoisFreaks database provides whois records from 1986.
Custom whois domain data is also available on the basis of Registrant
Specific Domains, Country-Specific Domains, and TLDs Specific
Domains.
Over 4.5 Million domains are registered each month.
WhoisFreaks database contains only unique Whois records of the
domains.
WhoisFreaks JSON response looks like this:
{
"status": true,
"domain_name": "google.com",
"query_time": "2021-03-12 08:03:53",
"whois_server": "whois.markmonitor.com",
"domain_registered": "yes",
"create_date": "1997-09-15",
"update_date": "2019-09-09",
"expiry_date": "2028-09-13",
"domain_registrar": {
"iana_id": "292",
"registrar_name": "MarkMonitor, Inc.",
"whois_server": "whois.markmonitor.com",
"website_url": "http://www.markmonitor.com",
"email_address": "abusecomplaints#markmonitor.com",
"phone_number": "+1.2083895770"
},
"registrant_contact": {
"company": "Google LLC",
"state": "CA",
"country_name": "United States",
"country_code": "US",
"email_address": "Select Request Email Form at https://domains.markmonitor.com/whois/google.com"
},
"administrative_contact": {
"company": "Google LLC",
"state": "CA",
"country_name": "United States",
"country_code": "US",
"email_address": "Select Request Email Form at https://domains.markmonitor.com/whois/google.com"
},
"technical_contact": {
"company": "Google LLC",
"state": "CA",
"country_name": "United States",
"country_code": "US",
"email_address": "Select Request Email Form at https://domains.markmonitor.com/whois/google.com"
},
"name_servers": [
"ns1.google.com",
"ns2.google.com",
"ns3.google.com",
"ns4.google.com"
],
"domain_status": [
"clientTransferProhibited",
"clientDeleteProhibited",
"serverDeleteProhibited",
"serverTransferProhibited",
"serverUpdateProhibited",
"clientUpdateProhibited"
],
"whois_raw_domain": "\nDomain Name: google.com\nRegistry Domain ID: 2138514_DOMAIN_COM-VRSN\nRegistrar WHOIS Server: whois.markmonitor.com\nRegistrar URL: http://www.markmonitor.com\nUpdated Date: 2019-09-09T08:39:04-0700\nCreation Date: 1997-09-15T00:00:00-0700\nRegistrar Registration Expiration Date: 2028-09-13T00:00:00-0700\nRegistrar: MarkMonitor, Inc.\nRegistrar IANA ID: 292\nRegistrar Abuse Contact Email: abusecomplaints#markmonitor.com\nRegistrar Abuse Contact Phone: +1.2083895770\nDomain Status: clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)\nDomain Status: clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)\nDomain Status: clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)\nDomain Status: serverUpdateProhibited (https://www.icann.org/epp#serverUpdateProhibited)\nDomain Status: serverTransferProhibited (https://www.icann.org/epp#serverTransferProhibited)\nDomain Status: serverDeleteProhibited (https://www.icann.org/epp#serverDeleteProhibited)\nRegistrant Organization: Google LLC\nRegistrant State/Province: CA\nRegistrant Country: US\nRegistrant Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com\nAdmin Organization: Google LLC\nAdmin State/Province: CA\nAdmin Country: US\nAdmin Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com\nTech Organization: Google LLC\nTech State/Province: CA\nTech Country: US\nTech Email: Select Request Email Form at https://domains.markmonitor.com/whois/google.com\nName Server: ns3.google.com\nName Server: ns2.google.com\nName Server: ns1.google.com\nName Server: ns4.google.com\nDNSSEC: unsigned\nURL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/\n>>> Last update of WHOIS database: 2021-03-11T22:57:36-0800 <<<\n\nFor more information on WHOIS status codes, please visit:\n https://www.icann.org/resources/pages/epp-status-codes\n\nIf you wish to contact this domainâs Registrant, Administrative, or Technical\ncontact, and such email address is not visible above, you may do so via our web\nform, pursuant to ICANNâs Temporary Specification. To verify that you are not a\nrobot, please enter your email address to receive a link to a page that\nfacilitates email communication with the relevant contact(s).\n\nWeb-based WHOIS:\n https://domains.markmonitor.com/whois\n\nIf you have a legitimate interest in viewing the non-public WHOIS details, send\nyour request and the reasons for your request to whoisrequest#markmonitor.com\nand specify the domain name in the subject line. We will review that request and\nmay ask for supporting documentation and explanation.\n\nThe data in MarkMonitorâs WHOIS database is provided for information purposes,\nand to assist persons in obtaining information about or related to a domain\nnameâs registration record. While MarkMonitor believes the data to be accurate,\nthe data is provided \"as is\" with no guarantee or warranties regarding its\naccuracy.\n\nBy submitting a WHOIS query, you agree that you will use this data only for\nlawful purposes and that, under no circumstances will you use this data to:\n (1) allow, enable, or otherwise support the transmission by email, telephone,\nor facsimile of mass, unsolicited, commercial advertising, or spam; or\n (2) enable high volume, automated, or electronic processes that send queries,\ndata, or email to MarkMonitor (or its systems) or the domain name contacts (or\nits systems).\n\nMarkMonitor reserves the right to modify these terms at any time.\n\nBy submitting this query, you agree to abide by this policy.\n\nMarkMonitor Domain Management(TM)\nProtecting companies and consumers in a digital world.\n\nVisit MarkMonitor at https://www.markmonitor.com\nContact us at +1.8007459229\nIn Europe, at +44.02032062220\n--",
"registry_data": {
"domain_name": "GOOGLE.COM",
"query_time": "2021-03-12 08:03:52",
"whois_server": "whois.verisign-grs.com",
"domain_registered": "yes",
"create_date": "1997-09-15",
"update_date": "2019-09-09",
"expiry_date": "2028-09-14",
"domain_registrar": {
"iana_id": "292",
"registrar_name": "MarkMonitor Inc.",
"whois_server": "whois.markmonitor.com",
"website_url": "http://www.markmonitor.com",
"email_address": "abusecomplaints#markmonitor.com",
"phone_number": "+1.2083895740"
},
"name_servers": [
"NS2.GOOGLE.COM",
"NS1.GOOGLE.COM",
"NS4.GOOGLE.COM",
"NS3.GOOGLE.COM"
],
"domain_status": [
"clientDeleteProhibited",
"clientTransferProhibited",
"serverDeleteProhibited",
"serverTransferProhibited",
"serverUpdateProhibited",
"clientUpdateProhibited"
],
"whois_raw_registery": "\n Domain Name: GOOGLE.COM\n Registry Domain ID: 2138514_DOMAIN_COM-VRSN\n Registrar WHOIS Server: whois.markmonitor.com\n Registrar URL: http://www.markmonitor.com\n Updated Date: 2019-09-09T15:39:04Z\n Creation Date: 1997-09-15T04:00:00Z\n Registry Expiry Date: 2028-09-14T04:00:00Z\n Registrar: MarkMonitor Inc.\n Registrar IANA ID: 292\n Registrar Abuse Contact Email: abusecomplaints#markmonitor.com\n Registrar Abuse Contact Phone: +1.2083895740\n Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited\n Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited\n Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited\n Domain Status: serverDeleteProhibited https://icann.org/epp#serverDeleteProhibited\n Domain Status: serverTransferProhibited https://icann.org/epp#serverTransferProhibited\n Domain Status: serverUpdateProhibited https://icann.org/epp#serverUpdateProhibited\n Name Server: NS1.GOOGLE.COM\n Name Server: NS2.GOOGLE.COM\n Name Server: NS3.GOOGLE.COM\n Name Server: NS4.GOOGLE.COM\n DNSSEC: unsigned\n URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/\n>>> Last update of whois database: 2021-03-12T07:03:38Z <<<\n\nFor more information on Whois status codes, please visit https://icann.org/epp\n\nNOTICE: The expiration date displayed in this record is the date the\nregistrar's sponsorship of the domain name registration in the registry is\ncurrently set to expire. This date does not necessarily reflect the expiration\ndate of the domain name registrant's agreement with the sponsoring\nregistrar. Users may consult the sponsoring registrar's Whois database to\nview the registrar's reported date of expiration for this registration.\n\nTERMS OF USE: You are not authorized to access or query our Whois\ndatabase through the use of electronic processes that are high-volume and\nautomated except as reasonably necessary to register domain names or\nmodify existing registrations; the Data in VeriSign Global Registry\nServices' (\"VeriSign\") Whois database is provided by VeriSign for\ninformation purposes only, and to assist persons in obtaining information\nabout or related to a domain name registration record. VeriSign does not\nguarantee its accuracy. By submitting a Whois query, you agree to abide\nby the following terms of use: You agree that you may use this Data only\nfor lawful purposes and that under no circumstances will you use this Data\nto: (1) allow, enable, or otherwise support the transmission of mass\nunsolicited, commercial advertising or solicitations via e-mail, telephone,\nor facsimile; or (2) enable high volume, automated, electronic processes\nthat apply to VeriSign (or its computer systems). The compilation,\nrepackaging, dissemination or other use of this Data is expressly\nprohibited without the prior written consent of VeriSign. You agree not to\nuse electronic processes that are automated and high-volume to access or\nquery the Whois database except as reasonably necessary to register\ndomain names or modify existing registrations. VeriSign reserves the right\nto restrict your access to the Whois database in its sole discretion to ensure\noperational stability. VeriSign may restrict or terminate your access to the\nWhois database for failure to abide by these terms of use. VeriSign\nreserves the right to modify these terms at any time.\n\nThe Registry database contains ONLY .COM, .NET, .EDU domains and\nRegistrars."
}
}
I hope it would be the best solution that will help you in purchasing the domain names.
Disclaimer: I work for WhoisFreaks API & Database.

For this you can also try the website www.whoisdb.co, Here you will get instant access to a 100M+ whois database. You will get whois information like
"num","domain_name","query_time","create_date","update_date","expiry_date","domain_registrar_id","domain_registrar_name","domain_registrar_whois","domain_registrar_url","registrant_name","registrant_company","registrant_address","registrant_city","registrant_state","registrant_zip","registrant_country","registrant_email","registrant_phone","registrant_fax","administrative_name","administrative_company","administrative_address","administrative_city","administrative_state","administrative_zip","administrative_country","administrative_email","administrative_phone","administrative_fax","technical_name","technical_company","technical_address","technical_city","technical_state","technical_zip","technical_country","technical_email","technical_phone","technical_fax","billing_name","billing_company","billing_address","billing_city","billing_state","billing_zip","billing_country","billing_email","billing_phone","billing_fax","name_server_1","name_server_2","name_server_3","name_server_4","domain_status_1","domain_status_2","domain_status_3","domain_status_4"
Some key hights
Download more than 50000 whois leads of newly registered domains everyday
Every day, more than 100,000 domains are added to the list created with whois leads information.
You can even use their API or Webhooks to get the databse
All database can also be downloaded in the form of xls file or zip file
OUTGOING WEBHOOK
When the event below is triggered inside the software, send and POST a JSON to the provided URL.
Preview of JSON
{
"webhook_event": "new_domain_registered",
"num": "value",
"domain_name": "value",
"query_time": "value",
"create_date": "value",
"update_date": "value",
"expiry_date": "value",
"domain_registrar_id": "value",
"domain_registrar_name": "value",
"domain_registrar_whois": "value",
"domain_registrar_url": "value",
"registrant_name": "value",
"registrant_company": "value",
"registrant_address": "value",
"registrant_city": "value",
"registrant_state": "value",
"registrant_zip": "value",
"registrant_country": "value",
"registrant_email": "value",
"registrant_phone": "value",
"registrant_fax": "value",
"administrative_name": "value",
"administrative_company": "value",
"administrative_address": "value",
"administrative_city": "value",
"administrative_state": "value",
"administrative_zip": "value",
"administrative_country": "value",
"administrative_email": "value",
"administrative_phone": "value",
"administrative_fax": "value",
"technical_name": "value",
"technical_company": "value",
"technical_address": "value",
"technical_city": "value",
"technical_state": "value",
"technical_zip": "value",
"technical_country": "value",
"technical_email": "value",
"technical_phone": "value",
"technical_fax": "value",
"billing_name": "value",
"billing_company": "value",
"billing_address": "value",
"billing_city": "value",
"billing_state": "value",
"billing_zip": "value",
"billing_country": "value",
"billing_email": "value",
"billing_phone": "value",
"billing_fax": "value",
"name_server_1": "value",
"name_server_2": "value",
"name_server_3": "value",
"name_server_4": "value",
"domain_status_1": "value",
"domain_status_2": "value",
"domain_status_3": "value",
"domain_status_4": "value"
}

This website allows to download lists of registered domain names: https://networksdb.io .
Some are free, some are paid for. It allows also to access aggregates WHOIS data for IP blocks to find which companies own what address blocks and the other way around.

Related

How to collect billing address through Stripe Payment Request Button

I'm creating a payment request like this:
const paymentRequest = stripe.paymentRequest({
country: config.stripeCountry,
currency: config.currency,
total: {
label: 'Total',
amount: parseInt((paymentAmounts.totalAmount * 100).toFixed())
},
requestShipping: true,
requestPayerEmail: true,
requestPayerName: true,
// It seems this one is not working
shippingOptions: config.shippingOptions
})
Now I'd like to collect a customer's billing address. But the doc says:
Requesting the payer’s name, email, or phone is optional, but highly
recommended, as it also results in collecting their billing address
for Apple Pay. The billing address can be used to perform address
verification and block fraudulent payments. For all other payment
methods, the billing address is automatically collected when
available.
So how shall I collect their billing address with the payer's name, email or phone?
For Apple Pay, requesting one of name, email, or phone (which you're already doing via requestPayerEmail and requestPayerName) triggers billing address to be collected. For other payment methods, you don't have to do anything.
In other words, you're all set as-is.
https://stripe.com/docs/stripe-js/reference#stripe-payment-request
We highly recommend you collect at least one of name, email, or phone as this also results in collection of billing address for Apple Pay. The billing address can be used to perform address verification and block fraudulent payments. For all other payment methods, the billing address is automatically collected when available.

How can I know status of a domain Registrar lock

When start a transfer in namecheap we can get domain Registrar lock.namecheap screenshot
How can I get this information?
Using whois I can only get clientTransferProhibited Does it mean domain is locked?
Domain Name: google.com
Registry Domain ID: 2138514_DOMAIN_COM-VRSN
Registrar WHOIS Server: whois.markmonitor.com
Registrar URL: http://www.markmonitor.com
Updated Date: 2018-02-21T10:45:07-0800
Creation Date: 1997-09-15T00:00:00-0700
Registrar Registration Expiration Date: 2020-09-13T21:00:00-0700
Registrar: MarkMonitor, Inc.
Registrar IANA ID: 292
Registrar Abuse Contact Email: abusecomplaints#markmonitor.com
Registrar Abuse Contact Phone: +1.2083895740
Domain Status: clientUpdateProhibited (https://www.icann.org/epp#clientUpdateProhibited)
Domain Status: clientTransferProhibited (https://www.icann.org/epp#clientTransferProhibited)
Domain Status: clientDeleteProhibited (https://www.icann.org/epp#clientDeleteProhibited)
Domain Status: serverUpdateProhibited (https://www.icann.org/epp#serverUpdateProhibited)
Domain Status: serverTransferProhibited (https://www.icann.org/epp#serverTransferProhibited)
Domain Status: serverDeleteProhibited (https://www.icann.org/epp#serverDeleteProhibited)
Registrant Organization: Google LLC
Registrant State/Province: CA
Registrant Country: US
Admin Organization: Google LLC
Admin State/Province: CA
Admin Country: US
Tech Organization: Google LLC
Tech State/Province: CA
Tech Country: US
Name Server: ns1.google.com
Name Server: ns3.google.com
Name Server: ns4.google.com
Name Server: ns2.google.com
DNSSEC: unsigned
URL of the ICANN WHOIS Data Problem Reporting System: http://wdprs.internic.net/

Does the TLS cert would require an common SAN

Based on the below reference link in configuring Haproxy with TLS:
Do i need to have the certificates generated with common SAN(Subject ALternate name) on all the target nodes (or)
Having the individual certs without any common SAN would work ?
https://serversforhackers.com/c/using-ssl-certificates-with-haproxy
Look at https://security.stackexchange.com/questions/172626/chrome-requires-san-names-in-certificate-when-will-other-browsers-ie-follow : some browsers (Chrome) require names to be in the SAN part as they disregard now completely the CN field
So even for a one domain certificate you need the domain both in the CN (as this is not optional) and in the SAN part.
It is also in the CAB Forum requirements, section 7.1.4.2.1 :
Certificate Field: extensions:subjectAltName
Required/Optional: Required
Contents: This extension MUST contain at least one entry.
Each entry MUST be either a dNSName containing the Fully-Qualified
Domain Name or an iPAddress containing the IP address of a server.
The CA MUST confirm that the Applicant controls the Fully-Qualified
Domain Name or IP address or has been granted the right to use it by
the Domain Name Registrant or IP address assignee, as appropriate.
Wildcard FQDNs are permitted.
Note that some other browsers, like Firefox, fallback to the CN instead, see https://bugzilla.mozilla.org/show_bug.cgi?id=1245280 and see beginning of patch at https://hg.mozilla.org/mozilla-central/rev/dc40f46fae48 for the security.pki.name_matching_mode configuration option.

Can you get/scrape emails from instagram accounts?

Someone is telling me that they can get all the email accounts associated to instagram accounts. Which would be a gold mind for marketing obviously.
They gave me my email associated to my email account as proof.
From my understanding, email addresses are hidden on instagram and this can not be done. But, i do not know how he got that email address as i use it ONLY for instagram.
I went and tried to do a password reset to see if they gave you the email when you request a pass and no it blocked out half my email address and i couldntmake it out.
So, can you get emails attached to instagram accounts? i do not see how this could be done
As far as I know, you can't. Instagram API does not provide it. And the "retrieve password" page and requests seems well done, so you don't have access to the e-mail (and you really should not).
This info below is the officially data response from Users endpoint.
{
"data": {
"id": "1574083",
"username": "snoopdogg",
"full_name": "Snoop Dogg",
"profile_picture": "http://distillery.s3.amazonaws.com/profiles/profile_1574083_75sq_1295469061.jpg",
"bio": "This is my bio",
"website": "http://snoopdogg.com",
"counts": {
"media": 1320,
"follows": 420,
"followed_by": 3410
}
}
If someone really knows your e-mail, I can imagine some options:
1) You use the same e-mail on Instagram and on another service that delivers e-mail (e.g. Facebook) and this guy has access to this service and had a lucky guess with you.
2) They googled you and discovered your main e-mail somewhere, and had a lucky guess.
3) They discovered a security vulnerability somewhere on Instagram code and now are exploring it. (but this is no good)
If I were you, I would create another account with a brand new e-mail and ask him again to "discover" this new one. If he can, the 3rd option is probably the right one. And if you do this, please share with us! :)

API '500' Error when delegating admin access on MySite (OneDrive / Office365)

I would value some advice as we are facing an issue with the Office 365 API where it returns 500 errors, when we request delegated administrative access to the Office 365 MySites Site Collection.
The use case scenario is the following:
We have independent organizations, 'A' and 'B' , which both own their respectives Azure Active Directroy accounts and O365/Azure subscriptions.
Organiation B wishes to utilise an application provide Organisation A which requires delegated administrative access to it's Office365 MySites Site Collection.
Organisation B successfully adds the app to their Active Directory Account by using the OAuth2 authorization process, granting the permissions that the application requires over their company active directory.
The final part is delegating access of an admin account in Organisation B to the Mysites site collection in Organisation B's Office 365 account. If we utilise the manual process via the administrative panel all works well, although programatically via the API we receive a '500' error.
We are using the following call to programmatically add the user as site collection administrator: (please note: I have used [ dummmy id's] below )
Endpoint URL:
https://<..org_a..>.my.sharepoint.com/personal/[[user_OrganisationA_com]]/_api/web/siteUsers/getByEmail(#u)?#u='[[info#_OrganisationA.com']]
Method: POST
Body: {'__metadata':{ 'type': 'SP.User' }, 'IsSiteAdmin':'true'}
HTTP Headers:
• "Content-Type": "application/json; odata=verbose"
• "X-HTTP-Method": "MERGE"
• "Authorization": "Bearer <OAuth2 token>"
• "Accept": "application/json"
On the request above we are trying to add info#_organisationB.com as site collection administrator to UserC's personal site in One Drive for Business service (UserC is identified by the email UserC#_organisationB.com). The user info#_organisationB.com has Global Administrator privileges in Organisation B's Office365 domain and UserC#_organisationB.com) is a basic user with no admin rights.
The call returns a 500 HTTP Status code (Internal Server Error) and the following message:
object(stdClass)[18]
public 'odata.error' =>
object(stdClass)[19]
public 'code' => string '-2146232832, Microsoft.SharePoint.SPException' (length=45)
public 'message' =>
object(stdClass)[20]
public 'lang' => string 'en-US' (length=5)
public 'value' => string 'You need to be a site collection administrator to set this property.' (length=68)
To our knowledge an API call should never return a 500 HTTP Status Code (Internal Server Error), if we are not allowed to do what we intend it should provide an Insufficiente permissions message or similar, however, the fact that we can manually apply this through the SharePoint Online management portal makes us believe we are hitting a bug that is stopping us from achieving what we intend.
Any advise would be appreciated!
Are you using SSL from the website you execute the request?

Resources