I have been trying to figure out how to programmatically check if a gmail account exists. Almost all searches lead to validation services like xverify or EmailOversight where you validate any email address on a cost per request basis.
What I am interested in is a way to do that directly, without a middleman. In other words, how do these validation services do it? Seems like there should be some sort of an API that google provides for those guys to ping to see if an email address is valid.
Please note that I am not interested in checking the syntax of an email address. So I am not looking for some kind of a regex solution.
Also, what I have tried is connecting to gmail.com MX record domains (e.g. alt3.gmail-smtp-in.l.google.com) and trying to extract the validity of an email address by running simple SMTP commands. Essentially what this article suggests: https://www.webdigi.co.uk/blog/2009/how-to-check-if-an-email-address-exists-without-sending-an-email/
But I cannot do that for any kind of volume. Gmail will start blocking your connection attempt after a certain number of connections. So that method is not scalable. That's why I feel like there has got to some other way.
*******ADDED*********
Here is why this question is different from How to check programmatically a email is existing or not
That post provides only one solution, and it's one I have already tried - using SMTP commands. Google will NOT allow to do that on any kind of scale. If I only had to validate a few emails, then that would be a sensible solution, but if I have 10,000, it is not.
Related
For legal reasons we can't send the email to HIBP in clear text.
Regarding "Domain Search" functionnality, there's no API (as far as I know).
It works by sending you multiple emails, no API, so hard to automate process around that.
I've also read one of Troy's old blog post about a Domain Search callback system he put in place for a specific case : https://www.troyhunt.com/have-i-been-pwned-goes-little-bit/ which looks good but no publicly available (that's my understanding).
So, if HIBP doesn't allow me to do that, I thought of other solutions based on it :
Firefox Monitor and 1pwd :
They use the k-anonymity principle (https://blog.mozilla.org/security/2018/06/25/scanning-breached-accounts-k-anonymity/) which consist to send only a hash of the searched email.
Unfortunately, that functionality is reserved to 1pwd & firefox monitor.
Am I missing a way to interact with HIBP without sending in clear the email ?
Thank you
Short answer: no.
Mozilla and 1Password use the k-anonymity model described here: https://www.troyhunt.com/were-baking-have-i-been-pwned-into-firefox-and-1password/
The reason I don't make that generally available is that every single k-anonymity search returns multiple results which would make it much easier for a nefarious party to abuse. Domain searches require verification of control at the time of search; an API that didn't require verification would also be open to misuse.
If you'd like to see new features, suggest them here (or vote on them if they already exist): https://haveibeenpwned.uservoice.com/
Just thinking about my registration flow for my Nodejs/MongoDB app and wondered whether I really need to check if email addresses are already in the database when users register.
I make my users verify their email address by clicking on an email that I will send to them. They must do this before logging in at all. Once they do click on this link then I will mark that email as verified in the database. With this in mind, I'm thinking that it doesn't really matter if someone registers with the same email address (for whatever reason) as it won't be marked as verified unless they can access the email account.
I'm wondering however if it will impact on performance when it comes to searching for users in the database via email address. For example, users login using email address. Would it be advisable in this case to create an index on both email and verified?
Have I overlooked something that could potentially be a security flaw?
Performance-wise it will probably not matter a great deal (you probably already have an index on the email field), but I don't think you should allow it anyway.
For one, it has the potential to add junk to your database. Secondly, it would allow existing users to create—by mistake or not—a second account with the same e-mail address, which could cause all sorts of issues.
You don't give a specific reason for wanting to allow duplicate e-mail addresses, but a duplicate check isn't really all that time-consuming to implement (you could also consider a unique index on email so the database will throw an error when someone is trying to reregister an existing address).
I have 2 Windows 2008 R2 boxes running in Microsoft Azure. My ASP.NET 4.0 site (let's imagine it's running at "example.com") has a standard Contact Us form.
When a user sends a Contact Us message, I use System.Net.Mail and SmtpDeliveryMethod.Network to deliver mail to an IIS6 SMTP server running on each box, which sends the mail to a Google Apps "enquiries#example.com" account, using the email address the user entered into the Contact Us form as the "From" address.
This was working beautifully for a year until I checked it today, and found this error in a .BDP file in the \Badmail folder:
550-5.7.1 Our system has detected an unusual rate of unsolicited mail originating from your IP address. To protect our users from spam, mail sent from your IP address has been blocked. Please visit http://www.google.com/mail/help/bulk_mail.html to review our Bulk Email Senders Guidelines.
Obviously Google upped their anti-spam strategies in the last 6 months - last time it worked was Feb 2013 (yeah, we don't get much mail luckily... yet).
I've read the Bulk Senders Guidelines linked above, but they're not really suited to my use case. My case is not sending emails from our server to users of our site (I simply use the Gmail API and send from our enquiries#example.com for that), but rather to collect users' enquiries so that we can easily respond by clicking Reply in that inbox.
I am looking for the easiest solution here. In response to the ones in Google's Bulk Senders Guidelines:
Use a consistent IP address to send bulk mail: I already do, doesn't seem to help
Reverse DNS: Godaddy, my domain and DNS provider doesn't seem to support them: http://support.godaddy.com/groups/domains-management-and-services/forum/topic/how-do-i-setup-reverse-dns/ Anyone know if there's a way?
Use the same address in the 'From:' header on every bulk mail you send: This is totally not my use case. I'll have different From headers in every email
SPF record: I think this only works if I am sending From ...#example.com every time. Is that right? My feeling is SPF doesn't help me here. Would love someone to enlighten me.
DKIM: This looks hellishly complicated, but I'll pursue it if someone thinks it can work in this case. Specifically is it OK that the From address doesn't match the "signing domain"? Anyone got any good "how to" links? And will this be sufficient for Google to un-blacklist me?
Sendgrid: Azure's preferred mail sending app. This means signing up, code changes, testing, and unknowns like "does Sendgrid allow any From address? It's non-trivial, and I'd like to avoid this, but again, will go there if it's what people think is the sanest option.
As a general answer to your questions, sending email on behalf of many different domains from one IP (e.g. example.net, example.org, and ex.co from 10.0.0.1) is generally seen as spammy behavior (and therefor not recommended).
Your points 1-5 only apply if you're sending from one domain. rDNS, SPF, and DKIM only improve delivery for one IP to one domain (in a generally 1:1) relationship.
Generally, the best way to avoid getting marked as spam in a situation like this is to set the From email as a consistent one that you actually control (e.g. enquery-sender#example.com), and then setting the Reply-To as the entered address (e.g. enquirer#someprovider.com). This way you consistently send from one domain, while still getting the benefit of replies going to the message originator (for example LinkedIn does it this way). Doing this will allow you to setup rDNS, SPF, and DKIM with benefit.
That said, if you decide that you don't want to use the recommended Reply-To method, you can use SendGrid to send from any arbitrary domain. It should not require any significant code change (just switching your current SMTP credentials to SendGrid's).
Disclaimer: I am a SendGrid employee.
It seems pretty typical to limit user accounts to unique email addresses. So on my user registration form, I am doing email validation and returning a message like
An account has already been registered for foo#bar.com
Then it occurred to me that an attacker could use this form to determine information about my users. Is there an alternative way to provide validation messages to my users without compromising security? It doesn't seem to me there is any way around it.
For most kinds of sites, I would expect that explicitly hiding this information would be a poor user experience trade-off. The better solution is to use CAPTCHA to help prevent war-dialing of email addresses.
The exception would be in cases where an attacker is seeking out information about a specific user (rather than just trying to find "some account"). As an example, if your site caters to people who have a strong interest in anonymity, and there are attackers who have a strong interest in finding out if a specific user is using the site, then the approach should be different. My approach would likely be to send an email to the address indicating the "already registered" error. The user experience annoyance would be outweighed by the user's anonymity interest.
Rob Napier's answer is correct. You should decide whether your users actually need that anonymity at the cost of degraded UX (in most cases they wouldn't care).
Here's how some big names do it:
AWS: Error: Account with this email already exists.
Apple ID: This email address is not available. Choose a different address.
Cloudflare: A user with that email already exists (Code: 1079)
Linkedin: Someone's already using that email.
Stack Overflow: Forgot your account’s password or having trouble logging into your Team? Enter your email address and we’ll send you a recovery link.
Simply tell them they cannot use the email address they have supplied ? You don't need to give any more reason than that? If they know it's not their email they might still guess it's an existing address but you haven't confirmed that.
Or
How about telling them you've sent an email to that address and they need to confirm - even if you haven't.
I can think of one way: you could ask for an email address and then send the link for a one-time registration form to that email address. You might need a captcha in there to stop spam. If the email is already in the system it could send a message saying that they already have an account.
I think this is unnecessary tho, unless your website is especially secret, like a support group for abuse victims.
I don't think it is really possible to create an error message without indicating that the email address is already in use, as that is the point of the email address.
These are the options I see:
1) Display a not so clear error message like "Email address is invalid"
2) Accept the registration and inform the user about his earlier account via email (I would suggest that)
In any case you can improve security thru captcha codes and throttling requests by the same client. If you care that much about the privacy of your users, forcing a unique email registration is maybe not suitable for your site.
You can receive this error message if you have already an existing email alias with this email or user foo#bar.com.
You can check that if the email alias exists then you will have to first remove it and then try to create it.
I was thinking about creating a service similar to posterous where users can post to a fixed address like post#domain.com and then authentication of the posts would be based on some combination of the from address and header signature. Posterous seems to be doing something clever so that they can detect if a message is being spoofed or sent from an unknown source.
Anyone know what they might be doing?
There are some basic heuristics you could use, to try to detect it.
The most basic thing would be to look at the originating host and headers of the previous emails and see if they match previous ones. If the poster's email domain uses SPF you can verify the IP address has permission to send on behalf of that domain. It's easy to spoof an email - it's harder to spoof an email and actually have it originate from a subnet that matches where your previous mail has come from. But that can be subverted, as well.
First off, the email you post to should be random and unrelated to any display name. This keeps random strangers on the net from knowing where to post in the first place.
You could also go far more sophisticated and implement some sorts of machine-learning-ish things (or even more straightforward heuristics) to pick up on style, but that would be much tougher to get right.
Or just be lame, and upon receipt of a post, fire back an email to the poster with a link to verify and/or a special token to send back as a subject (if they're posting from a phone, etc. with no browser).