What is the exact difference between the envelope and the email in smtp?
Why does the protocol need the envelope? In snail mail, the letter needs not contain addresses and is not visible to the postman (at least that's what you want to believe), so all the routing is made entirely based on the envelope and that is clear to me.
However in e-mail, the letter itself is parseable and has headers such as From: and To:. Why is that not enough to route email? Why do we need an envelope?
And what is the meaning of a divergence between the envelope and the email itself? Does it necessarily mean that someone is trying to game the system, or is it, under certain circumstances (what circumstances?), legal to have a difference between them.
Your clarifications are more than welcome.
Puzzled email receiver.
The recipient address in the SMTP RCPT TO: command is what mail transports use to determine the actual recipient of an email. The addresses in the To: and Cc: headers are there for mail readers to display to users who the recipients are, but they're not actually used by mail transports.
Most mail clients set the same addresses in SMTP RCPT TO: and MAIL FROM: commands that they insert in To:, Cc:, and From: headers, so the "envelope" addresses will be the same as the addresses in the headers.
Envelope and header addresses will usually agree for most legitimate mail. Notable exceptions are Bcc: addresses and mailing lists.
Spammers often forge header addresses to try to avoid spam filters.
The bottom line is that the overhead would be too great. When mail servers are processing millions of messages per day, it's much faster during the SMTP session (The Envelope) when the mail server can process individual commands.
If you didn't use an Envelope, you would have to push the entire message at the SMTP server, before you could find out if the message needs to be rejected. Some messages are quite large (I've seen 100+Meg messages working with GIS images).
By using an Envelope, we can issue specific 1 line commands against the mail server, and the mail server will tell us if we are successful or not, right then and there.
The envelope is used by the SMTP server and the message headers are used by a mail reader as everyone else has said.
What has not been said is that the RCPT TO: is used to route the message to a specific user regardless of where the user's name appears in the headers. The user does not necessarily need to appear in the To: or even the Cc: headers. Think of a Bcc: where the only thing that the receiver knows is who the message was from. In this case, the To: and CC: headers should be empty - hence the blind part of BCC. In another case, if an email message has the user mentioned in the CC list along with 10 other users, how can the SMTP routing pick the appropriate user. The answer is that it uses the RCPT line to route the message.
This is also used when routing via mailing lists. The To: header will contain the mailing list e-mail address. An SMTP system will generate separate messages for each user in the list each with a specific RCPT TO: user#host... envelope. In this case, the user's name will not even appear in any of the other headers.
An email message is very similar to a physical letter that you would send in the mail. There is an envelope, with To/From information, and there is the actual letter on the inside, with it's own To/From information. The envelope to/from information is the real information that is used for message delivery, for both email servers and post offices.
When an envelope comes into a post office, they inspect the To address on the envelope, and send it to the correct destination. The post office workers have no knowledge of the letter inside the envelope. The letting inside could have completely different To/From information than the envelope says. The Envelope could say the message is to Bob, but the letter inside may say it's for Alice. Or in real world: The envelope could say the message is to SomeCompanyName, and when secretary open envelope, letter inside say, it's is for Mr. Brown which work in SomeCompanyName.
Source: Email Envelope vs Email Header
Related
Using fuglu plugin and python, trying to parse thru' the suspect, i am trying to get the envelope recipients. eg. Sometimes I can send TO: members#domain.com, xyz#anotherdomain.com
How does fuglu decide who is the envelope recipient when members#domain.com is a group mail id
Postfix has already translated the group to individual ids on incoming relay server before it reaches the outgoing server with this fuglu plugin.
I need to access the envelope recipient address only. How can I get this value in fuglu?
In fuglu, you can get the list of envelope recipients using suspect.recipients.
Now from your question it seems like a message may previously have been sent to group#example.com and this group adress then rewrites to recipient1#example.com, recipient2#example.com, ...
Which one of these adresses fuglu sees in suspect.recipients depends on your mailserver/routing configuration. I understand that fuglu in your case runs on a second mailserver where the alias has already been expanded, so suspect.recipients will likely not contain group#example.com .
It may contain either all or only some of the expanded adresses if the server is configured to handle multiple recipients in the same transaction or if the mailserver is configured to split up the recipients ( destination_recipient_limit in postfix for example).
If your goal is to extract the original recipient address (group#example.com), this is actually quite hard, as this information may not even be available in the SMTP transaction anymore. Some servers write it into a received header:
Received: from [...]
by your.relay.example.com with SMTPS id blabla
for <group#example.com>
and in this case you could extract it from there with some regex magic. But if your inbound server does not add the for <original recipient> part you're out of luck. In this case you could fall back to extracting the To: header from the message using suspect.get_message_rep().get('To') ), which likely contains the original envelope recipient, but unfortunately this is not guaranteed.
I'm using phpmailer on my ERP and the from email is a noreply# and the reply-to is my client/company email.
Sometimes I'm receiving "Mail delivery failed: returning message to sender" that is normal but I'm receiving those to my noreply# and not the client email.
How can I force the phpmailer to send the "Mail delivery failed" to reply-to email?
Thank you for your help.
Bounces go to the return-path, not the reply-to address, because they are bounces, not replies.
It's not PHPMailer that's sending your bounces there, it's the receiver, and they are acting correctly.
You can set the return path indirectly using the Sender property which is used to set the envelope sender, which the receiver converts into a return-path before attempting delivery. Do not be tempted to set a return-path header yourself; it's the receiver's job.
$mail->Sender = 'bounces_go_here#example.com';
Beware though – this address is what SPF applies to, so you need to be very sure that your SPF records are set up correctly. For reliable bounce handling, I'd recommend making use of VERP addressing here too.
I've just set up SendGrid, authentificated my domain and did the sender verification.
I use Node.js to send email via the API.
When I make a test, the mail is sent to my outlook address. It takes 3 seconds to appear in Outlook, not in spam, in my inbox.
However, when I make the same test, 30 seconds later, the mail is processed by SendGrid, but I never receive in Outlook..
How can it be possible ?
please check your sendgrid activity tab. Processed doesnt mean that outlook received it. After processed there is another level of delivered. Does it say in your sendgrid activity log that it was delivered. It might say not and outlook refused it because they think one email after the other might be spam. (Keep this in mind when sendgrid sends emails it doesnt come from your email address, and some might mark that as spam)
At some messages in Gmail, the MESSAGE-ID header is missing and I need to use it as IN-REPLY-TO header when replying so I can keep the replies at the same conversation.
When those emails with missing MESSAGE-ID header are replied from the Gmail's WEB UI, the replies stay with the conversation and the reply in fact acquires a IN-REPLY-TO header something looks like <-4185615914882731559#unknownmsgid> I could not discover how is this generated.
I've been trying to find a solution to this problem quite a while, ended up trying a few desktop email clients, Airmail and Sparrow to see how they behave. If I ever reply a particular email which is missing the MESSAGE-ID header using those email clients, the sent reply also acquires the same IN-REPLY-TO header <-4185615914882731559#unknownmsgid> and the reply indeed stays at the same conversation as expected.
So, I would like to ask how they achieve this, do they somehow generate the IN-REPLY-TO header <-4185615914882731559#unknownmsgid>? Is there a rule or a workaround?
This is a classic problem, every mail reader has it. The classic solution is:
copy the References field from the message you reply to, if any.
make up an In-Reply-To field, perhaps with a made-up message-id, perhaps with the sender's address and the message's date.
Keep the subject, and append "re: ".
You may also copy the Thread-Index field, if there is one.
Gmail's and most other threading algorithms will then connect the dots.
Thread-Index is a Microsoft thing, noone else uses it, as far as I know. The format of the hex blob is not specified, but copying that will help probably help Exchange, if the original message comes from Exchange.
If you just want to send an email and make sure it threads with another email in that user's inbox (and not any other gmail users that may receive the reply) then I believe you can use the Gmail API's messages.send() method and set the message.threadId to match the thread you want it to thread with.
I don't think that e-mails delivered to your gmail account have no message id. My evidence is that I just tried sending one to myself via telnet without a message id, and gmail's mx server added this
Message-Id: <54412844.63bbb40a.0772.ffffcbdcSMTPIN_ADDED_MISSING#mx.google.com>
It may be that copying a message into gmail's imap store can result in a missing message id. Eg see https://productforums.google.com/forum/#!topic/gmail/w5kgRivwbIg (though that is very old, it shows that synchronising mail to a gmail IMAP account can have this result).
The basic rule of e-mail is that if an e-mail doesn't have a message id for whatever reason, one is created for it, which is what has happened here. Hence, when you reply, the made-up message id is used in the in-reply-to header.
I am using the nodemailer SMTP node module, a simple way to send emails through SMTP.
I am using this to send out text messages to various cell carriers, which will allow you to send a text message for free via email. An example email, which will text the Sprint phone (310) 987-6543:
3109876543#messaging.sprintpcs.com
These are all going out through one SMTP email address. This means all text replies from someone's phone will channel back to my email address.
A lot of people send these texts. I want to route the reply text back to the right person, using my own server side code.
The problem is distinguishing where the text came from. Ideally I could hide my own primary key ID for the sender in some sort of SMTP header, which would arrive at the recipients phone and come straight back in their text reply.
I don't really know anything about SMTP or email protocols - is this possible?
edit
As given in each of the answers below, in better email servers you can address to an email address as well as an extension to that address, such as:
// original email
sms-service#mysite.com
// extension
sms-service+a6e1087b#mysite.com
This would be a correct answer and would solve my problem, if only this worked in Exchange Server, which is the email server I am using. I don't know if there is a way to enable this or any other workaround.
Exchange server doesn't support plus addressing natively but it is possible do achieve this with a custom MTA, e.g. http://durdle.com/regexcatchall/
Source: https://serverfault.com/questions/195276/does-exchange-support-plussed-users-e-g-mynamespamexample-com-or-a-similar
Here are a few approaches that you might be able to use:
Unique from addresses
You could you use unique "from" or "reply-to" address for each each message (or for each recipient). For example "sms-service+a6e1087b#mysite.com". In that example, the hex would be a unique identifier that can be mapped to the conversation in your database.
Gmail supports such a scheme: emails targeted to [foo]+[whatever]#gmail.com will be delivered to [foo]#gmail.com and the receiver will know also get the [whatever] back. (more on this http://lifehacker.com/144397/instant-disposable-gmail-addresses)
Identify the user, not the conversation
One option is to not identify the conversation, but identify the phone number the reply comes back from instead. In many cases, the number from where the SMS reply was sent back from is enough to connect that message to the conversation. I'm sure the SMS gateway you're using will provide you with the number from where the reply came from.
Commercial SMS gateway
You can also opt to use a commercial SMS gateway (smsglobal.com comes to mind, though there are many others). These services have well defined APIs for sending a receiving replies. This naturally comes at a price, though. But along with the price, you do get more confidence as to the reliability of the service. A free SMS gateway might get shut down without much warning.
This common problem is traditionally implemented using the semantics of From: and Reply-To: headers of e-mails. This is in fact not (directly) related to SMTP, but rather to ARPA network text messages.
RFC 5322 is the current specification of these messages. It states:
When the "Reply-To:" field is present, it indicates the address(es) to which the author of the message suggests that replies be sent. In the absence of the "Reply-To:" field, replies SHOULD by default be sent to the mailbox(es) specified in the "From:" field unless otherwise specified by the person composing the reply.
So you simply need to encode the information you need into the Reply-To: header. Since you mention your own server-side code, you might be running an SMTP server yourself. In this case, you can put any address that will be routed to your SMTP server and encode the right person's address there.
Alternatively, if you rely on another SMTP server, you might want to use aliases. Postfix, and mail services such as GMail, typically allow address extensions, i.e. addresses of the form user+foo#domain.com, and you can encode the necessary information in the address extension.
For example, let's say that alice#domain.com wants to send a text to Sprint phone (310) 987-6543 through your service, you could send an e-mail to 3109876543#messaging.sprintpcs.com with:
Reply-To: alice%40domain.com#yourservice.com
Alternatively:
Reply-To: yourservice+alice%40domain.com#gmail.com
This will allow you to route the reply properly by looking at the destination of the reply e-mail.