/home/armin

Condensed Guide to Mail Delivery

2018-12-13

This guide serves as a refresher and aims to help you understand all the moving parts during an email message delivery to a mail server using SMTP and collection by a mail user agent (i.e. Thunderbird) using IMAP.

Format of an Email Message

Email messages are basic text files. They are composed out of two parts:

All lines in the message are terminated with CRLF. A line with a single dot on it signifies the end of the message (i.e. EOF).

A very plain email message looks like this:

From: Diana Prince <diana@amazonian.com>\r\n
To: Bruce Wayne <bruce@wayne.com>\r\n
Date: Tue, 27 Nov 2018 16:09:31 +0100\r\n
Subject: Batmobile insurance\r\n
\r\n
Dear Bruce,\r\n
\r\n
Seriously, how do you insure that tank of a car you drive?\r\n
\r\n
Love,\r\n
Diana\r\n
.\r\n

Mail Delivery Process

If we take the example message above, the process to deliver it from diana@amazonian.com to bruce@wayne.com would be:

  1. Diana’s mail client:
    • resolves the IP of amazonian.com’s mail server (via DNS/MX records),
    • connects, authenticates and provides the message to be delivered (via SMTP)
  2. amazonian.com’s mailserver resolves the IP of wayne.com’s mail server, connects and provides the email message
  3. wayne.com’s mailserver will then put the mail into Bruce’s mailbox
  4. next time Bruce opens his mail client, it connects to wayne.com’s mailserver, authenticates and downloads the email

MX Record and Mail Server IP Lookup

IP discovery for mail servers is done via mail exchanger (MX) records.

These DNS records indicate the machines responsible for handling a domain’s emails. For example:

$ dig amazonian.com mx

amazonian.com.		3600	IN	MX	10 mail1.amazonian.com.
amazonian.com.		3600	IN	MX	20 mail2.amazonian.com.
amazonian.com.		3600	IN	MX	30 mail3.amazonian.com.

indicates that @amazonian.com email is handled by mail{1,2,3}.amazonian.com and in that order since the 10, 20 and 30 are MX priority values.

Given those, we would do a second lookup for A or AAAA records for mail{1,2,3}.amazonian.com and proceed to connect to those.

$ dig mail1.amazonian.com a # (for AAAA records use "dig domain.com aaaa")

mail1.amazonian.com.		3600	IN	A	10.20.30.40

If you have only one machine you can configure the MX record to simply point to the same domain:

example.com.		3600	IN	MX	10 example.com.     # @example.com mail is handled by example.com
example.com.		3600	IN	A	1.2.3.4             # which has the IP address 1.2.3.4

SMTP Protocol

The SMTP protocol has three well known TCP ports:

In short, use port 465 for sending email, if that is not available use 587. Either way, make sure you’re sending over an encrypted connection.

Note that this does not guarantee security as your email might be sent unencrypted between other servers. If you need that type of security, use OpenPGP encryption.

Without a doubt, you've already seen the word `STARTTLS` somewhere when setting up an email client like Thunderbird. When using `STARTTLS` the connection will initially begin unencrypted and after the `STARTTLS` command is issued will be upgraded to an encrypted one. This means that the initial part of the session will be done over an unencrypted connection and (with poorly configured mail servers) might send your email unencrypted if the encryption handshake (started after the `STARTTLS` command) fails. For this reason, always prefer port 465 or encrypted-by-default connections.

Example SMTP Session

SMTP uses base64 encoding for authentication; you can generate that easily enough:

echo -n "username" | base64
echo -n "password" | base64

Below is an example SMTP session. I’ve prefixed lines you type in with “-->” and added a bit more vertical spacing for readability.

$ openssl s_client -connect amazonian.com:465

(... a wall of text about the TLS negotiation details ...)
250 HELP

--> EHLO Diana
250-amazonian.com Hello Diana [1.2.3.4], pleased to meet you
250-8BITMIME
250-ENHANCEDSTATUSCODES
250-SIZE 36700160
250-DSN
250-AUTH PLAIN LOGIN
250 HELP

--> AUTH LOGIN
334 VXNlcm5hbWU6

--> <your base64 encoded username>
334 UGFzc3dvcmQ6

--> <your base64 encoded password>
235 2.0.0: Authentication succeeded

--> MAIL FROM:<diana@amazonian.com>
250 2.1.0 <diana@amazonian.com>... Sender ok

--> RCPT TO:<bruce@wayne.com>
250 2.1.5 <bruce@wayne.com>... Recipient ok

--> DATA
354 Enter mail, end with "." on a line by itself
(...
    at this point we would enter the email message 
    from the above example with a ".\r\n" at the end
...)
S: 250 2.0.0 k7TKIBYb024731 Message accepted for delivery

--> QUIT
221 2.0.0 example.org closing connection 

You might’ve noticed by now that the sender and recipient have been specified twice:

The MAIL-FROM and RCPT-TO from the SMTP session are called an “SMTP envelope” and they are what mail servers use to identify the sender and recipient. For more info about the envelope read through this article.

The From and To are used by your MUA (i.e. Thunderbird) to display the sender and recipient.

At this point, our mail client has delivered the message to amazonian.com’s mail server which is ready to relay the message to wayne.com.

Consider the actors involved in the above transaction:

<Diana's Laptop>  --> connected to port 587 on -->  <amazonian.com mailserver>
IP: 1.1.1.1                                         IP: 1.2.3.4

Diana’s laptop will have whatever IP her ISP assigned her and her IP will change depending on where she’s connecting from (smartphone, laptop, home computer, work computer) meaning that there is absolutely no way for mail1.amazonian.com to verify it is indeed Diana just via the IP.

That is why we had to authenticate ourselves after connecting. However, when two mail servers are communicating there are a number of ways for them to verify that it is indeed the correct server talking to them. This verification process is done via:

Which is why they do not require authentication.

The amazonian.com mail server will now repeat the above SMTP session with wayne.com’s mail server (lets call it mail.wayne.com) with a few key differences:

One important thing to keep in mind when setting up your mail server is that you should configure it to only allow specific senders and always, always, always require authentication when sending email from any other machine (i.e. from another IP address).

Not doing so means you are setting up an open relay which allows any spammer to send mail through your machine.

Mail Server Authentication

As mentioned above, when two mail servers deliver mail between themselves they don’t use authentication but other mechanisms. There are three very important things you should take the time to set up:

DMARC is out of scope of this article but you can use any of the generators on the web to set that up. It is basically a TXT record on your domain which tells others how and where to report problems with emails from your domain.

Sender Policy Framework (SPF)

SPF is a special TXT DNS record that specifies a list of server IPs allowed to send email for a domain. It uses the TXT record type. An example record looks like this:

example.com.		3600	IN	TXT	"v=spf1 a mx -all"

The above value, decyphered, means:

You should really set up this record as it’s the first and most basic SPAM protection that many servers do. For a complete explanation, read the Wikipedia article on SPF.

DomainKeys Identified Mail (DKIM)

DKIM is a mechanism which uses a public/private keypair to sign and verify emails.

You keep the private key on your server and use it to sign emails and publish your public key in a special TXT record on the domain. Receiving mail servers can then find your public key using the MAIL-FROM value from the SMTP session and verify the email signature.

The signatures are simply additional header fields computed using the body of the mail message.

The full process for verifying email via DKIM is:

  1. publish our public key in a DNS TXT record
  2. the sending mail server signs the outgoing email using the private key in the DKIM-Signature header
  3. the receiving mail server fetches our public key via DNS and uses it to verify the email signature from the DKIM-Signature header
  4. if correct, the mail is from us, otherwise it’s rejected

The DKIM DNS record is normally in the form of <identifier>._dkim.yourdomain.com where the identifier is whatever you choose. Here’s an example using mail as the identifier:

mail._domainkey.example.com. 1799 IN	TXT	"k=rsa; t=s; p=<my very long public key>"

IMAP Protocol

IMAP is a protocol used to fetch and organize mail on a server.

That email message from the beginning of this guide will ultimately be stored somewhere accessible via IMAP.

It could be saved as a file inside a Maildir, appended to an mbox file, stored in a database, and so on. IMAP abstracts this away from your mail user agent which will use simple IMAP commands to fetch that email.

IMAP uses these ports:

You should without exception always use port 993.

IMAP Session Example

In the IMAP protocol each command you send to the server should be prefixed with a simple identifier. Every response you get will also have this prefix so you know what the server is responding to. This is because some commands work asynchronously and you might get the replies out-of-order/mixed with other replies.

In the example IMAP session below, all the commands prefixed with --> CMD<N> are the ones you type in (excluding the --> ). The CMD<N> is the identifier but you can use anything else (i.e. a1 works as well).

$ openssl s_client -connect mail.wayne.com:993 -crlf

(... a wall of text about the TLS negotiation details ...)
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE AUTH=PLAIN] Dovecot ready.

--> CMD1 LOGIN "bruce@wayne.com" "<bruce's password>"
CMD1 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY SPECIAL-USE] Logged in

--> CMD2 LIST "" *
* LIST (\HasNoChildren \Trash) "." Trash
* LIST (\HasNoChildren \Sent) "." Sent
* LIST (\HasNoChildren \Archive) "." Archive
* LIST (\HasNoChildren \Drafts) "." Drafts
* LIST (\HasNoChildren \Junk) "." Junk
* LIST (\HasNoChildren) "." INBOX
CMD2 OK List completed (0.004 + 0.000 + 0.004 secs).

--> CMD3 STATUS INBOX (MESSAGES)
* STATUS INBOX (MESSAGES 1)
CMD3 OK Status completed (0.006 + 0.000 + 0.005 secs).

--> CMD4 SELECT INBOX
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft Old)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft Old \*)] Flags permitted.
* 1 EXISTS
* 0 RECENT
* OK [UNSEEN 1] First unseen.
* OK [UIDVALIDITY 1543324688] UIDs valid
* OK [UIDNEXT 24] Predicted next UID
CMD4 OK [READ-WRITE] Select completed (0.003 + 0.000 + 0.002 secs).

--> CMD5 FETCH 1:* FLAGS
* 1 FETCH (FLAGS ())
CMD5 OK Fetch completed (0.003 + 0.000 + 0.002 secs).

--> CMD6 FETCH 1 BODY[]
* 1 FETCH (FLAGS (\Seen) BODY[] {2845}
(...
    here you will see the entire email file
...)
CMD6 OK Fetch completed (0.018 + 0.000 + 0.017 secs).

--> CMD7 LOGOUT
* BYE Logging out
CMD7 OK Logout completed (0.002 + 0.000 + 0.001 secs).
closed

What we’ve done above is:

Like regular email, IMAP uses \r\n for newlines hence the -crlf in the openssl command.