SMTP Authentication Update

They were heady days back in 2004 when the industry tried to reach consensus on a standard for SMTP authentication. The MARID (MTA Authentication Records In DNS) standards effort went down, partly in the flames of a political fight over patents with Microsoft, but the larger truth of the matter was that changes to heavily used standards like SMTP are bound to generate resistance from numerous sources, many of them with valid concerns.

But changes can come slowly, one site at a time over a period of several years. And sometimes they can be made more effectively when pushed by a big, strategic company than a standards body.

A little catchup on the terminology:

• SPF (Sender Policy Framework) is a standard for authenticating messages by checking special records stored in the DNS for the domain that the message claims to be from. But it only checks the message envelope (the MAIL FROM statement in SMTP) and thus misses quite a bit of SMTP fraud.
• SIDF (Sender ID Framework) is Microsofts approach to authenticating the entire message. The heart of it is the determination of a PRA (Purportedly Responsible Address) for the message and checking the domain of that address to see if the messagess sending agent is authorized. SIDF incorporates SPF as a first-stage test and stores further records in DNS in XML format.
• DKIM (DomainKeys Identified Mail) is the merger of two similar standards from Yahoo and Cisco. It too uses DNS-based records to authenticate a messages sender as authorized for the domain of purported sender but uses a public key cryptography scheme for the authentication, removing the need to track actual IP addresses of servers.

While the MARID dust was still settling I dismissed the possibility of any real success in the market for SIDF. Throughout the relatively short process it became clear that everyone, even people at Microsoft, felt that, in the long term, a crypto-based solution was better for a variety of reasons. Some argued that the alternatives (mostly what became SIDF) were quicker and easier to implement and that they therefore were appropriate as a transitional technology. Microsoft did go ahead and implement SIDF in their Hotmail service and they have some industry support

/zimages/3/28571.gifOften the best way to thwart spamming and phishing is to take down the domain names involved in it, but this is a hard thing to do under current rules. Click here to read more.

But in the meantime , others decided to dive in to the DKIM pool and swim. There are some big mail providers on the list, Yahoo of course being the biggest and AOL is there too. Most of the others are actually software companies, important ones like Sendmail and IBM and qmail.org (I know, not really a company). Notice also a fair amount of overlap with the SIDF list. But how many of the smaller ISPs and hosting services support DKIM or SIDF? Not many I think.

SPF support is much more widespread. Setting up SPF is easy and its one of those standards that everyone loves and professes to admire, even though it does far less than most people think it does. Consider this Slashdot article and discussion about promotion of DMIM by PayPal. Of course, as usual, the Slashdot writeup overstates the actual claims in the article, but whats interesting is in the discussions how participants who claim to be administering mail servers for real organizations.

There is a general lack of understanding that SPF deals only with envelopes, a lack of understanding that all these systems are designed to work only at the MTA level, and a general assumption that SIDF is technically unsound (with no elaboration of course).

Next page: SIDF and the numbers

SIDF and the numbers

There are technical difficulties with these standards, of course, especially with forwarding accounts. Many universities, for example, allow alumni to keep an e-mail address and forward the mail on to their own personal account. Many commercial services, like Spamex, work the same way. If I send mail “from” my usndh.edu (University of Southern North Dakota at Hoople) address but send it through my comcast.net mail server, then its going to fail any reasonable authentication check. There really isnt a good solution to this.

But all that aside, in the years since the standards positions hardened, have they had any effect? Microsoft argues that they have. Last week at the AOTS conference in Boston they announced that SIDF blocks 20 million fraudulent messages. Now when Microsoft says “Sender ID” in cases like this they often mean “SPF” which is incorporated as part of SIDF. In fact, its probably complete overlap now.

Its possible, for example, for a spammer to put SPF records on their junk domain and use a proper envelope when sending it, and then to use FROM: and SENDER: headers in the message with “microsoft.com” in them. Sender ID would detect this, although the SPF test would pass. So if youre sending out spam and spoofing FROM: addresses to do it, make sure not to spoof a Sender ID domain.

/zimages/3/28571.gifSome people think the answer is to throw out the old Internet and build a new one. Click here to read more on the idea.

Microsoft throws out a lot of other numbers:

• 98% of phishing messages are caught by Sender ID
• 90% of e-mail marketers have implemented Sender ID
• E-mail marketers who implement SIDF and have a positive reputation have “up to 85% fewer messages mistakenly marked as spam.”
• 3.8 billion out of the 4.5 billion messages sent to Hotmail every day are spam.
• 300 million of those messages to Hotmail come from domains with SPF records.

The message is as much—maybe more—to e-mail marketers than anyone else, but the point is that SIDF and SPF have substantial implementation and are working to improve the e-mail system. I can believe this. I can also believe that Microsofts ability to promote SIDF through the massive Exchange Server community has, and will continue to have, a positive effect on adoption by business.

So about 2 and a half years out from the MARID collapse, SMTP authentication is progressing as a real-world tool but slowly. Very large mail providers and senders are using it and benefiting from it, Some day a tipping point will arrive where it will be realized, generally, that you need to implement authentication and deal with any technical problems that result, but were not there yet.

Editors Note: This story was updated to correct an error in the description of SPF.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

/zimages/3/28571.gifCheck out eWEEK.coms for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at eWEEK.com Security Center Editor Larry Seltzers Weblog.

More from Larry Seltzer