One of my favorite books from public policy was "The Moon and the Ghetto" by Richard R. Nelson. Basically, it asks the rhetorical question, "If we can put a man on the moon why cant we solve social problems in the ghetto?" The answer, counterintuitive to some, is that putting a man on the moon is a relatively easy problem to solve, compared with poverty and crime. (If this seems obvious to you now, it may not have in the 60s and 70s.)
Some people are asking the same question about spam. After years of effort and zillions of dollars invested by major players, why is spam still dominating the e-mail landscape? Over the last few years spam has settled in at around 85 to 90 percent of all e-mail, and theres every reason to believe it will stay there. I think the answer has a lot to do with the answer Nelson gave as to why solving the problems of the inner city, no matter how much money you throw at them, is harder than putting a man on the moon.
The world of e-mail, like the real world of politics, is by now long-established with entrenched interests. Anything done to "improve" the system, like any public policy initiative, creates winners and losers. And very often the winners and losers arent who policy makers had in mind.
Policy initiatives are not necessarily embraced by the people they are intended to protect. Think of how we react to the statement, "Hi, were from the government and were here to help." Why should "Were from the IETF and were here to make e-mail better for you" be any different? How do we know they wont make it even worse?
What got me started on this was a discussion on a group run by the IETFs ASRG (Anti-Spam Research Group). As one participant put it, "[s]pam is fundamentally a social problem. Technological measures are an arms race—a good stopgap, perhaps, but still a stopgap. It will require social change, and thats slow to happen, especially when its major opponent is laziness."
I dont entirely agree with the "laziness" crack, but clearly the answer to spam is not technology, and not even mostly technology. It would be relatively easy to design a new Internet mail system that would be free, or relatively free of spam, but such a system would require measures that are unacceptable to enough decision makers that a sufficient consensus will not be attained, at least not from this generation of Internet leaders.