Amazon, eBay Get Attached to SOAP Messaging Spec

It seems inevitable that when you put Microsoft, IBM and Sun (and HP and TIBCO and BEA, etc.) in the same room over and over again, youll end up with some porky specifications. Sure enough, the Simple Object Access Protocol (along with its numerous related appendages) has been getting bigger and more confusing every year.

All of those evolving specifications can be conveniently ignored if you have the luxury of defining the terms of the transaction. For example, if my fictional company, Monolithic Entity Inc., wants to exchange data with its suppliers, it can define its own terms. Maybe well use our own home-brewed XML with compression and just bypass SOAP entirely. Or, perhaps well use a variant of SOAP 1.0 that relies mostly on attachments being sent out-of-band (not in the SOAP package).

This is exactly the way many businesses have handled their Web services transactions—using the tools available in their own way and not waiting (perhaps years) for the committee-driven standards to make their way into draft form.

Resource Library:

But the stakes are different when you start creating public-facing Web services. Both eBay and Amazon, which are at the forefront of large-scale Web services deployments, have realized this. If you want 20,000-plus developers to play with your Web services, you need standards that interoperate. Your developers may be using Java or .Net or Perl or Python, and there may be multiple SOAP tool kits available for each language. Suddenly, committee-driven standards dont seem so bad—provided they interoperate.

In turn, those 20,000 developers may bring in 100,000 SOAP requests an hour (or minute) with their applications. Performance and scalability arent issues that can be handled ad hoc in these kinds of environments.

One hurdle that has only recently been put to rest (hopefully) is how to handle attachments with SOAP messaging. By attachments, I mean the same thing as e-mail attachments—a binary file such as a JPEG photo. Some people call these things "blobs," or "opaque data," but Ill just call them attachments.

Attachments turn out to be important to both eBay and Amazon as their sellers are constantly dealing with various types of binary information and need an efficient—and interoperable—way of sending and receiving them via SOAP. But attachments have also been one of the blurriest appendages of the SOAP standard and generally have been neither efficient nor interoperable.

Between 2000 and 2003 there were no less than three proposals to handle attachments to SOAP messages. HP and Microsoft started things off in late 2000 with the SOAP with Attachments draft, commonly referred to as "SwA." This was a simple specification that let SOAP messages refer to out-of-band attachments that were MIME-encoded (just like e-mail attachments). But MIME (Multipurpose Internet Mail Extension) attachments are just base64-encoded, which tends to increase the attachment size by 30 percent or more. Not good.

In 2002, Microsoft came up with WS-attachments, which replaced MIME encoding with a new compressed format called DIME (Direct Internet Message Encapsulation). DIME was quickly implemented in Microsofts tool kits, but not widely outside of Microsoft.

In 2003, Microsoft and others proposed a new version of SwA called Proposed Infoset Addendum to SOAP Messages with Attachments. Commonly know as "PASwA," it suggests how MIME encoding (and associated compression mechanisms) could be integrated into the SOAP messaging stream.

The latest, and by all accounts, final salvo, came in mid-2003. The Message Transmission Optimization Mechanism, or MTOM, which was co-authored by IBM, Canon, BEA, and Microsoft, takes the better parts of most of the previous attachments specs and provides some clear guidance for interoperability. In fact, its coverage is somewhat broader (note that the title no longer refers to attachments but to "optimization") as it allows for various transports and compression methods and allows the attachment to be contained within the SOAP packet. Importantly, the attachment is packaged as an XML infoset, which means that it can be handled as a regular XML object as opposed to something thats (possibly) non-XML compliant.

The W3C MTOM Candidate Recommendation was published on August 26th. Microsoft is expected to drop DIME support (which they offer in their SOAP tool kit) in favor of MTOM and that effectively makes it the de facto standard.

Three years isnt too bad for a committee-driven specification, and, to be fair, this was new territory for messaging— no one really knew the best solution. EBay and Amazon are also covering new ground with their public-facing, high-volume Web services, and that will drive more immediate demand for interoperable standards.

Check out eWEEK.coms Web Services Center for the latest news, reviews and analysis in programming environments and developer tools.