eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Disruptive technologies are born for all sorts of reasons—good ideas, market pressure, economic opportunity, and sometimes just plain luck. Many of todays disruptive leaders only emerged when combined with other seemingly unrelated inventions. Wi-Fi and broadband (DSL and cable but not satellite) have prospered in a mutually symbiotic fashion. So too have weblogs and RSS.
For newbies, RSS feeds are XML text files generated by blogs, websites and other web servers that desktop clients—called RSS Readers or Weblog Readers—download on a set schedule, usually once an hour.
As RSS gains momentum, it begins to strain the boundaries of its current infrastructure. Feeds are increasingly containing full text, graphics, and even multimedia files. Strict constructionists are bemoaning the trend, suggesting that syndication is all about signaling rather than transporting. Those of us whove moved to RSS as the gateway to as much information as we can filter reject that notion.
It reminds me of the dawn of the Web, when Internet academics and geeks bemoaned the influx of the point-and-click crowd. Or the Notes administrator priesthood who kept the keys to creating new applications from us lowly users. Weblogs are just the most recent instantiation of that revolution, where users gained control of their pages, data, and voices.
Now we have the same argument from the entrenched Web crowd. RSS should just provide the abstract, the notification, the announcement of new content. But the ultimate target is the Web page, where eyeballs equal dollars. Its the old guard protecting last years business model – like the RIAA desperately poking a finger in the p2p dike or the MPAA outlawing the distribution of DVDs for Academy Award consideration.
Whos the Jack Valenti of this production? I hope its not Dave Winer, RSS founding father and author of one of the most popular RSS feeds, Scripting News. Notice I say RSS feed, not weblog. Since the day I first adopted an RSS reader (Radio Userland on the PC, NetNewsWire on the Mac), I have consumed Daves site through RSS.
Dave has always included the full text and graphics in his feed, but recently hes limited the entries to the current day. Im not sure why that is, but perhaps its because it limits the size and bandwidth usage of the RSS file he serves to his subscribers.
Note that my copy of NetNewsWire is set to download at the most rapid interval possible—30 minutes. Thats still not fast enough for me, relying as I do on RSS for timely notification of important posts. But bandwidth costs on popular sites like Daves make this kind of polling economically prohibitive. Or do they?
Heres how Dave put it one morning last week: “Now, should an aggregator be polling every 30 minutes? The convention early on was no more than once an hour. But newer aggregators either never heard of the convention or chose to ignore it. Some aggregators let the users scan whenever they want. Please dont do that. Once an hour is enough. Otherwise bandwidth bills wont scale.”
To be fair, Dave goes on to say there are good ways to optimize polling, but that progress in the RSS community is hard to come by. Hes probably right – but Im not dissuaded by that daunting prospect. RSS has forever altered the way I acquire information, and its disruptive quality can surely bond with another such technology to conquer this bottleneck.
Next page: Revolutionary peer-to-peer program to the rescue!
Revolutionary Peer
-to-Peer Program Could Solve Problem”>
One such candidate is peer-to-peer, as resurrected in the form of Bram Cohens BitTorrent. Its an elegant protocol for distributing files, one that takes advantage of “the unused upload capacity of your customers.” BitTorrent breaks up files into shards that are uploaded around the network as the file is downloaded by multiple clients. The more popular a file, the more endpoints exist. You download a file with BitTorrent by simultaneously collecting shards, assembling them together locally as they arrive.
Map this to RSS feeds: the more popular the feed, the more nodes on the network serving pieces of the feed. That would allow rapid downloads by many users by distributing the data across multiple sites. Its a digital Robin Hood, redistributing the wealth from the server to a network of peers. BitTorrent does cryptographic hashing of all data, so feed owners can be confident the file reaches its target unchanged.
But theres even more to this disruptive alliance: a small amount of special code known as a tracker sits inside the host Web site and emits information to help other downloaders find each other. As Bram Cohen describes: “[Trackers] speak a very simple protocol layered on top of HTTP in which a downloader sends information about what file its downloading, what port its listening on, and similar information, and the tracker responds with a list of contact information for peers which are downloading the same file.”
So youve got a list of peers connected via known ports, a trusted group of RSS feed subscribers, who can marshall their resources for additional economic benefit. That could take the form of an affinity group marketing their attention to an advertiser or political cause, a secure pool of computing resources for distributing confidential information, and a pathway for signaling information about new content on that particular subnetwork.
Even if none of this were to be leveraged, the combined polling of the group, if staggered by one minute intervals, would monitor a feed even though each client only polled once per hour. But once you were on the BitTorrent feed, you could shut off the polling after the first connection.
Next page: Two great disruptive technologies that taste great together.
Server
-Based Distribution Solutions”>
There are server solutions to the polling problem, including Dave Winers sub-element documented on the RSS 2.0 site maintained by Harvards Berkman Center. Or you could use a service like Dave Sifrys Technorati to inform you of updates as they occur. An email or instant message alert might be too disruptive for some, but building the service into the reader (or information router as I prefer to call it) could prove popular.
Sifry suggests how this might work:
“In order to facilitate an event-based mechanism to eliminate inefficient polling of RSS feeds, the subscribers information router can request a notification via XMPP (or other notification protocol) whenever the feed is updated. Technorati is notified of the feed update via XML-RPC ping call as the normal result of the CMS update procedure, and then proceeds to notify all of the information routers that the new content is available, along with the BitTorrent URL – then the information routers collectively download and distribute the new content via BitTorrent. The original feed is only downloaded when it is seeding the BitTorrent peers, and the subscribers get to take advantage of the highly available fresh content.”
There we have it: two disruptive technologies coming together to produce an outcome thats win-win for each other. For BitTorrents peer-to-peer part, the fact that its author controls RSS content takes that bit of digital McCarthyism off the table. And as for RSS, this solution requires no changes to the specification whatsoever. Its up to content providers to support it on the server (no problem, since it cuts costs) and aggregator/reader/routers to support it on the client (those who dont see the benefits aint paying attention).
As for me, Im off to listen to some Chris Lydon RSS audio, downloaded via Dave Winers Enclosure extension and copied to my Microsoft DRM-free iPod. Hi ho hi ho disruptively we go…
Discuss This in the eWEEK Forum
eWEEK.com Messaging & Collaboration Center Editor Steve Gillmor can be reached at [email protected].