Dealing With the RSS
Bandwidth Problem"> SourceForge.net and its news cousin Slashdot.org are on the lookout for overly zealous newsreaders as a way to combat bandwidth hogging by RSS. Slashdot.org has an aggressive policy where it will ban newsreaders that check for its feeds more than once every half hour. SourceForge.net, on the other hand, watches RSS traffic as part of its overall check for abuse. If any IP address is checking too many pages in too short of a time, it could be banned. McGovern declined to offer details on the threshold for banning but said the problem is not RSS itself."We like RSS feeds because theyre a great way to get our content out to more sites and get more traffic to our sites," McGovern said. Winer also suggested that one component of the RSS 2.0 specification already allows better cooperation between publishers and newsreaders. Called "time to live," or ttl, the element lets a content publisher tell a newsreader how often it should check for fresh content, Winer explained. However, the ttl element is based on the honor system, Winer said. For it to be successful, a newsreader must abide by the time-interval set in the ttl tag. Winer said one way to enforce it could be for a publisher to give newsreaders a period of time, say 60 days, in which to support the interval or face being banned. While a useful tool, ttl has other limitations, Reinacker said. It only covers one XML syndication specification, RSS 2.0, and not the other RSS varieties or the Atom format. Another way feed publishers can limit too much polling is the use of HTTP caching headers, where sites can notify readers whether any new content has been published since the last check, Reinacker said. NewsGator, and most major newsreaders, support HTTP caching headers, he said. NewsGator also sets updating by default at every hour but lets users change the setting. Reinacker said that newsreaders have to be careful about limiting users ability to poll more frequently for updates, or they could undermine the value of RSS. Some enterprises, for example, are using RSS for real-time notifications. "Publishers are paying a bandwidth cost right now, but in doing so theyre delivering content in a way that hasnt been done before," Reinacker said. "They are able to notify users of changes to their site as opposed to waiting for them to come to their site." Winer agreed that publishers have to expect to pay some cost for the extra bandwidth of RSS traffic. While he doesnt think that most blogs and publishers are facing an undue bandwidth burden, he said that the full industry does need to discuss a more collaborative solution. "Its basically when this industry is ready to face these issues that its going to happen," Winer said. Read more here about some enterprises are eyeing RSS for content distribution. Until then, perhaps those most effected by RSS bandwidth hogging wont be large Web sites such as MSDN but smaller bloggers who develop a decent following, such as Gary Lawrence Murphy. Murphy, who publishes the blog Teledyn, last November began noticing traffic spikes from his RSS feed. It reached a point where he was exceeding his ISPs daily bandwidth limit under his hosting plan. He also noticed that despite using HTTP caching, many of the implementations of newsreaders didnt follow the specification correctly, leading him to both "break" the way his feed delivered its time stamp and to pare it down by removing HTML and limiting posts to 180 characters. It stemmed the bandwidth tide, but Murphy said it should serve as a warning to other smaller blogs like his. "Any site that becomes popular is going to be killed by their RSS," Murphy said. Check out eWEEK.coms Messaging & Collaboration Center at http://messaging.eweek.com for more on IM and other collaboration technologies.
What is needed is better caching of news feeds so that they are not constantly being pulled from the single server of the publisher, he said.