Google, UAL Gaffe Underscores Need for Smarter Web Crawlers

Google and the Tribune are finger pointing, with Google saying the Sun-Sentinel should have included a dateline on the Tribune on the story, while the Tribune blamed Google's bot, adding that it asked Google to stop crawling its network of newspaper Web sites months go. Google denies this.

"The claim that the Tribune Company asked Google to stop crawling its newspaper Web sites is untrue," a Google spokesperson told me.

I turned to Search Engine Land's Danny Sullivan for clarity. He told me:

IDC's Susan Feldman had her own take on the matter, noting that the lack of a date for the article, which as Sullivan noted would have been added by humans, "tripped the whole train of events that eventually tripped up the automated trading programs." She said:

Feldman says the solution is to improve the newspaper site, the crawlers and the automated trading programs, as well as having a better understanding of the unforeseen consequences of getting the wrong information at the wrong time to the wrong people.

So clearly what we need are smarter algorithms and therefore smarter crawlers, not only from Google, but from the automatic trading brokers. We can't continue to have these gaffes because it will seriously disrupt Wall Street, the aorta that pumps the lifeblood through the country.

Look at how UAL is suffering from an article published six years ago. Imagine if something similar happened to Google or Microsoft.

That would be even more embarrassing, but maybe that's what it would take to galvanize the companies into improving their search algorithms.