Google, UAL Gaffe Underscores Need for Smarter Web Crawlers - Analysts Weigh In on the Google UAL Gaffe (
Page 2 of 2 )
Google and the Tribune are finger pointing, with Google saying the
Sun-Sentinel should have included a dateline on the Tribune on the
story, while the Tribune blamed Google's bot, adding that it asked
Google to stop crawling its network of newspaper Web sites months go.
Google denies this.
"The claim that the Tribune Company asked Google to stop crawling its
newspaper Web sites is untrue," a Google spokesperson told me.
I turned to Search Engine Land's Danny Sullivan for clarity. He told me:
Things like this have happened, as I've seen personally, but not in
such a big fashion. Better verification of dates, better working
between the news search engines and news sites would help, in
particular perhaps more dependence on feeds. But also, people doing the
basic amount of fact checking of a major story before feeding it into a
major wire service would have helped. That's where 90 percent of the
blame lies.
IDC's Susan Feldman had her own take on the matter, noting that the
lack of a date for the article, which as Sullivan noted would have been
added by humans, "tripped the whole train of events that eventually
tripped up the automated trading programs." She said:
While wary humans should be able to spot this kind of mistake,
computers can't unless they have been programmed to. And, to be honest,
apparently a lot of humans weren't wary enough to spot the lack of a
date either. So, human negligence kicked up the ranking of the article
as the very human rumor mill kicked in. This has been happening ever
since people started talking to each other. Think about the Teapot Dome
scandal, or the War of the Worlds. The problem with any automated
approach to processing information is that computers follow the rules
that are set by humans. If the rule to check the date or kick out a
document for human scrutiny is not in the rule base, then the computer
processes a document as if it is current, and that triggers alerts, and
trading problems.
Feldman says the solution is to improve the newspaper site, the
crawlers and the automated trading programs, as well as having a better
understanding of the unforeseen consequences of getting the wrong
information at the wrong time to the wrong people.
So clearly what we need are smarter algorithms and therefore smarter
crawlers, not only from Google, but from the automatic trading brokers.
We can't continue to have these gaffes because it will seriously
disrupt Wall Street, the aorta that pumps the lifeblood through the
country.
Look at how UAL is suffering from an article published six years ago.
Imagine if something similar happened to Google or Microsoft.
That would be even more embarrassing, but maybe that's what it would
take to galvanize the companies into improving their search algorithms.
Indexing & Search Engine News & Reviews 
