Is the Bing It On Challenge a Little Off?

People prefer Bing’s results to Google’s, asserts Microsoft in its “Bing It On” ads. A Yale professor is challenging those claims.

The Bing It On challenge, which first appeared last year, loads Bing and Google search results side by side. Users, according to Microsoft, prefer Bing’s results—both are stripped of their branding—nearly 2:1 over Google’s in the comparison.

Ian Ayres, a Yale law professor, is calling those claims bogus.

In an Oct. 1 post on Freakonomics, Ayres wrote that the “the 2:1 claim seemed implausible” when he first viewed the ads a year ago. “I would have thought the search results of these competitors would be largely identical, and that it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1.”

“To make matters worse, Microsoft has refused to release the results of its comparison Website, BingItOn.com,” stated Ayres. Admitting to being “slightly annoyed” in discovering that the claim was based on a study of a mere 1,000 participants, he said that he enlisted Yale law students to run an experiment using a similar sample size and the BingItOn.com Website.

“We found that, to the contrary of Microsoft’s claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were ‘ties’),” reported Ayres. Secondary tests, which involved randomly assigned participants and a mix of popular, Bing-suggested and self-suggested search terms, failed to come close to Bing’s 2:1 advantage.

Matt Wallaert a behavioral psychologist at Bing, fired back in an Oct. 2 blog post and picked apart Ayres’ argument.

In terms of the sample size, Walleart noted that in his Freakonomics post, “Ayres then links to a paper he put together with his grad students, in which they also use a sample size of 1,000 people. They then subdivide the sample into thirds for different treatments condition and yet still manage to meet conventional statistical tests using their sample.”

“A 1,000-person, truly representative sample is actually fairly large. As a comparison, the Gallup poll on presidential approval is just 1,500 people,” stated Walleart.

As for Microsoft’s apparent unwillingness to share Bing It On site data, Wallaert said “we don’t release it because we don’t track it.” Citing strong privacy protections, he argued that “unlike in an experiment, where people give informed consent to having their results tracked and used,” BingItOn.com visitors do not agree to participate in research. It would, therefore, be an unethical breach for Microsoft to base a study on such data if it existed, he added.

Wallaert also dismissed the notion that Bing It On suggested searches sway results in Bing’s favor. He argued that Ayres also muddied the waters by jumping between two claims, specifically that participants “chose Bing Web search results over Google nearly 2:1 in blind comparison tests” and that they “prefer Bing over Google for the Web’s top searches.”