Crowdsourcing Pitfalls

By Lukas Biewald  |  Posted 2010-09-29 Print this article Print

Crowdsourcing pitfalls

So, what are the downsides of crowdsourcing? One of the pitfalls that catch businesses by surprise is privacy concerns. Netflix crowdsourced the improvement of its movie recommendation algorithm. The algorithm is critical to its business effectively helping users find new movies-something crucial in order to retain its users. The contest: improve the existing ranking algorithm by 10 percent and win $1 million.

The contest attracted scientists from around the world. The Netflix data was so rich and deep that it promised to yield broader insights about data analysis. Eventually, the scientists formed teams which collaborated to win the prize. Unfortunately, Netflix's attempt to run the contest again was cancelled due to a lawsuit that challenged the release of user preference data. So, crowdsourcing does not always fit the data, and there may be special concerns around private information.

Another issue that comes up is unexpected bias. GalaxyZoo is a Website where amateur astronomers can label pictures of galaxies taken by telescopes. They created the largest database of galaxies ever assembled. The project led to many academic papers and discoveries. However, one finding was perplexing: there were more clockwise spiral galaxies than counterclockwise spiral galaxies. Astronomers wondered if there some galactic Coriolis effect that makes galaxies spiral clockwise. After running a mirrored set of the same images, researchers found that users had a subtle bias towards labeling ambiguous spirals galaxies as clockwise.

Lukas Biewald is the co-founder and CEO of CrowdFlower. Before co-founding CrowdFlower, Lukas was a senior scientist and manager within the ranking and management team at Powerset, a natural language search technology company later acquired by Microsoft in 2008. Lukas has also led the search relevance team for Yahoo Japan. He holds a Bachelor's degree in Mathematics and a Master's degree in Computer Science, both from Stanford University. Recently, Lukas won the Netexplorateur Award for Give Work, a collaboration with Samasource that brings digital work to refugees worldwide. He can be reached at

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel