One of the main obstacles is managing mined data and keeping sight of the purpose for collecting it. Usama Fayyad, chairman and co-founder of Revenue Science Inc., said he makes a "mess" whenever working on a data mining project, leaving "a trail of droppings thats of Biblical proportions." Within two to three days of initiating a project, it becomes easy to lose sight of the purpose and goals, he said. Another major obstacle, in Fayyads view, is a disconnect between the way data is represented in data stores and the way mining technologies work. Revenue Science, which changed its name in June from digiMine Inc., is based in Belleview, Wash.Despite their recognition of the hurdles that lie ahead for data mining, proponents largely remain optimistic about its long-term prospects. "Headlines have given data mining a bad name," quipped Gregory Piatetsky-Shapiro, president of KD Nuggets, noting recent news reports of a U.S. Senate vote to block funding for the Pentagons Terrorism Information Awareness project. "But even if the Senate bans data mining, remember we will still have knowledge discovery."
The publics concern about protecting privacy remains a key obstacle, and researchers increasingly see it as a problem that they must address themselves. Calling privacy a "do-or-die" issue, Rakesh Agrawal, a researcher at IBM Almaden Research Center near San Francisco, said that developers must accept responsibility for the technology they are creating. Additionally, new methods must be developed to safeguard against false positives, or inaccurate patterns in data, and there remains much work to be done in that field, he said.