The Quality of Our Ignorance Is Rising

By Peter Coffee  |  Posted 2003-04-14 Print this article Print

Coffee: Networks, business intelligence and other emerging IT domains depend on statistical smarts.

In the past week, various people have been telling me about their initiatives in both enterprise data mining and ultrawideband (UWB) wireless networks. Besides growing momentum, these IT domains share a common demand for a statistical approach to system design: data mining with its emphasis on inference and significance, UWB and other shared-spectrum techniques with their sensitivity to worst-case behavior of random events. Statistics are themselves perhaps the worst-case example of a business or technical discipline thats both too rarely used and too often misused. As we build systems that are larger, that work in less controlled environments, and whose operation involves larger numbers of independent decision makers, its vital for IT professionals to get smarter about our use of statistical methods--and about the inferences we make from the numbers we collect. Telecommunication carriers have developed a huge body of statistical expertise on systems like the wired telephone network, although this may prove misleading if we try to apply that experience to the new world of always-on unwired access. The transactions of Web services will be, for the most part, less sensitive to split-second timing than voice communications (which have to satisfy our wetwares idea of what sounds natural). On the other hand, our models of when people want to connect, and for how long, have already failed to keep up with even the cumbersome mode of dial-up Internet access. Well have a lot to learn about the new "normal" behavior when people can access the network at will--or when devices and software agents can access each other directly at any time.
Our need for broader statistical expertise is compounded by the tilt toward business intelligence and decision support--rather than mere bookkeeping--as the mission of new enterprise IT projects. Just as we would not willingly build a system that tried to do accounting with 8-bit integers, we owe it to business unit managers to warn them when theyre asking for a system that will mislead them with impressive but ill-founded analyses.

Peter Coffee is Director of Platform Research at, where he serves as a liaison with the developer community to define the opportunity and clarify developers' technical requirements on the company's evolving Apex Platform. Peter previously spent 18 years with eWEEK (formerly PC Week), the national news magazine of enterprise technology practice, where he reviewed software development tools and methods and wrote regular columns on emerging technologies and professional community issues.Before he began writing full-time in 1989, Peter spent eleven years in technical and management positions at Exxon and The Aerospace Corporation, including management of the latter company's first desktop computing planning team and applied research in applications of artificial intelligence techniques. He holds an engineering degree from MIT and an MBA from Pepperdine University, he has held teaching appointments in computer science, business analytics and information systems management at Pepperdine, UCLA, and Chapman College.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel