Intelligent Data Processing Is No Game

By Peter Coffee  |  Posted 2002-04-22 Print this article Print

Peter Coffee: Competitive pressure demands that the commercial sector now take a leadership role in developing exotic new algorithms, as well as feasible implementations of new ways to deal with data.

Programmers and hardware, taken together, form a culture that hasnt changed since the Space War game debuted in 1960 at MIT. That culture wants to take a small amount of input and do interesting tricks to produce amazing output, and its been highly successful in following that inclination: Computer games have been the most recession-resistant segment of the business. The sexy side of IT has always been more focused on computation than on data. Computer chess, once almost the definition of leading-edge computer science, only keeps track of 64 values at a time: What piece, if any, is on each square of the board? The rest is thinking, or the machine equivalent thereof, and its a lot of fun.
With instructions-per-second as the resulting measure of success, its no surprise that our IT infrastructure has grown up to be really good at various forms of arithmetic—but not so good at remembering what it did yesterday, or even five minutes ago. Weve built an enormous absent-minded professor, and only lately have we begun to talk about giving it a more durable and useful memory in the form of query-oriented file system technologies.
True, the commercial sector has enjoyed the resulting trajectory of technology: Unthinkable computational power, at equally unthinkable cost, enters the market in relatively price-insensitive realms such as defense and financial services, then works its way into mass-market applications with the necessary programming models and tools already well debugged. Microprocessor economies combine with mainframe sophistication to give us steam-engine power at a AAA-battery price. Very nice. What a shock, then, to find that competitive pressure demands that the commercial sector now take a leadership role in developing exotic new algorithms, as well as feasible implementations of new ways to deal with data. • Phrases such as "affinity analysis" and "sequence analysis" appear in press releases from companies such as Teradata, describing new facilities for predicting buyer behavior across product lines and over time. • Research into color perception leads to new ways of sorting colors and shades, streamlining inventory management for textile and apparel manufacturers. • Data-mining tools identify drug interactions with smaller numbers of experimental trials, speeding the delivery of pharmaceutical products to market. Its a combination of very abstract science and very boring storage and retrieval of information. The result is an interesting tension between two opposite trends. The technical risks are higher: We have to take on tasks before we know that they can be done at all, instead of just trying to do them in more cost-effective ways. At the same time, the engineering side of IT has become much more exacting. We have to deal with huge volumes of data, unforgiving levels of expected reliability, and a level of constant security challenge that makes Space War look like...well, like a game. E-mail eWEEK Technology Editor Peter Coffee
Peter Coffee is Director of Platform Research at, where he serves as a liaison with the developer community to define the opportunity and clarify developers' technical requirements on the company's evolving Apex Platform. Peter previously spent 18 years with eWEEK (formerly PC Week), the national news magazine of enterprise technology practice, where he reviewed software development tools and methods and wrote regular columns on emerging technologies and professional community issues.Before he began writing full-time in 1989, Peter spent eleven years in technical and management positions at Exxon and The Aerospace Corporation, including management of the latter company's first desktop computing planning team and applied research in applications of artificial intelligence techniques. He holds an engineering degree from MIT and an MBA from Pepperdine University, he has held teaching appointments in computer science, business analytics and information systems management at Pepperdine, UCLA, and Chapman College.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel