Programmers and hardware, taken together, form a culture that hasnt changed since the Space War game debuted in 1960 at MIT. That culture wants to take a small amount of input and do interesting tricks to produce amazing output, and its been highly successful in following that inclination: Computer games have been the most recession-resistant segment of the business.
The sexy side of IT has always been more focused on computation than on data. Computer chess, once almost the definition of leading-edge computer science, only keeps track of 64 values at a time: What piece, if any, is on each square of the board? The rest is thinking, or the machine equivalent thereof, and its a lot of fun.
With instructions-per-second as the resulting measure of success, its no surprise that our IT infrastructure has grown up to be really good at various forms of arithmetic—but not so good at remembering what it did yesterday, or even five minutes ago. Weve built an enormous absent-minded professor, and only lately have we begun to talk about giving it a more durable and useful memory in the form of query-oriented file system technologies.
True, the commercial sector has enjoyed the resulting trajectory of technology: Unthinkable computational power, at equally unthinkable cost, enters the market in relatively price-insensitive realms such as defense and financial services, then works its way into mass-market applications with the necessary programming models and tools already well debugged. Microprocessor economies combine with mainframe sophistication to give us steam-engine power at a AAA-battery price. Very nice.
What a shock, then, to find that competitive pressure demands that the commercial sector now take a leadership role in developing exotic new algorithms, as well as feasible implementations of new ways to deal with data.
• Phrases such as “affinity analysis” and “sequence analysis” appear in press releases from companies such as Teradata, describing new facilities for predicting buyer behavior across product lines and over time.
• Research into color perception leads to new ways of sorting colors and shades, streamlining inventory management for textile and apparel manufacturers.
• Data-mining tools identify drug interactions with smaller numbers of experimental trials, speeding the delivery of pharmaceutical products to market.
Its a combination of very abstract science and very boring storage and retrieval of information. The result is an interesting tension between two opposite trends.
The technical risks are higher: We have to take on tasks before we know that they can be done at all, instead of just trying to do them in more cost-effective ways.
At the same time, the engineering side of IT has become much more exacting. We have to deal with huge volumes of data, unforgiving levels of expected reliability, and a level of constant security challenge that makes Space War look like…well, like a game.
