Data Integration Is ITs Frontier

Even free-trade advocates are starting to tiptoe around the subject of tech-job outsourcing. In a presidential election year, its dangerous to suggest that the overseas export of any voters job is a long-term gain for the U.S. economy.

But for IT pros, the important trend is not the loss of jobs in slinging code or designing chips or providing generic technical support. The key to job security is certainly not to be found in learning a new programming language or mastering a new set of APIs.

Instead, I suggest that data integration into real-time applications is the task that will get the most leverage from new technologies, such as satellite location services or RFID tags. This will also pose the greatest challenge, only in part because of the sheer volume of data that these innovations introduce into enterprise systems.

Its plausible, therefore, that jobs in database administration will grow by 60 percent before the decade is out, according to the Bureau of Labor Statistics; the nature of that DBA task is evolving as well. The next-generation database guru wont be merely designing relational structures, or maintaining data capture and grooming procedures. Instead, tomorrows data specialist will be actively working with business units to decide which data need to be delivered to which workers, under what time constraints, to create distinctive competitive advantage.

The resulting operational knowledge will be much too valuable to share with an outsource service provider. "The sheer scale and volume of data, the variety of data sources and formats, the number of data owners, and the geographic distribution of the suppliers and consumers of data impose real challenges," observed Boeing Co. database expert Suryanarayana M. Sripada in a presentation to the Very Large Database Conference in Hong Kong, late in 2002: this statement described the situation in the aerospace industry, but could readily apply to many others.

Sripada is correct in identifying the vital need "to provide the right information, at the right time, to the right people, in the right context, in the right format," adding that "Integration of database and collaboration technologies is still a largely unrealized goal."

Looking over these forecasts and analyses, Im reminded of the controversy that began in 1948 with the publication of mathematician Norbert Wieners "Cybernetics," and its dire predictions about the future of labor in the era of computing machines. When digging a ditch was a task that required a strong back, an able-bodied man could make a living moving dirt. But as Wiener pointed out, "There is no rate of pay at which a United States pick-and-shovel laborer can live which is low enough to compete with the work of a steam shovel." Wiener was far ahead of his time in predicting that the same crossing of cost curves would soon take place in the work of the mind as well as the body.

"The modern industrial revolution is similarly bound to devalue the human brain, at least in its simpler and more routine decisions," he wrote, and people have quoted him ever since as a prophet of the doom of the middle class: they cite his warning that "Taking the second [industrial] revolution as accomplished, the average human being of mediocre attainments or less has nothing to sell that is worth anyones money to buy."

Sure enough, self-service scanners are replacing checkout clerks in stores; Amazon.com data-mining tools are replacing the expert guidance of experienced librarians or booksellers; and at the most general level, a Google search makes many kinds of expertise effectively achievable in hours rather than years. Should we be making plans to ship off the irrelevant middle third of our population to another planet, as did the people of the planet Golgafrincham in the second book of Douglas Adams "Hitchhiker" series?

Yes, bulldozers have reduced the number of ditch-digging jobs, but theyve also created jobs in designing golf courses: without a low marginal cost of moving a cubic yard of hill from one place to another, freedom to design what people will pay to enjoy would be greatly reduced. In the same way, when it costs less to write a line of code, theres more return to be gained from writing custom applications instead of tolerating the limits of an off-the-shelf solution. More code gets written, and that code is more specific to a particular business and requires higher-value knowledge to produce.

Wieners comments can also be taken as a warning that "mediocre attainments" would simply be insufficient for someone hoping to enjoy an interesting career at an attractive wage. In 1955, Robert Heinleins "Tunnel in the Sky" described a fictional future students lack of mathematical knowledge: "Being only about to finish high school, his training gone no farther than tensor calculus, statistical mechanics, simple transfinities, generalized geometries of six dimensions, and, on the practical side, analysis for electronics, primary cybernetics and robotics, and basic design of analog computers; he had had no advanced mathematics as yet." I dont know if Heinlein actually expected students of the following century to routinely enjoy such training, or if he was ironically commenting on lack of math and science emphasis in the schools of his own time, but either way the image is worth remembering.

"Mediocre attainments" invite a search for the same skills at a lower price. Strategic IT depends on a commitment to continuing learning.

Tell me what youre doing to prepare for the flood of real-time data.