In criminal investigations the first suspects are always the most obvious ones—the husband, the girlfriend, the accountant with power-of-attorney who cant be located right at the moment.
Not so when the problem under investigation is a database that suddenly starts taking queries but giving no answers back or, worse, spits out reports with data that makes no sense.
Like detective work, database troubleshooting relies heavily on the analysis of forensic evidence–both performance data from sniffers that deconstruct software processes, and environmental analysis that can identify troublemakers that would otherwise remain unsuspected.
Glenn Dunne, manager of data resources for $3.5 billion grocery chain Hannaford Bros., followed the evidence not to an error, but to an unfortunate coincidence of things that worked correctly.
“We had this one query that came back in 15 minutes or so a week before, when we ran it last. Suddenly its not coming back at all,” Dunne says.
The culprit turned out to be a backup sequence that was running at inopportune times, but finding that out took a lot of doing. No single person saw across enough systems to quickly identify the problem, which might not have been noticed if it hadnt goofed up that one query.
Every company that has to troubleshoot a database uses some form of database forensics. But simple analyses can lead to solutions for symptoms, not the root problem, warns Charlie Garry, senior program director for infrastructure strategies at Meta Group.
Databases stress servers, storage and other systems so heavily that a problem there often shows up as a database slowdown, Garry says.
Sometimes, however, the problem arises between keyboard and chair. For example, the users at one company who would add a “D” at the beginning of some names to indicate that the customer had died could read the data just fine, says Adrienne Tannenbaum, president of Database Design Solutions, a consultancy specializing in database design and optimization. The database didnt handle it nearly so well.
But problems like that can indicate a more serious one—that the database isnt answering the needs of the people its supposed to serve without jury-rigged changes. How can you tell when thats the problem? “Spreadsheets,” Tannenbaum says. “People will pull data out and manipulate it on their own. Too much spreadsheet use is a good indicator of bad design.”
To learn more about your database problems, click Here for Baselines interactive diagnostic quiz.