In life-and-death decisions, people want facts. Health insurance with “medium” coverage doesnt cut it at the emergency room, and the nations vague multicolor terror-alert system has generated plenty of nervous confusion.
Still, despite all of the money at stake, many businesses rely on simplistic weighted scores to measure and protect against risk. But a risk-assessment scale of “1 to 5” or “low to severe” is about as effective as a placebo, warns Doug Hubbard, chief executive of Hubbard Decision Research in Glen Ellyn, Ill.
Mathematical formulas are used to measure risks in everything from insurance coverage to stock portfolios, but CIOs rarely take this approach when trying to assess their information-technology risks, Hubbard says. “I.T. isnt some special case that defies all risk analysis,” he says. “Its just the last place to use it.”
Hubbard believes in training managers to measure risk the way actuaries do. Instead of rating a projects chance of technical success on a scale of 1 to 5, for example, a manager would forecast a 75% chance a project will cost between $1 million and $1.3 million, or an 85% chance it will be completed within 10 to 12 months.
Scientific measurements are ideal in theory, agrees Edward Hill, a managing director of Protiviti Inc., a risk and audit consulting firm based in Menlo Park, Calif. But most managers arent well versed in statistical research, and the time it would take them to learn it might not be worth the investment for their companies, he says.C
Another obstacle: assembling the hard numbers needed for more-exact measurements. Take information-security risks, for example. Data are often unreliable or unavailable because most computer crimes arent caught, and when they are, companies try to keep the details quiet, says Ed Roche of Barraclough Ltd., a New York information-technology consulting firm. “You cant make up the data,” he says. “If its not there, its not there.”
Hubbard thinks of it this way: Would you rather fly in a plane designed using mathematical formula or weighted score?
Getting more precise
Doug Hubbard of Hubbard Decision Research offers the following advice to beef up risk-assessment systems:
1. Remove ambiguity. Instead of rating “technology risk” on a scale of 1 to 5, drill in on the variables. Is there a chance the vendor wont be around to support the software in two years? Could implementation costs exceed the approved budget?
2. Bet on it. Pretend your risks are tied to hard-money wagers. People are better at assessing odds when dollars are involved, even if its a simulated exercise.
3. Study past forecasts. Work from historical numbers, not memory. People remember when theyre right, not when theyre wrong, which leads to overconfidence. Reviewing actual track records will temper that.
4. Find the experts. Recruit the statisticians in your company—actuaries, ratings specialists, market-analysis experts—who are already trained to do sophisticated statistical analysis.
How precise is your companys assessment of risk? Download the quiz to find out.