Examining the Regulators

Despite the strong action it took against Multidata, FDA watchers say the agency-one of the few empowered to regulate software in any fashion-still does not go far enough to insure the integrity of the computer programs that are the brains of medical devices.

"Thinking the FDA is some sort of watchdog group is an exaggeration," says Bob Morton, a software-quality expert and a former head of the FDA unit in charge of radiation-therapy equipment.

The FDA approves medical devices under one of two mechanisms: premarket approval or premarket notification.

Premarket approval is reserved for technologies that are radically different from anything on the market, such as a breakthrough pacemaker product. These products are run through scientific reviews and tests to ensure that they are both safe and effective. Products which fit into an existing category of devices are subject only to the premarket-notification process, under which the FDA neither tests products nor requires a manufacturer to conduct its own field trials.

Multidata won approval for its software in March 1997 under the notification process, since a number of radiation-treatment planning packages were already on the market.

In this procedure, a manufacturer such as Multidata submits paperwork that details how they designed, produced and tested the new product. Agency officials then pore over the documents looking for potential flaws, problems in testing or other likely trouble. If everything is in order, the FDA gives the company clearance to sell its product.

Last year, 3,500 of the 4,000 medical devices approved by the FDA came through the notification process.

"Can we rely on the FDA to police the medical-device industry in a very reliable way, via the premarket submissions and the inspections?" asks SoftwareCPRs Kusinitz. "No. I think were primarily dependent on the...good intentions of the medical-device manufacturers."

In their defense, FDA officials say its not feasible to test every product. "We regulate 10,000 different types of products. How do you come up with tests for 10,000 products?" says Murray, the FDA software expert. Besides, he says, the FDA has no conclusive data that outside testing would improve the quality of software anyway.

Critics, however, still take issue with the FDA being so reliant on human review.

"The problem with human review is that its not infallible," says Jonathan Jacky, a radiation-oncology research scientist now working at Microsoft Research, Microsofts computer-science research organization. Humans, he says, "just might overlook something."

The FDA admits that it doesnt have the manpower to look at every line of code and "things do get missed," according to Timothy Ulatowski, director of compliance at the FDA unit that oversees medical devices.

Indeed, some bugs are even allowed in the software. According to FDA documents, a list of all bugs left in a system must be submitted, plus documentation that those bugs arent a safety concern.

The FDA also asks manufacturers to submit a schedule for when they plan to fix the bugs. None of the bugs, however, can be considered a safety issue. Multidata says it submitted its bug list to the FDA and that it fixed all bugs.

The FDAs task will only get more difficult as time goes on. While the number of medical devices approved has remained steady-roughly 4,000 products a year for each of the last five years-the devices and their software are becoming more pervasive and more complex. Already, about half the medical devices approved for market contain software, and FDA watchers expect that percentage to grow. "Each iteration of [a] device tends to put more software in," says Morton.

And, Morton says, "devices are being released before theyre ready." He wont name companies or products, citing confidentiality agreements, "but its true," he says.

The FDA, in its defense, maintains that its up to the task. If anyone wants to know how tough the FDA is, says Ulatowski, "ask Multidata."

Next Page: Back in Panama.