AP Experiences Outage During Patching Process

 
 
By Brian Prince  |  Posted 2010-10-26
 
 
 

The Associated Press experienced a computer outage that spanned several hours Oct. 25 after its systems failed during an attempted security update.

According to a report by the AP, the problem arose around 3 p.m. EDT, when the news organization tried to apply a Microsoft patch to update security ahead of next week's national and state elections. As part of its patching process, the AP switched from its main system to its backup computers, which failed.

When engineers tried to revert to the main system, they ran into more problems, according to the AP report. The situation wasn't fixed until roughly 8 p.m. EDT. As a result of the outage, a news database that sends out stories, photos and video through the Web was shut down, the AP reported.

In a statement to eWEEK, the AP said there is no evidence the patch itself caused the problem.

"During the process of patching our SQL servers with the latest Microsoft patches, there was a data corruption in a disc drive," Jack Stokes, manager of media relations at the AP, wrote in an e-mail. "We are still working with Microsoft to determine what caused the data corruption that brought down both nodes of the server cluster."

Most of the approximately 1,500 U.S. newspapers that receive AP news utilize the Web feed, though some could have fallen back on old satellite technology on their premises that the AP once used instead of the Web, according to the report. The outage also affected online video customers.

Still, the AP reported that most broadcasters were still getting AP coverage during the outage, and Websites owned by the news cooperative's biggest Internet customers, such as Yahoo, Microsoft and Google, were unaffected.

*This story was updated with commentary from the Associated Press.

Rocket Fuel