Addressing an audience of several hundred members at an event jointly hosted here by the Churchill Club and the Commonwealth Club, Ballmer said the Redmond, Wash.-based software giant faces many competitors across the many different parts of its business.
But there are two big phenomena going on: the new open-source and advertising business models.
Open source is more about whether that is a business model that delivers superior innovation than a commercial company can deliver.
"So, there is competition with this new business model; there was also competition with the new advertising business model [embraced by Yahoo and Google]," he said.
While Microsoft cant embrace the open-source business model, it has embraced the new advertising one and wants to grow to the No. 1 market position in that space from its current No. 3 spot.
"The greatest competitive threat we face is our own ability to either embrace or compete with alternative business models," Ballmer said.
When asked by moderator Roger McNamee, the co-founder of Elevation Partners, to choose between IBM or open source as the threat he most worries about, Ballmer quipped that IBM is then no longer in the game.
Open source is a good-old fashioned engineering competitive threat, he said, and has managed to dominate in several markets.
Microsoft needs a better high-performance clustering product than Linux, and is working hard in this regard, with "a better product and a total cost of ownership being the key," he said.
"While it is hard to beat open source on the initial procurement side, it is easy to compete on the total cost of ownership front," he said.