Microsoft Move to Agility Hastens Bing's Deployment Cadence

Microsoft's move to Continuous Delivery helped Bing deployments speed up from monthly to daily and scaled the Bing team from 100 engineers to 600.

Bing search

When Microsoft's Bing engineering team decided to speed up its cadence of delivering code into production, it set out on a journey toward agility that initiated with Continuous Delivery (CD) as a key destination.

With a goal of trying to catch and surpass its closest leading competitor, the Bing team had to rethink the way it worked.

Craig Miller, technical advisor for Bing, said the effort began four years ago with a system of monthly deployments, 100 engineers, two to three weekly incidents—production issues that occur—and a pretty standard waterfall development system. That had worked fairly well for the early generation of Bing. But to continue to chase Google, something had to change, drastically.

"So we stepped back and decided to go after an order of magnitude improvement in our system," Miller told eWEEK. "I don't mean code velocity, but to look at the big picture as well and at what we call 'idea velocity' here in Bing."

According to a synopsis of the Bing team's move to agility and agile methods on Microsoft's Engineering Stories site, "When we began to make the leap to Continuous Delivery, we not only changed the way our developers write code—we fundamentally altered the way our business operates."

The CD method is an agile software engineering approach in which teams produce software in short cycles, ensuring that the software can be reliably released at any time. It aims at building, testing and releasing software faster, more frequently and with higher quality.

"At its core, CD represents a decision to pursue development processes that result in steady incremental rather than comprehensive change," said Charles King, principal analyst at Pund-IT. "If CD is implemented successfully, organizations can reduce the cost and time required to make changes, along with associated risks."

That's true for two reasons, he said. One is that making numerous, ongoing incremental changes tends to identify and correct potential problems more effectively, resulting in greater reliability and resiliency. The second is that since older features are altered and newer features are introduced more gradually in CD, it's easier to identify which deliver actual value for customers and which fail to do so.

In a blog post on the Bing transformation, Dr. Jan Pedersen, Microsoft's chief scientist for Bing and Information Platform R&D, said, "To accelerate feature deployment and innovation, we have invested much effort in overcoming software engineering challenges. The monthly deployment cadence has been gone for some time, taking with it both the old culture and most of the infrastructure. In its place, a highly distributed, parallelized, and agile system has risen, and this system has been a game-changer for both developers working on Bing features and the live site users. Bing Engineering has spearheaded this effort, and has produced a world-class ideation, development, validation, and experimentation system."

That system helped Miller's Bing engineering team scale from 100 engineers to 600, and from a cadence of monthly releases to daily releases.

"We evolved from monthly deployments to daily—which was our mantra, to now where we deploy 20 to 24 times a week," Miller said. "That's roughly four times a day. We have 600 engineers in the branch, and we have zero to occasionally one incident in a given week."

Agility is the speed at which ideas can transition from the whiteboard to the keyboard to the live site and to users, Miller said.

To ensure compliance with its standards, the team built a core focus on quality into its process, demanding that engineers think about the effect of their changes before they go out to customers.

"When you're this close to production ..., when your code goes in and literally within six or seven hours, it's fully rolled out worldwide, it actually connects the engineer to the customer much more closely than if you check in your code and a month later it shows up in production," Miller said. "It gives that engineer a feeling of real power.

Indeed, he noted that with Continuous Delivery, there is an inherent 24-hour cycle as part of the model.

"Within 24 hours you can write your code, you can get it deployed and you can get it live and experimented, and get feedback by the next day when you come in," Miller said. "Every day you get a signal back from the users on the code you just put out, so that you can then alter your behavior. It may mean you have a bug that you missed, or the feature just isn't working for the users. Or you see where you need to iterate and tweak a feature. That's a powerful connection to the users."

Instead of the term "fail fast" that is often used in discussions of agile development to describe the process of quickly learning what does not work, Miller said he prefers to use the term "learn fast." Code velocity is not the ultimate goal, he said. The real goal is idea velocity and being able to deliver as many ideas through the system safely and with value to the customer at every opportunity.