Torvalds: How to Keep Linux Kernel on Course

However, the process is not without its problems.

"People do occasionally complain. They worry about the fact that 2.6.x does get a lot of development, but the fact is that the kernel does have a lot of people involved, and we do end up doing a lot of work," Torvalds said, adding that while "some people worry that this equates to lots of new problems, I think weve been pretty successful at keeping the tree stable."

In fact, being forced to keep the tree stable almost all the time is actually one of the big upsides.

And, while mistakes inevitably happen, "we never diverge from stable very much, which has actually been a big relief. With some of our earlier development kernels we ended up rewriting some very core and fundamental code and they became rather unmanageable after a while, exactly because the big changes had made us diverge so far away from a stable base that it was hard to know even which direction to go to get back to stable," Torvalds said.

Novells Kroah-Hartman agreed, saying that in the past the even-numbered kernel releases were the stable releases and the odd-numbered releases were the development kernels, until the 2.4 kernel, which was "nasty, and we decided to rewrite the whole virtual manager in the 2.4.9 release," he said.

Read more here about the tussle over which virtual manager to include in the 2.4 kernel.

The 2.5 development tree also had all these "great new features," and the distributions all wanted them for their enterprise kernels as early as possible, which got "real nasty, real fast, and resulted in back-port hell, and we just did not want to do that again. The 2.4 kernel was not good; it was hell," he said.

There were some 961 unique kernel developers working on the 2.4 kernel, which had risen to more than 1,000 with the 2.6 kernel, all sending patches up the chain, Kroah-Hartman said.

At around the time of the 2.5.3 kernel release, Torvalds started using BitMover Inc.s BitKeeper software configuration management to manage Linux, which changed the process significantly and resulted in nightly snapshots of Linuxs tree now being available, the propagation of e-mails of all individual changes, and a faster feedback cycle.

"But the biggest thing was that Linus started trusting the sub-system maintainers more, which also helped speed up the process," he said.

When the 2.6 kernel was released it was the result of 680 development days, 27,149 different patches and some 6 million lines of code.

At the 2004 Linux Kernel Summit, some eight months after the stable kernel release, 1.23 million lines of code were added, 849,366 lines removed, meaning a third of the kernel was touched "showing that there was still a lot going on with the stable kernel," he said.

This resulted in the decision not to have separate stable development tree and the agreement that everything would go into the 2.6 kernel.

"So that was where we were a year ago. Whats happened since then? The time between kernel releases was too long and as there was no development kernel, there were a lot of changes taking place in the stable kernel and a lot of testing going on and people didnt feel quite so comfortable.

There was also too much time between kernel releases and security patches were coming out as just that, as patches and not updates, resulting in the formation of a security team.

Linux proposed odd and even releases, where one contained development stuff and the other was more stable release, but that was not well-received and resulted in the decision to have the -mm and .y trees, he said.

But, at the recent 2005 kernel summit, there was concern that people were not testing the release candidate, or –rc, kernels as many developers rightly felt they were not the real and final release candidate.

So the issue was addressed and it was decided that after 2.6.13 was released, expected in a few weeks, all the major patches and fixes and updates needed to be sent in the first week.

"These must already have been in the –mm tree for testing. After these are received, the kernel release candidate will be released and open for testing. From that point on we are going to take bug fixes only," he said.

Check out eWEEK.coms for the latest open-source news, reviews and analysis.