The rapid pace of Linux development appeared to hit a roadblock last year with the industrys decision to forestall development of the Linux 2.7 kernel. Linux vendors and developers wondered if tweaking a single, stable 2.6 kernel could work in practice.
According to open-source insiders, the move to create separate kernel trees for technology testing and bug fixes, which are then incorporated into the stable kernel when ready, has been a huge success, pleasing both kernel developers and the vendors who distribute the open-source operating system.
“Im certainly pleased, and judging from the reactions we had at the Linux Kernel Summit in Ottawa a few weeks ago, most everybody else is too,” Linus Torvalds, the founder of the Linux operating system, told eWEEK.
The biggest advantage of staying with 2.6.x was that developers do not have two different trees between which they need to port patches, which makes them happy, he said.
Linux vendors tend to like the move, because the upgrades are more gradual, rather than the huge, and potentially painful, jumps of the past.
This shift started at the 2004 Linux Kernel Summit with the decision to no longer have a separate kernel development tree, but to keep adding new features, technologies and patches to the existing 2.6 stable tree, Greg Kroah-Hartman, a Linux kernel developer with Novell, told attendees at the annual OReilly Open Source Convention last week.
That decision has spawned three separate 2.6 trees: The first is the mainline or stable kernel, known as 2.6.x and which is maintained by Torvalds; the second is known as the 2.6-mm tree, and is where technologies are tested before they get into the mainline kernel; and the third is the 2.6.x.y kernel (known as the .y kernel), which is for bug fixes only.
The .y kernel is governed by a set of strict rules, including that the fix has to be less than 100 lines, that it applies to something already in the mainline kernel, and there had to be a three-day public review of all patches to allow all involved parties time to express their view and to make sure that everyone bought in, Kroah-Hartman said.
There were 12 2.6.11.y releases, with just 507 lines added and 303 lines removed, “which is the way to just get things fixed and I think shows that this process is working. These fixes also go into the mainline release and the 2.6.11.y was dropped when the 2.6.12 release came out. There have been four 2.6.12.y releases so far, with the latest released last Friday,” he said.
The 2.6.12.y releases have had just 169 lines added and 199 lines removed.
“We are not back porting big experimental things into the .y-series,” he said.
But there needed to be a mechanism for testing new technologies, a place where they could be revised, updated and even removed before actually getting into the mainline kernel.
As such, it was decided that the -mm tree would be the place where things were tested before they got into the mainline kernel, Kroah-Hartman said.
For his part, Torvalds was upbeat about the changes.
“Of course, it has led to us having to be a bit more careful about things, and the 2.6.x.y releases end up being another sign of how weve changed our process a bit as a result of this all, but I think its been an unqualified success,” he told eWEEK.
These changes have significantly improved the kernel development process and have allowed the team to keep things less diverged, with people doing development off a base that is closer to what most people end up doing.
With the huge development splits required for the big rewrites under the old model, some developers had to work on both the development and the non-development kernel, which did not always please them, Torvalds said.
But he did leave the door open to create a 2.7 tree “if we hit some fundamental change that makes us split into a 2.7.x tree. We havent hit anything yet, and people seem to be doing well, but I want to point out that if something really fundamental rears its ugly head, we still accept the possibility that wed have to do a full unstable branch split.”
A Flawed Process
However, the process is not without its problems.
“People do occasionally complain. They worry about the fact that 2.6.x does get a lot of development, but the fact is that the kernel does have a lot of people involved, and we do end up doing a lot of work,” Torvalds said, adding that while “some people worry that this equates to lots of new problems, I think weve been pretty successful at keeping the tree stable.”
In fact, being forced to keep the tree stable almost all the time is actually one of the big upsides.
And, while mistakes inevitably happen, “we never diverge from stable very much, which has actually been a big relief. With some of our earlier development kernels we ended up rewriting some very core and fundamental code and they became rather unmanageable after a while, exactly because the big changes had made us diverge so far away from a stable base that it was hard to know even which direction to go to get back to stable,” Torvalds said.
Novells Kroah-Hartman agreed, saying that in the past the even-numbered kernel releases were the stable releases and the odd-numbered releases were the development kernels, until the 2.4 kernel, which was “nasty, and we decided to rewrite the whole virtual manager in the 2.4.9 release,” he said.
The 2.5 development tree also had all these “great new features,” and the distributions all wanted them for their enterprise kernels as early as possible, which got “real nasty, real fast, and resulted in back-port hell, and we just did not want to do that again. The 2.4 kernel was not good; it was hell,” he said.
There were some 961 unique kernel developers working on the 2.4 kernel, which had risen to more than 1,000 with the 2.6 kernel, all sending patches up the chain, Kroah-Hartman said.
At around the time of the 2.5.3 kernel release, Torvalds started using BitMover Inc.s BitKeeper software configuration management to manage Linux, which changed the process significantly and resulted in nightly snapshots of Linuxs tree now being available, the propagation of e-mails of all individual changes, and a faster feedback cycle.
“But the biggest thing was that Linus started trusting the sub-system maintainers more, which also helped speed up the process,” he said.
When the 2.6 kernel was released it was the result of 680 development days, 27,149 different patches and some 6 million lines of code.
At the 2004 Linux Kernel Summit, some eight months after the stable kernel release, 1.23 million lines of code were added, 849,366 lines removed, meaning a third of the kernel was touched “showing that there was still a lot going on with the stable kernel,” he said.
This resulted in the decision not to have separate stable development tree and the agreement that everything would go into the 2.6 kernel.
“So that was where we were a year ago. Whats happened since then? The time between kernel releases was too long and as there was no development kernel, there were a lot of changes taking place in the stable kernel and a lot of testing going on and people didnt feel quite so comfortable.
There was also too much time between kernel releases and security patches were coming out as just that, as patches and not updates, resulting in the formation of a security team.
Linux proposed odd and even releases, where one contained development stuff and the other was more stable release, but that was not well-received and resulted in the decision to have the -mm and .y trees, he said.
But, at the recent 2005 kernel summit, there was concern that people were not testing the release candidate, or –rc, kernels as many developers rightly felt they were not the real and final release candidate.
So the issue was addressed and it was decided that after 2.6.13 was released, expected in a few weeks, all the major patches and fixes and updates needed to be sent in the first week.
“These must already have been in the –mm tree for testing. After these are received, the kernel release candidate will be released and open for testing. From that point on we are going to take bug fixes only,” he said.
