Getting the best out of two worlds

Published Tuesday July 28th, 2009
14 Comments on Getting the best out of two worlds
Posted in Git, Qt

We all want to get access to new features, but we also want to get stable things. These two goals seems contradictory and in a sense they are. If you want to live in the edge you have to accept some “early adopter problems” but if you want to have only rock solid features, you need to let them mature and “gain experience”. With our new open development process people can get access to all of our newly developed features while they are still in the making, therefore gaining valuable insight on what’s coming in the next release. The problem appears when people uses that “bleeding edge” code to start developing things, how can we provide “stable bleeding edge” features?

The solution is more or less obvious, have a master branch where everything is committed and a master-stable branch that is slightly behind master but it is guaranteed to compile. It seems simple and problem free, however there is still one pending question. How “slightly behind” is master-stable from master? Up to last week, master-stable was manually updated, which led to fewer updates and not so “bleeding edge” code. From this week on we have implemented automatic update of master-stable, so now master-stable will be updated as soon as everything is ready, no human intervention required.

The system goes like this:

  1. Whenever there is a change committed to master, that change triggers a build in all of our tier 1 platforms.
  2. If the build is successful, then a build on all of our tier 2 platforms is triggered.
  3. And if that build is successful, our CI system will push changes from master to master-stable.

Voila, master-stable updates itself! So from now on you have no excuses not to use master-stable to check out what’s happening in Qt while enjoying a “stable bleeding edge” branch.

The avid reader would have noticed that it is the first time my name appears on this blog. I joined Trolltech on September 2007 and I’ve been the QA manager for Qt for the past two minor releases (4.4 and 4.5) and currently I’m busy handling Qt 4.6 πŸ™‚

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Git, Qt


Nils says:

This is a really cool system. Thank you!

I really appreciate what Nokia has done to open the Qt development and I’m very curious when the first Nokia phone will benefit from all this!

Carina Denkmann says:

How do you handle rebuilds, full or incremental? For example, someone changes a simple macro in qglobal.h I have to do a full rebuild of Qt. That happens quite often the last days πŸ™‚

Scorp1us says:

There is one small problem. “You forgot to hook up the doll.”

And by doll I mean the tar.gz snapshot facility. And by “forgot to hook up” I mean that whenever a commit to master or master -stable happens, any downloads are aborted. The problem with that is 1) my effective bandwidth means 15-20 minutes of transfer. commits to either branch are about 10-15 minutes apart, on average.

In order to download, I start a download and note the size. I cancel it, then find a download service that takes a url, it pulls it to its site, and I pull it from that site, verifying that the sizes match up.

It is a royal PITA. Please fix.

cduclos says:

@Nils: Thank you very much, we are also looking forward to that!
@Carina: Our CI system deletes all the object files, then it updates the source tree and then it starts a new build. Using incremental builds is possible, but at this stage (i.e. before feature freeze) is still risky to depend on that. There are still plenty of changes and lot of things happening in Qt, after feature freeze incremental builds are more reliable but there’s always the possibility that we need to change a macro here or there therefore requiring a full rebuild.

Joel says:

If I have well understood, you consider a change as having been QCed enough for being automatically pushed on a branch marked as stable… as soon as it builds everywhere (platforms of Tier I and II).

What about testing it, before pushing it?

bkelemen says:

Do you run any regression tests before updating master-stable?

cduclos says:

@Joel: No, our QA process is longer than that. But to allow people to get early access to our code, yet avoid those frustrating experiences of checking out a repository and finding out that it does not even compile we introduced this stage. If you read my post it clearly states that master-stable is guaranteed to compile, nothing else. If we had to go through all the QA process before pushing changes to master-stable, then people will be simply waiting for the next release πŸ™‚

AlekSi says:

You don’t even run unit tests after build?

cduclos says:

@bkelemen & @AlekSi: Yes, we do run unit tests but not on master-stable. In fact our unit test suite is run on master and changes are propagated to master-stable.

It is a chicken and egg problem. We want to give people access to our code as soon as possible, yet we want to do our “best effort” so people can use the bleeding edge code with certain level of confidence.

ariya says:

@Scorp1us: With git, it is better and much more efficient to update the repository than to grab the tarball.

Out of curiosity, what do you guys use for build hardware? I seem to recall hearing at some point that you’re using Incredibuild… does that mean that you’re depending on a big build network of everyone’s desktops, or do you have a “build monster”? (If so I’m curious what its general specs are…)

cduclos says:

@GordonSchumacher: Our test farm is not based on Incredibuild. It will be unfair to customers if we used a special set up that some people might not have. We do “single machine” builds.
The test farm is a mixture of machines, covering the range of platforms we support. We use virtualization technologies as much as possible. For instance all of our x86/x86_64 builds are performed in a blade system that is capable to hold around 36 virtual machines. That way we can cover a lot of platforms and run several builds in parallel. Our OSX builds are performed in Xserve machines from Apple, unfortunately there is no simple virtualization technology for Apple machines (license restrictions), so in this case we have to use hardware machines. Our Solaris builds are done using Sun machines and containers, which allow us to have several virtual machines in one physical machine. HPUX(i) builds are done using HP hardware, we are investigating virtualization technologies, but for now we use physical machines, same thing goes for Aix builds which are run in IBM hardware.

Vahan says:

Hi Carlos:

I am glad to see you are improving your development process:)
I believe that development process automating is always good, but from other hand we are in the risk of falling into a trap. Along with others in this thread I would like to express my thought concerning “stability” of master-stable branch. I think that “guaranteed to compile” is a very bold assumption about code quality. My experience in two-branch development model shows that if accidentally some bug passed to the stable branch and not detected in early stage, it will be, in some cases, very difficult to fix it in the future, especially in prerelease stage, even though builds are never failed. There is some kind of snowball effect…
You said that you run unit test on master branch and integrate changes into the master-stable. What I want to propose, if proposal makes sense in this case, is run nightly regression tests on both master and master-stable branch. This will slow down the process, but you will gain the benefit of release with less bugs, especially regression bugs. Finally, it is well known truth that bug fixing price depends on when the bug is found, so our effort is to discover bugs as soon as possible.

cduclos says:

@Vahan: The only thing we are trying to do when offering master-stable is to let people know that they can trust that code to compile. It happens very often that one gets the latest code from a project directly from the SCM and the code does not even compile. That’s what we are fixing with master stable.
Again, our QA cycle is more complicated and I think everybody agrees the quality of our releases proves that. We are just trying to make life easier for people using the code directly from our SCM, in case somebody needs something stable please use our released versions not our development versions.
Finally, I would like to stress that our unit test suite is run more than once a day and all changes go through it. All changes go through several other stages before being released as part of a new Qt release, but we are not talking about Qt releases here, we are talking about using the code directly from our SCM. Keep in mind that using code from the SCM is very different from using code from a release.

Commenting closed.

Get started today with Qt Download now