Please welcome qtintegration, the newest member of the team

Published Friday December 18th, 2009
9 Comments on Please welcome qtintegration, the newest member of the team
Posted in Git, Qt, Test

Some of you may have noticed a new committer to the Qt git repository. Someone who is tirelessly working night and day, but only seems to make merge commits. I am of course talking about Qt Continuous Integration System, who in addition to never getting any time off, never gets paid either.

This is because Qt Integration is a Python script, written by Simon and Paul, with invaluable assistance from Rohan and Jesper from the QA team. (And just to make sure: “invaluable” means “of immense value, so great that it cannot be measured”, and not “so small that it cannot be measured”.)

So what does this script do? First a little detour: If you are still reading this, you have probably been following the commit logs and seen commits of the type “Merge branch ‘4.6’ of scm.dev.nokia.troll.no:qt/oslo-staging-2 into 4.6”, but you may have wondered: What is a staging branch?

A staging branch is a git branch that is shared between a number of QtDF developers and is tested separately and then integrated into the public branch at intervals. The point of having staging branches is to catch build breakages and regressions before they make their way into the shared repository. Until now, the integration has been done manually, and there was no common baseline to compare against, so it was hard to compare the quality of branches: If there is a failure in one of two branches, is it introduced in the first, or fixed in the other?

Another problem we’ve had recently is that our test farm is now running on virtual machines, but most of the tests were written on real machines. Some tests end up exposing the difference, and suddenly turn unstable because of timing differences or other subtleties. We are fixing these tests of course, but in the meantime, this adds a whole new level of uncertainty to the interpretation of test results.

The Qt Integration script automates the process and solves the problems. The system works roughly like this:

1. Developers push changes into one of the integration branches.
2. Direct push access to Qt’s 4.6 branch is _disabled_.
3. The script takes one integration branch after the other, merges the branch with Qt 4.6 and runs the resulting merge through the autotest system.
4. After a successful build of Qt and the execution of the tests, the script compares against the previous 4.6 results.
5. If there are no new regressions (*), the merged branch is pushed to 4.6
6. However, there is a “blacklist” of test functions that are known to be unstable, and those are allowed to regress without blocking the integration.
7. Then the next staging branch is tested. (goto point 3 :-))

(*) No new regressions means: Tests are failing in 4.6 are allowed to also fail in the integration branch. No other tests are allowed to fail, unless they are blacklisted

Note that at any point, 4.6 contains a version that has already passed the testing process.

What this means is that in theory, the Qt 4.6 branch will always build, and the number of autotest failures will steadily decline. Yes, I did say “in theory”. The biggest limitation is that the system does not currently cover all the supported platforms. Currently we are doing Linux/X11, Linux/QWS, one Windows version, and one Mac version. We are going to add new platforms as we go along, but the system will probably never cover all the Tier 2 platforms. Any failures on the remaining platforms will be caught by the autotests running continuously on 4.6.

This means that we will still keep the 4.6-stable branch, just like before. The only difference is that it should now be even more stable. In theory 4.6-stable could go away in the future, if we could get the test cycle for all of our platforms to be fast enough. Also, please note that we are only running this on 4.6 at the moment. We will extend it to the master-going-to-become-4.7 branch in the not too distant future (but not this year).

Finally, I have to apologize for the lack of cool screenshots in this post. This is a script after all; just move along, nothing to see here…

OK, then, if you insist, here’s one of the script running, taken just moments ago:

Output from qt-integration script

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Git, Qt, Test

9 comments

Cobalto says:

Hi!
Really interesting learn more about your workflow! Could be more interesting if you publish this script on gitorious and tell more about which integration server you use, etc.

Wim Leers says:

Very nice to learn more about Qt’s continuous integration setup, which apparently strongly takes advantage of distributed version control features. Cool! πŸ™‚

AlexBravo says:

How long does it take to process one integration branch?

AlekSi says:

First of all, thanks for a detailed post. Some comments / questions:
1) +1 for “publish this script on gitorious”, it’s generally useful for all git users.
2) Is this blacklist public? May be we will help to reduce it.
3) When you saying “Tests are failing in 4.6 are allowed to also fail in the integration branch” you mean XFAILs? I hope you do because failing tests are big demotivators for CI.
4) And for cleaning you may want to use ‘git clean -xdf’ (looking at screenshot seems that you are not use it).

AlekSi says:

Oh, you still use KDE3? πŸ˜‰

Anon says:

Aw – I thought was going to be about some of Oliver Goffarts’ recent DE integration work πŸ™‚

Thiago Macieira says:

@AlekSi: it means that, if the test fails (as in FAIL, not XFAIL) while testing the 4.6 branch, it’s allowed to keep on failing on the tested branch. If it passes, it’s a fix.

If, on the other hand, the test is passing, but starts to fail, it’s a regression. And the integration will not continue (unless it’s one of the blacklisted tests that are unstable).

Most of the unstable tests are those that either depend on the window manager (like, getting focus) or the networking ones.

Simon says:

Aleksi, right now the black list is just a little text file in the directory where the script is located. But in order to make it work with different branches of Qt with different sets of tests it might make sense to move it into the Qt repository directly.

There’s no good reason to keep it private, except that we don’t have a good place to publish it. But if it were part of Qt then it’s no problem πŸ™‚

Commenting closed.

Get started today with Qt Download now