Status of Qt Modularization

Published Friday January 21st, 2011
17 Comments on Status of Qt Modularization
Posted in Git, Qt

This is a followup post to the Qt is going modular post in October. Work has gone steadily and we now have modularized most of the big pieces.

In the diagram below, you can see the current split. The modules that were not originally part of Qt have not been illustrated.

Some design decisions:

  • No source dependencies between modules are allowed, only header dependencies.
  • There are still many modules in the QtKernel repository. The reason for this choice is that separating these modules is very hard from a technical standpoint. There are numerous, private dependencies between them (especially QtCore and QtGui), and while we could keep them in separate repositories, it just doesn’t make sense when the dependencies are as heavy as they are. Truly separating them on a source code level would take too long, and right now it is more important that the rest of Qt is modularized. There is also nothing preventing us from modularizing them in a later project.
  • Class documentation stays with their module, since it is inside .cpp files, and naturally belong there anyway. Overview documentation and the qdoc tool itself go in a separate repository.
  • Examples and demos stay with the modules that they demonstrate, with a few exceptions when they depend on many modules. qtdemo is one such exception, and it will go into the doc repository.
  • Autodetection of platform features is still done by the configure script in QtKernel, even if it is used by a different module. This is again to save time on the project. The goal here is that the new maintainers of each module will start transitioning to their own configure scripts (or something different) when the switch to a modular Qt has been made. For example, QtWebKit already has the build-webkit script.
  • “git submodule” has been chosen as the way to provide a master repository containing all of the modularized ones.

Current status:

Most of the things in Qt have been modularized, and are building in Linux. The three big pieces of work we are currently looking at is building for other platforms, running autotests and adapting the modularization process to the Qt master branch (it is currently based on 4.7).

What will happen at the switch

Since Modularization is a very important project for us, we want to make the switch as soon as it is possible. This means that some work may be postponed until after the switch as long as it does not interfere with development. There is no exact date set, but somewhere between one and two months from now should be a good estimate. There will be more news about this when a solid date has been set.

When the switch takes place, what will happen is that all development in the existing Qt repository will be closed. No more commits allowed. Following that, new repositories will appear according to the diagram above. The old Qt repository will be archived so that it can be used to look at old history using the “Git graft” mechanism. However, the new repositories will all start out without any history of their own, with the exception of QtWebKit, which is already a separate repository and based on the original WebKit history.

This means that merge requests will start going to each individual module, and existing merge requests will unfortunately have to be resubmitted according to the new module layout.

Certainly there will be some difficulties in the first phases, but hopefully when they’re ironed out, we’ll have a Qt repository structure that is easier and more enjoyable to use and work on.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Git, Qt

17 comments

Simon Ruggier says:

Why not make each new repository a descendant of the original repository? I think this is important for viewing the history of files with git blame or similar. Bandwidth can still be saved during a clone, if necessary, using the –depth argument.

Christoph Vogtländer says:

Where will QtHelp be located? Still inside Assistant?

Mitchell Ferguson says:

What happened to QtSql? I can’t seem to find it there.

JarJarThomas says:

More important for me, when is there a real 64 bit release for windows that you do not have to compile on your own with changed dll names to get it working.

przemoc says:

@Simon Ruggier:
Because each repo would have the same old history even for files no longer concerning it. Sure, you can use alternates mechanism to solve space issues, but… I do not see why enforcing history continuation would be really a better solutions and I can see why it’s a bit flawed one.

Qt’s repo is a pretty healthy one, Qt’s code is of high standards, well documented and so on. Therefore looking back past modularization point will be rarely needed. We’ll get clean repositories that in short time reach 1000+ commits and they will be presumably more important for newcomers than old commits. And for those more into the past than present or future there are grafts.

But there is one important detail. The current repo will require at the end commit reflecting all the modularization changes. That means the old history will be available to the new combined repo (using submodules), but not module repos alone, unless glueing repos (*) will be provided for each module. So yes, grafts is not the best solution either.

* repos with 2 commits only – first being the last in current repo, second moving particular module code to the root of repo and removing useless rest

Question is whether having all Qt source code will be recommended way of working with it. If yes, then I wouldn’t bother with glueing repos. If no, then making them will be easy and will definitely please devs working on one module only and not wanting to checkout whole Qt.

There is also other solution than glueing repos. New repos with the history of particular modules development. It’s the most time consuming solution and worth only if devs will be encouraged to not checkout the whole Qt. I doubt so.

Kristian Amlie says:

@Christoph Vogtländer:
Yes, QtHelp will be inside assistant.

@Mitchell Ferguson:
QtSql will also be in QtKernel. Seems that it was overlooked in the diagram.

@przemoc:
All of the changes will happen in the old repository, then the new ones will be simply be pieces of that repository. This makes it easy to look at history with grafts, since it just looks like a big file move to Git.

gemfield says:

i’m a little worried about that one day the conficts between each modules occurs

Thiago Macieira says:

@Simon: we’ll solve the problem by using a graft instead. We have experience with this and it works rather well.

The new repositories, alone, start anew. If you don’t download the old repository, it’s like all files are added at a very advanced stage of development.

However, by downloading the current qt.git and by adding two little files to each new repository’s .git dir, all the history is restored and git blame, git log, etc., resume working. We will provide these files for everyone.

Thiago Macieira says:

@gemfield: I’m worried about people writing configure scripts.

The line “each module will start transitioning to their own configure scripts” makes me a bit worried. I hope you’re still standardizing on some tool for this and don’t let everyone write their own configure.sh?

Kristian Amlie says:

@Espen R:
That’s up to the maintainers of each module, but you’re right, we certainly won’t *encourage* them to write configure scripts. A standardized solution would be nice, but that is beyond the scope of this project.

Jason McDonald says:

Don’t forget that configure is done differently under Windows (with configure.exe).

Perhaps this is an opportunity to finally unify the Unix and Windows configure programs into something cross-platform, e.g. Perl.

Janne Anttila says:

“When the switch takes place, what will happen is that all development in the existing Qt repository will be closed. No more commits allowed.”
Do you actually mean no more commits to master, or does modularization impact the whole repository i.e. also 4.7 branch?

“… and existing merge requests will unfortunately have to be resubmitted according to the new module layout.”
If modularization impacts only master branch, merge requests for 4.7 could could be handled with current repository, right?

Janne Koskinen says:

configure for Qt is with sh and exe.
configure for QtMobility is sh and bat.
build-webkit for QtWebkit is Perl.

Having one usage format for all packages would be great. Extra plus if the language would be the same. I vote for Python 🙂

Janne Koskinen says:

Oh yeah.. Where is Qt Mobility in this picture ?

Richard says:

Your title is misleading, as since Qt3 the library is modularized in terms of the compilation. For Qt4, for example, one can run

./configure -opensource -release -stl -qt-sql-sqlite -no-qt3support -no-opengl -no-gif -no-libtiff -no-libjpeg -no-phonon -no-phonon-backend -no-webkit -no-scripttools -no-script -no-multimedia -no-audio-backend -nomake demos -nomake docs -nomake examples

To me, you refer here to modularization in the sense of module source code ownership, as the library is already modularized in terms of the building process. Okay, I assume that the manager(s) in charge of new releases (Thiago Macieira for up to 4.6.3 and Jason McDonald since version 4.7.0) can wash their hands more easily now, and do only some minor PR when a new version full of funny new bugs is released. I really hope that the module owners are serious senior software engineers (in terms of their job descriptions) and not a new generation of junior managers (with dual manager-programmer job descriptions) who actually cannot finalize anything without crying loud for help and/or excuses.

I understand that this blog entry deals with Nokia Qt team internal issues that potentially affect external git users. To me this is only a very small fraction of the modularization process impact. What the blog entry doesn’t address is how should be compiled new Qt library releases after this new modularization is in place. Your published design decisions do not address the methodology of compiling these new releases, and this is funny and laughable, as compilation and modularization go hand in hand. Addressing these issues at a later stage is non-professional, to say the least.

Kristian Amlie says:

@Jason McDonald:
Rewriting configure is something we want to do, but it’s not an easy job, and we decided to keep it out of the scope of this project. I believe there was talk of a separate project aimed at rewriting configure, but at the moment I don’t know the status of that.

@Janne Anttila:
The changes will be applied to both branches, 4.7 and master. 4.6 is dead, so it will receive no more commits regardless.

@Janne Koskinen:
Which build system will be used by the modules is entirely up to the new maintainers. We are just the starting gun. Compare the new modules to packages in a Linux distribution. They all have their own ways of building according to what suits them, yet they all operate together.
Qt Mobility is not included because it was not part of Qt to begin with.

@Richard:
Yes, you are right that modularization deals more with source code ownership than with actual modularized Qt DLLs. This is an important step for us in doing Open Governance, since some modules may be maintained by outside parties.

Commenting closed.

Get started today with Qt Download now