Qt is going modular

Published Tuesday October 26th, 2010
48 Comments on Qt is going modular
Posted in Contributors, Git, Qt

Recently a project called “Qt Modularization” was initiated. This is a project that aims to modularize Qt at every level. As you may know already, Qt is currently modularized on the DLL level; each module has its own DLL. However, the project as a whole is still monolithic; all the code is being hosted in a single repository, and you cannot build a leaf module without building the modules on which it depends at the same time. This project aims to change that, so that the modules are hosted in different repositories, with a separate maintainer for each, and the modules may have different release schedules.

This project was motivated by many factors: One is that we want to be able to release some modules  more frequently than others. This is particularly important for modules where development is very rapid, such as QtWebKit and QtDeclarative. Another reason is that having a single repository for all Qt code simply does not scale, neither from a technical nor administrative viewpoint. Having completely separate modules will allow teams and contributors to work independently on separate modules, and it also allows testers and developers to focus on building and testing one module, which can be based on stable base modules.

The actual way Qt will be divided is presented in the diagrams below. First you can see the current state of Qt and then what we are planning. We don’t have a solid release date yet, but the project is fairly high priority for us. Note that the Qt kernel module could be divided further at a later point, but we chose to keep those modules together for now, since they have quite heavy dependencies on each other.

Please note that this is just the current plan, and is subject to change.

Existing:

Planned:

Needless to say, this will have quite a substantial impact on how development of Qt happens. Gitorious will start hosting several repositories, and this will have an impact on everyone who has a clone of Qt. However, once the initial confusion is over, we hope that this change will make it easier to use Qt, work on Qt, and contribute to Qt.

For those who are not interested in looking at specific modules, but want the whole Qt package, we are looking at possibilities for adding a master repository which contains all the others. Currently we are considering git submodule and Android’s repo, but we are open to other suggestions as well. We will also continue to offer releases containing all current Qt modules.

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Contributors, Git, Qt

48 comments

Kelytha says:

Great idea!

I wish KDE would do the same with their Software Compilation.

Don says:

Makes a lot of sense.

What happens with the backwards compatibility guarantees ? For example the existing (Symbian) applications currently linked to the Qt 4.6.3 shipped on the phones ?

Scorp1us says:

I for one, welcome our new modularized overlords!

We’ve needed this for some time. I don’t get why DBUS is in the core, if that’s a Linux-only thing. Shouldn’t we put the Qt Mobility Services API in the core? Or am I confused between DBUS/OOPService/IPService?

2beers says:

I think QtMobility is ok as a separate module. WHat I don’t understand is if Qt kernel and XML,DBUS will be in the same module probably from qt 4.8

eli says:

Sounds nice.
But, what will you do when a new feature in say Declarative would require a small addition to the Kernel?

Apart from the breaking out of the tools and docs, in many ways this is back how things were in early Qt 4.x days. The newly separated modules are all new features like Quick and Webkit. Basically I don’t see how this can be a bad thing so good idea! I like that all the QML/Quick stuff is being grouped together so essential things like QtComponents don’t need to be retrieved separately.

It’s also good to see Qt3 Support split out too 🙂

Tsiolkovsky says:

@Kelytha I think KDE is going in the same direction with their libraries already. Of course any addition helping hand can speed this up. As far as I remember there is also a mailing list about this modularization of KDE, but can’t find it now. maybe someone else knows more about this?

Will Stokes says:

This sounds great. I manually turn off compiling demos and examples on Mac and won’t need to do so in the future. On Windows you can’t disable compiling these easily so that problem will go away (although these days I’m using precompiled binaries on Windows). It will also be nice to avoid compiling Designer, Linguist, etc., and instead simply use those and QtCreator precompiled by Nokia. Since I don’t use QML or QtDeclarative at this point this will mean 1) my installed copy of Qt will be smaller and 2) I can compile Qt significantly faster on my Mac. Excellent!

It also makes a lot of sense to split off the demos and examples since this way bugfixes for those can be pushed out quickly and easily and they really don’t need to be tied to the main Qt releases. My only concern is when you want to add a new demo/example for the newest Qt technology I guess you’ll have to put in a check to see if the version of Qt of new enough. That’s not really a big deal.

Benjamin says:

Shouldn’t examples, docs and demos be part of each module? For example I would expect the examples of WebKit to be close the the WebKit module so changes in the API can be reflected directly in improved example.

@Don: the compatibility guarantee remains. This modularization doesn’t mean that e.g. QtCore.dll, QtGui.dll, QtNetwork.dll will go away – they will still be there in the modular releases, so compatibility is retained.

I agree with Benjamin really.. I’d expect to see docs/tests/examples/demos for each component to be local to that component, having one big dumping ground for everyone directly goes against what you’re trying to do here.

Build times shouldn’t be a concern, as you should be able to turn those off anyhow (like you can currently via -nomake examples etc.)

Apart from that, good news.

Kristian Amlie says:

@eli: The modules will still have some dependency on each other. It’s hard to get rid of that. What we are planning is a system where each module depends on some stable version of another module. This means that an upgrade of the Qt kernel would still force an upgrade of all the others. However, an upgrade of QtDeclarative would not affect anything else, unless that module specifically needs a new Qt kernel version.

@Benjamin, Robin Burchell: The main reason for having examples & demos in separate repository is that this is largely a development driven split. Examples and demos are not crucial neither to develop nor deploy Qt, and fixes to them can be made independently. For someone learning about Qt and using the examples, he would probably be better off getting a public Qt release instead of a Git clone, which will contain all of that.

manudwarf says:

Sounds great !
Personally I sometimes contribute to sqldrivers plugins, and I get bored about having to clone the whole repository in order to modify 2 lines. Would it be possible for plugins to have their own repository ?

Best regards,

killerovsky says:

But what about “Qt Kernel” modularization?
Especially modularization of QtGui stuff?
Current QtGui.dll is _extremely_ huge and monolithic. And it contains overhead functionality for example for projects that is “pure QGraphicsScene-based”.
Example of possible QtGui modularization:
QtGuiCore (some basic painting stuff and some basic “QWidgetLite”)
QtGuiGraphicsView
QtGuiWidgets (all classical UI stuff like QMainWindow’s, toolbars, docks and so on)

This sounds like a great idea, and I look forward to watching things develop. I would love KDE to take a similar approach too, having used git submodules in my day job quite a bit now they aren’t too bad (although they can be difficult to work with). I haven’t really gotten to grips with repo, but it seems like it makes dealing with multiple git repositories easier once you have it up and running.

Kristian Amlie says:

@manudwarf: Plugins will most likely go together with the module that they belong to. It should still be quite a bit smaller to clone.

@killerovsky: It’s something we are considering, but we decided to leave it out of the project for the time being, because we know from experience that splitting those modules is harder than the rest. In addition, splitting the DLL would require us to move to Qt 5, because it is binary incompatible.

Ryan Paul says:

It seems like this approach to modularization creates some fragmentation risks. If different environments have different versions of individual components, third-party developers no longer have a predictable target. It would seem like this only makes sense in environments where the developer is shipping the entire Qt library stack with their application.

sierdzio says:

Great news, thanks!

You could also think of a tool for managing versions – so that when a new version of, say, QtWebKit comes out, people are notified (not only by blogs/RSS) and get the ability to download it for example from within Qt Creator.

Jeremy.Katz says:

@Robin & Benjamin: There’s some discussion going on about what documentation should be associated with. On one hand, I can see keeping the documentation for each module with that module. You want to know how to use the stuff you have, right? On the other hand, having documentation can guide decisions about what modules are of interest for a project. One way around it is to have module specific documentation with the module, and a separate documentation module to provide an overview.

The same goes for tests, examples, and demos. Some of them make sense in a vacuum. Some don’t.

Diego says:

Great! But I hope QT doesn’t make the same mistake X.org did. They modularized the code repositories way too much…

Benjamin says:

@Jeremy
Indeed, you are right some demos and the overview in the documentation just don’t make sense in any module. I still hope the examples and doc will stay close to the code.

aportale says:

@Will Stokes: If you do not want build demos and examples, (I do not want to most of the times), you can configure Qt with -nomake demos -nomake examples. That works for Windows aswell as for other platforms. And I would call that easy 🙂

Liang Qi says:

@Will Stokes: my common way is:
configure
cd src
make/nmake

Will Stokes says:

I’ve used -nomake demos and -nomake examples for years. I didn’t realize they finally worked when compiling on Windows. Thanks for the tip.

/me hopes that this will not result in qbuild-like building infrastructure. That was really a piece of crap, unfortunately.

Enrico Ros says:

I vote for “repo” as the git-overlay-to-fetch-them-all. It’s easy, stable, solid and standard in Android development (~170 git repositories) and ChromeOS as well. I hope more distributions will install it as a system-wide tool.
Thumbs up for this modularization!

Reed Hedges says:

I’m also curious about the details of version and dependency management. I’m sure you guys will come up with a good system, I just want to learn (steal?) from your ideas 🙂

@Kristian: I agree that this is a development driven split, but, who writes, (or at least maintains) a lot of those demos and examples? The developers working on the code in the first place. I personally worry that having them separately will lead to gradual degradation in quality as it’s “easier” to just not bother working on them.

Also, how do you plan to handle things like versioning across modules? There’s a lot of concerns there that I won’t touch on, but continuing on in the same vein: if examples/demos are in a separate repository, how do you plan to release them at the same time as the module they relate to?

Thomas says:

I’m a little skeptical. One of the advantages to Qt is that you get an entire platform with one simple to build/maintain dependency. You are making the world a lot more complex for a some scalability.

NuShrike says:

@Robin: I hope you guys are considering git subtrees to handle this. You still need to a “master” repo to version all the submodules. At least Git is up to this task versus Mercurial.

Meantime, still waiting for Qt5 then because as it is in Qt4.x, having all the rendering paths in one DLL kernel is so monolithic the footprint becomes ridiculous on embedded and Windows Mobile.

With qconfig configurations effectively rotting from each new Qt release since 4.5 (nobody regression tests against it anymore) it would be better to bake qconfig into Qt modularity. It’s rather poor to continue to recommend this broken process to others as the only way to solve Qt’s footprint issues.

All this Qt “progress” still doesn’t give a lean and clean OpenGLES rendering path without being encumbered by all the software rendering, multiple redundant buffer copies and conversions, and legacy code. Not all environments support gcc > 3.x so without fvisibility, Qt fat just gets worse.

Of course this has to be driven internally as any external submissions will just get rejected.

Jose Olcese says:

Regarding the Qt Kernel modularization (like splitting current DLLs in more DLLs) is there something planned or just a something that you’ll evaluate in the future?
Thanks

Lars Knoll says:

@Ryan: The fragmentation risk can be managed differently than with a monolithic development setup. The plan here is to have regular SDK releases that will pull the latest stable versions of all modules back together.

Modularization offers a lot of advantages when developing Qt further, as it splits the problem of stabilization and testing into smaller pieces and allows different parts of Qt to move at different speeds according to the needs we have with this module.

@NuShrike: I’m just an external redshirt, I don’t work for Nokia/QtSW (I work for Collabora, not that it really matters much), so at *this* point in time, I don’t have much ability to do too much of the consideration, given that as far as I know, pretty much all of this discussion has happened behind closed doors as there’s still no open external infrastructure to have these sorts of discussions, much to my chagrain (..sigh :().

Back to the rest of your point, though, I really don’t think that bringing things like that up on an unrelated blogpost are going to further your cause. If I was you, I’d be trying to publicize it through other means: bugreports to Qt, but as well as that, external pressures, like writing your own notes, with your own analyses of what the problem is, how it’s a problem, how it can be fixed, etc – and writing them into a decent blogpost/etc to try get attention focused on it.

It might work, it might not, but it’s better than being lost in the line noise here at least.

Cool stuff! I see Qt Kernel might be split up later – when if at all ever would you do that? I guess there is no way to do it and keep it BC?

Michael Pyne says:

Well I guess that means I’ll have some more changes to make for kdesrc-build… 😉

Gareth Stockwell says:

Will this project change the way either of the following are delivered?

1. QtMultimedia audio APIs (QAudioInput, QAudioOutput, etc), for which implementations are provided both in Qt (QtMultimedia.dll) and in QtMobility (QtMultimediaKit.dll)

2. Phonon: the primary copy of the source is hosted in its own repo (http://gitorious.org/phonon); Qt ships with a snapshot of this repo in src/3rdparty/phonon.

The first case causes confusion for developers (Are the two implementations identical? If not, which one should I link against). Both add extra work for developers, who must migrate patches between the two codelines.

As long as Phonon remains part of Qt, I expect that the Qt project will want to maintain its own branch, so that it can give the same quality guarantees for Phonon as for the rest of the system – but will this be part of the Qt Kernel, or in its own module?

As for the QAudioXxx APIs, I hope that this re-org will be the opportunity to get back to having just one copy – in Qt Mobility.

Tasuku Suzuki says:

Will translation be maintained in each module?

Keith Rusler says:

When is Qt3Support going to die? I personally think users by now already have moved from Qt3 to Qt4.

Keith Rusler says:

I forgot to mention this. You say some projects like QtDeclarative and QtWebKit are developed more faster. Does that mean they will not be part of Qt anymore, but more like them having their own version different from Qt?

Hugo says:

It would be also good to make QImage and its plugins a separate module. I make OpenGL applications and I have always to pack QtGui with my application to use only QImage.

Kristian Amlie says:

@Thomas: You can still do that. The pieces will still fit when you put them together, even if we split it into separate repositories.

@Espen Riskedal: Most likely it can’t be done without breaking binary compatibility as you say. But at some point down the road if Qt 5 comes along, we will definitely take another look at it to see if we can split it further.

@Tasuku Suzuki: Most likely, yes.

Philippe says:

With such a restructuration, isn’t it the time to break binary compatibility, and call it 5.0…?

Kristian Amlie says:

@Philippe: When we finally do break compatibility, we don’t want to do it for no reason. There are *a lot* of other things we want to do if we break compatibility, and if we added all that to this project, it would take much too long to complete.

I think this new approach will be good in the mid-term, even when in the near term this will involve having different repositories and getting used to work with the modules and dependencies.

Question: will this modular approach help getting the Qt SDK and Nokia SDK merged into just one SDK so the developers can download and install selected modules based on the desired platforms they want to develop for without having the “problem” we’re facing now that that a lot of tools (i.e. Qt Creator), examples, documentation and so is duplicated and installed twice if you decide to develop for all the platforms Qt is available for.

good_kid says:

I want to know why remove phonesim package from QT?And How can I access sim card in QT?please don’t tell me no way!

NuShrike says:

@Robin, thanks for the tips, but this is a topic about modularization so I would think my points are somewhat on topic. It is important because there are corporate projects I’m involved in where Qt could’ve been a contender, but the monolithic issues have hurt its deployability. This is after I’ve already done extensive multimedia development in Qt in another largish, public-released app.

As for bringing it up in Qt bug-reports, tried, invalidated, ignored, so I really don’t have any more investment to care.

bsd says:

I would prefer vanilla git as opposed to a third party porcelain.

Henrik Hartz says:

@killerovsky I agree – IMO the QWidget _derivates_ could be split off. Especially in embedded scenarios you might not need these, but have a very specialized UI, and very little room to spare.. Might require Qt 5 though.

Commenting closed.

Get started today with Qt Download now