To make, or not to make – QMake and beyond: Redux

Published Wednesday October 14th, 2009
32 Comments on To make, or not to make – QMake and beyond: Redux
Posted in Build system

The previous blog entry yesterday spawned so many sizable comments, that I felt they couldn’t be simply replied to in comments. So, here are some replies and comments to your comments, which you may comment on again 🙂

Many: “Have you considered the <insert favorite here> build system?”

Yes, we have.
Any Java-based system is automatically /ignored due to the vast overhead of running such a system, and the fact that we would then need a Java VM running on all the systems we support. A build system needs to be fast, and fairly lean.

We’re not really considering the autotools.

Max: “CMake gets a lot of “wow! it’s great!” comments, but I could write several blog posts about how it is in fact merely mediocre. I guess you guys agree since you are not considering it.”

This is incorrect. We have not said nor implied that we are disregarding CMake. CMake might not current fulfill our complete wishlist, but we are of course still considering it. In fact, we initiated discussions with KitWare (the guys behind CMake) weeks before this blog post, to see if our wishes are unitable with their plans for CMake. These discussions are still ongoing. In fact, Bill Hoffman (CTO & Vice President of KitWare) is currently at DevDays in Munich, where these discussions will continue.

nathan: “I think the idea of using JavaScript has all the many advantages”
Sylvain: “Javascript + JSON is really good, please go for it…”
Damien Arthur: “a build tool based on json would be sweet!”
Adam Higerd: “I like the idea of using ECMAScript / JSON notation. It’s simple and not overly verbose, but expressive enough to handle non-trivial setups.”
mariuz: “I would choose javascript or minimal python (waf like)”
spinynorman: “Please don’t base any replacement on JavaScript.”…”your core audience is developers familiar with desktop rather than web languages. I’d much rather have it Python based than JavaScript.”
Tim: “I think IDE integration is the most important factor to consider, and one that you can’t do right if your build file is a script.”

After having thought about build systems a lot, we tend to like the idea of JavaScript as the build language. The reasons are several:
1) Javascript lets you use both an imperative language, and a subset declarative syntax (JSON).
This is great for supporting IDEs, since you can say that a project is primarily defined in the declarative syntax, while still support more complex builds using a full imperative language. The IDE would then only support modifying the JSON of the project, so support changing compiler flags, defines, adding and removing files on various platforms etc, and for the non-declarative parts it could drop you out in a tab-completed syntax-highlighted editor with easy access to documentation 😉
2) No need to rely on 3rd party
JavaScriptCore can be bootstrapped into the product, so there’s no need for a 3rd party distribution. That also enables us more to structure the language usage, as you wouldn’t expose a boat load of libraries by default, which you would have with for example Python.
3) Native speed on many platforms
JavaScriptCore will compile the scripts into native code on many platforms, and increasing, so the script execution is fast.
4) Good debugging tools for JavaScript
JavaScript has plenty of debuggers readily available for it, so you can debug the whole build process from within the IDE, for example.
5) Solid user base, and growing
The World Wide Web usage of JavaScript is growing at tremendous pace. QMLs scripting language is JavaScript. So, on a team there will always be someone who already knows the language. This makes build system maintenance easier for both us and our users.

Peter: “It isn’t too hard to use CMake with QtScript, see http://sourceforge.net/projects/cmakescript/
A hello world looks like this:


project("hello")

// this starts the debugger
// Qt 4.5 needed
debugger

var name = "Hello";

add_library (name, "hello.cxx")


This is a very neat proof of concept, and I could definitely consider CMake with a JavaScript language. However, unless you also start changing some of the internals to make use of the language more effectively, then all you have a new language used in the same way as before, like the add_library() function call above. If the internals were modified to allow for the the JSON construct of the project, then we’re talking! This project is great, but at the current state, not much more than what we had with QMake + JavaScript, which we had before. (And I’m not sure KitWare would want to integrate the whole QtScript module into CMake 😉 )

DAVIDB: “Cmake – its good enough for KDE which handles like a billion lines of code in hundreds if not thousands of projects. It’s been around a while and has proven to be a very good cross platform configuration tool.”

Several features needed implementation before the KDE project could use CMake; and several more would be needed to properly support Qt. It’s not just about the missing features, but also about language. We like how easy QMake projects are for medium projects, although it can get ugly when you get into the medium-to-complicated projects. It’s also very quick to get started with. Often you can just run qmake -project in your project directory, and you have a .pro file which gets you most of the way, if not all.
You also have to consider all the customers that Qt has, and will have as the various phone ports move along. What we decide on will ultimately push them to switch at some point, although we will of course maintain QMake for some time.

Coder: “Why not take an existing Open Source cross-platform build system and contribute to it.”

If we find a project which maps nicely with wishlist and requirements we have, of course this is what we will do. We don’t want re-invent a wheel, if we can avoid it. However, we do have strong reasons for not just jumping on any of the ones discussed in their current state; and that’s also why we’re reaching out, to see what ya’ll think about the various tools out there, and QMake of course.

spinynorman: “whatever you do please provide a painless upgrade path – a way to import QMake .pro files into the new system”

The users are always our primary concern, and we would of course do everything we can to assist in the transition. Just like Qt 4 has the Qt3Support, maybe the new tool would have a QMake front-end alternative? It’s not impossible.

spinynorman: “please don’t have any hidden behaviors like special include or library paths (such as the libary paths that QMake itself decides to add to Makefiles).”

I fear you might be using QMake in the incorrect way if it’s adding library paths which you don’t want into the Makefiles.
For example, if you wish to use QMake on non-Qt projects, you should do CONFIG-=qt.

spinynorman + many: “Can’t QMake be fixed rather than replaced?”

Possibly. However, it would take considerable efforts to switch out the internals while keeping backwards compatibility, and at the current state of the QMake autotests I wouldn’t dare it. Though, I guess it would be doable if enough people consider it a much better alternative than something new. One thing is then, how do we fix up the language mistakes without destroying too many projects? Bring the QtScript module back into QMake?

Holger Schurig: “If you consider using your own tool, make it a library. So that it’s easyly integratable into some IDE.”

If we decide on writing our own, that would be the idea, yes. The library would contain the whole DAG, have functions to manipulate the project JSONs, and the IDE could walk the invalidated DAG nodes, both to compile them, and to update the code completer.

Holger Schurig: “QMake currently has several generators, e.g. unix (make) or windows (nmake). Maybe it can gain a qbuilder (or whatever the new name will be) generator. That would allow you to use qmake to convert a qmake-project into a qbuilder-project.”

That’s a novel idea. Only problem is that much of the project is “lost in translation”. Once the front-end parses it, much of the information is thrown away (system checks, conditions, loops etc), so the information the back-end sees is only valid for that one particular configuration.

Reed: “Jam (Boost Jam or Perforce) are pretty good”

Read this article: http://gamesfromwithin.com/the-quest-for-the-perfect-build-system-part-2

Cliff: “Ecmascript-Syntax is better than any strange language but it won’t fully fix the problem for IDEs in my eyes. Should the IDE evaluate the code?”

As mentioned above, if the build system was libified, the IDE would contain the whole build system logic, parser and DAG. So, it easily could reparse the project file if need be, and replace that project branch in the DAG. The DAG walkers would then take care to only rebuild the invalid nodes.

Craig Ringer: “CMake looks nice … at first. Like most build systems, it turns out to have some huge, ugly warts.”

Any build system will in someones eyes have these “ugly warts”. QMake certainly has a few of them. How we deal with them (reporting/complaining, sending patches) and how the team maintaining the tool handles them is key. It’s OK if a tool has some warts as long as a) you have possible work arounds, b) you can live it the issue, and/or c) the team is actively trying to fix these issues. I’m certain KitWare doesn’t like warts in their products, and are actively trying to fix them. 🙂

Craig Ringer: (about CMake): “I’ve had to copy almost every standard Find module I use into my source tree and modify it to handle one or more basic issues – improper caching of negative results, failure to handle debug library linkage on win32, failure to handle spaces in paths, etc.”

..And I assume you’ve contributed these patches back to KitWare, right?

Craig Scott: “Cross-compiling: I much prefer to compile on the platform itself. Once you have to integrate third party packages that need to be installed to supply bits of the build (eg libraries, headers, etc.), trying to set up cross-compiling gets too hard.”

Unfortunately, this is a reality which we (Qt) have to live with, increasingly. When we start targeting even more embedded systems, phones etc, we often have to do mixed cross-compilation. It’s good that not too many have to do this though 🙂

Uwe: “For me it would be more important to see a better documentation of the existing qmake, than a new build environment.”
Florian: “I strongly vote for rewritting QMake and for staying backward compatible (!). That mainly means that you need to write a good unit test for the existing QMake functionality and keep that running while refactoring.”…”In our company we have thousands of QMake profiles and it would really be a bad decision for us if you stop QMake!”

QMake certainly has a lot of undocumented features, and should be better documented. However, our organization is hitting the limits of its capabilities and maintainability, so we find it hard to continue at the current state. Either we seriously renovate QMake, by fixing its internals and shape up the language, or we move over to another tool and try to shape up that one. I guess most QMake users today would want us to do the former, but the question is then, would it require too many resources to get there; and in what time perspective?

Per: “Allow the easy and familiar ./configure interface to switches with integrated documentation.”

This has been one of the topics of discussion with KitWare, yes.

Chris: “How about a XML-based approach, like Ant?”

XML is too verbose, and impossible to maintain by hand. Yes, many of our users don’t use a GUI for development, and would hate to maintain XML files for building.

cartman:http://code.google.com/p/gyp/ might be the answer to all your problems”

It’s got the JSON part, yes, but a bit young and feature-lacking of a project if we decide to go with something established.

Sebastian: “Long ago, I looked at QMake and I believe at that time it wasn’t able to create full-blown VS project files that allowed to tune all usual settings from within the IDE, but some crippled wrapper projects that just called nmake and lacked the usual settings.”

Just had to respond to this one, since I wrote the whole vcproj/sln generator: QMake never ever created a nmake wrapper for VS project files. The first iteration of the vcproj generator parsed all known options at the time, and placed them mostly correctly in the vcproj format.

QtFan: “I’m curious, when you are talking about changing/replacing QMake, are there any thoughts about changing/extending MOC !?”

Moc is a different discussion altogether. And no, we’re not planning to change Moc. Extending it, maybe. Give us a detailed use-case, and we’ll talk about it.

Florian: “I think if you would create a new tool, we would NOT use it in our company and change to CMake instead, because CMake will never go away.”

There’s no reason why QMake should go away either. It’s Open-Source after all.

gwright: “Personally, I think qmake is great. However, there is one huge flaw with it which is a showstopper for at least me”…”he makefile generator backends for qmake are paired with specific linker versions”

Yes, this is somewhat true. However, it’s not impossible to fix, and we have a developer internally which is trying to cross-compile Qt for Windows on his Linux box using MinGW, and run that result directly from his VMWare image. The applications now run, but missing some graphics 🙂

Ok, that was some of my comments to your comments. Keep the comment flowing, we need to know your opinions!

Thanks!

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Build system

32 comments

poppafuze says:

Sorry about the anon post.

I think lua could be your friend on this quest, given the research I did a while back on build tools (they all suck). It was designed to embed, extend, has a specification (the ref manual), from a beginning as a configuration language. The syntax, parsing, etc. are all nailed down and mature. There are some nice libs; reading how the Shake test framework was done gives an idea of how libs can use each other. It’s light, clean, uses object prototyping (similar to Javascript), fast, has a small executable, and bootstrap-compiles from straight C. One of the build systems (below) does just that as its first task.

I can’t say any of these build systems are even close to being right for you. Some have elements that can be cherry-picked, I think. And they’re likely to be a long, long way from the IDE integration you’re looking for, but hey, couldn’t that use a little lua scripting, too?

http://primemover.sourceforge.net/ — why use JSON or YAML when the config file is a valid lua script anyway, and easy to read. The readme for pm hits some good feature points. It includes a copy of itself as a shell script that includes the lua source internally and bootstraps itself on the target. Single-file deployment.

http://premake.sourceforge.net/about — makes build files for different platforms (in case you can’t escape this)

There are others… http://lua-users.org/wiki/LuaBuildSystems (the hamster link is broken, so I’ll give it here… http://luaforge.net/projects/hamster/)

Maybe the best takeaway from this is a set of examples that use one language but have different ways of specifying their config, states, actions, and desired results. I’d offer that it suggests a base language with a lot of legs for the job.

Good luck!

QtFan says:

> Give us a detailed use-case, and we’ll talk about it.

Sorry for going off topic.

MOC is part of build system so maybe it is a good time to think about
adding some kind of extensions, “injection points” (seams), that can
enable generation of additional meta code.

I think (maybe wrong) that is the easiest way for other frameworks
that need to extend some Qt features to implement their functionality.
For example, I want (theoretically, to hard for one person) to implement
QIoC, a DI/IoC container for Qt, I can use additional meta code to
implement injection, weaving and other IoC things.

Please, be open minded as you are. You already borrowed good things
form other frameworks, so please take a look at Spring and Hibernate
form Java world (just ideas). Sending data form GUI to database and
back should not be so hard and time consuming job.

Once again, sorry for going off topic and thanks for your wonderful work!

Peter says:

CMake+QtScript: I assume Kitware will be skeptical about linking the cmake binary against Qt.
(they already use Qt for the cmake-gui on Linux)

But they are interested in extending CMake with a non-proprietary script-language:

Kitware’s chairman Ken Martin tried to incorporate Lua in cmake,
http://lua-users.org/lists/lua-l/2008-02/msg00403.html
http://www.cmake.org/Wiki/CMake:Experiments_With_Lua
(Lua: one of the fasted script-languages when not JITed, with a footprint of 100kB, famous in the game industry)

A project similar to CMakeScript is here:
http://www.assembla.com/wiki/show/CMakeLua

and a discussion on the cmake list:
http://www.mail-archive.com/cmake@cmake.org/msg12092.html
where I wrote Feb 2008:
“- maybe Trolltech finds it convenient to replace their
proprietary build-file generators, and join forces with
Kitware. Then it is possible that CMake is sometimes part
of Nokia/Symbian projects’ build process”

JubiluM says:


Either we seriously renovate QMake, by fixing its internals and shape up the language, or we move over to another tool and try to shape up that one. I guess most QMake users today would want us to do the former, but the question is then, would it require too many resources to get there; and in what time perspective?

Well I wonder will the second option really be any better? You will have to allocate resources anyway for the massive design and coding if you start from the scratch. And still can’t escape the complexity of the system that is intended to produce makefiles for huge variety of platforms. There will have to be be parsing of the input, logic, and generating of the output irrespective of the implementaition language.

And what do users get in change? Yet, we will have to learn yer another way of maintaining projects and some of us will have to get familiar with the scripting language of your choise (after all, I suppose that most of Qt-users are not webdesigners).

From the user standpoint I’ve found only one relevant point to support, for example Javascript, and that is the potential debugging capability of the build-process. However, why couldnt current qmake be made to be able to output a project build-configuration, let’s say, in structured html format, if you’d give it a -html – switch?

bencord0 says:

I’m starting off in the world of Qt, and my projects so far are quite small to medium sized.
What I like about QMake is that all I have to do is

tar xjvf blah.tar.bz2 or git clone blah.com/blah.git
qmake -project
qmake
make

and everything works fine. Even nicer is that everything works fine if I use the same commands in Windows (cmd.exe not just bash/cygwin), Linux and a Mac when I can get my hands on one.

In the highly heterogeneous environment that I use (windows, linux x86 and x86_64), I use a distributed program called icecream (http://en.opensuse.org/Icecream) which helps to add parallism.
During the ‘make’ stage, whenever a call to gcc is made, icecream intercepts this and passes the compilation job to the network. Each workstation has it’s own iceccd which accepts jobs and sends back the results, and one server has the scheduler which does (very simple) load balancing. Crosscompiling is easy too because icecream sends a compressed tarball of the build environment of the target (to get past any compiler version specific issues) to the computers that can help. Mixed crosscompiling could be achieved by extending this mechanism to include the crosscompiler in the tarball.

I like QMake as it is because it serves my purposes much better than anything else, and I don’t need to write a single line of the .pro file/build script and can just concentrate on coding.

Have you had a look at AAP http://www.a-a-p.org/ It’s Bram Molenaar’s (of vim fame) build tool for vim. We have used it extensively for C, C++, swig and xsl dependencies. Our project wasn’t very big, so I don’t know about performance.

It’s a really nice tool though. It’s Python based, so that might be a put-off.

Greetings and keep up the great work

Adam Treat says:

I vote for fixing CMake so it will work the way you want. JSON/JavaScript whatever. Collaborate with KitWare. I also vote for creating a tool to convert QMake to that will at least get people 90% of the way. Much like you did with Qt3->Qt4.

This time please start with good test suite.

Thanks,

Adam

ClintonStimpson says:

I’d support a decision for CMake too. Let us know if you need help with that.

I also vote for working with CMake to adapt it to satisfy Qt’s needs. CMake served us well from setting up relatively small Qt projects to larger libraries packaged on multiple platforms. It is also well tested by the KDE community across quite a few different platforms.

I recently started working at Kitware, but would have said the same before joining them. I think that this would be a great move for both the Qt and CMake communities. I would love to help make this happen, and work with some trolls on ironing out any issues 😉 (You guys are still Trolls aren’t you?)

I vote for improving CMake to satisfy Qt’s needs. I use CMake to build projects cross-platform, and while it’s a little weird to do it currently, it works fairly well, and I like how cmake works, being able to be automated or interactive in the build process. A thing that many buildsystems miss nowadays is the ability for the user to be interactive during the build process, such as while using ccmake instead the autobuild tool, cmake.

qmake is just a real huge PITA. Each time I have to deal with Qt projects, I hate qmake a little more every time. Cross compiling is a bitch with it, and sometimes regular compilation is as well…

While Lua or JavaScript would probably work well, I just wanted to make one comment about Python. In both of your blog posts, now, you’ve said that Python would bring a bunch of libraries along with it. This isn’t necessarily true. You do not have to bring the entire standard library into your build system. All you would need (if I understand correctly) is the python2X.so (or dll), and then whatever modules you want to include (sys and os might be good).

Or is there something beyond that I’m missing?

spinynorman says:

@Marius
Re: QMake adding library paths

On Linux QMake by default generates makefiles with LIBS = $$QMAKE_LIBDIR /usr/lib $$LIBS
What this means is that if I want to use a version of a library other than the one in /usr/lib I can’t just add “-L -l to LIBS in the .pro file (since the generated makefile will search /usr/lib first, and use the version there instead). I don’t like this sort of “hidden” behavior – I think it would be preferable if the .pro file was the only place where library search paths are specified, and certainly it should be documented alongside QMAKE_LIBDIR/LIBS if that’s not the case.

JubiluM says:


qmake is just a real huge PITA. Each time I have to deal with Qt projects, I hate qmake a little more every time. Cross compiling is a bitch with it, and sometimes regular compilation is as well…

I get the same feeling every time I’ve been forced to work with CMake. Even accomplishnig simple tasks can make you nuts because of some weird CMake anomaly.

And from Qt standpoint it would not be rational to start relying on some 3rd party solution that doesn’t carry you any further than qmake. qmake allready satisfies most of its users’ and Qt’s needs and has a wide userbase. I’d say that looking at the issue from many viewpoints, the way to go would be to focus on qmake. It is great already as it is. Improve it, redesign on some parts and fix what is not working.

And it would be great if Kitware and CMake community could start assisting to improve qmake. After all, there are several users using Vtk and Qt interconnected.

JohnA says:

I’m coming late too this party but what about the approach SCons has taken? (http://www.scons.org/)

Too Python-y?

Sylvain says:

It would be the best for Qt to build a new tool, free of any history, based on their experience on QMake.
… and a new tool based on javascript + JSON seems very promising
I am very excited to see such a tool.

I never liked CMAKE, I found it always too complex and I loved QMake because it was simple and completely managed the Qt specific parts.

Philippe says:

Nokia/Trolltech has always kept controls on its tools, eg. moc, linguist, qmake and recently QtCreator. Not to mention Nokia that has acquired Trolltech 😉
It would be curious to take another direction and depend eg. on CMake. My vote goes for a large refactoring of QMake, though I completly ignore the ins and outs. But I guess that everything you have done in QMake is not to throw away!…

SaroEngels says:

qmake needs some more fixes which cmake already includes – shadow builds with qmake are just a pain (and seem hard to maintain), and this is rather easy to have in cmake. CMake provides ways for us to compile not only KDE but also many 3rdparty libraries without the task to maintain a different build system for each platform. Qt can gain from these advantages of cmake especially when it comes to compiling its third party libraries (phonon, webkit, etc.).

Max Howell says:

Honestly, it sounds like you know the situation very well, and have excellent and proper aspirations. The other day I said you should build on an existing solution, but having reread your first post and this I frankly think Qt should roll their own and show the world the build system we all yearn for.

And regarding CMake, my admitedly sour opinion is mostly due to its ugly syntax and semantics and its custom language that is strange to use, painful to learn and full of bugs. So if the bugs were fixed and the language changed to Javascript, I’d like it a lot better.

However, I think you’ve already secretly decided to opt for a solution that doesn’t act as an intermediary to another build manager, which seems correct to me. Well perhaps, you’d still have to generate VSproj files, unless someone at MS gets a clue and makes their IDE more pluggable. Same goes for Xcode to some extent.

Thiago Macieira says:

To all that have expressed their votes in the blog: this is not a vote. 🙂

If you want to participate in the decision-making, you have to start doing so with code. Marius’s earlier blog listed the features we’re looking for. There is no buildsystem matching the requirements, so we’ll have to work to improve the existing ones (or write a new one) to match those requirements.

There are must-have requirements and nice-to-have features. I’ll let you find out which is which by common sense and by looking at what we’re trying to achieve with Qt.

Nach says:

When I think of a build system, I always find that the basic GNU Makefile syntax fits closest with what makes sense. The reasons why we have various wrappers around GNU Make is to assist with various configuration options, auto dependency generation, avoiding boiler plate code, simplifying certain cases which GNU Make’s syntax is complex for, and making certain more common exotic cases easier than writing it all in a Makefile.

While I find many solutions out there (autoconf/automake, qmake, cmake, whatever) all try to assist with shortcomings of writing everything directly in a Makefile, they add their own overhead and difficulties in many ways.

Cross compiling may not be very popular with some, but are bread and butter for many projects. Cross compiling generally amounts to calling a different compiler executable for a particular language, or using a special switch when calling the main compiler, as well as specifying some different paths, different object names, and some other platform switches.

For QMake, I have to create a whole new qmakespec and add it to my system directories to add a cross compiler. Then each developer on my project has to do the same if they also want to cross compile.

Contrast this with a Makefile, where all one needs to do is use a few variables, and add a few conditional statements to the build script which everyone is already using. Other developers should be able to download a project’s source and install whichever dependencies they need to build a project, without needing to modify system files.

When I played with CMake, it seemed even more complex than QMake to deal with in this area, and had some strange bugs which I couldn’t even pin down.

Another thing is that not every project will be C, C++, or other popular languages, or may be a combination of several languages, popular or not. Perhaps a company will have a special preprocessor they use in conjunction with a particular language. A build system should make it easy to add on a new compiler to handle building certain objects. GNU Make again makes this easy, I only have to specify which file extensions get handled by which compiler, and how to output them. For QMake, I didn’t even find information on this subject documented, and instead had to review the source code to QMake to figure it out, and even then it was a lot of trial and error, and seems more complex than it should be. CMake was even worse in this area, as one has to add files to their system directory.

This also gets more complex when the build process needs to build helper compilers/preprocessors in the process of building the whole project. GNU Make manages to figure out the dependanices right every time. For QMake, I could only get it working right by having it call an external build system to handle building the custom compiler, then feeding it all the objects necessary. Otherwise it tried building objects prior to the compiler for them being built.

As others said, this also needs to play very well with building outside libraries. All too often a build system can play nice with sub-building other components designed to build with the same build system, but fall flat when it comes to managing something alien.

The build system that would make me happy is something that would be easy and practically automatic for simple cases as QMake currently is, handle dependency generation (near) perfectly, understand the needs of more demanding cases, and not make them harder to implement than it would in a less advanced build system. One example would be allowing the build system to understand to run a certain developer supplied command to figure out dependencies for a certain language. Such as “nasm -M -DPLATFORM=%platform% ” and parse the Makefile dependencies it outputs.

The system should also support being able to add multiple passes of build options on certain files. Such as one source file producing two different objects, one time built with one set of options, one time with another. Or after creating a binary, run a compressor or code signer on it.

There’s also a couple of less important wishlist items I would have. it’d be nice if there’d be a way to tell the build system to build multiple targets at once and have them coexist. Such as “build linux cross-windows cross-freebsd”, and have it put everything in linux/ cross-windows/ and cross-freebsd/ as needed. Also package generation options, such as packaging executables with documentation and artwork into a single archive. Packaging the source tree into a single archive, but knowing to strip out directories which contain repository information, and *~ files.

It is unfortunate, but many advanced build systems can’t compete with GNU Make in ease of use for some cases in terms of adding a building routine. Some don’t allow calling external commands simply, such as “wget” to download some files, or “upx” to compress a binary when the build is done, or other cases I highlighted.

It seems unlikely anybody was expecting they were casting democratic votes, merely providing user feedback. I would hope that code commits aren’t the only metric being used to gauge user feedback.

Back on topic, I’ve been over to the Kitware shop, and was very impressed with their rigorous development methodology. The company I was working for at the time foolishly passed up an opportunity to work with them. Great crew.

Javascript does seem like the obvious choice.

spinynorman says:

I think JavaScript (or rather QtScript) is only the obvious choice in the sense of being the easy choice seeing as Qt already supports it. However, being the easy choice doesn’t necessarily make it the right one! I do expect it’s what we’re going to end up with regardless since Qt seems to be rapidly moving away from it’s desktop roots towards web/embedded (i.e. smartphone) usage, and QtScript therefore makes more sense than if it was still primarily targeting the desktop.

What’s going to make any QMake .pro file replacement easy to use is declarative vs procedural semantics and a close fit to the application domain (which really implies a custom mini-language). No doubt QtScript could be bent to do the job, but given the amount of usage that any development tool sees, I think a custom language is called for it it’s going to make life easier for us users (as it would do if well designed)!

johan says:

JS/JSON definitely shouldn’t be used in build system. Instead of this, CMake can be fixed to meet your expectations maybe?

Marius says:

@johan: There’s no arguments in your comment. Why do you think JS/JSON shouldn’t be used?
@spinynorman: The thing is, most build system which use a declarative language have some way of adding your own ‘custom tool’ or script, since the declarative language doesn’t handle everything a developer wants. So, instead of having it as part of the build system language, they add hooks into the system, to please the users who point out a issue/flaw with their designed declarative language. Then suddenly, instead of having a nice clean special language, which *was* designed to handle *your* cases well, is full of cryptic custom tools/steps to build a project properly. That’s why I rather like the JS/JSON idea since, 1) It already contains a declarative part which we can use to satisfy the IDE manipulation parts of the projects, and 2) has a well defined imperative language to handle all the special cases needed for all those different needs.
@bill_mcgonigle: No, code is not required to give good feedback. We do like patches though 🙂
@Nach: Simplicity is what we aim for. QMake is rather simple to use on medium projects, but gets rather hairy on larger projects. If we move on to a new system, we want to keep the simplicity.
@JohnA: Scons has a few issues with it, 1) Python, so we’d require Python on all platform which we support Qt, and require them to install that before Qt, 2) It’s very slow (though Waf is faster), 3) It has fewer backends than for example CMake, so if going for something already there, things like that count.

Anteru says:

I’d also vote for fixing CMake, mainly because even though CMake has lots of rough edges, it’s still the “best of the breed”. Adding a scripting language to it seems a natural extension, and JS sounds good for one big reason: It’s standardised, so there won’t be any changes to it — unlike Lua etc. This makes long-term support much easier.

Why CMake: Because even with all its flaws, it does work for quite a large amount of people. The Boost guys managed to build Boost with it, KDE is working with it — it can’t be that bad, and it shows that it can handle rather large builds as well. Getting to this point from scratch is going to take a large amount of time … and I really have the feeling that CMake just needs a proper project plan to get the required functionality (mostly front-end stuff, like the scripting interface), while the underlying ideas of CMake are sound and should be kept. So far, the guys at CMake seem to be very focused on backwards-compatibility, which IMAO hurts the progress. Maybe CMake 3 could go forward and break with this.

Re Python: While Python is great, I’d also vote against something which is not embedded into the build system. For instance, I have tools written in Python 3.1 now, and I don’t want to install Python 2.6 just for QPmake and translate those stuff later to Python 3.1; adding Python also introduces huge amounts of possibilities to side-step the build by adding custom logic, which is not a good thing as the logic starts to depend on the build system internals suddenly (something I have seen quite a lot with SCons).

ClintonStimpson says:

Has there been thought into exporting build information for those using Qt? CMake has the capability to create export files with information such that including it into another project is as simple as pretending it was part of the project.

The current method of CMake asking qmake questions about Qt, and parsing .pr* files is really awkward and doesn’t give enough information to support everything people have wanted to do. I mean, why did the QT_INSTALL_* paths get embedded into qmake, but never put into a text file?
Its a show stopper for cross compiling unless one plays around with the Qt installation. I did that a while ago, cross compiling an application that uses Qt.

Sure, its good to think about a build system for Qt, but one should also think about users of Qt.

So, if you’re considering JS/JSON, how simple are the build scripts going to be for the Qt examples? How simple would they be for any app using Qt?

superjordo says:

Please don’t roll your own unless you’ve got the backing. There are already too many systems, one more mediocre system will only muddy the waters. (FYI Even if it fits your purposes completely it can still be mediocre.) However, if Nokia/Qt Software management is backing you, and you feel like you can make The Next Big Thing, go for it. I have confidence in you guys with the proviso that you have the proper support.

Maurice says:

Please, if you reinvent a wheel : make it with the idea to fully replace CMake et other autotools.
I wouldn’t use a special tool for every library I use…
Here are my two favorite choices:
1 – use CMake and make it better (if CMake guys agree),
2 – (if CMake guys disagree above) fork CMake and add your missing feature, script langage, …

A nice feature would be to allow binding with most common languages (e.g. Javascript, ruby, lua, tcl, pithon, …)

Also, please don’t make error like Makefile by choosing a syntax with heavily depend on the blank characters (or tabulation ;-)…

JubiluM says:

@Marius: “Why do you think JS/JSON shouldn’t be used?”

And what are the arguments of using JS/JSON?
Debugging capability? Well I wouldn’t want to debug yet my my makefiles :). Seriously, could it be implemented also to the current qmake….for example dumping the make setup to the well structured, browsable html-file or some other similar idea?

Because of that JS/JSON as a language is “getting popular”? I still think that most of the Qt-developers are doing most of their job using C++ and are familiar with it. And make alike syntax is in the blood, kind of.

More, qmake (*.pro/*.pri) makefiles have a clear and easy visual representation, and it is rather easy to comprehend what’s going on in the makefiles. They describe the projects structure itself and that’s how it should be, in my opinion. If you make the makesystem script-alike, its more of a representation of programming the project and the project itself loses some the focus, visual appearance of makefiles clutteres bacause of the added progamming elements etc…

If support for scriptability is needed, an easy way could be added to execute scripts, Python, Lua, JS…could it? And…correct me if I don’t remember right that one can execute system commands from (q)makefiles, so can’t the feature already be used to execute scripts, if wanted?

Easier IDE-manipulation of the projects using declarative? Do you mean to embed gui-descriptions using declarative, into the makefile….if that’s it, I’ll run for my life :)! I mean, that would be a mess. Could you please share some light on the idea behind the argument…? Sorry if/for not “getting it”.

“Simplicity is what we aim for. QMake is rather simple to use on medium projects, but gets rather hairy on larger projects. If we move on to a new system, we want to keep the simplicity.”
Totally agree on the simplicity. But…will the possible change of the language/syntax of make-system/makefiles make the maintenance of larger projects less hairy? I mean, one still can’t get rid of the fact that one has to describe the project and all of it’s dependencies, rules for different platforms etc. etc.

Big projects will always be hairy, what counts more is how it is presented to the user….using IDE or makefiles. Why we fell in love with the qmake was it’s simplicity, clear syntax, easy maintenance also of bigger projects, it’s features…and the list goes on.

Extend, improve and redesign on some parts (plus learn from Competitors’ MistAKEs ) and you have a make of all times :)!

Jeremy Andrus says:

Perhaps I’ve missed something, but one of my biggest annoyances with the current QMake system is that when building IDE project files (specifically XCode and MSVS) there doesn’t seem to be a way to output _both_ your debug and release settings. These IDEs make it easy to graphically switch between different build configurations, and it would be nice if future versions of your cross-platform build system could integrate more completely in this area. Standard GNUMakefile generations works perfectly (albeit a bit clunky): you “make debug” or “make release” and everything happens the way you expect. This seems to be accomplished with multiple passes over the *.pro files – could you do the same thing for IDE project file generation?

Timothy Murphy says:

Hi,

I work on the Raptor build system for S60 and we are beginning to look at what we can do to help make the Qt/S60 experience better for people. Your decisions on build systems are going to have a major impact on us so I hope you don’t mind me putting in a couple of comments.

Raptor has a heck of a lot of S60 knowledge embedded in it – particularly in it’s makefiles – and has taken years to approach correctness so it’s not something one wants to dump straight away. I suppose I am biased. It also sort of good for building really big things and to do that successfully it has to be the one on top, as it were, not run by another tool but the one that’s in charge of the whole picture.

The problem is that it’s always going to be ugly to generate Raptor input files from something else like qmake, cmake or whatever. The mapping will always be a source of some sort of problem. So I was thinking that perhaps we could do something about adding support for whatever new system you come up with into Raptor so that there is never any need to generate the S60 file formats at all – i.e. read the info directly. I am speaking purely for myself here and utterly unofficially, but some of us would love to replace our legacy input formats with something more thought out. The complexity that they contain is needed but it doesn’t have to be so opaque and obscure and the same concepts don’t have to be implemented in different ways for each format. We use python so we like that better than javascript, but I don’t suppose its an insuperable problem.

To give you an idea of what we need:
* A hierarchy – projects, components, ordered layers (of components), systems (the whole OS and applications).
* Ability to describe what types of platform/architecture each item in the hierarchy should be built for.
* A very powerful way to represent the many phone models, architectures and toolchains which can be applied to each project to produce the different versions of that binary for different hardware. I.e. a way to express what settings apply to a particular project or component or layer or system or phone model.
* Some way to customise options for a particular project, for a particular “variant” of that project. e.g. “when fred.exe is built for armv7, set this compiler option X on rvct but Y if building with gcc”
* some way for the nodes in the hierarchy to communicate e.g. I write a component that supplies the settings needed to build a special type of program – a new kind of plugin dll or something – this needs to be put in some common known location before other projects can describe themselves as a “newplugin”. In symbian we call this “exporting” but in linus it would simply be “make install” except that in Symbian you do it before you build, not after you’ve finished (otherwise you couldn’t build everything in parallel).

I think that’s all I really need to say for now. The rest of what I have typed below is my rant about an ideal type of build system and now that I look at it again I’d feel guilty if I didn’t arn anyone about it being a rant. :-).

Regards,

Tim

I think that @Nach above made a good point about how powerful GNU make is and how we only avoid it because of trying to not have boilerplate code. We also try to hide complexity from people who don’t want to know it and we need a way to verify that people have put in a configuration that has a chance of working. Make doesn’t supply all of this but if it was just a little more powerful – had a few more functions – then you probably could have an awesome build system written directly in make.

So the usual choice is: massive power to describe complex tasks (the whole build system in make) or “noddy” (oversimplified) configuration files where you can’t build anything that isn’t standard or apply any variation that hasn’t been catered for.

What I am looking for is something that is multi-resolution. Something that can be used to describe the logical structure of some collection of software. For me it has to scale all the way from being able to describe a whole OS or, what the hell, several OSes, together with all their applications plus all the actions needed to access the source and produce installation packages and ROM images down to a simple project that make one binary. You shouldn’t have to repeat detail (boilerplate).
e.g.

p = Program(barney, sources=”barney.cpp”)

… should be enough for the common case and this should work across all OSes.

When there is something OS or architecture specific it should be simple to add it::

p = Program(barney, sources=”barney.cpp”)

with a=p.aspect(e32binary) {
a.uid3 = 0x10101010b
a.paging = data
}

with l=p.aspect(armlinux.armv5) {
l.armlinux_aspect.rvct.setcpu(5TE)
l.armlinux_aspect.gcc.setcpu(5TE)
}

The most important thing is that one should be able to define new standard build objects without modifying the build system e.g. some component in the build should be able to define a new type of program:

MySpecialGraphicsProgram is a NormalProgram with ExtraOptimisations set to on

Or something like that. The effect is to cut boilerplate down much more dramatically by letting everyone do it with ease – not just people who know the build system inside out and have commit access to it.

In other words, I’m thinking of some sort of object oriented way to add support for building new things so that no-one has to wait for the build system developers to add support but they can just get on and do it themselves if they want and it’s easy because they can reuse some existing thing.

The in-memory-representation of this information, once read in, should be useful to tools that want to do more than just build – e.g. generate documentation or re-factor the entire OS plus applications at a button-click in the way that IDEs can now help you re-factor Java or let you explore a visual tree representation of your system and it’s output.

This seems a lot like SCons to me but that has a reputation for being a bit slow. It’s also trying to implement the actual build whereas I am not worried as much about that – I mostly care about how to describe what you want so that you can play around with different ways to generate it most effectively. (e.g. write makefiles or generate machine code or whatever people come up with).

Anyhow that’s it for the moment. 🙂

Rubenvb says:

Ever since I learned Qt and qmake, I love it! Simple cross-platform syntax and great moc, uic and other implementations. Even my more complex projects had a kind of simplicity to them, that other project managements just can not have. Some gripes I have with the current qmake:
– when using qmake for non-Qt projects: makefiles cannot be used without the presence of the original Qt directory. Can qmake not take care of all the properties in the mkspec configuration and bake them into the makefile, as to eliminate the implicit Qt dependency in all qmake projects? I’d like to “qmake -macx -r” and be able to zip the whole project, open on a mac, and build by execcuting “make” (if the same versions are set and used in qmake and on the mac). Thàt is “code once, compile anywhere” (I know this isn’t Qt’s slogan exactly, but that’s what it should be!). I understand qmake isn’t a complete project management system, but it can be, with a few tweaks. A big step in the right direction would be to move all Qt-related paths in a makefile to only appear when CONFIG += qt is set.
– no way to check the (target) platform (x64 vs x86) and thus no way to implement specific preprocessor defines to use architecture-specific codepaths or let nasm operate in the correct bitness required.
– it’s already been said: better documentation. qmake is a powerful tool, but the docs could be better.

I’d love to have qmake stay, perhaps a complete make-over, with at least partial backwards compatibility. It’s the only cross-platform system that I can rely on to just work. But it should be made to work with non-Qt projects without the need for a full Qt source/SDK install.

Commenting closed.

Get started today with Qt Download now