Tobias Hunger

CMake support in Qt Creator (and elsewhere?)

Published Tuesday November 15th, 2016
35 Comments on CMake support in Qt Creator (and elsewhere?)
Posted in Dev Loop, QtCreator | Tags: ,

Kitware released CMake version 3.7 on Friday night. There is one feature mentioned at the very bottom of the feature list that makes this a really exciting release for people writing tools that integrate with CMake: The server-mode.

What is CMake server-mode?

The Qt Creator team has had contact with the CMake community for a long time now and was actively involved in discussions with CMake and other IDE and tools vendors. After many discussions the CMake community decided that the best way forward to support the use-cases of IDEs was to add a new interactive way to interact with CMake, the so called server-mode. There was some initial development started by Stephen Kelly, who made an impressive prototype.

At this point the Qt Creator team decided that this functionality is crucial to enable first class CMake support in Qt Creator. So I went ahead and jumped into CMake development. The result is a set of patches that implement a production-ready server-mode for CMake that was merged in time for the CMake 3.7 release. The server-mode currently focuses on core functionality like parsing the project structure, etc.

This development happened in close collaboration with developers from KDevelop and the CMake community.

I am committed to maintain CMake server-mode support going forward and hope to continue to develop this feature in close collaboration with all interested parties.

CMake version 3.7 is the first release that ships this code.

What does CMake 3.7 enable?

The first change in CMake 3.7 is that it now provides machine-readable output with basic information about the capabilities of the CMake binary (via “cmake -E capabilities”).

This includes static information like version number or whether server-mode is available, but also details on the supported generators.

IDEs can use this information to provide a better UI for selecting generators.

CMake server-mode (started via “cmake -E server”) allows IDEs to query a range of CMake settings and project information in a machine-readable form, many of which were not available to IDEs before. This enables a more detailed view of the CMake project.

The information is also taken from CMake directly, making sure to use the same code that is also used by the CMake generators to write the actual build-system files. So the information is more accurate than what was available before. This makes the code model more exactly represent what is actually going to be built.

When will we see these improvements in action?

I think the server-mode is the way forward for all tools that need to extract information from CMake. Interest in this topic seemed very strong in my workshop about CMake at QtCon this year. I hope to see CMake server-mode flourish going forward.

There is code that makes use of the new CMake 3.7 features in the master branch of Qt Creator. So this will be present in Qt Creator 4.3, planned for spring of 2017.

What will improve in Qt Creator?

Creator will continue to support any CMake version 3.0 (or later), just as current versions of Qt Creator do. But Qt Creator master does support CMake server-mode already and will make use of the better information. You will be able to switch between CMake binaries with and without server-mode support as needed.

Using CMake server-mode enabled binaries will have the following effects:

  • Generator selection is no longer limited to those generators with support for the “CodeBlocks” extra generator. Any generator can now be used.

    CMake Generator setup in Qt Creator

    CMake Generator setup in Qt Creator

  • The Generator selection UI now knows which options are available for each generator. E.g. the “Ninja” generator used above does not support Toolsets or Platform settings.
  • The project tree is much improved and knows about projects and build targets defined in CMake."<yoastmarkThe screenshots show the project tree from Qt Creator 4.2 and from Qt Creator master — with an old CMake and with CMake server-mode.
    Qt Creator master adds all headers found in include paths of a build target and not listed in the CMakeLists.txt to the project tree (for all versions of CMake). These headers are printed with a grey text color so they can be distinguished from sources known to CMake.
    In server-mode the project structure is more visible: The “scribus” project defined in the top-level CMakeLists.txt file is clearly visible. So are the different build targets. The “scribus_colormgmt_lib” build target is expanded and all files belonging to that target are listed.
  • The code model has way more accurate information about all targets. This sounds very simple, but has not always been the case for CMake projects up to now.

What will happen next with CMake server-mode?

Stephen Kelly has pioneered interesting features in his CMake server-mode implementation prototype. Many did not make it into the new CMake release.

Going forward I would like to see more of these powerful features. This includes debugging of CMakeLists.txt files and syntax highlighting driven by CMake itself. These require deep integration into CMake and will require a much deeper understanding of CMake internals than I currently have. So I hope CMake developers find the server-mode interesting enough to help out a bit here and there:-)

I would like to say thanks to Aleix Pol (KDevelop) and Stephen Kelly (CMake community) for all the feedback they provided. Finally I have to thank Brad King from Kitware for his patience while  reviewing my code.

Where can I get this?

Get CMake 3.7 here: https://cmake.org/download/
View the Qt Creator master sources: https://code.qt.io/cgit/qt-creator/qt-creator.git/
Or use git to clone the code: git clone git://code.qt.io/qt-creator/qt-creator.git

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Dev Loop, QtCreator | Tags: ,

35 comments

Milian Wolff says:

Many thanks Tobias for driving this forward. Much appreciated!

MetalMajor says:

Super super awesome!
Maybe support for renaming files would also be possible then?

We will see whether CMake will add such functionality to its server-mode. If that shows up there, then Creator will support it.

I am not too optimistic about that feature landing anytime soon though.

illva says:

You can’t…. rename files in 2016? ahahahahahaha

Qt Creator supports a wide range of build systems and can add/remove/rename files, provided the build system back-end supports that. Not all do.

I will be more than happy to add that as soon as CMake server-mode makes that feature available. I will help to implement that there, but this is again something that touches a lot of CMake internals and will need a lot of help, feedback, reviews and fixes from core CMake developers (if it can be done at all).

Lilian says:

I observed that recently QtCreator master crashes(it does not crash with QtCreator 4.2-beta1) on some CMakeLists.txt files.
Might it be related to this, or this was already available in 4.2-beta1?
I use CMake 3.6.2 btw.

None of this is in Qt Creator 4.2.

So any crashes you see might be related to the changes necessary in Qt Creator to accommodate CMake server-mode (and being able to switch between a non-server-mode CMake and a server-mode CMake). The code did not got into the 4.2 release exactly so that we have time to iron out bugs.

I would appreciate if you could find the time to file bug reports at https://bugreports.qt.io/ with information on how to re-produce the crashes you are seeing (e.g. by providing a link to the project if that is open source).

Roland says:

Thank you very much!
I tried it immediately but there seems to be a problem. Creator lets me do my Git work in a terminal and it should also let me do the CMake work in a terminal. If Creator meets a fully configured project it should accept the project as is. It may extract information, maybe create a temporary kit with the extracted information. For sure it should not overwrite the settings I have made with information from the default kit. This happened when I tried latest Creator from git/master. The build folder was destroyed (CMAKE_PREFIX_PATH was gone and other damage) when Creator was done with it. In general version 3 used to accept manual CMake work, the 4.x series is problematic for people with several manually configured build folders.

Qt Creator 4.x will push its settings into the build folders. That is by design: Users do expect the settings they make in the UI to be reflected in the builds they do.

And yes, Qt Creator does miss the functionality to import an existing build directory and set up a kit with exactly those settings. That is QTCREATORBUG-16899 for your reference:-)

But there is no way to not run CMake from Creator and have up-to-date information, so Creator will continue to do so (at least once when opening the project). CMake server-mode is not going to help with that: You do not even have files with the necessary information anymore and need to run CMake to get *any* information on the project. And that will change the build directory, as CMake is not (at least at this time) unable to extract any information from a build directory without also updating the files it manages in the build directory.

Having said that: Creator should leave *most* of your settings alone. To force it to leave all of them alone, you can change the CMake configuration settings in your kits. You get a list of all the settings that Creator applies there.

Roland says:

You are probably talking about QTCREATORBUG-16802.
It is easily possible to read CMake information without running CMake. CMake-Gui always does this. The information you need is in the text file CMakeCache.txt in your build folder. It contains CMAKE_PREFIX_PATH (=Qt location), compiler, build tool (Ninja/make). Simply all properties the user has set and wants to keep.

Creator does keep CMakeCache.txt intact — with the exception of the variables it has to set up to tell cmake about the compilers and Qt versions to use.

If you do not like that, you can configure your kit to not override any of the settings.

Setting up a kit with the information in CMakeCache.txt is missing, I already admitted that. I hope I can add that functionality in time for 4.3.

Thomas Sondergaard says:

Why a server-mode? Why not make cmake a library?

That is a decision of the CMake project, I just ran along with it:-)

But using a process does have some advantages: Having the CMake code in a separate process means that it can crash without bringing down the UI. Not that I expect CMake to crash, but bugs do happen, especially when you use a tool in ways that are not so well tested (e.g. a command line tool in server-mode;-).

It also enables to run the CMake binary you to build on the command line, so that you get exactly the same results in the IDE and in your terminal/build server/CI system.

Finally you can have different versions of CMake for different projects, which would be impossible with the IDE linking to one CMake library. You would be stuck with whatever version that is for all your projects then.

Philip S says:

This is nice to see! I was using cmake to build a large qt, c++, and fortan project. I recently switched to qbs and I’ll probably never go back to cmake. Cmake integration never seemed to work with my project very well… Overall, it’s nice to this development!

Carl says:

Good to see someone else doing the C++/Qt/FORTRAN blend. I tried QBS but bailed and went back to CMake, so VERY glad to see this.

Christian Kandeler says:

Hi Philip,

did you write your own qbs module to support the Fortran parts of the project? If so, would you be interested in contributing it upstream?

Mirko Boehm says:

Excellent, nice work! I think the decision to go for a server process instead of a library is the correct one, too. Keep it up! 🙂

Matthias says:

We have _many_ build targets. Is there a way to filter them out in the overview?
The tree-view will be cluttered otherwise.

Not yet.

Why would you want to filter them out? If they are not important enough to list, why do you have them?

All the files are attached to the build targets. What should happen with the files of filtered targets?

Matthias says:

We have more than 1000 build targets, all following a certain naming scheme.
Reducing the number is not feasible.

Having 1000 targets displayed in the “overview” would destroy the overview’s usefulness.

People who use QtCreator at work know what their desired build target is named and press Ctrl+K to type “cm my_target”.

Matthias says:

Maybe I was not specific enough.
What would be great is, if it is possible to hide all build targets in the overview.

Slava says:

Thank you guys, that is a really great news! Something I wanted to do myself for a long time, but never set myself up. A bit of polish to clang code-model and analyzer and it is going to be the perfect c++ code editor (and not just best in town as of now 🙂 ). I am really exicted to see it in action.

waddlesplash says:

I dunno. I rather like the present style “Projects” tree view, and I don’t think I like the newer model as much. Could the old way stick around as an option? Or could we at least get a tree-view “File System” panel, if not?

It represents the build system structure much more closely, and representing the build system structure is what the whole tree is made for.

*If* we can make CMake support manipulations of the tree (e.g. add a file, etc.), then such a close match is required so that you can actually specify exactly where you want your files added. So I do not plan to make this optional.

We do have bug reports open for a customizable tree view for navigation purposes. That is something we probably should add at some point, but entirely unrelated to the build system visualization.

Ryan Mulder says:

This looks like a wonderful improvement, and I’m eager to try it.
Do you know if QtCreator + CMake will now be able to support compiling a single file?

AFAIK CMake is not able to compile a single file, so Qt Creator will not support that either with CMake projects.

Alexander Drozdov says:

Hi Tobias!

Thanks for work. But… is it possible to add ability turn on “classic” look and fill for project tree with CMake Server Mode? I found that new look and fill unsusable for:
1) big project (out project contains about of 200k source files for different targets)
2) big project with complex structure and to much count of targets. Like LLVM/Clang.

New look and fill now more differ to directory tree, but we must still interact with browser and console for file adding, removing, renaming, copying and so on and manually fix appropriate CMakeLists.txt and others. This differences adds complexity to switch between project and file manager/console.

Also, with classic view it is simple to add ability add/rename/remove files to file system with user notify, that CMakeLists.txt should be updated manually if needed (I have changes that do that, it can be uploaded to Gerrit if needed). Idea: currently we have no way update CMakeList.txt, but we can simplify life of developer omits switching between IDE and file manager/console/etc.

My suggestion for targets/lists in project tree:
1) move it to separate subfolder, “Artifacts” for example : this interface can be used in future to assign files per target (but I think that it will be never), but currently it can helps to inspect which files assigns to some targets. Assing appropriate icon for that Virtual Folder.
2) display classic tree in general way.

Have you actually tried the new tree?

You still have the tree of CMakeLists.txt files, typically forming a structure very similar to your directory layout. The targets are attached to the CMakeLists.txt files and their files are below the target. That is pretty similar to the previous layout: The files tend to be in the same directory as their CMakeLists.txt files. If they are in subfolders, then that structure is preserved.

In the end you got the targets and projects as additional nodes in the tree. Compared to the number of files the number of targets is small — in almost all projects.

And no, I have not yet planned to enable switching back to the old tree.

Alexander Drozdov says:

> Have you actually tried the new tree?

Yes. I use master version of the QtC and maintain package in PPA.

> You still have the tree of CMakeLists.txt files, typically forming a structure very similar to your directory layout.

Yes. But I can see only directories with CMakeLists. If project contains only one root cmake lists, all files hidden by the some target. By the way, if CMakeLists.txt hidden deeply, now name of folder too long.

We discuss it in mail list: best work with cmake project – work with source tree as is. With additional info from cmake, like: targets to build, files per target for code model. No exclusion. CMake Server mode only standard way to omit CodeBlocks generation with more correct and structured info. And this info also read-only. And I do not believe that this behavior will be changes in near (100 years?) future.

By the way, currently you scan source tree for ALL files. But adds only headers, not all. Way? I think that user should have a way to add exclusion per-project for scanner based on MIME and Glob mask. But all scanned files should be displayed too. See to Clion, KDevelop.

> Compared to the number of files the number of targets is small — in almost all projects.

Look to the LLVM test folder. Too much targets! But no files at all 🙂 A lot of Utility targets without files also annoying.

> And no, I have not yet planned to enable switching back to the old tree.

Do you ready to accept such changes if it will be done by other people? You known: I do changes that needed by me 😉

Another open questions:
1) file adding: VERY very very needed ability. Currently, it more-less simple to implement with classic view
2) display all files in project tree. Not only provided by cmake. There is problem: for big projects tree regeneration requires so much time (mostly in ’emit fileListChanged()’).

Great that you actually already use this! Please keep the bug reports coming, I am sure there are a lot of issues to be fixed still:-)

Alexander Drozdov says:

Heh, I only work with my, “enhanced”, version of CMake Project Manager. It provides two major features for me:
1) all files in source tree
2) adding/renaming/removing

I regularly sync code base with upstream and for some annoying issues submits patches.

Currently, changes in upstream too complex, and my changes should be dropped and rewriting from scratch. It requires time. So, currently my plugin sources freezed and only adopts to compile with latest QtC.

Both of features above simple to implement for classic (old) look and fill, but for new one… I don’t know. Tree scanning and displaying should be customisable per-project too, it does not completed yet in my code, but it simple to implement for any look and fills.

Currently I have one note (it is not bug), according new tree scanning: source tree more/less permanent. It should not be keep per-build configuration. It project-specific and should be kept per-project and shares between configurations. Also, it should not be scans for every cmake run, but CMakeLists.txt changes is a good reason to do it (but build options – no).

Alexander Drozdov says:

Also, compromise variant:
Classic (old) tree with non-utility targets displayed above CMakeList.txt. This targets can be “sub-foldered” to CMakeLists.txt node. Targets also can sub-folder referenced files, but, in my opinion, it is not required (see KDevelop tree).

@Alexander Drozdov: How about lending a hand at getting all your plugin’s changes upstream, so that you no longer need to maintain it? 🙂

Alexander Drozdov says:

First one: separate processing for C and C++ flags in TeaLeafReader already in Gerrit 😉

Another need discussion:
1) Tree scanner. I can provide more-less complete solution only for TeaLeadReader. For big projects (in my case about 280k different files: configs, yaml, xml, sources, liker-scripts and so on) there is bottleneck, which freezes GUI for a long time: “emit fileListChanged()”. I do not known which runs when this signal emits. Also, is it possible to update tree gradually? For example: Tree scanner collects 1000 files, push it to list. Seems that tree scanner should only displays files in tree: no code model updates required.

2) Add/removing/renaming. It works only for project tree. CMakeLists.txt kept unmodified. Solution has a sense with previous feature only. Current implementation runs CMake on every changes, and this behavior unusable for Big projects. But new (in testing now) just changes project tree and outputs user notify. Also, it works only with classic view now: I have no ideas how to adopt it for new look and fill.

The freeze when updating the project tree is on the todo list for 4.3. I think I have an idea where the time is spent, but this needs some more profiling to validate.

Actually you should be able to stop cmake from being run for you (except during project start-up) with Qt Creator 4.1 and later. Check Tools>Options>Build & Run>CMake. You should be able to change that for each cmake command individually.

Commenting closed.

Get started today with Qt Download now