Building and testing on multiple platforms - introducing minicoin

When working on Qt, we need to write code that builds and runs on multiple platforms, with various compiler versions and platform SDKs, all the time. Building code, running tests, reproducing reported bugs, or testing packages is at best cumbersome and time consuming without easy access to the various machines locally. Keeping actual hardware around is an option that doesn’t scale particularly well. Maintaining a bunch of virtual machines is often a better option - but we still need to set those machines up, and find an efficient way to build and run our local code on them.

Building my local Qt 5 clone on different platforms to see if my latest local changes work (or at least compile) should be as simple as running “make”, perhaps with a few more options needed. Something like

qt5 $ minicoin run windows10 macos1014 ubuntu1804 build-qt

should bring up three machines, configure them using the same steps that we ask Qt developers to follow when they set up their local machines (or that we use in our CI system Coin - hence the name), and then run the build job for the code in the local directory.

This (and a few other things) is possible now with minicoin. We can define virtual machines in code that we can share with each other like any other piece of source code. Setting up a well-defined virtual machine within which we can build our code takes just a few minutes.

minicoin is a set of scripts and conventions on top of Vagrant, with the goal to make building and testing cross-platform code easy. It is now available under the MIT license at https://git.qt.io/vohilshe/minicoin.

A small detour through engineering of large-scale and distributed systems

While working with large-scale (thousands of hosts), distributed (globally) systems, one of my favourite, albeit somewhat gruesome, metaphors was that of “servers as cattle” vs “servers as pets”. Pet-servers are those we groom manually, we keep them alive, and we give them nice names by which to remember and call (ie ssh into) them. However, once you are dealing with hundreds of machines, manually managing their configuration is no longer an option. And once you have thousands of machines, something will break all the time, and you need to be able to provision new machines quickly, and automatically, without having to manually follow a list of complicated instructions.

When working with such systems, we use configuration management systems such as CFEngine, Chef, Puppet, or Ansible, to automate the provisioning and configuration of machines. When working in the cloud, the entire machine definition becomes “infrastructure as code”. With these tools, servers become cattle which - so the rather unvegetarian idea - is simply “taken behind the barn and shot” when it doesn’t behave like it should. We can simply bring a new machine, or an entire environment, up by running the code that defines it. We can use the same code to bring production, development, and testing environments up, and we can look at the code to see exactly what the differences between those environments are. The tooling in this space is fairly complex, but even so there is little focus on developers writing native code targeting multiple platforms.

For us as developers, the machine we write our code on is most likely a pet. Our primary workstation dying is the stuff for nightmares, and setting up a new machine will probably keep us busy for many days. But this amount of love and care is perhaps not required for those machines that we only need for checking whether our code builds and runs correctly. We don’t need our test machines to be around for a long time, and we want to know exactly how they are set up so that we can compare things. Applying the concepts from cloud computing and systems engineering to this problem lead me (back) to Vagrant, which is a popular tool to manage virtual machines locally and to share development environments.

Vagrant basics

Vagrant gives us all the mechanisms to define and manage virtual machines. It knows how to talk to a local hypervisor (such as VirtualBox or VMware) to manage the life-cycle of a machine, and how to apply machine-specific configurations. Vagrant is written in Ruby, and the way to define a virtual machine is to write a Vagrantfile, using Ruby code in a pseudo-declarative way:

Vagrant.configure("2") do |config|
    config.vm.box = "generic/ubuntu1804"
    config.vm.provision "shell",
        inline: "echo Hello, World!"
end

Running “vagrant up" in a directory with that Vagrantfile will launch a new machine based on Ubuntu 18.04 (downloading the machine image from the vagrantcloud first), and then run “echo Hello, World!” within that machine. Once the machine is up, you can ssh into it and mess it up; when done, just kill it with “vagrant destroy”, leaving no traces.

For provisioning, Vagrant can run scripts on the guest, execute configuration management tools to apply policies and run playbooks, upload files, build and run docker containers, etc. Other configurations, such as network, file sharing, or machine parameters such as RAM, can be defined as well, in a more or less hypervisor-independent format. A single Vagrantfile can define multiple machines, and each machine can be based on a different OS image.

However, Vagrant works on a fairly low level and each platform requires different provisioning steps, which makes it cumbersome and repetitive to do essentially the same thing in several different ways. Also, each guest OS has slightly different behaviours (for instance, where uploaded files end up, or where shared folders are located). Some OS’es don’t fully support all the capabilities (hello macOS), and of course running actual tasks is done different on each OS. Finally, Vagrant assumes that the current working directory is where the Vagrantfile lives, which is not practical for developing native code.

minicoin status

minicoin provides various abstractions that try to hide many of the various platform specific details, works around some of the guest OS limitations, and makes the definition of virtual machines fully declarative (using a YAML file; I'm by no means the first one with that idea, so shout-out to Scott Lowe). It defines a structure for providing standard provisioning steps (which I call “roles”) for configuring machines, and for jobs that can be executed on a machine. I hope the documentation gets you going, and I’d definitely like to hear your feedback. Implementing roles and jobs to support multiple platforms and distributions is sometimes just as complicated as writing cross-platform C++ code, but it's still a bit less complex than hacking on Qt.

We can’t give access to our ready-made machine images for Windows and macOS, but there are some scripts in "basebox" that I collected while setting up the various base boxes, and I’m happy to share my experiences if you want to set up your own (it's mostly about following the general Vagrant instructions about how to set up base boxes).

Of course, this is far from done. Building Qt and Qt applications with the various compilers and toolchains works quite well, and saves me a fair bit of time when touching platform specific code. However, working within the machines is still somewhat clunky, but it should become easier with more jobs defined. On the provisioning side, there is still a fair bit of work to be done before we can run our auto-tests reliably within a minicoin machine. I’ve experimented with different ways of setting up the build environments; from a simple shell script to install things, to “insert CD with installed software”, and using docker images (for example for setting up a box that builds a web-assembly, using Maurice’s excellent work with Using Docker to test WebAssembly).

Given the amount of discussions we have on the mailing list about “how to build things” (including documentation, where my journey into this rabbit hole started), perhaps this provides a mechanism for us to share our environments with each other. Ultimately, I’d like coin and minicoin to converge, at least for the definition of the environments - there are already "coin nodes" defined as boxes, but I'm not sure if this is the right approach. In the end, anyone that wants to work with or contribute to Qt should be able to build and run their code in a way that is fairly close to how the CI system does things.