The existing network test server has had limitations for years. Most notably, it is only available in Qt intranet. it is not accessible by Qt developers to run network tests from outside. Since the old network test server can’t easily be reproduced, the contributor relies on Qt CI to verify and debug their changes.
As for Qt internally, the server configurations are out of date and hard to be maintained. Apparently the network code and tests are out of sync. That’s why the test server doesn’t work with many legacy tests. It makes those tests blacklisted. Even we desire to do something, it is no way to remove the blacklist. Moreover, some services (e.q. Samba) don’t allow simultaneous access. When more than one virtual machine are running the test at the same time, flaky result occurs occasionally. (what a nightmare, sigh!)
Replace the monolithic test server with Docker containers
The challenge is that it needs not only consideration for the user environments but also the test framework of Qt. We want to make it ease to use for the developers., but we need to compromise under the current test framework.
After countless discussions, we have managed to make it works on both Ubuntu and macOS. The developers can now test networking code by using local containers to run the test servers. Besides, Qt CI is doing the test of QNetworkReply with it. If nothing unexpected happens, the Windows system with MinGW compiler is coming soon.
Requirements from developers
- Launch the relevant infrastructure transparently when doing make check
- Minimal setup required without modifying host settings (e.q. /etc/hosts)
- Test network code locally without login Qt intranet
- Provide a constant environment for each test
- Split the monolithic test server into several dedicated services
- Use Gerrit to review tests and the corresponding server configurations in the same space
- Progressively convert tests from using the old server to using the new approach one by one
- The server configurations vary depending on the repository branches
- Support Qt shadow builds as well
Requirements from Qt CI
- Coin loses network connectivity after provisioning
- Cache the provisioned images so that Coin won’t rebuild the images every time
- Run tests in parallel to speed up the process when staging changes
- The side-effect shall be completely isolated if the tests are using the same service at the same time
NOTE: To learn more about how to rework existing tests and create new test servers, https://wiki.qt.io/Network_Testing.
How to run tests
Just try it, and then you will see how it works!
The container-based solution shall launch itself transparently when building check target. If the user has installed docker engine on the host, the test servers will be built up inside separate docker containers, and then, the test case goes with the docker-based test servers. Otherwise, the tests will keep using the internal test server.
bash: ~/qtbase/tests/auto/network/access/qnetworkreply (dev) $ make check
Before starting the test servers, it requires the user to run the provisioning script in advance. The provisioning script is used to build up the docker images into the docker-cache. It handles the immutable parts of the server installation that rarely need adjustment.
What happened in the box
(1) Ensure that the docker images have been provisioned. If you get the massage below, please run the provisioning script, and then try again.
- Docker machine qt-test-server not found
- Docker image qt-test-server-* not found
(2) Use docker-compose to bring up the depending test servers
(3) Build test code with the predefined macro QT_TEST_SERVER
(4) Use mDNS to retrieve the IP address of each server and make sure their services are ready, and then start running test
(5) Destroy test servers and tidy away related files
It is not the perfect right now. There are still some works to be done. However, this is the first step, as well as the foundation that we can make it better step by step.
- Support multistage provisioning. Provision the test data in qtbase.
- Use Vagrant Docker provisioner to replace Boot2Docker