QTestLib, now with nice graphs pointing upwards

Published Friday December 5th, 2008
3 Comments on QTestLib, now with nice graphs pointing upwards
Posted in Aggregated, Qt, Test

One of the focus areas for 4.5 is performance. To that end, we decided to add a couple of benchmarking extentions to our beloved QTestLib.

There is exactly one new API, the QBENCHMARK macro:

    QString str1 = "foo";
    QString str2 = "foo";

    QBENCHMARK {
        str1.localeAwareCompare(str2);
    }

(Full example is available at $QTDIR/examples/qtestlib/tutorial5.)

Under the hood, the benchmark macro starts the timing measurements, and then repeats the code under test enough times to get a measurable result. Running this test produces the following output:

RESULT : TestBenchmark::simple():
     0.00094 msec per iteration (total: 31, iterations: 32768)

The default measuring back-end is based on QTime, and in this case we needed 32768 iterations in order to get a result. Repeating the code this many times is not ideal, since it takes a lot of time and might give inaccurate results due to caching effects. On select platforms we therefore have CPU tick counters available, which can be invoked by specifying “-tickcounter” on the command line:

./mytest -tickcounter
RESULT : TestBenchmark::simple():
     3,206 ticks per iteration (total: 3206, iterations: 1)

There is also callgrind-based measurer available on linux/x86 systems, which produces 100% reliable instruction counts, but does not take things like I/O waits into account.

Benchmarking a single API is fun, but to be useful we have to compare it against something. Let’s add some data:

void TestBenchmark::series_data()
{
    QTest::addColumn("useLocaleCompare");
    QTest::addColumn("stringSize");

    for (int i = 1; i < 10000; i += 2000) {
        QByteArray size = QByteArray::number(i);
        QTest::newRow(("locale aware compare--" + size).constData()) << true << i;
        QTest::newRow(("standard compare--" + size).constData()) << false << i;
    }
}

We're benchmarking QString::localeAwareCompare() vs QString::operator==(), for different string sizes. The test function now looks like this:

void TestBenchmark::series()
{
    QFETCH(bool, useLocaleCompare);
    QFETCH(int, stringSize);

    QString str1 = QString().fill('A', stringSize);
    QString str2 = QString().fill('A', stringSize);
    int result; // assign to result to prevent the optimizer from optimizing the compare away.
    if (useLocaleCompare) {
        QBENCHMARK {
            result = str1.localeAwareCompare(str2);
        }
    } else {
        QBENCHMARK {
            result = (str1 == str2);
        }
    }
}

This gives lots of results:

RESULT : TestBenchmark::series():"locale aware compare--1":
     0.00067 msec per iteration (total: 22, iterations: 32768)
RESULT : TestBenchmark::series():"standard compare--1":
     0.000019 msec per iteration (total: 41, iterations: 2097152)
RESULT : TestBenchmark::series():"locale aware compare--1001":
     0.016 msec per iteration (total: 33, iterations: 2048)
RESULT : TestBenchmark::series():"standard compare--1001":
     0.00056 msec per iteration (total: 37, iterations: 65536)
RESULT : TestBenchmark::series():"locale aware compare--2001":
     0.031 msec per iteration (total: 32, iterations: 1024)
...

It’s hard to make heads or tails of all this, so we decided to create a few tools to help us out. Unfortunately these tools didn’t make the cut for 4.5 (“Shipping Software Means Prioritizing”), but instead we’re making them available here on labs, with bugs and all. (run “git clone http://labs.trolltech.com/gitweb?p=qtestlib-tools” to download.)

So, to generate the nice graph pointing upwards, run the test in xml mode and then feed that xml to the generatereport tool.

./mytest -xml > results.xml
generatereport results.xml

A graph generated by QTestLib (adm: Added a smaller version of the graph)

Do you like this? Share it
Share on LinkedInGoogle+Share on FacebookTweet about this on Twitter

Posted in Aggregated, Qt, Test

3 comments

Joe says:

Would it be too much to ask for a pre-made graph for a couple of systems so we could be impressed without so much effort?

Philippe says:

You say “On select platforms we therefore have CPU tick counters available which can be invoked by specifying “-tickcounter” on the command line:”
My question: is this tick counter an independant Qt API? QTime’s implementation is slow and not too accurate, and I need something fast and more accurate.

Morten says:

Philippe: We use the cycle.h file included with fftw. You can the latest version directly from their web page: http://www.fftw.org/cycle.h. It has support for lots of platforms and has a liberal license.

Commenting closed.

Get started today with Qt Download now