The Man Who Benchmarks MariaDB Against the World

Jonathan Miller joined the MariaDB Foundation a year ago to do one thing: performance work, and to write about it. The tool he built to make that possible is TAF, the Test Automation Framework, and since the 4.0 release it has been downloaded more than 400 times in two weeks.

TAF lets anyone install MariaDB, MySQL and PostgreSQL side by side, run the identical benchmark against each, and hand the exact configuration to someone else so they can reproduce the result. It is open source, it is the tool Jonathan uses every day to investigate performance bug reports, and it is built so that others can extend it. I sat down with Jonathan to talk about where it came from, how it works, and what comes next.

Illustration of Jonathan Miller, MariaDB Foundation performance engineer, with a laptop in front of server racks

27 years of making databases faster

Jonathan has spent almost 27 years in database performance testing: close to five at Pervasive SQL, almost 21 at MySQL, and now nearly one at the MariaDB Foundation. Before any of that he served in the military, trained as a firefighter, and studied computer programming. His first jobs were in computer operations in Texas, working up to systems administrator and assistant database administrator.

In 2000 he joined Pervasive SQL, where he was put under a performance engineer within two months and took over performance testing. In 2005 he answered a Craigslist post from MySQL AB and became the second QA person the company ever hired. His first week was NDB Cluster training in California; his second was a developer meeting in Prague, where he met Monty and much of the early MySQL team.

He spent two years on NDB Cluster and replication QA, then moved into System QA to take over performance when the founders of Percona left. He stayed through the Sun and Oracle years, and for 11 years taught an annual class on QA for internationalisation, the double-byte problems that break single-byte applications in Asia. For 17 of those years he also helped run his local volunteer fire department, growing it from a rural district with a 35,000 dollar budget into a professional emergency service district with a 2.5 million dollar one.

When Oracle laid him off a little over a year ago, Anna Widenius called and he joined the MariaDB Foundation. She wanted someone who could do performance work and blog about it. That is the whole brief he was given, and TAF is what it took to deliver on it.

The third framework

TAF is the third benchmarking framework Jonathan has built. The first, Autobench 2, started life as a 10,000-line shell script handed to him when the Percona founders left MySQL. It had sub-functions and some Perl around it, but no real automation: set everything up by hand, run the script, hope nothing breaks. He calls it duct tape and glue.

Autobench 3 was the rewrite he and his manager in System QA argued for: almost pure Perl, everything in one unit, far easier to run. It was still a 10,000-line script, it only tested MySQL, and it only reported in XML. It was also internal. Inside MySQL, Jonathan built it for the team, and the team was the only one who ever saw it.

TAF started for the same reason, in his words: he had performance work coming and needed a tool he trusted. The difference is where it lives. At the Foundation the framework is published for the whole world to use, and he is open about how much that takes. Opening MariaDB to identical tests against MySQL and PostgreSQL means the results in some rare cases will show MariaDB behind.

His answer, and Anna’s, is that this is the point. If the world comes back and says MySQL or PostgreSQL is faster on some workload, that is now a place to look and an area to improve. The only way to get better is to know where you are lacking.

Everything is a plugin

The architectural difference between TAF and its predecessors is that the script is gone. The functionality is split into libraries, one for installs, one for running tests, one for actions, one for reporting, and the pieces that vary are plugins:

Database maker plugins: MariaDB, MySQL and PostgreSQL today. The database is plugin logic rather than hard-coded, so anyone can write a plugin for another engine.
Test suite plugins: sysbench for micro-benchmarks such as point selects and read-only or read-write mixes, and HammerDB (Steve Shaw’s framework) for TPC-C, the transaction processing standard everyone is graded by, and TPC-H, a read-only suite of 21 analytical queries. When you choose a suite, TAF loads it into itself, so the setup and run calls become sysbench’s or HammerDB’s own.
Profiler plugins: a perf profiler is built in, with more diagnostics possible without touching the suite or the database plugin.
Reporter plugins: JSON, CSV, text, or whatever you write. Reporters receive normalised results and never influence the run.

Two practical consequences follow. First, TAF can hold many installs at once: twelve versions of MariaDB, five of MySQL and three of PostgreSQL side by side, with the user jumping between them. Autobench 3 handled one install at a time. Second, extending it means adding a library or a plugin, not editing a monolith. A future action that pulls MariaDB source, builds a given commit and installs it is a new library, not a rewrite.

In practice that covers a wide range of work: comparing major versions from 10.x through 13.x, minor releases and patch levels, debug against non-debug builds; comparing parent and target commits; comparing MariaDB against MySQL and PostgreSQL with identical configuration, workload and environment; and swapping storage engines, for instance running the same sysbench case against InnoDB and then against TidesDB. The same configuration can run on bare metal, VMs, cloud instances or containers, and the results feed a backend that keeps a baseline and flags change over time.

Configuration provenance, or how to end the benchmark blog problem

Every performance blog has the same weakness. The author configured the server one way last month, another way today, and the reader who wants to reproduce the numbers never gets all the pieces. TAF’s answer is a single file.

TAF accepts every setting both as a property in a user property file and as a command-line option, and the command line always wins. The database configuration can be a property pointing to an external file, the configuration written inline in the property file, or a file named on the command line. Whichever source actually took effect is the one TAF writes into the user property file it generates at the end of the run.

That is configuration provenance: whatever was really used to run the test is what ends up in the file. Hand that file to anyone in the world, they install TAF and the same database versions, run it, and they have done exactly what you did. Jonathan credits Monty for pushing him to collapse everything into one file, and it is now the mechanism behind every MDEV test case he publishes.

Performance change testing, not regression testing

Ask Jonathan whether he does performance regression testing and he will correct you. He does performance change testing. If a build got faster, he wants to know why just as much as if it got slower, because a gain is sometimes a bug.

The result that has stayed with him longest came out of the daily and weekly build cycle at MySQL. A weekly I/O-bound run showed a gain of over 300 percent. Investigation found the commit had stopped writing undo records while still writing redo: the tests were doing half the work, and the database was broken. Functional testing had not caught it. Within 24 to 48 hours the developer had it back with the message that this needs fixing. Watching performance in both directions shows you both good and bad.

The recent example is a regression Mark Callaghan reported against a simple point lookup, a select by id from a set of ids. Jonathan reproduced it with TAF and confirmed the fix in 11.8, then showed it was missing from 12 and 13. The developers sent him all three commits with their parents, and he returned six builds and six test runs, parent against fix for each series, in under four hours. The fix delivered over 3x on that micro case. Macro workloads are made of micro operations, so wins like this add up.

The mdevs folder: reproducible bug reports for everyone

Jonathan now investigates the performance MDEVs, the bug reports in MariaDB’s tracker. Each investigation produces a test case, a user property file and usually a shell script that runs the whole thing. Those go into the mdevs folder of the TAF repository.

The effect is that a closed performance MDEV is no longer a comment thread with numbers in it. Anyone can pull the exact files Jonathan used, run them, and get the same result on their own hardware. Performance history becomes transparent instead of anecdotal.

How to prove your patch made MariaDB faster

The workflow for a contributor is the one MySQL developers used before pushing to main, and Jonathan had it fully automated there by 2025. With TAF it is simple:

Take the parent commit, the one your change applies to, and the commit with your change.
Build both and install both into TAF.
Run the same test case against each. The user property file guarantees it is the same.
Compare: what it was like before your code, what it is like with it.

Focus the test on the part of the code you touched. A skeptical engineer’s first question when TAF says version X is slower than version Y is always whether it reproduces, and Jonathan’s first step is the same. If it does, and there are commits A, B, C and D between X and Y, he walks backwards to find where the change really sits. Then he profiles.

Profiling in TAF is scheduled rather than immediate. On a five-minute run he waits two and a half minutes, then captures perf for 30 seconds, so the profile comes from the meat of the run rather than the startup noise. Profile both sides and you can see where time is being spent that was not spent before. At that point he hands it to the developers and performance architects. He is explicit about his lane: he does not change database code, he finds the problem, narrows it as far as it will go, and passes it on.

What comes after 4.0

Two things are in motion. The first is a front end. TAF already has a backend for automated change detection: you set a baseline, results flow in, and a comparison table records how each run compares. What it lacks is a GUI to look at it. The engineer who built the equivalent front end for the performance framework inside MySQL is now building one for TAF, donating his time to the Foundation.

The second is a new test suite Jonathan calls Rapid Query, a descendant of the Quick Query suite he built for Autobench 3. Many performance bugs arrive as a single statement: this query is ten times slower than it used to be. Rapid Query is a single-threaded suite where you provide the SQL and either the data or the SQL that generates it, and TAF times the execution. It is meant to make adding a test case as fast as pasting the customer’s query.

One laptop, one evening

Asked what someone with a single laptop can usefully do tonight, Jonathan’s answer is modest and concrete:

Install TAF.
Install whichever version of MariaDB you want to play with.
Build sysbench, the easiest suite on small hardware.
Run something cacheable: point selects, read-only, or select distinct ranges, on about eight tables of 1,000 rows each.

That is enough to learn the framework. Once you get onto real hardware, in his words, you can turn it loose and beat up some databases pretty hard.

If you want to contribute code, the most wanted contribution is a new test suite. Every additional benchmark, whether a standard like HammerDB and sysbench or something custom, tells the team more about how the database performs, and that is what makes it better.

It is not work, it is an obsession

I asked Jonathan what advice he would give someone starting in performance today, when frameworks like TAF and HammerDB exist and the first steps look nothing like his did. His answer was to decide what kind of performance work fits you, because there are many kinds, and then be consistent with it. Then he added the real reason he has lasted 27 years: he loves the work. That is why they could not get rid of him.

It is a sentiment most of us at the Foundation recognise. The work does not feel like work. It feels like a problem that needs solving, and a sandbox to solve it in.