Hunting Performance Regressions Through Years of MariaDB History

Performance regression investigations rarely follow a straight line.
Mark Callaghan has been providing me with some great MariaDB performance issues to investigate. These are exactly the kinds of problems I enjoy working on. A benchmark shows that something changed, sometimes years ago, and the job becomes figuring out where, why, and whether the performance was ever recovered.
MDEV-38049 turned into one of those investigations.
It also reinforced two things for me.
First, having access to the Hetzner Sponsor Host gives the MariaDB Foundation an awesome system for hunting performance problems. The AMD EPYC system provides enough CPU, memory, and consistency to repeatedly move backward and forward through MariaDB history while keeping the workload itself stable.
Second, this investigation exposed an area where TAF itself now needs to grow.
MDEV-38049
https://jira.mariadb.org/browse/MDEV-38049
TAF User Properties file: MDEV38049.properties
Thanks to Mark Callaghan for filing MDEV-38049 and providing another interesting performance problem to investigate.
The initial TAF testing showed several different behaviors.
POINTS-COVERED-PK and RANDOM-POINTS showed roughly a 50% performance regression compared with the 10.3.39 baseline.
The range workloads, RANGE-COVERED-PK, RANGE-COVERED-SI, and RANGE-NOTCOVERED-PK, were much closer to their original performance.
OLTP_RO was different again. That workload had a later performance impact associated with the move from latin1 to UTF8, particularly around the ORDER and DISTINCT range workloads.
Already, this was clearly not one simple regression.
Walking Through the History
The deeper POINTS-COVERED-PK investigation became a trip through years of MariaDB history.
I tested releases, individual commits, merge commits, parents of merges, and commits pulled in from other release branches.
The history was not linear.
The deeper investigation identified eight areas of performance regression and three areas of performance recovery.
That matters because looking only at release-to-release results can make a performance investigation appear much simpler than the underlying history really is.
A regression may appear.
Another change may make it worse.
Something else may recover part of the throughput.
Another merge may move that behavior into another branch.
By the time you look at a much later release, the result you are measuring can represent years of accumulated changes.
So I kept following the numbers.
Eventually that trail moved through main, 13.1, 12.3, and into 11.8.
The Recovery
The performance recovery led to work by Marko Mäkelä and co-author Alessandro Vetere:
db94f391be4e986be358be1e8e60c94758cc78ba
MDEV-33966: buf_page_make_young() is a contention point
The performance boundary was unmistakable.
Immediately before the recovery, POINTS-COVERED-PK was running at approximately 187K TPS.
At MDEV-33966, the same workload reached more than 402K TPS.
The original 10.3.39 baseline was approximately 386K TPS.
So the performance was not merely recovered. On this system and workload, POINTS-COVERED-PK moved slightly ahead of the original baseline.
RANDOM-POINTS recovered as well.
The range workloads remained at or near their original baseline performance.
That gave us the answer we had been looking for.
Our testing showed that MDEV-33966 recovered the throughput regression affecting the point workloads
The MDEV currently lists 11.8.10, 12.3.4, and 13.1.2 as fix versions. Those releases come after the 11.8.9, 12.3.3, and 13.1.1 versions I had used in the investigation, which explains why the recovery was not visible in those release results.
Excellent work by Marko Mäkelä and Alessandro Vetere.
The Hetzner Performance Hunting Ground
Investigations like this require hardware where I can run the same workload repeatedly while changing the software underneath it.
For this work, testing was performed on a Hetzner Sponsor Host provided to the MariaDB Foundation:
AMD EPYC 9454 48-Core Processor
48 physical cores / 96 CPUs
Single socket
124.98 GB RAM
AlmaLinux 9.8
The MDEV-38049 TAF profile uses eight tables with 10 million rows each, a 90 GB InnoDB buffer pool, 40 test threads, explicit warmup, and repeated iterations.
That gives me a substantial, memory-resident workload where performance differences are easy to expose and, just as importantly, repeat.
Having sponsor hardware available for this kind of work gives the MariaDB Foundation a great environment for performance archaeology.
Instead of asking whether a regression still exists, we can start asking exactly where it appeared, where it changed, and where it disappeared.
The Investigation Also Found a TAF Bottleneck
Bottleneck is me!
There was one frustrating part of this investigation.
Too much of the commit hunting was manual.
For every commit I wanted to test, I had to build it, create the installation, point TAF at that installation, initialize the database, run the workload, capture the result, and then repeat the process for the next commit.
When narrowing a regression, the process often looks something like this:
Take the top, middle, and bottom of a commit range.
Run them.
Determine which half contains the transition.
Take another set from that range.
Run them.
Eventually there may be only three commits left.
Run all three.
The performance-engineering decision about which commits to test is interesting.
Manually building, installing, and running every one of those commits is not.
The Next TAF Build Workflow
The idea is straightforward.
TAF should be able to receive a list of commits.
For each commit, TAF will invoke a source-build backend script with the information required to produce an installation: the commit ID, source location, build location, installation location, and other build properties.
Once that build is available, TAF can feed the generated installation into its existing lifecycle:
Build → Install → Initialize → Start → Run Tests (repeat)
Then repeat the complete lifecycle for the next commit.
The database-specific source builder can remain outside the core orchestration logic. A MariaDB builder knows how to build MariaDB. Another builder could implement the corresponding process for another database.
TAF only needs a well-defined contract with the builder and a successful installation to continue the test lifecycle.
That will turn a large amount of manual regression hunting into deterministic automation.
Instead of spending my time managing builds and installations, I can spend it looking at the results and determining which part of the history should be tested next.
And Eventually, Git Bisect
There is an obvious direction this could go later.
Once TAF can reliably take a commit, build it, install it, run a deterministic workload, and determine whether the resulting performance belongs on the good or bad side of a regression boundary, much of the machinery needed for automated performance bisection already exists.
I do not want to jump straight there.
The first requirement is simpler and immediately useful: reliably execute a user-provided list of commits.
But eventually I would like to explore integrating this workflow with git bisect.
That would allow TAF to participate not only in reproducing a regression, but in systematically narrowing the source history that introduced it.
In closing
A big thank you to Mark Callaghan for continuing to provide interesting MDEVs to investigate, and to Hetzner for providing an awesome host to investigate them with.
This investigation recovered the answer we were looking for, but it also exposed where the investigation process itself can improve.
Automating the build, install, and test cycle in TAF should let me spend less time managing commits and more time doing the part that matters: finding where performance changed and understanding why.
To all, “Happy benchmarking!”

Jonathan (Jeb) Miller has experience in military leadership, Fire/EMS leadership, computer operations leadership, teaching, and more than 27 years of database performance engineering (PervasiveSQL, MySQL, MariaDB).
TAF is the third benchmarking framework he has helped design and build.
#Hetzner #MariaDB #MySQL #PostgreSQL #Database #TAF #DatabasePerformance #Performance
Leave a Reply