Building a Public Performance Change Detection Library for MariaDB

Written by Written by Jonathan Miller 1 Comment on Building a Public Performance Change Detection Library for MariaDB

Performance engineering inside a database project is usually private. Teams run internal workloads, investigate changes, fix issues, and move on. The knowledge stays inside the company, and the outside world only sees a benchmark graph or a release note.

I know this pattern well — I spent more than twenty‑six years doing this work for both PervasiveSQL and MySQL.

The investigations were deep, the lessons were real, but almost none of it was visible to the outside world.

The surprises, the unexpected gains, the subtle drops, the “why did throughput collapse at 64 threads?” moments — they all stayed internal.

And that’s normal in most database companies.

But it’s also a missed opportunity.

With this commit, the MariaDB Foundation begins establishing the Performance MDEV Test Case Library inside the Test Automation Framework (TAF). The goal is simple:

Make performance change detection public, reproducible, and teachable.

This library will grow over time, but today marks the first step — adding a real, meaningful test case that exposed a behavioral change in the wild.

Before diving into the details, the Foundation would truly appreciate your input into the MariaDB Foundation: Annual Survey (Closes 10/15/2026)

Starting with a real case: MDEV‑32750 (AHI behavior change)

The first entry in the library is MDEV‑32750, a change discovered during a migration from MariaDB 10.4.10 to 10.11.4. The Adaptive Hash Index (AHI) behaved differently between versions, causing throughput shifts at higher thread counts. HammerDB TPROC‑C was the perfect workload for revealing this behavior — its mix of CPU‑bound and lock‑heavy OLTP operations makes AHI effects visible, measurable, and impossible to hide.

This wasn’t a synthetic benchmark. It was a real performance change found by a real engineer (thank you, Keshan).

We took that investigation and turned it into a deterministic, repeatable test case:

  • two properties files (AHI ON / AHI OFF)
  • a multi‑version runner script
  • a release‑testing script
  • a HammerDB/TPROC‑C workload tuned for reproducibility
  • deterministic CPU behavior
  • consistent thread counts
  • checksums for correctness
  • flamegraph generation for analysis

This is exactly the kind of case that belongs in a performance change detection library: a real behavioral change, captured and preserved so future engineers can detect it again.

A perfect example: when a “gain” isn’t a gain

One of the clearest examples of why a public performance change detection library matters came from MySQL’s automated performance change detection farm. We had daily runs, weekly runs, long‑run stability checks — a full production system designed to surface any behavioral change, good or bad.

During one of the weekly I/O‑bound workloads, the farm suddenly reported a huge improvement — more than +55% throughput. On the dashboard it looked like a breakthrough.

But breakthroughs in database performance rarely appear out of nowhere.

I was on performance duty that week, overseeing what the farm was producing, and this “improvement” immediately stood out.

The improvement wasn’t real.

A commit had accidentally stopped writing the undo log.

The workload looked faster because the database was silently skipping work.

Once the bug was fixed, performance returned exactly to where it had been before. The “improvement” vanished, because it had never been real.

This is the essence of change detection:

  • A drop might be a real problem.
  • A gain might be a real problem.
  • A surprise is always worth investigating.

And this is exactly the kind of case that belongs in a performance change detection library — a real behavioral change, captured so future engineers can detect it again.

Why a framework needs test cases

A test automation framework is useful. It gives structure, repeatability, and a place to run workloads. But a framework by itself is only okay. It’s a tool — an empty container.

The real value appears when the framework carries test cases that matter.

A framework with built‑in test cases for real performance behaviors — optimizer changes, storage engine behavior shifts, concurrency surprises, AHI toggles, IO scheduling quirks, migration differences — is where the framework stops being a tool and starts becoming a treasure for a database maker.

Performance engineers generate test cases every day:

  • daily sanity checks
  • weekly change sweeps
  • migration investigations
  • odd behaviors spotted during tuning
  • throughput shifts at specific thread counts
  • optimizer plans that evolve between minor releases
  • storage engine behaviors that change under load

These aren’t synthetic benchmarks. They’re real engineering moments — the places where someone learned something important about the database.

Those moments should never be lost.

They should become test cases.

A public library meant to teach the world

By building this library publicly inside TAF, the Foundation is doing more than testing MariaDB. We’re showing other database makers, benchmark practitioners, and performance engineers how to build their own libraries of performance change detection cases.

The structure is intentionally simple:

  • mdevs/ — cases tied to real MDEVs
  • future folders for:
    • daily checks
    • weekly sweeps
    • release validation
    • pre‑commit checks
    • investigation workloads
    • long‑run stability tests

Each folder can hold properties files tuned for a specific level of seriousness in change detection:

  • quick checks for “did anything obvious change?”
  • deeper sweeps for “did anything subtle shift?”
  • targeted MDEV cases for “did this specific behavior stay fixed?”
  • heavy release runs for “is this version safe to ship?”

The plan is to build this one step at a time, starting with real MDEV cases like MDEV‑32750, then expanding outward.

As we build out change detection for MariaDB, we teach the world how to build change detection for their databases too.

The release‑testing script

Alongside the MDEV‑32750 multi‑version runner, this commit also introduces a release‑testing script designed for single‑version validation.

This script:

  • auto‑detects the TAF root
  • locates the MDEV properties directory
  • constructs deterministic test‑case tags
  • runs selected MDEV cases against the active MariaDB installation
  • logs results cleanly
  • keeps the workflow simple and reproducible

This is the script you use when validating a release candidate or checking whether a behavioral change is expected or unexpected.

It’s intentionally minimal — no version switching, no installation logic, no matrix testing. Just run the cases you care about on the version you’re validating.

Hardware Sponsor Acknowledgment — Hetzner.com

TAF 4.0 was engineered and validated on high‑performance hosts provided by Hetzner.

hz-bench

  • Intel i5‑13500 (20 vCPUs / 14 cores)
  • 62 GB RAM
  • AlmaLinux 9.7

hz-bench2

  • AMD EPYC 9454 (96 vCPUs / 48 cores)
  • 125 GB RAM
  • AlmaLinux 9.8

These systems enabled full multi‑database workload testing, reproducible performance runs, and the scale required to expose real behavior across MariaDB, MySQL, and PostgreSQL.

We appreciate Hetzner’s outstanding support.

Thank you

Thank you for your interest in performance testing and change detection. Whether you’re a database maker, a benchmark enthusiast, or someone who simply wants to understand how and why database behavior shifts over time, your curiosity is what keeps this work moving forward. The more people who care about measuring change, the better databases become for everyone.

If you’re interested in contributing your own performance change detection cases, we’d love to see them. Real workloads, real surprises, and real behavioral changes are exactly what make this library valuable. As the structure grows, we’ll document clear ways for the community to add cases, share investigations, and help expand MariaDB’s public performance memory.

About the Author
Jonathan (Jeb) Miller has experience in military leadership, Fire/EMS leadership, computer operations leadership, teaching, and more than 27 years of database performance engineering (PervasiveSQL, MySQL, MariaDB).

TAF is the third benchmarking framework he has helped design and build.