MariaDB Foundation’s Test Automation Framework (TAF) 4.0 Release

Written by Written by Jonathan Miller 1 Comment on MariaDB Foundation’s Test Automation Framework (TAF) 4.0 Release

MariaDB Foundation’s Performance Test Automation Framework (TAF) 4.0 Release represents the biggest changes to the framework since its creation.

This release reshapes how we organize, run, and report database workloads across MariaDB, MySQL, and now PostgreSQL.

The improvements are not just incremental — they fundamentally change how results are collected, tagged, and analyzed across the entire automation pipeline.

Several of the changes in TAF 4.0 were driven directly by requests from Monty, the creator of both MySQL and MariaDB.

Monty’s recent MDEV‑39497 request also contributed to new features such as --test-case-tag, which helps organize large test runs and keep track of individual cases more reliably. Current results attached to MDEV39497 where generated with a beta TAF 4.0.

Additional improvements came from pull requests submitted by contributors across the community, and from sponsors who provided engineers, and hardware making this work possible.

Before diving into the details of the TAF 4.0 release, we’d appreciate your input in the MariaDB Foundation Annual Survey

TAF 4.0 GitHub Release

Major Implementations in TAF 4.0

1) PostgreSQL Support (Beta)

TAF now includes end‑to‑end PostgreSQL support:

  • PostgreSQL Database Plugin (PR #7)
  • Updated sysbench‑lua for PostgreSQL builds
  • Updated RUN SQL library for PostgreSQL

This expands TAF’s multi‑database testing capabilities and ensures consistent behavior across MariaDB, MySQL, and PostgreSQL.

This feature is in beta — feedback and pull requests are welcome.

2) Embedded DB Configuration in User Properties

TAF now supports embedding database configuration directly inside the user properties file:

[db_config_start]
...
[db_config_end]

Precedence order:

  1. CLI overrides
  2. Inline DB config block
  3. db_config_file reference

The winning configuration is written into the auto‑generated test‑case properties file, archived with results, and passed to reporters and backend ingestion. This eliminates configuration drift and ensures each test case is fully self‑contained.

3) Auto‑generated Database Configuration File

TAF now generates a temporary DB configuration file at runtime:

  • built from the winning configuration
  • used to start the database
  • archived with results
  • included in the auto‑generated test‑case properties
  • passed to reporters and backend ingestion

This guarantees reproducibility and traceability.

4) Auto‑generated Test Case User Properties Files

Each test case now produces a standalone properties file containing:

  • Original properties (minus db_config_file)
  • CLI overrides converted into properties
  • The winning DB configuration

This makes test cases diff‑friendly and fully reproducible.

5) README.txt Improvements

Each test‑case README now includes:

  • the DB configuration used
  • the auto‑generated test‑case properties file

Reporters and backend ingestion now rely on run metadata rather than external configuration files.

6) Recovery Mode Improvements

Recovery mode now re‑runs iteration 1 of the recovered data store to eliminate skew and ensure synchronization. (PR #9)

7) New TAF Usage Options

  • --test-case-tag / taf.test_case_tag
  • --dump-run-state / taf.dump_run_state
  • --exec-script-file-before-tests
  • --exec-script-file-after-tests

These improve documentation, reproducibility, and automation.

8) New Database Configuration Files

New comparable, minimal, default, analytics, and OLTP configuration files for:

  • MariaDB
  • MySQL
  • PostgreSQL

9) Backend and Reporter Improvements

Backend now retrieves DB configuration from run metadata.

Reporters use raw run data for configuration.

This resolves long‑standing provenance issues.

10) ResultsCompareRaw.pl (New)

A new tool for regenerating HTML from raw results using any metric:

--metric-name=<metric_name>

11) System Saturation Collector Scripts

  • taf_collect_mpstat.sh
  • taf_collect_perf_sched.sh
  • taf_collect_pidstat.sh
  • taf_collect_sarq.sh
  • taf_start_collectors.sh
  • taf_stop_collectors.sh

12) Test Suite Improvements

All suites:

  • added client_cpu_affinity

HammerDB TPROC‑C / TPROC‑H:

  • added state directory cleanup
  • removed timing options not applicable to HammerDB

sysbench_lua:

  • added NormalizeDBType

13) New TidesDB Configuration and Test‑Case Properties

Thanks to Alex Padula creator and author of TidesDB (PR #8):

Configuration files:

  • innodb_compare_local.cnf
  • tides_compare_local.cnf
  • tides_compare_v10.cnf

Properties files:

  • sysbench_local_tidesdb_innodb.properties
  • sysbench_tidesdb10_innodb_rocks_compare.properties

14) RenameResultsTestCaseTag Utility (New)

Normalizes archived result directories and associated raw/text/CSV/JSON files based on the authoritative test_case_tag.

Patches and Fixes

Test Suite Framework

  • fixed README metadata timing
  • moved dynamic fields to WriteReadmeEnd

DatabaseSoftwareInstall

Major hardening:

  • correct package‑list parsing
  • warnings → errors
  • refuse install if directory exists
  • hard fail on invalid package validation
  • hard fail on missing install_root
  • improved ACTIVE selection logic

MariaDB / MySQL Startup (SSL)

Rewrote daemon wrapper to fix SSL initialization failures.

HammerDB TPROC‑C SSL Configuration

Corrected dictionary and parameter usage for SSL.

Contributor Acknowledgements

Virtuozzo (PR #7, PR #9)

  • Lukas Oliva — PostgreSQL plugin
  • Viktor Ganeles — recovery‑mode improvements

Virtuozzo is a MariaDB Foundation sponsor, and we appreciate their continued support and engineering contributions to TAF.

PostgreSQL Testing & Patches

  • Amrendra Kumar — early PG testing, .deb installation fixes

TidesDB (PR #8)

  • Alex Padula — configuration files and properties

Hardware Sponsor Acknowledgment — Hetzner.com

TAF 4.0 was engineered and validated on high‑performance hosts provided by Hetzner.

hz-bench

  • Intel i5‑13500 (20 vCPUs / 14 cores)
  • 62 GB RAM
  • AlmaLinux 9.7

hz-bench2

  • AMD EPYC 9454 (96 vCPUs / 48 cores)
  • 125 GB RAM
  • AlmaLinux 9.8

These systems enabled full multi‑database workload testing, reproducible performance runs, and the scale required to expose real behavior across MariaDB, MySQL, and PostgreSQL.

We appreciate Hetzner’s outstanding support.

In Closing

To MySQL, PostgreSQL, and all database vendors, please take TAF performance testing framework and use it.

The MariaDB Foundation released it openly because performance change detection should benefit the entire database ecosystem, not just one project.

If TAF helps you prevent regressions, validate improvements, or simply understand your engine more clearly, that’s exactly the point.

Users deserve predictable performance, and TAF exists to make that possible for all.

About the Author
Jonathan (Jeb) Miller has experience in military leadership, Fire/EMS leadership, computer operations leadership, teaching, and more than 27 years of database performance engineering (PervasiveSQL, MySQL, MariaDB).

TAF is the third benchmarking framework he has helped design and build.