MariaDB Foundation Releases Alpha of the Test Automation Framework (TAF)

Written by Written by Jonathan Miller Leave a comment on MariaDB Foundation Releases Alpha of the Test Automation Framework (TAF)

The MariaDB Foundation is releasing the alpha version of the Test Automation Framework (TAF), an open-source benchmarking framework designed for clarity, repeatability, and vendor-neutral testing. TAF provides a structured way to run database benchmarks using consistent workloads, configuration, and reporting pipelines, making results easier to reproduce and discuss.

TAF has evolved into a modular system built around three plugin types: database maker plugins, test suite plugins, and report plugins. This architecture keeps the core stable while allowing contributors to extend the framework with new database engines, workloads, and reporting formats.

The alpha includes maker plugins for MariaDB and MySQL, test suites for Sysbench and HammerDB (TPROC-C and TPROC-H), and a set of report plugins for raw text, charts, and combined summaries. All configuration is driven by simple user properties files, ensuring that benchmark runs are reproducible and shareable. The alpha also includes user test case properties and database configuration files contributed by TidesDB, an early reviewer of the framework.

Why the Foundation Is Investing in TAF?

The Foundation is investing in TAF to:

  • Strengthen performance benchmarking, transparency, and the overall MariaDB community.
  • Provide an open, reproducible way to compare database makers using the same workloads, configuration, and reporting pipeline. Shareable properties files define each run, enabling reproducible and verifiable results.
  • Detect performance change early and consistently. TAF supports automated comparisons across runs, helping identify changes — up or down — before they reach users. This addresses a long-standing pain point in other ecosystems where performance drift often goes unnoticed.
  • Lower the barrier to meaningful benchmarking. Many tools require deep, tool-specific knowledge; TAF unifies them under a consistent workflow, making it easier for more people to contribute, following in the footsteps of experts like Steve Shaw and Mark Callaghan.
  • Enable automated performance-change detection at scale. With a front-end driver such as Jenkins and a back-end for automated comparisons to baselines, organizations can build continuous performance pipelines that validate changes and share results openly.

What Is Included in the Alpha?

The alpha release contains:

  • Database maker plugins for MariaDB and MySQL
  • Test suite plugins for Sysbench and HammerDB (TPROC-C and TPROC-H)
  • Report plugins for raw text, charts, and combined summaries
  • User properties and database configuration files contributed by TidesDB
  • A complete help system and QuickStart guides
  • A stable plugin architecture ready for community contributions

The framework is ready for real workloads, experimentation, and early feedback.

TAF: https://github.com/MariaDB/TAF

Call for Community Feedback

This alpha release is intentionally early. The Foundation wants to ensure the framework evolves in the right direction, and community input is essential. We encourage users, developers, and benchmark practitioners to try TAF, explore the plugin model, and share their experiences.

Community feedback is always appreciated.

About the Author
Jonathan (Jeb) Miller has experience in military leadership, Fire/EMS leadership, computer operations leadership, teaching, and more than 26 years of database performance engineering (PervasiveSQL, MySQL, MariaDB). TAF is the third benchmarking framework he has helped design and build.

#MariaDB #MariaDBFoundation #TAF #TestAutomationFramework #OpenSource #DatabasePerformance #Benchmarking #PerformanceTesting #WorkloadTesting #ReproducibleBenchmarks #Automation #DevOps #ContinuousTesting #PerformanceChangeDetection #DatabaseMakers #Sysbench #HammerDB #BenchmarkKit #MySQL #PostgreSQL #OpenBenchmarking #ReproducibleResearch #DBOps #EngineeringExcellence #CommunityDriven #ContributeToOpenSource