MariaDB Foundation releases the BETA of the Test Automation Framework (TAF) 2.5

Written by Written by Jonathan Miller 1 Comment on MariaDB Foundation releases the BETA of the Test Automation Framework (TAF) 2.5

The MariaDB Foundation is releasing the BETA version of the Test Automation Framework (TAF) 2.5. This release represents a significant architectural upgrade, strengthening the framework’s lifecycle model, profiling capabilities, extraction and install pipeline, and reporting consistency. The focus of the BETA is on determinism, clarity, and contributor‑proof behavior across the entire workflow.

TAF continues to serve as an open, reproducible framework for evaluating MariaDB builds, running performance workloads, collecting diagnostics, and validating behavior across versions. The 2.5 BETA builds on the earlier Alpha release by refining the internal architecture and expanding the framework’s ability to support community testing.

https://github.com/MariaDB/TAF

Unified and explicit lifecycle model

TAF 2.5 introduces a formal database lifecycle model that defines how the database behaves before each iteration. The new database_iteration_mode supports four explicit modes:

  • preserve
  • restart
  • reset
  • restore

These modes remove ambiguity and ensure that each iteration begins from a well‑defined state. The restore mode is backed by a new restore‑image subsystem capable of copying or unpacking pre‑built images in multiple formats. This enables reproducible iteration baselines and consistent behavior across test runs.

Additional lifecycle improvements include:

  • explicit db_start_wait and db_stop_wait controls
  • corrected handling of database configuration files
  • SSL detection and support for --skip-ssl
  • integration of lifecycle behavior with warmup and test setup sequencing

The result is a deterministic iteration model that is easier to reason about and more reliable under automation.

Modular profiling subsystem

The BETA introduces a modular profiling subsystem designed to integrate cleanly with the iteration lifecycle. Profilers are now implemented as plugins under libs/profile_libs, with a unified runner that manages setup, execution, and output isolation.

The initial plugin wraps Linux perf and supports:

  • per‑iteration profiling
  • optional continuous profiling
  • configurable start delays and durations
  • automatic flamegraph generation

Profiling output is isolated per iteration, and profiler failures no longer affect the main run. This provides a more robust and extensible foundation for performance analysis.

Expanded SQL and OS‑script hooks

TAF 2.5 adds new hook points that allow SQL and OS‑level actions to run at key stages of the lifecycle:

  • after database start
  • before and after test setup
  • before and after each iteration

These hooks make it possible to integrate diagnostics, environment preparation, and custom logic without modifying the core framework. They also improve the framework’s ability to support complex test scenarios and engine‑specific workflows.

Strengthened extraction and install pipeline

The extraction and install pipeline has been significantly improved for clarity, safety, and determinism. Key enhancements include:

  • stronger validation and error handling
  • support for .tar.xz and nested bundle tarballs
  • deterministic directory layout normalization
  • safer temporary directory cleanup
  • explicit install directory naming rules
  • early safety checks to prevent accidental overwrites

These improvements make the install process more predictable and easier to automate, especially in continuous testing environments.

Correct warmup and setup sequencing

The BETA corrects long‑standing issues in warmup and test setup sequencing. The new test_setup_mode integrates directly with the lifecycle model and supports:

  • skip
  • once
  • per_test
  • per_iter

Warmup iterations now behave consistently and no longer interfere with reporting or iteration accounting. This ensures that test setup and warmup behavior is predictable across workloads.

Expanded MariaDB SQL diagnostics

TAF 2.5 includes a significantly expanded set of MariaDB SQL diagnostics covering:

  • performance_schema
  • InnoDB metrics
  • global status and variables
  • explain plan suites
  • storage engine statistics

These diagnostics provide deeper visibility into server behavior during benchmarking and regression analysis, improving the framework’s usefulness for both developers and testers.

Licensing and metadata updates

All modules in the framework now include GPLv2+ headers. Metadata has been updated, and the framework revision has been bumped to 2.5. This ensures consistent licensing across the entire codebase.

Updated properties and driver options

The default properties have been updated to include:

  • lifecycle controls
  • restore‑image configuration
  • profiling defaults
  • warmup and setup‑mode options
  • SQL and OS‑script hook configuration

Deprecated flags have been removed.
The main driver (taf.pl) has been updated to expose the new lifecycle, profiling, and hook options.

New configuration and diagnostic files

The BETA includes new:

  • sysbench Lua scripts
  • MariaDB and MySQL configuration files
  • profiling libraries and flamegraph tooling
  • SQL diagnostics for explain plans, performance schema, InnoDB, and status

These additions expand the framework’s coverage and improve out‑of‑the‑box usability.

Summary

The TAF 2.5 BETA release is a structural upgrade. It focuses on determinism, clarity, and architectural correctness rather than feature expansion. The result is a framework that is easier to maintain, easier to extend, and more predictable for contributors and users.

The Foundation welcomes community feedback as TAF continues to mature.

About the Author
Jonathan (Jeb) Miller has experience in military leadership, Fire/EMS leadership, computer operations leadership, teaching, and more than 26 years of database performance engineering (PervasiveSQL, MySQL, MariaDB). TAF is the third benchmarking framework he has helped design and build.

#MariaDB #MariaDBFoundation #TAF #TestAutomationFramework #OpenSource #DatabasePerformance #Benchmarking #PerformanceTesting #WorkloadTesting #ReproducibleBenchmarks #Automation #DevOps #ContinuousTesting #PerformanceChangeDetection #DatabaseMakers #Sysbench #HammerDB #BenchmarkKit #MySQL #PostgreSQL #OpenBenchmarking #ReproducibleResearch #DBOps #EngineeringExcellence #CommunityDriven #ContributeToOpenSource