TAF 3.0 — Results Backend With Automated Performance Change Detection

Written by Written by Jonathan Miller 1 Comment on TAF 3.0 — Results Backend With Automated Performance Change Detection

TAF 3.0 introduces the new TAF Results Backend, a structured results database and parser pipeline that delivers fully automated performance change detection. This system uses a deterministic workload hash, schema‑driven baselines, and stored‑procedure‑driven comparison. No procedural comparison code. No special‑case logic. Everything is clean and automatic.

Workload Hash

Every test run gets a workload hash and parser builds it from:

  • test + suite identity
  • system identity
  • database maker + engine
  • database version normalized to major only (MariaDB, MySQL, PG, any maker)
  • configuration identity
  • workload parameters (threads, rows, tables, ranges, warmup, connector)
  • harness + client versions
  • iteration count
  • requested duration

This hash is the identity of the workload.
Runs with the same hash belong to the same workload group.

Baseline

A baseline is simply a prior run with the same workload hash that has been marked as a baseline.

Each baseline has:

  • baseline date
  • baseline comment
  • full metrics
  • full iterations

TAF keeps all baselines, giving you a complete history.

When A New Run Arrives

  • The workload hash groups it
  • The backend finds the latest baseline for that hash
  • That baseline becomes the reference for automated compare

BackendParser.jar

What the Java Parser (BackendParser.jar) Actually Performs:

  • Parse raw results
  • Convert raw output into structured rows for test_run, iterations, and metrics.
  • Ingest data via stored procedures
  • The parser calls SPs to insert test_run, iteration rows, and metric rows.
  • Compute the workload hash and assign it
  • The parser builds the identity string, computes the SHA‑256 hash, and assigns it to the test_run through the ingestion SP.
  • Call the stored procedures that perform automated compare
  • The parser calls SPs that:
    • locate baselines with the same workload hash
    • select the latest baseline
    • compute metric deltas
    • compute duration drift
    • apply warning / fail / gain thresholds
    • classify the result
    • store the comparison outcome

The parser calls the check_for_baseline stored procedure to compute comparisons.

Database Plugins — Version 3.0

Database plugins were upgraded to support the makers plugins and runtime architecture.

  • MariaDB Plugin v3.0
    • Added runtime_dir support
    • Auto‑derived bindir, libdir, plugin_dir
    • Exported PATH + LD_LIBRARY_PATH
    • Updated bootstrap, install, start, stop logic
    • Improved error handling and validation
  • MySQL Plugin v3.0
    • Added runtime_dir support
    • Auto‑derived bindir, libdir, plugin_dir
    • Validated install root + runtime_dir
    • Exported environment for correct binary/library resolution
    • Improved version + SSL flag detection
    • Strengthened runtime path normalization

Runtime Directory Architecture

TAF 3.0 introduces option db_runtime_dir as the authoritative runtime location:

  • pid files
  • sockets
  • logs
  • bootstrap artifacts
  • install‑db logs
  • error logs

New Files and Settings Added in TAF 3.0

TAF 3.0 adds a small set of backend, parser, configuration, and comparison files that enable the new Results Backend and automated performance change detection. Below is a short, accurate description of what each item is in the release.

Backend / Parser Components

  • backend_reporter.pm
    • The reporter plugin that interfaces directly with the TAF Results Backend, piping results from the current run into the backend and triggering automated compare. It can be included as a standard taf.report_plugin. The backend must already be up, reachable, and configured so the reporter can connect.
  • backend.conf
    • Configuration file for the parser (connection info, schema name, runtime settings).
  • backend_parser
    • Directory containing the parser implementation, including source/ and working/.
  • backend_parser/source
    • Full Java source tree for the parser.
    • Users compile their own parser from this directory.
  • backend_parser/working
    • Runtime directory containing build artifacts produced by the parser build script.
  • backend_sql
    • Directory containing backend SQL artifacts.
  • create_backend.sql
    • SQL file that creates the backend schema, tables, stored procedures, and compare logic.
  • build_backend_parser.sh
    • Script that compiles the parser Java sources and produces the parser JAR.
    • Users run this to build their own parser.
  • parse_raw_text_results.sh
    • Script that invokes the parser on raw text result files and pipes them into the backend.

New TAF Properties

These properties expose backend, parser, runtime, and threshold settings to the framework.

  • Backend + Parser
    • taf.backend_config
      • Path to the backend configuration file (connection, schema, runtime settings).
    • taf.backend_parser_jar
      • Path to the parser JAR that TAF launches to ingest results.
    • taf.java_bin
      • Java executable used to run the parser.
    • taf.jdbc_jar
      • JDBC driver required for the parser to connect to the backend database.
    • taf.json_jar
      • JSON library required by the parser for structured result parsing.
      • These tell TAF where the backend is, where the parser is, and how to run it.
  • Runtime
    • taf.db_runtime_dir
      • Directory where the database plugin places runtime artifacts (pid, logs, sockets).
    • taf.db_process_pid
      • PID of a database process that was already running outside of TAF; passed to the CPU‑rest monitor so it knows which external DB instance to watch for rest before moving to the next iteration/thread count.
  • Automated Compare Thresholds
    • taf.results_threshold_warning_pct
      • Percentage regression that triggers a WARNING classification.
    • taf.results_threshold_fail_pct
      • Percentage regression that triggers a FAIL classification.
    • taf.results_threshold_gain_pct
      • Percentage improvement that triggers a GAIN classification.
    • taf.results_iteration_allowed_duration_drift_pct
      • Allowed iteration‑duration drift when validating baseline compatibility.

New Script

  • database_size.sql
    • SQL script for collecting database size metrics.

In Closing

Thanks for being here and for continuing to push TAF forward.

We hope the changes in TAF 3.0 make your benchmarking, regression hunting, and database development faster, cleaner, and more predictable. This release is a foundation, and like every part of TAF, it grows through the work of people who care about performance, correctness, and reproducibility.

For anyone setting up the new Results Backend, we’ve included a detailed PDF in taf-perl/help (TAF-PERL_Results_Backend.pdf). It walks through backend creation, configuration, parser setup, ingestion flow, and automated compare behavior in far more depth than this blog. If you’re deploying the backend for the first time, that document will guide you through every step.

We are always looking for contributors. Some current needs include:

  • PostgreSQL database plugin
  • Web site / UI for the new TAF Results Backend
  • New profiler plugins
  • New report plugins
  • New test suite plugins

All contributions are welcome, encouraged, and absolutely appreciated. If you build something, improve something, or even just point out something that should exist, you’re helping move the entire benchmarking ecosystem forward. Thank you for being part of it.

And one final point: it really does not matter whether a change is a regression or a gain — every change needs to be understood. Automated detection is only the beginning. The real value comes from knowing why something moved, and ensuring that every shift in performance is visible, explainable, and never ignored. You never know when a GAIN is caused by a functional regression, and that’s exactly why TAF treats all change as equally important.

Reflection

I keep thinking about something Sergei Golubchik said to me not long ago, about wanting a box in the corner that just runs the automation and handles the results. I spent years saying that exact line to Trim Pershad when I was pushing MySQL System QA toward real, continuous performance testing. When I finally got moved into System QA for MySQL Server performance change detection, Trim said, “Make it happen.” It took a few versions to get the discipline, the tooling, and the flow right, but by the time I left we had more than 80 hosts and VMs running continuous MySQL performance change detection tests. Hearing Sergei say the same words back to me decades later made me smile, because it reminded me that all of this — the framework, the backend, the reproducibility, the comparisons — starts with the same simple idea: put a box in the corner and let it tell the truth.

PS: Database Providers

To MySQL, PostgreSQL, and all database vendors — please take this performance testing framework and use it. The MariaDB Foundation released it openly because performance change detection should benefit the entire database ecosystem, not just one project. If this helps you prevent regressions, validate improvements, or simply understand your engine more clearly, that’s exactly the point. Users deserve predictable performance, and TAF exists to make that possible for all.

About the Author
Jonathan (Jeb) Miller has experience in military leadership, Fire/EMS leadership, computer operations leadership, teaching, and more than 26 years of database performance engineering (PervasiveSQL, MySQL, MariaDB). TAF is the third benchmarking framework he has helped design and build.