Author: Jonathan Miller
-
TAF 3.0 — Results Backend With Automated Performance Change Detection
TAF 3.0 introduces the new TAF Results Backend, a structured results database and parser pipeline that delivers fully automated performance change detection. This system uses a deterministic workload hash, schema‑driven baselines, and stored‑procedure‑driven comparison. No procedural comparison code. No special‑case logic. Everything is clean and automatic.
Workload Hash
Every test run gets a workload hash and parser builds it from:
This hash is the identity of the workload.Runs with the same hash belong to the same workload group.
Baseline
A […]
-
Simple tool to build MariaDB commits for performance-change analysis
Tracking down changes in database performance is one of the hardest parts of engineering, especially when the change is buried somewhere in a long commit history.
To make this work easier and more repeatable, I put together a small but important tool:
MariaDBCommitBuilder.sh
This script does one thing well:
Given a commit hash, it checks out that commit, builds it cleanly, and packages it in a deterministic way so the Test Automation Framework (TAF) can run consistent performance tests.
Why […]
-
MariaDB Foundation: Bringing TPC-B Back To Life
Pervasive PSQL
TPC-B
Pervasive PSQL
TPC-C
https://en.wikipedia.org/wiki/Pervasive_Software
During that time, about seventy percent of all regressions came from TPC-B, with the remaining issues coming from ATOMICs and TPC-C. TPC-B was simple, direct, and very sensitive to performance changes. It caught issues that other workloads did not.
Moving to MySQL AB, and working on the MySQL NDB Cluster and MySQL Global Replication teams, I created a TPC-B style test that could be used inside mysql-test.
That version is still visible today:
-
MariaDB Foundation releases the BETA of the Test Automation Framework (TAF) 2.5
The MariaDB Foundation is releasing the BETA version of the Test Automation Framework (TAF) 2.5. This release represents a significant architectural upgrade, strengthening the framework’s lifecycle model, profiling capabilities, extraction and install pipeline, and reporting consistency. The focus of the BETA is on determinism, clarity, and contributor‑proof behavior across the entire workflow.
TAF continues to serve as an open, reproducible framework for evaluating MariaDB builds, running performance workloads, collecting diagnostics, and validating behavior across versions. The 2.5 BETA builds on […]
-
MariaDB Foundation Releases Alpha of the Test Automation Framework (TAF)
The MariaDB Foundation is releasing the alpha version of the Test Automation Framework (TAF), an open-source benchmarking framework designed for clarity, repeatability, and vendor-neutral testing. TAF provides a structured way to run database benchmarks using consistent workloads, configuration, and reporting pipelines, making results easier to reproduce and discuss.
TAF has evolved into a modular system built around three plugin types: database maker plugins, test suite plugins, and report plugins. This architecture keeps the core stable while allowing contributors to extend […]
-
The Real Operational Cost of Vacuuming in PostgreSQL
There was a time when PostgreSQL’s own developers were far more open about the real cost of their MVCC design. Back in the 8.1 era, the documentation spelled out the resource drain, the vacuum overhead, and the “cold comfort” of wraparound risk in plain language. I first read that line in 2003, and it stuck with me for more than twenty years. I could not imagine any serious operations team wanting a database that required a separate background process to […]
-
Why sysbench‑tpcc results on outdated hardware should not be presented as a valid OLTP vendor comparison
Benchmark results only have meaning when the workload, hardware, and methodology are clearly defined and reproducible. When those elements are unclear or incomplete, the conclusions can easily mislead readers into assuming the results represent something they do not.
That is the core issue with the recent Percona post comparing MySQL, Percona Server, and MariaDB.
This is not about disputing Percona’s numbers. Their results may be valid for their environment.
The problem is that the post presents the results in a […]