MariaDB Foundation: Bringing TPC-B Back To Life

When I joined Pervasive PSQL, one of the first performance test cases I was introduced to was TPC-B. It was already implemented inside Pervasive PSQL and it quickly became one of the most important tools in my daily work. A little later, another developer and I wrote the Pervasive PSQL’s TPC-C implementation in C++. Between those two workloads, and a few others, I spent nearly five years performing change testing. (https://en.wikipedia.org/wiki/Pervasive_Software)

During that time, about seventy percent of all regressions came from TPC-B, with the remaining issues coming from ATOMICs and TPC-C. TPC-B was simple, direct, and very sensitive to performance changes. It caught issues that other workloads did not.

Moving to MySQL AB, and working on the MySQL NDB Cluster and MySQL Global Replication teams, I created a TPC-B style test that could be used inside mysql-test.

That version is still visible today:

Even though the TPC organization discontinued TPC-B, and many vendors stopped using it, the workload itself never stopped being useful.

For simple select, update, and insert sensitivity, TPC-B can still detect regressions faster than TPC-C.

Many MariaDB users may know sysbench or TPC-C, but not TPC-B. 

TPC-B was a transaction processing benchmark built around simple banking style operations. Its value was not complexity, but sensitivity. A small number of predictable selects, updates, and inserts could expose regressions very quickly.

TPCB: https://www.tpc.org/tpcb/default5.asp

Sysbench-lua already provides everything needed to simulate TPC-B without writing new Lua. Because of that, I added two new test cases to the TAF sysbench-lua test suite.

TPCB_KEY     Updates hits key values

TPCB_NO_KEY  Updates hit non key values

These follow the original TPC-B pattern of three point selects, three updates, and one insert. 

The only difference is whether the update hits a key or non key column. With the insert, a delete occurs first, matching the original TPC-B behavior.

This brings a piece of historical performance testing back into modern automation where it still has real value.

The TAF commit is here:   

Followup patch, correcting updates from one to three:

I want to thank Amrendra for his recent contribution to TAF. He fixed an issue with oltp_scan when selecting BMK sysbench, and improved TAF Utilities to use the correct hostname call.

MariaDB Foundation provides TAF as fully open source. Anyone can download it, use it, and help improve it.

The goal is simple. Help TAF grow and become the standard for benchmark automation.

TPC-B is one workload. The backend is what makes those workload results comparable and actionable over time.

Work is ongoing on a backend for benchmark results. The goal is to make it easy to store performance results, compare runs, detect regressions, and automate performance reporting.

The backend parser can ingest results from a TAF plugin library or from raw text output.

Examples from taf_backend:

We are in the final development stages now, testing and documenting. The Foundation hopes to have the backend released soon.