How MariaDB Server is tested
MariaDB Foundation is commited to ensuring MariaDB Server has a thriving community of developers and contributors. A software project cannot be maintained without proper tests. mysql-test-run is our standard testing toolkit for MariaDB Server. What it (mostly) does is run queries against one or more servers and compare their output to the expected one. This checks both behaviour and data consistency. The main principle is the server should always return the same data that is put into it.
One problem with this testing method is that it only covers the hosts’ environment. That means that if you are running Debian on x86_64 architecture, you are testing x86_64 architecture on a Debian Linux distribution. MariaDB Server must run on many platforms from x86 to ARM, Power, MIPS, IBM mainframes and others. MariaDB Server must also run on all popular operating systems including Linux and FreeBSD, Windows, OS X and others. Each one of these architectures must be properly tested to minimize bugs.
The build infrastructure has historically run on a deployment of buildbot hosted at buildbot.askmonty.org, currently a shared resource between MariaDB Foundation and MariaDB Corporation.
Testing infrastructure available to community
The MariaDB Foundation is commited to improving our testing efforts. This includes providing the same level of testing for the community. We do this by upgrading to a newer version of buildbot, which we now host at buildbot.mariadb.org. The new buildbot has more functionality and allows us to provide the same level of testing to community pull requests. For a while now, we have been running tests for pull-requests as well. Contributors can now check the correctness of their patches directly on GitHub!
As MariaDB Foundation is a non-profit, we could not make this project happen without our sponsors. We are now running 3 large dedicated builders, one donated by Hetzner since the project started more than a year ago and one Power9 machine donated by IBM. Recently, we have expanded our worker fleet with a powerful ARM64 machine donated by Huawei.
For a full list of sponsors, please visit buildbot.mariadb.org/#/sponsor/. Each machine, big or small helps the system perform better and run more tests!
Plans going forward
We have a broader scope for buildbot.mariadb.org. Additionally we plan to implement protected branches and staging trees. With these changes we can ensure that every commit on the main branches passes all tests in all environments.
We will be adding more builder configurations, extending automation and fixing sporadic failures. We are looking for feedback from the community! Let us know what else you would like to see from buildbot.mariadb.org on our mailing lists maria-developers and maria-discuss.