At the MariaDB Foundation, we want MariaDB Server to be a model citizen in the Open Source world. For now, there is a sizeable gap between dreams and reality. But that doesn’t stop us from striving to improve. Let me here describe some of our challenges, and share some visions of where we want to be.
Continuous Integration
One pain point is the state of the development tree. A model citizen would ensure that the tree can always be built. Every day, all the features under development could be tested by the community, on all platforms. All code pushes would be committed only on condition that the code not only compiles on the individual developer’s laptop, but on all platforms. And on condition that the code passes the regression tests. On all platforms.
We’re not yet in a situation where the MariaDB Server tree always passes all tests on all platforms, but we’re working on it. Every developer, every contributor should be able to rely on the main development branch to work, so that the bugs I encounter when developing are mine only.
Our continuous integration (CI) is based on Buildbot, and theoretically, it enables us to go far towards being model citizens. But in practice, we cannot run all tests on all commits as a precondition for pushing the code. It takes too long (several hours) to run the tests, and particularly sporadic failures are a challenge, as they don’t happen every time.
Our MariaDB Server’s key role is to function in the environment of our users, and the interoperability of MariaDB Server with the user’s data and software is something that will remain of critical importance. For popular Linux distributions and client connectors, we perform some tests ourselves. However, we’ll need to provide the ability for interested parties to take part in our CI process.
It’s difficult to predict when a feature will be ready
According to old school wisdom, you cannot define both a deadline, a feature set, and a quality level. Pick two. And old school has it right.
However, as users are always asking for them all – features, by a deadline, without bugs – we will inevitably have to strike some compromise between the three.
The pain point here is agreeing on what defines a release. We do not compromise on quality, but shall we release at a fixed deadline or when a certain feature set is implemented?
This discussion has existed since MariaDB Server was forked off MySQL. Until MariaDB 10.1, we did feature-based releases. Now, there is a broad understanding that users expect fairly regular releases at a fairly frequent cadence. Since 10.2, we have done time-based releases, with various levels of success. For 10.6, we abandoned the time constraint, and shifted in a feature-based direction again. However, irregular feature-based releases with long intervals between them create a vicious circle of delaying a release even more, just a little bit, to get just one more feature in. But a release cannot have it all, we must not try to create what the Germans call an “eierlegende Wollmilchsau”. An egg laying wool milk sow is wishful thinking.
Missing a train is expensive
But time-based releases have their drawbacks too. If a feature isn’t ready by the release deadline, it will have to wait for the next release. This is called the “train model”. And we all hate missing a train.
Still, there are trains and there are trains. Missing a subway train or a tram isn’t nearly as upsetting. You just wait five-ten minutes for the next one. It might look that changing the train model to accommodate more frequent releases — shall we call it the “tram model”? — would significantly reduce the pain of missing a release. Unfortunately, it is not that simple.
Quality needs time too
There is a fundamental contradiction between releasing more frequently and releasing a good quality, mature product. First, we release an alpha version of the new MariaDB. Some users start trying it out, they find and report bugs. When the stream of incoming bugs has slowed to a trickle, we release a beta version of MariaDB. More users download and try it. More bugs are being discovered and reported. And later we produce a Release Candidate version to reach even more users — it is supposed to be almost, but not quite, of the production quality. And after fixing those last bugs we declare this new MariaDB version stable (or GA for General Availability).
This means a release has to mature on its own schedule, we cannot simply will it to be stable to save the time. But perhaps it might be possible to compress the time somewhat by doing intensive internal testing before the very first release. Ideally, intensive targeted testing can allow us to fast-forward through early maturity phases, discovering those bugs in days and weeks, not months.
Early Access
Still, no internal testing can tell us what users do — whether new features solve user problems, whether they fit into user applications, whether they do what users expect them to. Internal testing can only tell us whether a feature works as we designed, but only users can tell whether we have designed it correctly. It is absolutely crucial to provide early access to new features to be able to discover and correct those problems that no internal testing can find.
There is a lot to be won
MariaDB Server is doing quite well, quality wise. We manage to keep up with bugs, even though MariaDB popularity is growing (according to Debian’s popcon). This means, one could say the quality is actually getting better.
Yet, there is quite a lot of potential to get closer to being a model Open Source citizen, through changes in the engineering practices – in the areas of continuous integration, testing, and striking the right balance between timeliness and feature set. Some of the culture changes need to come from within, but part of our desire to be model open source citizens is to get input from the community.
There will be changes coming up. We won’t be able to solve all of these overnight, and we don’t expect everything to work perfectly the first time. For further plans and progress reports on incremental improvement, watch this space!