MariaDB has been using a pluggable storage engine architecture for a long time and whilst this means great flexibility in choosing and managing the right storage engines for specific use cases, it also means they are easier to develop and therefore there’s an expectation that more engines will be created.
More storage engines means the MariaDB Server itself needs to be as flexible as possible to accommodate all sorts of functionalities that storage engines may need. One area where the MariaDB Server proved to be not that welcoming was in making available all the compression libraries needed by storage engines.
As MariaDB is getting more storage engines and as both new and existing engines are getting more features, MariaDB can potentially use more and more compression libraries for various purposes.
InnoDB, RocksDB, Mroonga – they all can use different sets of compression libraries. From the MariaDB Server’s perspective, this was a kind of an “in a pickle” situation. Compiling all these libraries in would result in a lot of runtime/rpm/deb dependencies, most of which would never be used by most of the users. Not compiling them in, would result in requests from users to compile them in (thus making them available in some storage engine features).
Most users don’t use all the compression libraries, many users use some of these libraries.
This project was tracked in Jira ticket MDEV-12933 and in summer 2020, it got accepted as a project for the Google Summer of Code program. Kartik Soneji, mentored by Robert Bindar, implemented a solution for loading compression libraries on request, at runtime, without creating packaging dependencies.
The initial solution implemented by Kartik Soneji got picked up by Sergei Golubchik who fine-tuned and integrated it into MariaDB Server. “Compression provider plugins” offered in this MariaDB 10.7 preview-release is the result of this work.
New architecture
Before MDEV-12933, for a compression algorithm to work, MariaDB had to be linked with the corresponding compression library. If for instance, a InnoDB user wanted to use the LZO compression, the user had to either use our .rpm packages for openSUSE or compile MariaDB from sources. Because neither our binary tarballs, nor our .deb packages and our other .rpm packages were compiled with LZO support. If one wanted LZ4 support — it meant using .deb packages or .rpm on Fedora. And no packages from mariadb.org provided both LZO and LZ4.
But now, with MDEV-12933 in, one does not have to choose anymore. MariaDB comes with the support for all compression algorithms compiled in, but with none of the dependencies. Compressions algorithms are disabled until the corresponding plugin is loaded. Their installation doesn’t impact the normal workings of the MariaDB Server for other users too, so everybody is happy.
Server packages have no dependencies either, neither .rpm nor .deb server packages require any compression libraries to be installed. If you want to use compression — you need to install the corresponding provider plugin package. This is very important to understand — if you have used InnoDB compression before, after upgrading to 10.7 you must install compression provider plugins that you need, otherwise your tables will be unreadable.
Installing/removing a compression provider plugin looks exactly like for any other MariaDB plugin. Here’s an example for the LZ4 algorithm, and please note that this preview was shipped with multiple provider plugins (bzip2, lzma, lz4, lzo and snappy), feel free to try those as well.
set global innodb_compression_algorithm = lz4; # Throws error when plugin isn't loaded ERROR 42000: Variable 'innodb_compression_algorithm' can't be set to the value of 'lz4' install soname 'provider_lz4'; select plugin_name, plugin_status from information_schema.plugins where plugin_name='provider_lz4'; # Check plugin status or use SHOW PLUGINS plugin_name plugin_status provider_lz4 ACTIVE set global innodb_compression_algorithm = lz4; # Now it works
And, like with any other plugin, if you install an .rpm or .deb package with a plugin, it will be auto-installed via the .cnf file.
Once you have the provider plugin installed, creating a InnoDB compression-enabled table should be trivial:
set global innodb_compression_algorithm = lz4; create table t (a int, b text) engine=InnoDB page_compressed=1; insert t1 (a, b) values (0, "abc");
Note that provider plugins cannot be unloaded. InnoDB will not expect a compression library to disappear right under its feet in the middle of a table access, so while you can load a provider plugin any time, the only way to get rid of it is to restart the server.
Note that currently only InnoDB and Mroonga use compression provider plugins. RocksDB still links with compression libraries directly, because RocksDB packages include separate utility executables (sst_dump and mariadb-lld) and they cannot load server plugins.
How to try the 10.7 provider plugins preview yourself?
Tarball
Go to tarball downloads.
Container
To test this preview release with as little effort as possible, you can run the container called quay.io/mariadb-foundation/mariadb-devel:10.7-mdev-12933-provider-plugins with the same interface as the Docker Library mariadb image.
Feedback welcome
If you come across any problems in this feature preview, with the design, or edge cases that don’t work as expected, please let us know with a JIRA bug/feature request on the MDEV project. You are welcome to chat about it on Zulip.
See also the Knowledge Base page for further details.