Heads up: Renewing MariaDB Downloads

Time to renew downloads.mariadb.org! We are embarking on a long project. A large part of our user base mainly interacts with us through downloading new versions. Renewing MariaDB Downloads is the biggest-impact project of all of 2020, for that part of our user base.

Projects like this often get launched when external and internal impulses coincide. Our users have asked for a more consistent, simpler user experience. Advanced users would like a programmatic interface (a.k.a. REST API) for part of the functionality. And the current codebase needs refactoring attention, as it has served past its best-before date.

We have a plan for what to do and how to do it. But we don’t claim our picture of user needs is perfect. Hence, before starting to code, we want to give everyone the opportunity of giving us input. Caveat: We promise to listen, but we don’t promise to implement all wishes.

Where we stand: How a new MariaDB version is released today

Allow me a technical detour. For most MariaDB users, this is merely to satisfy your curiosity. If you’re interested in “what’s in it for the MariaDB user”, skip ahead to “Improving the user experience”.

OK, here goes: Currently, MariaDB download pages are based on Python additions to Django code from way-back-when, when the entity providing the downloads was still called Monty Program Ab. We believe about 15% of that Python code is still relevant and usable, whereas 85 % of the code relates to other aspects than downloads.

Our intent is to put the “harvested”, reusable code as new code under Github as Open Source. Not that we intend to make it into a product, but we do believe in openness. Publishing the code as we develop it seems like the natural thing to do. Our development tasks will also be publicly available on Jira under the MFDL abbreviation (for MariaDB Foundation Downloads).

Architecture-wise, Github notifies the current Continuous Integration system, affectionately known as the “old” Buildbot. From that one, built packages are uploaded to an archive site The many releases on that site are identified by an unique ID of the final commit. Affectionate name: commit hash. Upon release, the appropriate binaries are uploaded to the primary mirror, from which all secondary mirrors take it. For historical reasons, the primary MariaDB mirror is hosted by Oregon State University’s Open Source Labs, http://ftp-osl.osuosl.org. We are grateful for their work and efforts.

The metadata used for downloads.mariadb.org is contained in a Python Pickle for deployment. Once downloads have propagated to secondary mirrors, this pickle is uploaded into the download site. Finally, the release notifications can be sent out, and a new release of MariaDB has seen the light of day.

Internal woes: How we suffer behind the scenes

Refactoring old code is a favourite pastime for programmers. From Henrik Ingo’s great Drizzle Eulogy https://openlife.cc/blogs/2019/december/lets-rewrite-everything-scratch-drizzle-eulogy (about a reduced-functionality fork of MySQL), we know that refactoring reasons are usually not user centric. Nonetheless, we do suffer from code still being on Python 2. Python Software Foundation has long since declared End of Life (EOL) for Python 2, and by now, even the declared date of 1 Jan 2020 has passed. We rely on some third-party plugins no longer maintained, and on old SSL versions. Add the lack of a testing framework plus a great deal of dead code emanating from functionality moved to new systems over the years, and you’ll understand our desire for refactoring.

Improving the user experience: Consistency, fewer clicks

Making the user interface consistent with the look and feel of the rest of the website could perhaps be called “usability refactoring”, and that one is most definitely in the interest of the user base. But it’s not just about making the existing functionality more consistent.

Most importantly, we want to detect your operating system so you will primarily be offered download packages of the operating system that you are downloading from. The goal is to optimise the process for the regular user. Select the product, press Next at most once (eg. selecting the release), and the download should start.

Currently, the choices users face are overwhelming. We intend for there to be a guide on the page to help choose the right download. Questions frequently asked of us include “what are debug symbols”, “msi vs zip”, “what is glibc and which version do I have”, “do I need the systemd version”, “when to use repos vs binary”, and “do I download the package from here or use the repo config tool”. We intend to simplify, where we can.

We will also make an effort for the default mirrors provided to be more sensible.

Our role models here are the usual suspects: MySQL, Postgres, MongoDB, and the bootable disk images for Ubuntu and Fedora.

BTW, we have been asked for other improvements of the overall download experience. While not part of the Download website renewal itself, they are on our radar screen:

A loud wish, which I personally share, is for us to provide directly downloadable OSX .dmg files.

And speaking of packaging for other operating systems: MDEV-21432 requests downloadable ARM packages. Duly noted.

Another sensible wish is for Minimum Viable Downloads (MVDs, if you will), of 50 MB or less. This is for low-bandwidth situations, or minimalist VMs / containers. And it saves space on everyone’s hard disks, including all the mirrors.

Obviously, a big package will slow down a Docker container and make it difficult to use. With the debug server and test suite included, the smallest MariaDB package with binaries included today is about 900 MB. The source tarball being much smaller doesn’t quite help, as the source tarball takes 20 mins on a laptop CPU to build. Given that Debian packages are much smaller, we do see the need of doing a honey-I-shrunk-the-packages. One thing we’re exploring is using xz instead of gzip, which gives much better compression (at a sustainable cost of added CPU when decompressing). Opinions welcome!

Improving the high-end user experience: REST API

Expert users tell us our site doesn’t provide us with the same functionality as others do. Giuseppe Maxia (who incidentally also was great at articulating the MVD need above) pointed out the need for a programmable way of starting a download, be it as simple as the website providing cut-and-pasteable copylinks. With a wget, you expect to get a tarball, but currently we provide you with an index.html file. This annoyance was solved ages ago in other download sites, adds Giuseppe.

Our plan is for a REST API to do more than just copy-and-pasteable URLs. We’ll add endpoints with JSON responses, exposed to outside users:

A list of available “products”
A list of available versions per product
The latest version of a product
A list of available architectures, OSes, file types – per product
A list of available mirrors for a binary
Fetching the binary (hi Giuseppe!)
Fetching the signature of the binary
A single URL per product, containing the latest release (like with WordPress), perhaps separated into latest-stable and latest-development

This is clearly no rocket science and just what one expects of a modern download site.