A mirror manager system to simplify the distribution of MariaDB binaries and packages

The MariaDB Foundation relies on public mirrors to distribute binaries and packages to the world. An overview of the full list can be seen at https://mirmon.mariadb.org. If you would like to volunteer to become a mirror, take a look at https://mariadb.com/kb/en/mirror-sites-for-mariadb/.

Our download page already automatically suggests one of those mirrors for you to download our binaries. Same goes for the repository configuration tool; see https://mariadb.org/download.

The purpose of this blog post is to present you a new system that we have been testing since December 2021 and that we believe is now ready for public adoption.

Mirrorbits

VLC folks developed a tool that can act as “a geographical download redirector”. And we decided to give it a try since they seem to successfully rely on it to distribute the famous video player.

The system is managed by us and is hosted by our cloud provider, Hetzner, in Germany. It is available at https://mirror.mariadb.org. On a first look, it seems like just another nginx auto-indexed website.

But if you take a look at https://mirror.mariadb.org/mirrorstats, then you can see all the mirrors behind this service with their status.

How does it work?

The mirrorbits tools will regularly scan (via FTP or RSYNC) a list of defined mirrors and compare their content with its local content. If the content is available (and has the same sha256), the mirror can be elected to serve the content.

Mirrorbits will then use a Maxmind geoip database to decide which mirror to pick on every request it receives. You can see this “redirect” decision by adding ?mirrorlist at the end of the URL, example with https://mirror.mariadb.org/repo/10.6/debian/sha256sums.txt?mirrorlist.

And finally, but probably most importantly, unavailable mirrors are automatically disabled by mirrorbits if it cannot reach them (ping every minute).

(deb|rpm).mariadb.org for repository configuration

For ease of use, we have created new sub-domains (CNAME on mirror.mariadb.org with URL rewriting) that you can use.

For Debian based distribution, you can use deb.mariadb.org in your sources.list. Here is an example of /etc/apt/sources.list.d/mariadb.list file for Debian Bullseye and MariaDB 10.6:

deb https://deb.mariadb.org/10.6/debian bullseye main

Same goes for Red Hat based distributions, the domain rpm.mariadb.org can be used, and here is an example of /etc/yum.repos.d/mariadb.repo for Fedora 34 and MariaDB 10.7:

[mariadb]
name = MariaDB
baseurl = https://rpm.mariadb.org/10.7/fedora34-amd64
gpgkey=https://rpm.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

Some drawbacks

Internet is a best-effort network, what is working at T can be down at T+1s (and running again at T+2s). Our mirrorbits system may be able to ping a mirror but you not (and vice et versa).

This system is a clear SPOF and if the mirrorbits system has downtime then we might impact lots of people using it.

Mirrorbits needs FTP or RSYNC available on mirrors in order to be able to scan them, but not all mirrors provide it, so, it only uses a smaller number of mirrors.

For every file that you want to download, a first request will need to reach the service that is, as already said, located in Germany, so, it may add some latency for you if you are living very far from this country.

For those reasons, we consider this system as an alternative to what already exists and if you prefer to manually pick a mirror that works for you, please continue doing so!

CI context

In a CI context, where you may not know where your CI runner/workers are (geographically), we believe that this service makes a lot of sense, and in that context we highly recommend it.