ColumnStore Native Part of MariaDB 10.5

On Wednesday 24 June 2020, MariaDB Server 10.5 was released GA. While there are several cool new features included, this first 10.5 blog is about the groundbreaking new component, ColumnStore.

The big picture

ColumnStore brings data warehousing to the world of MariaDB Server. It’s a columnar storage engine (Wikipedia: Column-oriented DBMS), enabling what is often called HTAP (Wikipedia: Hybrid Transactional/Analytical Processing). The Columnar data type stores data by columns, not by rows, enabling quick analytical reporting over huge data volumes.

The news

The ColumnStore engine is part of MariaDB Server 10.5. Earlier, ColumnStore was available to the open source community through a separate fork of MariaDB. This meant connecting data between the two required the use of non GPL versions of MaxScale. Now, once installed, moving data from InnoDB to ColumnStore is as easy as moving data from any MariaDB storage engine to another one. 

What you need to know

While MariaDB 10.5 is GA, the ColumnStore storage engine is beta, which is the typical process we follow when adding new storage engines. Independent maturity states is a benefit of the Storage Engine architecture. The integration of ColumnStore into 10.5 is done and as mature as the overall MariaDB Server 10.5 itself is. Hence, work to fix any issues found in the new ColumnStore storage engine can proceed independently, and ColumnStore can be declared GA in the following months. (Note that one can use the INFORMATION_SCHEMA.PLUGINS table to see maturity of plugins).

ColumnStore is an integrated part of MariaDB Server 10.5 and can be deployed similar to other storage engines such as MyRocks or Spider. To enable ColumnStore, you need to plug it in, which means a few more installation steps (see getting started below).

MariaDB ColumnStore does not currently support all the same operating systems as MariaDB Server (currently no Windows, no Mac OS X). Dependent on the flavour of Linux you use, download and installation will be different.

Why this is important for the open source community

From a MariaDB community perspective, getting analytical capabilities into MariaDB Server is great news. It opens up an entire new class of applications, available with the same skill set that MariaDB users already have. So far, Open Source database users have not had the luxury of hybrid transactional/analytical processing within the same product. They have had to turn to proprietary solutions, such as various flavours of Oracle Database, IBM Db2, and Microsoft SQL Server, or Platform as a Service solutions like Amazon Redshift or Microsoft Azure SQL Data Warehouse. 

From a MariaDB Foundation perspective, the ColumnStore engine has already been immersed in the open development model for a while. The project’s code is stored on GitHub (https://github.com/mariadb-corporation/mariadb-columnstore-engine/) and the issues are managed in Jira (https://jira.mariadb.org/projects/MCOL). There are many community contributions, the largest individual one being  https://jira.mariadb.org/browse/MCOL-265, support for TIMESTAMP (https://github.com/mariadb-corporation/mariadb-columnstore-engine/pull/739). MCOL tasks have been offered for Google Summer of Code. Hence we are talking about a first-class citizen of the MariaDB world, which isn’t much of a surprise, given that work on integrating ColumnStore into MariaDB Server started several years ago.

Getting started

Trying out ColumnStore starts the usual way with installing 10.5 packages from https://downloads.mariadb.org/mariadb/repositories/, then make ColumnStore executable, and install ColumnStore with either yum or apt. Find step by step instructions for deployment linked from https://mariadb.com/kb/en/columnstore-getting-started/ (and we know there are quite a few steps; the ColumnStore team is looking into simplification possibilities).

Though ColumnStore is newly available in MariaDB Server, it has been available for years from MariaDB Corporation as part of their enterprise-focused product offering. MariaDB Corporation has made its ColumnStore documentation available. To make it easy to find, we have put together a collection of pointers on https://mariadb.com/kb/en/mariadb-columnstore/. 

How to provide feedback

The previously mentioned ColumnStore Jira can be used for community feedback of any sort.

Documentation feedback on ColumnStore can be submitted using the BugHerd link in the lower right-hand corner of documentation pages (https://mariadb.com/docs/ see “Load Feedback Tool -> Send Feedback …).

Thank you!

All in all: Thank you MariaDB Corporation for successfully integrating ColumnStore to be a full component of MariaDB Server 10.5! The MariaDB Foundation sees this as validation of two things close to our heart: the open development model of MariaDB Server, and the open Storage Engine Architecture. Good job!