In the name of the MariaDB Foundation, I would like to thank Tencent for their significant contributions to the MariaDB Server in 2020. The permission to, and encouragement of, Tencent staff to contribute towards MariaDB shows a superior and practical understanding of the value proposition of open source in delivering value to everyone at the same time as reducing software maintenance costs.
About Tencent’s DB Kernel Team
Tencent has a cloud native database kernel R&D team, focusing on database kernel optimisation and architecture evolution. The team is experienced (over ten years of database R&D experience) and serves hundreds of thousands of enterprise users.
The Tencent team focuses on computing engine optimisation (outline and plan cache optimisation), storage engine optimisation, software and hardware integrated design. For the continuous performance optimisation of storage engines, Tencent works on secondary cache architecture, adaptive hash indices (AHI), transaction system locks, availability and various aspects of HA usage. As for software and hardware integrated design, they work on redo log and binlog optimisations based on NVDIMM.
To learn more, look at the video CDB & CynosDB with hardware and software integrated design (https://mariadb.org/fest2020/cynosdb/; also available on Bilibili). It was presented by Xinlong Li, Principal Engineer of Cloud Database at Tencent Cloud. You can also read the Zulip chat that happened during Xinlong Li’s presentation in Sep 2020. Of course, you can also learn a lot directly from Tencent’s web pages at https://cloud.tencent.com/product/cdb and https://cloud.tencent.com/product/cynosdb.
What MariaDB did with Tencent’s contributions
The Tencent contributions, while not always accepted in their original form, highlighted to the server developers the need for significant improvement in the specific functions of the server code.
Let me walk through the contributions one by one!
Drop Table Force
In addressing this multi year old improvement you highlighted the need for MariaDB to be resilient against sporadic errors and race conditions in the creation of tables. Inconsistent states of tables that exist in a partial form are a danger to the stability of the server. By introducing this feature it’s always possible for a database administrator to take control and eliminate the table even if it only partially exists. Database administrators can be left to recreate tables of the same name without a fear of triggering an anomalous condition in the server.
This contribution is a significant benefit to the user community.
Asynchronous Drop Table
The issue addressed here is that dropping large tables, rebuilding tables, adding and dropping indexes all requires significant time and during that time excessive mutexes are held. This improvement increases the concurrency of operations related to Innodb tables. The importance of getting this addressed will benefit users as their datasets grow and changes need to occur.
Accelerate buffer pool init
Practical experience in starting and stopping large instances of MariaDB by Tencent shows that even basic things like initializing a buffer pool need to be improved. The code contributed happened to coincide with a concurrent effort to address other similar bottlenecks in code. The reminder of the continued need to improve ensured that this effort didn’t stagnate and did result in improved startup time of large instances.
Parallel Initialize Rollback Segment
Less common paths like rollback segments are often left unoptimized when benchmarks consider throughput of query cases. Practical experience from Tencent led to the development of this effort. Assumptions over relative speed of storage and threads means some previous assumptions need to be re-measured. MariaDB developers await further bench-marking to show the value of this contribution.
Redo log for NVDIMM in MariaDB
The exploration of new technologies such as NVDIMM and its astute application in critical areas of MariaDB shows an attentive detail to new technologies and the operations of MariaDB by Tencent. This combined with an expertise in other concurrent development efforts will eventually materialize into a better server performance where such hardware exists. Tencent’s contribution on this journey is appreciated and with the appropriate benchmarks complete its value will be better understood.
Concluding notes
With a year of initiatives from Tencent in addressing issues of often older bug reports/feature requests shows the true value of the Tencent’s expertise and willingness to work with MariaDB on delivering a better server for everyone.
Thank you, Tencent, for this fantastic year of contributions! We look forward to many more.