This article is by the Tencent Game DBA Team, and has been translated from the original.
Tencent Game DBA Team (the DBA Team for short) has been serving the game business for a number of years. The mission of the DBA Team is to provide stable and efficient online storage services for Tencent Games. As the storage requirements from the game business evolve, we keep extending features of our database and improving its performance. We spared no effort to provide reliable, scalable and performant database solutions.
We currently maintain several products:
- TenDB: the MySQL branch maintained by the DBA team. It features single machine performance improvements and core functionalities vital to the game business, such as Instant Add Column for InnoDB.
- TenDB Cluster: a MySQL Cluster solution based on TenDB and the Spider Storage Engine. It can provide linear scalability in performance (CPU/MEM/IO). Its high reliability has been proven by numerous popular Tencent games.
- Tendis: a Redis Cluster solution based on RocksDB and Twemproxy. It provides a high performance, linear scalable and persistent cache service.
The open source dream of the DBA Team
We have been using MySQL for more than 10 years, dating back to version 4.0.12. Now we maintain tens of thousands of MySQL instances. As the instance number grows, we accumulate tons of experience in MySQL database management and maintenance. Moreover we are able to harness and hack the MySQL source code.
All DBAs in our team make their living from maintaining MySQL databases and other open source software. The official MySQL server, and our TenDB and TenDB Cluster contribute a non-negligible factor to the great success of Tencent’s game business.
Therefore, we sincerely want to contribute to the MySQL community. We dream that we can evolve together with the MySQL community. “We have a dream”! This faith has been around for ten years.
While making this dream into reality, confusion in how to feed back to the community haunted us from time to time, but we are more and more determined. “Unique and massively used inventions are the best contribution to the open source community!” This quote from our high manager Xiaochun Cui, clarified the vision and encouraged us to pursue our dream. Inspired by this quote, after ten years’ exploration, we finally started a brand new chapter of our open source dream.
Overcoming difficulties in pursuing our dream
There are three methods to contribute to the open source community:
- Be MySQL users: report bugs we find in our testing and production environments, and feedback new features required by business.
- Be MySQL evangelists: share technologies of using MySQL at massive scale, which promotes the popularity of MySQL.
- Be MySQL contributors: submit new features already proven in large scale product environments to the official MySQL team, and push them into incoming official releases.
After 10 years’ involvement, we maintain tens of thousands of MySQL instances in production, and have accumulated plenty of experience in using MySQL well and efficiently. Even more, we have hacked the MySQL source code and developed new features badly needed and massively used by our businesses.
In recent years, we have actively participated in technology conferences, including the China MySQL User Group (ACMUG) and DTCC, to share our experiences and technologies. We are making our way as a MySQL evangelist.
Now we are ready to practice method 3. We know it may be quite challenging, but most valuable. We contribute our experience and features back to the community, enabling all MySQL users to benefit, as we have benefited from the community in the past 10 years.
In fact, back in December 2012, we open sourced the Instant Add Column for InnoDB feature in TenDB1.1 on Github. But we gradually stopped maintaining the code and the community on Github since we were too busy with our work to spare enough time. We decided to submit our features to the official MySQL team.
Dream come true with MariaDB
DBAs in the MySQL ecosystem are very fortunate. We have plenty of choices, either as MySQL users or as committers. At present, we have the official MySQL from Oracle, MariaDB from Monty (Michael Widenius), and third-party branches such as Percona Server. As long as the whole community can benefit from our work, we are glad to have a try. In fact, we have made a number of efforts.
In August 2015, we submitted two features on Replication to the MySQL Replication Team and the MySQL mailing list. The official MySQL team gave us positive responses, and regarded our features as the missing default ones. However, due to many factors, these features have not been merged into the main branch. We will try our best to cooperate with the official team to make our dream come true.
In July 2016, with the help of Yanwei Zhou, the president of ACMUG, and Lixun Peng, member of MariaDB Foundation, we were lucky to have the chance to meet Monty, the creator of MySQL and MariaDB. Monty was working on his new project, MariaDB. We spent the whole afternoon explaining to him the business scenarios of Tencent Games, and the design and core features of TenDB and TenDB Cluster, our database products customized for game businesses. Monty illustrated new features to us in the upcoming
MariaDB version, the development milestones, and how the MariaDB Foundation works.
The scenario on that afternoon was basically like this: a gray-haired coder seriously listening to a few young engineers explaining design and features. When there was ambiguity due to the language barrier, they turned to the whiteboard and drew some charts for illustration. Even more, they opened a terminal and discussed the code directly. Monty would point out the problem with a specific line. Wouldn’t you be touched if you were there?
In the end, we expressed our wish: submit a few features from TenDB and TenDB Cluster to MariaDB’s main branch. Monty expressed his support and encouraged us to contribute to the community. After that, we followed the MariaDB code guidelines, and submitted our pull requests to MariaDB on Github. Our dream began to come true.
MariaDB 10.2.3 released !
So far, we have submitted three pull requests to MariaDB. MariaDB 10.2.3 has already merged two of them: (Highlighted in the Figure above)
- Restrict the speed of reading binlog from Master
https://github.com/MariaDB/server/pull/246 - Compressed binary log:
https://github.com/MariaDB/server/pull/247
In addition, as for transparent compression of InnoDB blob data, MariaDB is considering a storage independent implementation, while keeping the InnoDB row format compatible with TenDB. - Big column compression(innodb):
https://github.com/MariaDB/server/pull/261
Many thanks to Sergey Vojtovich and Kristian Nielsen for reviewing our code and giving suggestions for improvement.
Further cooperation with MariaDB in the future
1.Merge TenDB’s unique feature Instant Add Column for InnoDB into the main MariaDB branch
As for the Instant Add Column for InnoDB feature, we had a comprehensive discussion via email and on JIRA issue MDEV-11369 with Marko Mäkelä from MariaDB. Moreover, we had a face to face discussion at the MariaDB Developers (Un)Conference in October 2017 Eventually, it was agreed that MariaDB 10.3 integrate TenDB’s Instant Add Column for InnoDB feature. We will submit a pull request to MariaDB soon.
2. Motivate MariaDB to Release a GA version of the Spider Storage Engine
The Spider storage engine is a distributed storage engine developed by Japanese database expert Kentoku Shiba, aiming at solving scalability problems in MySQL. It has been integrated into MariaDB since 10.0, but remains gamma maturity.
In early 2014, the Tencent Game DBA Team started from Spider 3.1 to develop a highly customized version, which, after three years of iteration, turned in to TenDB Cluster. TenDB Cluster solved a number of performance and stability issues in the original Spider, and added features such as connection pool, global increment ID, parallel query, query condition pushdown and so forth. TenDB Cluster has been widely used in the vital online storage service of Tencent Games.
Currently we are working closely with Kentoku to integrate features including connection pool and force index pushdown, as well as a few bug fixes, into Spider 3.3. In the future, the Spider storage engine will be included in MariaDB 10.3 as stable. Kentoku is sharing at MariaDB M|17 in the following figure:
Refer to The status of Tencent patches into Spider to see details of features already merged.
3. Promote MariaDB in the Chinese mainland to help more MySQL DBAs to know and use MariaDB, and to bring more contributors to MariaDB.
The Tencent Game DBA Team will continue to make an effort to contribute the source code of features that are widely proven in the production environment of our game business to MariaDB. We expect more MariaDB users to benefit from this code and these features. Meanwhile, we will actively use good features from the MariaDB community in our production environment. With close cooperation with the MariaDB community, we would like to explore and solve challenging problems encountered in core database
operations of our game business.
Donating to the MariaDB Foundation
The mission of the MariaDB Foundation is to provide infrastructure support for the MariaDB project, users and the developer community. At the same time, it enhances database technologies, including implementations of standards and interactions with other databases.
Since August 2016, the Tencent Game DBA team has been a bronze sponsor of the MariaDB Foundation. We were the first Chinese organization to sponsor the Foundation.
Why donate to the MariaDB Foundation?
In July 2016, when Monty, the “father of MySQL & MariaDB”, visited Tencent, he illustrated how the MariaDB Foundation operates; the MariaDB Foundation is a non-profit organization. In order to attract excellent code and features from worldwide, the MariaDB Foundation needs to hire engineers to review code from the community to make sure they follow the code guidelines, and to write strict tests for this code to make sure that new features are stable enough. Monty expects companies that benefit from MariaDB’s open source community to donate to the MariaDB Foundation to keep it running. We strongly agreed with Monty’s idea and the mission of the MariaDB Foundation, so we made the first donation as a team shortly after. In the near future, we will increase the donation fund to become a gold member of the MariaDB Foundation, to cooperate more closely with the MariaDB Community.
FAQ
You may still feel a little confused why we did this after reading through the whole article.
Q: Why develop a customized version based on MySQL?
A: Games developed or distributed by Tencent rely heavily on MySQL Community Server as the storage layer. Our MySQL instances in production exceeded 10,000 several years ago. While enjoying the simplicity and flexibility of MySQL, we clearly know that MySQL’s feature set is not as rich as some proprietary database products. This disadvantage could usually be overcome by purchasing commercial services, which would nevertheless incur a cost of many millions of dollars per year. Not
surprisingly, we did not make the purchase. Instead we use the strategy already practiced by many other large internet companies; maintain a customized version and develop features that could solve problems that widely emerge in the game business. We can benefit from this strategy in two ways. On the one hand, through these in-house developed features, we can fundamentally solve some problems that have existed a long time and that greatly hurt us; on the other hand, we can cultivate engineers who are able to harness the internals of open source databases, and they can solve problems and challenges that constantly emerge as Tencent’s game business evolves.
Q: Why open source code and cooperate with MariaDB?
A: Several considerations motivate us to open source our code. Firstly, it’s a mandatory requirement of the GPL license that code developed based on a GPL licensed project should be open sourced. Secondly, by participating in an influential community, we can help more people or companies benefit from our work. Besides that, we expect to solve difficulties with maintaining a customized version by open sourcing. Due to resource constraints, we want to save the effort of merging important features from the community into our branch, or developing similar features already existing in the community. If we get our code merged by the community, then every time the community releases a major GA version, we can quickly benefit from contributions from the whole community, and don’t have to lose our own unique features that are important to our business. In the long run, we don’t have to explain what TenDB is and we don’t need to persuade the game developers to use TenDB, while they insist on a MariaDB or official MySQL, since they are pretty much the same thing.
Q: Why choose to solve the scalability problem of MySQL at the storage engine level?
A: The high development speed of the internet business requires the database layer to scale
fast, and it would be ideal if the process of scaling is transparent to applications. Many NoSQL or NewSql products aiming at satisfying such requirements have arisen in recent years. However, for the game business, most developers still choose MySQL protocol compatible databases. The rigorousness and stability of RDBMS (like MySQL and MariaDB), along with the flexibility of the SQL protocol, are two major factors which game developers pay special attention to when choosing a storage product. Implementation of data sharding at the storage layer enables game developers to use TenDB Cluster without modifying a single line of code most of the time, which makes it possible that games can be deployed into production very fast, and capacity or performance can be scaled out quickly as the business develops at a high speed.
Q: How widely are Ten Series Products used in other Tencent businesses besides games?
A:After TenDB Cluster covered a large portion of popular mobile games, we shared our experiences of using TenDB Cluster in a company-wide tech event. DBAs of TenPay (Tencent Online Payment Business) showed their interest as they were investigating MySQL scalability solutions at that time. Soon, they deployed TenDB Cluster in numerous production environments where there were many more reads than writes. Besides Tenpay, the Tencent Online media business is also a heavy user of TenDB Cluster. In late 2016’s ACMUG event, they learnt of TenDB Cluster. After a comprehensive review, they found TenDB Cluster was a great fit for Tencent Video’s (Tencent’s online streaming video business) log storage, especially as TenDB Cluster enables DBAs to employ a sharding solution transparent to the application, which completely separates database scaling from the application code. This transparency makes it possible for DBAs to independently scale up databases and provide a more responsive service when businesses experience explosive development.
Q: Will Ten series products be actively maintained in the long term?
A: This question is an important one when considering Ten series products are reviewed as
candidate solutions by other business lines inside Tencent. In December 2011, the Tencent Game DBA Team started to maintain an independent internal MySQL branch, aka TenDB 1.0. We iterated TenDB 1.0 to today’s 2.x. Till now, TenDB is completely compatible with MySQL 5.5 and MySQL 5.6. As for TenDB Cluster, its development and operation dates back to 2014. We iterated TenDB Cluster from 1.0 to today’s 1.9, and keep developing new features needed by business, and fixing bugs. Now we are working hard to push core features of these two products into the main MariaDB tree. We name our products “Ten” from Tenable, which means sustainable storage products.