My fellow testers and others who run RQG tests on MySQL flavors might be interested in some additions that are being used for MariaDB testing. While none of them is a major breakthrough, maybe they will make somebody’s life a little easier.
RQG Introduction
A quick introduction for those who have never heard of RQG, but are still curious what this blog post is about.
RQG stands for Random Query Generator, also known as randgen — an open-source product, available under the GPL v2 license. Quoting its home page on Launchpad, it is a “pseudo-random data and query generator that can be used to test any Perl DBI, JDBC or ODBC-compatible SQL server, in particular MySQL, but also JavaDB and PostgreSQL”.
The framework was created by my former colleague Philip Stoev, who not only developed a great tool, but also managed to make it a very important part of the MySQL testing routine. I think by now all MySQL-based products use it to some extent; and to those that don’t, I recommend to try it out.
Along with data and query generation, the framework features “pluggable” modules of different types, which perform various checks during or after test execution. This article is mainly about adding more of them to the existing set. When you see ‘validator’ or ‘reporter’ below, it means the type of the corresponding module.
RQG documentation can be found on GitHub.
About these MariaDB additions
All listed additions have been pushed into the main randgen tree on Launchpad, along with some other stuff that is more specific for MariaDB and thus less interesting for the general public. However by the time you are reading it, those versions might already be a little outdated. Since the randgen tree is now self-regulated, I feel like it can easily fall into the “too many cooks” trap, so I have chosen the most conservative approach for merging into the main tree. Whenever a change affects anything but a MariaDB-specific component, I will send an email to the randgen mailing list with the description, and if there is at least one voice against the change, it will not be merged. The downside is that it makes merges a bit fussy, so will not be happening often. Moreover, rejected changes will never make it to the main tree.
Most recent versions, as well as all new and rejected patches can be found in our work tree.
Galera mode
RQG is very useful for running tests on a Galera cluster, but doing it is a pain: start all the nodes, run an instance of RQG for each node, check results, don’t forget to shut down the nodes at the end… Probably most of us who have had to do it by now have scripted some parts of it, but still, it felt all wrong, so I finally decided to make it “native”. Now Galera cluster can be started by RQG itself, with a topology configurable via a command-line option.
See https://kb.askmonty.org/en/rqg-extensions-for-mariadb-features/#galera-mode
This is just the first drop, which is likely to be fixed and improved later.
CheckFieldValue validator
If you need a simple one-time check that does not deserve implementing a separate validator, you might try this one.
Adding a specifically formatted comment to a query triggers a validation. The validation is defined in the comment itself: it says which field in which row in the result set should be checked, and provides the condition (currently simple numeric comparison: ‘=‘, ‘<‘, ‘>‘, ‘<=‘, ‘>=‘).
See https://kb.askmonty.org/en/rqg-extensions-for-mariadb-features/#checkfieldvalue-validator
Here is a real example of how it can be used: MDEV-4578. A user reports sporadic bizarre values in I_S.processlist.time field. You can of course write a validator for it, and I initially did, but I soon realized that it was inconvenient since I wanted to keep modifying the checking query. So instead I added the SELECT from the I_S to the grammar, with the condition that the time field must be less than some ridiculously big value.
The set of possible validations will be expanded on need basis.
SlaveCrashRecovery reporter
The reporter is for testing crash-safety of master=>slave replication. It crashes the slave periodically, restarts it and makes sure it starts all right.
See https://kb.askmonty.org/en/rqg-extensions-for-mariadb-features/#slavecrashrecovery-reporter
I am using it, along with its sibling MariadbGtidCrashSafety reporter, for testing MariaDB implementation of GTID.
BinlogConsistency reporter
The reporter checks that the contents of the binary log(s) correctly reflects the data at the end of the test flow. After the main test flow is finished, the reporter creates a data dump of the server, shuts down the server, starts a new one with the same options, but on a clean data directory, replays binary logs of the first server into the new one, creates a data dump of the new server, and compares it with the previous dump.
See https://kb.askmonty.org/en/rqg-extensions-for-mariadb-features/#binlogconsistency-reporter
I was using it for testing of binlog changes in MariaDB 10.0 (see for example MDEV-181, MDEV-232)
CrashRecovery reporter
A new implementation of the old Recovery reporter, more suitable for current server versions. Instead of hard-coded set of options for the restarted server, it uses the same configuration as the initial server had.
See https://kb.askmonty.org/en/rqg-extensions-for-mariadb-features/#crashrecovery-reporter
Comparison of VIEW Algorithms
RQG allowed to configure a view type to be used during the test, but you could only create views of the same type on servers that you were comparing. I found it to be a shortage while testing the extension of MERGE views implemented in MariaDB 10.0 (see MDEV-3862): I needed to compare MERGE views on one server with TEMPTABLE views on another server. So, --views1 and --views2 options have now been added. It might be not the most elegant solution, but it is in line with other similar options in RQG, and it does the trick.
See https://kb.askmonty.org/en/rqg-extensions-for-mariadb-features/#comparison-of-view-algorithms
Multiple redefining grammars
I found that my set of redefining grammars started growing beyond reason because I only could provide one at a time, and often I would need to combine several, so I would have to create a new grammar for each combination.
So, now you can add as many --redefine= options as you want, all grammars will be applied on top of the main grammar.
See https://kb.askmonty.org/en/rqg-extensions-for-mariadb-features/#multiple-redefining-grammars
New grammar keywords _basetable and _view
The RQG keyword _table means any table or view from the test dataset. Often it’s good enough, but sometimes it badly increases the number of erroneous statements, e.g. with the “not a base table” error. Now tables and views can be differentiated in a grammar depending on what the goal is: _table can still be used for both tables and views, while _basetable will only pick tables, and _view will only pick views.
Feedback
If you have a feedback on the changes, you can leave comments at RQG extensions for MariaDB Features.