Benchmarking MariaDB-5.3.4
Last weekend Vadim from Percona published his MariaDB 5.3.4 benchmark results. As the new benchmark guy at Monty Program I take this oportunity to add some more results of my own.
One question in the comments to Vadim was if it is fair to compare MariaDB-5.3 with MySQL-5.5. Or if this comparison should be done with MySQL-5.1. The answer is: it does not matter much. MySQL-5.5 and MySQL-5.1 show very similar results in the Sysbench OLTP benchmark.
So I created a Sysbench environment pretty much like Vadims and tested the following versions of the MySQL Server:
- MariaDB-5.3.4 – the Monty Program release candidate, both with XtraDB and the InnoDB plugin
- Percona-Server 5.1.61 because it is based on the same XtraDB version as MariaDB-5.3
- Percona-Server 5.5.20 – the current Percona flagship
- MySQL-5.5.20 – the current Oracle flagship
The result:
Indeed MariaDB-5.3.4 scales significantly worse than MySQL-5.5.20. However Percona-Server 5.1.61 behaves nearly identical and Percona-Server 5.5.20 is only slightly better. If we run MariaDB-5.3.4 with the InnoDB plugin instead of the default XtraDB, then MariaDB outperforms Percona-Server 5.5.20.
As a consequence, we will deliver the InnoDB plugin with future binary releases of MariaDB. MySQL-5.1.61 with the InnoDB plugin (not shown in the graph) is between MariaDB-5.3.4(InnoDB-plugin) and MySQL-5.5.20. We are currently investigating why MySQL-5.1 is faster than MariaDB-5.3, even with the same storage engine.
Please keep in mind that Sysbench is a highly synthetical benchmark. Real world workload will probably yield different results. Also this benchmark used solid state disks. A storage system with spinning disks might saturate much earlier and would then flatten out the peaks.
If I should draw a conclusion, then I would say that for highest concurrency levels you should go for MySQL-5.5. Or wait for MariaDB-5.5 🙂
If your workload includes complex (sub)queries, then you will probably benefit more from MariaDBs new optimizer features. And if you are running replication, then MariaDB 5.3 with group-commit will greatly outperform any other MySQL version.
Benchmark details: sysbench OLTP multi-table. 16 tables with 250K rows each. 20G InnoDB buffer pool. Data and logs on SSD. Dual socket hexacore Xeon with HT enabled (24 logical cores). You can download the tarball with benchmark scripts, config and raw results.
When 5.5 came out, Oracle claimed 369% better performance on sysbench RW than 5.1, with or without plugin.
http://downloads.mysql.com/presentations/mysql-sunday/MySQL_OOW2010_Keynote_Final.pdf
(page 16 here)
Percona Server claims to be 40% faster than MySQL in overall throughput, though on a different benchmark.
http://www.percona.com/software/percona-server/benchmarks/
This result here shows that none of this would be true, 5.1 and 5.5 performance is similar, while Percona Server is 40% *slower* in overall throughput, and XtraDB by Percona being factor #1 in slowing down MariaDB.
That’s funny.
I only believe benchmarks I run myself from now on..
In this benchmark are activated the new 5.5 features like multiple buffer pools and new I/O threads?
I do know, I have not seen my.cnf or sysbench parameters from Oracle, only the graph.
I/O threads yes (8 readers and 8 writers). Multiple buffer pools no.
> This result here shows that none of this would be true, 5.1 and 5.5 performance is similar…
369% I think is for 1024 threads, which wasn’t tested here. At 256, results appear to be similar.
No, actually the page 16 here
http://downloads.mysql.com/presentations/mysql-sunday/MySQL_OOW2010_Keynote_Final.pdf
has ~4000 TPS for 5.5, ~2500 TPS for 5.1/plugin, and ~1000 TPS for 5.1/builtin
for 256 concurrent clients
The Percona white paper that claims to be 40% faster over same MySQL version is focused on SSD performance. The white paper makes it clear that on spinning disks there will be much less of a benefit.
Since this test was in fact using SSD, it’s a good question why this 40% gain can’t be seen here? The answer is, Percona adds specific SSD opitmized InnoDB flushing algorithm (and maybe something else) that needs to be specifically turned on. Since this test probably used same configuration file for all tests, it didn’t benefit from this optimization.
Axel here doesn’t say if SSD was SAS connected or PCIe connected – makes a big difference. Also not explicit, it seems this data set actually fits into memory?
That Percona Server would be significantly slower than the matching MySQL version seems very unlikely and suggests there is a flaw in this test. (I didn’t comment on Vadim’s test, but likewise there could obviously be some flaw in that test too.)
It is true that no extra configuration was used to help XtraDB. But then OTOH Vadim didn’t do that too. Indeed it would be interesting to rerun the benchmark with optimized XtraDB settings.
The disk used in this test is a RAID0 over 3 SAS SSDs. The data set indeed fits into memory (again: just like in Vadims test). The OLTP rw benchmark writes significant amounts of data. And while the log is consecutive I/O, the checkpointing is not. I don’t remember the number of I/Os per sec but I’ve seen up to 100M/s writes to the disk. You need a rather big disk cache and/or many disks if you want to stem this amount of random writes with spinning disks.
Did you test Oracle MySQL Enterprise or Community version?
This test used the Community Server from Oracle. Also everything was compiled from source with the same (release build) options to eliminate any influence of compiler and libraries used.
In those tests it did not make a difference. Up to 5.5 there are no functional differences between Community and Enterprise. This will change when the thread pool becomes GA because Oracle (unlike MariaDB) will not put this into the Community Server.
Ilari, the two are identical, save for the commercial license on the Enterprise edition.
Marc: Since last year they are not. Performance-wise the closed source, Enterprise-only thread pool plugin would make a difference: http://www.mysql.com/products/enterprise/scalability.html
It’s worthwile to note that MariaDB 5.5 comes with an open source version of a thread pool 🙂
Thanks Axel for your benchmarks, really interesting indeed. How does MariaDB 5.2.10 fit into the above comparisons ? I see a new addition on the KB for the how to for compiling MariaDB 5.3.3 with vanilla InnoDB plugin http://kb.askmonty.org/en/how-to-compile-mariadb-with-vanilla-xtradb. Here’s hoping to you folks figuring what is happening to improve the performance and hope MariaDB 5.5 comes soon 🙂
MariaDB-5.2, MariaDB-5.3 and Percona-5.1 all behaved nearly identical (except some statistical fluctuations). In fact this was expected because all 3 use the same XtraDB version. It seems XtraDB is the limiting factor here because MariaDB-5.3.4 was so much better with the InnoDB plugin. The latter is quite strange because XtraDB is based on the InnoDB plugin code.
This is about vanilla XtraDB, not vanilla InnoDB-plugin. MariaDB source code contains the InnoDB plugin, so anybody who compiles his own binaries can have it. It’s just that our binary packages don’t include it yet.
Great analysis and presentation of results. Providing the best performance at various sizes and loads is a real challenge, it is nice to see that MariaDB will include the InnoDB plugin going foward. It is also refreshing to see your recommendation of someone elses product (MySQL-5.5), as you say “for now”.
BTW, the MariaDB group commit patch is also in Percona Server 5.5.