Author Archives: Roman Nozdrin
Magento normally uses InnoDB for orders, order items, invoices, and other transactional data. However, analytical reports have another workload. I wanted to check whether DuckDB can improve a real Magento report without replacing the operational database.
For this experiment, I generated one million Magento orders and two million order items. Then I executed two complete Magento Orders Report queries in three storage configurations.
Short result
- 1,000,000 orders
- 2,000,000 order items
- 18.464 seconds on InnoDB
- 8.267 seconds with orders on DuckDB
- 0.292 seconds with orders and items on DuckDB
- 63.2× measured improvement for this dataset and report
Why I started this experiment
The Magento database has a mostly transactional workload.
…
Continue reading “Trying DuckDB for Magento Analytics: An Experiment with One Million Orders”
This work started from a conversation with Monty Widenius at Meet Magento Czech in August. We discussed possible ways to improve the scalability of the InnoDB B+Tree, especially its insert path and page splits.
Monty explained several ideas about reducing synchronization during structural changes and allowing independent parts of the tree to make progress concurrently. We did not have a whiteboard, so we made the first design sketch on the back of a dark chocolate wrapper.
The sketch contained a small B+Tree, several page links, and the basic idea of moving structural work outside the globally serialized path.
…
Continue reading “From a Chocolate Wrapper to Concurrent InnoDB Page Splits”
MariaDB already allows developers to add new Pluggable Data Types and scalar Plugin Functions. One missing piece has been Pluggable Aggregate Functions operating on PDTs. That matters for functionality such as HyperLogLog, where an extension needs to aggregate values into a custom statistical sketch while preserving its native SQL type.
MDEV-40672 closes that architectural gap. Aggregate functions can now participate in MariaDB’s native aggregation infrastructure, including DISTINCT, window execution, prepared statements and Pluggable Data Types.
MariaDB already supports MariaDB_FUNCTION_PLUGIN, a mechanism for registering SQL functions that create regular server Item objects and can behave almost like built-in functions.
…
An early look at the DuckDB storage engine for MariaDB — columnar, vectorized analytics that live right next to your transactional tables.
The problem
MariaDB’s InnoDB is excellent at what it was built for: transactions. Row-by-row inserts, updates, point lookups, strong consistency. But the moment you ask it to scan tens of millions of rows for a multi-way join with a few aggregations, a row store has to work hard.
The usual answer is to stand up a separate analytical system, then build ETL pipelines to copy data into it.
…
Continue reading “DuckDB Storage Engine for MariaDB. When the Sea Lion Learns to Quack.”