From a Production Problem to MariaDB: Headout’s Open-Source Contribution Journey
When we talk about open source contributions, we often focus immediately on the code.
How many lines were changed?
What feature was added?
What do the benchmarks show?
But sometimes, the most interesting part is not the feature itself. It is the journey that transforms a production problem inside one company into an improvement available to an entire community.
This is exactly what happened with Headout, Arcadiy Ivanov (Karellen, Inc), and MariaDB Server.
The technical result is support for BLOB values in in-memory internal temporary tables. This is an important improvement, of course, particularly for workloads involving JSON [1][2].
Most of the technical details were already covered in the previous article.
Here, I want to focus on how this contribution happened, why Headout decided to work upstream, and what both sides learned during the process.
It Started With a Real Production Problem
Headout operates a substantial workload on MariaDB using AWS RDS.
The platform handles around 10,000 sustained queries per second, with peaks of approximately 30,000. At this scale, database performance is not something that can be delegated to one person and forgotten. Headout does not have a traditional DBA or SRE team responsible for every database issue. Instead, its engineering pods share responsibility for the performance of their services.
As the platform grew, the team began using JSON more frequently. Aggregating and denormalizing information into JSON helped accelerate several join-heavy queries.
That optimization worked, but it also exposed another bottleneck.
AWS RDS metrics showed that the system was creating approximately 3,000 internal temporary tables every second. Many of the affected queries contained JSON columns, UNION operations, or common table expressions, and some of their temporary tables were spilling to disk.
This is a familiar story in performance engineering: improving one part of a workload often reveals the next limitation.
The team had successfully accelerated its joins, but the increased use of JSON meant that more internal temporary tables contained BLOB data. The existing in-memory path could not handle those values, so the workload moved to a more expensive path.
Because Headout runs on AWS RDS, the team could not simply reconfigure the underlying host or place the temporary directory on a RAM disk. The infrastructure was managed by the cloud provider.
More importantly, this was not a problem that could be cleanly solved by rewriting one application query.
Headout consists of multiple autonomous engineering pods. A solution requiring broad application changes would have been disruptive, difficult to coordinate, and expensive to maintain.
The preferred approach was therefore clear: solve the limitation as close as possible to its source and make the change minimally invasive for the applications that use the database.
What Moving These Temporary Tables to Memory Changes
A picture is worth a thousand words—here are a few graphs taken from the benchmarks conducted by Arcadiy, while we wait for a blog post dedicated to these benchmarks that explains how they were conducted and provides a better analysis:
Amazing results, isn’t it?
“Let Me Give It a Try”
Arcadiy had already been working with Headout’s platform team on database and application performance. His usual approach was not to propose a large rewrite whenever the system encountered a limitation, but to look for targeted improvements that could solve the problem without disrupting the work of multiple engineering teams.
After the RDS metrics exposed the temporary-table bottleneck, the team discussed what could realistically be changed. Arcadiy decided to investigate whether the limitation could be removed directly in MariaDB.
The first working draft came together quickly: roughly 16 hours on the first day, followed by another day spent mainly tracking down a specific bug. But that initial implementation was only the beginning.
It is difficult to put an exact number on the iterations that followed. Arcadiy and Monty exchanged feedback frequently, sometimes several times in the same day. Monty reviewed the implementation, suggested changes, and contributed related fixes and improvements across the affected subsystems. Elena, one of the test engineers at MariaDB, also uncovered numerous corner cases that the original tests did not cover.
This is an important part of upstream development that is easily hidden by the final commit. The process was not a simple sequence of “write patch, submit patch, merge patch.” It was a continuous cycle of implementation, review, testing, correction, and refinement.
Arcadiy also used Claude Opus 4.6 during the coding process, but not as an autonomous replacement for engineering work. The process began with research and a detailed explanation of the intended design: how MariaDB’s fixed buffers would be extended for BLOB values, how those buffers should be linked and collected, and how three different types of BLOB records should behave.
The coding assistant could then help implement individual changes, but only under close supervision. Arcadiy noted that it sometimes proposed unnecessarily complicated or unsuitable implementation strategies, making active control and technical judgment essential.
That experience illustrates a useful distinction. AI tools can accelerate code production, exploration, and repetitive implementation tasks, but they do not eliminate the need for a developer who understands the architecture, defines the constraints, rejects poor approaches, and verifies the results.
And even after the code behaved correctly for Headout’s workload, it still needed to become suitable for MariaDB itself. Tests had to be expanded, interactions with other engines and configurations had to be checked, and the implementation had to survive review by people with deep knowledge of the surrounding code.
Writing something that solves one production problem is one step. Turning it into a change that can be safely maintained as part of a widely used database server is the real contribution.
A Patch Intended for 10.11 Finds Another Path
The original work was designed against MariaDB 10.11.
Arcadiy deliberately structured the change to be self-contained and minimally disruptive. The hope was that it might eventually be included in a 10.11 maintenance release and then become available through AWS RDS.
This was an important consideration for Headout.
Running a private build of MariaDB inside a managed service is not an option. To receive the RDS improvement, Headout needed the feature to be part of an official MariaDB Server release that AWS could adopt.
Rather than targeting a stable maintenance branch immediately, the feature was accepted into the MariaDB 13.1 preview series. Once it has matured and received more testing, it may become a candidate for inclusion in a future stable series (and this is where you can help by testing it with your workload).
From the outside, this might appear to be a simple change of version number.
From a contributor’s perspective, it represents one of the essential parts of upstream development: a patch must fit not only the contributor’s requirements, but also the project’s release strategy, compatibility rules, maintenance expectations, and technical roadmap.
This is why communication with maintainers matters so much.
Working Directly With the People Who Know the Code
Arcadiy described the work on this feature with Monty as an “absolutely great experience” and a “joy.”
They worked in close contact, including on weekends, repeatedly iterating on the implementation. Monty suggested several optimizations, including improvements related to memory allocation and zero-copy handling.
For Arcadiy, the value was not only that the feature was accepted. It was also the opportunity to work directly with someone who knows the relevant MariaDB Server code intimately.
That kind of interaction is difficult to replace.
Documentation can explain what code currently does. Tests can show whether a change breaks expected behavior. But an experienced maintainer can often explain why the code has its current shape, which assumptions must be preserved, and where an apparently simple modification could create problems elsewhere.
This is one of the greatest advantages of contributing upstream.
A company does not merely hand-code a project. Its engineers gain access to the project’s accumulated technical knowledge, while maintainers gain access to real production use cases that may be difficult to reproduce internally.
The resulting implementation is usually better than either side would have produced alone.
I also asked Monty what he took away from this experience and how he felt about those long iterations and discussions with Arcadiy. This is his answer:
This was my first experience to work with someone who was heavily
using an LLM for code generation for a larger project. It did tech me
a lot of what LLM’s are good at and where they fail (in producing good
enough code). The initial PR was just a start, There was a lot of
human collaboration needed to get this into a something that was
shippable. I am very thankful for Arcadiy did implement all my
change requests and 90% of all my suggestions.
For me this was important, as the author of the Memory engine, as
I wanted to ensure that the new code would be as good as if I had
written it myself.From this experience, here is short a list where OPUS 4.6 failed and
Arcadiy and me had to find ways to fix:
- It did not reuse code properly. It created similar code paths and
functions that already existed. This happened many times during
the project.- The SQL level did know that the Memory engine could not store or index
blobs. This was also reflected in the interface for creating internal
temporary Memory tables.
Instead of adding/changing the SQL level to handle indexed blobs,
Claude instead changed the handler interface for memory table to
change the structures to fit its needs. This was ugly and error prone
and was one of the first things I fixed.
This is also something I have seen several times with Claude. It likes
to fix after the fact instead of fixing the original problem.
This is probably because fixing the original problem require structural
changes that affects more code and is thus harder to do.- Loops and other logical structures could be written to be faster and with
less code (which we did).- The LLM is not full proof. Several times it did create faulty code.
Extensive test cases, reviews buildbot and QA was needed to get the code
work properly.Arcadiy also helped me setup my Claude-MariaDB environment so that I could
better use LLM to review and check code and do some code transformations.I also have found Claude to be good in creating test case, do an
initial code review, explain code, write code commits and function
comments. As this case shows, with proper human interaction and a
reviewer that understands all aspect of the code, it can also solve
more complex solutions.Arcadiy as proven that with LLM’s also people with no knowledge of the
Monty Widenius
internals of an Open Source project can give meaningful big
contributions to the project. This is great and I hope that he will
inspire others to do the same !
Open Source Is Also a Maintenance Decision
Headout already had a culture of contributing to projects it depends on.
The company’s engineers have previously contributed to projects such as React and React Native, and Headout also publishes several open-source libraries of its own.
Rachit Watts, Headout’s CTO, explained the philosophy very simply: a company can fork a project, fix its own problem, and keep the result private—but it is usually better to give the improvement back so that everyone benefits.
There is also a very practical reason.
Maintaining a private fork creates a permanent obligation.
Every upstream release must be merged with the internal changes. Every conflict must be resolved again. Every engineer joining the team must understand why the private modification exists. When people leave, knowledge about the patch can disappear while the maintenance burden remains.
As Arcadiy pointed out during our discussion, contributing the feature upstream is therefore not only an open-source ideal. It is a better managerial decision.
Once the feature is properly integrated into MariaDB, it becomes part of the normal testing and maintenance process. Future changes to the surrounding code can take the feature into account. Headout no longer has to carry the entire burden alone.
Upstream contribution turns a private liability into shared infrastructure.
The Process Was Not Perfect
It would be easy to present this story as a completely smooth journey, but that would not provide the full picture.
Before this feature, Arcadiy had submitted fixes for other problems encountered by Headout, including issues involving the audit plugin and InnoDB locking behavior.
Some of those reports and fixes waited for several months before receiving attention.
Open-source maintainers have limited capacity. They must balance new features, regressions, security issues, release work, reviews, testing, and community requests. From the contributor’s side, however, a technically valid patch waiting in a queue can still be frustrating—especially when the problem affects production.
Arcadiy noted that responsiveness had really improved recently, but his experience illustrates an important part of the contribution that projects sometimes underestimate: submitting the patch is only the beginning.
A healthy contribution process also requires acknowledgment, communication, prioritization, and a reasonably clear path forward.
The collaboration on the in-memory BLOB work was very positive because the feedback loop was direct and continuous. Earlier bug-fix submissions had been slower.
Both experiences are useful.
The first shows what effective collaboration can achieve. The second shows where the project can continue to improve.
Why This Collaboration Matters
MariaDB receives code from many different contributors, but contributions originating from large production workloads are particularly valuable.
Headout did not develop this improvement as an academic exercise. The work started because observable production metrics showed a specific limit at scale.
The company brought the use case, workload, constraints, initial implementation, and benchmarks.
MariaDB’s maintainers brought their knowledge of the server internals, design history, release process, and long-term maintenance requirements.
Neither side approached the collaboration as a simple customer-and-vendor transaction.
Headout was not merely asking MariaDB to fix a problem. Its engineers invested time in understanding the problem and implementing a solution.
MariaDB was not merely accepting a code dump. Maintainers reviewed the work, proposed changes, and helped shape it into something suitable for the wider project.
This is the model we want to encourage.
Open source works best when users are not only consumers and maintainers are not treated only as a free support department. The strongest results appear when both sides bring something to the table.
And Now?
The feature is available in the MariaDB 13.1 preview, where it can receive broader testing and feedback.
For Headout, the remaining challenge is deployment. Because the company uses AWS RDS, it must wait until an appropriate MariaDB release containing the feature becomes available through the managed service.
This limitation also reinforces the point that upstream contribution was the only sustainable route. A private MariaDB patch would not help very much when the production database is operated by a cloud provider.
The experience has not discouraged the team from contributing again.
Quite the opposite.
Arcadiy has already submitted another proposal for a broader optimization that could significantly improve some of Headout’s query-retrieval paths.
During our conversation, he described the work on MariaDB as stimulating and said he would be happy to continue the collaboration.
This may be the most important outcome of the entire process.
A successful contribution does more than add one feature. It creates relationships, establishes trust, and makes the next contribution easier.
Conclusion
The most valuable part of this story is not only that MariaDB Server gained a useful improvement. It is that a real production limitation became the starting point for a productive collaboration between users, engineers, and maintainers.
Headout brought a demanding workload, a clearly identified problem, and the willingness to invest in a solution. Arcadiy turned those observations into working code. MariaDB’s maintainers contributed their server expertise, challenged parts of the implementation, and helped prepare the change for a much wider audience.
That is how open source should work. Companies do not have to choose between solving their own problems and helping the community. By contributing upstream, they can do both while reducing the long-term cost of private patches and local forks.
So, use MariaDB Server, test the preview releases, report the problems you encounter, and share the improvements you build. A contribution does not always need to begin as a grand idea. Sometimes it starts with one slow query, one production metric, or one limitation that somebody decides is worth fixing properly.
Your problem may be shared by many other users—and your contribution could become the next improvement that benefits the entire MariaDB community.
