I recently helped two of our sponsors simultaneously, DBS Bank and MariaDB Plc, with a recent issue they were experiencing. This actually helped us add support for an extra cloud vendor. But before I tell the story, I first need to give a bit of background.
S3 Engine History
Back in 2019, we added the S3 engine to MariaDB Server. “S3” stands for “Simple Storage Service”, it was developed by Amazon, and it is pretty much the standard for storing objects (typically files) in the cloud. Almost every cloud vendor has their own object storage with an S3-compatible REST API.
In 2019, I worked for the MariaDB Corporation, leading the MariaDB ColumnStore project. We needed an S3 storage backend for ColumnStore, at the same time, Monty was about to work on an S3 engine for MariaDB Server. We cross-team collaborated on these projects and divided-up the work. Amazon has a great SDK to use with all the parts of their cloud. But it has a licence which is incompatible with MariaDB Sever’s, so we couldn’t easily use this SDK. In addition, we only needed a subset of the vast functionality that S3 provides, which itself was a subset of the SDK. We therefore decided to create our own library that would provide the parts of the API we needed.
I very quickly created the library to give us the required functionality, called libmarias3 (I didn’t have a lot of time to think of a cool name). My team developed the ColumnStore storage manager using this library, whilst Monty developed the S3 engine using the library.
The S3 engine itself is based on the Aria engine, it is a read-only archival storage for your data. It has the ability via ALTER TABLE to pull and push the tables in and out of local / remote storage. If you are in the cloud and need to store large amounts of reporting data, it can be a lot cheaper to do it this way than having everything on local storage.
Over time, we increased compatibility with other cloud vendors, who have slightly tweaked the way their S3 implementation works when compared to Amazon’s. Of course, I moved on and now work for the MariaDB Foundation.
The issue at hand
We were asked by two of our sponsors, MariaDB Plc and DBS Bank, to help them with a compatibility issue with the S3 engine and Huawei OceanStor. I was the right person for the job because I knew libmarias3 very well, and adding support for Huawei OceanStor could help other MariaDB Server users in the future.
The core problem (ticket MDEV-34867) was that libcurl underneath libmarias3 was sending a header default Content-Type: application/x-www-form-urlencoded header in the REST requests. The mime types of the files aren’t that important, so we didn’t implement correct mime type handling, and this worked out for every vendor used so far. But it turns out that Huawei OceanStor will reject requests that use this header.
The fix for this was two-fold. The first was libmarias3 now has an option to disable sending that specific header when a mime type has not been set. Along with this, I added support for setting and retrieving mime types for files. The second part of the fix was a new option in the S3 engine called S3-provider. This option allows a kind of “quirks mode” for specific cloud vendors. Therefore, setting S3-provider=Huawei will disable sending the Content-Type: header when one has not been set.
Conclusion
The unique position the Foundation has means that we can easily help with issues affecting multiple sponsors such as this. Enabling us to support even more external storage vendors and allowing our users more choice.
The fix itself will be coming soon in an update to 10.6 onwards.