Protect your data with MariaDB Server and Intel® TDX

Written by Written by Răzvan Vărzaru Leave a comment on Protect your data with MariaDB Server and Intel® TDX

In 2023, Intel launched Intel® Trust Domain Extensions (Intel® TDX), their latest Confidential Computing technology that ensures data in use protection for hardware-isolated virtual machines and verified hardware and software integrity through attestation.

In this blog post, we will show that MariaDB Server can run correctly on an Intel® TDX enabled VM and that the in-memory data is protected from outside actors, even from the host OS running the VM. By coupling Intel’s Confidential Computing technology with MariaDB Server build in data-at-rest and data-in-transit protection, customers can gain an end-to-end protected solution.

This advancement is crucial as the growing use of cloud services is inscreasing the need to protect sensitive data. Cloud providers now offer Confidential Computing services based on Intel® TDX. For more technology specific details, see Intel’s website.

Prologue

This year, we engaged in discussions with Intel representatives at SUSECON, leading us to collaborate on testing MariaDB Server with Intel® TDX. As part of this partnership we will publish several blog posts highlighting the combination of MariaDB Server and Intel® TDX.

This specific post will demonstrate how data stored in the server’s memory is protected from unauthorized access, even from attackers exploiting kernel vulnerabilities or rogue administrative users.

We will make use of Intel® TDX from two perspectives:

  • A DBA configuring MariaDB on their own VM to store sensitive banking information.
  • A third-party hosting provider.

Now, let’s keep the introduction brief and delve into the technical details!

Host Configuration

We’re working on a Ubuntu 24.04 OS installed on top of a system equipped with a CPU from the Xeon Scalable 5th Gen Emerald Rapids family.

~/tdx$ cat /etc/os-release 
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"

~/tdx$ lscpu
Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          45 bits physical, 57 bits virtual
  Byte Order:             Little Endian
CPU(s):                   224
  On-line CPU(s) list:    0-223
Vendor ID:                GenuineIntel
  Model name:             INTEL(R) XEON(R) PLATINUM 8570
    CPU family:           6
    Model:                207
    Thread(s) per core:   2
    Core(s) per socket:   56
    Socket(s):            2
    Stepping:             2

We’re following Canonical’s instructions for setting up the host OS and guest OS. If you’re looking for a more detailed alternative, Intel offers excellent documentation on setting up Intel® TDX.

On the host side, the BIOS settings were already in place for us as described in Canonical’s instructions. If you’re setting up your own bare-metal instance, remember to enable TDX in the BIOS settings.

We just need to run the configuration script.

# Clone the Canonical's Intel® TDX repository
git clone -b noble-24.04 https://github.com/canonical/tdx.git

# Setup Intel® TDX in host OS
cd tdx
sudo ./setup-tdx-host.sh

# Reboot
sudo reboot

# Verify that Intel® TDX is enabled
$ sudo dmesg | grep -i tdx
[    5.340815] virt/tdx: BIOS enabled: private KeyID range [64, 128)
...
...
[   21.984224] virt/tdx: module initialized

Configure VMs

First, we need to generate the guest images that are later used by QEMU. Canonical provides a script to generate guest images with Intel® TDX tools inside, with the baseline image being an ubuntu-24.04 cloud image, which does not require any kernel changes.

# Creating an Intel® TDX qcow2 disk image
cd tdx/guest-tools/image/
sudo ./create-td-image.sh

To later compare the VM protection provided by Intel® TDX with a regular VM, we also create a regular disk image, by simply removing Intel® TDX tooling from the image creation script.

We will end up with two disk images, under the repository tree.

:~/tdx/guest-tools/image$ ls -l *qcow*
-rwxrwxrwx 1 root root 1903362048 Oct 17 23:30 regular-guest-ubuntu-24.04-generic.qcow2
-rwxrwxrwx 1 root root 4116250624 Oct 17 06:23 tdx-guest-ubuntu-24.04-generic.qcow2

In the Canonical TDX repository we can also find two sample domain XML’s for booting both an Intel® TDX protected VM (also called Trust Domain) and a regular VM. I’ve added hostfwd=tcp::0-:3306 to the XML’s to let QEMU assign a random port for the client-server connection.

The recommended way for booting both VM’s is using the libvirt wrapper tdvirsh provided in the repository.

./tdvirsh new \
    --td-image ./image/regular-guest-ubuntu-24.04-generic.qcow2 \
    --xml-template ./regular_vm.xml.template

./tdvirsh new \
    --td-image ./image/tdx-guest-ubuntu-24.04-generic.qcow2 \
    --xml-template ./trust_domain.xml.template

As a result both VM’s will start and both will be assigned random ports for SSH and MariaDB Server in the example below.

./tdvirsh list

Id   Name                                                        State 
--------------------------------------------------------------------------- 
27   tdvirsh-trust_domain-ae41a94c-43f3-4bf0-9c30-b66a13ca6b42   running (ssh:38231 45017, cid:3)
30   tdvirsh-regular_vm-00c643ab-205d-41b5-873d-a60800b284f3     running (ssh:45545 38053, cid:4)

Using SSH to connect into the Intel® TDX protected VM, we can easily verify that Intel® TDX is enabled inside this VM.

root@tdx-guest:~# sudo dmesg | grep -i tdx
[    0.000000] tdx: Guest detected
[    0.000000] DMI: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 2024.02-3+tdx1.0 07/03/2024
[    0.278473] process: using TDX aware idle routine
[    0.330149] Memory Encryption Features active: Intel TDX
[    6.454503] systemd[1]: Detected confidential virtualization tdx.
[    6.470694] systemd[1]: Hostname set to <tdx-guest>.

Configure MariaDB server

We will configure MariaDB Server 11.4.3 on both VM’s. Since 11.4.1 the connection between the client and the server has SSL enabled by default, which automatically brings us data-in transit protection. The server will generate certificates in memory and use them. For further reading we’ve blog posted about it.

Note that Intel® TDX does not inherently protect data that is written to disk, i.e. data-at-rest. There are two options to protect data in this state: (1) application-level file encryption and (2) guest OS-based file encryption. We opt for the first option as this is already provided by MariaDB Server.

In more details, we will set up the server with data-at-rest protection for the InnoDB storage engine using the file-key-management plugin.

Overall, we perform the following steps:

  • Install the server. You can download the latest MariaDB Server 11.4 from the downloads page.
  • Generate a keyfile for data-at-rest protection
  • Enable data-at-rest protection in the server configuration file
  • Bind to a public interface to allow remote connections
  • Restart the server to apply changes

Steps executed inside both VM’s after installing MariaDB Server:

# Keyfile generation
root@tdx-guest:~# mkdir /etc/mysql/encryption
root@tdx-guest:~# (echo -n "1;" ; openssl rand -hex 32 ) | sudo tee -a  /etc/mysql/encryption/keyfile
1;56f9e484de0016b19c4de543430e729367838033db1d3ea724dc9624f1f3ae4b

# Server configuration
root@tdx-guest:~# cat <<EOL >> /etc/mysql/my.cnf

[mariadb]
plugin_load_add = file_key_management
loose_file_key_management_filename = /etc/mysql/encryption/keyfile
innodb_encrypt_log = ON
innodb_encrypt_temporary_tables = ON
innodb_encryption_threads = 4
bind-address = 0.0.0.0
EOL

root@tdx-guest:~# systemctl restart mariadb

The last step is to create a user for the remote connection.

CREATE USER 'user12'@'%' IDENTIFIED BY 'password123';
GRANT ALL PRIVILEGES ON *.* TO 'user12'@'%' WITH GRANT OPTION;

Dataset preparation.

For our deployment, we will use the mariadb command-line client installed in the host OS to interact with the database server. In an actual production environment, the client would be installed on another machine.

The following steps will be repeated for both VM’s. First, let’s verify that MariaDB Server is configured to protect data in transit and at rest.

# Connect to the server
mariadb -uuser12 -ppassword123 -h localhost -P #FW_PORT#

# Verify that the connection is secured (data-in-transit protection)
MariaDB [(none)]> SHOW STATUS LIKE 'Ssl_cipher';
+---------------+------------------------+
| Variable_name | Value                  |
+---------------+------------------------+
| Ssl_cipher    | TLS_AES_256_GCM_SHA384 |
+---------------+------------------------+

# Check that the encryption plugin is ON (data-at-rest protection)
MariaDB [(none)]> SHOW VARIABLES LIKE '%encryption%';
+------------------------------------------+---------+
| Variable_name                            | Value   |
+------------------------------------------+---------+
| file_key_management_encryption_algorithm | aes_cbc |
| innodb_default_encryption_key_id         | 1       |
| innodb_encryption_rotate_key_age         | 1       |
| innodb_encryption_rotation_iops          | 100     |
| innodb_encryption_threads                | 4       |
+------------------------------------------+---------+

MariaDB [(none)]> SELECT PLUGIN_NAME, PLUGIN_STATUS FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME LIKE "%key%";
+---------------------+---------------+
| PLUGIN_NAME         | PLUGIN_STATUS |
+---------------------+---------------+
| file_key_management | ACTIVE        |
+---------------------+---------------+

While setting up our database, we create encrypted tables to ensure data-at-rest protection and add some dummy data to the tables. We execute the following from the client:

# Prepare dataset
CREATE DATABASE Customers;
USE Customers;

CREATE TABLE Individuals (
    ID INT AUTO_INCREMENT PRIMARY KEY,
    Name VARCHAR(100) NOT NULL,
    Email VARCHAR(100),
    PhoneNumber VARCHAR(15) NOT NULL,
    Address VARCHAR(255)
) ENGINE=InnoDB, ENCRYPTED='YES';

INSERT INTO Individuals (Name, Email, PhoneNumber, Address)
VALUES ('Razvan Varzaru', 'razvan@provider.com', '+40700060000', '456 Real Rd, Bucharest, Romania');

INSERT INTO Individuals (Name, Email, PhoneNumber, Address)
VALUES ('John Doe', 'john.doe@provider.com', '+40700060001', '123 Fake St, Springfield, USA');

CREATE TABLE Accounts (
    AccountID INT AUTO_INCREMENT PRIMARY KEY,
    CustomerID INT,
    AccountNumber VARCHAR(50) NOT NULL,
    Currency VARCHAR(3) NOT NULL,
    Balance DECIMAL(15, 2) NOT NULL,
    FOREIGN KEY (CustomerID) REFERENCES Individuals(ID)
) ENGINE=InnoDB, ENCRYPTED='YES';

INSERT INTO Accounts (CustomerID, AccountNumber, Currency, Balance)
VALUES (1, 'ACC123456789', 'EUR', 1000.00);

INSERT INTO Accounts (CustomerID, AccountNumber, Currency, Balance)
VALUES (2, 'ACC444555666', 'GBP', 1500.00);

If we examine the data, we see that sensitive account information is present:

MariaDB [Customers]> SELECT * FROM Individuals;
+----+----------------+-----------------------+--------------+---------------------------------+
| ID | Name           | Email                 | PhoneNumber  | Address                         |
+----+----------------+-----------------------+--------------+---------------------------------+
|  1 | Razvan Varzaru | razvan@provider.com   | +40700060000 | 456 Real Rd, Bucharest, Romania |
|  2 | John Doe       | john.doe@provider.com | +40700060001 | 123 Fake St, Springfield, USA   |
+----+----------------+-----------------------+--------------+---------------------------------+

MariaDB [Customers]> SELECT * FROM Accounts;
+-----------+------------+---------------+----------+---------+
| AccountID | CustomerID | AccountNumber | Currency | Balance |
+-----------+------------+---------------+----------+---------+
|         1 |          1 | ACC123456789  | EUR      | 1000.00 |
|         2 |          2 | ACC444555666  | GBP      | 1500.00 |
+-----------+------------+---------------+----------+---------+

Memory dump

Now it’s time to check out how this setup protects data-in-transit, data-at-rest and data-in-use. To show this, we perform a strong attack assuming a malicious administrator with full host OS privileges. Such an administrator can perform a memory dump of every running VM.

First, the regular VM without Intel® TDX protection:

# Memory dump of the regular domain
:~/memdumps$ virsh dump tdvirsh-regular_vm-00c643ab-205d-41b5-873d-a60800b284f3 ./regular-dom-mem-dump.file --memory-only

# Suppose ACC123 it's just a standard pattern for identifying the bank and the account type.

:~/memdumps$ strings regular-dom-mem-dump.file | grep -aE -B 5 -A 5 'ACC123'

Balance
Razvan Varzaru
razvan@provider.com
+40700060000
456 Real Rd, Bucharest, Romania
ACC123456789
1000.00i
John Doe
john.doe@provider.com
+40700060001
123 Fake St, Springfield, USA
--
AUE1
default-character-set=utf8mb3
default-collation=utf8mb3_general_ci
infimum
supremum
ACC123456789EUR
ACC444555666GBP
I:6769510
E:ID_MM_CANDIDATE=1
G:systemd
Q:systemd

...
...
# and there's more output to it

With enough patience and further investigation into the results, we can discover that Razvan has the account number ACC123456789 and a balance of 1,000.00 EUR.

As we can see, an administrator can basically read all the sensitive data, even with data-at-rest and data-in-transit protection.

Now let’s try the same on the Intel® TDX protected VM.

# Memory dump of the TD
:~/memdumps$ virsh dump tdvirsh-trust_domain-ae41a94c-43f3-4bf0-9c30-b66a13ca6b42 ./tdx-dom-mem-dump.file --memory-only

:~/memdumps$ strings tdx-dom-mem-dump.file | grep -aE -B 5 -A 5 'ACC123'
# Nothing!
:~/memdumps$ strings tdx-dom-mem-dump.file | grep -aE -B 5 -A 5 'Razvan'
:~/memdumps$
# Nothing x2!

Indeed, it works seamlessly. With Intel® TDX and a properly configured MariaDB Server installation, data protection is effectively ensured, at rest, in transit and in use. This setup ensures that no external actor can access or read the sensitive information stored within the Intel® TDX protected MariaDB Server.

Still, there are certain limits of the state we are in now:

  • we say that data is protected in use by doing a main memory dump but if you deploy the same workload at any external infrastructure provider, you normally do not have host OS access and thus cannot perform this test. Therefore, to address this, MariaDB Server can be combined with attestation to remotely verify the protection guarantees.
  • the key used for data-at-rest protection is created inside the TD. As a result, this requires a VM administrator that is allowed to login. Additionally, this key is written to a file which is not protected by Intel® TDX.

In conclusion, by leveraging the features of Intel® TDX, we can fully protect our data from unauthorized access and maintain the integrity of our information assets. We will explore in a future blog post how we can address the limits mentioned above.