Rethinking PAM

Written by Written by Daniel Black Leave a comment on Rethinking PAM

PAM, Pluggable Authentication Modules, originated in Solaris and became common place in all Linux, BSDs, and AIX, HP-UX and macOS. MariaDB has had PAM support as an option since MariaDB 5.2.10 becoming stable in 10.0.11. So if this is all old and boring why reconsider it? The environment in which Linux operates with PAM has changed in the last 15 years.

A stack of blocks
Basic Blocks: https://publicdomainvectors.org/en/free-clipart/Four-colorful-building-blocks-vector-image/29819.html

MariaDB’s PAM plugin version 1 (as its become known) has a restriction, because the pam module loaded was loaded into the mariadbd executable, it was subject to the restricted permissions, normally just the user, that mariadbd runs. This user cannot read /etc/shadow for instance which needs to be read on Linux OSes to determine a user password. Other credential stores may be exposed, and if the pam module can read them, so can a MariaDB user with FILE privileges, (LOAD DATA INFILE….).

Along came PAMv2 in MariaDB 10.4.0 (stable in 10.4.10) that made a suid helper executable, runnable by the MariaDB, that now has full access to perform lookups. This also adds durability that a crashing pam plugin no-longer crashes the server. Another important feature was because its a separate process, it works with PAM modules that don’t function correctly performing concurrent authentication from different threads. Moving on quite a few years and the number of Linux and systemd provided controls that restrict a MariaDB systemd service have increased for the good of security isolation, some by default, some added, but many are still unimplemented as inherited processes, including the suid helper, become hindered by such a restriction.

Advances in Linux kernel protection added features like “no new privileges”, a convenient prctl on Linux processes acts as a safeguard of privilege escalation, now acts as a barrier on hardening systemd services.

So what choices do we have? One is to put MariaDB PAM as a separate package that has lower hardening so that PAM can function. Not ideal.

SELinux and AppArmor

As a local Linux PAM authentication is a common approach for external authentication, for pam_unix.so it uses a unix_chkpwd suid executable that can read /etc/shadow. Before this is executed the pam_unix.so, it does the syscall setuid(0) (requesting root privileges), which fails because mariadbd is a low privilege process. It continues and executes the unix_chkpwd, where this process, despite its root suid status, changes permission to the calling user, and relied on that users access to check the password. So we need to lower the permissions on /etc/shadow. Didn’t we want to avoid this? Yes, but please continue.

With SELinux and AppArmor, are we protected against SQL access to the this file? Yes as it runs out. Linux Security Modules, of which SELinux and AppArmor are examples, by design, act as constraints that extend file system restrictions only. They cannot grant things that a filesystem ACL prevents. Lets look at it:

Install the auth_pam_v1 plugin:

INSTALL SONAME 'auth_pam_v1'

Identify / create a basic pam configuration /etc/pam.d/mariadb_basic:

auth required pam_unix.so
account required pam_unix.so

Create a user that maps to an existing unix user:

create user bob@localhost identified via pam using 'mariadb_basic';

It cannot login yet (do test) as the MariaDB process cannot access /etc/shadow, so lets give it permissions:

# setfacl -m u:mysql:r /etc/shadow

Can we log in? Yes:

$ mariadb -p -u bob
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 15
Server version: 11.8.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select user(), current_user()\G
*************************** 1. row ***************************
        user(): bob@localhost
current_user(): bob@localhost
1 row in set (0.000 sec)

As importantly, can we read the /etc/shadow file? This requires FILE permissions in MariaDB. So as a MariaDB use with such privileges;

MariaDB [(none)]> SELECT LOAD_FILE('/etc/shadow')\G
*************************** 1. row ***************************
LOAD_FILE('/etc/shadow'): NULL
1 row in set (0.000 sec)

If you see the contents of /etc/shadow here ,you don’t have a Linux Security Module (LSM) like SELinux or AppArmor providing an additional layer of protection over the filesystem permissions and exposing this information exposes users and hashed passwords which might be able to be brute force reversed into clear text passwords.

How did it work for PAM authentication? Despite the pam_unix.so‘s helper unix_checkpwd dropping privileges, the LSM granted permission for it to read the /etc/shadow file.

With AppArmor it has a permission like:

profile unix-chkpwd /{,usr/}{,s}bin/unix_chkpwd {
...
/etc/shadow r,
...
}

SELinux, by the runtime domain on unix_chkpwd and the /etc/shadow filesystem label, has the rule:

allow chkpwd_t shadow_t:file read_file_perms;

The mariadbd executable does not have such a permission.

What are the other options?

SSSD – System Security Services Daemon using PAM

SSSD is written as service layer for having a very feature-full experience and its accessible by PAM. How does this help us? Well the pam_sssd.so uses a DBUS IPC on the authentication, so like the PAMv2 helper of MariaDB, its running as a separate service. It also runs as a separate user – sssd. Like our PAMv2 plugin, it has process separation to protect against crashing the MariaDB server.

Let’s start with basic pam configuration /etc/pam.d/mariadb:

auth    required    pam_sss.so domains=mariadb
account required    pam_sss.so domains=mariadb

And a basic /etc/sssd/sssd.conf file:

[sssd]
services = pam
domains = mariadb

[domain/mariadb]
id_provider = proxy
proxy_lib_name = files
proxy_pam_target = sssd-shadowutils

I’m using a sssd-proxy as a proxy that takes, for the domain mariadb, and performs a authentication with the sssd-shadowutils pam service. This happens to be installed by the sssd-common package but is otherwise is almost as basic as the pam_unix.so in the first example above.

Start the sssd service:

systemctl start sssd.service

And grant the sssd unix user access to the shadow file. Like MariaDB the sssd service processes are low privilege:

setfacl -m u:sssd:r /etc/shadow

Recreate our MariaDB user to use the sss based pam service:

create or replace user bob@localhost  IDENTIFIED VIA pam USING 'mariadb'

And connect:

$ mariadb -u bob -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 18
Server version: 11.8.8-MariaDB MariaDB Server

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select user(), current_user()\G
*************************** 1. row ***************************
        user(): bob@localhost
current_user(): bob@localhost
1 row in set (0.000 sec)

For all the amazing facilities of SSSD this is a rather a boring an overengineered way to use it as a proxy separator. Its backend functionality as centralized authentication service responder with a localised credential cache means it can replace some pam_ldap and provide integration with Windows AD, FreeIPA and OpenLDAP services in a way that is locally cached. All of these are fully functional and usable by MariaDB via PAM with a different sssd.conf configuration.

If you want something simple:

  • check your PAM plugin supports concurrency;
  • Use a LSM capable Linux distro;
  • and ensure it is enabled.

Conclusions

With a couple of going back to PAM v1 options available, we can harden our MariaDB service and the environment which is runs by using the systemd exposed features of the Linux kernel with systemctl edit mariadb.service:

[Service]
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true

RestrictRealtime=true
LockPersonality=true
RestrictNamespaces=true

RestrictRealtime=true
LockPersonality=true

With PAMv1 looking viable again, MDEV-15547 – auth_pam: add PAM_RHOST to pam info work can continue.

For the end user this shows that SELinux and Apparmor are valuable in their extension of the security of a Linux system to provided differentiated access for different executables under the same Linux user.

Credits