Create statefulset MariaDB application in K8s

Written by Written by Anel Husakovic 2 Comments on Create statefulset MariaDB application in K8s

In the previous blog we created a stateless application, deployed with K8s resource Deployment, which allows one to replicate the application, but where data is lost when Pods are restarted, meaning there were no data consistency. In the same blog we used PersistentVolumeClaim for dynamic provisioning of PersistentVolume, but we used Deployment, meant for stateless application, and this way is *not recommended* for statefulset application where each replica should have its own persistent volume. The proper way to achieve that is through the Statefulset resource and this post we will cover that.

In K8s one can create a stateful application, an application like a database, which needs to save data to persistent disk storage for use by the server/clients/other applications, to keep track of its state and to be able to replicate and be used in distributed systems. The stateful application is deployed using the K8s resource called StatefulSet.

StatefulSet deploys Pods based on the container specification, like Deployments, but maintains a sticky identity for each Pod. Pods are created from the same specification, but are not interchangeable, and have a persistent identifier across rescheduling, meaning that when a Pod dies it gets replaced by a new Pod, but keeps its identity.

Statefulset example

Let’s see how the configuration file might look like (find it on GitHub).

apiVersion: v1
kind: Service
metadata:
  name: mariadb-service
  labels:
    app: mariadb
spec:
  ports:
  - port: 3306
    name: mariadb-port
  clusterIP: None
  selector:
    app: mariadb
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mariadb-sts
spec:
  serviceName: "mariadb-service"
  replicas: 3
  selector:
    matchLabels:
      app: mariadb
  template:
    metadata:
      labels:
        app: mariadb
    spec:
      containers:
      - name: mariadb
        image: mariadb
        ports:
        - containerPort: 3306
          name: mariadb-port
        env:
        # Using Secrets
        - name: MARIADB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mariadb-secret
              key: mariadb-root-password
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/mysql/
  volumeClaimTemplates:
  - metadata:
      name: datadir
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 300M

From above, first we have created the Service, namely the Headless type (Cluster IP equals None) service. StatefulSet needs to have this service to be responsible for the network identity of the Pods, but we need to create it. It is used for DNS lookups between MariaDB Pods and clients within a cluster.

VolumeClaimTemplates is a list of claims that Pods are allowed to reference. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. A claim in this list takes precedence over any volumes in the template with the same name. So we have created datadir PersistentVolumeClaim, dynamically provisioned to mount in the container path, the default data directory. For each VolumeClaimTemplate entry defined in a StatefulSet, each Pod receives one PersistentVolumeClaim. In the above example each Pod receives a single PersistentVolume with a StorageClass of default (standard) and 300 MB of provisioned storage. The name of the Pod will be prefixed to the name of the mounted volume (like datadir-mariadb-sts-0).

Apply configuration files and verify

Let’s deploy Statefulset by first creating the Secret and afterwards deploying the manifest from above.

# Create the Secret
$ kubectl apply -f mariadb-secret.yaml 
secret/mariadb-secret created

# Create service/sts
$ kubectl apply -f mariadb-sts.yaml 
service/mariadb-service created
statefulset.apps/mariadb-sts created

# Verify sts
$ kubectl get sts
NAME          READY   AGE
mariadb-sts   1/3     8s

# Use wide option
$ kubectl get statefulset mariadb-sts -o wide
NAME          READY   AGE   CONTAINERS   IMAGES
mariadb-sts   3/3     19m   mariadb      mariadb

# Verify pods (wait until all are in running state)
$ kubectl get pods
NAME            READY   STATUS    RESTARTS   AGE
mariadb-sts-0   1/1     Running   0          2m29s
mariadb-sts-1   1/1     Running   0          2m23s
mariadb-sts-2   1/1     Running   0          2m18s

# Verify service
$ kubectl get svc -l app=mariadb
NAME              TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
mariadb-service   ClusterIP   None         <none>        3306/TCP   22s

Now we have deployed a stateful MariaDB application. Note that Pods created are ordered by number prefix, not by random hash value.

Test the stateful application

Each Pod  has a stable hostname based on its ordinal index:

$ for i in 0 1 2; do kubectl exec "mariadb-sts-$i" -- bash -c "hostname"; done
mariadb-sts-0
mariadb-sts-1
mariadb-sts-2

To get the Fully Qualified Domain Name (FQDN) of each Pod in the StatefulSet use the following command:

$ for i in 0 1 ; do kubectl exec "mariadb-sts-$i" -- hostname -f; done
mariadb-sts-0.mariadb-service.default.svc.cluster.local
mariadb-sts-1.mariadb-service.default.svc.cluster.local

The mariadb-service Service creates a domain for all of the Pods, mariadb-service.default.svc.cluster.local.

Let’s now scale down and scale up the application by scaling the number of replicas and observe the results

# Scale down
$ kubectl scale sts mariadb-sts --replicas=2
# Watch the Pods
$ kubectl get pods -w
NAME            READY   STATUS    RESTARTS   AGE
mariadb-sts-0   1/1     Running   0          4m27s
mariadb-sts-1   1/1     Running   0          4m21s
mariadb-sts-2   1/1     Running   0          4m16s
mariadb-sts-2   1/1     Terminating   0          4m38s

# Scale up
$ kubectl scale sts mariadb-sts --replicas=4
# Watch the Pods
$ kubectl get pods -w
NAME            READY   STATUS    RESTARTS   AGE
mariadb-sts-0   1/1     Running   0          7m19s
mariadb-sts-1   1/1     Running   0          7m13s
mariadb-sts-2   0/1     Pending   0          0s
mariadb-sts-2   0/1     Pending   0          0s
mariadb-sts-2   0/1     ContainerCreating   0          0s
mariadb-sts-2   1/1     Running             0          5s
mariadb-sts-3   0/1     Pending             0          0s
mariadb-sts-3   0/1     Pending             0          0s
mariadb-sts-3   0/1     Pending             0          1s
mariadb-sts-3   0/1     ContainerCreating   0          1s
mariadb-sts-3   1/1     Running             0          4s

# Alternatively here we can use the kubectl edit command to change the number of replicas

We can conclude that when scaling down, the last replica is terminated, while when scaling up the number prefix is increased in an ordinal way, and no Pod is created after the previous Pod is in the “Running” state (compare Pod 2 and 3).

Pods can be deleted with the kubectl delete command and will be recreated with the same ordinal number prefix

# Since Pods retain their sticky identity
# let's remove Pod with ordinal index 0
$ kubectl delete pod mariadb-sts-0
pod "mariadb-sts-0" deleted

# Watch the Pods during deletion
$ kubectl get pods -w
NAME            READY   STATUS    RESTARTS   AGE
mariadb-sts-0   1/1     Running   0          31m
mariadb-sts-1   1/1     Running   0          31m
mariadb-sts-2   1/1     Running   0          24m
mariadb-sts-3   1/1     Running   0          24m
mariadb-sts-0   1/1     Terminating   0          31m
mariadb-sts-0   0/1     Terminating   0          31m
mariadb-sts-0   0/1     Terminating   0          31m
mariadb-sts-0   0/1     Terminating   0          31m
mariadb-sts-0   0/1     Pending       0          0s
mariadb-sts-0   0/1     Pending       0          0s
mariadb-sts-0   0/1     ContainerCreating   0          0s
mariadb-sts-0   1/1     Running             0          4s

Since PersistentVolumeClaims are dynamically provisioned, we can look for them too

$ kubectl get pvc -l app=mariadb
NAME                    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-mariadb-sts-0   Bound    pvc-bdc2be42-f3b6-467e-a3b2-ad5bf882c556   300M       RWO            standard       33m
datadir-mariadb-sts-1   Bound    pvc-f05d44c6-0bfe-4ba4-a6c4-8312341b8367   300M       RWO            standard       33m
datadir-mariadb-sts-2   Bound    pvc-6573cd4d-8e83-4273-8764-33fb2bce6633   300M       RWO            standard       33m

Each Pod has its own persistent volume created, so let’s test that. Create sample data in the first Pod

$ kubectl exec -it mariadb-sts-0 -- mariadb -uroot -psecret -e "create database if not exists mytest0; use mytest0; create table t(t int); insert into t values (1),(2); select * from t;" 
+------+
| t    |
+------+
|    1 |
|    2 |
+------+

Now we will be able to get data from that Pod, but not from the other Pods. In order to get the same data as in the first Pod, one can use MariaDB’s replication feature.

$ kubectl exec mariadb-sts-0 -- mariadb -uroot -psecret -e "show databases like '%test%'; use mytest0; select * from t;"
Database (%test%)
mytest0
t
1
2
$ kubectl exec mariadb-sts-1 -- mariadb -uroot -psecret -e "show databases like '%test%'; use mytest0; select * from t;"
ERROR 1049 (42000) at line 1: Unknown database 'mytest0'
command terminated with exit code 1

When deleting the Pod try for homework to verify that the persistent volume will remain.

To delete the statefulset use the kubectl delete statefulset command.

Conclusion

This blog showed how to create a MariaDB Statefulset application and how to work with it.

With this blog we have finished small series of blogs on MariaDB & K8s.

You are welcome to chat about it on Zulip.

Read more