In the previous blog we created a stateless application, deployed with K8s resource Deployment, which allows one to replicate the application, but where data is lost when Pods are restarted, meaning there were no data consistency. In the same blog we used PersistentVolumeClaim for dynamic provisioning of PersistentVolume, but we used Deployment, meant for stateless application, and this way is *not recommended* for statefulset application where each replica should have its own persistent volume. The proper way to achieve that is through the Statefulset resource and this post we will cover that.
In K8s one can create a stateful application, an application like a database, which needs to save data to persistent disk storage for use by the server/clients/other applications, to keep track of its state and to be able to replicate and be used in distributed systems. The stateful application is deployed using the K8s resource called StatefulSet.
StatefulSet deploys Pods based on the container specification, like Deployments, but maintains a sticky identity for each Pod. Pods are created from the same specification, but are not interchangeable, and have a persistent identifier across rescheduling, meaning that when a Pod dies it gets replaced by a new Pod, but keeps its identity.
Let’s see how the configuration file might look like (find it on GitHub).
apiVersion: v1 kind: Service metadata: name: mariadb-service labels: app: mariadb spec: ports: - port: 3306 name: mariadb-port clusterIP: None selector: app: mariadb --- apiVersion: apps/v1 kind: StatefulSet metadata: name: mariadb-sts spec: serviceName: "mariadb-service" replicas: 3 selector: matchLabels: app: mariadb template: metadata: labels: app: mariadb spec: containers: - name: mariadb image: mariadb ports: - containerPort: 3306 name: mariadb-port env: # Using Secrets - name: MARIADB_ROOT_PASSWORD valueFrom: secretKeyRef: name: mariadb-secret key: mariadb-root-password volumeMounts: - name: datadir mountPath: /var/lib/mysql/ volumeClaimTemplates: - metadata: name: datadir spec: accessModes: [ "ReadWriteOnce" ] resources: requests: storage: 300M
From above, first we have created the Service, namely the Headless type (Cluster IP equals
None) service. StatefulSet needs to have this service to be responsible for the network identity of the Pods, but we need to create it. It is used for DNS lookups between MariaDB Pods and clients within a cluster.
VolumeClaimTemplates is a list of claims that Pods are allowed to reference. Every claim in this list must have at least one matching (by name) volumeMount in one container in the template. A claim in this list takes precedence over any volumes in the template with the same name. So we have created
datadir PersistentVolumeClaim, dynamically provisioned to mount in the container path, the default data directory. For each VolumeClaimTemplate entry defined in a StatefulSet, each Pod receives one PersistentVolumeClaim. In the above example each Pod receives a single PersistentVolume with a StorageClass of default (standard) and 300 MB of provisioned storage. The name of the Pod will be prefixed to the name of the mounted volume (like
Let’s deploy Statefulset by first creating the Secret and afterwards deploying the manifest from above.
# Create the Secret $ kubectl apply -f mariadb-secret.yaml secret/mariadb-secret created # Create service/sts $ kubectl apply -f mariadb-sts.yaml service/mariadb-service created statefulset.apps/mariadb-sts created # Verify sts $ kubectl get sts NAME READY AGE mariadb-sts 1/3 8s # Use wide option $ kubectl get statefulset mariadb-sts -o wide NAME READY AGE CONTAINERS IMAGES mariadb-sts 3/3 19m mariadb mariadb # Verify pods (wait until all are in running state) $ kubectl get pods NAME READY STATUS RESTARTS AGE mariadb-sts-0 1/1 Running 0 2m29s mariadb-sts-1 1/1 Running 0 2m23s mariadb-sts-2 1/1 Running 0 2m18s # Verify service $ kubectl get svc -l app=mariadb NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE mariadb-service ClusterIP None <none> 3306/TCP 22s
Now we have deployed a stateful MariaDB application. Note that Pods created are ordered by number prefix, not by random hash value.
Each Pod has a stable hostname based on its ordinal index:
$ for i in 0 1 2; do kubectl exec "mariadb-sts-$i" -- bash -c "hostname"; done mariadb-sts-0 mariadb-sts-1 mariadb-sts-2
To get the Fully Qualified Domain Name (FQDN) of each Pod in the StatefulSet use the following command:
$ for i in 0 1 ; do kubectl exec "mariadb-sts-$i" -- hostname -f; done mariadb-sts-0.mariadb-service.default.svc.cluster.local mariadb-sts-1.mariadb-service.default.svc.cluster.local
mariadb-service Service creates a domain for all of the Pods,
Let’s now scale down and scale up the application by scaling the number of replicas and observe the results
# Scale down $ kubectl scale sts mariadb-sts --replicas=2 # Watch the Pods $ kubectl get pods -w NAME READY STATUS RESTARTS AGE mariadb-sts-0 1/1 Running 0 4m27s mariadb-sts-1 1/1 Running 0 4m21s mariadb-sts-2 1/1 Running 0 4m16s mariadb-sts-2 1/1 Terminating 0 4m38s # Scale up $ kubectl scale sts mariadb-sts --replicas=4 # Watch the Pods $ kubectl get pods -w NAME READY STATUS RESTARTS AGE mariadb-sts-0 1/1 Running 0 7m19s mariadb-sts-1 1/1 Running 0 7m13s mariadb-sts-2 0/1 Pending 0 0s mariadb-sts-2 0/1 Pending 0 0s mariadb-sts-2 0/1 ContainerCreating 0 0s mariadb-sts-2 1/1 Running 0 5s mariadb-sts-3 0/1 Pending 0 0s mariadb-sts-3 0/1 Pending 0 0s mariadb-sts-3 0/1 Pending 0 1s mariadb-sts-3 0/1 ContainerCreating 0 1s mariadb-sts-3 1/1 Running 0 4s # Alternatively here we can use the kubectl edit command to change the number of replicas
We can conclude that when scaling down, the last replica is terminated, while when scaling up the number prefix is increased in an ordinal way, and no Pod is created after the previous Pod is in the “Running” state (compare Pod 2 and 3).
Pods can be deleted with the kubectl delete command and will be recreated with the same ordinal number prefix
# Since Pods retain their sticky identity # let's remove Pod with ordinal index 0 $ kubectl delete pod mariadb-sts-0 pod "mariadb-sts-0" deleted # Watch the Pods during deletion $ kubectl get pods -w NAME READY STATUS RESTARTS AGE mariadb-sts-0 1/1 Running 0 31m mariadb-sts-1 1/1 Running 0 31m mariadb-sts-2 1/1 Running 0 24m mariadb-sts-3 1/1 Running 0 24m mariadb-sts-0 1/1 Terminating 0 31m mariadb-sts-0 0/1 Terminating 0 31m mariadb-sts-0 0/1 Terminating 0 31m mariadb-sts-0 0/1 Terminating 0 31m mariadb-sts-0 0/1 Pending 0 0s mariadb-sts-0 0/1 Pending 0 0s mariadb-sts-0 0/1 ContainerCreating 0 0s mariadb-sts-0 1/1 Running 0 4s
Since PersistentVolumeClaims are dynamically provisioned, we can look for them too
$ kubectl get pvc -l app=mariadb NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE datadir-mariadb-sts-0 Bound pvc-bdc2be42-f3b6-467e-a3b2-ad5bf882c556 300M RWO standard 33m datadir-mariadb-sts-1 Bound pvc-f05d44c6-0bfe-4ba4-a6c4-8312341b8367 300M RWO standard 33m datadir-mariadb-sts-2 Bound pvc-6573cd4d-8e83-4273-8764-33fb2bce6633 300M RWO standard 33m
Each Pod has its own persistent volume created, so let’s test that. Create sample data in the first Pod
$ kubectl exec -it mariadb-sts-0 -- mariadb -uroot -psecret -e "create database if not exists mytest0; use mytest0; create table t(t int); insert into t values (1),(2); select * from t;" +------+ | t | +------+ | 1 | | 2 | +------+
Now we will be able to get data from that Pod, but not from the other Pods. In order to get the same data as in the first Pod, one can use MariaDB’s replication feature.
$ kubectl exec mariadb-sts-0 -- mariadb -uroot -psecret -e "show databases like '%test%'; use mytest0; select * from t;" Database (%test%) mytest0 t 1 2 $ kubectl exec mariadb-sts-1 -- mariadb -uroot -psecret -e "show databases like '%test%'; use mytest0; select * from t;" ERROR 1049 (42000) at line 1: Unknown database 'mytest0' command terminated with exit code 1
When deleting the Pod try for homework to verify that the persistent volume will remain.
To delete the statefulset use the
kubectl delete statefulset command.
This blog showed how to create a MariaDB Statefulset application and how to work with it.
With this blog we have finished small series of blogs on MariaDB & K8s.
You are welcome to chat about it on Zulip.
- Start MariaDB in K8s
- MariaDB & K8s: Communication between containers/Deployments
- MariaDB & K8s: Create a Secret and use it in MariaDB deployment
- MariaDB & K8s: Deploy MariaDB and WordPress using Persistent Volumes
- Create statefulset MariaDB application in K8s
- MariaDB replication using containers
- MariaDB & K8s: How to replicate MariaDB in K8s