Configure mTLS Connection in Zalando Postgres Operator

Posted on Aug 11, 2022

Overview

We want to protect the communications between Postgres server and the clients. Our goal in tutorial will be mTLS.

SSL/TLS

Transport Layer Security (TLS), and its now-deprecated predecessor, Secure Sockets Layer (SSL), are cryptographic protocols designed to provide communications security over a computer network.

mTLS

Mutual Transport Layer Security (mTLS) is a process that establishes an encrypted TLS connection in which both parties use X.509 digital certificates to authenticate each other. MTLS can help mitigate the risk of moving services to the cloud and can help prevent malicious third parties from imitating genuine apps.

Install cert-manager to cluster

First of all we need to install cert-manager to our Kubernetes cluster:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.2/cert-manager.yaml

For more information refer to the official documentation.

Create Issuers and certificates

Create Self-Signed Issuer

There are two types of issuers in cert-manager, Issuer and ClusterIssuer. Issuer is used for one namespace and ClusterIssuer for multiple namepaces. In this tutorial we will use just an Issuer. Let’s create postgres-operator namespace first:

kubectl create ns postgres-operator

Create an Issuer:

self-signed-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-issuer
  namespace: postgres-operator
spec:
  selfSigned: {}

Deploy the Issuer:

kubectl create -f certs-manifests/self-signed-issuer.yaml

Check:

kubectl get issuer
NAME                READY   AGE
selfsigned-issuer   True    5s

Create Self-Signed Root CA certificate

Create CA (root) certificate.

self-signed-ca-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
 name: ca-cert
 namespace: postgres-operator
spec:
  secretName: ca-cert
  commonName: "root.postgres-operator.svc.cluster.local"
  isCA: true
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: Issuer
    group: cert-manager.io
  dnsNames:
  - "root.postgres-operator.svc.cluster.local"

Create:

kubectl create -f self-signed-ca-cert.yaml

Check the certificate:

kubectl get cert ca-cert
NAME      READY   SECRET    AGE
ca-cert   True    ca-cert   24s

Check the secret:

kubectl get secret ca-cert
NAME      TYPE                DATA   AGE
ca-cert   kubernetes.io/tls   3      55s

Secret contains three files ca.crt, tls.crt and tls.key.

Test that the certificate is valid:

openssl x509 -in <(kubectl get secret ca-cert \
  -o jsonpath='{.data.tls\.crt}' | base64 -d) \
  -text -noout

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            45:d4:2f:07:f8:63:14:37:37:9f:20:33:8a:a2:f5:af
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = root.postgres-operator.svc.cluster.local
        Validity
            Not Before: Jul  5 17:57:27 2022 GMT
            Not After : Oct  3 17:57:27 2022 GMT
        Subject: CN = root.postgres-operator.svc.cluster.local
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:55:67:43:5a:63:3f:0b:8d:a5:21:dc:7d:d8:62:
                    06:d0:ea:69:c2:d2:c7:a5:a9:e0:f8:51:ec:0b:66:
                    48:6c:d0:9c:21:ee:8f:e5:9e:9d:93:2b:c4:71:33:
                    75:2d:69:76:a8:db:4d:5f:a7:5b:02:4d:40:78:42:
                    af:1d:ef:f6:8a
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment, Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Subject Key Identifier:
                54:78:B8:CD:D0:EB:97:87:45:98:D9:47:FC:6C:E6:0D:C9:B3:DB:68
            X509v3 Subject Alternative Name:
                DNS:root.postgres-operator.svc.cluster.local
    Signature Algorithm: ecdsa-with-SHA256
         30:44:02:20:73:88:9b:65:00:ad:3a:c8:b8:52:87:33:86:e6:
         3f:dd:ca:99:95:cf:c1:38:88:e9:77:1e:7b:65:66:d0:38:c7:
         02:20:5e:d3:13:5c:0e:39:c9:9f:f5:6e:49:0f:be:90:c2:61:
         b3:8d:9c:59:e6:11:e2:23:11:63:92:e1:2f:34:c4:4e

Create Self-Signed CA Issuer

Now we have CA certificate then let’s create another Issuer which will based on the CA certificate. We will use this CA Issuer in order to issue the certificates for server and client(s).

self-signed-ca-issuer.yaml

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: selfsigned-ca-issuer
  namespace: postgres-operator
spec:
  ca:
    secretName: ca-cert

Create:

kubectl create -f self-signed-ca-issuer.yaml

Check CA Issuer:

kubectl get issuer
NAME                   READY   AGE
selfsigned-ca-issuer   True    5s
selfsigned-issuer      True    7m54s

Create Self-Signed Server certificate

Create self-signed certificate for server based on the CA Issuer.

self-signed-server-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-server-cert
  namespace: postgres-operator
spec:
  secretName: selfsigned-server-cert
  commonName: "zalando-postgres-cluster.postgres-operator.svc.cluster.local"
  isCA: false
  dnsNames:
  - "zalando-postgres-cluster.postgres-operator.svc.cluster.local"
  issuerRef:
    name: selfsigned-ca-issuer

Create:

kubectl create -f self-signed-server-cert.yaml

Check server certificate:

kubectl get cert
NAME                     READY   SECRET                   AGE
ca-cert                  True    ca-cert                  9m1s
selfsigned-server-cert   True    selfsigned-server-cert   9s

Check server secret:

kubectl get secret selfsigned-server-cert
NAME                     TYPE                DATA   AGE
selfsigned-server-cert   kubernetes.io/tls   3      35s

Create Self-Signed Client certificate

verify-ca SSL mode

If the parameter sslmode is set to verify-ca, libpq will verify that the server is trustworthy by checking the certificate chain up to the root certificate stored on the client.

For this mode the following configuration pg_hba.conf is required:

hostssl all             all           all      md5  clientcert=verify-ca

self-signed-client-verify-ca-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-client-cert
  namespace: postgres-operator
spec:
  secretName: selfsigned-client-cert
  commonName: "client.postgres-operator.svc.cluster.local"
  isCA: false
  dnsNames:
  - "client.postgres-operator.svc.cluster.local"
  issuerRef:
    name: selfsigned-ca-issuer

Create:

kubectl create -f self-signed-client-verify-ca-cert.yaml

verify-full SSL mode

If sslmode is set to verify-full, libpq will also verify that the server host name matches the name stored in the server certificate. The SSL connection will fail if the server certificate cannot be verified. verify-full is recommended in most security-sensitive environments.

For this mode the following configuration pg_hba.conf is required:

hostssl all             all           all      cert

cert here is auth-method which the same as trust clientcert=verify-full.

self-signed-client-verify-full-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-client-for-ca-cert
  namespace: postgres-operator
spec:
  secretName: selfsigned-client-for-ca-cert
  commonName: "postgres"
  isCA: false
  dnsNames:
  - "postgres"
  issuerRef:
    name: selfsigned-ca-issuer

Create:

kubectl create -f self-signed-client-verify-full-cert.yaml

Check certificate:

kubectl get certs
NAME                            READY   SECRET                          AGE
ca-cert                         True    ca-cert                         34m
selfsigned-client-for-ca-cert   True    selfsigned-client-for-ca-cert   11s
selfsigned-server-cert          True    selfsigned-server-cert          25m

Check secret:

kubectl get secret selfsigned-client-for-ca-cert
NAME                            TYPE                DATA   AGE
selfsigned-client-for-ca-cert   kubernetes.io/tls   3      2m14s

Install Zalando Postgres Operator

Clone Zalando Postgres Operator and checkout to the v1.8.2 branch:

git clone https://github.com/zalando/postgres-operator.git
cd postgres-operator
git checkout v1.8.2

Install operator:

helm install postgres-operator ./charts/postgres-operator

Check Pods:

kubectl get pods

Get operator configuration:

kubectl get operatorconfiguration
NAME                IMAGE                                               CLUSTER-LABEL   SERVICE-ACCOUNT   MIN-INSTANCES   AGE
postgres-operator   registry.opensource.zalan.do/acid/spilo-14:2.1-p3   cluster-name    postgres-pod      -1              28s

Configure CR.

minimal-postgres-manifest.yaml

apiVersion: "acid.zalan.do/v1"
kind: postgresql
metadata:
  name: zalando-postgres-cluster
  namespace: postgres-operator
spec:
  spiloFSGroup: 103  # <-- this option must be enabled for the group permissions
  tls:
    secretName: "selfsigned-server-cert"
    caSecretName: "ca-cert"
    caFile: "ca.crt"
  teamId: "zalando"
  volume:
    size: 1Gi
  numberOfInstances: 2
  users:
    zalando:  # database owner
    - superuser
    - createdb
    foo_user: []  # role for application foo
  databases:
    foo: zalando  # dbname: owner
  preparedDatabases:
    bar: {}
  postgresql:
    version: "14"
    parameters:
      log_connections: "ON"
      log_directory: /var/log/postgresql/
      log_disconnections: "ON"
      log_min_messages: debug5
  patroni:
    initdb:
      encoding: "UTF8"
      locale: "en_US.UTF-8"
      data-checksums: "true"
    pg_hba:
      - local   all             all                             trust
      - hostssl all             +zalandos    127.0.0.1/32       pam
      - host    all             all          127.0.0.1/32       pam
      - hostssl all             +zalandos    ::1/128            pam
      - host    all             all          ::1/128            pam
      - local   replication     standby                         trust
      - hostssl replication     standby all                     pam
      - hostnossl all           all   all          reject
      - hostssl all             +zalandos     all      pam
      - hostssl all             all           all      cert

Create Postgres cluster:

kubectl create -f manifests/minimal-postgres-manifest.yaml

Check Pods:

kubectl get pods

Check the logs:

kubectl logs zalando-postgres-cluster-0 -f

Connect to the Postgres:

kubectl exec -it zalando-postgres-cluster-0 -- bash

psql -U postgres

Deploy Ubuntu as a client for Postgres

ubuntu-pod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu
  namespace: postgres-operator
  labels:
    app: ubuntu
spec:
  containers:
  - name: ubuntu
    image: ubuntu:latest
    command: ["/bin/sleep", "3650d"]
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - name: client
      mountPath: "/tls/client"
      readOnly: true
    - name: ca-cert
      mountPath: "/tls/ca-cert"
      readOnly: true
  volumes:
  - name: client
    secret:
      secretName: selfsigned-client-verify-full-cert
  - name: ca-cert
    secret:
      secretName: ca-cert
  restartPolicy: Always

Exec to the Ubuntu pod:

kubectl exec -it ubuntu -- bash

Install Postgres client:

apt update && apt install postgresql-client -y

Create ~/.postgresql directory (default for psql). Copy root.crt cert and client cert with private key:

mkdir ~/.postgresql && \
cd ~/.postgresql

cat /tls/ca-cert/ca.crt > root.crt && \
cat /tls/client/tls.crt > client.crt && \
cat /tls/client/tls.key > client.key && \
chmod 600 client.key

Check the connection:

psql -U postgres -h zalando-postgres-cluster.postgres-operator.svc.cluster.local -d "sslmode=verify-full dbname=postgres sslrootcert=root.crt sslcert=client.crt sslkey=client.key"
psql (14.4 (Ubuntu 14.4-0ubuntu0.22.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

postgres=#

Let’s do the additional checks:

postgres=# \d pg_stat_ssl
              View "pg_catalog.pg_stat_ssl"
    Column     |  Type   | Collation | Nullable | Default
---------------+---------+-----------+----------+---------
 pid           | integer |           |          |
 ssl           | boolean |           |          |
 version       | text    |           |          |
 cipher        | text    |           |          |
 bits          | integer |           |          |
 client_dn     | text    |           |          |
 client_serial | numeric |           |          |
 issuer_dn     | text    |           |          |

postgres=# \x
Expanded display is on.

postgres=# SELECT * FROM pg_stat_ssl;
...
-[ RECORD 2 ]-+---------------------------------------------
pid           | 391
ssl           | t
version       | TLSv1.3
cipher        | TLS_AES_256_GCM_SHA384
bits          | 256
client_dn     | /CN=postgres
client_serial | 279330686703367488231752209583344471391
issuer_dn     | /CN=root.postgres-operator.svc.cluster.local

postgres=#

That’s all for today.