Security

Kubernetes RBAC

Install the requisite Operator RBAC resources, as a Kubernetes cluster admin user, by running a Makefile target:

make installrbac

This script creates the following RBAC resources on your Kubernetes cluster:

Setting Definition
Custom Resource Definitions (crd.yaml) pgbackups
pgclusters
pgpolicies
pgreplicas
pgtasks
pgupgrades
Cluster Roles (cluster-roles.yaml) pgopclusterrole
pgopclusterrolecrd
Cluster Role Bindings (cluster-roles-bindings.yaml) pgopclusterbinding
pgopclusterbindingcrd
Service Account (service-accounts.yaml) postgres-operator
pgo-backrest
Roles (rbac.yaml) pgo-role
pgo-backrest-role
Role Bindings (rbac.yaml) pgo-backrest-role-binding
pgo-role-binding

Note that the cluster role bindings have a naming convention of pgopclusterbinding-$PGO_OPERATOR_NAMESPACE and pgopclusterbindingcrd-$PGO_OPERATOR_NAMESPACE. The PGO_OPERATOR_NAMESPACE environment variable is added to make each cluster role binding name unique and to support more than a single Operator being deployed on the same Kube cluster.

Operator RBAC

The conf/postgresql-operator/pgorole file is read at start up time when the operator is deployed to the Kubernetes cluster. This file defines the Operator roles whereby Operator API users can be authorized.

The conf/postgresql-operator/pgouser file is read at start up time also and contains username, password, role, and namespace information as follows:

username:password:pgoadmin:
pgouser1:password:pgoadmin:pgouser1
pgouser2:password:pgoadmin:pgouser2
pgouser3:password:pgoadmin:pgouser1,pgouser2
readonlyuser:password:pgoreader:

The format of the pgouser server file is:

<username>:<password>:<role>:<namespace,namespace>

The namespace is a comma separated list of namespaces that user has access to. If you do not specify a namespace, then all namespaces is assumed, meaning this user can access any namespace that the Operator is watching.

A user creates a .pgouser file in their $HOME directory to identify themselves to the Operator. An entry in .pgouser will need to match entries in the conf/postgresql-operator/pgouser file. A sample .pgouser file contains the following:

username:password

The format of the .pgouser client file is:

<username>:<password>

The users pgouser file can also be located at: /etc/pgo/pgouser or it can be found at a path specified by the PGOUSER environment variable.

If the user tries to access a namespace that they are not configured for within the server side pgouser file then they will get an error message as follows:

Error: user [pgouser1] is not allowed access to namespace [pgouser2]

The following list shows the current complete list of possible pgo permissions that you can specify within the pgorole file when creating roles:

Permission Description
ApplyPolicy allow pgo apply
Cat allow pgo cat
CreateBackup allow pgo backup
CreateBenchmark allow pgo create benchmark
CreateCluster allow pgo create cluster
CreateDump allow pgo create pgdump
CreateFailover allow pgo failover
CreatePgbouncer allow pgo create pgbouncer
CreatePgpool allow pgo create pgpool
CreatePolicy allow pgo create policy
CreateSchedule allow pgo create schedule
CreateUpgrade allow pgo upgrade
CreateUser allow pgo create user
DeleteBackup allow pgo delete backup
DeleteBenchmark allow pgo delete benchmark
DeleteCluster allow pgo delete cluster
DeletePgbouncer allow pgo delete pgbouncer
DeletePgpool allow pgo delete pgpool
DeletePolicy allow pgo delete policy
DeleteSchedule allow pgo delete schedule
DeleteUpgrade allow pgo delete upgrade
DeleteUser allow pgo delete user
DfCluster allow pgo df
Label allow pgo label
Load allow pgo load
Ls allow pgo ls
Reload allow pgo reload
Restore allow pgo restore
RestoreDump allow pgo restore for pgdumps
ShowBackup allow pgo show backup
ShowBenchmark allow pgo show benchmark
ShowCluster allow pgo show cluster
ShowConfig allow pgo show config
ShowPolicy allow pgo show policy
ShowPVC allow pgo show pvc
ShowSchedule allow pgo show schedule
ShowNamespace allow pgo show namespace
ShowUpgrade allow pgo show upgrade
ShowWorkflow allow pgo show workflow
Status allow pgo status
TestCluster allow pgo test
UpdateCluster allow pgo update cluster
User allow pgo user
Version allow pgo version

If the user is unauthorized for a pgo command, the user will get back this response:

Error:  Authentication Failed: 401 

Making Security Changes

The Operator today requires you to make Operator user security changes in the pgouser and pgorole files, and for those changes to take effect you are required to re-deploy the Operator:

make deployoperator

This will recreate the pgo-config ConfigMap that stores these files and is mounted by the Operator during its initialization.

API Security

The Operator REST API is encrypted with keys stored in the pgo.tls Secret.

The pgo.tls Secret can be generated prior to starting the Operator or you can let the Operator generate the Secret for you if the Secret does not exist.

Adjust the default keys to meet your security requirements using your own keys. The pgo.tls Secret is created when you run:

make deployoperator

The keys are generated when the RBAC script is executed by the cluster admin:

make installrbac

In some scenarios like an OLM deployment, it is preferable for the Operator to generate the Secret keys at runtime, if the pgo.tls Secret does not exit when the Operator starts, a new TLS Secret will be generated. In this scenario, you can extract the generated Secret TLS keys using:

kubectl cp <pgo-namespace>/<pgo-pod>:/tmp/server.key /tmp/server.key -c apiserver
kubectl cp <pgo-namespace>/<pgo-pod>:/tmp/server.crt /tmp/server.crt -c apiserver

example of the command below:

kubectl cp pgo/postgres-operator-585584f57d-ntwr5:tmp/server.key /tmp/server.key -c apiserver
kubectl cp pgo/postgres-operator-585584f57d-ntwr5:tmp/server.crt /tmp/server.crt -c apiserver

This server.key and server.crt can then be used to access the pgo-apiserver from the pgo CLI by setting the following variables in your client environment:

export PGO_CA_CERT=/tmp/server.crt
export PGO_CLIENT_CERT=/tmp/server.crt
export PGO_CLIENT_KEY=/tmp/server.key

You can view the TLS secret using:

kubectl get secret pgo.tls -n pgo

or

oc get secret pgo.tls -n pgo

If you create the Secret outside of the Operator, for example using the default installation script, the key and cert that are generated by the default installation are found here:

$PGOROOT/conf/postgres-operator/server.crt 
$PGOROOT/conf/postgres-operator/server.key 

The key and cert are generated using the deploy/gen-api-keys.sh script. That script gets executed when running:

make installrbac

You can extract the server.key and server.crt from the Secret using the following:

oc get secret pgo.tls -n $PGO_OPERATOR_NAMESPACE -o jsonpath='{.data.tls\.key}' | base64 --decode > /tmp/server.key
oc get secret pgo.tls -n $PGO_OPERATOR_NAMESPACE -o jsonpath='{.data.tls\.crt}' | base64 --decode > /tmp/server.crt

This server.key and server.crt can then be used to access the pgo-apiserver REST API from the pgo CLI on your client host.