OPAのチュートリアル

eksworkshop.comのOPAのチュートリアルをやってみたメモ。

コンポーネント	バージョン	備考
eksctl	0.31.0
Kubernetes バージョン	1.18
プラットフォームのバージョン	eks.2
Gatekeeper	3.1.0

参考リンク

• Using Open Policy Agent (OPA) for policy-based control in EKS
• Using Gatekeeper as a drop-in Pod Security Policy replacement in Amazon EKS
• OPA Gatekeeper: Policy and Governance for Kubernetes

手順

Gatekeeperのデプロイ

OPA Gatekeeperをデプロイする。最新は3.2.1だが、eksworkshopは3.1。

$ kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.1/deploy/gatekeeper.yaml
namespace/gatekeeper-system created
customresourcedefinition.apiextensions.k8s.io/configs.config.gatekeeper.sh created
customresourcedefinition.apiextensions.k8s.io/constraintpodstatuses.status.gatekeeper.sh created
customresourcedefinition.apiextensions.k8s.io/constrainttemplatepodstatuses.status.gatekeeper.sh created
customresourcedefinition.apiextensions.k8s.io/constrainttemplates.templates.gatekeeper.sh created
serviceaccount/gatekeeper-admin created
role.rbac.authorization.k8s.io/gatekeeper-manager-role created
clusterrole.rbac.authorization.k8s.io/gatekeeper-manager-role created
rolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/gatekeeper-manager-rolebinding created
secret/gatekeeper-webhook-server-cert created
service/gatekeeper-webhook-service created
deployment.apps/gatekeeper-audit created
deployment.apps/gatekeeper-controller-manager created
validatingwebhookconfiguration.admissionregistration.k8s.io/gatekeeper-validating-webhook-configuration created

Podを確認する。レプリカが3つも動いている。Helmチャートだとこの辺りカスタマイズできそう。

$ kubectl get pods -n gatekeeper-system
NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-audit-576f6d6f8d-mpsjn                1/1     Running   0          45s
gatekeeper-controller-manager-85d8bf48c9-288r8   1/1     Running   0          45s
gatekeeper-controller-manager-85d8bf48c9-5zphf   1/1     Running   0          45s
gatekeeper-controller-manager-85d8bf48c9-wjft8   1/1     Running   0          45s

ルールの作成

ConstraintTemplateのマニフェストを作成する。ConstraintTemplateはクラスターワイドのリソース。

cat <<EOF > constrainttemplate.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8spspprivilegedcontainer
spec:
  crd:
    spec:
      names:
        kind: K8sPSPPrivilegedContainer
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8spspprivileged

        violation[{"msg": msg, "details": {}}] {
            c := input_containers[_]
            c.securityContext.privileged
            msg := sprintf("Privileged container is not allowed: %v, securityContext: %v", [c.name, c.securityContext])
        }

        input_containers[c] {
            c := input.review.object.spec.containers[_]
        }

        input_containers[c] {
            c := input.review.object.spec.initContainers[_]
        }
EOF

ConstraintTemplateを作成する。

$ kubectl apply -f constrainttemplate.yaml
constrainttemplate.templates.gatekeeper.sh/k8spspprivilegedcontainer created
$ kubectl get constrainttemplate
NAME                        AGE
k8spspprivilegedcontainer   48s

Constraintのマニフェストを作成する。こちらもクラスターワイドのリソース。ConstraintTemplateをどのリソースに適用するかを指定する。

KindがConstraintではないのがわかりにくい。

cat <<EOF > constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPPrivilegedContainer
metadata:
  name: psp-privileged-container
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
EOF

Constraintを作成する

$ kubectl apply -f constraint.yaml
k8spspprivilegedcontainer.constraints.gatekeeper.sh/psp-privileged-container created
$ kubectl get constraint
NAME                       AGE
psp-privileged-container   22s

試しに特権Podを作ってみる。

cat <<EOF > example1.yaml
apiVersion: v1
kind: Pod
metadata:
  name: bad-nginx
  labels:
    app: bad-nginx
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      privileged: true
EOF
kubectl apply -f example1.yaml

ちゃんと弾かれる。

$ kubectl apply -f example1.yaml
Error from server ([denied by psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {"privileged": true}): error when creating "example1.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {"privileged": true}

Deploymentの場合はどうなるか確認する。

cat <<EOF > example2.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: bad-nginx
  name: bad-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bad-nginx
  template:
    metadata:
      name: bad-nginx
      labels:
        app: bad-nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        securityContext:
          privileged: true
EOF
kubectl apply -f example2.yaml

この場合Deployment自体は作成できるが、Podは作られない。

$ kubectl apply -f example2.yaml
deployment.apps/bad-nginx created
$ kubectl get deploy
NAME        READY   UP-TO-DATE   AVAILABLE   AGE
bad-nginx   0/1     0            0           82s
$ kubectl get rs
NAME                   DESIRED   CURRENT   READY   AGE
bad-nginx-6cdffbc58f   1         0         0       87s
$ kubectl get pod
No resources found in default namespace.

イベントでブロックされていることが確認できる。

$ kubectl get ev --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON              OBJECT                            MESSAGE
2m38s       Normal    ScalingReplicaSet   deployment/bad-nginx              Scaled up replica set bad-nginx-6cdffbc58f to 1
76s         Warning   FailedCreate        replicaset/bad-nginx-6cdffbc58f   Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [denied by psp-privileged-container] Privileged container is not allowed: nginx, securityContext: {"privileged": true}

Deploymentを削除する。

$ kubectl delete -f example2.yaml
deployment.apps "bad-nginx" deleted

現在特権で動いているkube-proxyを停止してみると、予想通り起動してこない。

$ kubectl get pod -n kube-system
NAME                                            READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-6c4d8d8f64-rr4cb   1/1     Running   0          21d
aws-node-lzzv4                                  1/1     Running   0          6d16h
aws-node-nxwvh                                  1/1     Running   0          6d16h
coredns-86f7d88d77-9nkn6                        1/1     Running   0          21d
coredns-86f7d88d77-p7x2h                        1/1     Running   0          21d
external-dns-7855554c94-hvl48                   1/1     Running   0          21d
kube-proxy-2q6wb                                1/1     Running   0          21d
kube-proxy-9rvdt                                1/1     Running   0          21d
metrics-server-5f956b6d5f-2lqst                 1/1     Running   0          7d
$ kubectl delete pod -n kube-system kube-proxy-2q6wb
pod "kube-proxy-2q6wb" deleted
$ kubectl get pod -n kube-system
NAME                                            READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-6c4d8d8f64-rr4cb   1/1     Running   0          21d
aws-node-lzzv4                                  1/1     Running   0          6d16h
aws-node-nxwvh                                  1/1     Running   0          6d16h
coredns-86f7d88d77-9nkn6                        1/1     Running   0          21d
coredns-86f7d88d77-p7x2h                        1/1     Running   0          21d
external-dns-7855554c94-hvl48                   1/1     Running   0          21d
kube-proxy-9rvdt                                1/1     Running   0          21d
metrics-server-5f956b6d5f-2lqst                 1/1     Running   0          7d

イベントを確認すると、ブロックされていることが確認できる。

$ kubectl get ev -n kube-system --sort-by=.lastTimestamp
LAST SEEN   TYPE      REASON         OBJECT                 MESSAGE
2m11s       Normal    Killing        pod/kube-proxy-2q6wb   Stopping container kube-proxy
45s         Warning   FailedCreate   daemonset/kube-proxy   Error creating: admission webhook "validation.gatekeeper.sh" denied the request: [denied by psp-privileged-container] Privileged container is not allowed: kube-proxy, securityContext: {"privileged": true}

ワークロードを対象としてルールを作成してみる。

cat <<EOF > constrainttemplate2.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8spspprivilegedworkload
spec:
  crd:
    spec:
      names:
        kind: K8sPSPPrivilegedWorkload
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8spspprivileged

        violation[{"msg": msg, "details": {}}] {
            c := input_containers[_]
            c.securityContext.privileged
            msg := sprintf("Privileged container is not allowed: %v, securityContext: %v", [c.name, c.securityContext])
        }

        input_containers[c] {
            c := input.review.object.spec.template.spec.containers[_]
        }

        input_containers[c] {
            c := input.review.object.spec.template.spec.initContainers[_]
        }
EOF
kubectl apply -f constrainttemplate2.yaml
cat <<EOF > constraint2.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPPrivilegedWorkload
metadata:
  name: psp-privileged-workload
spec:
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["ReplicaSet"]
      - apiGroups: ["apps"]
        kinds: ["Deployment"]
      - apiGroups: ["apps"]
        kinds: ["DaemonSet"]
      - apiGroups: ["apps"]
        kinds: ["StatefulSet"]
      - apiGroups: ["batch"]
        kinds: ["Job"]
EOF
kubectl apply -f constraint2.yaml

これでもう一度Deploymentを作成してみると、ブロックできた。

$ kubectl apply -f example2.yaml
Error from server ([denied by psp-privileged-workload] Privileged container is not allowed: nginx, securityContext: {"privileged": true}): error when creating "example2.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [denied by psp-privileged-workload] Privileged container is not allowed: nginx, securityContext: {"privileged": true}

この辺りの説明を読むと、matchでセレクターを上手く使うことで除外は設定できそう。

先ほどのPodの制約について、kube-proxyを除外してみる。k8s-appラベルがついているものを除外する。

cat <<EOF > constraint.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sPSPPrivilegedContainer
metadata:
  name: psp-privileged-container
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    labelSelector:
      matchExpressions:
        - key: k8s-app
          operator: DoesNotExist
EOF
kubectl apply -f constraint.yaml

この条件があれば、適応を除外できる。

$ kubectl get pod -n kube-system
NAME                                            READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-6c4d8d8f64-rr4cb   1/1     Running   0          21d
aws-node-lzzv4                                  1/1     Running   0          6d17h
aws-node-nxwvh                                  1/1     Running   0          6d17h
coredns-86f7d88d77-9nkn6                        1/1     Running   0          21d
coredns-86f7d88d77-p7x2h                        1/1     Running   0          21d
external-dns-7855554c94-hvl48                   1/1     Running   0          21d
kube-proxy-r5jjz                                1/1     Running   0          29m
kube-proxy-r6zc7                                1/1     Running   0          2m15s
metrics-server-5f956b6d5f-2lqst                 1/1     Running   0          7d1h
$ kubectl delete pod -n kube-system kube-proxy-r5jjz
pod "kube-proxy-r5jjz" deleted
$ kubectl get pod -n kube-system
NAME                                            READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-6c4d8d8f64-rr4cb   1/1     Running   0          21d
aws-node-lzzv4                                  1/1     Running   0          6d17h
aws-node-nxwvh                                  1/1     Running   0          6d17h
coredns-86f7d88d77-9nkn6                        1/1     Running   0          21d
coredns-86f7d88d77-p7x2h                        1/1     Running   0          21d
external-dns-7855554c94-hvl48                   1/1     Running   0          21d
kube-proxy-fsbn8                                1/1     Running   0          13s
kube-proxy-r6zc7                                1/1     Running   0          2m50s
metrics-server-5f956b6d5f-2lqst                 1/1     Running   0          7d1h