Kyvernoを試す - sotoiwa’s blog

Kyvernoを試してみたメモ。AWSブログにしたがって試す。

参考リンク

Kyvernoのインストール

クラスターを作成する。

eksctl create cluster -f cluster.yaml

マニフェストまたはHelmでインストールが可能。

$ helm search repo kyverno
NAME            CHART VERSION   APP VERSION     DESCRIPTION                        
kyverno/kyverno v1.3.3          v1.3.3          Kubernetes Native Policy Management

今回はマニフェストで導入する。

$ kubectl create -f https://raw.githubusercontent.com/kyverno/kyverno/master/definitions/release/install.yaml
namespace/kyverno created
customresourcedefinition.apiextensions.k8s.io/clusterpolicies.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/clusterpolicyreports.wgpolicyk8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterreportchangerequests.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/generaterequests.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/policies.kyverno.io created
customresourcedefinition.apiextensions.k8s.io/policyreports.wgpolicyk8s.io created
customresourcedefinition.apiextensions.k8s.io/reportchangerequests.kyverno.io created
serviceaccount/kyverno-service-account created
clusterrole.rbac.authorization.k8s.io/kyverno:admin-policies created
clusterrole.rbac.authorization.k8s.io/kyverno:admin-policyreport created
clusterrole.rbac.authorization.k8s.io/kyverno:admin-reportchangerequest created
clusterrole.rbac.authorization.k8s.io/kyverno:customresources created
clusterrole.rbac.authorization.k8s.io/kyverno:generatecontroller created
clusterrole.rbac.authorization.k8s.io/kyverno:policycontroller created
clusterrole.rbac.authorization.k8s.io/kyverno:userinfo created
clusterrole.rbac.authorization.k8s.io/kyverno:webhook created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:customresources created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:generatecontroller created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:policycontroller created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:userinfo created
clusterrolebinding.rbac.authorization.k8s.io/kyverno:webhook created
configmap/init-config created
service/kyverno-svc created
deployment.apps/kyverno created

特権コンテナの禁止

以下のようなポリシーを作成する。

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged
  annotations:
    policies.kyverno.io/category: Security
    policies.kyverno.io/description: Privileged containers are defined as any
      container where the container uid 0 is mapped to the host’s uid 0.
      A process within a privileged container can get unrestricted host access.
      With `securityContext.allowPrivilegeEscalation` enabled, a process can
      gain privileges from its parent.
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-privileged
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Privileged mode is not allowed. Set privileged to false"
      pattern:
        spec:
          containers:
          - =(securityContext):
              =(privileged): false
  - name: validate-allowPrivilegeEscalation
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Privileged mode is not allowed. Set allowPrivilegeEscalation to false"
      pattern:
        spec:
          containers:
          - securityContext:
              allowPrivilegeEscalation: false

$ kubectl create -f disallow_privileged.yaml
clusterpolicy.kyverno.io/disallow-privileged created

作成したポリシーを確認する。

$ kubectl get cpol
NAME                  BACKGROUND   ACTION
disallow-privileged   true         enforce

自動生成の機能でアノテーションが追加されているし、Pod以外の定義も追加されている。

$ k get cpol disallow-privileged -o yaml | k neat
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  annotations:
    pod-policies.kyverno.io/autogen-controllers: DaemonSet,Deployment,Job,StatefulSet,CronJob
    policies.kyverno.io/category: Security
    policies.kyverno.io/description: Privileged containers are defined as any container
      where the container uid 0 is mapped to the host’s uid 0. A process within a
      privileged container can get unrestricted host access. With `securityContext.allowPrivilegeEscalation`
      enabled, a process can gain privileges from its parent.
  name: disallow-privileged
spec:
  background: true
  rules:
  - match:
      resources:
        kinds:
        - Pod
    name: validate-privileged
    validate:
      message: Privileged mode is not allowed. Set privileged to false
      pattern:
        spec:
          containers:
          - =(securityContext):
              =(privileged): false
  - match:
      resources:
        kinds:
        - Pod
    name: validate-allowPrivilegeEscalation
    validate:
      message: Privileged mode is not allowed. Set allowPrivilegeEscalation to false
      pattern:
        spec:
          containers:
          - securityContext:
              allowPrivilegeEscalation: false
  - match:
      resources:
        kinds:
        - DaemonSet
        - Deployment
        - Job
        - StatefulSet
    name: autogen-validate-privileged
    validate:
      message: Privileged mode is not allowed. Set privileged to false
      pattern:
        spec:
          template:
            spec:
              containers:
              - =(securityContext):
                  =(privileged): false
  - match:
      resources:
        kinds:
        - CronJob
    name: autogen-cronjob-validate-privileged
    validate:
      message: Privileged mode is not allowed. Set privileged to false
      pattern:
        spec:
          jobTemplate:
            spec:
              template:
                spec:
                  containers:
                  - =(securityContext):
                      =(privileged): false
  - match:
      resources:
        kinds:
        - DaemonSet
        - Deployment
        - Job
        - StatefulSet
    name: autogen-validate-allowPrivilegeEscalation
    validate:
      message: Privileged mode is not allowed. Set allowPrivilegeEscalation to false
      pattern:
        spec:
          template:
            spec:
              containers:
              - securityContext:
                  allowPrivilegeEscalation: false
  - match:
      resources:
        kinds:
        - CronJob
    name: autogen-cronjob-validate-allowPrivilegeEscalation
    validate:
      message: Privileged mode is not allowed. Set allowPrivilegeEscalation to false
      pattern:
        spec:
          jobTemplate:
            spec:
              template:
                spec:
                  containers:
                  - securityContext:
                      allowPrivilegeEscalation: false
  validationFailureAction: enforce

特権昇格が可能なNginxのPodを作成してみる。

apiVersion: v1
kind: Pod
metadata:
  name: nginx-privileged
  labels:
    app: nginx-privileged
spec:
  containers:
  - name: nginx
    image: nginx
    securityContext:
      allowPrivilegeEscalation: true

拒否される。

$ kubectl create -f nginx-privileged.yaml
Error from server: error when creating "nginx-privileged.yaml": admission webhook "validate.kyverno.svc" denied the request: 

resource Pod/default/nginx-privileged was blocked due to the following policies

disallow-privileged:
  validate-allowPrivilegeEscalation: 'validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule validate-allowPrivilegeEscalation failed at path /spec/containers/0/securityContext/allowPrivilegeEscalation/'

同じPodSpecのDeployment定義で試す。

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx-privileged
  name: nginx-privileged
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-privileged
  template:
    metadata:
      labels:
        app: nginx-privileged
    spec:
      containers:
      - image: nginx
        name: nginx
        securityContext:
          allowPrivilegeEscalation: true

ちゃんと拒否される。

$ k create -f nginx-privileged-deploy.yaml
Error from server: error when creating "nginx-privileged-deploy.yaml": admission webhook "validate.kyverno.svc" denied the request: 

resource Deployment/default/nginx-privileged was blocked due to the following policies

disallow-privileged:
  autogen-validate-allowPrivilegeEscalation: 'validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule autogen-validate-allowPrivilegeEscalation failed at path /spec/template/spec/containers/0/securityContext/allowPrivilegeEscalation/'

不明なイメージレジストリの禁止

不明なイメージレジストリを禁止する以下のポリシーを作成する。

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: deployment-pod-valid-registry
  labels:
    app: kyverno
  annotations:
    policies.kyverno.io/category: Compliance
    policies.kyverno.io/description: Rules to enforce correct image source registry
spec:
  validationFailureAction: enforce
  rules:
  - name: validate-registries
    match:
      resources:
        kinds:
        - Pod
    validate:
      message: "Unknown image registry"
      pattern:
        spec:
          containers:
          - image: "k8s.gcr.io/* | gallery.ecr.aws/*"

$ k create -f unknown-image-registry.yaml
clusterpolicy.kyverno.io/deployment-pod-valid-registry created

Docker HubからイメージをPullするPod定義で試す。

apiVersion: v1
kind: Pod
metadata:
  name: mongo
  labels:
    app: mongo
spec:
  containers:
  - name: mongo
    image: mongo:latest

拒否される。

$ k create -f mongo.yaml
Error from server: error when creating "mongo.yaml": admission webhook "validate.kyverno.svc" denied the request: 

resource Pod/default/mongo was blocked due to the following policies

deployment-pod-valid-registry:
  validate-registries: 'validation error: Unknown image registry. Rule validate-registries failed at path /spec/containers/0/image/'
disallow-privileged:
  validate-allowPrivilegeEscalation: 'validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule validate-allowPrivilegeEscalation failed at path /spec/containers/0/securityContext/'

ArgoCDを使用した自動化

ArgoCDのインストールがこけるので、ここまでに作成したポリシーを削除する。

$ k delete cpol --all
clusterpolicy.kyverno.io "deployment-pod-valid-registry" deleted
clusterpolicy.kyverno.io "disallow-privileged" deleted

ArgoCDをインストールする。

$ kubectl create namespace argocd
namespace/argocd created
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-dex-server created
serviceaccount/argocd-redis created
serviceaccount/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-dex-server created
role.rbac.authorization.k8s.io/argocd-redis created
role.rbac.authorization.k8s.io/argocd-server created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
configmap/argocd-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
secret/argocd-secret created
service/argocd-dex-server created
service/argocd-metrics created
service/argocd-redis created
service/argocd-repo-server created
service/argocd-server created
service/argocd-server-metrics created
deployment.apps/argocd-dex-server created
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
statefulset.apps/argocd-application-controller created

ArgoCDをType: LoadBalancerで外部公開する。

kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
export ARGOCD_SERVER=`kubectl get svc argocd-server -n argocd -o json | jq --raw-output .status.loadBalancer.ingress[0].hostname`

パスワードを取得する。

ARGO_PWD=`kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2`

UIにログインしてNEW APPを作成する。

f:id:sotoiwa:20210219023808p:plain

ソースにサンプルリポジトリhttps://github.com/texanraj/kyverno/tree/master/samples/core_best_practicesを入力する。その他以下のように設定してCREATEする。

f:id:sotoiwa:20210219023827p:plain

f:id:sotoiwa:20210219023848p:plain

f:id:sotoiwa:20210219023902p:plain

SYNCとSYNCHRONIZEをクリックして同期する。

f:id:sotoiwa:20210219023918p:plain

なお、自動生成部分があるため、ちゃんと同期しない。

f:id:sotoiwa:20210219023937p:plain

ポリシーを確認する。

$ kubectl get cpol
NAME                         BACKGROUND   ACTION
disallow-bind-mounts         true         audit
disallow-docker-sock-mount   true         audit
disallow-helm-tiller         true         audit
disallow-host-network-port   true         audit
disallow-new-capabilities    true         audit
disallow-privileged          true         enforce
disallow-root-user           true         enforce
disallow-sysctls             true         enforce
require-ro-rootfs            true         audit

ポリシー違反のレポートはPolicyReport/ClusterPolicyReportを確認する。

$ kubectl get cpolr
NAME                  PASS   FAIL   WARN   ERROR   SKIP   AGE
clusterpolicyreport   0      0      0      0       0      40m
$ kubectl get polr -A
NAMESPACE   NAME             PASS   FAIL   WARN   ERROR   SKIP   AGE
argocd      polr-ns-argocd   36     14     0      0       0      40m

$ kubectl describe polr -A | grep -i "status: \+fail" -B10
  Message:        validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule autogen-validate-allowPrivilegeEscalation failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         disallow-privileged
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-dex-server
    Namespace:    argocd
    UID:          b246153b-2852-46e6-bd7d-6b076bbc5734
  Rule:           autogen-validate-allowPrivilegeEscalation
  Scored:         true
  Status:         fail
--
  Message:        validation error: Root filesystem must be read-only. Rule autogen-validate-readOnlyRootFilesystem failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         require-ro-rootfs
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-repo-server
    Namespace:    argocd
    UID:          6d4581bb-3571-40f3-a5a8-83d0d4292e9b
  Rule:           autogen-validate-readOnlyRootFilesystem
  Scored:         true
  Status:         fail
--
  Message:        validation error: Root filesystem must be read-only. Rule autogen-validate-readOnlyRootFilesystem failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         require-ro-rootfs
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-dex-server
    Namespace:    argocd
    UID:          b246153b-2852-46e6-bd7d-6b076bbc5734
  Rule:           autogen-validate-readOnlyRootFilesystem
  Scored:         true
  Status:         fail
--
  Message:        validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule autogen-validate-allowPrivilegeEscalation failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         disallow-privileged
  Resources:
    API Version:  apps/v1
    Kind:         StatefulSet
    Name:         argocd-application-controller
    Namespace:    argocd
    UID:          883b6d14-ea19-4f31-b0b9-c89efed157e6
  Rule:           autogen-validate-allowPrivilegeEscalation
  Scored:         true
  Status:         fail
--
  Message:        validation error: Root filesystem must be read-only. Rule autogen-validate-readOnlyRootFilesystem failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         require-ro-rootfs
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-server
    Namespace:    argocd
    UID:          085a25b7-a84c-42bf-9098-0e27acb38486
  Rule:           autogen-validate-readOnlyRootFilesystem
  Scored:         true
  Status:         fail
--
  Message:        validation error: Root filesystem must be read-only. Rule autogen-validate-readOnlyRootFilesystem failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         require-ro-rootfs
  Resources:
    API Version:  apps/v1
    Kind:         StatefulSet
    Name:         argocd-application-controller
    Namespace:    argocd
    UID:          883b6d14-ea19-4f31-b0b9-c89efed157e6
  Rule:           autogen-validate-readOnlyRootFilesystem
  Scored:         true
  Status:         fail
--
  Message:        validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule autogen-validate-allowPrivilegeEscalation failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         disallow-privileged
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-server
    Namespace:    argocd
    UID:          085a25b7-a84c-42bf-9098-0e27acb38486
  Rule:           autogen-validate-allowPrivilegeEscalation
  Scored:         true
  Status:         fail
--
  Message:        validation error: Running as root is not allowed. Set runAsNonRoot to true, or use runAsUser. Rule autogen-validate-runAsNonRoot[0] failed at path /spec/template/spec/securityContext/runAsNonRoot/. Rule autogen-validate-runAsNonRoot[1] failed at path /spec/template/spec/securityContext/runAsUser/. Rule autogen-validate-runAsNonRoot[2] failed at path /spec/template/spec/containers/0/securityContext/. Rule autogen-validate-runAsNonRoot[3] failed at path /spec/template/spec/containers/0/securityContext/.
  Policy:         disallow-root-user
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-repo-server
    Namespace:    argocd
    UID:          6d4581bb-3571-40f3-a5a8-83d0d4292e9b
  Rule:           autogen-validate-runAsNonRoot
  Scored:         true
  Status:         fail
--
  Message:        validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule autogen-validate-allowPrivilegeEscalation failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         disallow-privileged
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-repo-server
    Namespace:    argocd
    UID:          6d4581bb-3571-40f3-a5a8-83d0d4292e9b
  Rule:           autogen-validate-allowPrivilegeEscalation
  Scored:         true
  Status:         fail
--
  Message:        validation error: Running as root is not allowed. Set runAsNonRoot to true, or use runAsUser. Rule autogen-validate-runAsNonRoot[0] failed at path /spec/template/spec/securityContext/runAsNonRoot/. Rule autogen-validate-runAsNonRoot[1] failed at path /spec/template/spec/securityContext/runAsUser/. Rule autogen-validate-runAsNonRoot[2] failed at path /spec/template/spec/containers/0/securityContext/. Rule autogen-validate-runAsNonRoot[3] failed at path /spec/template/spec/containers/0/securityContext/.
  Policy:         disallow-root-user
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-server
    Namespace:    argocd
    UID:          085a25b7-a84c-42bf-9098-0e27acb38486
  Rule:           autogen-validate-runAsNonRoot
  Scored:         true
  Status:         fail
--
  Message:        validation error: Privileged mode is not allowed. Set allowPrivilegeEscalation to false. Rule autogen-validate-allowPrivilegeEscalation failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         disallow-privileged
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-redis
    Namespace:    argocd
    UID:          3920ea11-3a4f-4e8f-9272-fc0165fc894e
  Rule:           autogen-validate-allowPrivilegeEscalation
  Scored:         true
  Status:         fail
--
  Message:        validation error: Running as root is not allowed. Set runAsNonRoot to true, or use runAsUser. Rule autogen-validate-runAsNonRoot[0] failed at path /spec/template/spec/securityContext/runAsNonRoot/. Rule autogen-validate-runAsNonRoot[1] failed at path /spec/template/spec/securityContext/runAsUser/. Rule autogen-validate-runAsNonRoot[2] failed at path /spec/template/spec/containers/0/securityContext/. Rule autogen-validate-runAsNonRoot[3] failed at path /spec/template/spec/containers/0/securityContext/.
  Policy:         disallow-root-user
  Resources:
    API Version:  apps/v1
    Kind:         StatefulSet
    Name:         argocd-application-controller
    Namespace:    argocd
    UID:          883b6d14-ea19-4f31-b0b9-c89efed157e6
  Rule:           autogen-validate-runAsNonRoot
  Scored:         true
  Status:         fail
--
  Message:        validation error: Root filesystem must be read-only. Rule autogen-validate-readOnlyRootFilesystem failed at path /spec/template/spec/containers/0/securityContext/
  Policy:         require-ro-rootfs
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-redis
    Namespace:    argocd
    UID:          3920ea11-3a4f-4e8f-9272-fc0165fc894e
  Rule:           autogen-validate-readOnlyRootFilesystem
  Scored:         true
  Status:         fail
--
  Message:        validation error: Running as root is not allowed. Set runAsNonRoot to true, or use runAsUser. Rule autogen-validate-runAsNonRoot[0] failed at path /spec/template/spec/securityContext/runAsNonRoot/. Rule autogen-validate-runAsNonRoot[1] failed at path /spec/template/spec/securityContext/runAsUser/. Rule autogen-validate-runAsNonRoot[2] failed at path /spec/template/spec/containers/0/securityContext/. Rule autogen-validate-runAsNonRoot[3] failed at path /spec/template/spec/containers/0/securityContext/.
  Policy:         disallow-root-user
  Resources:
    API Version:  apps/v1
    Kind:         Deployment
    Name:         argocd-dex-server
    Namespace:    argocd
    UID:          b246153b-2852-46e6-bd7d-6b076bbc5734
  Rule:           autogen-validate-runAsNonRoot
  Scored:         true
  Status:         fail