AWS Secrets & Configuration Providerを試す

以下のブログを試したメモ。

（6/28にSecretとのSyncまでやり直して更新）

• https://aws.amazon.com/jp/blogs/security/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/
• https://aws.amazon.com/jp/blogs/news/how-to-use-aws-secrets-configuration-provider-with-kubernetes-secrets-store-csi-driver/

準備

クラスターを作成する。

cat <<EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ascp
  region: ap-northeast-1
  version: "1.19"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    ssh:
      allow: true
      publicKeyName: default
      enableSsm: true

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true
EOF

eksctl create cluster -f cluster.yaml

前提条件を準備する。

まず、Secrets Managerでシークレットを作る。

aws secretsmanager create-secret \
  --region ap-northeast-1 \
  --name mysecret/mypasswd \
  --secret-string '{"username":"admin","password":"abcdef"}'

このシークレットにアクセスできるポリシーを作る。

cat <<EOF > mysecret-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetResourcePolicy",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": "arn:aws:secretsmanager:ap-northeast-1:XXXXXXXXXXXX:secret:mysecret*"
        }
    ]
}
EOF

aws iam create-policy --policy-name mysecret-policy --policy-document file://mysecret-policy.json

test Namespaceとtest ServiceAccountを作る。

$ k create ns test
namespace/test created
$ kubens test
Context "sotosugi@ascp.ap-northeast-1.eksctl.io" modified.
Active namespace is "test".
$ k create sa test
serviceaccount/test created

ウォークスルー

Step 1: IAM Roles for Service Accounts (IRSA) を使って Pod へのアクセスを制限する

OICDプロバイダーは作成済み。

ServiceAccount用のロールを作成する。

$ eksctl create iamserviceaccount --name test --namespace test --cluster ascp --attach-policy-arn arn:aws:iam::XXXXXXXXXXXX:policy/mysecret-policy --approve --override-existing-serviceaccounts
2021-06-27 23:24:27 [ℹ]  eksctl version 0.54.0
2021-06-27 23:24:27 [ℹ]  using region ap-northeast-1
2021-06-27 23:24:30 [ℹ]  1 existing iamserviceaccount(s) (kube-system/aws-node) will be excluded
2021-06-27 23:24:30 [ℹ]  1 iamserviceaccount (test/test) was included (based on the include/exclude rules)
2021-06-27 23:24:30 [!]  metadata of serviceaccounts that exist in Kubernetes will be updated, as --override-existing-serviceaccounts was set
2021-06-27 23:24:30 [ℹ]  1 task: { 2 sequential sub-tasks: { create IAM role for serviceaccount "test/test", create serviceaccount "test/test" } }
2021-06-27 23:24:30 [ℹ]  building iamserviceaccount stack "eksctl-ascp-addon-iamserviceaccount-test-test"
2021-06-27 23:24:30 [ℹ]  deploying stack "eksctl-ascp-addon-iamserviceaccount-test-test"
2021-06-27 23:24:30 [ℹ]  waiting for CloudFormation stack "eksctl-ascp-addon-iamserviceaccount-test-test"
2021-06-27 23:24:47 [ℹ]  waiting for CloudFormation stack "eksctl-ascp-addon-iamserviceaccount-test-test"
2021-06-27 23:25:05 [ℹ]  waiting for CloudFormation stack "eksctl-ascp-addon-iamserviceaccount-test-test"
2021-06-27 23:25:08 [ℹ]  serviceaccount "test/test" already exists
2021-06-27 23:25:08 [ℹ]  updated serviceaccount "test/test"

Step 2: Kubernetes Secrets Store CSI driver をインストールする

チャートレポジトリを追加する。

$ helm repo add secrets-store-csi-driver https://raw.githubusercontent.com/kubernetes-sigs/secrets-store-csi-driver/master/charts
"secrets-store-csi-driver" has been added to your repositories

チャートを確認する。

$ helm search repo secrets-store-csi-driver
NAME                                                    CHART VERSION   APP VERSION     DESCRIPTION                                       
secrets-store-csi-driver/secrets-store-csi-driver       0.0.23          0.0.23          A Helm chart to install the SecretsStore CSI Dr...

チャートのパラメータを確認する。

$ helm inspect values secrets-store-csi-driver/secrets-store-csi-driver
linux:
  enabled: true
  image:
    repository: k8s.gcr.io/csi-secrets-store/driver
    tag: v0.0.23
    pullPolicy: IfNotPresent

  ## Prevent the CSI driver from being scheduled on virtual-kublet nodes
  affinity: 
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: type
            operator: NotIn
            values:
            - virtual-kubelet

  driver:
    resources:
      limits:
        cpu: 200m
        memory: 200Mi
      requests:
        cpu: 50m
        memory: 100Mi

  registrarImage:
    repository: k8s.gcr.io/sig-storage/csi-node-driver-registrar
    tag: v2.2.0
    pullPolicy: IfNotPresent

  registrar:
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 10m
        memory: 20Mi
    logVerbosity: 5

  livenessProbeImage:
    repository: k8s.gcr.io/sig-storage/livenessprobe
    tag: v2.3.0
    pullPolicy: IfNotPresent

  livenessProbe:
    resources:
      limits:
        cpu: 100m
        memory: 100Mi
      requests:
        cpu: 10m
        memory: 20Mi


  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

  kubeletRootDir: /var/lib/kubelet
  providersDir: /etc/kubernetes/secrets-store-csi-providers
  nodeSelector: {}
  tolerations: []
  metricsAddr: ":8095"
  env: []
  priorityClassName: ""
  daemonsetAnnotations: {}
  podAnnotations: {}
  podLabels: {}

  # volumes is a list of volumes made available to secrets store csi driver.
  volumes: null
  #   - name: foo
  #     emptyDir: {}

  # volumeMounts is a list of volumeMounts for secrets store csi driver.
  volumeMounts: null
  #   - name: foo
  #     mountPath: /bar
  #     readOnly: true

windows:
  enabled: false
  image:
    repository: k8s.gcr.io/csi-secrets-store/driver
    tag: v0.0.23
    pullPolicy: IfNotPresent

  ## Prevent the CSI driver from being scheduled on virtual-kublet nodes
  affinity: 
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: type
            operator: NotIn
            values:
            - virtual-kubelet

  driver:
    resources:
      limits:
        cpu: 400m
        memory: 400Mi
      requests:
        cpu: 50m
        memory: 100Mi

  registrarImage:
    repository: k8s.gcr.io/sig-storage/csi-node-driver-registrar
    tag: v2.2.0
    pullPolicy: IfNotPresent

  registrar:
    resources:
      limits:
        cpu: 200m
        memory: 200Mi
      requests:
        cpu: 10m
        memory: 20Mi
    logVerbosity: 5

  livenessProbeImage:
    repository: k8s.gcr.io/sig-storage/livenessprobe
    tag: v2.3.0
    pullPolicy: IfNotPresent

  livenessProbe:
    resources:
      limits:
        cpu: 200m
        memory: 200Mi
      requests:
        cpu: 10m
        memory: 20Mi

  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1

  kubeletRootDir: C:\var\lib\kubelet
  providersDir: C:\k\secrets-store-csi-providers
  nodeSelector: {}
  tolerations: []
  metricsAddr: ":8095"
  env: []
  priorityClassName: ""
  daemonsetAnnotations: {}
  podAnnotations: {}
  podLabels: {}

  # volumes is a list of volumes made available to secrets store csi driver.
  volumes: null
  #   - name: foo
  #     emptyDir: {}

  # volumeMounts is a list of volumeMounts for secrets store csi driver.
  volumeMounts: null
  #   - name: foo
  #     mountPath: /bar
  #     readOnly: true

# log level. Uses V logs (klog)
logVerbosity: 0

# logging format JSON
logFormatJSON: false

livenessProbe:
  port: 9808
  logLevel: 2

## Maximum size in bytes of gRPC response from plugins
maxCallRecvMsgSize: 4194304

## Install Default RBAC roles and bindings
rbac:
  install: true
  pspEnabled: false

## Install RBAC roles and bindings required for K8S Secrets syncing if true
syncSecret:
  enabled: false

## Enable secret rotation feature [alpha]
enableSecretRotation: false

## Secret rotation poll interval duration
rotationPollInterval:

## Filtered watch nodePublishSecretRef secrets
filteredWatchSecret: false

## Provider HealthCheck
providerHealthCheck: false

## Provider HealthCheck interval
providerHealthCheckInterval: 2m

imagePullSecrets: []

デフォルト設定でデプロイする。

$ helm -n kube-system install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver
NAME: csi-secrets-store
LAST DEPLOYED: Sun Jun 27 23:26:47 2021
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Secrets Store CSI Driver is getting deployed to your cluster.

To verify that Secrets Store CSI Driver has started, run:

  kubectl --namespace=kube-system get pods -l "app=secrets-store-csi-driver"

Now you can follow these steps https://secrets-store-csi-driver.sigs.k8s.io/getting-started/usage.html
to create a SecretProviderClass resource, and a deployment using the SecretProviderClass.

インストールを確認する。DaemonSetが動いている。

$ kubectl get po --namespace=kube-system
NAME                                               READY   STATUS    RESTARTS   AGE
aws-node-7p247                                     1/1     Running   0          37m
aws-node-w6sq5                                     1/1     Running   0          37m
coredns-59847d77c8-fjwsx                           1/1     Running   0          51m
coredns-59847d77c8-qhlj5                           1/1     Running   0          51m
csi-secrets-store-secrets-store-csi-driver-9867x   3/3     Running   0          17s
csi-secrets-store-secrets-store-csi-driver-fjhl9   3/3     Running   0          17s
kube-proxy-ppq4b                                   1/1     Running   0          37m
kube-proxy-wkv2h                                   1/1     Running   0          37m

CRDを確認する。

$ kubectl get crd
NAME                                                        CREATED AT
eniconfigs.crd.k8s.amazonaws.com                            2021-06-27T13:35:51Z
secretproviderclasses.secrets-store.csi.x-k8s.io            2021-06-27T14:26:48Z
secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io   2021-06-27T14:26:48Z
securitygrouppolicies.vpcresources.k8s.aws                  2021-06-27T13:35:54Z

Step 3: AWS Secrets & Configuration Provider をインストールします

AWS Secrets & Configuration Provider をインストールする。

$ kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
serviceaccount/csi-secrets-store-provider-aws created
clusterrole.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-secrets-store-provider-aws-cluster-rolebinding created
daemonset.apps/csi-secrets-store-provider-aws created

確認する。

$ k get pod -n kube-system
NAME                                               READY   STATUS    RESTARTS   AGE
aws-node-7p247                                     1/1     Running   0          38m
aws-node-w6sq5                                     1/1     Running   0          38m
coredns-59847d77c8-fjwsx                           1/1     Running   0          52m
coredns-59847d77c8-qhlj5                           1/1     Running   0          52m
csi-secrets-store-provider-aws-6n65t               1/1     Running   0          13s
csi-secrets-store-provider-aws-nrn2g               1/1     Running   0          13s
csi-secrets-store-secrets-store-csi-driver-9867x   3/3     Running   0          79s
csi-secrets-store-secrets-store-csi-driver-fjhl9   3/3     Running   0          79s
kube-proxy-ppq4b                                   1/1     Running   0          38m
kube-proxy-wkv2h                                   1/1     Running   0          38m

Step 4: SecretProviderClass カスタムリソースを作成してデプロイする

SecretProviderClass カスタムリソースを作成する。

cat <<EOF > aws-secrets.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets
spec:
  provider: aws
  parameters:                    # provider-specific parameters
    objects:  |
      - objectName: "mysecret/mypasswd"
        objectType: "secretsmanager"
EOF

$ k apply -f aws-secrets.yaml
secretproviderclass.secrets-store.csi.x-k8s.io/aws-secrets created

Step 5: 構成されたシークレットに基づいてボリュームをマウントするように Pod を構成してデプロイする

cat <<EOF > nginx-secrets-store-inline.yaml
kind: Pod
apiVersion: v1
metadata:
  name: nginx-secrets-store-inline
spec:
  serviceAccountName: test
  containers:
  - image: nginx
    name: nginx
    volumeMounts:
    - name: mysecret
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
  - name: mysecret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "aws-secrets"
EOF

$ k apply -f nginx-secrets-store-inline.yaml
pod/nginx-secrets-store-inline created

マウントされたことを確認する。

$ kubectl exec -it nginx-secrets-store-inline -- ls /mnt/secrets-store/
mysecret_mypasswd
$ kubectl exec -it nginx-secrets-store-inline -- cat /mnt/secrets-store/mysecret_mypasswd
{"username":"admin","password":"abcdef"}

その他

Secretと同期させたりもできるようなので確認する。インストール時に有効にする必要がある。念のため全部インストールし直す。

helm -n kube-system delete csi-secrets-store
kubectl delete -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml
helm -n kube-system upgrade --install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver \
  --set syncSecret.enabled=true
kubectl apply -f https://raw.githubusercontent.com/aws/secrets-store-csi-driver-provider-aws/main/deployment/aws-provider-installer.yaml

ExternalSecretsのようにExternalSecretリソースを作るとSecretリソースが作られるわけではなく、Podがボリュームマウントする際にSecretリソースにも同期するという動作になる。

cat <<EOF > aws-secrets-sync.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: aws-secrets-sync
spec:
  provider: aws
  secretObjects:
  - secretName: mysecret
    type: Opaque
    data:
    - objectName: myalias
      key: mykey
  parameters:
    objects: |
      - objectName: "mysecret/mypasswd"
        objectAlias: myalias
        objectType: "secretsmanager"
EOF

$ k apply -f aws-secrets-sync.yaml
secretproviderclass.secrets-store.csi.x-k8s.io/aws-secrets-sync created

cat <<EOF > nginx-secrets-store-sync.yaml
kind: Pod
apiVersion: v1
metadata:
  name: nginx-secrets-store-sync
spec:
  serviceAccountName: test
  containers:
  - image: nginx
    name: nginx
    env:
    - name: MYSECRET
      valueFrom:
        secretKeyRef:
          name: mysecret
          key: mykey
    volumeMounts:
    - name: mysecret
      mountPath: "/mnt/secrets-store"
      readOnly: true
  volumes:
  - name: mysecret
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: "aws-secrets-sync"
EOF

$ k apply -f nginx-secrets-store-sync.yaml
pod/nginx-secrets-store-sync created

確認する。

$ k get po
NAME                         READY   STATUS    RESTARTS   AGE
nginx-secrets-store-inline   1/1     Running   0          5m43s
nginx-secrets-store-sync     1/1     Running   0          5s
$ k exec -it nginx-secrets-store-sync -- env | grep MYSECRET
MYSECRET={"username":"admin","password":"abcdef"}
$ k get secret
NAME                  TYPE                                  DATA   AGE
default-token-8dd6x   kubernetes.io/service-account-token   3      119m
mysecret              Opaque                                1      78s
test-token-45fqm      kubernetes.io/service-account-token   3      119m
$ k get secret mysecret -o yaml
apiVersion: v1
data:
  mykey: eyJ1c2VybmFtZSI6ImFkbWluIiwicGFzc3dvcmQiOiJhYmNkZWYifQ==
kind: Secret
metadata:
  creationTimestamp: "2021-06-27T16:21:39Z"
  labels:
    secrets-store.csi.k8s.io/managed: "true"
  name: mysecret
  namespace: test
  ownerReferences:
  - apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
    kind: SecretProviderClassPodStatus
    name: nginx-secrets-store-sync-test-aws-secrets-sync
    uid: bec8e05c-1dab-46af-9784-cd7884264986
  resourceVersion: "28885"
  selfLink: /api/v1/namespaces/test/secrets/mysecret
  uid: a978b70a-7a5e-43bb-bb24-bb138ea18659
type: Opaque
$ k get secret mysecret -o json | jq -r '.data.mykey' | base64 --decode
{"username":"admin","password":"abcdef"}

Podを消すとSecretも消える。

$ k delete po --all
pod "nginx-secrets-store-inline" deleted
pod "nginx-secrets-store-sync" deleted
$ k get secret
NAME                  TYPE                                  DATA   AGE
default-token-8dd6x   kubernetes.io/service-account-token   3      134m
test-token-45fqm      kubernetes.io/service-account-token   3      133m