kubeadmで作成したクラスターでetcdでのSecretの暗号化を有効にする方法のメモ。
デフォルト状態
デフォルト状態ではSecretは暗号化されていないので、etcdctlを使ってetcdに直接アクセスすれば見ることができる。
Secretを作成する。
root@cks-master:~# k create secret generic secret1 --from-literal=user=admin secret/secret1 created
root@cks-master:~# k get secret secret1 -o yaml
apiVersion: v1
data:
user: YWRtaW4=
kind: Secret
metadata:
creationTimestamp: "2020-12-28T16:29:53Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:user: {}
f:type: {}
manager: kubectl-create
operation: Update
time: "2020-12-28T16:29:53Z"
name: secret1
namespace: default
resourceVersion: "2187"
selfLink: /api/v1/namespaces/default/secrets/secret1
uid: 5f47402f-e1eb-44f8-a672-c4a56806b626
type: Opaque
etcdctlをインストールする。
apt-get install -y etcd-client
kube-apiserverがetcdに接続している情報を確認する。
root@cks-master:~# cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep etcd
- --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
- --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
- --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
- --etcd-servers=https://127.0.0.1:2379
この情報を使ってetcdに接続する。
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--key=/etc/kubernetes/pki/apiserver-etcd-client.key \
--endpoints=https://127.0.0.1:2379 \
endpoint health
root@cks-master:~# ETCDCTL_API=3 etcdctl \ > --cacert=/etc/kubernetes/pki/etcd/ca.crt \ > --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \ > --key=/etc/kubernetes/pki/apiserver-etcd-client.key \ > --endpoints=https://127.0.0.1:2379 \ > endpoint health https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 1.523763ms
Secretを読み取る。
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--key=/etc/kubernetes/pki/apiserver-etcd-client.key \
--endpoints=https://127.0.0.1:2379 \
get /registry/secrets/default/secret1
root@cks-master:/var/log/containers# ETCDCTL_API=3 etcdctl \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
> --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
> --endpoints=https://127.0.0.1:2379 \
> get /registry/secrets/default/secret1
/registry/secrets/default/secret1
k8s
v1Secret�
�
secret1default"*$5f47402f-e1eb-44f8-a672-c4a56806b6262����z�_
kubectl-createUpdatev����FieldsV1:-
+{"f:data":{".":{},"f:user":{}},"f:type":{}}
useradminOpaque"
暗号化
設定ファイルを置くディレクトリを作成する。
mkdir -p /etc/kubernetes/etcd cd /etc/kubernetes/etcd
暗号化に使うキーを作成する。
# KEY=$(echo -n passwordpassword | base64) KEY=$(head -c 32 /dev/urandom | base64)
EncryptionConfigurationのyamlを作成する。
cat <<EOF > /etc/kubernetes/etcd/ec.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
- secrets
providers:
- aescbc:
keys:
- name: key1
secret: ${KEY}
- identity: {}
EOF
kube-apiserverの起動引数でこのyamlを渡す。ファイルをマウントしてあげる必要もあることに注意。
vi /etc/kubernetes/manifests/kube-apiserver.yaml
（省略）
- command:
- kube-apiserver
- --encryption-provider-config=/etc/kubernetes/etcd/ec.yaml
（省略）
volumeMounts:
- mountPath: /etc/kubernetes/etcd
name: etcd
readOnly: true
（省略）
volumes:
- hostPath:
path: /etc/kubernetes/etcd
type: DirectoryOrCreate
name: etcd
既存のSecretは暗号化されない。暗号化時は一番上のプロバイダーが使われ、復号時は上から順番に使われる。一番下にidentity: {}の設定があるので、暗号化されていないものも読み取りが可能。
root@cks-master:/var/log/containers# ETCDCTL_API=3 etcdctl \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
> --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
> --endpoints=https://127.0.0.1:2379 \
> get /registry/secrets/default/secret1
/registry/secrets/default/secret1
k8s
v1Secret�
�
secret1default"*$5f47402f-e1eb-44f8-a672-c4a56806b6262����z�_
kubectl-createUpdatev����FieldsV1:-
+{"f:data":{".":{},"f:user":{}},"f:type":{}}
useradminOpaque"
新しくSecretを作成する。
root@cks-master:/var/log/containers# k create secret generic very-secure --from-literal=cc=1234 secret/very-secure created
root@cks-master:/var/log/containers# k get secret very-secure -o yaml
apiVersion: v1
data:
cc: MTIzNA==
kind: Secret
metadata:
creationTimestamp: "2020-12-28T17:49:20Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:cc: {}
f:type: {}
manager: kubectl-create
operation: Update
time: "2020-12-28T17:49:20Z"
name: very-secure
namespace: default
resourceVersion: "12256"
selfLink: /api/v1/namespaces/default/secrets/very-secure
uid: 1883c04d-0e66-4179-816d-b9401174c125
type: Opaque
etcdctlで直接見てみる。
ETCDCTL_API=3 etcdctl \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
--key=/etc/kubernetes/pki/apiserver-etcd-client.key \
--endpoints=https://127.0.0.1:2379 \
get /registry/secrets/default/very-secure
root@cks-master:/var/log/containers# ETCDCTL_API=3 etcdctl \
> --cacert=/etc/kubernetes/pki/etcd/ca.crt \
> --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
> --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
> --endpoints=https://127.0.0.1:2379 \
> get /registry/secrets/default/very-secure
/registry/secrets/default/very-secure
k8s:enc:aescbc:v1:key1:�1� �l�Tk.8�谶
���[A�d��h�U0m9���J�(v2��Э|Ь�����u�����"�pe���oؖA&�7���^'�Us�
P��%͖gOO/jJ�ʰif'��!��)K���M��
j�ܿN;e��j������ �w��}��}�r�E?���~s� ��R�N��p�L��,
�ƙZ���v�+B�ק��涻�(�Y�W*ݭ?YN/&�K{��`��?�Ktz�ď�����I�
既存のSecretの暗号化
全部読み取って、置き換える。一番下のidentity: {}で暗号化されていないものも読み取られて、保存されるときには一番上の設定で暗号化される。
k get secret -A -o yaml | k replace -f -
root@cks-master:/etc/kubernetes/manifests# k get secret -A -o yaml | k replace -f - secret/default-token-grb2j replaced secret/secret1 replaced secret/secret2 replaced secret/very-secure replaced secret/default-token-hbjtd replaced secret/default-token-7n9bx replaced secret/attachdetach-controller-token-vndrq replaced secret/bootstrap-signer-token-7pfcp replaced secret/bootstrap-token-jsf0sg replaced secret/bootstrap-token-xmh1o0 replaced secret/certificate-controller-token-8vfv8 replaced secret/clusterrole-aggregation-controller-token-h2tnb replaced secret/coredns-token-khz9t replaced secret/cronjob-controller-token-ntwtd replaced secret/daemon-set-controller-token-vd4dj replaced secret/default-token-8glxq replaced secret/deployment-controller-token-hxbfp replaced secret/disruption-controller-token-jhq8d replaced secret/endpoint-controller-token-69pxz replaced secret/endpointslice-controller-token-xp5gr replaced secret/endpointslicemirroring-controller-token-9dnsj replaced secret/expand-controller-token-5vwd2 replaced secret/generic-garbage-collector-token-qgt5j replaced secret/horizontal-pod-autoscaler-token-ghvpj replaced secret/job-controller-token-sb5dp replaced secret/kube-proxy-token-nzcmr replaced secret/namespace-controller-token-cl54l replaced secret/node-controller-token-vmhjm replaced secret/persistent-volume-binder-token-6fv9z replaced secret/pod-garbage-collector-token-25dbr replaced secret/pv-protection-controller-token-n9p4z replaced secret/pvc-protection-controller-token-qw9d7 replaced secret/replicaset-controller-token-52drf replaced secret/replication-controller-token-5fzp8 replaced secret/resourcequota-controller-token-f2vfc replaced secret/service-account-controller-token-dqrsn replaced secret/service-controller-token-j8w9n replaced secret/statefulset-controller-token-5twbl replaced secret/token-cleaner-token-2q8jc replaced secret/ttl-controller-token-4w56d replaced secret/weave-net-token-vk5lm replaced
全部暗号化できたら、identity: {}は消してもOK。
