kubeadmクラスターでauditログを有効化するメモ。
参考リンク
- https://kubernetes.io/docs/tasks/debug-application-cluster/audit/
- https://github.com/killer-sh/cks-course-environment/tree/master/course-content/runtime-security/auditing
手順
ディレクトリを作成する。
mkdir -p /etc/kubernetes/audit cd /etc/kubernetes/audit
ポリシーファイルを作成する。
vi policy.yaml
Metadataレベルで全てをロギングする例。
apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
kube-apiserverのマニフェストを変更する。
（省略） spec: containers: - command: - kube-apiserver - --audit-policy-file=/etc/kubernetes/audit/policy.yaml - --audit-log-path=/var/log/audit.log （省略） volumeMounts: - mountPath: /etc/kubernetes/audit name: k8s-audit readOnly: true - mountPath: /var/log/audit.log name: k8s-audit-log readOnly: false （省略） volumes: - hostPath: path: /etc/kubernetes/audit type: DirectoryOrCreate name: k8s-audit - hostPath: path: /var/log/audit.log type: FileOrCreate name: k8s-audit-log （省略）
確認
Secretを作成してログを確認する。createの場合はResponseCompleteのステージしかない？
root@cks-master:/etc/kubernetes/manifests# k create secret generic very-secure --from-literal=user=admin
secret/very-secure created
root@cks-master:/etc/kubernetes/manifests# cat /var/log/audit.log | grep very-secure
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5094c9ec-69bb-4d76-be4e-021287d2c370","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets?fieldManager=kubectl-create","verb":"create","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.146.0.6"],"userAgent":"kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a","objectRef":{"resource":"secrets","namespace":"default","name":"very-secure","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2021-01-03T21:15:42.512381Z","stageTimestamp":"2021-01-03T21:15:42.519040Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
root@cks-master:/etc/kubernetes/manifests#
jqで整形してみると以下のような形式。
{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "5094c9ec-69bb-4d76-be4e-021287d2c370", "stage": "ResponseComplete", "requestURI": "/api/v1/namespaces/default/secrets?fieldManager=kubectl-create", "verb": "create", "user": { "username": "kubernetes-admin", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "10.146.0.6" ], "userAgent": "kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a", "objectRef": { "resource": "secrets", "namespace": "default", "name": "very-secure", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 201 }, "requestReceivedTimestamp": "2021-01-03T21:15:42.512381Z", "stageTimestamp": "2021-01-03T21:15:42.519040Z", "annotations": { "authorization.k8s.io/decision": "allow", "authorization.k8s.io/reason": "" } }
Secretを適当にeditしてみる。getのRequestReceivedとResponseComplete、patchのRequestReceivedとResponseCompleteが記録されている。
root@cks-master:/etc/kubernetes/manifests# k edit secret very-secure
secret/very-secure edited
root@cks-master:/etc/kubernetes/manifests# cat /var/log/audit.log | grep very-secure
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5094c9ec-69bb-4d76-be4e-021287d2c370","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets?fieldManager=kubectl-create","verb":"create","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.146.0.6"],"userAgent":"kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a","objectRef":{"resource":"secrets","namespace":"default","name":"very-secure","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2021-01-03T21:15:42.512381Z","stageTimestamp":"2021-01-03T21:15:42.519040Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"cec16d21-a336-473f-9c61-aa56acfad7d0","stage":"RequestReceived","requestURI":"/api/v1/namespaces/default/secrets/very-secure","verb":"get","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.146.0.6"],"userAgent":"kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a","objectRef":{"resource":"secrets","namespace":"default","name":"very-secure","apiVersion":"v1"},"requestReceivedTimestamp":"2021-01-03T21:20:20.876437Z","stageTimestamp":"2021-01-03T21:20:20.876437Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"cec16d21-a336-473f-9c61-aa56acfad7d0","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets/very-secure","verb":"get","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.146.0.6"],"userAgent":"kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a","objectRef":{"resource":"secrets","namespace":"default","name":"very-secure","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2021-01-03T21:20:20.876437Z","stageTimestamp":"2021-01-03T21:20:20.878489Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"80f4192b-0533-43d9-90a3-e4f33aaee2c5","stage":"RequestReceived","requestURI":"/api/v1/namespaces/default/secrets/very-secure?fieldManager=kubectl-edit","verb":"patch","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.146.0.6"],"userAgent":"kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a","objectRef":{"resource":"secrets","namespace":"default","name":"very-secure","apiVersion":"v1"},"requestReceivedTimestamp":"2021-01-03T21:20:33.854711Z","stageTimestamp":"2021-01-03T21:20:33.854711Z"}
{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"80f4192b-0533-43d9-90a3-e4f33aaee2c5","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/default/secrets/very-secure?fieldManager=kubectl-edit","verb":"patch","user":{"username":"kubernetes-admin","groups":["system:masters","system:authenticated"]},"sourceIPs":["10.146.0.6"],"userAgent":"kubectl/v1.19.3 (linux/amd64) kubernetes/1e11e4a","objectRef":{"resource":"secrets","namespace":"default","name":"very-secure","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2021-01-03T21:20:33.854711Z","stageTimestamp":"2021-01-03T21:20:33.860414Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":""}}
root@cks-master:/etc/kubernetes/manifests#
ポリシーのカスタマイズ
以下のルールでポリシーを作成する。
- RequestReceivedステージのログは記録しない
- get, watch, listのログは記録しない
- SecretについてはMetadataレベルで記録する
- その他はRequestResponseレベルで記録する
apiVersion: audit.k8s.io/v1 # This is required. kind: Policy omitStages: - "RequestReceived" rules: - level: None verbs: ["get", "watch", "list"] - level: Metadata resources: - group: "" # core API group resources: ["secrets"] - level: RequestResponse
