kube-benchを使ってEKSのCISベンチマークを実行してみたメモ。EKS Workshopにトピックがあるのでそちらにしたがって実施する。
| コンポーネント | バージョン | 備考 |
|---|---|---|
| eksctl | 0.27.0 | |
| Kubernetes バージョン | 1.17 | |
| プラットフォームのバージョン | eks.3 | |
| kube-bench | 0.3.1 |
参考情報
- Amazon EKS の CIS ベンチマークを発表
- Introducing The CIS Amazon EKS Benchmark
- Amazon EKS Best Practices Guide for Security
- CIS EKS Benchmark assessment using kube-bench
- aquasecurity/kube-bench
準備
クラスターを作成する。
eksctl create cluster \ --name=mycluster \ --version 1.17 \ --nodes=2 --managed \ --ssh-access --ssh-public-key=default
ベンチマークの実行
ベンチマークはノードでツールを実行するか、Kubernetes Jobとして実行することができる。
- コントロールプレーンのログ記録設定（Section 2）
- ノードのセキュリティ設定（Section 3）
- ポリシー（Section 4）
- マネージドサービス（Section 5）
の 4 つのセクションで構成されているが、ノード上でツールを実行する場合は、ノード（Section 3）が対象となり、Jobとして実行した場合は明示的にnodeを指定しなければポリシー（Section 4）とマネージドサービス（Section 5）もチェックも実行されているが、ただし、Not Scoredとなっており実際にはチェックは実行されていない。EKS Workshopに記載の出力とは異なるのでバグかも知れない。
ノード上で実行
ノードにsshでログインする。
rpmをダウンロードしてインストールする。
wget https://github.com/aquasecurity/kube-bench/releases/download/v0.3.1/kube-bench_0.3.1_linux_amd64.rpm
アセスメントを実行する。
kube-bench --benchmark "eks-1.0"
以下のように出力される。
[ec2-user@ip-192-168-41-143 ~]$ kube-bench --benchmark "eks-1.0" [INFO] 3 Worker Node Security Configuration [INFO] 3.1 Worker Node Configuration Files [PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored) [PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored) [PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) [PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored) [INFO] 3.2 Kubelet [PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored) [PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored) [PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored) [PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored) [PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored) [PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored) [PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored) [PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored) [WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored) [PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored) [PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) == Remediations == 3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service == Summary == 14 checks PASS 0 checks FAIL 1 checks WARN 0 checks INFO [ec2-user@ip-192-168-41-143 ~]$
デフォルトでほぼPASSしている。WARNとなっている1件については以下のプルリクエストがあるがマージされていない。
kube-benchをアンインストールする。
sudo yum remove kube-bench -y
Jobとして実行
DockerHubのイメージで実行
Jobのマニフェストを作成する。
cat << EOF > job-eks.yaml --- apiVersion: batch/v1 kind: Job metadata: name: kube-bench spec: template: spec: hostPID: true containers: - name: kube-bench image: aquasec/kube-bench:latest command: ["kube-bench", "--benchmark", "eks-1.0"] volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet readOnly: true - name: etc-systemd mountPath: /etc/systemd readOnly: true - name: etc-kubernetes mountPath: /etc/kubernetes readOnly: true restartPolicy: Never volumes: - name: var-lib-kubelet hostPath: path: "/var/lib/kubelet" - name: etc-systemd hostPath: path: "/etc/systemd" - name: etc-kubernetes hostPath: path: "/etc/kubernetes" EOF
Jobを作成する。
kubectl apply -f job-eks.yaml
ログを確認する。
$ kubectl get pod NAME READY STATUS RESTARTS AGE kube-bench-qf7j9 0/1 Completed 0 20s $ kubectl logs kube-bench-qf7j9 [INFO] 3 Worker Node Security Configuration [INFO] 3.1 Worker Node Configuration Files [PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored) [PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored) [PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) [PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored) [INFO] 3.2 Kubelet [PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored) [PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored) [PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored) [PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored) [PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored) [PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored) [PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored) [PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored) [WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored) [PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored) [PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) == Remediations == 3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service == Summary == 14 checks PASS 0 checks FAIL 1 checks WARN 0 checks INFO [INFO] 4 Policies [INFO] 4.1 RBAC and Service Accounts [WARN] 4.1.1 Ensure that the cluster-admin role is only used where required (Not Scored) [WARN] 4.1.2 Minimize access to secrets (Not Scored) [WARN] 4.1.3 Minimize wildcard use in Roles and ClusterRoles (Not Scored) [WARN] 4.1.4 Minimize access to create pods (Not Scored) [WARN] 4.1.5 Ensure that default service accounts are not actively used. (Not Scored) [WARN] 4.1.6 Ensure that Service Account Tokens are only mounted where necessary (Not Scored) [INFO] 4.2 Pod Security Policies [WARN] 4.2.1 Minimize the admission of privileged containers (Not Scored) [WARN] 4.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Not Scored) [WARN] 4.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Not Scored) [WARN] 4.2.4 Minimize the admission of containers wishing to share the host network namespace (Not Scored) [WARN] 4.2.5 Minimize the admission of containers with allowPrivilegeEscalation (Not Scored) [WARN] 4.2.6 Minimize the admission of root containers (Not Scored) [WARN] 4.2.7 Minimize the admission of containers with the NET_RAW capability (Not Scored) [WARN] 4.2.8 Minimize the admission of containers with added capabilities (Not Scored) [WARN] 4.2.9 Minimize the admission of containers with capabilities assigned (Not Scored) [INFO] 4.3 CNI Plugin [WARN] 4.3.1 Ensure that the latest CNI version is used (Not Scored) [WARN] 4.3.2 Ensure that all Namespaces have Network Policies defined (Not Scored) [INFO] 4.4 Secrets Management [WARN] 4.4.1 Prefer using secrets as files over secrets as environment variables (Not Scored) [WARN] 4.4.2 Consider external secret storage (Not Scored) [INFO] 4.5 Extensible Admission Control [WARN] 4.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Not Scored) [INFO] 4.6 General Policies [WARN] 4.6.1 Create administrative boundaries between resources using namespaces (Not Scored) [WARN] 4.6.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Not Scored) [WARN] 4.6.3 Apply Security Context to Your Pods and Containers (Not Scored) [WARN] 4.6.4 The default namespace should not be used (Not Scored) == Remediations == 4.1.1 Identify all clusterrolebindings to the cluster-admin role. Check if they are used and if they need this role or if they could use a role with fewer privileges. Where possible, first bind users to a lower privileged role and then remove the clusterrolebinding to the cluster-admin role : kubectl delete clusterrolebinding [name] 4.1.2 Where possible, remove get, list and watch access to secret objects in the cluster. 4.1.3 Where possible replace any use of wildcards in clusterroles and roles with specific objects or actions. 4.1.4 4.1.5 Create explicit service accounts wherever a Kubernetes workload requires specific access to the Kubernetes API server. Modify the configuration of each default service account to include this value automountServiceAccountToken: false 4.1.6 Modify the definition of pods and service accounts which do not need to mount service account tokens to disable it. 4.2.1 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.privileged field is omitted or set to false. 4.2.2 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostPID field is omitted or set to false. 4.2.3 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostIPC field is omitted or set to false. 4.2.4 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostNetwork field is omitted or set to false. 4.2.5 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.allowPrivilegeEscalation field is omitted or set to false. 4.2.6 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.runAsUser.rule is set to either MustRunAsNonRoot or MustRunAs with the range of UIDs not including 0. 4.2.7 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.requiredDropCapabilities is set to include either NET_RAW or ALL. 4.2.8 Ensure that allowedCapabilities is not present in PSPs for the cluster unless it is set to an empty array. 4.2.9 Review the use of capabilites in applications runnning on your cluster. Where a namespace contains applications which do not require any Linux capabities to operate consider adding a PSP which forbids the admission of containers which do not drop all capabilities. 4.3.1 Review the documentation of AWS CNI plugin, and ensure latest CNI version is used. 4.3.2 Follow the documentation and create NetworkPolicy objects as you need them. 4.4.1 If possible, rewrite application code to read secrets from mounted secret files, rather than from environment variables. 4.4.2 Refer to the secrets management options offered by your cloud provider or a third-party secrets management solution. 4.5.1 Follow the Kubernetes documentation and setup image provenance. 4.6.1 Follow the documentation and create namespaces for objects in your deployment as you need them. 4.6.2 Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you would need to enable alpha features in the apiserver by passing "--feature- gates=AllAlpha=true" argument. Edit the /etc/kubernetes/apiserver file on the master node and set the KUBE_API_ARGS parameter to "--feature-gates=AllAlpha=true" KUBE_API_ARGS="--feature-gates=AllAlpha=true" Based on your system, restart the kube-apiserver service. For example: systemctl restart kube-apiserver.service Use annotations to enable the docker/default seccomp profile in your pod definitions. An example is as below: apiVersion: v1 kind: Pod metadata: name: trustworthy-pod annotations: seccomp.security.alpha.kubernetes.io/pod: docker/default spec: containers: - name: trustworthy-container image: sotrustworthy:latest 4.6.3 Follow the Kubernetes documentation and apply security contexts to your pods. For a suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker Containers. 4.6.4 Ensure that namespaces are created to allow for appropriate segregation of Kubernetes resources and that all new resources are created in a specific namespace. == Summary == 0 checks PASS 0 checks FAIL 24 checks WARN 0 checks INFO [INFO] 5 Managed Services [INFO] 5.1 Image Registry and Image Scanning [WARN] 5.1.1 Ensure Image Vulnerability Scanning using Amazon ECR image scanning or a third-party provider (Not Scored) [WARN] 5.1.2 Minimize user access to Amazon ECR (Not Scored) [WARN] 5.1.3 Minimize cluster access to read-only for Amazon ECR (Not Scored) [WARN] 5.1.4 Minimize Container Registries to only those approved (Not Scored) [INFO] 5.2 Identity and Access Management (IAM) [WARN] 5.2.1 Prefer using dedicated Amazon EKS Service Accounts (Not Scored) [INFO] 5.3 AWS Key Management Service (AWS KMS) [WARN] 5.3.1 Ensure Kubernetes Secrets are encrypted using Customer Master Keys (CMKs) managed in AWS KMS (Not Scored) [INFO] 5.4 Cluster Networking [WARN] 5.4.1 Restrict Access to the Control Plane Endpoint (Not Scored) [WARN] 5.4.2 Ensure clusters are created with Private Endpoint Enabled and Public Access Disabled (Not Scored) [WARN] 5.4.3 Ensure clusters are created with Private Nodes (Not Scored) [WARN] 5.4.4 Ensure Network Policy is Enabled and set as appropriate (Not Scored) [WARN] 5.4.5 Encrypt traffic to HTTPS load balancers with TLS certificates (Not Scored) [INFO] 5.5 Authentication and Authorization [WARN] 5.5.1 Manage Kubernetes RBAC users with AWS IAM Authenticator for Kubernetes (Not Scored) [INFO] 5.6 Other Cluster Configurations [WARN] 5.6.1 Consider Fargate for running untrusted workloads (Not Scored) == Remediations == 5.1.1 5.1.2 5.1.3 5.1.4 5.2.1 5.3.1 5.4.1 5.4.2 5.4.3 5.4.4 5.4.5 5.5.1 5.6.1 == Summary == 0 checks PASS 0 checks FAIL 13 checks WARN 0 checks INFO $
Section 4と5もチェックされているが、Not Scoredなので実際にはチェックされていない。ノードのみを対象とする場合は、このjob-eks.yamlの例のように、command: ["kube-bench", "node", "--benchmark", "eks-1.0"]としてnodeに限定すればよさそう。
Jobを削除する。
kubectl delete -f job-eks.yaml
nodeに限定してやってみる。マニフェストを作成する。
cat << EOF > job-eks-node.yaml --- apiVersion: batch/v1 kind: Job metadata: name: kube-bench-node spec: template: spec: hostPID: true containers: - name: kube-bench image: aquasec/kube-bench:latest command: ["kube-bench", "node", "--benchmark", "eks-1.0"] volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet readOnly: true - name: etc-systemd mountPath: /etc/systemd readOnly: true - name: etc-kubernetes mountPath: /etc/kubernetes readOnly: true restartPolicy: Never volumes: - name: var-lib-kubelet hostPath: path: "/var/lib/kubelet" - name: etc-systemd hostPath: path: "/etc/systemd" - name: etc-kubernetes hostPath: path: "/etc/kubernetes" EOF
ジョブを作成する。
kubectl apply -f job-eks-node.yaml
ログを確認する。
$ kubectl get pod NAME READY STATUS RESTARTS AGE kube-bench-node-qc9wb 0/1 Completed 0 9s $ kubectl logs kube-bench-node-qc9wb [INFO] 3 Worker Node Security Configuration [INFO] 3.1 Worker Node Configuration Files [PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored) [PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored) [PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) [PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored) [INFO] 3.2 Kubelet [PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored) [PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored) [PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored) [PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored) [PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored) [PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored) [PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored) [PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored) [WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored) [PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored) [PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) == Remediations == 3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service == Summary == 14 checks PASS 0 checks FAIL 1 checks WARN 0 checks INFO $
Section 3に限定された。
ジョブを削除する。
kubectl delete -f job-eks-node.yaml
イメージをビルドしてECRから実行
ECRリポジトリを作成する。リポジトリのURIを変数に入れ、ECRにログインしておく。
aws ecr create-repository --repository-name k8s/kube-bench --image-tag-mutability MUTABLE repo=$(aws ecr describe-repositories --repository-names k8s/kube-bench --query 'repositories[0].repositoryUri' --output text) ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) AWS_REGION=$(aws configure get region) aws ecr get-login-password | docker login --username AWS --password-stdin https://${ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
イメージをbuildしてECRにpushする。
git clone https://github.com/aquasecurity/kube-bench.git cd kube-bench docker build -t k8s/kube-bench . docker tag k8s/kube-bench:latest ${repo}:latest docker push ${repo}:latest
Jobのマニフェストを作成する。
cd ../ cat << EOF > job-eks-ecr.yaml --- apiVersion: batch/v1 kind: Job metadata: name: kube-bench-ecr spec: template: spec: hostPID: true containers: - name: kube-bench # Push the image to your ECR and then refer to it here image: ${repo}:latest command: ["kube-bench", "node", "--benchmark", "eks-1.0"] volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet readOnly: true - name: etc-systemd mountPath: /etc/systemd readOnly: true - name: etc-kubernetes mountPath: /etc/kubernetes readOnly: true restartPolicy: Never volumes: - name: var-lib-kubelet hostPath: path: "/var/lib/kubelet" - name: etc-systemd hostPath: path: "/etc/systemd" - name: etc-kubernetes hostPath: path: "/etc/kubernetes" EOF
Jobを作成する。
kubectl apply -f job-eks-ecr.yaml
ログを確認する。
$ kubectl get pod NAME READY STATUS RESTARTS AGE kube-bench-ecr-q48bt 0/1 Completed 0 22s $ kubectl logs kube-bench-ecr-q48bt [INFO] 3 Worker Node Security Configuration [INFO] 3.1 Worker Node Configuration Files [PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored) [PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored) [PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) [PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored) [INFO] 3.2 Kubelet [PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored) [PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored) [PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored) [PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored) [PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored) [PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored) [PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored) [PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored) [WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored) [PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored) [PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) == Remediations == 3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service == Summary == 14 checks PASS 0 checks FAIL 1 checks WARN 0 checks INFO $
ジョブを削除する。
kubectl delete -f job-eks-ecr.yaml
デバッグ実行
cat << EOF > job-debug-eks.yaml --- apiVersion: batch/v1 kind: Job metadata: name: kube-bench-debug spec: template: spec: hostPID: true containers: - name: kube-bench image: aquasec/kube-bench:latest command: ["kube-bench", "-v", "3", "--logtostderr", "--benchmark", "eks-1.0"] volumeMounts: - name: var-lib-kubelet mountPath: /var/lib/kubelet readOnly: true - name: etc-systemd mountPath: /etc/systemd readOnly: true - name: etc-kubernetes mountPath: /etc/kubernetes readOnly: true restartPolicy: Never volumes: - name: var-lib-kubelet hostPath: path: "/var/lib/kubelet" - name: etc-systemd hostPath: path: "/etc/systemd" - name: etc-kubernetes hostPath: path: "/etc/kubernetes" EOF
$ kubectl get pod NAME READY STATUS RESTARTS AGE kube-bench-debug-krzkb 0/1 Completed 0 6s $ kubectl logs kube-bench-debug-krzkb I0907 14:55:45.608940 1391 common.go:303] Kubernetes version: "" to Benchmark version: "eks-1.0" I0907 14:55:45.608970 1391 root.go:71] Running checks for benchmark eks-1.0 I0907 14:55:45.608976 1391 common.go:318] Checking if the current node is running master components I0907 14:55:45.608995 1391 util.go:69] ps - proc: "kube-apiserver" I0907 14:55:45.613878 1391 util.go:73] [/bin/ps -C kube-apiserver -o cmd --no-headers]: exit status 1 I0907 14:55:45.613890 1391 util.go:76] ps - returning: "" I0907 14:55:45.613917 1391 util.go:218] reFirstWord.Match() I0907 14:55:45.613923 1391 util.go:248] executable 'kube-apiserver' not running I0907 14:55:45.613930 1391 util.go:69] ps - proc: "hyperkube" I0907 14:55:45.618931 1391 util.go:73] [/bin/ps -C hyperkube -o cmd --no-headers]: exit status 1 I0907 14:55:45.618944 1391 util.go:76] ps - returning: "" I0907 14:55:45.618977 1391 util.go:218] reFirstWord.Match() I0907 14:55:45.618982 1391 util.go:248] executable 'hyperkube apiserver' not running I0907 14:55:45.618991 1391 util.go:69] ps - proc: "hyperkube" I0907 14:55:45.622457 1391 util.go:73] [/bin/ps -C hyperkube -o cmd --no-headers]: exit status 1 I0907 14:55:45.622468 1391 util.go:76] ps - returning: "" I0907 14:55:45.622506 1391 util.go:218] reFirstWord.Match() I0907 14:55:45.622514 1391 util.go:248] executable 'hyperkube kube-apiserver' not running I0907 14:55:45.622520 1391 util.go:69] ps - proc: "apiserver" I0907 14:55:45.626161 1391 util.go:73] [/bin/ps -C apiserver -o cmd --no-headers]: exit status 1 （省略） I0907 14:55:45.695981 1391 check.go:267] Command "/bin/cat /etc/kubernetes/kubelet/kubelet-config.json" - Output: "{\n \"kind\": \"KubeletConfiguration\",\n \"apiVersion\": \"kubelet.config.k8s.io/v1beta1\",\n \"address\": \"0.0.0.0\",\n \"authentication\": {\n \"anonymous\": {\n \"enabled\": false\n },\n \"webhook\": {\n \"cacheTTL\": \"2m0s\",\n \"enabled\": true\n },\n \"x509\": {\n \"clientCAFile\": \"/etc/kubernetes/pki/ca.crt\"\n }\n },\n \"authorization\": {\n \"mode\": \"Webhook\",\n \"webhook\": {\n \"cacheAuthorizedTTL\": \"5m0s\",\n \"cacheUnauthorizedTTL\": \"30s\"\n }\n },\n \"clusterDomain\": \"cluster.local\",\n \"hairpinMode\": \"hairpin-veth\",\n \"readOnlyPort\": 0,\n \"cgroupDriver\": \"cgroupfs\",\n \"cgroupRoot\": \"/\",\n \"featureGates\": {\n \"RotateKubeletServerCertificate\": true\n },\n \"protectKernelDefaults\": true,\n \"serializeImagePulls\": false,\n \"serverTLSBootstrap\": true,\n \"tlsCipherSuites\": [\n \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\",\n \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\",\n \"TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305\",\n \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\",\n \"TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305\",\n \"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\",\n \"TLS_RSA_WITH_AES_256_GCM_SHA384\",\n \"TLS_RSA_WITH_AES_128_GCM_SHA256\"\n ],\n \"clusterDNS\": [\n \"10.100.0.10\"\n ],\n \"evictionHard\": {\n \"memory.available\": \"100Mi\",\n \"nodefs.available\": \"10%\",\n \"nodefs.inodesFree\": \"5%\"\n },\n \"kubeReserved\": {\n \"cpu\": \"70m\",\n \"ephemeral-storage\": \"1Gi\",\n \"memory\": \"574Mi\"\n },\n \"maxPods\": 29\n}\n" I0907 14:55:45.696015 1391 check.go:202] 1 tests I0907 14:55:45.696096 1391 test.go:129] In flagTestItem.findValue , match false, s UID PID PPID C STIME TTY TIME CMD root 3854 1 0 13:41 ? 00:00:44 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime docker --network-plugin cni --node-ip=192.168.41.143 --pod-infra-container-image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1 --node-labels=alpha.eksctl.io/cluster-name=mycluster,alpha.eksctl.io/nodegroup-name=ng-85ed444a,eks.amazonaws.com/nodegroup=ng-85ed444a,eks.amazonaws.com/nodegroup-image=ami-08c36bc5fca5ac740,eks.amazonaws.com/sourceLaunchTemplateId=lt-00b537332c0796b19,eks.amazonaws.com/sourceLaunchTemplateVersion=1, t.Flag RotateKubeletServerCertificate I0907 14:55:45.696120 1391 test.go:197] flagFound false I0907 14:55:45.696196 1391 test.go:147] In pathTestItem.findValue true I0907 14:55:45.696207 1391 test.go:197] flagFound true I0907 14:55:45.696245 1391 check.go:245] Returning from execute on tests: finalOutput &check.testOutput{testResult:true, flagFound:false, actualResult:"", ExpectedResult:"'true' is equal to 'true'"} I0907 14:55:45.696258 1391 check.go:173] Check.ID: 3.2.11 Command: "/bin/cat /etc/kubernetes/kubelet/kubelet-config.json" TestResult: true State: "PASS" I0907 14:55:45.696313 1391 root.go:110] == Running policies checks == I0907 14:55:45.696347 1391 util.go:116] Looking for config specific CIS version "eks-1.0" I0907 14:55:45.696389 1391 util.go:120] Looking for file: cfg/eks-1.0/policies.yaml I0907 14:55:45.696457 1391 common.go:235] Using config file: cfg/eks-1.0/config.yaml I0907 14:55:45.696519 1391 common.go:80] Using test file: cfg/eks-1.0/policies.yaml I0907 14:55:45.697096 1391 root.go:121] == Running managed services checks == I0907 14:55:45.697109 1391 util.go:116] Looking for config specific CIS version "eks-1.0" I0907 14:55:45.697116 1391 util.go:120] Looking for file: cfg/eks-1.0/managedservices.yaml I0907 14:55:45.697196 1391 common.go:235] Using config file: cfg/eks-1.0/config.yaml I0907 14:55:45.697223 1391 common.go:80] Using test file: cfg/eks-1.0/managedservices.yaml [INFO] 3 Worker Node Security Configuration [INFO] 3.1 Worker Node Configuration Files [PASS] 3.1.1 Ensure that the proxy kubeconfig file permissions are set to 644 or more restrictive (Scored) [PASS] 3.1.2 Ensure that the proxy kubeconfig file ownership is set to root:root (Scored) [PASS] 3.1.3 Ensure that the kubelet configuration file has permissions set to 644 or more restrictive (Scored) [PASS] 3.1.4 Ensure that the kubelet configuration file ownership is set to root:root (Scored) [INFO] 3.2 Kubelet [PASS] 3.2.1 Ensure that the --anonymous-auth argument is set to false (Scored) [PASS] 3.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Scored) [PASS] 3.2.3 Ensure that the --client-ca-file argument is set as appropriate (Scored) [PASS] 3.2.4 Ensure that the --read-only-port argument is set to 0 (Scored) [PASS] 3.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Scored) [PASS] 3.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Scored) [PASS] 3.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Scored) [PASS] 3.2.8 Ensure that the --hostname-override argument is not set (Scored) [WARN] 3.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Scored) [PASS] 3.2.10 Ensure that the --rotate-certificates argument is not set to false (Scored) [PASS] 3.2.11 Ensure that the RotateKubeletServerCertificate argument is set to true (Scored) == Remediations == 3.2.9 If using a Kubelet config file, edit the file to set eventRecordQPS: to an appropriate level. If using command line arguments, edit the kubelet service file /etc/systemd/system/kubelet.service on each worker node and set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. Based on your system, restart the kubelet service. For example: systemctl daemon-reload systemctl restart kubelet.service == Summary == 14 checks PASS 0 checks FAIL 1 checks WARN 0 checks INFO [INFO] 4 Policies [INFO] 4.1 RBAC and Service Accounts [WARN] 4.1.1 Ensure that the cluster-admin role is only used where required (Not Scored) [WARN] 4.1.2 Minimize access to secrets (Not Scored) [WARN] 4.1.3 Minimize wildcard use in Roles and ClusterRoles (Not Scored) [WARN] 4.1.4 Minimize access to create pods (Not Scored) [WARN] 4.1.5 Ensure that default service accounts are not actively used. (Not Scored) [WARN] 4.1.6 Ensure that Service Account Tokens are only mounted where necessary (Not Scored) [INFO] 4.2 Pod Security Policies [WARN] 4.2.1 Minimize the admission of privileged containers (Not Scored) [WARN] 4.2.2 Minimize the admission of containers wishing to share the host process ID namespace (Not Scored) [WARN] 4.2.3 Minimize the admission of containers wishing to share the host IPC namespace (Not Scored) [WARN] 4.2.4 Minimize the admission of containers wishing to share the host network namespace (Not Scored) [WARN] 4.2.5 Minimize the admission of containers with allowPrivilegeEscalation (Not Scored) [WARN] 4.2.6 Minimize the admission of root containers (Not Scored) [WARN] 4.2.7 Minimize the admission of containers with the NET_RAW capability (Not Scored) [WARN] 4.2.8 Minimize the admission of containers with added capabilities (Not Scored) [WARN] 4.2.9 Minimize the admission of containers with capabilities assigned (Not Scored) [INFO] 4.3 CNI Plugin [WARN] 4.3.1 Ensure that the latest CNI version is used (Not Scored) [WARN] 4.3.2 Ensure that all Namespaces have Network Policies defined (Not Scored) [INFO] 4.4 Secrets Management [WARN] 4.4.1 Prefer using secrets as files over secrets as environment variables (Not Scored) [WARN] 4.4.2 Consider external secret storage (Not Scored) [INFO] 4.5 Extensible Admission Control [WARN] 4.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Not Scored) [INFO] 4.6 General Policies [WARN] 4.6.1 Create administrative boundaries between resources using namespaces (Not Scored) [WARN] 4.6.2 Ensure that the seccomp profile is set to docker/default in your pod definitions (Not Scored) [WARN] 4.6.3 Apply Security Context to Your Pods and Containers (Not Scored) [WARN] 4.6.4 The default namespace should not be used (Not Scored) == Remediations == 4.1.1 Identify all clusterrolebindings to the cluster-admin role. Check if they are used and if they need this role or if they could use a role with fewer privileges. Where possible, first bind users to a lower privileged role and then remove the clusterrolebinding to the cluster-admin role : kubectl delete clusterrolebinding [name] 4.1.2 Where possible, remove get, list and watch access to secret objects in the cluster. 4.1.3 Where possible replace any use of wildcards in clusterroles and roles with specific objects or actions. 4.1.4 4.1.5 Create explicit service accounts wherever a Kubernetes workload requires specific access to the Kubernetes API server. Modify the configuration of each default service account to include this value automountServiceAccountToken: false 4.1.6 Modify the definition of pods and service accounts which do not need to mount service account tokens to disable it. 4.2.1 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.privileged field is omitted or set to false. 4.2.2 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostPID field is omitted or set to false. 4.2.3 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostIPC field is omitted or set to false. 4.2.4 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.hostNetwork field is omitted or set to false. 4.2.5 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.allowPrivilegeEscalation field is omitted or set to false. 4.2.6 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.runAsUser.rule is set to either MustRunAsNonRoot or MustRunAs with the range of UIDs not including 0. 4.2.7 Create a PSP as described in the Kubernetes documentation, ensuring that the .spec.requiredDropCapabilities is set to include either NET_RAW or ALL. 4.2.8 Ensure that allowedCapabilities is not present in PSPs for the cluster unless it is set to an empty array. 4.2.9 Review the use of capabilites in applications runnning on your cluster. Where a namespace contains applications which do not require any Linux capabities to operate consider adding a PSP which forbids the admission of containers which do not drop all capabilities. 4.3.1 Review the documentation of AWS CNI plugin, and ensure latest CNI version is used. 4.3.2 Follow the documentation and create NetworkPolicy objects as you need them. 4.4.1 If possible, rewrite application code to read secrets from mounted secret files, rather than from environment variables. 4.4.2 Refer to the secrets management options offered by your cloud provider or a third-party secrets management solution. 4.5.1 Follow the Kubernetes documentation and setup image provenance. 4.6.1 Follow the documentation and create namespaces for objects in your deployment as you need them. 4.6.2 Seccomp is an alpha feature currently. By default, all alpha features are disabled. So, you would need to enable alpha features in the apiserver by passing "--feature- gates=AllAlpha=true" argument. Edit the /etc/kubernetes/apiserver file on the master node and set the KUBE_API_ARGS parameter to "--feature-gates=AllAlpha=true" KUBE_API_ARGS="--feature-gates=AllAlpha=true" Based on your system, restart the kube-apiserver service. For example: systemctl restart kube-apiserver.service Use annotations to enable the docker/default seccomp profile in your pod definitions. An example is as below: apiVersion: v1 kind: Pod metadata: name: trustworthy-pod annotations: seccomp.security.alpha.kubernetes.io/pod: docker/default spec: containers: - name: trustworthy-container image: sotrustworthy:latest 4.6.3 Follow the Kubernetes documentation and apply security contexts to your pods. For a suggested list of security contexts, you may refer to the CIS Security Benchmark for Docker Containers. 4.6.4 Ensure that namespaces are created to allow for appropriate segregation of Kubernetes resources and that all new resources are created in a specific namespace. == Summary == 0 checks PASS 0 checks FAIL 24 checks WARN 0 checks INFO [INFO] 5 Managed Services [INFO] 5.1 Image Registry and Image Scanning [WARN] 5.1.1 Ensure Image Vulnerability Scanning using Amazon ECR image scanning or a third-party provider (Not Scored) [WARN] 5.1.2 Minimize user access to Amazon ECR (Not Scored) [WARN] 5.1.3 Minimize cluster access to read-only for Amazon ECR (Not Scored) [WARN] 5.1.4 Minimize Container Registries to only those approved (Not Scored) [INFO] 5.2 Identity and Access Management (IAM) [WARN] 5.2.1 Prefer using dedicated Amazon EKS Service Accounts (Not Scored) [INFO] 5.3 AWS Key Management Service (AWS KMS) [WARN] 5.3.1 Ensure Kubernetes Secrets are encrypted using Customer Master Keys (CMKs) managed in AWS KMS (Not Scored) [INFO] 5.4 Cluster Networking [WARN] 5.4.1 Restrict Access to the Control Plane Endpoint (Not Scored) [WARN] 5.4.2 Ensure clusters are created with Private Endpoint Enabled and Public Access Disabled (Not Scored) [WARN] 5.4.3 Ensure clusters are created with Private Nodes (Not Scored) [WARN] 5.4.4 Ensure Network Policy is Enabled and set as appropriate (Not Scored) [WARN] 5.4.5 Encrypt traffic to HTTPS load balancers with TLS certificates (Not Scored) [INFO] 5.5 Authentication and Authorization [WARN] 5.5.1 Manage Kubernetes RBAC users with AWS IAM Authenticator for Kubernetes (Not Scored) [INFO] 5.6 Other Cluster Configurations [WARN] 5.6.1 Consider Fargate for running untrusted workloads (Not Scored) == Remediations == 5.1.1 5.1.2 5.1.3 5.1.4 5.2.1 5.3.1 5.4.1 5.4.2 5.4.3 5.4.4 5.4.5 5.5.1 5.6.1 == Summary == 0 checks PASS 0 checks FAIL 13 checks WARN 0 checks INFO $
Section 3のみデバッグの出力がされている。Section 4と5はNot Scoredでデバッグログも出ていない。
