非 root ユーザーで Docker コンテナを実行したときのケーパビリティー

非 root ユーザーで Docker コンテナを実行したときにそもそもケーパビリティーがあったかの確認メモ。

準備

イメージを作成する。

FROM ubuntu
RUN apt-get update && apt-get install -y \
    libcap-ng-utils
docker build -t pscap-ubuntu .

root ユーザー

コマンドを直接実行すると何も出なかった。

$ docker run --rm -it pscap-ubuntu pscap -a
sotosugi@a483e7dfd6ad:~/temp
$

そのためシェルを起動してから確認する。

$ docker run --rm -it pscap-ubuntu /bin/bash
root@d34942bc7179:/# pscap -a
ppid  pid   name        command           capabilities
0     1     root        bash              chown, dac_override, fowner, fsetid, kill, setgid, setuid, setpcap, net_bind_service, net_raw, sys_chroot, mknod, audit_write, setfcap
root@d34942bc7179:/#

root ユーザー + --privileged

$ docker run --privileged --rm -it pscap-ubuntu /bin/bash
root@b625b9fd08a9:/# pscap -a
ppid  pid   name        command           capabilities
0     1     root        bash              chown, dac_override, dac_read_search, fowner, fsetid, kill, setgid, setuid, setpcap, linux_immutable, net_bind_service, net_broadcast, net_admin, net_raw, ipc_lock, ipc_owner, sys_module, sys_rawio, sys_chroot, sys_ptrace, sys_pacct, sys_admin, sys_boot, sys_nice, sys_resource, sys_time, sys_tty_config, mknod, lease, audit_write, audit_control, setfcap, mac_override, mac_admin, syslog, wake_alarm, block_suspend, audit_read
root@b625b9fd08a9:/#

非 root ユーザー

$ docker run --user nobody --rm -it pscap-ubuntu /bin/bash
nobody@c3bee3f5f933:/$ pscap -a
nobody@c3bee3f5f933:/$

非 root ユーザー + --privileged

$ docker run --privileged --user nobody --rm -it pscap-ubuntu /bin/bash
nobody@3125c31b4e07:/$ pscap -a
nobody@3125c31b4e07:/$

まとめ

非 root ユーザーで実行している場合はそもそもケーパビリティーがなかった。なのでわざわざ DROP を書く必要はない。少なくとも Docker の場合は。Kuberentes や Fargate の場合も同じだったかは確認したい。

Karpener を試す

Karpener を試してみるメモ。

公式には Getting Started with Terraform もあるが、eksctl での Getting Started に沿ってやってみる。

コンポーネント バージョン 備考
eksctl 0.79.0
Kubernetes バージョン 1.21
プラットフォームのバージョン eks.4
Karpenter 0.5.4
Karpenter チャート 0.5.4

準備

Karpenter を利用するための準備をする。

クラスターの作成

クラスターを作成する。MNG も作っている。

export CLUSTER_NAME=karpenter-demo
export AWS_DEFAULT_REGION=ap-northeast-1
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
cat <<EOF > cluster.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: ${CLUSTER_NAME}
  region: ${AWS_DEFAULT_REGION}
  version: "1.21"
managedNodeGroups:
  - instanceType: m5.large
    amiFamily: AmazonLinux2
    name: ${CLUSTER_NAME}-ng
    desiredCapacity: 1
    minSize: 1
    maxSize: 10
iam:
  withOIDC: true
EOF
eksctl create cluster -f cluster.yaml

サブネットへのタグ付け

Kaerpenter がサブネットを見つけるのにタグが必要なのでタグ付けする。キーが決まっていて値は何でもよさそう。

SUBNET_IDS=$(aws cloudformation describe-stacks \
    --stack-name eksctl-${CLUSTER_NAME}-cluster \
    --query 'Stacks[].Outputs[?OutputKey==`SubnetsPrivate`].OutputValue' \
    --output text)
aws ec2 create-tags \
    --resources $(echo $SUBNET_IDS | tr ',' '\n') \
    --tags Key="kubernetes.io/cluster/${CLUSTER_NAME}",Value=

一番下が追加したタグで、プライベートサブネットにのみ追加している。

f:id:sotoiwa:20220118070730p:plain

IAM ロールの作成

IAM ロールを作成する。この CloudFormation では、ノード用の IAM ロールと、Karpenter コントローラー用の IAM ポリシーを作っている。ノード用のロールとは、Karpenter がプロビジョニングするノードに付与するロール。

TEMPOUT=$(mktemp)
curl -fsSL https://karpenter.sh/docs/getting-started/cloudformation.yaml > $TEMPOUT \
&& aws cloudformation deploy \
  --stack-name Karpenter-${CLUSTER_NAME} \
  --template-file ${TEMPOUT} \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides ClusterName=${CLUSTER_NAME}

ノードに Kubernetes への権限を与えるための aws-auth ConfigMap を変更する。

eksctl create iamidentitymapping \
  --username system:node:{{EC2PrivateDNSName}} \
  --cluster  ${CLUSTER_NAME} \
  --arn arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME} \
  --group system:bootstrappers \
  --group system:nodes

確認する。

$ k -n kube-system get cm aws-auth -oyaml
apiVersion: v1
data:
  mapRoles: |
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::XXXXXXXXXXXX:role/eksctl-karpenter-demo-nodegroup-k-NodeInstanceRole-1C7PSQCKU617K
      username: system:node:{{EC2PrivateDNSName}}
    - groups:
      - system:bootstrappers
      - system:nodes
      rolearn: arn:aws:iam::XXXXXXXXXXXX:role/KarpenterNodeRole-karpenter-demo
      username: system:node:{{EC2PrivateDNSName}}
  mapUsers: |
    []
kind: ConfigMap
metadata:
  creationTimestamp: "2022-01-16T20:41:52Z"
  name: aws-auth
  namespace: kube-system
  resourceVersion: "11499"
  uid: fe62e41b-95f6-47a7-b6ae-c57458cb8640

IRSA でコントローラーに付与する IAM ロールを作成する。

eksctl create iamserviceaccount \
  --cluster $CLUSTER_NAME --name karpenter --namespace karpenter \
  --attach-policy-arn arn:aws:iam::$AWS_ACCOUNT_ID:policy/KarpenterControllerPolicy-$CLUSTER_NAME \
  --approve

EC2 スポット用の Service Linked ロールを作成する。これはアカウントで 1 回だけ必要。エラーになったので、作成済みだった。

aws iam create-service-linked-role --aws-service-name spot.amazonaws.com

Karpenter のデプロイ

Helm チャートリポジトリを登録する。

helm repo add karpenter https://charts.karpenter.sh
helm repo update

チャートを確認する。

$ helm search repo karpenter
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
karpenter/karpenter     0.5.4           0.5.4           A Helm chart for https://github.com/aws/karpent...

Karpenter をインストールする。

$ helm upgrade --install karpenter karpenter/karpenter --namespace karpenter \
  --create-namespace --set serviceAccount.create=false --version 0.5.4 \
  --set controller.clusterName=${CLUSTER_NAME} \
  --set controller.clusterEndpoint=$(aws eks describe-cluster --name ${CLUSTER_NAME} --query "cluster.endpoint" --output json) \
  --wait
Release "karpenter" does not exist. Installing it now.
NAME: karpenter
LAST DEPLOYED: Tue Jan 18 05:11:39 2022
NAMESPACE: karpenter
STATUS: deployed
REVISION: 1
TEST SUITE: None

Pod を確認する。

$ k -n karpenter get po
NAME                                    READY   STATUS    RESTARTS   AGE
karpenter-controller-7b77564cb8-f9gcg   1/1     Running   0          105s
karpenter-webhook-7d847d4df5-t8jb4      1/1     Running   0          105s

(オプション)デバッグログを有効にする。

kubectl patch configmap config-logging -n karpenter --patch '{"data":{"loglevel.controller":"debug"}}'

Prometheus と Grafana のインストール(オプション)

チャートリポジトリを登録する。

helm repo add grafana-charts https://grafana.github.io/helm-charts
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

チャートを確認する。

$ helm search repo prometheus-community/prometheus
NAME                                                    CHART VERSION   APP VERSION     DESCRIPTION                                       
prometheus-community/prometheus                         15.0.4          2.31.1          Prometheus is a monitoring system and time seri...
prometheus-community/prometheus-adapter                 3.0.1           v0.9.1          A Helm chart for k8s prometheus adapter           
prometheus-community/prometheus-blackbox-exporter       5.3.1           0.19.0          Prometheus Blackbox Exporter                      
prometheus-community/prometheus-cloudwatch-expo...      0.18.0          0.12.2          A Helm chart for prometheus cloudwatch-exporter   
prometheus-community/prometheus-consul-exporter         0.5.0           0.4.0           A Helm chart for the Prometheus Consul Exporter   
prometheus-community/prometheus-couchdb-exporter        0.2.0           1.0             A Helm chart to export the metrics from couchdb...
prometheus-community/prometheus-druid-exporter          0.11.0          v0.8.0          Druid exporter to monitor druid metrics with Pr...
prometheus-community/prometheus-elasticsearch-e...      4.11.0          1.3.0           Elasticsearch stats exporter for Prometheus       
prometheus-community/prometheus-json-exporter           0.1.0           1.0.2           Install prometheus-json-exporter                  
prometheus-community/prometheus-kafka-exporter          1.5.0           v1.4.1          A Helm chart to export the metrics from Kafka i...
prometheus-community/prometheus-mongodb-exporter        2.9.0           v0.10.0         A Prometheus exporter for MongoDB metrics         
prometheus-community/prometheus-mysql-exporter          1.5.0           v0.12.1         A Helm chart for prometheus mysql exporter with...
prometheus-community/prometheus-nats-exporter           2.9.0           0.9.0           A Helm chart for prometheus-nats-exporter         
prometheus-community/prometheus-node-exporter           2.5.0           1.3.1           A Helm chart for prometheus node-exporter         
prometheus-community/prometheus-operator                9.3.2           0.38.1          DEPRECATED - This chart will be renamed. See ht...
prometheus-community/prometheus-pingdom-exporter        2.4.1           20190610-1      A Helm chart for Prometheus Pingdom Exporter      
prometheus-community/prometheus-postgres-exporter       2.4.0           0.10.0          A Helm chart for prometheus postgres-exporter     
prometheus-community/prometheus-pushgateway             1.14.0          1.4.2           A Helm chart for prometheus pushgateway           
prometheus-community/prometheus-rabbitmq-exporter       1.0.0           v0.29.0         Rabbitmq metrics exporter for prometheus          
prometheus-community/prometheus-redis-exporter          4.6.0           1.27.0          Prometheus exporter for Redis metrics             
prometheus-community/prometheus-snmp-exporter           0.1.5           0.19.0          Prometheus SNMP Exporter                          
prometheus-community/prometheus-stackdriver-exp...      1.12.0          0.11.0          Stackdriver exporter for Prometheus               
prometheus-community/prometheus-statsd-exporter         0.4.2           0.22.1          A Helm chart for prometheus stats-exporter        
prometheus-community/prometheus-to-sd                   0.4.0           0.5.2           Scrape metrics stored in prometheus format and ...
$ helm search repo grafana-charts/grafana
NAME                                    CHART VERSION   APP VERSION     DESCRIPTION                                       
grafana-charts/grafana                  6.20.5          8.3.4           The leading tool for querying and visualizing t...
grafana-charts/grafana-agent-operator   0.1.4           0.20.0          A Helm chart for Grafana Agent Operator     

Namespace を作成する。

$ kubectl create namespace monitoring
namespace/monitoring created

Prometheus をデプロイする。

$ curl -fsSL https://karpenter.sh/docs/getting-started/prometheus-values.yaml | tee prometheus-values.yaml
alertmanager:
  persistentVolume:
    enabled: false

server:
  fullnameOverride: prometheus-server
  persistentVolume:
    enabled: false

extraScrapeConfigs: |
    - job_name: karpenter
      static_configs:
      - targets:
        - karpenter-metrics.karpenter:8080
$ helm install --namespace monitoring prometheus prometheus-community/prometheus --values prometheus-values.yaml
NAME: prometheus
LAST DEPLOYED: Tue Jan 18 05:20:58 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9090
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Server pod is terminated.                             #####
#################################################################################


The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.monitoring.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the AlertManager pod is terminated.                       #####
#################################################################################
#################################################################################
######   WARNING: Pod Security Policy has been moved to a global property.  #####
######            use .Values.podSecurityPolicy.enabled with pod-based      #####
######            annotations                                               #####
######            (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.monitoring.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

Grafana をデプロイする。

$ curl -fsSL https://karpenter.sh/docs/getting-started/grafana-values.yaml | tee grafana-values.yaml
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      version: 1
      url: http://prometheus-server:80
      access: proxy
$ helm install --namespace monitoring grafana grafana-charts/grafana --values grafana-values.yaml
W0118 05:22:13.264815   16206 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0118 05:22:14.344142   16206 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0118 05:22:26.410996   16206 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
W0118 05:22:26.411453   16206 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
NAME: grafana
LAST DEPLOYED: Tue Jan 18 05:22:06 2022
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.monitoring.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:

     export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace monitoring port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin
#################################################################################
######   WARNING: Persistence is disabled!!! You will lose your data when   #####
######            the Grafana pod is terminated.                            #####
#################################################################################

Pod を確認する。

$ k -n monitoring get po
NAME                                             READY   STATUS    RESTARTS   AGE
grafana-859fcfbbfc-bkmrx                         1/1     Running   0          42s
prometheus-alertmanager-74cb99ff8-sdsvp          2/2     Running   0          106s
prometheus-kube-state-metrics-68b6c8b5c5-78967   1/1     Running   0          106s
prometheus-node-exporter-wqp9k                   1/1     Running   0          106s
prometheus-pushgateway-7ff8d6b8c4-cbd7k          1/1     Running   0          106s
prometheus-server-74d79774cd-mccv9               2/2     Running   0          106s

Grafana のパスワードを取得する。

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode

Grafana にはポートフォワードでアクセスする。

kubectl port-forward --namespace monitoring svc/grafana 3000:80

ダッシュボードをインポートしておく。

f:id:sotoiwa:20220118070832p:plain

f:id:sotoiwa:20220118070847p:plain

Provisioner の作成

デフォルトの Provisioner を作成する。

$ cat <<EOF | kubectl apply -f -
apiVersion: karpenter.sh/v1alpha5
kind: Provisioner
metadata:
  name: default
spec:
  requirements:
    - key: karpenter.sh/capacity-type
      operator: In
      values: ["spot"]
  limits:
    resources:
      cpu: 1000
  provider:
    instanceProfile: KarpenterNodeInstanceProfile-${CLUSTER_NAME}
  ttlSecondsAfterEmpty: 30
EOF
provisioner.karpenter.sh/default created

確認する。

$ k get Provisioner
NAME      AGE
default   21s

Karpenter の利用

pause イメージで CPU 1 を要求する Pod の Deployment をレプリカ 1 で作成する。

$ cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 0
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: inflate
          image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
          resources:
            requests:
              cpu: 1
EOF
deployment.apps/inflate created

Deployment をスケールする。

$ kubectl scale deployment inflate --replicas 5
deployment.apps/inflate scaled

コントローラーのログを確認する。

$ kubectl logs -f -n karpenter $(kubectl get pods -n karpenter -l karpenter=controller -o name)
2022-01-17T20:12:07.548Z        INFO    Successfully created the logger.
2022-01-17T20:12:07.548Z        INFO    Logging level set to: info
{"level":"info","ts":1642450327.6526809,"logger":"fallback","caller":"injection/injection.go:61","msg":"Starting informers..."}
I0117 20:12:07.670361       1 leaderelection.go:243] attempting to acquire leader lease karpenter/karpenter-leader-election...
2022-01-17T20:12:07.670Z        INFO    controller      starting metrics server {"commit": "7e79a67", "path": "/metrics"}
I0117 20:12:07.715692       1 leaderelection.go:253] successfully acquired lease karpenter/karpenter-leader-election
2022-01-17T20:12:07.771Z        INFO    controller.controller.counter   Starting EventSource    {"commit": "7e79a67", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.771Z        INFO    controller.controller.counter   Starting EventSource    {"commit": "7e79a67", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.771Z        INFO    controller.controller.counter   Starting Controller     {"commit": "7e79a67", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-01-17T20:12:07.772Z        INFO    controller.controller.provisioning      Starting EventSource    {"commit": "7e79a67", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.772Z        INFO    controller.controller.provisioning      Starting Controller     {"commit": "7e79a67", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner"}
2022-01-17T20:12:07.772Z        INFO    controller.controller.volume    Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "PersistentVolumeClaim", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.772Z        INFO    controller.controller.volume    Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "PersistentVolumeClaim", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.772Z        INFO    controller.controller.volume    Starting Controller     {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "PersistentVolumeClaim"}
2022-01-17T20:12:07.772Z        INFO    controller.controller.termination       Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.772Z        INFO    controller.controller.termination       Starting Controller     {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node"}
2022-01-17T20:12:07.773Z        INFO    controller.controller.node      Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.773Z        INFO    controller.controller.node      Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.773Z        INFO    controller.controller.node      Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.773Z        INFO    controller.controller.node      Starting Controller     {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node"}
2022-01-17T20:12:07.773Z        INFO    controller.controller.podmetrics        Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.773Z        INFO    controller.controller.podmetrics        Starting Controller     {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Pod"}
2022-01-17T20:12:07.773Z        INFO    controller.controller.nodemetrics       Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.773Z        INFO    controller.controller.nodemetrics       Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.773Z        INFO    controller.controller.nodemetrics       Starting EventSource    {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "source": "kind source: /, Kind="}
2022-01-17T20:12:07.773Z        INFO    controller.controller.nodemetrics       Starting Controller     {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node"}
2022-01-17T20:12:07.872Z        INFO    controller.controller.provisioning      Starting workers        {"commit": "7e79a67", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 10}
2022-01-17T20:12:07.873Z        INFO    controller.controller.termination       Starting workers        {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-01-17T20:12:07.873Z        INFO    controller.controller.counter   Starting workers        {"commit": "7e79a67", "reconciler group": "karpenter.sh", "reconciler kind": "Provisioner", "worker count": 10}
2022-01-17T20:12:07.873Z        INFO    controller.controller.volume    Starting workers        {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "PersistentVolumeClaim", "worker count": 1}
2022-01-17T20:12:07.888Z        INFO    controller.controller.podmetrics        Starting workers        {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Pod", "worker count": 1}
2022-01-17T20:12:07.891Z        INFO    controller.controller.node      Starting workers        {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "worker count": 10}
2022-01-17T20:12:07.891Z        INFO    controller.controller.nodemetrics       Starting workers        {"commit": "7e79a67", "reconciler group": "", "reconciler kind": "Node", "worker count": 1}
2022-01-17T20:15:12.525Z        INFO    controller      Updating logging level for controller from info to debug.       {"commit": "7e79a67"}
2022-01-17T20:35:26.973Z        DEBUG   controller.provisioning Discovered 324 EC2 instance types       {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:35:27.216Z        DEBUG   controller.provisioning Discovered subnets: [subnet-021d3b8da085b55a9 (ap-northeast-1c) subnet-02e70ec727156b94f (ap-northeast-1d) subnet-0d529a446897f24a2 (ap-northeast-1a)]        {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:35:27.336Z        DEBUG   controller.provisioning Discovered EC2 instance types zonal offerings   {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:35:27.341Z        INFO    controller.provisioning Waiting for unschedulable pods  {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:40:28.066Z        DEBUG   controller.provisioning Discovered 324 EC2 instance types       {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:40:28.287Z        DEBUG   controller.provisioning Discovered subnets: [subnet-021d3b8da085b55a9 (ap-northeast-1c) subnet-02e70ec727156b94f (ap-northeast-1d) subnet-0d529a446897f24a2 (ap-northeast-1a)]        {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:40:28.406Z        DEBUG   controller.provisioning Discovered EC2 instance types zonal offerings   {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:56.769Z        INFO    controller.provisioning Batched 5 pods in 1.080460937s  {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:56.852Z        DEBUG   controller.provisioning Discovered subnets: [subnet-021d3b8da085b55a9 (ap-northeast-1c) subnet-02e70ec727156b94f (ap-northeast-1d) subnet-0d529a446897f24a2 (ap-northeast-1a)]        {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:56.967Z        DEBUG   controller.provisioning Excluding instance type t3a.nano because there are not enough resources for kubelet and system overhead {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:56.974Z        DEBUG   controller.provisioning Excluding instance type t3.nano because there are not enough resources for kubelet and system overhead  {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:56.979Z        INFO    controller.provisioning Computed packing of 1 node(s) for 5 pod(s) with instance type option(s) [c1.xlarge c4.2xlarge c3.2xlarge c5a.2xlarge c5.2xlarge c6i.2xlarge c5d.2xlarge c5n.2xlarge m3.2xlarge m5n.2xlarge m6i.2xlarge m5zn.2xlarge t3.2xlarge m5d.2xlarge m5a.2xlarge m5.2xlarge t3a.2xlarge m5ad.2xlarge m4.2xlarge m5dn.2xlarge]   {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:57.138Z        DEBUG   controller.provisioning Discovered security groups: [sg-0ef398d023a1842ba]      {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:57.140Z        DEBUG   controller.provisioning Discovered kubernetes version 1.21      {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:57.211Z        DEBUG   controller.provisioning Discovered ami ami-0b7bbd8c99e925a83 for query /aws/service/eks/optimized-ami/1.21/amazon-linux-2/recommended/image_id  {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:57.211Z        DEBUG   controller.provisioning Discovered caBundle, length 1066        {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:41:57.360Z        DEBUG   controller.provisioning Created launch template, Karpenter-karpenter-demo-17569880151886454907  {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:42:01.109Z        INFO    controller.provisioning Launched instance: i-0473f37e849fdd8ec, hostname: ip-192-168-133-127.ap-northeast-1.compute.internal, type: t3a.2xlarge, zone: ap-northeast-1d, capacityType: spot    {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:42:01.191Z        INFO    controller.provisioning Bound 5 pod(s) to node ip-192-168-133-127.ap-northeast-1.compute.internal       {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T20:42:01.191Z        INFO    controller.provisioning Waiting for unschedulable pods  {"commit": "7e79a67", "provisioner": "default"}

5 つの Pod が収容できるノードがプロビジョニングされた。

$ k get po -o wide
NAME                       READY   STATUS    RESTARTS   AGE     IP                NODE                                                 NOMINATED NODE   READINESS GATES
inflate-6b88c9fb68-7h4vw   1/1     Running   0          2m52s   192.168.128.88    ip-192-168-133-127.ap-northeast-1.compute.internal   <none>           <none>
inflate-6b88c9fb68-c6hmr   1/1     Running   0          2m52s   192.168.159.221   ip-192-168-133-127.ap-northeast-1.compute.internal   <none>           <none>
inflate-6b88c9fb68-cqd9v   1/1     Running   0          2m52s   192.168.137.10    ip-192-168-133-127.ap-northeast-1.compute.internal   <none>           <none>
inflate-6b88c9fb68-ws5jq   1/1     Running   0          2m52s   192.168.159.185   ip-192-168-133-127.ap-northeast-1.compute.internal   <none>           <none>
inflate-6b88c9fb68-xj6pc   1/1     Running   0          2m52s   192.168.137.151   ip-192-168-133-127.ap-northeast-1.compute.internal   <none>           <none>
$ k get node -o wide
NAME                                                 STATUS   ROLES    AGE    VERSION               INTERNAL-IP       EXTERNAL-IP    OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
ip-192-168-133-127.ap-northeast-1.compute.internal   Ready    <none>   3m4s   v1.21.5-eks-bc4871b   192.168.133.127   <none>         Amazon Linux 2   5.4.162-86.275.amzn2.x86_64   containerd://1.4.6
ip-192-168-72-151.ap-northeast-1.compute.internal    Ready    <none>   24h    v1.21.5-eks-bc4871b   192.168.72.151    54.92.61.102   Amazon Linux 2   5.4.162-86.275.amzn2.x86_64   docker://20.10.7

この時の動きとしては以下のようになっていた。

  • Karpenter はさっさとノードの登録と、Pod のスケジューリングを実施
  • ノードのステータスは Unknown で、Pod のステータスは Pending でスケジューリング済み
  • しばらくしてノードが NotReady になると Pod は ContainerCreating となる
  • ノードが Ready になると Pod は Running になる

以下気になった点。

  • コンテナランタイムが Containerd ではないのは、Karpanter が起動テンプレートを作っていて、そのユーザーデータで Containerd を指定しているから
  • Public サブネットと、Private サブネットがそれぞれ 3 つの VPC で、MNG は Public サブネットに配置する構成になっていた
  • Private サブネットにのみタグ付けしたので、Karpenter は Private サブネットにノードを作成している

EC2 コンソールでインスタンスを確認する。先ほど作成した IAM ロールがアタッチされている。このロールを Provisioner で指定していた。

f:id:sotoiwa:20220118070910p:plain

EC2 Fleet API を利用しているらしいので、describe-fleets してみたが何も表示されなかった。

$ aws ec2 describe-fleets
{
    "Fleets": []
}

インスタンスのタグには aws:ec2:fleet-id が付与されている。

f:id:sotoiwa:20220118070926p:plain

Provisioner で spot を指定していたので、スポットリクエストを確認したらあった。

f:id:sotoiwa:20220118070942p:plain

$ aws ec2 describe-spot-instance-requests
{
    "SpotInstanceRequests": [
        {
            "CreateTime": "2022-01-17T20:41:59+00:00",
            "InstanceId": "i-0473f37e849fdd8ec",
            "LaunchSpecification": {
                "SecurityGroups": [
                    {
                        "GroupName": "eks-cluster-sg-karpenter-demo-619918540",
                        "GroupId": "sg-0ef398d023a1842ba"
                    }
                ],
                "IamInstanceProfile": {
                    "Arn": "arn:aws:iam::XXXXXXXXXXXX:instance-profile/KarpenterNodeInstanceProfile-karpenter-demo",
                    "Name": "KarpenterNodeInstanceProfile-karpenter-demo"
                },
                "ImageId": "ami-0b7bbd8c99e925a83",
                "InstanceType": "t3a.2xlarge",
                "NetworkInterfaces": [
                    {
                        "DeleteOnTermination": true,
                        "DeviceIndex": 0,
                        "SubnetId": "subnet-02e70ec727156b94f"
                    }
                ],
                "Placement": {
                    "AvailabilityZone": "ap-northeast-1d",
                    "Tenancy": "default"
                },
                "Monitoring": {
                    "Enabled": false
                }
            },
            "LaunchedAvailabilityZone": "ap-northeast-1d",
            "ProductDescription": "Linux/UNIX",
            "SpotInstanceRequestId": "sir-r8268drq",
            "SpotPrice": "0.391700",
            "State": "active",
            "Status": {
                "Code": "fulfilled",
                "Message": "Your spot request is fulfilled.",
                "UpdateTime": "2022-01-17T21:08:39+00:00"
            },
            "Tags": [],
            "Type": "one-time",
            "InstanceInterruptionBehavior": "terminate"
        }
    ]
}

以下のコマンドはエラーになった。ID が sir- なので、Spot Fleet Request ではなく、Spot Instance Request なのかと思われる。

$ aws ec2 describe-spot-fleet-instances --spot-fleet-request-id sir-r8268drq

An error occurred (InvalidParameterValue) when calling the DescribeSpotFleetInstances operation: 1 validation error detected: Value 'sir-r8268drq' at 'spotFleetRequestId' failed to satisfy constraint: Member must satisfy regular expression pattern: ^(sfr|fleet)-[a-z0-9-]{36}\b

Deployment を削除する。Provisioner で ttlSecondsAfterEmpty: 30 を指定していたので、Pod がいなくなってから 30 秒経つとノードが削除される。

$ kubectl delete deployment inflate
deployment.apps "inflate" deleted

ログを確認する。

$ kubectl logs -f -n karpenter $(kubectl get pods -n karpenter -l karpenter=controller -o name)
...
2022-01-17T21:15:35.530Z        DEBUG   controller.provisioning Discovered 324 EC2 instance types       {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:15:35.630Z        DEBUG   controller.provisioning Discovered subnets: [subnet-021d3b8da085b55a9 (ap-northeast-1c) subnet-02e70ec727156b94f (ap-northeast-1d) subnet-0d529a446897f24a2 (ap-northeast-1a)]        {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:15:35.757Z        DEBUG   controller.provisioning Discovered EC2 instance types zonal offerings   {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:20:36.489Z        DEBUG   controller.provisioning Discovered 324 EC2 instance types       {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:20:36.698Z        DEBUG   controller.provisioning Discovered subnets: [subnet-021d3b8da085b55a9 (ap-northeast-1c) subnet-02e70ec727156b94f (ap-northeast-1d) subnet-0d529a446897f24a2 (ap-northeast-1a)]        {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:20:36.818Z        DEBUG   controller.provisioning Discovered EC2 instance types zonal offerings   {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:25:33.626Z        INFO    controller.node Added TTL to empty node {"commit": "7e79a67", "node": "ip-192-168-133-127.ap-northeast-1.compute.internal"}
2022-01-17T21:25:37.567Z        DEBUG   controller.provisioning Discovered 324 EC2 instance types       {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:25:37.713Z        DEBUG   controller.provisioning Discovered subnets: [subnet-021d3b8da085b55a9 (ap-northeast-1c) subnet-02e70ec727156b94f (ap-northeast-1d) subnet-0d529a446897f24a2 (ap-northeast-1a)]        {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:25:37.845Z        DEBUG   controller.provisioning Discovered EC2 instance types zonal offerings   {"commit": "7e79a67", "provisioner": "default"}
2022-01-17T21:26:03.646Z        INFO    controller.node Triggering termination after 30s for empty node {"commit": "7e79a67", "node": "ip-192-168-133-127.ap-northeast-1.compute.internal"}
2022-01-17T21:26:03.685Z        INFO    controller.termination  Cordoned node   {"commit": "7e79a67", "node": "ip-192-168-133-127.ap-northeast-1.compute.internal"}
2022-01-17T21:26:03.964Z        INFO    controller.termination  Deleted node    {"commit": "7e79a67", "node": "ip-192-168-133-127.ap-northeast-1.compute.internal"}

ノードを確認する。削除された。

$ k get node -o wide
NAME                                                STATUS   ROLES    AGE   VERSION               INTERNAL-IP      EXTERNAL-IP    OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
ip-192-168-72-151.ap-northeast-1.compute.internal   Ready    <none>   24h   v1.21.5-eks-bc4871b   192.168.72.151   54.92.61.102   Amazon Linux 2   5.4.162-86.275.amzn2.x86_64   docker://20.10.7

EC2 Fleet API との関連については、describe-fleets で ID を指定してみるとちゃんととれた。Spot Fleet からではなく、EC2 Fleet からスポットインスタンスを起動しているということらしい。

$ aws ec2 describe-fleets --fleet-ids fleet-728040e2-5e26-0924-843a-ab8a64977bca
{
    "Fleets": [
        {
            "ActivityStatus": "fulfilled",
            "CreateTime": "2022-01-18T00:48:51+00:00",
            "FleetId": "fleet-728040e2-5e26-0924-843a-ab8a64977bca",
            "FleetState": "active",
            "ClientToken": "EC2Fleet-7b91c7a1-6281-4d8c-a02d-cd930fefe667",
            "FulfilledCapacity": 1.0,
            "FulfilledOnDemandCapacity": 0.0,
            "LaunchTemplateConfigs": [
                {
                    "LaunchTemplateSpecification": {
                        "LaunchTemplateName": "Karpenter-karpenter-demo-17569880151886454907",
                        "Version": "$Default"
                    },
                    "Overrides": [
                        {
                            "InstanceType": "c1.xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 0.0
                        },
                        {
                            "InstanceType": "c1.xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 0.0
                        },
                        {
                            "InstanceType": "c4.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 1.0
                        },
                        {
                            "InstanceType": "c4.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 1.0
                        },
                        {
                            "InstanceType": "c4.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 1.0
                        },
                        {
                            "InstanceType": "c3.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 2.0
                        },
                        {
                            "InstanceType": "c3.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 2.0
                        },
                        {
                            "InstanceType": "c5.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 3.0
                        },
                        {
                            "InstanceType": "c5.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 3.0
                        },
                        {
                            "InstanceType": "c5.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 3.0
                        },
                        {
                            "InstanceType": "c5a.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 4.0
                        },
                        {
                            "InstanceType": "c5a.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 4.0
                        },
                        {
                            "InstanceType": "c5d.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 5.0
                        },
                        {
                            "InstanceType": "c5d.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 5.0
                        },
                        {
                            "InstanceType": "c5d.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 5.0
                        },
                        {
                            "InstanceType": "c6i.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 6.0
                        },
                        {
                            "InstanceType": "c6i.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 6.0
                        },
                        {
                            "InstanceType": "c5n.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 7.0
                        },
                        {
                            "InstanceType": "c5n.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 7.0
                        },
                        {
                            "InstanceType": "c5n.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 7.0
                        },
                        {
                            "InstanceType": "m3.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 8.0
                        },
                        {
                            "InstanceType": "m3.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 8.0
                        },
                        {
                            "InstanceType": "m5n.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 9.0
                        },
                        {
                            "InstanceType": "m5n.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 9.0
                        },
                        {
                            "InstanceType": "m5n.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 9.0
                        },
                        {
                            "InstanceType": "m5dn.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 10.0
                        },
                        {
                            "InstanceType": "m5dn.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 10.0
                        },
                        {
                            "InstanceType": "m5dn.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 10.0
                        },
                        {
                            "InstanceType": "m5.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 11.0
                        },
                        {
                            "InstanceType": "m5.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 11.0
                        },
                        {
                            "InstanceType": "m5.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 11.0
                        },
                        {
                            "InstanceType": "m5a.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 12.0
                        },
                        {
                            "InstanceType": "m5a.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 12.0
                        },
                        {
                            "InstanceType": "m5a.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 12.0
                        },
                        {
                            "InstanceType": "m6i.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 13.0
                        },
                        {
                            "InstanceType": "m6i.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 13.0
                        },
                        {
                            "InstanceType": "m5zn.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 14.0
                        },
                        {
                            "InstanceType": "m5zn.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 14.0
                        },
                        {
                            "InstanceType": "t3.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 15.0
                        },
                        {
                            "InstanceType": "t3.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 15.0
                        },
                        {
                            "InstanceType": "t3.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 15.0
                        },
                        {
                            "InstanceType": "m5ad.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 16.0
                        },
                        {
                            "InstanceType": "m5ad.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 16.0
                        },
                        {
                            "InstanceType": "m5ad.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 16.0
                        },
                        {
                            "InstanceType": "m5d.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 17.0
                        },
                        {
                            "InstanceType": "m5d.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 17.0
                        },
                        {
                            "InstanceType": "m5d.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 17.0
                        },
                        {
                            "InstanceType": "m4.2xlarge",
                            "SubnetId": "subnet-021d3b8da085b55a9",
                            "AvailabilityZone": "ap-northeast-1c",
                            "Priority": 18.0
                        },
                        {
                            "InstanceType": "m4.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 18.0
                        },
                        {
                            "InstanceType": "m4.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 18.0
                        },
                        {
                            "InstanceType": "t3a.2xlarge",
                            "SubnetId": "subnet-0d529a446897f24a2",
                            "AvailabilityZone": "ap-northeast-1a",
                            "Priority": 19.0
                        },
                        {
                            "InstanceType": "t3a.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 19.0
                        }
                    ]
                }
            ],
            "TargetCapacitySpecification": {
                "TotalTargetCapacity": 1,
                "OnDemandTargetCapacity": 0,
                "SpotTargetCapacity": 0,
                "DefaultTargetCapacityType": "spot"
            },
            "TerminateInstancesWithExpiration": false,
            "Type": "instant",
            "ValidFrom": "1970-01-01T00:00:00+00:00",
            "ValidUntil": "1970-01-01T00:00:00+00:00",
            "ReplaceUnhealthyInstances": false,
            "SpotOptions": {
                "AllocationStrategy": "capacity-optimized-prioritized"
            },
            "OnDemandOptions": {
                "AllocationStrategy": "lowest-price"
            },
            "Errors": [],
            "Instances": [
                {
                    "LaunchTemplateAndOverrides": {
                        "LaunchTemplateSpecification": {
                            "LaunchTemplateId": "lt-0773aa9ea2bdcafc1",
                            "Version": "1"
                        },
                        "Overrides": {
                            "InstanceType": "t3a.2xlarge",
                            "SubnetId": "subnet-02e70ec727156b94f",
                            "AvailabilityZone": "ap-northeast-1d",
                            "Priority": 19.0
                        }
                    },
                    "Lifecycle": "spot",
                    "InstanceIds": [
                        "i-0021fb5c588e50edd"
                    ],
                    "InstanceType": "t3a.2xlarge"
                }
            ]
        }
    ]
}

PodSecurity アドミッションを試す

1.23 でベータに昇格した PodSecurity アドミッションを以下のブログにしたがって試す。

準備

Kind でクラスターを起動する。

$ kind create cluster --image kindest/node:v1.23.0
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.23.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

クラスターを確認する。

$ kubectl cluster-info --context kind-kind
Kubernetes control plane is running at https://127.0.0.1:57523
CoreDNS is running at https://127.0.0.1:57523/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl get nodes
NAME                 STATUS   ROLES                  AGE   VERSION
kind-control-plane   Ready    control-plane,master   11m   v1.23.0

PodSecurity アドミッションがデフォルトで有効のリストに入っていることを確認する。

$ kubectl -n kube-system exec kube-apiserver-kind-control-plane -it -- kube-apiserver -h | grep "default enabled ones"
      --enable-admission-plugins strings       admission plugins that should be enabled in addition to default enabled ones (NamespaceLifecycle, LimitRanger, ServiceAccount, TaintNodesByCondition, PodSecurity, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, RuntimeClass, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, ResourceQuota). Comma-delimited list of admission plugins: AlwaysAdmit, AlwaysDeny, AlwaysPullImages, CertificateApproval, CertificateSigning, CertificateSubjectRestriction, DefaultIngressClass, DefaultStorageClass, DefaultTolerationSeconds, DenyServiceExternalIPs, EventRateLimit, ExtendedResourceToleration, ImagePolicyWebhook, LimitPodHardAntiAffinityTopology, LimitRanger, MutatingAdmissionWebhook, NamespaceAutoProvision, NamespaceExists, NamespaceLifecycle, NodeRestriction, OwnerReferencesPermissionEnforcement, PersistentVolumeClaimResize, PersistentVolumeLabel, PodNodeSelector, PodSecurity, PodSecurityPolicy, PodTolerationRestriction, Priority, ResourceQuota, RuntimeClass, SecurityContextDeny, ServiceAccount, StorageObjectInUseProtection, TaintNodesByCondition, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.

Namespace を作成する。

$ kubectl create namespace verify-pod-security
namespace/verify-pod-security created

Privileged レベル

Privileged レベルという章タイトルだが、実際には Baseline レベルの Pod なので注意。

Namespace にラベルを付与する。restricted を強制し、監査ログもとる。

$ kubectl label --overwrite ns verify-pod-security \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted
namespace/verify-pod-security labeled
キー
pod-security.kubernetes.io/enforce restricted
pod-security.kubernetes.io/audit restricted
pod-security.kubernetes.io/warn 未指定

allowPrivilegeEscalation: truebaseline では許可されているが、restricted では禁止なので、これから作ろうとしている Pod は baseline な Pod である。ここでは特権 Pod を作るというテストをやろうとしているように思えるので、意図からずれていそうに思えるが、後半に以下の記載があった。

UPDATE: The baseline policy permits allowPrivilegeEscalation. While I cannot see the Pod Security default levels of enforcement, they are there. Let's try to provide a manifest that violates the baseline by requesting hostNetwork access.

baseline な Pod を作成する。

$ cat <<EOF | kubectl -n verify-pod-security apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-privileged
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      allowPrivilegeEscalation: true
EOF
Error from server (Forbidden): error when creating "STDIN": pods "busybox-privileged" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

restricted に違反するので作成できない。

ラベルを変更する。privileged を強制(すべて許可)し、baseline に違反した場合に警告とログを取る。

$ kubectl label --overwrite ns verify-pod-security \
  pod-security.kubernetes.io/enforce=privileged \
  pod-security.kubernetes.io/warn=baseline \
  pod-security.kubernetes.io/audit=baseline
namespace/verify-pod-security labeled
キー
pod-security.kubernetes.io/enforce privileged
pod-security.kubernetes.io/audit baseline
pod-security.kubernetes.io/warn baseline

もう一度 baseline な Pod を作成する。

$ cat <<EOF | kubectl -n verify-pod-security apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-privileged
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      allowPrivilegeEscalation: true
EOF
pod/busybox-privileged created

privileged に違反していないので作成でき、baseline に違反していないので警告も出ない。

Pod を削除する。

$ kubectl -n verify-pod-security delete pod busybox-privileged
pod "busybox-privileged" deleted

Baseline レベル

ラベルを変更し、restricted を強制する。最初と同じだが、warn のレベルを指定していないので、そこは前のステップで指定した baseline になっている。

$ kubectl label --overwrite ns verify-pod-security \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted
namespace/verify-pod-security labeled
キー
pod-security.kubernetes.io/enforce restricted
pod-security.kubernetes.io/audit restricted
pod-security.kubernetes.io/warn baseline

baseline な Pod を作成する。

$ cat <<EOF | kubectl -n verify-pod-security apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-baseline
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
          - NET_BIND_SERVICE
          - CHOWN
EOF
Error from server (Forbidden): error when creating "STDIN": pods "busybox-baseline" is forbidden: violates PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]; container "busybox" must not include "CHOWN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

restricted に違反するので作成できない。

ラベルを再び変更し、baseline を強制する。warn のレベルは restricted に変更する。

$ kubectl label --overwrite ns verify-pod-security \
  pod-security.kubernetes.io/enforce=baseline \
  pod-security.kubernetes.io/warn=restricted \
  pod-security.kubernetes.io/audit=restricted
namespace/verify-pod-security labeled
キー
pod-security.kubernetes.io/enforce baseline
pod-security.kubernetes.io/audit restricted
pod-security.kubernetes.io/warn restricted

baseline な Pod を作成する。

$ cat <<EOF | kubectl -n verify-pod-security apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-baseline
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
          - NET_BIND_SERVICE
          - CHOWN
EOF
Warning: would violate PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]; container "busybox" must not include "CHOWN" in securityContext.capabilities.add), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/busybox-baseline created

baseline に違反していないので作成できるが、restricted に違反するので警告が出る。

Pod を削除する。

$ kubectl -n verify-pod-security delete pod busybox-baseline
pod "busybox-baseline" deleted

Restricted レベル

再びラベルを変更して restricted を強制する。

$ kubectl label --overwrite ns verify-pod-security \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/audit=restricted
namespace/verify-pod-security labeled
キー
pod-security.kubernetes.io/enforce restricted
pod-security.kubernetes.io/audit restricted
pod-security.kubernetes.io/warn restricted

baseline な Pod を作成する。

$ cat <<EOF | kubectl -n verify-pod-security apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-restricted
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        add:
          - NET_BIND_SERVICE
EOF
Error from server (Forbidden): error when creating "STDIN": pods "busybox-restricted" is forbidden: violates PodSecurity "restricted:latest": unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

restricted に違反するので作成できない。

マニフェストを変更して restricted に適合するようにして Pod を作成する。

$ cat <<EOF | kubectl -n verify-pod-security apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-restricted
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
        add:
          - NET_BIND_SERVICE
EOF
pod/busybox-restricted created

Pod は作成できたが、実行に失敗している。

$ kubectl -n verify-pod-security get pods
NAME                 READY   STATUS                       RESTARTS   AGE
busybox-restricted   0/1     CreateContainerConfigError   0          48s

Pod の詳細を確認する。

$ kubectl -n verify-pod-security describe pod busybox-restricted
Name:         busybox-restricted
Namespace:    verify-pod-security
...
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  103s                default-scheduler  Successfully assigned verify-pod-security/busybox-restricted to kind-control-plane
  Normal   Pulled     101s                kubelet            Successfully pulled image "busybox" in 1.7259118s
  Normal   Pulled     99s                 kubelet            Successfully pulled image "busybox" in 1.7005545s
  Normal   Pulled     86s                 kubelet            Successfully pulled image "busybox" in 1.5442063s
  Normal   Pulled     69s                 kubelet            Successfully pulled image "busybox" in 1.6893928s
  Normal   Pulled     56s                 kubelet            Successfully pulled image "busybox" in 1.6754146s
  Normal   Pulled     40s                 kubelet            Successfully pulled image "busybox" in 1.5302337s
  Normal   Pulled     25s                 kubelet            Successfully pulled image "busybox" in 1.661749s
  Normal   Pulling    11s (x8 over 103s)  kubelet            Pulling image "busybox"
  Warning  Failed     9s (x8 over 101s)   kubelet            Error: container has runAsNonRoot and image will run as root (pod: "busybox-restricted_verify-pod-security(b6afe0b4-83c8-4838-9444-4f8969444c68)", container: busybox)
  Normal   Pulled     9s                  kubelet            Successfully pulled image "busybox" in 1.6799024s

runAsNonRoot: true を指定しているが、root で実行されているのでエラーになっている。

Pod を一旦削除する。

$ kubectl -n verify-pod-security delete pod busybox-restricted
pod "busybox-restricted" deleted

runAsUser を指定して Pod を再作成する。

$ cat <<EOF | kubectl -n verify-pod-security apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-restricted
spec:
  securityContext:
    runAsUser: 65534
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      seccompProfile:
        type: RuntimeDefault
      runAsNonRoot: true
      allowPrivilegeEscalation: false
      capabilities:
        drop:
          - ALL
        add:
          - NET_BIND_SERVICE
EOF
pod/busybox-restricted created

今度は実行できたことも確認する。

$ kubectl -n verify-pod-security get pods
NAME                 READY   STATUS    RESTARTS   AGE
busybox-restricted   1/1     Running   0          52s

Namespace を削除する。

$ kubectl delete namespace verify-pod-security
namespace "verify-pod-security" deleted

クラスターワイドでのポリシー適用

クラスター全体での適用には、AdmissionConfiguration の変更が必要だが、ランタイム変更はできないので、クラスターを削除する。

$ kind delete cluster
Deleting cluster "kind" ...

設定ファイルを作成する。デフォルトを設定している。kube-system を例外に指定している。

$ cat <<EOF > pod-security.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: AdmissionConfiguration
plugins:
- name: PodSecurity
  configuration:
    apiVersion: pod-security.admission.config.k8s.io/v1beta1
    kind: PodSecurityConfiguration
    defaults:
      enforce: "baseline"
      enforce-version: "latest"
      audit: "baseline"
      audit-version: "latest"
      warn: "restricted"
      warn-version: "latest"
    exemptions:
      # Array of authenticated usernames to exempt.
      usernames: []
      # Array of runtime class names to exempt.
      runtimeClasses: []
      # Array of namespaces to exempt.
      namespaces: [kube-system]
EOF

これを API server に渡す Kind クラスターの設定ファイルを作成する。

cat <<EOF > kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    apiServer:
        # enable admission-control-config flag on the API server
        extraArgs:
          admission-control-config-file: /etc/kubernetes/policies/pod-security.yaml
        # mount new file / directories on the control plane
        extraVolumes:
          - name: policies
            hostPath: /etc/kubernetes/policies
            mountPath: /etc/kubernetes/policies
            readOnly: true
            pathType: "DirectoryOrCreate"
  # mount the local file on the control plane
  extraMounts:
  - hostPath: ./pod-security.yaml
    containerPath: /etc/kubernetes/policies/pod-security.yaml
    readOnly: true
EOF

クラスターを作成する。

$ kind create cluster --image kindest/node:v1.23.0 --config kind-config.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.23.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Namespace を作成する。

$ kubectl create namespace test-defaults
namespace/test-defaults created
$ kubectl describe namespace test-defaults
Name:         test-defaults
Labels:       kubernetes.io/metadata.name=test-defaults
Annotations:  <none>
Status:       Active

No resource quota.

No LimitRange resource.

ラベルはないが、以下と同等がデフォルトになっているはずである。

キー
pod-security.kubernetes.io/enforce baseline
pod-security.kubernetes.io/audit baseline
pod-security.kubernetes.io/warn restricted

baseline レベルの Pod を作成する。

$ cat <<EOF | kubectl -n test-defaults apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-privileged
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
    securityContext:
      allowPrivilegeEscalation: true
EOF
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "busybox" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "busybox" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "busybox" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "busybox" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/busybox-privileged created

baseline に違反していないので作成できるが、restricted に違反するので警告が出ている。

Pod を削除する。

$ kubectl -n test-defaults delete pod/busybox-privileged
pod "busybox-privileged" deleted

今度は privileged な Pod を作成する。

$ cat <<EOF | kubectl -n test-defaults apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox-privileged
spec:
  containers:
  - name: busybox
    image: busybox
    args:
    - sleep
    - "1000000"
  hostNetwork: true
EOF
Error from server (Forbidden): error when creating "STDIN": pods "busybox-privileged" is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true)

baseline に違反するので確かに拒否された。

メトリクスを見てみる。

$ kubectl get --raw /metrics | grep pod_security_evaluations_total
# HELP pod_security_evaluations_total [ALPHA] Number of policy evaluations that occurred, not counting ignored or exempt requests.
# TYPE pod_security_evaluations_total counter
pod_security_evaluations_total{decision="allow",mode="enforce",policy_level="baseline",policy_version="latest",request_operation="create",resource="pod",subresource=""} 2
pod_security_evaluations_total{decision="allow",mode="enforce",policy_level="privileged",policy_version="latest",request_operation="create",resource="pod",subresource=""} 0
pod_security_evaluations_total{decision="allow",mode="enforce",policy_level="privileged",policy_version="latest",request_operation="update",resource="pod",subresource=""} 0
pod_security_evaluations_total{decision="deny",mode="audit",policy_level="baseline",policy_version="latest",request_operation="create",resource="pod",subresource=""} 1
pod_security_evaluations_total{decision="deny",mode="enforce",policy_level="baseline",policy_version="latest",request_operation="create",resource="pod",subresource=""} 1
pod_security_evaluations_total{decision="deny",mode="warn",policy_level="restricted",policy_version="latest",request_operation="create",resource="controller",subresource=""} 2
pod_security_evaluations_total{decision="deny",mode="warn",policy_level="restricted",policy_version="latest",request_operation="create",resource="pod",subresource=""} 2

監査ログを確認したいところだが、監査ログの有効化が必要なので、今回はスキップ。

ワークロードリソース

Pod ではなく Deployment だとどうなのか確認する。

privileged な Pod を持つ Deployment を作成する。

$ cat <<EOF | kubectl -n test-defaults apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      hostNetwork: true
      containers:
      - image: nginx
        name: nginx
EOF
Warning: would violate PodSecurity "restricted:latest": host namespaces (hostNetwork=true), allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx created

警告は出すが Deployment リソース自体は作成された。当然 Pod は作成されない。

$ k get deploy -n test-defaults
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   0/1     0            0           73s
$ k get po -n test-defaults
No resources found in test-defaults namespace.

Deployment を削除する。

$ k delete deploy nginx -n test-defaults
deployment.apps "nginx" deleted

baseline な Pod を持つ Deployment を作成する。

$ cat <<EOF | kubectl -n test-defaults apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        securityContext:
          allowPrivilegeEscalation: true
EOF
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "nginx" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/nginx created

この場合も警告が出された。Pod は作成される。

$ k get deploy -n test-defaults
NAME    READY   UP-TO-DATE   AVAILABLE   AGE
nginx   1/1     1            1           119s
$ k get po -n test-defaults
NAME                     READY   STATUS    RESTARTS   AGE
nginx-688777c4cf-ftn7m   1/1     Running   0          2m

つまり、warn は Deployemnt にも有効だが、enforce は有効でないと考えてよさそう。

読書メモ: マンガでわかる! 仮説思考

会社のSlackのコンサルタントとしてのスキルを高めるにはどうしたらよいのかというスレッドの中で紹介された中で一番とっつきやすそうだったので買った本。

何事にも慎重な日本人は、情報収集や分析を重視し、意志決定を先延ばししがちです。

ちょうど自分の資料作成方法が、この状態になっていると感じた。時間ばかりかかって何もできあがらない。いろいろ調べてから資料を作ろうとするのではなく、資料のアウトラインを先に作ってそこを埋めてくべきだ。

選択肢を広げる情報に価値はない

最近のプロジェクトで、お客様から、いろいろ選択肢はあるけれど、結局どうしたらよいのかとよく言われていた。選択肢を掲示してお客様に選んでもらうのでよい場面もあるかも知れないが、選択肢を絞ってこうすべきと言えるのがコンサルタントの価値である。

論点思考とは、解くべき問題(論点)を定義し発見すること

解決できないときは、そもそも問題設定を間違えている可能性があるため、視座を上げて問題を考える必要がある。イシューツリーを書こう。

上司や顧客から問題点の改善を求められたとき、その依頼主があまり関心を持っていないところ、気づいていないところに真の問題が潜んでいることが多い

これはCustomer Obsessionとしてよく言われていることだ。

コンサルタントは、解決できて、簡単に実行でき、効果の高い問題のことを、「筋がいい問題(論点)だ」と表現します。

この3つの観点でも問題設定を見直すことが必要。

半分はマンガだが、読書時間は1時間半くらい。読み飛ばしてはおらず、わりとちゃんと読んでしまった。

sysbencnでMySQLに負荷をかけるメモ

Cloud9からAurora MySQLにsysbenchで負荷をかけるメモ。

DevOps Guru for RDSで何らかのインサイトを出したいがために実施。

参考リンク

手順

Cloud9からAuroraに接続できるようにする。セキュリティグループの設定を忘れずに。

Admin:~/environment $ ENDPOINT=<エンドポイント>
Admin:~/environment $ mysql -h ${ENDPOINT} -P 3306 -u admin -p
Enter password: 
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MySQL connection id is 18
Server version: 5.7.12 MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MySQL [(none)]> 

mydbデータベースを作っておく。

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
4 rows in set (0.01 sec)

MySQL [(none)]> create database mydb;
Query OK, 1 row affected (0.01 sec)

MySQL [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| mydb               |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
5 rows in set (0.00 sec)

MySQL [(none)]> 

sysbenchをインストールする。

Admin:~/environment $ curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash
Detected operating system as amzn/2.
Checking for curl...
Detected curl...
Downloading repository file: https://packagecloud.io/install/repositories/akopytov/sysbench/config_file.repo?os=amzn&dist=2&source=script
done.
Installing pygpgme to verify GPG signatures...
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
akopytov_sysbench-source/signature                                                                                                                                                                      |  833 B  00:00:00     
Retrieving key from https://packagecloud.io/akopytov/sysbench/gpgkey
Importing GPG key 0x04DCFD39:
 Userid     : "https://packagecloud.io/akopytov/sysbench-prerelease (https://packagecloud.io/docs#gpg_signing) <support@packagecloud.io>"
 Fingerprint: 9789 8d69 f99e e5ca c462 a0f8 cf10 4890 04dc fd39
 From       : https://packagecloud.io/akopytov/sysbench/gpgkey
akopytov_sysbench-source/signature                                                                                                                                                                      | 1.0 kB  00:00:00 !!! 
amzn2-core                                                                                                                                                                                              | 3.7 kB  00:00:00     
akopytov_sysbench-source/primary                                                                                                                                                                        | 2.0 kB  00:00:01     
akopytov_sysbench-source                                                                                                                                                                                                 15/15
234 packages excluded due to repository priority protections
Package pygpgme-0.3-9.amzn2.0.3.x86_64 already installed and latest version
Nothing to do
Installing yum-utils...
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
234 packages excluded due to repository priority protections
Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
Nothing to do
Generating yum cache for akopytov_sysbench...
Importing GPG key 0x04DCFD39:
 Userid     : "https://packagecloud.io/akopytov/sysbench-prerelease (https://packagecloud.io/docs#gpg_signing) <support@packagecloud.io>"
 Fingerprint: 9789 8d69 f99e e5ca c462 a0f8 cf10 4890 04dc fd39
 From       : https://packagecloud.io/akopytov/sysbench/gpgkey
Generating yum cache for akopytov_sysbench-source...

The repository is setup! You can now install packages.
Admin:~/environment $ sudo yum -y install sysbench
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
234 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package sysbench.x86_64 0:1.0.20-1.el7 will be installed
--> Processing Dependency: libpq.so.5()(64bit) for package: sysbench-1.0.20-1.el7.x86_64
--> Running transaction check
---> Package postgresql-libs.x86_64 0:9.2.24-6.amzn2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

===============================================================================================================================================================================================================================
 Package                                                 Arch                                           Version                                                Repository                                                 Size
===============================================================================================================================================================================================================================
Installing:
 sysbench                                                x86_64                                         1.0.20-1.el7                                           akopytov_sysbench                                         430 k
Installing for dependencies:
 postgresql-libs                                         x86_64                                         9.2.24-6.amzn2                                         amzn2-core                                                232 k

Transaction Summary
===============================================================================================================================================================================================================================
Install  1 Package (+1 Dependent package)

Total download size: 662 k
Installed size: 1.8 M
Downloading packages:
(1/2): postgresql-libs-9.2.24-6.amzn2.x86_64.rpm                                                                                                                                                        | 232 kB  00:00:00     
(2/2): sysbench-1.0.20-1.el7.x86_64.rpm                                                                                                                                                                 | 430 kB  00:00:00     
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                          983 kB/s | 662 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : postgresql-libs-9.2.24-6.amzn2.x86_64                                                                                                                                                                       1/2 
  Installing : sysbench-1.0.20-1.el7.x86_64                                                                                                                                                                                2/2 
  Verifying  : sysbench-1.0.20-1.el7.x86_64                                                                                                                                                                                1/2 
  Verifying  : postgresql-libs-9.2.24-6.amzn2.x86_64                                                                                                                                                                       2/2 

Installed:
  sysbench.x86_64 0:1.0.20-1.el7                                                                                                                                                                                               

Dependency Installed:
  postgresql-libs.x86_64 0:9.2.24-6.amzn2                                                                                                                                                                                      

Complete!

テーブルを作成する。

PASSWORD=<パスワード>
sysbench --db-driver=mysql \
      --mysql-host=$ENDPOINT  \
      --mysql-user=admin \
      --mysql-password=$PASSWORD \
      --mysql-db=mydb \
      oltp_read_write \
      prepare
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Creating table 'sbtest1'...
Inserting 10000 records into 'sbtest1'
Creating a secondary index on 'sbtest1'...

負荷をかける。

sysbench --db-driver=mysql \
      --mysql-host=$ENDPOINT  \
      --mysql-user=admin \
      --mysql-password=$PASSWORD \
      --mysql-db=mydb \
      oltp_read_write \
      run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 1
Initializing random number generator from current time


Initializing worker threads...

Threads started!

SQL statistics:
    queries performed:
        read:                            3486
        write:                           996
        other:                           498
        total:                           4980
    transactions:                        249    (24.80 per sec.)
    queries:                             4980   (496.10 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          10.0364s
    total number of events:              249

Latency (ms):
         min:                                   37.91
         avg:                                   40.30
         max:                                   48.68
         95th percentile:                       43.39
         sum:                                10033.87

Threads fairness:
    events (avg/stddev):           249.0000/0.00
    execution time (avg/stddev):   10.0339/0.00

これだと同時実行数が1なのでDB Load(平均アクティブセッション数)をもとにインサイトを出すDevOps Guru for RDSは何も出さないと思われる。

実行時間を5分、同時実行数を20にしてみる。

sysbench --db-driver=mysql \
      --mysql-host=$ENDPOINT  \
      --mysql-user=admin \
      --mysql-password=$PASSWORD \
      --mysql-db=mydb \
      --threads=20 \
      --time=300 \
      oltp_read_write \
      run
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 20
Initializing random number generator from current time


Initializing worker threads...

Threads started!

SQL statistics:
    queries performed:
        read:                            1682296
        write:                           467487
        other:                           235659
        total:                           2385442
    transactions:                        115495 (384.92 per sec.)
    queries:                             2385442 (7950.15 per sec.)
    ignored errors:                      4669   (15.56 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          300.0482s
    total number of events:              115495

Latency (ms):
         min:                                   38.30
         avg:                                   51.95
         max:                                  204.73
         95th percentile:                       75.82
         sum:                              6000173.52

Threads fairness:
    events (avg/stddev):           5774.7500/15.97
    execution time (avg/stddev):   300.0087/0.02

パフォーマンスインサイトで負荷がかかったことが確認できた。

f:id:sotoiwa:20220106180231p:plain

さらにスレッドを100、300と増やしてみたところ、DevOps Guruのインサイトも出てきた。しかしこれはDB Loadではないので、DevOps Guru for RDSのインサイトではなく、普通のDevOps Guruのインサイトのように思える。

f:id:sotoiwa:20220106182031p:plain

f:id:sotoiwa:20220106182046p:plain

f:id:sotoiwa:20220106182059p:plain

読書メモ: 遅読家のための読書術

積ん読がたまっているので、どうにか速くたくさん本を読めないか読んだ本のメモ。

主張としては、真面目に本を読まず、フローとして音楽を聴くように本を読むべしということ。

本書は、これと同じように「音楽を聴くように本が読める」状態を作っていくことをゴールにしています。

遅読家というのは、読書に対する「真面目さ」を捨て切れない人

そして、息を吸うだけではなく吐くことが必要、すなわちアウトプットが必要で、そのためのやり方も述べている。この主張はとても納得できたので、こうやってこのブログに書いている。

読書を呼吸として捉える場合、読む(吸う)だけでなく、書く(吐く)ことが大切

引用をすることによって、その本のどこに心動かされたのか、どんな文章が気になったのかか可視化されます。

アウトプットのやり方としては引用を重視し、本の中で一番印象に残った文章を引用し、それについての感想を書くことを進めている。なのでこのブログでも引用を多用している。

まず、1ラインエッセンスを書き写し、その下に30〜40字程度で1ラインレビューメモをしましょう。

1日1冊、だいたい60分くらいで1冊の本を読み、毎週6冊の本を読み、12冊毎にレビューの振り返りをすることをおすすめしているが、なかなかそこまではできないが、なるべく本を気楽に読んで、読んだ後は引用を使ったメモを残していこうと思う。

この本はだいたい2時間くらいで読めた。

CDKのアップデート

CDKのアップデートがいつもよくわからなくなるのでメモ。

参考リンク

CDK CLIのアップデート

npm-check-updatesを導入する。

npm install -g npm-check-updates

GlobalにインストールされているCDKのアップデートをチェックする。

$ ncu -g aws-cdk                  
[====================] 1/1 100%

 aws-cdk  2.1.0  →  2.2.0     

ncu itself cannot upgrade global packages. Run the following to upgrade all global packages: 

npm -g install aws-cdk@2.2.0

アップデートする。

npm -g install aws-cdk@2.2.0
# 以下でもたぶん結果は同じ
# npm -g install aws-cdk
# npm -g update aws-cdk

プロジェクトのCDKの依存性のアップデート

  • npm outdatedは古いパッケージを教えてくれる
  • npm updateは更新を行うが、package.jsonを更新してくれない
  • npm update^付きのバージョン指定(互換性のあるバージョン指定)の場合にメジャーバージョンアップをしない
  • npm audit fix脆弱性のあるときしか使えないが、package.jsonは更新してくれるが、メジャーバージョンアップをしない
  • npm audit fix --force脆弱性のあるときしか使えないが、package.jsonは更新してくれ、メジャーバージョンアップもする

いずれも完全ではないので、npm-check-updatesを使う方がよい。

更新をチェックする。

ncu

package.jsonを書き換える。特定のパッケージのみだけ書き換えたいときは引数でパッケージ名を指定する。

ncu -u

書き換えたpackage.jsonをもとにパッケージをインストールする。

npm install

不要になった依存性の削除

depcheckをインストールする。

npm install -g depcheck

不要な依存性をチェックする。

$ depcheck                    
Unused devDependencies
* @types/node
* ts-node

不要なものはuninstallする。