ADOTとContainer Insights Prometheusを試す

以下のAWSブログにしたがってAWS Distro for OpenTelemetryでPrometheusメトリクスをCloudWatchに送る方法を試したメモ。

• Introducing CloudWatch Container Insights Prometheus Support with AWS Distro for OpenTelemetry on Amazon ECS and Amazon EKS

クラスターの作成

1.21でクラスターを作成する。ノードなしで作成する。

cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: adot
  region: ap-northeast-1
  version: "1.21"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true
EOF

eksctl create cluster -f cluster.yaml

ノードを作成する。

cat << "EOF" > managed-ng-1.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: adot
  region: ap-northeast-1

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EOF

eksctl create nodegroup -f managed-ng-1.yaml

Adminロールにも権限をつけておく。

CLUSTER_NAME="adot"
USER_NAME="Admin:{{SessionName}}"
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/Admin"
eksctl create iamidentitymapping --cluster ${CLUSTER_NAME} --arn ${ROLE_ARN} --username ${USER_NAME} --group system:masters

ADOT のデプロイ

オールインワンの設定を使ってデプロイする。

export CLUSTER_NAME=adot
export AWS_REGION=ap-northeast-1
wget https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/eks/otel-container-insights-prometheus.yaml
cat otel-container-insights-prometheus.yaml |sed "s/{{region}}/$AWS_REGION/g" | sed "s/{{cluster_name}}/$CLUSTER_NAME/g" | kubectl apply -f -

namespace/aws-otel-eks created
serviceaccount/aws-otel-collector created
clusterrole.rbac.authorization.k8s.io/otel-prometheus-role created
clusterrolebinding.rbac.authorization.k8s.io/otel-prometheus-role-binding created
deployment.apps/aws-otel-collector created

Podを確認する。

$ k get po -n aws-otel-eks
NAME                                  READY   STATUS    RESTARTS   AGE
aws-otel-collector-685565dd6c-gmjcn   1/1     Running   0          10m

どのIAM権限が必要なのかはっきりわからない。

• https://github.com/aws-observability/aws-otel-collector/blob/main/docs/developers/eks-demo.md
• https://aws-otel.github.io/docs/getting-started/container-insights/eks-prometheus
• https://aws-otel.github.io/docs/setup/eks

CloudWatchAgentServerPolicyをIRSAでアタッチする。

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=adot
NAMESPACE=aws-otel-eks
SERVICE_ACCOUNT=aws-otel-collector
POLICY_ARN="arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
eksctl create iamserviceaccount \
    --name ${SERVICE_ACCOUNT} \
    --namespace ${NAMESPACE} \
    --cluster ${CLUSTER_NAME} \
    --attach-policy-arn ${POLICY_ARN} \
    --override-existing-serviceaccounts \
    --approve

Podを再起動する。

k delete -n aws-otel-eks po --all

サンプルアプリケーションのデプロイ

• https://github.com/aws-observability/aws-otel-collector/blob/main/docs/developers/container-insights-eks-jmx.md

リポジトリをクローンする。

git clone https://github.com/aws-observability/aws-otel-test-framework.git
cd aws-otel-test-framework/sample-apps/jmx/

ECRにログインする。

AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
AWS_REGION=$(aws configure get region)
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com

リポジトリを作成する。

aws ecr create-repository --repository-name prometheus-sample-tomcat-jmx \
  --image-scanning-configuration scanOnPush=true \
  --region ${AWS_REGION}

イメージをビルドしてPushする。

docker build -t ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/prometheus-sample-tomcat-jmx:latest .
docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/prometheus-sample-tomcat-jmx:latest

アプリケーションをデプロイする。

SAMPLE_TRAFFIC_NAMESPACE=javajmx-sample
curl https://raw.githubusercontent.com/aws-observability/aws-otel-test-framework/terraform/sample-apps/jmx/examples/prometheus-metrics-sample.yaml |
sed "s/{{aws_account_id}}/${AWS_ACCOUNT_ID}/g" |
sed "s/{{region}}/${AWS_REGION}/g" |
sed "s/{{namespace}}/$SAMPLE_TRAFFIC_NAMESPACE/g" |
kubectl apply -f -

namespace/javajmx-sample created
deployment.apps/tomcat-example created
service/tomcat-war-example created
pod/tomcat-traffic-generator created
pod/tomcat-bad-traffic-generator created

Podを確認する。

$ k get po -n javajmx-sample
NAME                              READY   STATUS    RESTARTS   AGE
tomcat-bad-traffic-generator      1/1     Running   0          5m39s
tomcat-example-78cf48d869-khb6n   1/1     Running   0          5m39s
tomcat-traffic-generator          1/1     Running   0          5m39s

CloudWatchコンソールを確認する。

パフォーマンスログが送られ、メトリクスはとれているが、Container Insightsのダッシュボードにadotクラスターが出てこない。普通のContainer Insightsのメトリクス収集をデプロイする必要がありそう。

先ほどデプロイしたPrometheus用のものと、普通のコンテナメトリクス用のマニフェストが以下にあるが、独立していそう。

• https://github.com/aws-observability/aws-otel-collector/tree/main/deployment-template/eks

ServiceAccountも違うし、DaemonSetか、Deploymentかという点でも違う。

デプロイする。

kubectl apply -f https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/eks/otel-container-insights-infra.yaml

namespace/aws-otel-eks unchanged
serviceaccount/aws-otel-sa created
clusterrole.rbac.authorization.k8s.io/aoc-agent-role created
clusterrolebinding.rbac.authorization.k8s.io/aoc-agent-role-binding created
configmap/otel-agent-conf created
daemonset.apps/aws-otel-eks-ci created

Podを確認する。

$ k get po -n aws-otel-eks
NAME                                  READY   STATUS    RESTARTS   AGE
aws-otel-collector-685565dd6c-crh5f   1/1     Running   0          24m
aws-otel-eks-ci-4tsc5                 1/1     Running   0          57s
aws-otel-eks-ci-lgp9r                 1/1     Running   0          57s

CloudWatchAgentServerPolicyをIRSAでアタッチする。

AWD_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=adot
NAMESPACE=aws-otel-eks
SERVICE_ACCOUNT=aws-otel-sa
POLICY_ARN="arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
eksctl create iamserviceaccount \
    --name ${SERVICE_ACCOUNT} \
    --namespace ${NAMESPACE} \
    --cluster ${CLUSTER_NAME} \
    --attach-policy-arn ${POLICY_ARN} \
    --override-existing-serviceaccounts \
    --approve

Podを再起動しておく。

k -n aws-otel-eks delete po --all

これでadotクラスターが出てくるようになった。

JMXのメトリクスも見られた。

EKSでContainerdを試す

EKSでContainerdを試して特にロギングまわりの違いを確認するメモ。

• Amazon EKS が Kubernetes 1.21 のサポートを開始
• https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html#containerd-bootstrap
• What is the difference between docker and containerd about logging?
• 3-shake SRE Tech Talk #2
• https://github.com/aws/containers-roadmap/issues/313

クラスターの作成

クラスターを作成する。

cat << EOF > cluster.yaml
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: containerd
  region: ap-northeast-1
  version: "1.21"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true
EOF

eksctl create cluster -f cluster.yaml

Dockerのノードグループを作成する。

cat << "EOF" > managed-ng-docker.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: containerd
  region: ap-northeast-1

managedNodeGroups:
  - name: managed-ng-docker
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EOF

eksctl create nodegroup -f managed-ng-docker.yaml

Containerdのノードグループを作成する。

EKS_VERSION=1.21
AMI_ID=$(aws ssm get-parameter \
    --name /aws/service/eks/optimized-ami/${EKS_VERSION}/amazon-linux-2/recommended/image_id \
    --query "Parameter.Value" --output text)
CLUSTER_NAME=containerd
cat << EOF > managed-ng-containerd.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ${CLUSTER_NAME}
  region: ap-northeast-1

managedNodeGroups:
  - name: managed-ng-containerd
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    ami: ${AMI_ID}
    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh ${CLUSTER_NAME} --container-runtime containerd
EOF

eksctl create nodegroup -f managed-ng-containerd.yaml

ノードを確認する。

$ k get node -o wide
NAME                                              STATUS   ROLES    AGE   VERSION               INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
ip-10-0-107-68.ap-northeast-1.compute.internal    Ready    <none>   21h   v1.21.2-eks-55daa9d   10.0.107.68    <none>        Amazon Linux 2   5.4.141-67.229.amzn2.x86_64   docker://19.3.13
ip-10-0-112-116.ap-northeast-1.compute.internal   Ready    <none>   23m   v1.21.2-eks-55daa9d   10.0.112.116   <none>        Amazon Linux 2   5.4.141-67.229.amzn2.x86_64   containerd://1.4.6
ip-10-0-88-133.ap-northeast-1.compute.internal    Ready    <none>   23m   v1.21.2-eks-55daa9d   10.0.88.133    <none>        Amazon Linux 2   5.4.141-67.229.amzn2.x86_64   containerd://1.4.6
ip-10-0-93-32.ap-northeast-1.compute.internal     Ready    <none>   21h   v1.21.2-eks-55daa9d   10.0.93.32     <none>        Amazon Linux 2   5.4.141-67.229.amzn2.x86_64   docker://19.3.13

Container Insightsのデプロイ

特にログ収集まわりが気になるので、Container Insightsをデプロイする。

• Amazon EKS と Kubernetes での Container Insights のセットアップ

メトリクス

メトリクスはCloudWatchエージェントかADOTかどちらか選べるようになっている。ADOTでセットアップする。

$ curl https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/eks/otel-container-insights-infra.yaml |
> kubectl apply -f -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6993  100  6993    0     0   8644      0 --:--:-- --:--:-- --:--:--  8665
namespace/aws-otel-eks created
serviceaccount/aws-otel-sa created
clusterrole.rbac.authorization.k8s.io/aoc-agent-role created
clusterrolebinding.rbac.authorization.k8s.io/aoc-agent-role-binding created
configmap/otel-agent-conf created
daemonset.apps/aws-otel-eks-ci created

Podを確認する。

$ kubectl get pods -l name=aws-otel-eks-ci -n aws-otel-eks
NAME                    READY   STATUS    RESTARTS   AGE
aws-otel-eks-ci-6z8jf   1/1     Running   0          54s
aws-otel-eks-ci-8bwwb   1/1     Running   0          54s
aws-otel-eks-ci-gbpxf   1/1     Running   0          54s
aws-otel-eks-ci-znqrv   1/1     Running   0          54s

CloudWatchAgentServerPolicyをIRSAでアタッチする。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=containerd
NAMESPACE=aws-otel-eks
SERVICE_ACCOUNT=aws-otel-sa
POLICY_ARN="arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
eksctl create iamserviceaccount \
    --name ${SERVICE_ACCOUNT} \
    --namespace ${NAMESPACE} \
    --cluster ${CLUSTER_NAME} \
    --attach-policy-arn ${POLICY_ARN} \
    --override-existing-serviceaccounts \
    --approve

Podを再起動しておく。

k -n aws-otel-eks delete po --all

ロギング

fluent-bitによるログ収集をセットアップする。

Namespaceを作成する。

kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml

ConfigMapを作成する。

ClusterName=containerd
RegionName=ap-northeast-1
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch

fluent-bitのDaemonSetを作成する。

$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml
serviceaccount/fluent-bit created
clusterrole.rbac.authorization.k8s.io/fluent-bit-role created
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-role-binding created
configmap/fluent-bit-config created
daemonset.apps/fluent-bit created

CloudWatchAgentServerPolicyをIRSAでアタッチする。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=containerd
NAMESPACE=amazon-cloudwatch
SERVICE_ACCOUNT=fluent-bit
POLICY_ARN="arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
eksctl create iamserviceaccount \
    --name ${SERVICE_ACCOUNT} \
    --namespace ${NAMESPACE} \
    --cluster ${CLUSTER_NAME} \
    --attach-policy-arn ${POLICY_ARN} \
    --override-existing-serviceaccounts \
    --approve

Podを再起動しておく。

k -n amazon-cloudwatch delete po --all

差異の確認

hey-yoをデプロイする。

k create deployment hey-yo --image=public.ecr.aws/toricls/everlasting-hey-yo:latest --replicas=4

確認する。

$ k get po -o wide
NAME                      READY   STATUS    RESTARTS   AGE   IP             NODE                                              NOMINATED NODE   READINESS GATES
hey-yo-5749db6dc8-4rtv5   1/1     Running   0          29s   10.0.86.17     ip-10-0-88-133.ap-northeast-1.compute.internal    <none>           <none>
hey-yo-5749db6dc8-kdkm9   1/1     Running   0          29s   10.0.114.125   ip-10-0-112-116.ap-northeast-1.compute.internal   <none>           <none>
hey-yo-5749db6dc8-kzstv   1/1     Running   0          29s   10.0.101.41    ip-10-0-107-68.ap-northeast-1.compute.internal    <none>           <none>
hey-yo-5749db6dc8-wchcb   1/1     Running   0          29s   10.0.69.158    ip-10-0-93-32.ap-northeast-1.compute.internal     <none>           <none>

CloudWatch Logs

CloudWatch Logsでhey-yoのログを確認する。

ログストリーム名には違いがない。

CloudWatch Logs上で確認できるログイベントはDockerの場合が以下。

{
    "log": "Hey, Yo!\n",
    "stream": "stdout",
    "kubernetes": {
        "pod_name": "hey-yo-5749db6dc8-kzstv",
        "namespace_name": "default",
        "pod_id": "29b38e87-ee7d-4847-be73-2e281429c7bc",
        "host": "ip-10-0-107-68.ap-northeast-1.compute.internal",
        "container_name": "everlasting-hey-yo",
        "docker_id": "370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07",
        "container_hash": "public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",
        "container_image": "public.ecr.aws/toricls/everlasting-hey-yo:latest"
    }
}

containerdの場合が以下。

{
    "log": "2021-09-13T21:30:13.223922107Z stdout F Hey, Yo!",
    "kubernetes": {
        "pod_name": "hey-yo-5749db6dc8-kdkm9",
        "namespace_name": "default",
        "pod_id": "62c0657f-1292-476a-a43e-6b971a7f8909",
        "host": "ip-10-0-112-116.ap-northeast-1.compute.internal",
        "container_name": "everlasting-hey-yo",
        "docker_id": "30aac73f69f88093e5c6191c1361336b021a4661adae375c42048ee5814ee28f",
        "container_hash": "public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",
        "container_image": "public.ecr.aws/toricls/everlasting-hey-yo:latest"
    }
}

logフィールドの内容と、streamフィールドの有無に違いがある。

ノード上のログ

各ノードにログインして直接確認する。

docker

ログローテーションの設定は/etc/docker/daemon.jsonにある。

[ec2-user@ip-10-0-107-68 ~]$ cat /etc/docker/daemon.json
{
  "bridge": "none",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  },
  "live-restore": true,
  "max-concurrent-downloads": 10,
  "default-ulimits": {
    "memlock": {
      "Hard": -1,
      "Name": "memlock",
      "Soft": -1
    }
  }
}

/var/log/containers/は/var/log/pods/へのシンボリックリンクであり、/var/log/pods/も/var/lib/docker/containers/へのシンボリックリンクとなっている。

[ec2-user@ip-10-0-107-68 ~]$ ls -l /var/log/containers/hey-yo-5749db6dc8-kzstv_default_everlasting-hey-yo-370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07.log
lrwxrwxrwx 1 root root 107 Sep 13 21:29 /var/log/containers/hey-yo-5749db6dc8-kzstv_default_everlasting-hey-yo-370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07.log -> /var/log/pods/default_hey-yo-5749db6dc8-kzstv_29b38e87-ee7d-4847-be73-2e281429c7bc/everlasting-hey-yo/0.log

[ec2-user@ip-10-0-107-68 ~]$ ls -l /var/log/pods/default_hey-yo-5749db6dc8-kzstv_29b38e87-ee7d-4847-be73-2e281429c7bc/everlasting-hey-yo/0.log
lrwxrwxrwx 1 root root 165 Sep 13 21:29 /var/log/pods/default_hey-yo-5749db6dc8-kzstv_29b38e87-ee7d-4847-be73-2e281429c7bc/everlasting-hey-yo/0.log -> /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log

[ec2-user@ip-10-0-107-68 ~]$ sudo ls -l /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log
-rw-r----- 1 root root 73117 Sep 13 21:44 /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log

/var/lib/docker/containers/以下のログが実体で、json形式になっている。

[ec2-user@ip-10-0-107-68 ~]$ sudo tail /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:15.410476113Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:16.410806875Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:17.411347021Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:18.411845789Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:19.412326515Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:20.412898298Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:21.413820362Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:22.413840922Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:23.422186054Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:24.414696293Z"}

containerd

こちらのノードも、docker自体はインストールされているが、起動していていない。

[ec2-user@ip-10-0-112-116 ~]$ systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: https://docs.docker.com

ローテーションはkubeletも役割となり、kubeletの設定の起動引数--container-log-max-filesと--container-log-max-sizeで設定できるが、kubeletの引数にはローテーションのオプションはない。デフォルトの5と10Miが使用される。

[ec2-user@ip-10-0-112-116 ~]$ ps -ef | grep kubelet | grep -v grep
root      3342     1  0 Sep12 ?        00:13:02 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime remote --container-runtime-endpoint unix:///run/dockershim.sock --network-plugin cni --node-ip=10.0.112.116 --pod-infra-container-image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1 --v=2

kubeletの設定ファイルにもなし。

[ec2-user@ip-10-0-112-116 ~]$ cat /etc/kubernetes/kubelet/kubelet-config.json
{
  "kind": "KubeletConfiguration",
  "apiVersion": "kubelet.config.k8s.io/v1beta1",
  "address": "0.0.0.0",
  "authentication": {
    "anonymous": {
      "enabled": false
    },
    "webhook": {
      "cacheTTL": "2m0s",
      "enabled": true
    },
    "x509": {
      "clientCAFile": "/etc/kubernetes/pki/ca.crt"
    }
  },
  "authorization": {
    "mode": "Webhook",
    "webhook": {
      "cacheAuthorizedTTL": "5m0s",
      "cacheUnauthorizedTTL": "30s"
    }
  },
  "clusterDomain": "cluster.local",
  "hairpinMode": "hairpin-veth",
  "readOnlyPort": 0,
  "cgroupDriver": "cgroupfs",
  "cgroupRoot": "/",
  "featureGates": {
    "RotateKubeletServerCertificate": true
  },
  "protectKernelDefaults": true,
  "serializeImagePulls": false,
  "serverTLSBootstrap": true,
  "tlsCipherSuites": [
    "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
    "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305",
    "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305",
    "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
    "TLS_RSA_WITH_AES_256_GCM_SHA384",
    "TLS_RSA_WITH_AES_128_GCM_SHA256"
  ],
  "clusterDNS": [
    "172.20.0.10"
  ],
  "evictionHard": {
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },
  "kubeReserved": {
    "cpu": "70m",
    "ephemeral-storage": "1Gi",
    "memory": "574Mi"
  },
  "maxPods": 29
}

/var/log/containers/は/var/log/pods/へのシンボリックリンクであり、/var/log/pods/以下のファイルが実体となっている。

[ec2-user@ip-10-0-112-116 ~]$ ls -l /var/log/containers/hey-yo-5749db6dc8-kdkm9_default_everlasting-hey-yo-30aac73f69f88093e5c6191c1361336b021a4661adae375c42048ee5814ee28f.log
lrwxrwxrwx 1 root root 107 Sep 13 21:29 /var/log/containers/hey-yo-5749db6dc8-kdkm9_default_everlasting-hey-yo-30aac73f69f88093e5c6191c1361336b021a4661adae375c42048ee5814ee28f.log -> /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log

[ec2-user@ip-10-0-112-116 ~]$ ls -l /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log
-rw-r----- 1 root root 89647 Sep 13 22:00 /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log

ログの形式がjsonではなくなっている。FはFullで、分割された場合はPartialのPになるらしい。分割されるサイズは16KiB。

[ec2-user@ip-10-0-112-116 ~]$ sudo tail /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log
2021-09-13T22:00:26.158635117Z stdout F Hey, Yo!
2021-09-13T22:00:27.159121471Z stdout F Hey, Yo!
2021-09-13T22:00:28.159559657Z stdout F Hey, Yo!
2021-09-13T22:00:29.160404057Z stdout F Hey, Yo!
2021-09-13T22:00:30.160645506Z stdout F Hey, Yo!
2021-09-13T22:00:31.16125444Z stdout F Hey, Yo!
2021-09-13T22:00:32.161720548Z stdout F Hey, Yo!
2021-09-13T22:00:33.162336488Z stdout F Hey, Yo!
2021-09-13T22:00:34.163012543Z stdout F Hey, Yo!
2021-09-13T22:00:35.163265301Z stdout F Hey, Yo!

fluent-bitの設定はこれ。

data:
  fluent-bit.conf: |
    [SERVICE]
        Flush                     5
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               ${HTTP_SERVER}
        HTTP_Listen               0.0.0.0
        HTTP_Port                 ${HTTP_PORT}
        storage.path              /var/fluent-bit/state/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 5M
        
    @INCLUDE application-log.conf
    @INCLUDE dataplane-log.conf
    @INCLUDE host-log.conf
  
  application-log.conf: |
    [INPUT]
        Name                tail
        Tag                 application.*
        Exclude_Path        /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        Path                /var/log/containers/*.log
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_container.db
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      ${READ_FROM_HEAD}
    [INPUT]
        Name                tail
        Tag                 application.*
        Path                /var/log/containers/fluent-bit*
        Parser              docker
        DB                  /var/fluent-bit/state/flb_log.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}
    [INPUT]
        Name                tail
        Tag                 application.*
        Path                /var/log/containers/cloudwatch-agent*
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  cwagent_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_cwagent.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}
    [FILTER]
        Name                kubernetes
        Match               application.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_Tag_Prefix     application.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              Off
        Annotations         Off
    [OUTPUT]
        Name                cloudwatch_logs
        Match               application.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/application
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
  dataplane-log.conf: |
    [INPUT]
        Name                systemd
        Tag                 dataplane.systemd.*
        Systemd_Filter      _SYSTEMD_UNIT=docker.service
        Systemd_Filter      _SYSTEMD_UNIT=kubelet.service
        DB                  /var/fluent-bit/state/systemd.db
        Path                /var/log/journal
        Read_From_Tail      ${READ_FROM_TAIL}
    [INPUT]
        Name                tail
        Tag                 dataplane.tail.*
        Path                /var/log/containers/aws-node*, /var/log/containers/kube-proxy*
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_dataplane_tail.db
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_from_Head      ${READ_FROM_HEAD}
    [FILTER]
        Name                modify
        Match               dataplane.systemd.*
        Rename              _HOSTNAME                   hostname
        Rename              _SYSTEMD_UNIT               systemd_unit
        Rename              MESSAGE                     message
        Remove_regex        ^((?!hostname|systemd_unit|message).)*$
    [FILTER]
        Name                aws
        Match               dataplane.*
        imds_version        v1
    [OUTPUT]
        Name                cloudwatch_logs
        Match               dataplane.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/dataplane
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
    
  host-log.conf: |
    [INPUT]
        Name                tail
        Tag                 host.dmesg
        Path                /var/log/dmesg
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_dmesg.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}
    [INPUT]
        Name                tail
        Tag                 host.messages
        Path                /var/log/messages
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_messages.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}
    [INPUT]
        Name                tail
        Tag                 host.secure
        Path                /var/log/secure
        Parser              syslog
        DB                  /var/fluent-bit/state/flb_secure.db
        Mem_Buf_Limit       5MB
        Skip_Long_Lines     On
        Refresh_Interval    10
        Read_from_Head      ${READ_FROM_HEAD}
    [FILTER]
        Name                aws
        Match               host.*
        imds_version        v1
    [OUTPUT]
        Name                cloudwatch_logs
        Match               host.*
        region              ${AWS_REGION}
        log_group_name      /aws/containerinsights/${CLUSTER_NAME}/host
        log_stream_prefix   ${HOST_NAME}.
        auto_create_group   true
        extra_user_agent    container-insights
  parsers.conf: |
    [PARSER]
        Name                docker
        Format              json
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ
    [PARSER]
        Name                syslog
        Format              regex
        Regex               ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key            time
        Time_Format         %b %d %H:%M:%S
    [PARSER]
        Name                container_firstline
        Format              regex
        Regex               (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ
    [PARSER]
        Name                cwagent_firstline
        Format              regex
        Regex               (?<log>(?<="log":")\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=})
        Time_Key            time
        Time_Format         %Y-%m-%dT%H:%M:%S.%LZ

EKS Anywhereを試す

EKS AnywhereとEKS Connectorを雑に試すメモ。

参考リンク

• EKS Anywhereのファーストインプレッション
• EKS AnywhereをProductionでデプロイした話
• https://anywhere.eks.amazonaws.com/docs/
• https://docs.aws.amazon.com/eks/latest/userguide/eks-connector.html

手順

Dockerプロバイダーでローカルで試したいが、手元のMacではメモリが足りなそうなので、要件に合うUbuntuのEC2インスタンスを立てて試す。

以下は立てたインスタンスにログインして作業する。

準備

Docker CEをインストールする。

sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get update
sudo apt-get install \
    apt-transport-https \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo \
  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

ubuntuユーザーをdockerグループに追加する。

sudo usermod -aG docker ubuntu

一度ログアウトして再ログインし、Dockerのインストールを確認。

ubuntu@ip-172-31-34-33:~$ docker version
Client: Docker Engine - Community
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:54:27 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:33 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

必要なツールをインストールする。

eksctlをインストールする。

curl "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" \
    --silent --location \
    | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin/

ubuntu@ip-172-31-34-33:~$ eksctl version
0.66.0

eksctl-anywhereプラグインをインストールする。

export EKSA_RELEASE="0.5.0" OS="$(uname -s | tr A-Z a-z)"
curl "https://anywhere-assets.eks.amazonaws.com/releases/eks-a/1/artifacts/eks-a/v${EKSA_RELEASE}/${OS}/eksctl-anywhere-v${EKSA_RELEASE}-${OS}-amd64.tar.gz" \
    --silent --location \
    | tar xz ./eksctl-anywhere
sudo mv ./eksctl-anywhere /usr/local/bin/

ubuntu@ip-172-31-34-33:~$ eksctl anywhere version
v0.5.0

kubectlもインストールする。

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/kubectl
chmod +x kubectl
sudo mv kubectl /usr/local/bin/

ローカルクラスターの作成

ローカルクラスターを作成する。

Cluster Configを作成する。

CLUSTER_NAME=dev-cluster
eksctl anywhere generate clusterconfig $CLUSTER_NAME \
   --provider docker > $CLUSTER_NAME.yaml

作成されたファイルを確認する。

ubuntu@ip-172-31-34-33:~$ cat $CLUSTER_NAME.yaml
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: Cluster
metadata:
  name: dev-cluster
spec:
  clusterNetwork:
    cni: cilium
    pods:
      cidrBlocks:
      - 192.168.0.0/16
    services:
      cidrBlocks:
      - 10.96.0.0/12
  controlPlaneConfiguration:
    count: 1
  datacenterRef:
    kind: DockerDatacenterConfig
    name: dev-cluster
  externalEtcdConfiguration:
    count: 1
  kubernetesVersion: "1.21"
  workerNodeGroupConfigurations:
  - count: 1

---
apiVersion: anywhere.eks.amazonaws.com/v1alpha1
kind: DockerDatacenterConfig
metadata:
  name: dev-cluster
spec: {}

---

クラスターを作成する。Bootstrap Clusterが作られ、そこからCluster APIでWorkload Clusterが作られている。

ubuntu@ip-172-31-34-33:~$ eksctl anywhere create cluster -f $CLUSTER_NAME.yaml
Performing setup and validations
Warning: The docker infrastructure provider is meant for local development and testing only
✅ Docker Provider setup is valid
Creating new bootstrap cluster
Installing cluster-api providers on bootstrap cluster
Provider specific setup
Creating new workload cluster
Installing networking on workload cluster
Installing storage class on workload cluster
Installing cluster-api providers on workload cluster
Moving cluster management from bootstrap to workload cluster
Installing EKS-A custom components (CRD and controller) on workload cluster
Creating EKS-A CRDs instances on workload cluster
Installing AddonManager and GitOps Toolkit on workload cluster
GitOps field not specified, bootstrap flux skipped
Writing cluster config file
Deleting bootstrap cluster
🎉 Cluster created!

ローカルにKUBECONFIGファイルが作成されているので確認する。

ubuntu@ip-172-31-34-33:~$ cat ${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://127.0.0.1:44563

  name: dev-cluster
contexts:
- context:
    cluster: dev-cluster
    user: dev-cluster-admin
  name: dev-cluster-admin@dev-cluster
current-context: dev-cluster-admin@dev-cluster
kind: Config
preferences: {}
users:
- name: dev-cluster-admin
  user:
    client-certificate-data: (snip)
    client-key-data: (snip)

クラスターに接続する。

export KUBECONFIG=${PWD}/${CLUSTER_NAME}/${CLUSTER_NAME}-eks-a-cluster.kubeconfig

クラスターの様子を見てみる。

ubuntu@ip-172-31-34-33:~$ kubectl get ns
NAME                                STATUS   AGE
capd-system                         Active   8m27s
capi-kubeadm-bootstrap-system       Active   8m37s
capi-kubeadm-control-plane-system   Active   8m30s
capi-system                         Active   8m38s
capi-webhook-system                 Active   8m40s
cert-manager                        Active   9m20s
default                             Active   10m
eksa-system                         Active   7m42s
etcdadm-bootstrap-provider-system   Active   8m36s
etcdadm-controller-system           Active   8m33s
kube-node-lease                     Active   10m
kube-public                         Active   10m
kube-system                         Active   10m

ubuntu@ip-172-31-34-33:~$ kubectl get node
NAME                                STATUS   ROLES                  AGE   VERSION
dev-cluster-l6r28                   Ready    control-plane,master   13m   v1.21.2-eks-1-21-4
dev-cluster-md-0-8475954cd9-28n48   Ready    <none>                 12m   v1.21.2-eks-1-21-4

ubuntu@ip-172-31-34-33:~$ kubectl get po -A
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS   AGE
capd-system                         capd-controller-manager-659dd5f8bc-thw5j                         2/2     Running   0          13m
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-69889cb844-glfq2       2/2     Running   0          13m
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-6ddc66fb75-fvfdp   2/2     Running   0          13m
capi-system                         capi-controller-manager-db59f5789-t5c9f                          2/2     Running   0          13m
capi-webhook-system                 capi-controller-manager-64b8c548db-9b4qh                         2/2     Running   0          13m
capi-webhook-system                 capi-kubeadm-bootstrap-controller-manager-68b8cc9759-7vnx5       2/2     Running   0          13m
capi-webhook-system                 capi-kubeadm-control-plane-controller-manager-7dc88f767d-k7jkq   2/2     Running   0          13m
cert-manager                        cert-manager-5f6b885b4-tp8vn                                     1/1     Running   0          14m
cert-manager                        cert-manager-cainjector-bb6d9bcb5-2lfhp                          1/1     Running   0          14m
cert-manager                        cert-manager-webhook-56cbc8f5b8-9sxg9                            1/1     Running   0          14m
eksa-system                         eksa-controller-manager-6769764b45-vgr5j                         2/2     Running   0          12m
etcdadm-bootstrap-provider-system   etcdadm-bootstrap-provider-controller-manager-54476b7bf9-8vfnp   2/2     Running   0          13m
etcdadm-controller-system           etcdadm-controller-controller-manager-d5795556-qr668             2/2     Running   0          13m
kube-system                         cilium-cmpn5                                                     1/1     Running   0          14m
kube-system                         cilium-operator-6bf46cc6c6-75bqs                                 1/1     Running   0          14m
kube-system                         cilium-operator-6bf46cc6c6-f5d8m                                 1/1     Running   0          14m
kube-system                         cilium-zdxrf                                                     1/1     Running   0          14m
kube-system                         coredns-7c68f85774-6vm9m                                         1/1     Running   0          15m
kube-system                         coredns-7c68f85774-9tcdr                                         1/1     Running   0          15m
kube-system                         kube-apiserver-dev-cluster-l6r28                                 1/1     Running   0          15m
kube-system                         kube-controller-manager-dev-cluster-l6r28                        1/1     Running   0          15m
kube-system                         kube-proxy-74p8l                                                 1/1     Running   0          15m
kube-system                         kube-proxy-nlghh                                                 1/1     Running   0          14m
kube-system                         kube-scheduler-dev-cluster-l6r28                                 1/1     Running   0          15m

kindで動いている。コンテナが4つ起動している。Workerノード、Masterノード、etcdノード、ロードバランサー。

ubuntu@ip-172-31-34-33:~$ docker ps
CONTAINER ID   IMAGE                                                                                COMMAND                  CREATED          STATUS          PORTS                                  NAMES
01722a50e6f5   public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.21.2-eks-d-1-21-4-eks-a-1   "/usr/local/bin/entr…"   15 minutes ago   Up 15 minutes                                          dev-cluster-md-0-8475954cd9-28n48
27be70abeb6b   public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.21.2-eks-d-1-21-4-eks-a-1   "/usr/local/bin/entr…"   16 minutes ago   Up 16 minutes   42591/tcp, 127.0.0.1:42591->6443/tcp   dev-cluster-l6r28
631d71cf079b   public.ecr.aws/eks-anywhere/kubernetes-sigs/kind/node:v1.21.2-eks-d-1-21-4-eks-a-1   "/usr/local/bin/entr…"   16 minutes ago   Up 16 minutes                                          dev-cluster-etcd-lgz9t
4fef8b84f08e   kindest/haproxy:v20210715-a6da3463                                                   "haproxy -sf 7 -W -d…"   16 minutes ago   Up 16 minutes   44563/tcp, 0.0.0.0:44563->6443/tcp     dev-cluster-lb

テストワークロードのデプロイ

テストワークロードをデプロイする。

ubuntu@ip-172-31-34-33:~$ kubectl apply -f "https://anywhere.eks.amazonaws.com/manifests/hello-eks-a.yaml"
deployment.apps/hello-eks-a created
service/hello-eks-a created

Podを確認する。

ubuntu@ip-172-31-34-33:~$ kubectl get pods -l app=hello-eks-a
NAME                          READY   STATUS    RESTARTS   AGE
hello-eks-a-9644dd8dc-pl5t2   1/1     Running   0          9s

ログを確認する。

ubuntu@ip-172-31-34-33:~$ kubectl logs -l app=hello-eks-a
2021/09/09 21:08:31 [notice] 1#1: using the "epoll" event method
2021/09/09 21:08:31 [notice] 1#1: nginx/1.21.1
2021/09/09 21:08:31 [notice] 1#1: built by gcc 10.3.1 20210424 (Alpine 10.3.1_git20210424)
2021/09/09 21:08:31 [notice] 1#1: OS: Linux 5.4.0-1045-aws
2021/09/09 21:08:31 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2021/09/09 21:08:31 [notice] 1#1: start worker processes
2021/09/09 21:08:31 [notice] 1#1: start worker process 37
2021/09/09 21:08:31 [notice] 1#1: start worker process 38
2021/09/09 21:08:31 [notice] 1#1: start worker process 39
2021/09/09 21:08:31 [notice] 1#1: start worker process 40

ポートフォワードする。

kubectl port-forward deploy/hello-eks-a 8000:80

別のターミナルからcurlしてみる。

ubuntu@ip-172-31-34-33:~$ curl localhost:8000
⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢

Thank you for using

███████╗██╗  ██╗███████╗
██╔════╝██║ ██╔╝██╔════╝
█████╗  █████╔╝ ███████╗
██╔══╝  ██╔═██╗ ╚════██║
███████╗██║  ██╗███████║
╚══════╝╚═╝  ╚═╝╚══════╝

 █████╗ ███╗   ██╗██╗   ██╗██╗    ██╗██╗  ██╗███████╗██████╗ ███████╗
██╔══██╗████╗  ██║╚██╗ ██╔╝██║    ██║██║  ██║██╔════╝██╔══██╗██╔════╝
███████║██╔██╗ ██║ ╚████╔╝ ██║ █╗ ██║███████║█████╗  ██████╔╝█████╗
██╔══██║██║╚██╗██║  ╚██╔╝  ██║███╗██║██╔══██║██╔══╝  ██╔══██╗██╔══╝
██║  ██║██║ ╚████║   ██║   ╚███╔███╔╝██║  ██║███████╗██║  ██║███████╗
╚═╝  ╚═╝╚═╝  ╚═══╝   ╚═╝    ╚══╝╚══╝ ╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚══════╝

You have successfully deployed the hello-eks-a pod hello-eks-a-9644dd8dc-pl5t2

For more information check out
https://anywhere.eks.amazonaws.com

⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢⬡⬢

EKS Connector

EKS Connectorも試してみる。

以下はローカルPCでの作業。

Service-linked Roleを作成する。

$ aws iam create-service-linked-role --aws-service-name eks-connector.amazonaws.com
{
    "Role": {
        "Path": "/aws-service-role/eks-connector.amazonaws.com/",
        "RoleName": "AWSServiceRoleForAmazonEKSConnector",
        "RoleId": "AROASYSBLVT2C6G5MXYRD",
        "Arn": "arn:aws:iam::XXXXXXXXXXXX:role/aws-service-role/eks-connector.amazonaws.com/AWSServiceRoleForAmazonEKSConnector",
        "CreateDate": "2021-09-09T21:19:47+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Action": [
                        "sts:AssumeRole"
                    ],
                    "Effect": "Allow",
                    "Principal": {
                        "Service": [
                            "eks-connector.amazonaws.com"
                        ]
                    }
                }
            ]
        }
    }
}

EKS Connector AgentのIAMロールを作成する。

信頼ポリシーのjsonを作成する。

cat << EOF > eks-connector-agent-trust-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SSMAccess",
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "ssm.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

IAMロールを作成する。

aws iam create-role \
    --role-name AmazonEKSConnectorAgentRole \
    --assume-role-policy-document file://eks-connector-agent-trust-policy.json

IAMポリシーのjsonを作成する。

cat << EOF > eks-connector-agent-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SsmControlChannel",
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateControlChannel"
            ],
            "Resource": "arn:aws:eks:*:*:cluster/*"
        },
        {
            "Sid": "ssmDataplaneOperations",
            "Effect": "Allow",
            "Action": [
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenDataChannel",
                "ssmmessages:OpenControlChannel"
            ],
            "Resource": "*"
        }
    ]
}
EOF

IAMロールにインラインポリシーをアタッチする。

aws iam put-role-policy \
    --role-name AmazonEKSConnectorAgentRole \
    --policy-name AmazonEKSConnectorAgentPolicy \
    --policy-document file://eks-connector-agent-policy.json

クラスターを登録する。

CLIでもできるがマネジメントコンソールでやってみる。「クラスターを追加」のボタンから「登録」を選択する。

プロバイダーがいくつか選べるようになっている。ここではEKS Anywhereを選択する。

先ほど作成したEKS Connector用のロールを選択して登録する。

yamlをダウンロードして適用するように言われる。

とりあえずローカルPCにyamlをダウンロードする。

---
apiVersion: v1
kind: Namespace
metadata:
  name: eks-connector
---
apiVersion: v1
kind: Secret
metadata:
  name: eks-connector-activation-config
  namespace: eks-connector
type: Opaque
data:
  code: VGVKTzhyR2d4ZTRpbWczTVBBSFI=
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: eks-connector-secret-access
  namespace: eks-connector
rules:
  - apiGroups: [ "" ]
    resources:
      - secrets
    verbs: [ "get", "update" ]
    resourceNames:
      - eks-connector-state-0
      - eks-connector-state-1
  - apiGroups: [ "" ]
    resources:
      - secrets
    verbs: [ "create" ]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: eks-connector
  namespace: eks-connector
automountServiceAccountToken: false
---
apiVersion: v1
kind: Secret
type: kubernetes.io/service-account-token
metadata:
  name: eks-connector-token
  namespace: eks-connector
  annotations:
    kubernetes.io/service-account.name: eks-connector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: eks-connector-secret-access
  namespace: eks-connector
subjects:
  - kind: ServiceAccount
    name: eks-connector
roleRef:
  kind: Role
  name: eks-connector-secret-access
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: eks-connector-agent
  namespace: eks-connector
data:
  amazon-ssm-agent.json: |
    {
      "Profile": {
        "KeyAutoRotateDays": 7
      },
      "Agent": {
        "ContainerMode": true
      },
      "Identity": {
        "ConsumptionOrder": [
            "OnPrem"
        ]
      }
    }
  seelog.xml: |
    <seelog type="adaptive" mininterval="2000000" maxinterval="100000000" critmsgcount="500" minlevel="info">
      <exceptions>
        <exception filepattern="test*" minlevel="error"/>
      </exceptions>
      <outputs formatid="fmtinfo">
        <console formatid="fmtinfo"/>
        <rollingfile type="size" filename="/var/log/amazon/ssm/amazon-ssm-agent.log" maxsize="30000000" maxrolls="5"/>
        <filter levels="error,critical" formatid="fmterror">
            <console formatid="fmterror"/>
            <rollingfile type="size" filename="/var/log/amazon/ssm/errors.log" maxsize="10000000" maxrolls="5"/>
        </filter>
      </outputs>
      <formats>
        <format id="fmterror" format="%Date %Time %LEVEL [%FuncShort @ %File.%Line] %Msg%n"/>
        <format id="fmtdebug" format="%Date %Time %LEVEL [%FuncShort @ %File.%Line] %Msg%n"/>
        <format id="fmtinfo" format="%Date %Time %LEVEL %Msg%n"/>
      </formats>
    </seelog>
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: eks-connector
  name: eks-connector
  labels:
    app: eks-connector
spec:
  replicas: 2
  selector:
    matchLabels:
      app: eks-connector
  serviceName: "eks-connector"
  template:
    metadata:
      labels:
        app: eks-connector
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - eks-connector
                topologyKey: "kubernetes.io/hostname"
      serviceAccountName: eks-connector
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
        - key: CriticalAddonsOnly
          operator: Exists
      initContainers:
        - name: connector-init
          image: public.ecr.aws/eks-connector/eks-connector:0.0.2
          imagePullPolicy: IfNotPresent
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: EKS_ACTIVATION_CODE
              valueFrom:
                secretKeyRef:
                  name: eks-connector-activation-config
                  key: code
            - name: EKS_ACTIVATION_ID
              value: hogehoge
          args:
            - "init"
            - "--activation.id=$(EKS_ACTIVATION_ID)"
            - "--activation.code=$(EKS_ACTIVATION_CODE)"
            - "--agent.region=ap-northeast-1"
          volumeMounts:
            - name: service-account-token
              mountPath: /var/run/secrets/kubernetes.io/serviceaccount
            - name: eks-agent-vault
              mountPath: /var/lib/amazon/ssm/Vault
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - all
      containers:
        - name: connector-agent
          image: public.ecr.aws/amazon-ssm-agent/amazon-ssm-agent:3.1.90.0
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: eks-connector-shared
              mountPath: /var/eks/shared
            - name: eks-agent-vault
              mountPath: /var/lib/amazon/ssm/Vault
            - name: eks-agent-config
              mountPath: /etc/amazon/ssm/amazon-ssm-agent.json
              subPath: amazon-ssm-agent.json
            - name: eks-agent-config
              mountPath: /etc/amazon/ssm/seelog.xml
              subPath: seelog.xml
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              add:
                - DAC_OVERRIDE
              drop:
                - all
        - name: connector-proxy
          image: public.ecr.aws/eks-connector/eks-connector:0.0.2
          imagePullPolicy: IfNotPresent
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
          args:
            - "server"
          volumeMounts:
            - name: service-account-token
              mountPath: /var/run/secrets/kubernetes.io/serviceaccount
            - name: eks-connector-shared
              mountPath: /var/eks/shared
            - name: eks-agent-vault
              mountPath: /var/lib/amazon/ssm/Vault
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              add:
                - DAC_OVERRIDE
              drop:
                - all
      volumes:
        - name: eks-connector-shared
          emptyDir: { }
        - name: eks-agent-vault
          emptyDir: { }
        - name: eks-agent-config
          configMap:
            name: eks-connector-agent
        - name: service-account-token
          secret:
            secretName: eks-connector-token

ローカルPCはクラスターに接続できないので、このファイルをクラスターに接続できるEC2インスタンスに持っていく。Cluster Configとファイル名が同じなので注意。

このファイルをapplyする。

ubuntu@ip-172-31-34-33:~$ kubectl apply -f dev-cluster-regist.yaml
namespace/eks-connector created
secret/eks-connector-activation-config created
role.rbac.authorization.k8s.io/eks-connector-secret-access created
serviceaccount/eks-connector created
secret/eks-connector-token created
rolebinding.rbac.authorization.k8s.io/eks-connector-secret-access created
configmap/eks-connector-agent created
statefulset.apps/eks-connector created

クラスターがActiveになったが、権限がないと言われる。

クラスターを見る権限を付与する。

• マネジメントコンソールを使用しているユーザーにはeks:AccessKubernetesApiが必要。今回はAdministratorAccessを持つユーザーを使用している。
• EKS ConnectorのServiceAccountが、IAMユーザー/ロールになりすます（impersonate）権限が必要。

eks-connector ClusterRole のテンプレートをダウンロードする。

curl -o eks-connector-clusterrole.yaml https://amazon-eks.s3.us-west-2.amazonaws.com/eks-connector/manifests/eks-connector-console-roles/eks-connector-clusterrole.yaml

マニフェストを確認する。

ubuntu@ip-172-31-34-33:~$ cat eks-connector-clusterrole.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks-connector-service
subjects:
  - kind: ServiceAccount
    name: eks-connector
    namespace: eks-connector
roleRef:
  kind: ClusterRole
  name: eks-connector-service
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks-connector-service
rules:
  - apiGroups: [ "" ]
    resources:
      - users
    verbs:
      - impersonate
    resourceNames:
      # TODO: 1. ADD your IAM identity arn here
      - "%IAM_ARN%"

"%IAM_ARN%"のところに、マネジメントコンソールにアクセスしているIAMユーザー/ロールのARNを記載し、マニフェストを適用する。

ubuntu@ip-172-31-34-33:~$ kubectl apply -f eks-connector-clusterrole.yaml
clusterrolebinding.rbac.authorization.k8s.io/eks-connector-service created
clusterrole.rbac.authorization.k8s.io/eks-connector-service created

これでエラーが少し変わった。接続できているが、クラスター内の権限がないので見られない。

すべてのNamespaceのリソースを見る権限を与えるテンプレートをダウンロードする。

curl -o eks-connector-console-dashboard-full-access-group.yaml https://amazon-eks.s3.us-west-2.amazonaws.com/eks-connector/manifests/eks-connector-console-roles/eks-connector-console-dashboard-full-access-group.yaml

マニフェストを確認する。

ubuntu@ip-172-31-34-33:~$ cat eks-connector-console-dashboard-full-access-group.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: eks-connector-console-dashboard-full-access-clusterrole
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - pods
  - events
  verbs:
  - get
  - list
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - statefulsets
  - replicasets
  verbs:
  - get
  - list
- apiGroups:
  - batch
  resources:
  - jobs
  verbs:
  - get
  - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: eks-connector-console-dashboard-full-access-clusterrole-binding
subjects:
- kind: User
  name: "%IAM_ARN%"
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: eks-connector-console-dashboard-full-access-clusterrole
  apiGroup: rbac.authorization.k8s.io
---

また%IAM_ARN%のところを書き換える必要がある。書き換えて適用する。

ubuntu@ip-172-31-34-33:~$ kubectl apply -f eks-connector-console-dashboard-full-access-group.yaml
clusterrole.rbac.authorization.k8s.io/eks-connector-console-dashboard-full-access-clusterrole created
clusterrolebinding.rbac.authorization.k8s.io/eks-connector-console-dashboard-full-access-clusterrole-binding created

これでエラーが消えてリソースが見られるようになった。

kubectlでPodが公開しているポートにアクセス

CNI Plugin Helperが公開しているメトリクスにkubectlで直接アクセスしてみるメモ。

kubectl port-forward

メトリクスは61678ポートで公開されているのでこのポートにポートフォワードし、http://localhost:61678/metrics にアクセスする。

$ kubectl port-forward aws-node-hpd82 61678
Forwarding from 127.0.0.1:61678 -> 61678
Forwarding from [::1]:61678 -> 61678
Handling connection for 61678
Handling connection for 61678
Handling connection for 61678

kubectl get --raw

apiserverのプロキシを利用してアクセスする。ポート番号は指定しなくても大丈夫。

$ kubectl get --raw /api/v1/namespaces/kube-system/pods/aws-node-hpd82:61678/proxy/metrics
# HELP awscni_add_ip_req_count The number of add IP address requests
# TYPE awscni_add_ip_req_count counter
awscni_add_ip_req_count 14
# HELP awscni_assigned_ip_addresses The number of IP addresses assigned to pods
# TYPE awscni_assigned_ip_addresses gauge
awscni_assigned_ip_addresses 14
# HELP awscni_assigned_ip_per_ipv4cidr The total number of IP addresses assigned per cidr
# TYPE awscni_assigned_ip_per_ipv4cidr gauge
awscni_assigned_ip_per_ipv4cidr{cidr="192.168.192.64/28"} 12
awscni_assigned_ip_per_ipv4cidr{cidr="192.168.246.0/28"} 2
# HELP awscni_aws_api_latency_ms AWS API call latency in ms
# TYPE awscni_aws_api_latency_ms summary
awscni_aws_api_latency_ms_sum{api="DescribeNetworkInterfaces",error="false",status="200"} 252
awscni_aws_api_latency_ms_count{api="DescribeNetworkInterfaces",error="false",status="200"} 1
awscni_aws_api_latency_ms_sum{api="GetMetadata",error="false",status="200"} 9617
awscni_aws_api_latency_ms_count{api="GetMetadata",error="false",status="200"} 44774
awscni_aws_api_latency_ms_sum{api="ModifyNetworkInterfaceAttribute",error="false",status="200"} 478
awscni_aws_api_latency_ms_count{api="ModifyNetworkInterfaceAttribute",error="false",status="200"} 1
# HELP awscni_del_ip_req_count The number of delete IP address requests
# TYPE awscni_del_ip_req_count counter
awscni_del_ip_req_count{reason="PodDeleted"} 14
# HELP awscni_eni_allocated The number of ENIs allocated
# TYPE awscni_eni_allocated gauge
awscni_eni_allocated 1
# HELP awscni_eni_max The maximum number of ENIs that can be attached to the instance, accounting for unmanaged ENIs
# TYPE awscni_eni_max gauge
awscni_eni_max 3
# HELP awscni_force_removed_enis The number of ENIs force removed while they had assigned pods
# TYPE awscni_force_removed_enis counter
awscni_force_removed_enis 0
# HELP awscni_force_removed_ips The number of IPs force removed while they had assigned pods
# TYPE awscni_force_removed_ips counter
awscni_force_removed_ips 0
# HELP awscni_ip_max The maximum number of IP addresses that can be allocated to the instance
# TYPE awscni_ip_max gauge
awscni_ip_max 432
# HELP awscni_ipamd_action_inprogress The number of ipamd actions in progress
# TYPE awscni_ipamd_action_inprogress gauge
awscni_ipamd_action_inprogress{fn="nodeIPPoolReconcile"} 0
awscni_ipamd_action_inprogress{fn="nodeInit"} 0
# HELP awscni_reconcile_count The number of times ipamd reconciles on ENIs and IP/Prefix addresses
# TYPE awscni_reconcile_count counter
awscni_reconcile_count{fn="eniDataStorePoolReconcileAdd"} 8134
# HELP awscni_total_ip_addresses The total number of IP addresses
# TYPE awscni_total_ip_addresses gauge
awscni_total_ip_addresses 32
# HELP awscni_total_ipv4_prefixes The total number of IPv4 prefixes
# TYPE awscni_total_ipv4_prefixes gauge
awscni_total_ipv4_prefixes 2
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.1927e-05
go_gc_duration_seconds{quantile="0.25"} 3.381e-05
go_gc_duration_seconds{quantile="0.5"} 4.1182e-05
go_gc_duration_seconds{quantile="0.75"} 5.5597e-05
go_gc_duration_seconds{quantile="1"} 0.002618703
go_gc_duration_seconds_sum 0.147290686
go_gc_duration_seconds_count 2040
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 35
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.14.15"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 6.42528e+06
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 6.437947936e+09
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.635703e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.7240465e+07
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 9.738090810849369e-06
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.594504e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 6.42528e+06
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.697536e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 9.37984e+06
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 33536
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 5.3952512e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.63552e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6310513580072515e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.7274001e+07
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 3472
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 140352
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 163840
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 1.264744e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 438913
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 753664
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 753664
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.2958208e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 8
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 101.08
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 19
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 7.008256e+07
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.63080734446e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 7.87320832e+08
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 4
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0

ExternelDNSを試す

ExternalDNSを試すメモ。

コンポーネント	バージョン
EKS	1.21
プラットフォームバージョン	eks.2
AWS Load Balancer Controller	v2.2.4
AWS Load Balancer Controllerチャート	1.2.7
ExternalDNS	v0.7.6

• https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws-sd.md

クラスターの作成

1.21でクラスターを作成する。ノードなしで作成する。

cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: externaldns
  region: ap-northeast-1
  version: "1.21"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EOF

eksctl create cluster -f cluster.yaml

AWS Load Balancer Controller

必須ではないがAWS Load Balancer Controllerを入れる。

• https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/aws-load-balancer-controller.html

curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json
aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

IAMロールとServiceAccountを作成する。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=externaldns
eksctl create iamserviceaccount \
  --cluster=${CLUSTER_NAME} \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/AWSLoadBalancerControllerIAMPolicy \
  --override-existing-serviceaccounts \
  --approve

HelmでAWS Load Balancer Controllerをインストールする。

TargetGroupBindingのカスタムリソース定義をインストールする。

$ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"
customresourcedefinition.apiextensions.k8s.io/ingressclassparams.elbv2.k8s.aws created
customresourcedefinition.apiextensions.k8s.io/targetgroupbindings.elbv2.k8s.aws created

リポジトリを最新化する。

helm repo add eks https://aws.github.io/eks-charts
helm repo update

コントローラーをデプロイする。

helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=${CLUSTER_NAME} \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  -n kube-system

Release "aws-load-balancer-controller" does not exist. Installing it now.
NAME: aws-load-balancer-controller
LAST DEPLOYED: Mon Sep  6 03:49:20 2021
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!

確認する。

$ kubectl get deployment -n kube-system aws-load-balancer-controller
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           21s

ExternalDNS

パブリックなCloud Map名前空間を作成する。

• https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws-sd.md
• https://aws.amazon.com/jp/premiumsupport/knowledge-center/create-subdomain-route-53/

aws servicediscovery create-public-dns-namespace --name "external-dns-test.sotosugi.com"

確認する。

$ aws servicediscovery list-namespaces
{
    "Namespaces": [
        {
            "Id": "ns-3driiduhotfeurah",
            "Arn": "arn:aws:servicediscovery:ap-northeast-1:XXXXXXXXXXXX:namespace/ns-3driiduhotfeurah",
            "Name": "external-dns-test.sotosugi.com",
            "Type": "DNS_PUBLIC",
            "Properties": {
                "DnsProperties": {
                    "HostedZoneId": "Z0678027REBIAZYHXZWX",
                    "SOA": {
                        "TTL": 60
                    }
                },
                "HttpProperties": {
                    "HttpName": "external-dns-test.sotosugi.com"
                }
            },
            "CreateDate": "2021-09-06T04:39:50.799000+09:00"
        }
    ]
}

ExternalDNSをデプロイする。なお、Helmチャートもある。ここで指定しているイメージのバージョンはちょっと古そう。

cat << EOF > external-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.7.6
        env:
          - name: AWS_REGION
            value: ap-northeast-1 # put your CloudMap NameSpace region
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=external-dns-test.sotosugi.com # Makes ExternalDNS see only the namespaces that match the specified domain. Omit the filter if you want to process all available namespaces.
        - --provider=aws-sd
        - --aws-zone-type=public # Only look at public namespaces. Valid values are public, private, or no value for both)
        - --txt-owner-id=my-identifier
EOF

kubectl apply -f external-dns.yaml -n kube-system

IAMポリシーを作成する。

cat << EOF > iam-policy-external-dns.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetHostedZone",
        "route53:ListHostedZonesByName",
        "route53:CreateHostedZone",
        "route53:DeleteHostedZone",
        "route53:ChangeResourceRecordSets",
        "route53:CreateHealthCheck",
        "route53:GetHealthCheck",
        "route53:DeleteHealthCheck",
        "route53:UpdateHealthCheck",
        "ec2:DescribeVpcs",
        "ec2:DescribeRegions",
        "servicediscovery:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

aws iam create-policy \
    --policy-name ExternalDNSIAMPolicy \
    --policy-document file://iam-policy-external-dns.json

IAMロールとServiceAccountを作成する。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=externaldns
eksctl create iamserviceaccount \
  --cluster=${CLUSTER_NAME} \
  --namespace=kube-system \
  --name=external-dns \
  --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/ExternalDNSIAMPolicy \
  --override-existing-serviceaccounts \
  --approve

external-dnsのPodを再起動し、ログを確認する。

$ k logs -n kube-system external-dns-cc9498875-v7ptx
time="2021-09-05T19:48:26Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws-sd GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[external-dns-test.sotosugi.com] ExcludeDomains:[] ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:my-identifier TXTPrefix: TXTSuffix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME]}"
time="2021-09-05T19:48:26Z" level=info msg="Instantiating new Kubernetes client"
time="2021-09-05T19:48:26Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-09-05T19:48:26Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2021-09-05T19:48:28Z" level=info msg="Registry \"txt\" cannot be used with AWS Cloud Map. Switching to \"aws-sd\"."
time="2021-09-05T19:48:35Z" level=info msg="All records are already up to date"

Serviceの確認

LoadBalancerタイプのサービスを作成する。

cat << EOF > nginx-lb.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF

k apply -f nginx-lb.yaml

確認する。

$ k get po,svc
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-bclwg   1/1     Running   0          15m

NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP                                                                    PORT(S)        AGE
service/kubernetes   ClusterIP      172.20.0.1      <none>                                                                         443/TCP        103m
service/nginx        LoadBalancer   172.20.192.78   ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com   80:31224/TCP   15m

ログを確認する。

time="2021-09-05T19:49:33Z" level=info msg="Creating a new service \"nginx\" in \"ns-3driiduhotfeurah\" namespace"
time="2021-09-05T19:49:33Z" level=info msg="Registering a new instance \"ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"

サービスとサービスインスタンスがCloud Mapに登録されている。

Route53にも登録されている。

名前解決できてブラウザでもアクセスできることを確認する。

削除する。

k delete -f nginx-lb.yaml

続いてNodePortサービスで試す。

cat << EOF > nginx-np.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: NodePort
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF

k apply -f nginx-np.yaml

確認する。

$ k get po,svc
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-25c5f   1/1     Running   0          91s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   172.20.0.1      <none>        443/TCP        110m
service/nginx        NodePort    172.20.74.220   <none>        80:31544/TCP   91s

ログに出力される。

time="2021-09-05T19:54:35Z" level=info msg="De-registering an instance \"ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T19:55:36Z" level=info msg="Registering a new instance \"10.0.119.198\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T19:55:36Z" level=info msg="Registering a new instance \"10.0.69.80\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"

$ k get node
NAME                                              STATUS   ROLES    AGE   VERSION
ip-10-0-119-198.ap-northeast-1.compute.internal   Ready    <none>   96m   v1.21.2-eks-55daa9d
ip-10-0-69-80.ap-northeast-1.compute.internal     Ready    <none>   96m   v1.21.2-eks-55daa9d

この場合はノードのIPが登録されている。

これだと外からアクセスできるわけではない。アクセスできる場所にいたとしても、NodePortがわからないとアクセスできない。

$ nslookup nginx.external-dns-test.sotosugi.com
Server:         172.17.192.154
Address:        172.17.192.154#53

Non-authoritative answer:
Name:   nginx.external-dns-test.sotosugi.com
Address: 10.0.69.80

削除する。

k delete -f nginx-np.yaml

続いてClusterIPで試す。

cat << EOF > nginx-clusterip.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF

k apply -f nginx-clusterip.yaml

確認する。

$ k get po,svc
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-vd9d4   1/1     Running   0          22s

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   172.20.0.1    <none>        443/TCP   116m
service/nginx        ClusterIP   172.20.0.51   <none>        80/TCP    22s

しばらく待っても登録される様子がない。LoadBalancerかNodePortでないとだめっぽい。

time="2021-09-05T20:00:40Z" level=info msg="De-registering an instance \"10.0.119.198\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T20:00:40Z" level=info msg="De-registering an instance \"10.0.69.80\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T20:01:41Z" level=info msg="All records are already up to date"
time="2021-09-05T20:02:41Z" level=info msg="All records are already up to date"
time="2021-09-05T20:03:42Z" level=info msg="All records are already up to date"

k delete -f nginx-clusterip.yaml

他にExternalNameはサポートしているようだが、使い道がよくわからない。

• https://github.com/kubernetes-sigs/external-dns/blob/master/docs/faq.md#which-kubernetes-objects-are-supported
• https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/externalname.md

Ingressの確認

Ingressを作成する。

cat << EOF > nginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
  - host: nginx.external-dns-test.sotosugi.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF

k apply -f nginx-ingress.yaml

確認する。

$ k get po,svc,ing
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-rc7jh   1/1     Running   0          7m26s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   172.20.0.1       <none>        443/TCP   142m
service/nginx        ClusterIP   172.20.124.151   <none>        80/TCP    7m26s

NAME                              CLASS    HOSTS                                  ADDRESS                                                                    PORTS   AGE
ingress.networking.k8s.io/nginx   <none>   nginx.external-dns-test.sotosugi.com   k8s-default-nginx-8007b1159a-1140119829.ap-northeast-1.elb.amazonaws.com   80      6m58s

ログ出力される。

time="2021-09-05T20:28:54Z" level=info msg="Updating service \"nginx\""
time="2021-09-05T20:28:54Z" level=info msg="Registering a new instance \"k8s-default-nginx-8007b1159a-1140119829.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"

PodからcAdvisorのメトリクスにアクセスする

前回に関連して、手動でPodからcAdvisorのメトリクスにアクセスしてみるメモ。

追記

こんなブログがあった。

• https://aws.amazon.com/jp/blogs/containers/monitoring-amazon-eks-on-aws-fargate-using-prometheus-and-grafana/

Podを起動するとかしなくても、普通にkubectl get --rawを叩けばよかった。

手順

defaultネームスペースに適当なPod（nginx）を起動する。

$ k run nginx --image=nginx
pod/nginx created

Podが起動したらPodの中にexecする。

k exec -it nginx -- bash

kubectlをインストールする。

curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.21.2/bin/linux/amd64/kubectl
chmod +x ./kubectl
mv ./kubectl /usr/local/bin/kubectl

別ターミナルからノードを確認する。

$ k get node
NAME                                                      STATUS   ROLES    AGE     VERSION
fargate-ip-10-0-126-158.ap-northeast-1.compute.internal   Ready    <none>   3d13h   v1.21.2-13+d2965f0db10712
fargate-ip-10-0-126-87.ap-northeast-1.compute.internal    Ready    <none>   3d13h   v1.21.2-13+d2965f0db10712
fargate-ip-10-0-68-162.ap-northeast-1.compute.internal    Ready    <none>   3d13h   v1.21.2-13+d2965f0db10712
fargate-ip-10-0-93-100.ap-northeast-1.compute.internal    Ready    <none>   2m32s   v1.21.2-13+d2965f0db10712
fargate-ip-10-0-96-193.ap-northeast-1.compute.internal    Ready    <none>   21m     v1.21.2-13+d2965f0db10712
fargate-ip-10-0-96-248.ap-northeast-1.compute.internal    Ready    <none>   3d13h   v1.21.2-13+d2965f0db10712
fargate-ip-10-0-99-56.ap-northeast-1.compute.internal     Ready    <none>   3d13h   v1.21.2-13+d2965f0db10712

kubecltでアクセスする。kubectlを使うのはトークンのマウントとかを勝手にやってくれるから。

root@nginx:/# NODE="fargate-ip-10-0-126-158.ap-northeast-1.compute.internal"
root@nginx:/# kubectl get --raw /api/v1/nodes/${NODE}/proxy/metrics/cadvisor
Error from server (Forbidden): nodes "fargate-ip-10-0-126-158.ap-northeast-1.compute.internal" is forbidden: User "system:serviceaccount:default:default" cannot get resource "nodes/proxy" in API group "" at the cluster scope
root@nginx:/#

権限が足りてないので付与する。

$ k create clusterrole test-cadvisor-metrics --verb=get --resource=node/proxy
clusterrole.rbac.authorization.k8s.io/test-cadvisor-metrics created
$ k create clusterrolebinding test-cadvisor-metrics --clusterrole=test-cadvisor-metrics --serviceaccount=default:default
clusterrolebinding.rbac.authorization.k8s.io/test-cadvisor-metrics created

これでメトリクスにアクセスできた。/metricsの許可は不要のようだ。

root@nginx:/# kubectl get --raw /api/v1/nodes/${NODE}/proxy/metrics/cadvisor
# HELP cadvisor_version_info A metric with a constant '1' value labeled by kernel version, OS version, docker version, cadvisor version & cadvisor revision.
# TYPE cadvisor_version_info gauge
cadvisor_version_info{cadvisorRevision="",cadvisorVersion="",dockerVersion="Unknown",kernelVersion="4.14.238-182.422.amzn2.x86_64",osVersion="Amazon Linux 2"} 1
# HELP container_blkio_device_usage_total Blkio Device bytes usage
# TYPE container_blkio_device_usage_total counter
container_blkio_device_usage_total{container="",device="/dev/nvme1n1",id="/",image="",major="259",minor="0",name="",namespace="",operation="Async",pod=""} 942080 1629753493544
container_blkio_device_usage_total{container="",device="/dev/nvme1n1",id="/",image="",major="259",minor="0",name="",namespace="",operation="Read",pod=""} 135168 1629753493544
container_blkio_device_usage_total{container="",device="/dev/nvme1n1",id="/",image="",major="259",minor="0",name="",namespace="",operation="Sync",pod=""} 2.883584e+07 1629753493544
container_blkio_device_usage_total{container="",device="/dev/nvme1n1",id="/",image="",major="259",minor="0",name="",namespace="",operation="Total",pod=""} 2.977792e+07 1629753493544
container_blkio_device_usage_total{container="",device="/dev/nvme1n1",id="/",image="",major="259",minor="0",name="",namespace="",operation="Write",pod=""} 2.9642752e+07 1629753493544
# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge
container_cpu_load_average_10s{container="",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_cpu_load_average_10s{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_cpu_load_average_10s{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_cpu_load_average_10s{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
container_cpu_load_average_10s{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_cpu_load_average_10s{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753483841
# HELP container_cpu_system_seconds_total Cumulative system cpu time consumed in seconds.
# TYPE container_cpu_system_seconds_total counter
container_cpu_system_seconds_total{container="",id="/",image="",name="",namespace="",pod=""} 1596.91 1629753493544
container_cpu_system_seconds_total{container="",id="/kubepods",image="",name="",namespace="",pod=""} 107.68 1629753493584
container_cpu_system_seconds_total{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 107.68 1629753493879
container_cpu_system_seconds_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 107.68 1629753484928
container_cpu_system_seconds_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0.01 1629753492567
container_cpu_system_seconds_total{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_cpu_system_seconds_total{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 107.66 1629753483841
# HELP container_cpu_usage_seconds_total Cumulative cpu time consumed in seconds.
# TYPE container_cpu_usage_seconds_total counter
container_cpu_usage_seconds_total{container="",cpu="total",id="/",image="",name="",namespace="",pod=""} 4456.774163395 1629753493544
container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods",image="",name="",namespace="",pod=""} 129.048032364 1629753493584
container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 129.048420139 1629753493879
container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 129.044671067 1629753484928
container_cpu_usage_seconds_total{container="",cpu="total",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0.198619685 1629753492567
container_cpu_usage_seconds_total{container="hey-yo",cpu="total",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 128.84559462 1629753483841
# HELP container_cpu_user_seconds_total Cumulative user cpu time consumed in seconds.
# TYPE container_cpu_user_seconds_total counter
container_cpu_user_seconds_total{container="",id="/",image="",name="",namespace="",pod=""} 2386.4 1629753493544
container_cpu_user_seconds_total{container="",id="/kubepods",image="",name="",namespace="",pod=""} 21.89 1629753493584
container_cpu_user_seconds_total{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 21.89 1629753493879
container_cpu_user_seconds_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 21.89 1629753484928
container_cpu_user_seconds_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0.17 1629753492567
container_cpu_user_seconds_total{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_cpu_user_seconds_total{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 21.71 1629753483841
# HELP container_file_descriptors Number of open file descriptors for the container.
# TYPE container_file_descriptors gauge
container_file_descriptors{container="",id="/",image="",name="",namespace="",pod=""} 228 1629753493544
container_file_descriptors{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_file_descriptors{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_file_descriptors{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_file_descriptors{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 3 1629753492567
container_file_descriptors{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_file_descriptors{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 7 1629753483841
# HELP container_fs_inodes_free Number of available Inodes
# TYPE container_fs_inodes_free gauge
container_fs_inodes_free{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 497068 1629753493544
container_fs_inodes_free{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 1.749617e+06 1629753493544
container_fs_inodes_free{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 497223 1629753493544
container_fs_inodes_free{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 497208 1629753493544
container_fs_inodes_free{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 1.749617e+06 1629753493544
# HELP container_fs_inodes_total Number of Inodes
# TYPE container_fs_inodes_total gauge
container_fs_inodes_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 497224 1629753493544
container_fs_inodes_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 1.96608e+06 1629753493544
container_fs_inodes_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 497224 1629753493544
container_fs_inodes_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 497224 1629753493544
container_fs_inodes_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 1.96608e+06 1629753493544
# HELP container_fs_io_current Number of I/Os currently in progress
# TYPE container_fs_io_current gauge
container_fs_io_current{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_current{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_current{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_current{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_current{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_io_time_seconds_total Cumulative count of seconds spent doing I/Os
# TYPE container_fs_io_time_seconds_total counter
container_fs_io_time_seconds_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_seconds_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_seconds_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_seconds_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_seconds_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_io_time_weighted_seconds_total Cumulative weighted I/O time in seconds
# TYPE container_fs_io_time_weighted_seconds_total counter
container_fs_io_time_weighted_seconds_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_weighted_seconds_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_weighted_seconds_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_weighted_seconds_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_io_time_weighted_seconds_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_limit_bytes Number of bytes that can be consumed by the container on this filesystem.
# TYPE container_fs_limit_bytes gauge
container_fs_limit_bytes{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 6.7108864e+07 1629753493544
container_fs_limit_bytes{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 3.1572529152e+10 1629753493544
container_fs_limit_bytes{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 2.036629504e+09 1629753493544
container_fs_limit_bytes{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 2.036629504e+09 1629753493544
container_fs_limit_bytes{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 3.1572529152e+10 1629753493544
# HELP container_fs_read_seconds_total Cumulative count of seconds spent reading
# TYPE container_fs_read_seconds_total counter
container_fs_read_seconds_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_read_seconds_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_read_seconds_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_read_seconds_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_read_seconds_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_reads_bytes_total Cumulative count of bytes read
# TYPE container_fs_reads_bytes_total counter
container_fs_reads_bytes_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 135168 1629753493544
# HELP container_fs_reads_merged_total Cumulative count of reads merged
# TYPE container_fs_reads_merged_total counter
container_fs_reads_merged_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_reads_merged_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_reads_merged_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_reads_merged_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_reads_merged_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_reads_total Cumulative count of reads completed
# TYPE container_fs_reads_total counter
container_fs_reads_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_reads_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 33 1629753493544
container_fs_reads_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_reads_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_reads_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_sector_reads_total Cumulative count of sector reads completed
# TYPE container_fs_sector_reads_total counter
container_fs_sector_reads_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_reads_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_reads_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_reads_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_reads_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_sector_writes_total Cumulative count of sector writes completed
# TYPE container_fs_sector_writes_total counter
container_fs_sector_writes_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_writes_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_writes_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_writes_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_sector_writes_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_usage_bytes Number of bytes that are consumed by the container on this filesystem.
# TYPE container_fs_usage_bytes gauge
container_fs_usage_bytes{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_usage_bytes{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 1.1741827072e+10 1629753493544
container_fs_usage_bytes{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_usage_bytes{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_usage_bytes{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 1.1741827072e+10 1629753493544
# HELP container_fs_write_seconds_total Cumulative count of seconds spent writing
# TYPE container_fs_write_seconds_total counter
container_fs_write_seconds_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_write_seconds_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_write_seconds_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_write_seconds_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_write_seconds_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_writes_bytes_total Cumulative count of bytes written
# TYPE container_fs_writes_bytes_total counter
container_fs_writes_bytes_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 2.9642752e+07 1629753493544
# HELP container_fs_writes_merged_total Cumulative count of writes merged
# TYPE container_fs_writes_merged_total counter
container_fs_writes_merged_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_writes_merged_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_writes_merged_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_writes_merged_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_writes_merged_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_fs_writes_total Cumulative count of writes completed
# TYPE container_fs_writes_total counter
container_fs_writes_total{container="",device="/dev",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_writes_total{container="",device="/dev/nvme1n1",id="/",image="",name="",namespace="",pod=""} 1707 1629753493544
container_fs_writes_total{container="",device="/dev/shm",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_writes_total{container="",device="/sys/fs/cgroup",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_fs_writes_total{container="",device="overlay_0-37",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
# HELP container_last_seen Last time a container was seen by the exporter
# TYPE container_last_seen gauge
container_last_seen{container="",id="/",image="",name="",namespace="",pod=""} 1.629753494e+09 1629753494616
container_last_seen{container="",id="/kubepods",image="",name="",namespace="",pod=""} 1.629753494e+09 1629753494618
container_last_seen{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 1.629753494e+09 1629753494618
container_last_seen{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.629753494e+09 1629753494618
container_last_seen{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.629753494e+09 1629753494616
container_last_seen{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 1.629753494e+09 1629753494618
container_last_seen{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.629753494e+09 1629753494616
# HELP container_memory_cache Number of bytes of page cache memory.
# TYPE container_memory_cache gauge
container_memory_cache{container="",id="/",image="",name="",namespace="",pod=""} 4.5666304e+07 1629753493544
container_memory_cache{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_memory_cache{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_memory_cache{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_memory_cache{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
container_memory_cache{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_cache{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753483841
# HELP container_memory_failcnt Number of memory usage hits limits
# TYPE container_memory_failcnt counter
container_memory_failcnt{container="",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_memory_failcnt{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_memory_failcnt{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_memory_failcnt{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_memory_failcnt{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
container_memory_failcnt{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_failcnt{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753483841
# HELP container_memory_failures_total Cumulative count of memory allocation failures.
# TYPE container_memory_failures_total counter
container_memory_failures_total{container="",failure_type="pgfault",id="/",image="",name="",namespace="",pod="",scope="container"} 3.8620428e+07 1629753493544
container_memory_failures_total{container="",failure_type="pgfault",id="/",image="",name="",namespace="",pod="",scope="hierarchy"} 3.8620428e+07 1629753493544
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods",image="",name="",namespace="",pod="",scope="container"} 0 1629753493584
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods",image="",name="",namespace="",pod="",scope="hierarchy"} 0 1629753493584
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/besteffort",image="",name="",namespace="",pod="",scope="container"} 0 1629753493879
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/besteffort",image="",name="",namespace="",pod="",scope="hierarchy"} 0 1629753493879
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="container"} 0 1629753484928
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="hierarchy"} 0 1629753484928
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="container"} 2178 1629753492567
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="hierarchy"} 2178 1629753492567
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/burstable",image="",name="",namespace="",pod="",scope="container"} 0 1629753490555
container_memory_failures_total{container="",failure_type="pgfault",id="/kubepods/burstable",image="",name="",namespace="",pod="",scope="hierarchy"} 0 1629753490555
container_memory_failures_total{container="",failure_type="pgmajfault",id="/",image="",name="",namespace="",pod="",scope="container"} 0 1629753493544
container_memory_failures_total{container="",failure_type="pgmajfault",id="/",image="",name="",namespace="",pod="",scope="hierarchy"} 0 1629753493544
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods",image="",name="",namespace="",pod="",scope="container"} 0 1629753493584
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods",image="",name="",namespace="",pod="",scope="hierarchy"} 0 1629753493584
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/besteffort",image="",name="",namespace="",pod="",scope="container"} 0 1629753493879
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/besteffort",image="",name="",namespace="",pod="",scope="hierarchy"} 0 1629753493879
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="container"} 0 1629753484928
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="hierarchy"} 0 1629753484928
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="container"} 0 1629753492567
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="hierarchy"} 0 1629753492567
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/burstable",image="",name="",namespace="",pod="",scope="container"} 0 1629753490555
container_memory_failures_total{container="",failure_type="pgmajfault",id="/kubepods/burstable",image="",name="",namespace="",pod="",scope="hierarchy"} 0 1629753490555
container_memory_failures_total{container="hey-yo",failure_type="pgfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="container"} 1.300167e+07 1629753483841
container_memory_failures_total{container="hey-yo",failure_type="pgfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="hierarchy"} 1.300167e+07 1629753483841
container_memory_failures_total{container="hey-yo",failure_type="pgmajfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="container"} 0 1629753483841
container_memory_failures_total{container="hey-yo",failure_type="pgmajfault",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",scope="hierarchy"} 0 1629753483841
# HELP container_memory_mapped_file Size of memory mapped files in bytes.
# TYPE container_memory_mapped_file gauge
container_memory_mapped_file{container="",id="/",image="",name="",namespace="",pod=""} 1.35168e+06 1629753493544
container_memory_mapped_file{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_memory_mapped_file{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_memory_mapped_file{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_memory_mapped_file{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
container_memory_mapped_file{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_mapped_file{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753483841
# HELP container_memory_max_usage_bytes Maximum memory usage recorded in bytes
# TYPE container_memory_max_usage_bytes gauge
container_memory_max_usage_bytes{container="",id="/",image="",name="",namespace="",pod=""} 1.95485696e+08 1629753493544
container_memory_max_usage_bytes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 8.265728e+06 1629753493584
container_memory_max_usage_bytes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 8.265728e+06 1629753493879
container_memory_max_usage_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 8.265728e+06 1629753484928
container_memory_max_usage_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 8.265728e+06 1629753492567
container_memory_max_usage_bytes{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_max_usage_bytes{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 7.573504e+06 1629753483841
# HELP container_memory_rss Size of RSS in bytes.
# TYPE container_memory_rss gauge
container_memory_rss{container="",id="/",image="",name="",namespace="",pod=""} 1.31739648e+08 1629753493544
container_memory_rss{container="",id="/kubepods",image="",name="",namespace="",pod=""} 69632 1629753493584
container_memory_rss{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 69632 1629753493879
container_memory_rss{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 69632 1629753484928
container_memory_rss{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 24576 1629753492567
container_memory_rss{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_rss{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 45056 1629753483841
# HELP container_memory_swap Container swap usage in bytes.
# TYPE container_memory_swap gauge
container_memory_swap{container="",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_memory_swap{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_memory_swap{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_memory_swap{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_memory_swap{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
container_memory_swap{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_swap{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753483841
# HELP container_memory_usage_bytes Current memory usage in bytes, including all memory regardless of when it was accessed
# TYPE container_memory_usage_bytes gauge
container_memory_usage_bytes{container="",id="/",image="",name="",namespace="",pod=""} 1.9120128e+08 1629753493544
container_memory_usage_bytes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 1.835008e+06 1629753493584
container_memory_usage_bytes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 1.933312e+06 1629753493879
container_memory_usage_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.900544e+06 1629753484928
container_memory_usage_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 667648 1629753492567
container_memory_usage_bytes{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_usage_bytes{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.24928e+06 1629753483841
# HELP container_memory_working_set_bytes Current working set in bytes.
# TYPE container_memory_working_set_bytes gauge
container_memory_working_set_bytes{container="",id="/",image="",name="",namespace="",pod=""} 1.38346496e+08 1629753493544
container_memory_working_set_bytes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 1.835008e+06 1629753493584
container_memory_working_set_bytes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 1.933312e+06 1629753493879
container_memory_working_set_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.900544e+06 1629753484928
container_memory_working_set_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 667648 1629753492567
container_memory_working_set_bytes{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_memory_working_set_bytes{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.24928e+06 1629753483841
# HELP container_network_receive_bytes_total Cumulative count of bytes received
# TYPE container_network_receive_bytes_total counter
container_network_receive_bytes_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 117964 1629753493544
container_network_receive_bytes_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 557511 1629753493544
container_network_receive_bytes_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 1.533601e+06 1629753493544
container_network_receive_bytes_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 6.81933182e+08 1629753493544
container_network_receive_bytes_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 13912 1629753492567
# HELP container_network_receive_errors_total Cumulative count of errors encountered while receiving
# TYPE container_network_receive_errors_total counter
container_network_receive_errors_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_errors_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_errors_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_errors_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_errors_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
# HELP container_network_receive_packets_dropped_total Cumulative count of packets dropped while receiving
# TYPE container_network_receive_packets_dropped_total counter
container_network_receive_packets_dropped_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_packets_dropped_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_packets_dropped_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_packets_dropped_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 0 1629753493544
container_network_receive_packets_dropped_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
# HELP container_network_receive_packets_total Cumulative count of packets received
# TYPE container_network_receive_packets_total counter
container_network_receive_packets_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 1245 1629753493544
container_network_receive_packets_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 6286 1629753493544
container_network_receive_packets_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 5206 1629753493544
container_network_receive_packets_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 969703 1629753493544
container_network_receive_packets_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 196 1629753492567
# HELP container_network_transmit_bytes_total Cumulative count of bytes transmitted
# TYPE container_network_transmit_bytes_total counter
container_network_transmit_bytes_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 32428 1629753493544
container_network_transmit_bytes_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 1.439774e+06 1629753493544
container_network_transmit_bytes_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 597537 1629753493544
container_network_transmit_bytes_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 1.40968731e+08 1629753493544
container_network_transmit_bytes_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 6956 1629753492567
# HELP container_network_transmit_errors_total Cumulative count of errors encountered while transmitting
# TYPE container_network_transmit_errors_total counter
container_network_transmit_errors_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_errors_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_errors_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_errors_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_errors_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
# HELP container_network_transmit_packets_dropped_total Cumulative count of packets dropped while transmitting
# TYPE container_network_transmit_packets_dropped_total counter
container_network_transmit_packets_dropped_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_packets_dropped_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_packets_dropped_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_packets_dropped_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 0 1629753493544
container_network_transmit_packets_dropped_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
# HELP container_network_transmit_packets_total Cumulative count of packets transmitted
# TYPE container_network_transmit_packets_total counter
container_network_transmit_packets_total{container="",id="/",image="",interface="eks-bridge",name="",namespace="",pod=""} 341 1629753493544
container_network_transmit_packets_total{container="",id="/",image="",interface="eks-veth",name="",namespace="",pod=""} 5163 1629753493544
container_network_transmit_packets_total{container="",id="/",image="",interface="eth0",name="",namespace="",pod=""} 6734 1629753493544
container_network_transmit_packets_total{container="",id="/",image="",interface="eth1",name="",namespace="",pod=""} 612243 1629753493544
container_network_transmit_packets_total{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",interface="eth0",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 98 1629753492567
# HELP container_processes Number of processes running inside the container.
# TYPE container_processes gauge
container_processes{container="",id="/",image="",name="",namespace="",pod=""} 17 1629753493544
container_processes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_processes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_processes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_processes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1 1629753492567
container_processes{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_processes{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 2 1629753483841
# HELP container_scrape_error 1 if there was an error while getting container metrics, 0 otherwise
# TYPE container_scrape_error gauge
container_scrape_error 0
# HELP container_sockets Number of open sockets for the container.
# TYPE container_sockets gauge
container_sockets{container="",id="/",image="",name="",namespace="",pod=""} 32 1629753493544
container_sockets{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0 1629753493584
container_sockets{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_sockets{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_sockets{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
container_sockets{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_sockets{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753483841
# HELP container_spec_cpu_period CPU period of the container.
# TYPE container_spec_cpu_period gauge
container_spec_cpu_period{container="",id="/",image="",name="",namespace="",pod=""} 100000
container_spec_cpu_period{container="",id="/kubepods",image="",name="",namespace="",pod=""} 100000
container_spec_cpu_period{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 100000
container_spec_cpu_period{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 100000
container_spec_cpu_period{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 100000
container_spec_cpu_period{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 100000
container_spec_cpu_period{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 100000
# HELP container_spec_cpu_shares CPU share of the container.
# TYPE container_spec_cpu_shares gauge
container_spec_cpu_shares{container="",id="/",image="",name="",namespace="",pod=""} 2
container_spec_cpu_shares{container="",id="/kubepods",image="",name="",namespace="",pod=""} 2048
container_spec_cpu_shares{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 2
container_spec_cpu_shares{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 2
container_spec_cpu_shares{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 2
container_spec_cpu_shares{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 2
container_spec_cpu_shares{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 2
# HELP container_spec_memory_limit_bytes Memory limit for the container.
# TYPE container_spec_memory_limit_bytes gauge
container_spec_memory_limit_bytes{container="",id="/",image="",name="",namespace="",pod=""} 4.073259008e+09
container_spec_memory_limit_bytes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 4.073259008e+09
container_spec_memory_limit_bytes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0
container_spec_memory_limit_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
container_spec_memory_limit_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
container_spec_memory_limit_bytes{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0
container_spec_memory_limit_bytes{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
# HELP container_spec_memory_reservation_limit_bytes Memory reservation limit for the container.
# TYPE container_spec_memory_reservation_limit_bytes gauge
container_spec_memory_reservation_limit_bytes{container="",id="/",image="",name="",namespace="",pod=""} 0
container_spec_memory_reservation_limit_bytes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0
container_spec_memory_reservation_limit_bytes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0
container_spec_memory_reservation_limit_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
container_spec_memory_reservation_limit_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
container_spec_memory_reservation_limit_bytes{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0
container_spec_memory_reservation_limit_bytes{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
# HELP container_spec_memory_swap_limit_bytes Memory swap limit for the container.
# TYPE container_spec_memory_swap_limit_bytes gauge
container_spec_memory_swap_limit_bytes{container="",id="/",image="",name="",namespace="",pod=""} 0
container_spec_memory_swap_limit_bytes{container="",id="/kubepods",image="",name="",namespace="",pod=""} 0
container_spec_memory_swap_limit_bytes{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0
container_spec_memory_swap_limit_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
container_spec_memory_swap_limit_bytes{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
container_spec_memory_swap_limit_bytes{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0
container_spec_memory_swap_limit_bytes{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0
# HELP container_start_time_seconds Start time of the container since unix epoch in seconds.
# TYPE container_start_time_seconds gauge
container_start_time_seconds{container="",id="/",image="",name="",namespace="",pod=""} 1.629446536e+09
container_start_time_seconds{container="",id="/kubepods",image="",name="",namespace="",pod=""} 1.629446547e+09
container_start_time_seconds{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 1.629446547e+09
container_start_time_seconds{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.629446557e+09
container_start_time_seconds{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.629446558e+09
container_start_time_seconds{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 1.629446547e+09
container_start_time_seconds{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1.629446563e+09
# HELP container_tasks_state Number of tasks in given state
# TYPE container_tasks_state gauge
container_tasks_state{container="",id="/",image="",name="",namespace="",pod="",state="iowaiting"} 0 1629753493544
container_tasks_state{container="",id="/",image="",name="",namespace="",pod="",state="running"} 0 1629753493544
container_tasks_state{container="",id="/",image="",name="",namespace="",pod="",state="sleeping"} 0 1629753493544
container_tasks_state{container="",id="/",image="",name="",namespace="",pod="",state="stopped"} 0 1629753493544
container_tasks_state{container="",id="/",image="",name="",namespace="",pod="",state="uninterruptible"} 0 1629753493544
container_tasks_state{container="",id="/kubepods",image="",name="",namespace="",pod="",state="iowaiting"} 0 1629753493584
container_tasks_state{container="",id="/kubepods",image="",name="",namespace="",pod="",state="running"} 0 1629753493584
container_tasks_state{container="",id="/kubepods",image="",name="",namespace="",pod="",state="sleeping"} 0 1629753493584
container_tasks_state{container="",id="/kubepods",image="",name="",namespace="",pod="",state="stopped"} 0 1629753493584
container_tasks_state{container="",id="/kubepods",image="",name="",namespace="",pod="",state="uninterruptible"} 0 1629753493584
container_tasks_state{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod="",state="iowaiting"} 0 1629753493879
container_tasks_state{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod="",state="running"} 0 1629753493879
container_tasks_state{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod="",state="sleeping"} 0 1629753493879
container_tasks_state{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod="",state="stopped"} 0 1629753493879
container_tasks_state{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod="",state="uninterruptible"} 0 1629753493879
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="iowaiting"} 0 1629753484928
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="running"} 0 1629753484928
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="sleeping"} 0 1629753484928
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="stopped"} 0 1629753484928
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="uninterruptible"} 0 1629753484928
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="iowaiting"} 0 1629753492567
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="running"} 0 1629753492567
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="sleeping"} 0 1629753492567
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="stopped"} 0 1629753492567
container_tasks_state{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="uninterruptible"} 0 1629753492567
container_tasks_state{container="",id="/kubepods/burstable",image="",name="",namespace="",pod="",state="iowaiting"} 0 1629753490555
container_tasks_state{container="",id="/kubepods/burstable",image="",name="",namespace="",pod="",state="running"} 0 1629753490555
container_tasks_state{container="",id="/kubepods/burstable",image="",name="",namespace="",pod="",state="sleeping"} 0 1629753490555
container_tasks_state{container="",id="/kubepods/burstable",image="",name="",namespace="",pod="",state="stopped"} 0 1629753490555
container_tasks_state{container="",id="/kubepods/burstable",image="",name="",namespace="",pod="",state="uninterruptible"} 0 1629753490555
container_tasks_state{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="iowaiting"} 0 1629753483841
container_tasks_state{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="running"} 0 1629753483841
container_tasks_state{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="sleeping"} 0 1629753483841
container_tasks_state{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="stopped"} 0 1629753483841
container_tasks_state{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",state="uninterruptible"} 0 1629753483841
# HELP container_threads Number of threads running inside the container
# TYPE container_threads gauge
container_threads{container="",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_threads{container="",id="/kubepods",image="",name="",namespace="",pod=""} 3 1629753493584
container_threads{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 3 1629753493879
container_threads{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 3 1629753484928
container_threads{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 1 1629753492567
container_threads{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_threads{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 2 1629753483841
# HELP container_threads_max Maximum number of threads allowed inside the container, infinity if value is zero
# TYPE container_threads_max gauge
container_threads_max{container="",id="/",image="",name="",namespace="",pod=""} 0 1629753493544
container_threads_max{container="",id="/kubepods",image="",name="",namespace="",pod=""} 32768 1629753493584
container_threads_max{container="",id="/kubepods/besteffort",image="",name="",namespace="",pod=""} 0 1629753493879
container_threads_max{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e",image="",name="",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753484928
container_threads_max{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753492567
container_threads_max{container="",id="/kubepods/burstable",image="",name="",namespace="",pod=""} 0 1629753490555
container_threads_max{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx"} 0 1629753483841
# HELP container_ulimits_soft Soft ulimit values for the container root process. Unlimited if -1, except priority and nice
# TYPE container_ulimits_soft gauge
container_ulimits_soft{container="",id="/",image="",name="",namespace="",pod="",ulimit="max_open_files"} 1024 1629753493544
container_ulimits_soft{container="",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",image="sha256:feb6125b3ec84f115efdf32f32da4c7b8c65b0fe8fa68bfc9a1a424c6da4c1ac",name="f84d06c9144b7e22f2f70edf0d270922712fa44c05fadd7dd6e6203b900927c2",namespace="default",pod="sample-app-55947f9c75-fqcgx",ulimit="max_open_files"} 1024 1629753492567
container_ulimits_soft{container="hey-yo",id="/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",image="public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",name="e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",namespace="default",pod="sample-app-55947f9c75-fqcgx",ulimit="max_open_files"} 1024 1629753483841
# HELP machine_cpu_cores Number of logical CPU cores.
# TYPE machine_cpu_cores gauge
machine_cpu_cores{boot_id="d7d11784-acb8-4a7c-ad45-3daed84cd7d8",machine_id="",system_uuid="EC261256-573E-E41D-18AB-D33663DBEDD2"} 2
# HELP machine_cpu_physical_cores Number of physical CPU cores.
# TYPE machine_cpu_physical_cores gauge
machine_cpu_physical_cores{boot_id="d7d11784-acb8-4a7c-ad45-3daed84cd7d8",machine_id="",system_uuid="EC261256-573E-E41D-18AB-D33663DBEDD2"} 1
# HELP machine_cpu_sockets Number of CPU sockets.
# TYPE machine_cpu_sockets gauge
machine_cpu_sockets{boot_id="d7d11784-acb8-4a7c-ad45-3daed84cd7d8",machine_id="",system_uuid="EC261256-573E-E41D-18AB-D33663DBEDD2"} 1
# HELP machine_memory_bytes Amount of memory installed on the machine.
# TYPE machine_memory_bytes gauge
machine_memory_bytes{boot_id="d7d11784-acb8-4a7c-ad45-3daed84cd7d8",machine_id="",system_uuid="EC261256-573E-E41D-18AB-D33663DBEDD2"} 4.073259008e+09
# HELP machine_nvm_avg_power_budget_watts NVM power budget.
# TYPE machine_nvm_avg_power_budget_watts gauge
machine_nvm_avg_power_budget_watts{boot_id="d7d11784-acb8-4a7c-ad45-3daed84cd7d8",machine_id="",system_uuid="EC261256-573E-E41D-18AB-D33663DBEDD2"} 0
# HELP machine_nvm_capacity NVM capacity value labeled by NVM mode (memory mode or app direct mode).
# TYPE machine_nvm_capacity gauge
machine_nvm_capacity{boot_id="d7d11784-acb8-4a7c-ad45-3daed84cd7d8",machine_id="",mode="app_direct_mode",system_uuid="EC261256-573E-E41D-18AB-D33663DBEDD2"} 0
machine_nvm_capacity{boot_id="d7d11784-acb8-4a7c-ad45-3daed84cd7d8",machine_id="",mode="memory_mode",system_uuid="EC261256-573E-E41D-18AB-D33663DBEDD2"} 0
# HELP machine_scrape_error 1 if there was an error while getting machine metrics, 0 otherwise.
# TYPE machine_scrape_error gauge
machine_scrape_error 0
root@nginx:/#

kubelet自体のメトリクスもある。

root@nginx:/# kubectl get --raw /api/v1/nodes/${NODE}/proxy/metrics
# HELP apiserver_audit_event_total [ALPHA] Counter of audit events generated and sent to the audit backend.
# TYPE apiserver_audit_event_total counter
apiserver_audit_event_total 0
# HELP apiserver_audit_requests_rejected_total [ALPHA] Counter of apiserver requests rejected due to an error in audit logging backend.
# TYPE apiserver_audit_requests_rejected_total counter
apiserver_audit_requests_rejected_total 0
# HELP apiserver_client_certificate_expiration_seconds [ALPHA] Distribution of the remaining lifetime on the certificate used to authenticate a request.
# TYPE apiserver_client_certificate_expiration_seconds histogram
apiserver_client_certificate_expiration_seconds_bucket{le="0"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="3600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="7200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="21600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="43200"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="86400"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="172800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="345600"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="604800"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="2.592e+06"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="7.776e+06"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="1.5552e+07"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="3.1104e+07"} 0
apiserver_client_certificate_expiration_seconds_bucket{le="+Inf"} 5096
apiserver_client_certificate_expiration_seconds_sum 1.599037534522057e+11
apiserver_client_certificate_expiration_seconds_count 5096
# HELP apiserver_envelope_encryption_dek_cache_fill_percent [ALPHA] Percent of the cache slots currently occupied by cached DEKs.
# TYPE apiserver_envelope_encryption_dek_cache_fill_percent gauge
apiserver_envelope_encryption_dek_cache_fill_percent 0
# HELP apiserver_storage_data_key_generation_duration_seconds [ALPHA] Latencies in seconds of data encryption key(DEK) generation operations.
# TYPE apiserver_storage_data_key_generation_duration_seconds histogram
apiserver_storage_data_key_generation_duration_seconds_bucket{le="5e-06"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="1e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="2e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="4e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="8e-05"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00016"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00032"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00064"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00128"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00256"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.00512"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.01024"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.02048"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="0.04096"} 0
apiserver_storage_data_key_generation_duration_seconds_bucket{le="+Inf"} 0
apiserver_storage_data_key_generation_duration_seconds_sum 0
apiserver_storage_data_key_generation_duration_seconds_count 0
# HELP apiserver_storage_data_key_generation_failures_total [ALPHA] Total number of failed data encryption key(DEK) generation operations.
# TYPE apiserver_storage_data_key_generation_failures_total counter
apiserver_storage_data_key_generation_failures_total 0
# HELP apiserver_storage_envelope_transformation_cache_misses_total [ALPHA] Total number of cache misses while accessing key decryption key(KEK).
# TYPE apiserver_storage_envelope_transformation_cache_misses_total counter
apiserver_storage_envelope_transformation_cache_misses_total 0
# HELP get_token_count [ALPHA] Counter of total Token() requests to the alternate token source
# TYPE get_token_count counter
get_token_count 0
# HELP get_token_fail_count [ALPHA] Counter of failed Token() requests to the alternate token source
# TYPE get_token_fail_count counter
get_token_fail_count 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.4e-05
go_gc_duration_seconds{quantile="0.25"} 5.3015e-05
go_gc_duration_seconds{quantile="0.5"} 5.6621e-05
go_gc_duration_seconds{quantile="0.75"} 7.5345e-05
go_gc_duration_seconds{quantile="1"} 0.001590065
go_gc_duration_seconds_sum 2.780554172
go_gc_duration_seconds_count 15589
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 137
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.16.5"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.1622056e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.65390246312e+11
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 3.165877e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 1.03193896e+09
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.00017269101449158018
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 6.035824e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.1622056e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 4.0722432e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 2.4223744e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 111868
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 3.8600704e+07
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.4946176e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.6297535836485524e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 1.032050828e+09
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 2400
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 318104
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 376832
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 2.4510336e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 711651
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 2.162688e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 2.162688e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.7415432e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 15
# HELP kubelet_certificate_manager_server_rotation_seconds [ALPHA] Histogram of the number of seconds the previous certificate lived before being rotated.
# TYPE kubelet_certificate_manager_server_rotation_seconds histogram
kubelet_certificate_manager_server_rotation_seconds_bucket{le="60"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="3600"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="14400"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="86400"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="604800"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="2.592e+06"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="7.776e+06"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="1.5552e+07"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="3.1104e+07"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="1.24416e+08"} 0
kubelet_certificate_manager_server_rotation_seconds_bucket{le="+Inf"} 0
kubelet_certificate_manager_server_rotation_seconds_sum 0
kubelet_certificate_manager_server_rotation_seconds_count 0
# HELP kubelet_certificate_manager_server_ttl_seconds [ALPHA] Gauge of the shortest TTL (time-to-live) of the Kubelet's serving certificate. The value is in seconds until certificate expiry (negative if already expired). If serving certificate is invalid or unused, the value will be +INF.
# TYPE kubelet_certificate_manager_server_ttl_seconds gauge
kubelet_certificate_manager_server_ttl_seconds 3.1228618e+07
# HELP kubelet_cgroup_manager_duration_seconds [ALPHA] Duration in seconds for cgroup manager operations. Broken down by method.
# TYPE kubelet_cgroup_manager_duration_seconds histogram
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.005"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.01"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.025"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.05"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.1"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.25"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="0.5"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="1"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="2.5"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="5"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="10"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="create",le="+Inf"} 4
kubelet_cgroup_manager_duration_seconds_sum{operation_type="create"} 0.004228409999999999
kubelet_cgroup_manager_duration_seconds_count{operation_type="create"} 4
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.005"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.01"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.025"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.05"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.1"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.25"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="0.5"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="1"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="2.5"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="5"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="10"} 10243
kubelet_cgroup_manager_duration_seconds_bucket{operation_type="update",le="+Inf"} 10243
kubelet_cgroup_manager_duration_seconds_sum{operation_type="update"} 1.470650264999995
kubelet_cgroup_manager_duration_seconds_count{operation_type="update"} 10243
# HELP kubelet_container_log_filesystem_used_bytes [ALPHA] Bytes used by the container's logs on the filesystem.
# TYPE kubelet_container_log_filesystem_used_bytes gauge
kubelet_container_log_filesystem_used_bytes{container="hey-yo",namespace="default",pod="sample-app-55947f9c75-fqcgx",uid="d568a774-833c-4bd0-bd94-9d9e5adaba4e"} 1.5015936e+07
# HELP kubelet_containers_per_pod_count [ALPHA] The number of containers per pod.
# TYPE kubelet_containers_per_pod_count histogram
kubelet_containers_per_pod_count_bucket{le="1"} 1
kubelet_containers_per_pod_count_bucket{le="2"} 1
kubelet_containers_per_pod_count_bucket{le="4"} 1
kubelet_containers_per_pod_count_bucket{le="8"} 1
kubelet_containers_per_pod_count_bucket{le="16"} 1
kubelet_containers_per_pod_count_bucket{le="+Inf"} 1
kubelet_containers_per_pod_count_sum 1
kubelet_containers_per_pod_count_count 1
# HELP kubelet_http_inflight_requests [ALPHA] Number of the inflight http requests
# TYPE kubelet_http_inflight_requests gauge
kubelet_http_inflight_requests{long_running="false",method="GET",path="metrics",server_type="readwrite"} 1
kubelet_http_inflight_requests{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite"} 0
# HELP kubelet_http_requests_duration_seconds [ALPHA] Duration in seconds to serve http requests
# TYPE kubelet_http_requests_duration_seconds histogram
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="0.005"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="0.01"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="0.025"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="0.05"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="0.1"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="0.25"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="0.5"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="1"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="2.5"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="5"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="10"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics",server_type="readwrite",le="+Inf"} 1
kubelet_http_requests_duration_seconds_sum{long_running="false",method="GET",path="metrics",server_type="readwrite"} 1.0139e-05
kubelet_http_requests_duration_seconds_count{long_running="false",method="GET",path="metrics",server_type="readwrite"} 1
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="0.005"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="0.01"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="0.025"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="0.05"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="0.1"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="0.25"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="0.5"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="1"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="2.5"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="5"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="10"} 5094
kubelet_http_requests_duration_seconds_bucket{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite",le="+Inf"} 5094
kubelet_http_requests_duration_seconds_sum{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite"} 0.012162604000000025
kubelet_http_requests_duration_seconds_count{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite"} 5094
# HELP kubelet_http_requests_total [ALPHA] Number of the http requests received since the server started
# TYPE kubelet_http_requests_total counter
kubelet_http_requests_total{long_running="false",method="GET",path="metrics",server_type="readwrite"} 2
kubelet_http_requests_total{long_running="false",method="GET",path="metrics/cadvisor",server_type="readwrite"} 5094
# HELP kubelet_node_config_error [ALPHA] This metric is true (1) if the node is experiencing a configuration-related error, false (0) otherwise.
# TYPE kubelet_node_config_error gauge
kubelet_node_config_error 0
# HELP kubelet_node_name [ALPHA] The node's name. The count is always 1.
# TYPE kubelet_node_name gauge
kubelet_node_name{node="fargate-ip-10-0-126-158.ap-northeast-1.compute.internal"} 1
# HELP kubelet_pleg_discard_events [ALPHA] The number of discard events in PLEG.
# TYPE kubelet_pleg_discard_events counter
kubelet_pleg_discard_events 0
# HELP kubelet_pleg_last_seen_seconds [ALPHA] Timestamp in seconds when PLEG was last seen active.
# TYPE kubelet_pleg_last_seen_seconds gauge
kubelet_pleg_last_seen_seconds 1.629753601e+09
# HELP kubelet_pleg_relist_duration_seconds [ALPHA] Duration in seconds for relisting pods in PLEG.
# TYPE kubelet_pleg_relist_duration_seconds histogram
kubelet_pleg_relist_duration_seconds_bucket{le="0.005"} 305904
kubelet_pleg_relist_duration_seconds_bucket{le="0.01"} 306450
kubelet_pleg_relist_duration_seconds_bucket{le="0.025"} 306582
kubelet_pleg_relist_duration_seconds_bucket{le="0.05"} 306614
kubelet_pleg_relist_duration_seconds_bucket{le="0.1"} 306621
kubelet_pleg_relist_duration_seconds_bucket{le="0.25"} 306621
kubelet_pleg_relist_duration_seconds_bucket{le="0.5"} 306621
kubelet_pleg_relist_duration_seconds_bucket{le="1"} 306621
kubelet_pleg_relist_duration_seconds_bucket{le="2.5"} 306621
kubelet_pleg_relist_duration_seconds_bucket{le="5"} 306621
kubelet_pleg_relist_duration_seconds_bucket{le="10"} 306621
kubelet_pleg_relist_duration_seconds_bucket{le="+Inf"} 306621
kubelet_pleg_relist_duration_seconds_sum 240.78444733399377
kubelet_pleg_relist_duration_seconds_count 306621
# HELP kubelet_pleg_relist_interval_seconds [ALPHA] Interval in seconds between relisting in PLEG.
# TYPE kubelet_pleg_relist_interval_seconds histogram
kubelet_pleg_relist_interval_seconds_bucket{le="0.005"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.01"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.025"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.05"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.1"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.25"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="0.5"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="1"} 0
kubelet_pleg_relist_interval_seconds_bucket{le="2.5"} 306620
kubelet_pleg_relist_interval_seconds_bucket{le="5"} 306620
kubelet_pleg_relist_interval_seconds_bucket{le="10"} 306620
kubelet_pleg_relist_interval_seconds_bucket{le="+Inf"} 306620
kubelet_pleg_relist_interval_seconds_sum 307052.8130905001
kubelet_pleg_relist_interval_seconds_count 306620
# HELP kubelet_pod_start_duration_seconds [ALPHA] Duration in seconds for a single pod to go from pending to running.
# TYPE kubelet_pod_start_duration_seconds histogram
kubelet_pod_start_duration_seconds_bucket{le="0.005"} 1
kubelet_pod_start_duration_seconds_bucket{le="0.01"} 1
kubelet_pod_start_duration_seconds_bucket{le="0.025"} 1
kubelet_pod_start_duration_seconds_bucket{le="0.05"} 1
kubelet_pod_start_duration_seconds_bucket{le="0.1"} 1
kubelet_pod_start_duration_seconds_bucket{le="0.25"} 1
kubelet_pod_start_duration_seconds_bucket{le="0.5"} 1
kubelet_pod_start_duration_seconds_bucket{le="1"} 1
kubelet_pod_start_duration_seconds_bucket{le="2.5"} 1
kubelet_pod_start_duration_seconds_bucket{le="5"} 1
kubelet_pod_start_duration_seconds_bucket{le="10"} 2
kubelet_pod_start_duration_seconds_bucket{le="+Inf"} 2
kubelet_pod_start_duration_seconds_sum 5.905271405000001
kubelet_pod_start_duration_seconds_count 2
# HELP kubelet_pod_worker_start_duration_seconds [ALPHA] Duration in seconds from seeing a pod to starting a worker.
# TYPE kubelet_pod_worker_start_duration_seconds histogram
kubelet_pod_worker_start_duration_seconds_bucket{le="0.005"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="0.01"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="0.025"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="0.05"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="0.1"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="0.25"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="0.5"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="1"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="2.5"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="5"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="10"} 1
kubelet_pod_worker_start_duration_seconds_bucket{le="+Inf"} 1
kubelet_pod_worker_start_duration_seconds_sum 0.000675364
kubelet_pod_worker_start_duration_seconds_count 1
# HELP kubelet_run_podsandbox_duration_seconds [ALPHA] Duration in seconds of the run_podsandbox operations. Broken down by RuntimeClass.Handler.
# TYPE kubelet_run_podsandbox_duration_seconds histogram
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="0.005"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="0.01"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="0.025"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="0.05"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="0.1"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="0.25"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="0.5"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="1"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="2.5"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="5"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="10"} 1
kubelet_run_podsandbox_duration_seconds_bucket{runtime_handler="",le="+Inf"} 1
kubelet_run_podsandbox_duration_seconds_sum{runtime_handler=""} 1.1761e-05
kubelet_run_podsandbox_duration_seconds_count{runtime_handler=""} 1
# HELP kubelet_running_containers [ALPHA] Number of containers currently running
# TYPE kubelet_running_containers gauge
kubelet_running_containers{container_state="running"} 1
# HELP kubelet_running_pods [ALPHA] Number of pods currently running
# TYPE kubelet_running_pods gauge
kubelet_running_pods 1
# HELP kubelet_runtime_operations_duration_seconds [ALPHA] Duration in seconds of runtime operations. Broken down by operation type.
# TYPE kubelet_runtime_operations_duration_seconds histogram
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.005"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.0125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.03125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.078125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.1953125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="0.48828125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="1.220703125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="3.0517578125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="7.62939453125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="19.073486328125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="47.6837158203125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="119.20928955078125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="298.0232238769531"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="745.0580596923828"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="container_status",le="+Inf"} 1
kubelet_runtime_operations_duration_seconds_sum{operation_type="container_status"} 0.000372997
kubelet_runtime_operations_duration_seconds_count{operation_type="container_status"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.0125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.03125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.078125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.1953125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="0.48828125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="1.220703125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="3.0517578125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="7.62939453125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="19.073486328125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="47.6837158203125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="119.20928955078125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="298.0232238769531"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="745.0580596923828"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="create_container",le="+Inf"} 1
kubelet_runtime_operations_duration_seconds_sum{operation_type="create_container"} 0.077435627
kubelet_runtime_operations_duration_seconds_count{operation_type="create_container"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.005"} 1025
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.0125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.03125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.078125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.1953125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="0.48828125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="1.220703125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="3.0517578125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="7.62939453125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="19.073486328125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="47.6837158203125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="119.20928955078125"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="298.0232238769531"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="745.0580596923828"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="image_status",le="+Inf"} 1026
kubelet_runtime_operations_duration_seconds_sum{operation_type="image_status"} 0.4683193340000005
kubelet_runtime_operations_duration_seconds_count{operation_type="image_status"} 1026
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.005"} 566909
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.0125"} 568279
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.03125"} 568475
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.078125"} 568489
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.1953125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="0.48828125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="1.220703125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="3.0517578125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="7.62939453125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="19.073486328125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="47.6837158203125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="119.20928955078125"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="298.0232238769531"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="745.0580596923828"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_containers",le="+Inf"} 568498
kubelet_runtime_operations_duration_seconds_sum{operation_type="list_containers"} 207.19300179799828
kubelet_runtime_operations_duration_seconds_count{operation_type="list_containers"} 568498
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.005"} 11246
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.0125"} 11255
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.03125"} 11256
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.078125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.1953125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="0.48828125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="1.220703125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="3.0517578125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="7.62939453125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="19.073486328125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="47.6837158203125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="119.20928955078125"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="298.0232238769531"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="745.0580596923828"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_images",le="+Inf"} 11259
kubelet_runtime_operations_duration_seconds_sum{operation_type="list_images"} 6.0245582380000195
kubelet_runtime_operations_duration_seconds_count{operation_type="list_images"} 11259
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.005"} 560046
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.0125"} 562892
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.03125"} 563346
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.078125"} 563374
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.1953125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="0.48828125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="1.220703125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="3.0517578125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="7.62939453125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="19.073486328125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="47.6837158203125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="119.20928955078125"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="298.0232238769531"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="745.0580596923828"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="list_podsandbox",le="+Inf"} 563380
kubelet_runtime_operations_duration_seconds_sum{operation_type="list_podsandbox"} 301.2414022120098
kubelet_runtime_operations_duration_seconds_count{operation_type="list_podsandbox"} 563380
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.005"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.0125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.03125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.078125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.1953125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="0.48828125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="1.220703125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="3.0517578125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="7.62939453125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="19.073486328125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="47.6837158203125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="119.20928955078125"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="298.0232238769531"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="745.0580596923828"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="podsandbox_status",le="+Inf"} 3
kubelet_runtime_operations_duration_seconds_sum{operation_type="podsandbox_status"} 0.002917863
kubelet_runtime_operations_duration_seconds_count{operation_type="podsandbox_status"} 3
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.0125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.03125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.078125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.1953125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="0.48828125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="1.220703125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="3.0517578125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="7.62939453125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="19.073486328125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="47.6837158203125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="119.20928955078125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="298.0232238769531"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="745.0580596923828"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="pull_image",le="+Inf"} 1
kubelet_runtime_operations_duration_seconds_sum{operation_type="pull_image"} 4.154795128
kubelet_runtime_operations_duration_seconds_count{operation_type="pull_image"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.0125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.03125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.078125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.1953125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="0.48828125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="1.220703125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="3.0517578125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="7.62939453125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="19.073486328125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="47.6837158203125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="119.20928955078125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="298.0232238769531"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="745.0580596923828"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="run_podsandbox",le="+Inf"} 1
kubelet_runtime_operations_duration_seconds_sum{operation_type="run_podsandbox"} 0.66189436
kubelet_runtime_operations_duration_seconds_count{operation_type="run_podsandbox"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.005"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.0125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.03125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.078125"} 0
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.1953125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="0.48828125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="1.220703125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="3.0517578125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="7.62939453125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="19.073486328125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="47.6837158203125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="119.20928955078125"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="298.0232238769531"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="745.0580596923828"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="start_container",le="+Inf"} 1
kubelet_runtime_operations_duration_seconds_sum{operation_type="start_container"} 0.142976735
kubelet_runtime_operations_duration_seconds_count{operation_type="start_container"} 1
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.005"} 61337
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.0125"} 61384
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.03125"} 61394
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.078125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.1953125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="0.48828125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="1.220703125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="3.0517578125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="7.62939453125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="19.073486328125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="47.6837158203125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="119.20928955078125"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="298.0232238769531"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="745.0580596923828"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="status",le="+Inf"} 61398
kubelet_runtime_operations_duration_seconds_sum{operation_type="status"} 25.787556264000212
kubelet_runtime_operations_duration_seconds_count{operation_type="status"} 61398
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.005"} 30516
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.0125"} 30552
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.03125"} 30560
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.078125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.1953125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="0.48828125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="1.220703125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="3.0517578125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="7.62939453125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="19.073486328125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="47.6837158203125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="119.20928955078125"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="298.0232238769531"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="745.0580596923828"} 30565
kubelet_runtime_operations_duration_seconds_bucket{operation_type="version",le="+Inf"} 30565
kubelet_runtime_operations_duration_seconds_sum{operation_type="version"} 12.40910683900004
kubelet_runtime_operations_duration_seconds_count{operation_type="version"} 30565
# HELP kubelet_runtime_operations_total [ALPHA] Cumulative number of runtime operations by operation type.
# TYPE kubelet_runtime_operations_total counter
kubelet_runtime_operations_total{operation_type="container_status"} 1
kubelet_runtime_operations_total{operation_type="create_container"} 1
kubelet_runtime_operations_total{operation_type="image_status"} 1026
kubelet_runtime_operations_total{operation_type="list_containers"} 568498
kubelet_runtime_operations_total{operation_type="list_images"} 11259
kubelet_runtime_operations_total{operation_type="list_podsandbox"} 563380
kubelet_runtime_operations_total{operation_type="podsandbox_status"} 3
kubelet_runtime_operations_total{operation_type="pull_image"} 1
kubelet_runtime_operations_total{operation_type="run_podsandbox"} 1
kubelet_runtime_operations_total{operation_type="start_container"} 1
kubelet_runtime_operations_total{operation_type="status"} 61398
kubelet_runtime_operations_total{operation_type="version"} 30565
# HELP kubelet_server_expiration_renew_errors [ALPHA] Counter of certificate renewal errors.
# TYPE kubelet_server_expiration_renew_errors counter
kubelet_server_expiration_renew_errors 0
# HELP kubernetes_build_info [ALPHA] A metric with a constant '1' value labeled by major, minor, git version, git commit, git tree state, build date, Go version, and compiler from which Kubernetes was built, and platform on which it is running.
# TYPE kubernetes_build_info gauge
kubernetes_build_info{build_date="2021-06-26T01:01:23Z",compiler="gc",git_commit="d2965f0db1071203c6f5bc662c2827c71fc8b20d",git_tree_state="clean",git_version="v1.21.2-13+d2965f0db10712",go_version="go1.16.5",major="1",minor="21+",platform="linux/amd64"} 1
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 2778.44
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 47
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.04656896e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.62944654145e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.86032128e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes -1
# HELP rest_client_exec_plugin_certificate_rotation_age [ALPHA] Histogram of the number of seconds the last auth exec plugin client certificate lived before being rotated. If auth exec plugin client certificates are unused, histogram will contain no data.
# TYPE rest_client_exec_plugin_certificate_rotation_age histogram
rest_client_exec_plugin_certificate_rotation_age_bucket{le="600"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="1800"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="3600"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="14400"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="86400"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="604800"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="2.592e+06"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="7.776e+06"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="1.5552e+07"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="3.1104e+07"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="1.24416e+08"} 0
rest_client_exec_plugin_certificate_rotation_age_bucket{le="+Inf"} 0
rest_client_exec_plugin_certificate_rotation_age_sum 0
rest_client_exec_plugin_certificate_rotation_age_count 0
# HELP rest_client_exec_plugin_ttl_seconds [ALPHA] Gauge of the shortest TTL (time-to-live) of the client certificate(s) managed by the auth exec plugin. The value is in seconds until certificate expiry (negative if already expired). If auth exec plugins are unused or manage no TLS certificates, the value will be +INF.
# TYPE rest_client_exec_plugin_ttl_seconds gauge
rest_client_exec_plugin_ttl_seconds +Inf
# HELP rest_client_request_duration_seconds [ALPHA] Request latency in seconds. Broken down by verb and URL.
# TYPE rest_client_request_duration_seconds histogram
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 0.012561949
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/configmaps?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.016"} 9
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.032"} 10
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.064"} 11
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.128"} 11
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.256"} 11
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="0.512"} 11
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST",le="+Inf"} 11
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST"} 0.17859485800000002
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events",verb="POST"} 11
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.016"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.032"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.064"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.128"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.256"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="0.512"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH",le="+Inf"} 3
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH"} 0.047552486000000005
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/events/%7Bname%7D",verb="PATCH"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.016"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.032"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.064"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.128"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.256"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="0.512"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET",le="+Inf"} 2
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET"} 0.025553367
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D",verb="GET"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.032"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.064"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.128"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.256"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="0.512"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH",le="+Inf"} 2
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH"} 0.032464837999999996
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/pods/%7Bname%7D/status",verb="PATCH"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.032"} 73
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.064"} 106
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.128"} 106
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.256"} 106
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="0.512"} 106
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST",le="+Inf"} 106
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST"} 3.172797795
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/namespaces/%7Bnamespace%7D/serviceaccounts/%7Bname%7D/token",verb="POST"} 106
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST"} 0.012779883
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes",verb="POST"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.008"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.016"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.032"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.064"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.128"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.256"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="0.512"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET",le="+Inf"} 3
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET"} 0.215241154
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D",verb="GET"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.016"} 748
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.032"} 1017
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.064"} 1021
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.128"} 1021
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.256"} 1021
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="0.512"} 1021
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH",le="+Inf"} 1021
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH"} 15.329850517999999
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D/status?timeout=10s",verb="PATCH"} 1021
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.008"} 577
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.016"} 1058
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.032"} 1839
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.064"} 29713
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.128"} 30558
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.256"} 30562
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="0.512"} 30563
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET",le="+Inf"} 30563
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET"} 1366.9477784879973
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?resourceVersion=%7Bvalue%7D&timeout=10s",verb="GET"} 30563
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.032"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.064"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.128"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.256"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="0.512"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET",le="+Inf"} 2
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET"} 0.033942869
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes/%7Bname%7D?timeout=10s",verb="GET"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.032"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.064"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 0.066807294
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/nodes?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.008"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 0.006331423
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/pods?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.032"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 0.049692239
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/api/v1/services?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.016"} 27
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.032"} 795
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.064"} 848
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.128"} 850
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.256"} 850
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="0.512"} 850
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST",le="+Inf"} 850
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST"} 20.07588460900002
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/authorization.k8s.io/v1/subjectaccessreviews",verb="POST"} 850
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST"} 0.025049219
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests",verb="POST"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.032"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.064"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.128"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET"} 0.143930012
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.032"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.064"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.128"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 0.197744508
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/certificates.k8s.io/v1/certificatesigningrequests?fieldSelector=%7Bvalue%7D&limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.064"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.128"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.256"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="0.512"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET",le="+Inf"} 2
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET"} 0.069254943
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="GET"} 2
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.008"} 41
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.016"} 11453
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.032"} 27078
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.064"} 29760
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.128"} 30033
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.256"} 30038
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="0.512"} 30039
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT",le="+Inf"} 30039
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT"} 627.2769822009973
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases/%7Bname%7D?timeout=10s",verb="PUT"} 30039
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST"} 0.012071454
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/coordination.k8s.io/v1/namespaces/%7Bnamespace%7D/leases?timeout=10s",verb="POST"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 0.030991665
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/node.k8s.io/v1/runtimeclasses?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.032"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.064"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.128"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 0.030554967
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csidrivers?limit=%7Bvalue%7D&resourceVersion=%7Bvalue%7D",verb="GET"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.016"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.032"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.064"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.128"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.256"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="0.512"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST",le="+Inf"} 1
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST"} 0.205479009
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes",verb="POST"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.001"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.002"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.004"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.008"} 0
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.016"} 1
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.032"} 3
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.064"} 4
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.128"} 4
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.256"} 4
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="0.512"} 4
rest_client_request_duration_seconds_bucket{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET",le="+Inf"} 4
rest_client_request_duration_seconds_sum{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET"} 0.106632214
rest_client_request_duration_seconds_count{url="https://178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com/apis/storage.k8s.io/v1/csinodes/%7Bname%7D",verb="GET"} 4
# HELP rest_client_requests_total [ALPHA] Number of HTTP requests, partitioned by status code, method, and host.
# TYPE rest_client_requests_total counter
rest_client_requests_total{code="200",host="178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com",method="GET"} 34663
rest_client_requests_total{code="200",host="178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com",method="PATCH"} 1026
rest_client_requests_total{code="200",host="178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com",method="PUT"} 30039
rest_client_requests_total{code="201",host="178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com",method="POST"} 971
rest_client_requests_total{code="404",host="178A26A2AE4E43C7EA7E66F83EE970C2.yl4.ap-northeast-1.eks.amazonaws.com",method="GET"} 9
# HELP storage_operation_duration_seconds [ALPHA] Storage operation duration
# TYPE storage_operation_duration_seconds histogram
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="0.1"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="0.25"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="0.5"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="1"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="2.5"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="5"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="10"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="15"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="25"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="50"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="120"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="300"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="600"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected",le="+Inf"} 1
storage_operation_duration_seconds_sum{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected"} 0.000130952
storage_operation_duration_seconds_count{migrated="false",operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected"} 1
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="0.1"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="0.25"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="0.5"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="1"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="2.5"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="5"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="10"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="15"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="25"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="50"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="120"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="300"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="600"} 4064
storage_operation_duration_seconds_bucket{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected",le="+Inf"} 4064
storage_operation_duration_seconds_sum{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected"} 4.443604261000001
storage_operation_duration_seconds_count{migrated="false",operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected"} 4064
# HELP storage_operation_status_count [ALPHA] Storage operation return statuses count (Deprecated since 1.21.0)
# TYPE storage_operation_status_count counter
storage_operation_status_count{operation_name="verify_controller_attached_volume",status="success",volume_plugin="kubernetes.io/projected"} 1
storage_operation_status_count{operation_name="volume_mount",status="success",volume_plugin="kubernetes.io/projected"} 4064
# HELP volume_manager_total_volumes [ALPHA] Number of volumes in Volume Manager
# TYPE volume_manager_total_volumes gauge
volume_manager_total_volumes{plugin_name="kubernetes.io/projected",state="actual_state_of_world"} 1
volume_manager_total_volumes{plugin_name="kubernetes.io/projected",state="desired_state_of_world"} 1
# HELP workqueue_adds_total [ALPHA] Total number of adds handled by workqueue
# TYPE workqueue_adds_total counter
workqueue_adds_total{name="DynamicCABundle-client-ca-bundle"} 5118
# HELP workqueue_depth [ALPHA] Current depth of workqueue
# TYPE workqueue_depth gauge
workqueue_depth{name="DynamicCABundle-client-ca-bundle"} 0
# HELP workqueue_longest_running_processor_seconds [ALPHA] How many seconds has the longest running processor for workqueue been running.
# TYPE workqueue_longest_running_processor_seconds gauge
workqueue_longest_running_processor_seconds{name="DynamicCABundle-client-ca-bundle"} 0
# HELP workqueue_queue_duration_seconds [ALPHA] How long in seconds an item stays in workqueue before being requested.
# TYPE workqueue_queue_duration_seconds histogram
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-08"} 0
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-07"} 0
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-06"} 0
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-06"} 2170
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-05"} 5105
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.001"} 5118
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.01"} 5118
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.1"} 5118
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1"} 5118
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="10"} 5118
workqueue_queue_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="+Inf"} 5118
workqueue_queue_duration_seconds_sum{name="DynamicCABundle-client-ca-bundle"} 0.07022981399999992
workqueue_queue_duration_seconds_count{name="DynamicCABundle-client-ca-bundle"} 5118
# HELP workqueue_retries_total [ALPHA] Total number of retries handled by workqueue
# TYPE workqueue_retries_total counter
workqueue_retries_total{name="DynamicCABundle-client-ca-bundle"} 0
# HELP workqueue_unfinished_work_seconds [ALPHA] How many seconds of work has done that is in progress and hasn't been observed by work_duration. Large values indicate stuck threads. One can deduce the number of stuck threads by observing the rate at which this increases.
# TYPE workqueue_unfinished_work_seconds gauge
workqueue_unfinished_work_seconds{name="DynamicCABundle-client-ca-bundle"} 0
# HELP workqueue_work_duration_seconds [ALPHA] How long in seconds processing an item from workqueue takes.
# TYPE workqueue_work_duration_seconds histogram
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-08"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-07"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1e-06"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-06"} 0
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="9.999999999999999e-05"} 4996
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.001"} 5115
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.01"} 5118
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="0.1"} 5118
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="1"} 5118
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="10"} 5118
workqueue_work_duration_seconds_bucket{name="DynamicCABundle-client-ca-bundle",le="+Inf"} 5118
workqueue_work_duration_seconds_sum{name="DynamicCABundle-client-ca-bundle"} 0.3032986319999991
workqueue_work_duration_seconds_count{name="DynamicCABundle-client-ca-bundle"} 5118
root@nginx:/#

参考

• https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/
• https://kubernetes.io/docs/concepts/cluster-administration/proxies/

EKS on FargateでPodのメトリクスをCloudWatchに収集する

Container InsightsはまだEKS on Fargateをサポートしていないが、Prometheusメトリクスモニタリングを使用してFargate上のPodのメトリクスを取得してみるメモ。

• https://github.com/aws/containers-roadmap/issues/920#issuecomment-812021685

クラスターの作成

1.21でクラスターを作成する。ノードなしで作成する。

cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: container-insights-fargate
  region: ap-northeast-1
  version: "1.21"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true

fargateProfiles:
  - name: fp-default
    selectors:
      - namespace: default
      - namespace: kube-system
      - namespace: amazon-cloudwatch
EOF

eksctl create cluster -f cluster.yaml

ロギング

Fargateでのマネージドなロギングを設定する。

• https://docs.aws.amazon.com/ja_jp/eks/latest/userguide/fargate-logging.html

まずNamespaceを作成する。

cat << EOF > aws-observability-namespace.yaml
kind: Namespace
apiVersion: v1
metadata:
  name: aws-observability
  labels:
    aws-observability: enabled
EOF

kubectl apply -f aws-observability-namespace.yaml

ConfigMapを作成する。

cat << EOF > aws-logging-cloudwatch-configmap.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: aws-logging
  namespace: aws-observability
  labels:
data:
  output.conf: |
    [OUTPUT]
        Name cloudwatch_logs
        Match   *
        region ap-northeast-1
        log_group_name fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit-
        auto_create_group true

  parsers.conf: |
    [PARSER]
        Name crio
        Format Regex
        Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z

  filters.conf: |
     [FILTER]
        Name parser
        Match *
        Key_name log
        Parser crio
        Reserve_Data On
        Preserve_Key On
EOF

kubectl apply -f aws-logging-cloudwatch-configmap.yaml

IAMポリシーをダウンロードする。

curl -o permissions.json https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json

IAMポリシーを作成する。

aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json

IAMポリシーをPod実行ロールにアタッチする。

aws iam list-roles | jq -r '.Roles[].RoleName | select( . | contains("FargatePodExecutionRole") )'
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
aws iam attach-role-policy \
  --policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/eks-fargate-logging-policy \
  --role-name eksctl-container-insights-FargatePodExecutionRole-1M61R5UHYEUL

サンプルPodをデプロイする。

cat << EOF > sample-app-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hey-yo
  template:
    metadata:
      labels:
        app: hey-yo
    spec:
      containers:
        - name: hey-yo
          image: public.ecr.aws/toricls/everlasting-hey-yo:latest
EOF

kubectl apply -f sample-app-deployment.yaml

メトリクス

Container InsightsのPrometheusメトリクスのモニタリングを設定する。

• https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-Setup.html
• https://github.com/aws/containers-roadmap/issues/920#issuecomment-812021685
• https://github.com/aws-samples/amazon-cloudwatch-container-insights/tree/master/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus

IAMロールを作成する。

eksctl create iamserviceaccount \
  --name cwagent-prometheus \
  --namespace amazon-cloudwatch \
  --cluster container-insights-fargate \
  --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
  --approve \
  --override-existing-serviceaccounts

Fargateプロファイルは作成済みなのでスキップ。

Deploymentのマニフェストをダウンロードして、クラスター名とリージョン名を書き換える。

wget https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks-fargate.yaml
sed -i -e "s/{{cluster_name}}/container-insights-fargate/;s/{{region_name}}/ap-northeast-1/" prometheus-eks-fargate.yaml

例のコメントにあるような設定を追加する。

cwagentconfig.jsonのmetric_declarationの下に以下を追加する。

                {
                  "source_labels": ["job"],
                  "label_matcher": "^kubernetes-nodes-cadvisor$",
                  "dimensions": [["ClusterName","namespace","pod","container"]],
                  "metric_selectors": [
                    "^container_cpu_usage_seconds_total$",
                    "^container_memory_usage_bytes$"
                  ]
                },
                {
                  "source_labels": ["job"],
                  "label_matcher": "^kubernetes-nodes-cadvisor$",
                  "dimensions": [["ClusterName","namespace","pod"]],
                  "metric_selectors": [
                    "^container_network_(receive|transmit)_bytes_total$"
                  ]
                },

prometheus.yamlのscrape_configsの下に下記を追加する。

    - job_name: kubernetes-nodes-cadvisor
      sample_limit: 10000
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - replacement: kubernetes.default.svc:443
        target_label: __address__
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        replacement: /api/v1/nodes/$1/proxy/metrics/cadvisor
        target_label: __metrics_path__
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      metric_relabel_configs:
      - source_labels: [namespace]
        action: drop
        regex: ^(amazon-cloudwatch|kube-system)$
      - source_labels: [pod]
        regex: (.+)(-\w+-\w+)|(.+)(-\w+)|(.+)
        replacement: ${1}${3}${5}
        target_label: pod
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true

Deploymentを作成する。

kubectl apply -f prometheus-eks-fargate.yaml

Podを確認する。

$ k get po -A
NAMESPACE           NAME                                  READY   STATUS    RESTARTS   AGE
amazon-cloudwatch   cwagent-prometheus-79db458586-lq9q6   1/1     Running   0          3m50s
default             sample-app-55947f9c75-bmddr           1/1     Running   0          26m
default             sample-app-55947f9c75-fqcgx           1/1     Running   0          26m
default             sample-app-55947f9c75-t88nm           1/1     Running   0          26m
kube-system         coredns-9f6f89c76-9ccp4               1/1     Running   0          51m
kube-system         coredns-9f6f89c76-lgkqs               1/1     Running   0          51m

CloudWatch Logsにログが収集できていることを確認する。

EMFを含まないログが多いが、たまにEMFを含むログが送られてきている。

こんな感じでEMFのログ取得できている。

{
    "CloudWatchMetrics": [
        {
            "Metrics": [
                {
                    "Name": "container_memory_usage_bytes"
                }
            ],
            "Dimensions": [
                [
                    "ClusterName",
                    "container",
                    "namespace",
                    "pod"
                ]
            ],
            "Namespace": "ContainerInsights/Prometheus"
        }
    ],
    "ClusterName": "container-insights-fargate",
    "Timestamp": "1629448193180",
    "Version": "0",
    "beta_kubernetes_io_arch": "amd64",
    "beta_kubernetes_io_os": "linux",
    "container": "hey-yo",
    "container_cpu_load_average_10s": 0,
    "container_file_descriptors": 7,
    "container_last_seen": 1629448195,
    "container_memory_cache": 0,
    "container_memory_mapped_file": 0,
    "container_memory_max_usage_bytes": 7573504,
    "container_memory_rss": 122880,
    "container_memory_swap": 0,
    "container_memory_usage_bytes": 1069056,
    "container_memory_working_set_bytes": 1069056,
    "container_processes": 2,
    "container_sockets": 0,
    "container_spec_cpu_period": 100000,
    "container_spec_cpu_shares": 2,
    "container_spec_memory_limit_bytes": 0,
    "container_spec_memory_reservation_limit_bytes": 0,
    "container_spec_memory_swap_limit_bytes": 0,
    "container_start_time_seconds": 1629446563,
    "container_threads": 2,
    "container_threads_max": 0,
    "eks_amazonaws_com_compute_type": "fargate",
    "failure_domain_beta_kubernetes_io_region": "ap-northeast-1",
    "failure_domain_beta_kubernetes_io_zone": "ap-northeast-1c",
    "id": "/kubepods/besteffort/podd568a774-833c-4bd0-bd94-9d9e5adaba4e/e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",
    "image": "public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c",
    "instance": "fargate-ip-10-0-126-158.ap-northeast-1.compute.internal",
    "job": "kubernetes-nodes-cadvisor",
    "kubernetes_io_arch": "amd64",
    "kubernetes_io_hostname": "ip-10-0-126-158.ap-northeast-1.compute.internal",
    "kubernetes_io_os": "linux",
    "name": "e078e206de637e5da80f357f49949739a5be1d10d758d381250b94017adf4860",
    "namespace": "default",
    "pod": "sample-app",
    "prom_metric_type": "gauge",
    "topology_kubernetes_io_region": "ap-northeast-1",
    "topology_kubernetes_io_zone": "ap-northeast-1c"
}

この方法では、Container InsightsのダッシュボードにPodのメトリクスが出るわけではない。今回のクラスターが選択肢に出てこない。

メトリクスとしてはとれている。

ただし、container_cpu_usage_seconds_totalとかなので、そのままでは使いにくい値になっているし、個別のPodやコンテナではなく集約された値になっているので、用途が限られそう。

追記

Prometheus メトリクスの変換についてユーザーガイドに記載があった。

• https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/ContainerInsights-Prometheus-metrics-conversion.html

カウンタータイプのメトリクスは、前回のスクレイプからのデルタが送信される仕様になっている。スクレイプ間隔は1分なので、container_cpu_usage_seconds_totalを60で割るとCPU使用率になると思われる。

Metric Mathで計算したところ、それっぽい値になっている。

ここで表示しているのはnginxコンテナのメトリクスで、nginxのPodを起動して中でstressコマンドを実行して上限までCPUを使っている状態。topコマンドの結果とだいたい一致している。

$ k top pod -n default
NAME                          CPU(cores)   MEMORY(bytes)
nginx                         228m         51Mi
sample-app-55947f9c75-bmddr   1m           0Mi
sample-app-55947f9c75-fqcgx   1m           1Mi
sample-app-55947f9c75-t88nm   1m           1Mi