EFKによるロギングのチュートリアル

eksworkshopのEFKスタックによるロギングのチュートリアルをやってみたメモ。

コンポーネント バージョン 備考
eksctl 0.32.0
Kubernetes バージョン 1.18
プラットフォームのバージョン eks.3
Elasticsearch のバージョン 7.4
AWS for Fluent Bit 2.9.0

参考リンク

EKSクラスターの準備

クラスターを作成する。

CLUSTER_NAME="efk"
eksctl create cluster \
    --name=${CLUSTER_NAME} \
    --version 1.18 \
    --nodes=2 --managed \
    --ssh-access --ssh-public-key=default

IRSAの準備をする。

eksctl utils associate-iam-oidc-provider \
    --cluster ${CLUSTER_NAME} \
    --approve

IAMポリシーを作成する。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
AWS_REGION=$(aws configure get region)
ES_DOMAIN_NAME="eksworkshop-logging"
cat <<EOF > fluent-bit-policy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:ESHttp*"
            ],
            "Resource": "arn:aws:es:${AWS_REGION}:${ACCOUNT_ID}:domain/${ES_DOMAIN_NAME}",
            "Effect": "Allow"
        }
    ]
}
EOF
aws iam create-policy   \
    --policy-name fluent-bit-policy \
    --policy-document file://fluent-bit-policy.json

logging Namespaceにfluent-bit ServiceAccountを作成する。

kubectl create namespace logging
eksctl create iamserviceaccount \
    --name fluent-bit \
    --namespace logging \
    --cluster ${CLUSTER_NAME} \
    --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/fluent-bit-policy" \
    --approve \
    --override-existing-serviceaccounts

ServiceAccountにアノテーションが設定されていることを確認する。

$ kubectl -n logging describe sa fluent-bit
Name:                fluent-bit
Namespace:           logging
Labels:              <none>
Annotations:         eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/eksctl-efk-addon-iamserviceaccount-logging-f-Role1-6IW7F7GLNKAK
Image pull secrets:  <none>
Mountable secrets:   fluent-bit-token-8jzrw
Tokens:              fluent-bit-token-8jzrw
Events:              <none>

Elasticsearchクラスターの作成

AWS CLIに渡すjsonファイルを作成する。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
AWS_REGION=$(aws configure get region)
ES_DOMAIN_NAME="eksworkshop-logging"
ES_VERSION="7.4"
ES_DOMAIN_USER="eksworkshop"
ES_DOMAIN_PASSWORD="$(openssl rand -base64 12)_Ek1$"
cat <<EOF > es_domain.json
{
    "DomainName": "${ES_DOMAIN_NAME}",
    "ElasticsearchVersion": "${ES_VERSION}",
    "ElasticsearchClusterConfig": {
        "InstanceType": "r5.large.elasticsearch",
        "InstanceCount": 1,
            "DedicatedMasterEnabled": false,
            "ZoneAwarenessEnabled": false,
            "WarmEnabled": false
        },
    "EBSOptions": {
        "EBSEnabled": true,
        "VolumeType": "gp2",
        "VolumeSize": 100
    },
    "AccessPolicies":  "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:ESHttp*\",\"Resource\":\"arn:aws:es:${AWS_REGION}:${ACCOUNT_ID}:domain/${ES_DOMAIN_NAME}/*\"}]}",
    "SnapshotOptions": {},
    "CognitoOptions": {
        "Enabled": false
    },
    "EncryptionAtRestOptions": {
        "Enabled": true
    },
    "NodeToNodeEncryptionOptions": {
        "Enabled": true
    },
    "DomainEndpointOptions": {
        "EnforceHTTPS": true,
        "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07"
    },
    "AdvancedSecurityOptions": {
        "Enabled": true,
        "InternalUserDatabaseEnabled": true,
        "MasterUserOptions": {
            "MasterUserName": "${ES_DOMAIN_USER}",
            "MasterUserPassword": "${ES_DOMAIN_PASSWORD}"
        }
    }
}
EOF

Elasticsearchクラスターを作成する。

$ aws es create-elasticsearch-domain --cli-input-json file://es_domain.json
{
    "DomainStatus": {
        "DomainId": "XXXXXXXXXXXX/eksworkshop-logging",
        "DomainName": "eksworkshop-logging",
        "ARN": "arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/eksworkshop-logging",
        "Created": true,
        "Deleted": false,
        "Processing": true,
        "UpgradeProcessing": false,
        "ElasticsearchVersion": "7.4",
        "ElasticsearchClusterConfig": {
            "InstanceType": "r5.large.elasticsearch",
            "InstanceCount": 1,
            "DedicatedMasterEnabled": false,
            "ZoneAwarenessEnabled": false,
            "WarmEnabled": false
        },
        "EBSOptions": {
            "EBSEnabled": true,
            "VolumeType": "gp2",
            "VolumeSize": 100
        },
        "AccessPolicies": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:ESHttp*\",\"Resource\":\"arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/eksworkshop-logging/*\"}]}",
        "SnapshotOptions": {},
        "CognitoOptions": {
            "Enabled": false
        },
        "EncryptionAtRestOptions": {
            "Enabled": true,
            "KmsKeyId": "arn:aws:kms:ap-northeast-1:XXXXXXXXXXXX:key/3d12c962-344d-4ba1-8c10-ad0a3c2d8075"
        },
        "NodeToNodeEncryptionOptions": {
            "Enabled": true
        },
        "AdvancedOptions": {
            "rest.action.multi.allow_explicit_index": "true"
        },
        "ServiceSoftwareOptions": {
            "CurrentVersion": "",
            "NewVersion": "",
            "UpdateAvailable": false,
            "Cancellable": false,
            "UpdateStatus": "COMPLETED",
            "Description": "There is no software update available for this domain.",
            "AutomatedUpdateDate": "1970-01-01T09:00:00+09:00",
            "OptionalDeployment": true
        },
        "DomainEndpointOptions": {
            "EnforceHTTPS": true,
            "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
            "CustomEndpointEnabled": false
        },
        "AdvancedSecurityOptions": {
            "Enabled": true,
            "InternalUserDatabaseEnabled": true
        }
    }
}

クラスターが起動するのを待つ。

Fluent BitからElasticsearchへのアクセスを構成する。バックエンドロールにFluent Bitに割り当てるIAMロールのARNを追加する。

FLUENTBIT_ROLE=$(eksctl get iamserviceaccount --cluster ${CLUSTER_NAME} --namespace logging -o json | jq '.iam.serviceAccounts[].status.roleARN' -r)
ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint")
curl -sS -u "${ES_DOMAIN_USER}:${ES_DOMAIN_PASSWORD}" \
    -X PATCH \
    https://${ES_ENDPOINT}/_opendistro/_security/api/rolesmapping/all_access?pretty \
    -H 'Content-Type: application/json' \
    -d'
[
  {
    "op": "add", "path": "/backend_roles", "value": ["'${FLUENTBIT_ROLE}'"]
  }
]
'

以下が出力される。

{
  "status" : "OK",
  "message" : "'all_access' updated."
}

Fluent Bitのデプロイ

マニフェストを作成する。

AWS_REGION=$(aws configure get region)
ES_DOMAIN_NAME="eksworkshop-logging"
ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint")
cat <<EOF > fluent-bit.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${ES_ENDPOINT}
        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ${AWS_REGION}
        Retry_Limit     6

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"

spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: amazon/aws-for-fluent-bit:2.5.0
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        - name: FLUENT_ELASTICSEARCH_HOST
          value: "elasticsearch"
        - name: FLUENT_ELASTICSEARCH_PORT
          value: "9200"
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"
EOF

マニフェストをapplyする。

$ k apply -f fluent-bit.yaml
clusterrole.rbac.authorization.k8s.io/fluent-bit-read created
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read created
configmap/fluent-bit-config created
daemonset.apps/fluent-bit created

Podを確認する。

$ k get pod -n logging
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-s42r5   1/1     Running   0          22s
fluent-bit-sx7kw   1/1     Running   0          22s

Kibana

アクセスに必要な情報を確認する。

echo "Kibana URL: https://${ES_ENDPOINT}/_plugin/kibana/
Kibana user: ${ES_DOMAIN_USER}
Kibana password: ${ES_DOMAIN_PASSWORD}"

ユーザーとパスワードを入力してログインする。

f:id:sotoiwa:20201126164225p:plain

"Explore on my own"をクリック。

f:id:sotoiwa:20201126164238p:plain

右下の"Connect to your Elasticsearch index"をクリック。

f:id:sotoiwa:20201126164251p:plain

Index patternでは*fluent-bit*を指定して次へ。

f:id:sotoiwa:20201126164306p:plain

Time Filter field nameでは@timestampを選択してインデックスパターンを作成する。

f:id:sotoiwa:20201126164321p:plain

Discoverタブでログが表示できることを確認する。

f:id:sotoiwa:20201126164335p:plain

おまけ

Fluent Bitのマニフェストを少しだけ変えてみる。

  • イメージのバージョンを2.5.0から2.9.0に変更
  • イメージの取得もとをECRに変更
  • 不要そうな環境変数を削除
  • 不要そうなtolerations設定を削除

不要そうな設定はプルリクを送ったらマージされた。

ECRにあるイメージの一覧を取得する。

$ aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/ | jq -r '.Parameters[].Value'
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.0.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.1.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.2.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.4.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.5.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.6.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.6.1
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.7.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.9.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:latest
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.1.1
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.3.0
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.3.1
906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.8.0

マニフェストを作成する。

IMAGE="906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.9.0"
AWS_REGION=$(aws configure get region)
ES_DOMAIN_NAME="eksworkshop-logging"
ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint")
cat <<EOF > fluent-bit.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: logging
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     50MB
        Skip_Long_Lines   On
        Refresh_Interval  10

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           *
        Host            ${ES_ENDPOINT}
        Port            443
        TLS             On
        AWS_Auth        On
        AWS_Region      ${AWS_REGION}
        Retry_Limit     6

  parsers.conf: |
    [PARSER]
        Name   apache
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache2
        Format regex
        Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   apache_error
        Format regex
        Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

    [PARSER]
        Name   nginx
        Format regex
        Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name   json
        Format json
        Time_Key time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

    [PARSER]
        Name        syslog
        Format      regex
        Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
        Time_Key    time
        Time_Format %b %d %H:%M:%S
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"

spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
      - name: fluent-bit
        image: ${IMAGE}
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
EOF

更新する。

$ k apply -f fluent-bit.yaml
clusterrole.rbac.authorization.k8s.io/fluent-bit-read unchanged
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read unchanged
configmap/fluent-bit-config unchanged
daemonset.apps/fluent-bit configured

Podを確認する。

$ k get pod -n logging
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-c8zvq   1/1     Running   0          14s
fluent-bit-d655m   1/1     Running   0          33s

これでも新しいログがKibanaで確認できたので問題なさそう。