eksworkshopのEFKスタックによるロギングのチュートリアルをやってみたメモ。
コンポーネント | バージョン | 備考 |
---|---|---|
eksctl | 0.32.0 | |
Kubernetes バージョン | 1.18 | |
プラットフォームのバージョン | eks.3 | |
Elasticsearch のバージョン | 7.4 | |
AWS for Fluent Bit | 2.9.0 |
参考リンク
- Logging with Elasticsearch, Fluent Bit, and Kibana (EFK)
- Fluent Bit による集中コンテナロギング
- AWS for Fluent Bit による Kubernetes ロギング
- How to Set up Log Forwarding in a Kubernetes Cluster Using Fluent Bit
EKSクラスターの準備
クラスターを作成する。
CLUSTER_NAME="efk" eksctl create cluster \ --name=${CLUSTER_NAME} \ --version 1.18 \ --nodes=2 --managed \ --ssh-access --ssh-public-key=default
IRSAの準備をする。
eksctl utils associate-iam-oidc-provider \ --cluster ${CLUSTER_NAME} \ --approve
IAMポリシーを作成する。
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) AWS_REGION=$(aws configure get region) ES_DOMAIN_NAME="eksworkshop-logging" cat <<EOF > fluent-bit-policy.json { "Version": "2012-10-17", "Statement": [ { "Action": [ "es:ESHttp*" ], "Resource": "arn:aws:es:${AWS_REGION}:${ACCOUNT_ID}:domain/${ES_DOMAIN_NAME}", "Effect": "Allow" } ] } EOF aws iam create-policy \ --policy-name fluent-bit-policy \ --policy-document file://fluent-bit-policy.json
logging
Namespaceにfluent-bit
ServiceAccountを作成する。
kubectl create namespace logging eksctl create iamserviceaccount \ --name fluent-bit \ --namespace logging \ --cluster ${CLUSTER_NAME} \ --attach-policy-arn "arn:aws:iam::${ACCOUNT_ID}:policy/fluent-bit-policy" \ --approve \ --override-existing-serviceaccounts
ServiceAccountにアノテーションが設定されていることを確認する。
$ kubectl -n logging describe sa fluent-bit Name: fluent-bit Namespace: logging Labels: <none> Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::XXXXXXXXXXXX:role/eksctl-efk-addon-iamserviceaccount-logging-f-Role1-6IW7F7GLNKAK Image pull secrets: <none> Mountable secrets: fluent-bit-token-8jzrw Tokens: fluent-bit-token-8jzrw Events: <none>
Elasticsearchクラスターの作成
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) AWS_REGION=$(aws configure get region) ES_DOMAIN_NAME="eksworkshop-logging" ES_VERSION="7.4" ES_DOMAIN_USER="eksworkshop" ES_DOMAIN_PASSWORD="$(openssl rand -base64 12)_Ek1$" cat <<EOF > es_domain.json { "DomainName": "${ES_DOMAIN_NAME}", "ElasticsearchVersion": "${ES_VERSION}", "ElasticsearchClusterConfig": { "InstanceType": "r5.large.elasticsearch", "InstanceCount": 1, "DedicatedMasterEnabled": false, "ZoneAwarenessEnabled": false, "WarmEnabled": false }, "EBSOptions": { "EBSEnabled": true, "VolumeType": "gp2", "VolumeSize": 100 }, "AccessPolicies": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:ESHttp*\",\"Resource\":\"arn:aws:es:${AWS_REGION}:${ACCOUNT_ID}:domain/${ES_DOMAIN_NAME}/*\"}]}", "SnapshotOptions": {}, "CognitoOptions": { "Enabled": false }, "EncryptionAtRestOptions": { "Enabled": true }, "NodeToNodeEncryptionOptions": { "Enabled": true }, "DomainEndpointOptions": { "EnforceHTTPS": true, "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07" }, "AdvancedSecurityOptions": { "Enabled": true, "InternalUserDatabaseEnabled": true, "MasterUserOptions": { "MasterUserName": "${ES_DOMAIN_USER}", "MasterUserPassword": "${ES_DOMAIN_PASSWORD}" } } } EOF
Elasticsearchクラスターを作成する。
$ aws es create-elasticsearch-domain --cli-input-json file://es_domain.json { "DomainStatus": { "DomainId": "XXXXXXXXXXXX/eksworkshop-logging", "DomainName": "eksworkshop-logging", "ARN": "arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/eksworkshop-logging", "Created": true, "Deleted": false, "Processing": true, "UpgradeProcessing": false, "ElasticsearchVersion": "7.4", "ElasticsearchClusterConfig": { "InstanceType": "r5.large.elasticsearch", "InstanceCount": 1, "DedicatedMasterEnabled": false, "ZoneAwarenessEnabled": false, "WarmEnabled": false }, "EBSOptions": { "EBSEnabled": true, "VolumeType": "gp2", "VolumeSize": 100 }, "AccessPolicies": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Action\":\"es:ESHttp*\",\"Resource\":\"arn:aws:es:ap-northeast-1:XXXXXXXXXXXX:domain/eksworkshop-logging/*\"}]}", "SnapshotOptions": {}, "CognitoOptions": { "Enabled": false }, "EncryptionAtRestOptions": { "Enabled": true, "KmsKeyId": "arn:aws:kms:ap-northeast-1:XXXXXXXXXXXX:key/3d12c962-344d-4ba1-8c10-ad0a3c2d8075" }, "NodeToNodeEncryptionOptions": { "Enabled": true }, "AdvancedOptions": { "rest.action.multi.allow_explicit_index": "true" }, "ServiceSoftwareOptions": { "CurrentVersion": "", "NewVersion": "", "UpdateAvailable": false, "Cancellable": false, "UpdateStatus": "COMPLETED", "Description": "There is no software update available for this domain.", "AutomatedUpdateDate": "1970-01-01T09:00:00+09:00", "OptionalDeployment": true }, "DomainEndpointOptions": { "EnforceHTTPS": true, "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07", "CustomEndpointEnabled": false }, "AdvancedSecurityOptions": { "Enabled": true, "InternalUserDatabaseEnabled": true } } }
クラスターが起動するのを待つ。
Fluent BitからElasticsearchへのアクセスを構成する。バックエンドロールにFluent Bitに割り当てるIAMロールのARNを追加する。
FLUENTBIT_ROLE=$(eksctl get iamserviceaccount --cluster ${CLUSTER_NAME} --namespace logging -o json | jq '.iam.serviceAccounts[].status.roleARN' -r) ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint") curl -sS -u "${ES_DOMAIN_USER}:${ES_DOMAIN_PASSWORD}" \ -X PATCH \ https://${ES_ENDPOINT}/_opendistro/_security/api/rolesmapping/all_access?pretty \ -H 'Content-Type: application/json' \ -d' [ { "op": "add", "path": "/backend_roles", "value": ["'${FLUENTBIT_ROLE}'"] } ] '
以下が出力される。
{ "status" : "OK", "message" : "'all_access' updated." }
Fluent Bitのデプロイ
マニフェストを作成する。
AWS_REGION=$(aws configure get region) ES_DOMAIN_NAME="eksworkshop-logging" ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint") cat <<EOF > fluent-bit.yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluent-bit-read rules: - apiGroups: [""] resources: - namespaces - pods verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: fluent-bit-read roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluent-bit-read subjects: - kind: ServiceAccount name: fluent-bit namespace: logging --- apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: logging labels: k8s-app: fluent-bit data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE filter-kubernetes.conf @INCLUDE output-elasticsearch.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 50MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off output-elasticsearch.conf: | [OUTPUT] Name es Match * Host ${ES_ENDPOINT} Port 443 TLS On AWS_Auth On AWS_Region ${AWS_REGION} Retry_Limit 6 parsers.conf: | [PARSER] Name apache Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S --- apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit namespace: logging labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: k8s-app: fluent-bit-logging template: metadata: labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" annotations: prometheus.io/scrape: "true" prometheus.io/port: "2020" prometheus.io/path: /api/v1/metrics/prometheus spec: containers: - name: fluent-bit image: amazon/aws-for-fluent-bit:2.5.0 imagePullPolicy: Always ports: - containerPort: 2020 env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluent-bit-config mountPath: /fluent-bit/etc/ terminationGracePeriodSeconds: 10 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluent-bit-config configMap: name: fluent-bit-config serviceAccountName: fluent-bit tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - operator: "Exists" effect: "NoExecute" - operator: "Exists" effect: "NoSchedule" EOF
マニフェストをapplyする。
$ k apply -f fluent-bit.yaml clusterrole.rbac.authorization.k8s.io/fluent-bit-read created clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read created configmap/fluent-bit-config created daemonset.apps/fluent-bit created
Podを確認する。
$ k get pod -n logging NAME READY STATUS RESTARTS AGE fluent-bit-s42r5 1/1 Running 0 22s fluent-bit-sx7kw 1/1 Running 0 22s
Kibana
アクセスに必要な情報を確認する。
echo "Kibana URL: https://${ES_ENDPOINT}/_plugin/kibana/ Kibana user: ${ES_DOMAIN_USER} Kibana password: ${ES_DOMAIN_PASSWORD}"
ユーザーとパスワードを入力してログインする。
"Explore on my own"をクリック。
右下の"Connect to your Elasticsearch index"をクリック。
Index patternでは*fluent-bit*
を指定して次へ。
Time Filter field nameでは@timestamp
を選択してインデックスパターンを作成する。
Discoverタブでログが表示できることを確認する。
おまけ
Fluent Bitのマニフェストを少しだけ変えてみる。
- イメージのバージョンを2.5.0から2.9.0に変更
- イメージの取得もとをECRに変更
- 不要そうな環境変数を削除
- 不要そうなtolerations設定を削除
不要そうな設定はプルリクを送ったらマージされた。
ECRにあるイメージの一覧を取得する。
$ aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/ | jq -r '.Parameters[].Value' 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.0.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.1.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.2.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.4.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.5.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.6.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.6.1 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.7.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.9.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:latest 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.1.1 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.3.0 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.3.1 906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.8.0
マニフェストを作成する。
IMAGE="906394416424.dkr.ecr.ap-northeast-1.amazonaws.com/aws-for-fluent-bit:2.9.0" AWS_REGION=$(aws configure get region) ES_DOMAIN_NAME="eksworkshop-logging" ES_ENDPOINT=$(aws es describe-elasticsearch-domain --domain-name ${ES_DOMAIN_NAME} --output text --query "DomainStatus.Endpoint") cat <<EOF > fluent-bit.yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: fluent-bit-read rules: - apiGroups: [""] resources: - namespaces - pods verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: fluent-bit-read roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: fluent-bit-read subjects: - kind: ServiceAccount name: fluent-bit namespace: logging --- apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config namespace: logging labels: k8s-app: fluent-bit data: # Configuration files: server, input, filters and output # ====================================================== fluent-bit.conf: | [SERVICE] Flush 1 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020 @INCLUDE input-kubernetes.conf @INCLUDE filter-kubernetes.conf @INCLUDE output-elasticsearch.conf input-kubernetes.conf: | [INPUT] Name tail Tag kube.* Path /var/log/containers/*.log Parser docker DB /var/log/flb_kube.db Mem_Buf_Limit 50MB Skip_Long_Lines On Refresh_Interval 10 filter-kubernetes.conf: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off output-elasticsearch.conf: | [OUTPUT] Name es Match * Host ${ES_ENDPOINT} Port 443 TLS On AWS_Auth On AWS_Region ${AWS_REGION} Retry_Limit 6 parsers.conf: | [PARSER] Name apache Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache2 Format regex Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name apache_error Format regex Regex ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$ [PARSER] Name nginx Format regex Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$ Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name json Format json Time_Key time Time_Format %d/%b/%Y:%H:%M:%S %z [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep On [PARSER] Name syslog Format regex Regex ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S --- apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit namespace: logging labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" spec: selector: matchLabels: k8s-app: fluent-bit-logging template: metadata: labels: k8s-app: fluent-bit-logging version: v1 kubernetes.io/cluster-service: "true" annotations: prometheus.io/scrape: "true" prometheus.io/port: "2020" prometheus.io/path: /api/v1/metrics/prometheus spec: containers: - name: fluent-bit image: ${IMAGE} imagePullPolicy: Always ports: - containerPort: 2020 env: volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluent-bit-config mountPath: /fluent-bit/etc/ terminationGracePeriodSeconds: 10 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluent-bit-config configMap: name: fluent-bit-config serviceAccountName: fluent-bit EOF
更新する。
$ k apply -f fluent-bit.yaml clusterrole.rbac.authorization.k8s.io/fluent-bit-read unchanged clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read unchanged configmap/fluent-bit-config unchanged daemonset.apps/fluent-bit configured
Podを確認する。
$ k get pod -n logging NAME READY STATUS RESTARTS AGE fluent-bit-c8zvq 1/1 Running 0 14s fluent-bit-d655m 1/1 Running 0 33s
これでも新しいログがKibanaで確認できたので問題なさそう。