Fluent Bit のチュートリアルをやってみる

以下のブログに従って、Fluent Bit によるログの分割を試す。

Splitting an application’s logs into multiple streams: a Fluent tutorial

EKS で試すことにする。

コンポーネント	バージョン	備考
eksctl	0.86.0
Kubernetes バージョン	1.21
プラットフォームのバージョン	eks.4
AWS for Fluent Bit	2.23.0
Fluent Bit	1.8.13

クラスターの作成

1.21 のクラスターをノードなしで作成する。

CLUSTER_NAME="fluent-tutorial"
cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ${CLUSTER_NAME}
  region: ap-northeast-1
  version: "1.21"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true
EOF

eksctl create cluster -f cluster.yaml

ノードを作成する。

cat << EOF > managed-ng-1.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ${CLUSTER_NAME}
  region: ap-northeast-1

managedNodeGroups:
  - name: managed-ng-1
    minSize: 1
    maxSize: 2
    desiredCapacity: 1
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EOF

eksctl create nodegroup -f managed-ng-1.yaml

Admin ロールにも権限をつけておく。

USER_NAME="Admin:{{SessionName}}"
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/Admin"
eksctl create iamidentitymapping --cluster ${CLUSTER_NAME} --arn ${ROLE_ARN} --username ${USER_NAME} --group system:masters

サンプルアプリのデプロイ

リポジトリをクローンする。

git clone https://github.com/aws-samples/amazon-ecs-firelens-examples.git
cd amazon-ecs-firelens-examples/examples/splitting-log-streams/app

サンプルアプリのログの量が多いので、Sleep 時間を 100 ms から 1000 ms に変える。

       time.Sleep(1000 * time.Millisecond)

サンプルアプリのイメージをビルドして ECR に置く。

repository_name="fluent-tutorial-sample-app"
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
AWS_REGION=$(aws configure get region)
aws ecr get-login-password | docker login --username AWS --password-stdin https://${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
aws ecr create-repository --repository-name ${repository_name}
docker build -t ${repository_name} .
docker tag ${repository_name} ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repository_name}
docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repository_name}

アプリをローカルで実行して動作を確認する。

$ docker run --rm -it fluent-tutorial-sample-app
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:01Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:01Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:01Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:02Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:02Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:02Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:03Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:03Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:03Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:04Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:04Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:04Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:05Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:05Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:05Z","user":"TheDoctor"}

アプリを EKS にデプロイする。

$ k create deployment app --image=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repository_name}
deployment.apps/app created
$ k get po
NAME                  READY   STATUS    RESTARTS   AGE
app-f9bfb5fc9-hq24n   1/1     Running   0          36s
$ k logs --tail 10 app-f9bfb5fc9-hq24n
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:14Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:31:15Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:15Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:15Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:31:16Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:16Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:16Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:31:17Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:17Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:17Z","user":"TheDoctor"}

Fluent Bit のデプロイ

Fluent Bit の公式ドキュメントを参考にデプロイする。

https://docs.fluentbit.io/manual/installation/kubernetes

ロールを作成する。ここで適用しているマニフェストはここにある。

$ kubectl create namespace logging
namespace/logging created
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
serviceaccount/fluent-bit created
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/fluent-bit-read created
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read created

IRSA で fluent-bit の ServiceAccount にポリシーをアタッチする。

eksctl create iamserviceaccount \
    --name fluent-bit \
    --namespace logging \
    --cluster ${CLUSTER_NAME} \
    --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
    --override-existing-serviceaccounts \
    --approve

DaemonSet をデプロイする。Elasticsearch 向けのマニフェストを参考にカスタマイズする。イメージも AWS for Fluent Bit に変える。必要な ConfigMap がないがこの後作る。

cat << EOF > fluent-bit-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: fluent-bit
        image: public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
        imagePullPolicy: Always
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
EOF

$ k apply -f fluent-bit-ds.yaml
daemonset.apps/fluent-bit created

ConfigMap を作成する。公式ドキュメントの宛先が Elasticsearch のサンプルと今回のブログ記事の設定サンプルを参考にする。

cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File  parser.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/app*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [OUTPUT]
        Name   stdout
        Match  *

  parser.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On
EOF

/var/log/containers 以下のログは以下のような命名規則となる。
- <コンテナ名>-<コンテナ ID>.log
分かり易くするため、取得するログを今回のサンプルアプリのログに絞る。
インプットの Tail プラグインにはタグ拡張という機能があり、Tag に * を含めると絶対パスで置き換えられる。
- その際、 / は . に置き換えられるため、タグは例えば以下のようになる。
- app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log
- kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log
理解のためにまずは stdout に出す。

$ k apply -f fluent-bit-configmap.yaml 
configmap/fluent-bit-config created

Pod を削除して再起動する。

k -n logging delete po --all

Pod を確認する。

$ k -n logging get po
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-sfrr8   1/1     Running   0          12s

ログを確認する。デフォルトの JSON ではない形式で標準出力に出した時の 2 番目の項目がタグと思われる。

$ k -n logging logs --tail 10 fluent-bit-czzqr
[10] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973521.003913853, {"log"=>"{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:41Z","user":"TheMaster"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:41.003913853Z"}]
[11] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973521.003989795, {"log"=>"{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:41Z","user":"TheDoctor"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:41.003989795Z"}]
[12] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973522.003846128, {"log"=>"{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:38:42Z"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:42.003846128Z"}]
[13] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973522.003880772, {"log"=>"{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:42Z","user":"TheMaster"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:42.003880772Z"}]
[14] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973522.003886896, {"log"=>"{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:42Z","user":"TheDoctor"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:42.003886896Z"}]

json を展開するためのパーサーを追加する。

cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File  parser.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/app*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name          parser
        Match         *
        Key_Name      log
        Parser        json
        Reserve_Data  True

    [OUTPUT]
        Name   stdout
        Match  *

  parser.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

    [PARSER]
        Name    json
        Format  json
EOF

k apply -f fluent-bit-configmap.yaml

Pod を削除して再起動する。

k -n logging delete po --all

Pod を確認する。

$ k -n logging get po
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-rndfn   1/1     Running   0          16s

ログを確認する。展開できている。

$ k -n logging logs --tail 10 fluent-bit-rndfn
[5] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973664.038677292, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:04Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:04.038677292Z"}]
[6] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973665.039015055, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T04:41:05Z", "stream"=>"stderr", "time"=>"2022-03-11T04:41:05.039015055Z"}]
[7] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973665.039047556, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:05Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T04:41:05.039047556Z"}]
[8] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973665.039054043, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:05Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:05.039054043Z"}]
[9] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973666.039108181, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T04:41:06Z", "stream"=>"stderr", "time"=>"2022-03-11T04:41:06.039108181Z"}]
[10] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973666.039151357, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:06Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T04:41:06.039151357Z"}]
[11] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973666.039158422, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:06Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:06.039158422Z"}]
[12] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973667.039251204, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T04:41:07Z", "stream"=>"stderr", "time"=>"2022-03-11T04:41:07.039251204Z"}]
[13] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973667.039285859, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:07Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T04:41:07.039285859Z"}]
[14] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973667.039291790, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:07Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:07.03929179Z"}]

アウトプットに CloudWatch Logs を追加する。Core 機能のプラグインを使う。

cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File  parser.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/app*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name          parser
        Match         *
        Key_Name      log
        Parser        json
        Reserve_Data  True

    [OUTPUT]
        Name   stdout
        Match  *

    [OUTPUT]
        Name               cloudwatch_logs
        Match              *
        region             ap-northeast-1
        log_group_name     streams-example
        log_stream_prefix  log-level-
        auto_create_group  On

  parser.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

    [PARSER]
        Name    json
        Format  json
EOF

ロググループ名は固定で指定している。
log_stream_prefix + タグがログストリーム名になる。

k apply -f fluent-bit-configmap.yaml

Pod を削除して再起動する。

k -n logging delete po --all

Pod を確認する。

$ k -n logging get po
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-2dlg9   1/1     Running   0          10s

CloudWatch Logs でログを確認する。

f:id:sotoiwa:20220311161418p:plain

f:id:sotoiwa:20220311161431p:plain

ストリームプロセッシング

ブログ記事の例を参考にしてストリームプロセッシングを追加する。

cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File  parser.conf
        Streams_File  stream_processing.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/app*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name          parser
        Match         *
        Key_Name      log
        Parser        json
        Reserve_Data  True

    [OUTPUT]
        Name   stdout
        Match  logs.*

    [OUTPUT]
        Name               cloudwatch_logs
        Match              logs.*
        region             ap-northeast-1
        log_group_name     streams-example
        log_stream_prefix  log-level-
        auto_create_group  On

  parser.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

    [PARSER]
        Name    json
        Format  json

  stream_processing.conf: |
    [STREAM_TASK]
        Name   debug_logs
        Exec   CREATE STREAM debug WITH (tag='logs.debug') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'debug';

    [STREAM_TASK]
        Name   info_logs
        Exec   CREATE STREAM info WITH (tag='logs.info') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'info';

    [STREAM_TASK]
        Name   warn_logs
        Exec   CREATE STREAM warning WITH (tag='logs.warning') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'warning';

    [STREAM_TASK]
        Name   error_logs
        Exec   CREATE STREAM error WITH (tag='logs.error') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'error';

    [STREAM_TASK]
        Name   fatal_logs
        Exec   CREATE STREAM fatal WITH (tag='logs.fatal') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'fatal';
EOF

アウトプットでのマッチ条件を変え、オリジナルのストリームは出力しないようにする。
クエリするタグを今回に合わせて変更する。

k apply -f fluent-bit-configmap.yaml

Pod を削除して再起動する。

k -n logging delete po --all

Pod を確認する。

$ k -n logging get po
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-s52jq   1/1     Running   0          14s

ログを確認する。タグが変わっていることが確認できる。

$ k -n logging logs --tail 10 fluent-bit-s52jq
[0] logs.info: [1646980187.321632337, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:47Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:47.321632337Z"}]
[1] logs.info: [1646980188.321767575, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:48Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:48.321767575Z"}]
[2] logs.info: [1646980189.321911383, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:49Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:49.321911383Z"}]
[3] logs.info: [1646980190.322012971, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:50Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:50.322012971Z"}]
[4] logs.info: [1646980191.322113388, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:51Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:51.322113388Z"}]
[0] logs.warning: [1646980187.321664882, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:47Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:47.321664882Z"}]
[1] logs.warning: [1646980188.321804527, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:48Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:48.321804527Z"}]
[2] logs.warning: [1646980189.321949090, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:49Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:49.32194909Z"}]
[3] logs.warning: [1646980190.322057547, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:50Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:50.322057547Z"}]
[4] logs.warning: [1646980191.322148437, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:51Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:51.322148437Z"}]

CloudWatch Logs でログを確認する。

f:id:sotoiwa:20220311161454p:plain

f:id:sotoiwa:20220311161541p:plain

Rewrite Tag フィルター

ブログ記事は Fluentd での設定例だが、このまま Fluent Bit の Rewrite Tag フィルターで同じことをやってみる。

cat << "EOF" > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  fluent-bit.conf: |
    [SERVICE]
        Parsers_File  parser.conf

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/app*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name          parser
        Match         *
        Key_Name      log
        Parser        json
        Reserve_Data  True

    [FILTER]
        Name          rewrite_tag
        Match         kube.var.log.containers.app*
        Rule          $level debug logs.debug false

    [FILTER]
        Name          rewrite_tag
        Match         kube.var.log.containers.app*
        Rule          $level info logs.info false

    [FILTER]
        Name          rewrite_tag
        Match         kube.var.log.containers.app*
        Rule          $level warn logs.warn false

    [FILTER]
        Name          rewrite_tag
        Match         kube.var.log.containers.app*
        Rule          $level error logs.error false

    [FILTER]
        Name          rewrite_tag
        Match         kube.var.log.containers.app*
        Rule          $level fatal logs.fatal false

    [OUTPUT]
        Name   stdout
        Match  logs.*

    [OUTPUT]
        Name               cloudwatch_logs
        Match              logs.*
        region             ap-northeast-1
        log_group_name     streams-example
        log_stream_prefix  log-level-
        auto_create_group  On

  parser.conf: |
    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

    [PARSER]
        Name    json
        Format  json
EOF

rewrite_tag の Rule の 1 つめの引数はログのキーで、JSON パースしないで log に対してマッチを擦ることも可能。
2 つめの引数はマッチさせる正規表現
3 つめの引数は新しいタグ
3 つめの引数はオリジナルのストリームを残すかどうか

k apply -f fluent-bit-configmap.yaml

Pod を削除して再起動する。

k -n logging delete po --all

ロググループを一度削除する。

aws logs delete-log-group --log-group-name streams-example

Pod を確認する。

$ k -n logging get po
NAME               READY   STATUS    RESTARTS   AGE
fluent-bit-rsjmd   1/1     Running   0          14s

ログを確認する。タグが変わっていることが確認できる。

$ k -n logging logs --tail 10 fluent-bit-rsjmd
[3] logs.debug: [1646982484.219075724, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:04Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:04.219075724Z"}]
[4] logs.debug: [1646982485.218921402, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:05Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:05.218921402Z"}]
[0] logs.info: [1646982484.218772031, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T07:08:04Z", "stream"=>"stderr", "time"=>"2022-03-11T07:08:04.218772031Z"}]
[0] logs.debug: [1646982486.219027579, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:06Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:06.219027579Z"}]
[1] logs.debug: [1646982487.219176543, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:07Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:07.219176543Z"}]
[2] logs.debug: [1646982488.219530324, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:08Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:08.219530324Z"}]
[3] logs.debug: [1646982489.219424415, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:09Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:09.219424415Z"}]
[4] logs.debug: [1646982490.219738572, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:10Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:10.219738572Z"}]
[0] logs.info: [1646982488.219244483, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T07:08:08Z", "stream"=>"stderr", "time"=>"2022-03-11T07:08:08.219244483Z"}]
[1] logs.info: [1646982490.219489820, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T07:08:10Z", "stream"=>"stderr", "time"=>"2022-03-11T07:08:10.21948982Z"}]

CloudWatch Logs でログを確認する。

f:id:sotoiwa:20220311161558p:plain

f:id:sotoiwa:20220311161613p:plain

Fluent Logger ライブラリについては省略。