以下のブログに従って、Fluent Bit によるログの分割を試す。
EKS で試すことにする。
| コンポーネント |
バージョン |
備考 |
| eksctl |
0.86.0 |
|
| Kubernetes バージョン |
1.21 |
|
| プラットフォームのバージョン |
eks.4 |
|
| AWS for Fluent Bit |
2.23.0 |
|
| Fluent Bit |
1.8.13 |
|
1.21 のクラスターをノードなしで作成する。
CLUSTER_NAME="fluent-tutorial"
cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${CLUSTER_NAME}
region: ap-northeast-1
version: "1.21"
vpc:
cidr: "10.0.0.0/16"
availabilityZones:
- ap-northeast-1a
- ap-northeast-1c
cloudWatch:
clusterLogging:
enableTypes: ["*"]
iam:
withOIDC: true
EOF
eksctl create cluster -f cluster.yaml
ノードを作成する。
cat << EOF > managed-ng-1.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${CLUSTER_NAME}
region: ap-northeast-1
managedNodeGroups:
- name: managed-ng-1
minSize: 1
maxSize: 2
desiredCapacity: 1
privateNetworking: true
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EOF
eksctl create nodegroup -f managed-ng-1.yaml
Admin ロールにも権限をつけておく。
USER_NAME="Admin:{{SessionName}}"
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/Admin"
eksctl create iamidentitymapping --cluster ${CLUSTER_NAME} --arn ${ROLE_ARN} --username ${USER_NAME} --group system:masters
サンプルアプリのデプロイ
リポジトリをクローンする。
git clone https://github.com/aws-samples/amazon-ecs-firelens-examples.git
cd amazon-ecs-firelens-examples/examples/splitting-log-streams/app
サンプルアプリのログの量が多いので、Sleep 時間を 100 ms から 1000 ms に変える。
time.Sleep(1000 * time.Millisecond)
サンプルアプリのイメージをビルドして ECR に置く。
repository_name="fluent-tutorial-sample-app"
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
AWS_REGION=$(aws configure get region)
aws ecr get-login-password | docker login --username AWS --password-stdin https://${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
aws ecr create-repository --repository-name ${repository_name}
docker build -t ${repository_name} .
docker tag ${repository_name} ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repository_name}
docker push ${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repository_name}
アプリをローカルで実行して動作を確認する。
$ docker run --rm -it fluent-tutorial-sample-app
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:01Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:01Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:01Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:02Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:02Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:02Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:03Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:03Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:03Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:04Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:04Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:04Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:25:05Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:05Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:25:05Z","user":"TheDoctor"}
アプリを EKS にデプロイする。
$ k create deployment app --image=${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com/${repository_name}
deployment.apps/app created
$ k get po
NAME READY STATUS RESTARTS AGE
app-f9bfb5fc9-hq24n 1/1 Running 0 36s
$ k logs --tail 10 app-f9bfb5fc9-hq24n
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:14Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:31:15Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:15Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:15Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:31:16Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:16Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:16Z","user":"TheDoctor"}
{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:31:17Z"}
{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:17Z","user":"TheMaster"}
{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:31:17Z","user":"TheDoctor"}
Fluent Bit のデプロイ
Fluent Bit の公式ドキュメントを参考にデプロイする。
ロールを作成する。ここで適用しているマニフェストはここにある。
$ kubectl create namespace logging
namespace/logging created
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-service-account.yaml
serviceaccount/fluent-bit created
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role.yaml
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRole is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRole
clusterrole.rbac.authorization.k8s.io/fluent-bit-read created
$ kubectl create -f https://raw.githubusercontent.com/fluent/fluent-bit-kubernetes-logging/master/fluent-bit-role-binding.yaml
Warning: rbac.authorization.k8s.io/v1beta1 ClusterRoleBinding is deprecated in v1.17+, unavailable in v1.22+; use rbac.authorization.k8s.io/v1 ClusterRoleBinding
clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-read created
IRSA で fluent-bit の ServiceAccount にポリシーをアタッチする。
eksctl create iamserviceaccount \
--name fluent-bit \
--namespace logging \
--cluster ${CLUSTER_NAME} \
--attach-policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy \
--override-existing-serviceaccounts \
--approve
DaemonSet をデプロイする。Elasticsearch 向けのマニフェストを参考にカスタマイズする。イメージも AWS for Fluent Bit に変える。必要な ConfigMap がないがこの後作る。
cat << EOF > fluent-bit-ds.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluent-bit
namespace: logging
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
selector:
matchLabels:
k8s-app: fluent-bit-logging
template:
metadata:
labels:
k8s-app: fluent-bit-logging
version: v1
kubernetes.io/cluster-service: "true"
spec:
containers:
- name: fluent-bit
image: public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
imagePullPolicy: Always
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: fluent-bit-config
mountPath: /fluent-bit/etc/
terminationGracePeriodSeconds: 10
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: fluent-bit-config
configMap:
name: fluent-bit-config
serviceAccountName: fluent-bit
EOF
$ k apply -f fluent-bit-ds.yaml
daemonset.apps/fluent-bit created
ConfigMap を作成する。公式ドキュメントの宛先が Elasticsearch のサンプルと今回のブログ記事の設定サンプルを参考にする。
cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Parsers_File parser.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/app*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[OUTPUT]
Name stdout
Match *
parser.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
EOF
- /var/log/containers 以下のログは以下のような命名規則となる。
- 分かり易くするため、取得するログを今回のサンプルアプリのログに絞る。
- インプットの Tail プラグインにはタグ拡張という機能があり、Tag に
* を含めると絶対パスで置き換えられる。
- その際、
/ は . に置き換えられるため、タグは例えば以下のようになる。
- app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log
- kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log
- 理解のためにまずは stdout に出す。
$ k apply -f fluent-bit-configmap.yaml
configmap/fluent-bit-config created
Pod を削除して再起動する。
k -n logging delete po --all
Pod を確認する。
$ k -n logging get po
NAME READY STATUS RESTARTS AGE
fluent-bit-sfrr8 1/1 Running 0 12s
ログを確認する。デフォルトの JSON ではない形式で標準出力に出した時の 2 番目の項目がタグと思われる。
$ k -n logging logs --tail 10 fluent-bit-czzqr
[10] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973521.003913853, {"log"=>"{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:41Z","user":"TheMaster"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:41.003913853Z"}]
[11] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973521.003989795, {"log"=>"{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:41Z","user":"TheDoctor"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:41.003989795Z"}]
[12] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973522.003846128, {"log"=>"{"level":"info","msg":"Got a request","path":"/","requestID":"45234523","time":"2022-03-11T04:38:42Z"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:42.003846128Z"}]
[13] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973522.003880772, {"log"=>"{"level":"warning","msg":"Access denied","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:42Z","user":"TheMaster"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:42.003880772Z"}]
[14] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973522.003886896, {"log"=>"{"level":"debug","msg":"Admin access","path":"/tardis","requestID":"546745643","time":"2022-03-11T04:38:42Z","user":"TheDoctor"}
", "stream"=>"stderr", "time"=>"2022-03-11T04:38:42.003886896Z"}]
json を展開するためのパーサーを追加する。
cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Parsers_File parser.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/app*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name parser
Match *
Key_Name log
Parser json
Reserve_Data True
[OUTPUT]
Name stdout
Match *
parser.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name json
Format json
EOF
k apply -f fluent-bit-configmap.yaml
Pod を削除して再起動する。
k -n logging delete po --all
Pod を確認する。
$ k -n logging get po
NAME READY STATUS RESTARTS AGE
fluent-bit-rndfn 1/1 Running 0 16s
ログを確認する。展開できている。
$ k -n logging logs --tail 10 fluent-bit-rndfn
[5] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973664.038677292, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:04Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:04.038677292Z"}]
[6] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973665.039015055, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T04:41:05Z", "stream"=>"stderr", "time"=>"2022-03-11T04:41:05.039015055Z"}]
[7] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973665.039047556, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:05Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T04:41:05.039047556Z"}]
[8] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973665.039054043, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:05Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:05.039054043Z"}]
[9] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973666.039108181, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T04:41:06Z", "stream"=>"stderr", "time"=>"2022-03-11T04:41:06.039108181Z"}]
[10] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973666.039151357, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:06Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T04:41:06.039151357Z"}]
[11] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973666.039158422, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:06Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:06.039158422Z"}]
[12] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973667.039251204, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T04:41:07Z", "stream"=>"stderr", "time"=>"2022-03-11T04:41:07.039251204Z"}]
[13] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973667.039285859, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:07Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T04:41:07.039285859Z"}]
[14] kube.var.log.containers.app-f9bfb5fc9-hq24n_default_fluent-tutorial-sample-app-320f20b2a3f8d6f77a4bee66d619263bce2d6efbee15407a85195376dc8c0cf4.log: [1646973667.039291790, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T04:41:07Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T04:41:07.03929179Z"}]
アウトプットに CloudWatch Logs を追加する。Core 機能のプラグインを使う。
cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Parsers_File parser.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/app*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name parser
Match *
Key_Name log
Parser json
Reserve_Data True
[OUTPUT]
Name stdout
Match *
[OUTPUT]
Name cloudwatch_logs
Match *
region ap-northeast-1
log_group_name streams-example
log_stream_prefix log-level-
auto_create_group On
parser.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name json
Format json
EOF
- ロググループ名は固定で指定している。
- log_stream_prefix + タグがログストリーム名になる。
k apply -f fluent-bit-configmap.yaml
Pod を削除して再起動する。
k -n logging delete po --all
Pod を確認する。
$ k -n logging get po
NAME READY STATUS RESTARTS AGE
fluent-bit-2dlg9 1/1 Running 0 10s
CloudWatch Logs でログを確認する。
ストリームプロセッシング
ブログ記事の例を参考にしてストリームプロセッシングを追加する。
cat << EOF > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Parsers_File parser.conf
Streams_File stream_processing.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/app*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name parser
Match *
Key_Name log
Parser json
Reserve_Data True
[OUTPUT]
Name stdout
Match logs.*
[OUTPUT]
Name cloudwatch_logs
Match logs.*
region ap-northeast-1
log_group_name streams-example
log_stream_prefix log-level-
auto_create_group On
parser.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name json
Format json
stream_processing.conf: |
[STREAM_TASK]
Name debug_logs
Exec CREATE STREAM debug WITH (tag='logs.debug') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'debug';
[STREAM_TASK]
Name info_logs
Exec CREATE STREAM info WITH (tag='logs.info') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'info';
[STREAM_TASK]
Name warn_logs
Exec CREATE STREAM warning WITH (tag='logs.warning') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'warning';
[STREAM_TASK]
Name error_logs
Exec CREATE STREAM error WITH (tag='logs.error') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'error';
[STREAM_TASK]
Name fatal_logs
Exec CREATE STREAM fatal WITH (tag='logs.fatal') AS SELECT * from TAG:'kube.var.log.containers.app*' WHERE level = 'fatal';
EOF
- アウトプットでのマッチ条件を変え、オリジナルのストリームは出力しないようにする。
- クエリするタグを今回に合わせて変更する。
k apply -f fluent-bit-configmap.yaml
Pod を削除して再起動する。
k -n logging delete po --all
Pod を確認する。
$ k -n logging get po
NAME READY STATUS RESTARTS AGE
fluent-bit-s52jq 1/1 Running 0 14s
ログを確認する。タグが変わっていることが確認できる。
$ k -n logging logs --tail 10 fluent-bit-s52jq
[0] logs.info: [1646980187.321632337, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:47Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:47.321632337Z"}]
[1] logs.info: [1646980188.321767575, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:48Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:48.321767575Z"}]
[2] logs.info: [1646980189.321911383, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:49Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:49.321911383Z"}]
[3] logs.info: [1646980190.322012971, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:50Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:50.322012971Z"}]
[4] logs.info: [1646980191.322113388, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T06:29:51Z", "stream"=>"stderr", "time"=>"2022-03-11T06:29:51.322113388Z"}]
[0] logs.warning: [1646980187.321664882, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:47Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:47.321664882Z"}]
[1] logs.warning: [1646980188.321804527, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:48Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:48.321804527Z"}]
[2] logs.warning: [1646980189.321949090, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:49Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:49.32194909Z"}]
[3] logs.warning: [1646980190.322057547, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:50Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:50.322057547Z"}]
[4] logs.warning: [1646980191.322148437, {"level"=>"warning", "msg"=>"Access denied", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T06:29:51Z", "user"=>"TheMaster", "stream"=>"stderr", "time"=>"2022-03-11T06:29:51.322148437Z"}]
CloudWatch Logs でログを確認する。
ブログ記事は Fluentd での設定例だが、このまま Fluent Bit の Rewrite Tag フィルターで同じことをやってみる。
cat << "EOF" > fluent-bit-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: fluent-bit-config
namespace: logging
labels:
k8s-app: fluent-bit
data:
fluent-bit.conf: |
[SERVICE]
Parsers_File parser.conf
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/app*.log
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[FILTER]
Name parser
Match *
Key_Name log
Parser json
Reserve_Data True
[FILTER]
Name rewrite_tag
Match kube.var.log.containers.app*
Rule $level debug logs.debug false
[FILTER]
Name rewrite_tag
Match kube.var.log.containers.app*
Rule $level info logs.info false
[FILTER]
Name rewrite_tag
Match kube.var.log.containers.app*
Rule $level warn logs.warn false
[FILTER]
Name rewrite_tag
Match kube.var.log.containers.app*
Rule $level error logs.error false
[FILTER]
Name rewrite_tag
Match kube.var.log.containers.app*
Rule $level fatal logs.fatal false
[OUTPUT]
Name stdout
Match logs.*
[OUTPUT]
Name cloudwatch_logs
Match logs.*
region ap-northeast-1
log_group_name streams-example
log_stream_prefix log-level-
auto_create_group On
parser.conf: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
[PARSER]
Name json
Format json
EOF
- rewrite_tag の Rule の 1 つめの引数はログのキーで、JSON パースしないで log に対してマッチを擦ることも可能。
- 2 つめの引数はマッチさせる正規表現
- 3 つめの引数は新しいタグ
- 3 つめの引数はオリジナルのストリームを残すかどうか
k apply -f fluent-bit-configmap.yaml
Pod を削除して再起動する。
k -n logging delete po --all
ロググループを一度削除する。
aws logs delete-log-group --log-group-name streams-example
Pod を確認する。
$ k -n logging get po
NAME READY STATUS RESTARTS AGE
fluent-bit-rsjmd 1/1 Running 0 14s
ログを確認する。タグが変わっていることが確認できる。
$ k -n logging logs --tail 10 fluent-bit-rsjmd
[3] logs.debug: [1646982484.219075724, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:04Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:04.219075724Z"}]
[4] logs.debug: [1646982485.218921402, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:05Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:05.218921402Z"}]
[0] logs.info: [1646982484.218772031, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T07:08:04Z", "stream"=>"stderr", "time"=>"2022-03-11T07:08:04.218772031Z"}]
[0] logs.debug: [1646982486.219027579, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:06Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:06.219027579Z"}]
[1] logs.debug: [1646982487.219176543, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:07Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:07.219176543Z"}]
[2] logs.debug: [1646982488.219530324, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:08Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:08.219530324Z"}]
[3] logs.debug: [1646982489.219424415, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:09Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:09.219424415Z"}]
[4] logs.debug: [1646982490.219738572, {"level"=>"debug", "msg"=>"Admin access", "path"=>"/tardis", "requestID"=>"546745643", "time"=>"2022-03-11T07:08:10Z", "user"=>"TheDoctor", "stream"=>"stderr", "time"=>"2022-03-11T07:08:10.219738572Z"}]
[0] logs.info: [1646982488.219244483, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T07:08:08Z", "stream"=>"stderr", "time"=>"2022-03-11T07:08:08.219244483Z"}]
[1] logs.info: [1646982490.219489820, {"level"=>"info", "msg"=>"Got a request", "path"=>"/", "requestID"=>"45234523", "time"=>"2022-03-11T07:08:10Z", "stream"=>"stderr", "time"=>"2022-03-11T07:08:10.21948982Z"}]
CloudWatch Logs でログを確認する。
Fluent Logger ライブラリについては省略。
