EKSでContainerdを試して特にロギングまわりの違いを確認するメモ。
- Amazon EKS が Kubernetes 1.21 のサポートを開始
- https://docs.aws.amazon.com/eks/latest/userguide/eks-optimized-ami.html#containerd-bootstrap
- What is the difference between docker and containerd about logging?
- 3-shake SRE Tech Talk #2
- https://github.com/aws/containers-roadmap/issues/313
クラスターの作成
クラスターを作成する。
cat << EOF > cluster.yaml --- apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: containerd region: ap-northeast-1 version: "1.21" vpc: cidr: "10.0.0.0/16" availabilityZones: - ap-northeast-1a - ap-northeast-1c cloudWatch: clusterLogging: enableTypes: ["*"] iam: withOIDC: true EOF
eksctl create cluster -f cluster.yaml
Dockerのノードグループを作成する。
cat << "EOF" > managed-ng-docker.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: containerd region: ap-northeast-1 managedNodeGroups: - name: managed-ng-docker minSize: 2 maxSize: 2 desiredCapacity: 2 privateNetworking: true iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore EOF
eksctl create nodegroup -f managed-ng-docker.yaml
Containerdのノードグループを作成する。
EKS_VERSION=1.21
AMI_ID=$(aws ssm get-parameter \
--name /aws/service/eks/optimized-ami/${EKS_VERSION}/amazon-linux-2/recommended/image_id \
--query "Parameter.Value" --output text)
CLUSTER_NAME=containerd
cat << EOF > managed-ng-containerd.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: ${CLUSTER_NAME}
region: ap-northeast-1
managedNodeGroups:
- name: managed-ng-containerd
minSize: 2
maxSize: 2
desiredCapacity: 2
privateNetworking: true
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
ami: ${AMI_ID}
overrideBootstrapCommand: |
#!/bin/bash
/etc/eks/bootstrap.sh ${CLUSTER_NAME} --container-runtime containerd
EOF
eksctl create nodegroup -f managed-ng-containerd.yaml
ノードを確認する。
$ k get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-107-68.ap-northeast-1.compute.internal Ready <none> 21h v1.21.2-eks-55daa9d 10.0.107.68 <none> Amazon Linux 2 5.4.141-67.229.amzn2.x86_64 docker://19.3.13 ip-10-0-112-116.ap-northeast-1.compute.internal Ready <none> 23m v1.21.2-eks-55daa9d 10.0.112.116 <none> Amazon Linux 2 5.4.141-67.229.amzn2.x86_64 containerd://1.4.6 ip-10-0-88-133.ap-northeast-1.compute.internal Ready <none> 23m v1.21.2-eks-55daa9d 10.0.88.133 <none> Amazon Linux 2 5.4.141-67.229.amzn2.x86_64 containerd://1.4.6 ip-10-0-93-32.ap-northeast-1.compute.internal Ready <none> 21h v1.21.2-eks-55daa9d 10.0.93.32 <none> Amazon Linux 2 5.4.141-67.229.amzn2.x86_64 docker://19.3.13
Container Insightsのデプロイ
特にログ収集まわりが気になるので、Container Insightsをデプロイする。
メトリクス
メトリクスはCloudWatchエージェントかADOTかどちらか選べるようになっている。ADOTでセットアップする。
$ curl https://raw.githubusercontent.com/aws-observability/aws-otel-collector/main/deployment-template/eks/otel-container-insights-infra.yaml |
> kubectl apply -f -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6993 100 6993 0 0 8644 0 --:--:-- --:--:-- --:--:-- 8665
namespace/aws-otel-eks created
serviceaccount/aws-otel-sa created
clusterrole.rbac.authorization.k8s.io/aoc-agent-role created
clusterrolebinding.rbac.authorization.k8s.io/aoc-agent-role-binding created
configmap/otel-agent-conf created
daemonset.apps/aws-otel-eks-ci created
Podを確認する。
$ kubectl get pods -l name=aws-otel-eks-ci -n aws-otel-eks NAME READY STATUS RESTARTS AGE aws-otel-eks-ci-6z8jf 1/1 Running 0 54s aws-otel-eks-ci-8bwwb 1/1 Running 0 54s aws-otel-eks-ci-gbpxf 1/1 Running 0 54s aws-otel-eks-ci-znqrv 1/1 Running 0 54s
CloudWatchAgentServerPolicyをIRSAでアタッチする。
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=containerd
NAMESPACE=aws-otel-eks
SERVICE_ACCOUNT=aws-otel-sa
POLICY_ARN="arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
eksctl create iamserviceaccount \
--name ${SERVICE_ACCOUNT} \
--namespace ${NAMESPACE} \
--cluster ${CLUSTER_NAME} \
--attach-policy-arn ${POLICY_ARN} \
--override-existing-serviceaccounts \
--approve
Podを再起動しておく。
k -n aws-otel-eks delete po --all
ロギング
fluent-bitによるログ収集をセットアップする。
Namespaceを作成する。
kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/cloudwatch-namespace.yaml
ConfigMapを作成する。
ClusterName=containerd
RegionName=ap-northeast-1
FluentBitHttpPort='2020'
FluentBitReadFromHead='Off'
[[ ${FluentBitReadFromHead} = 'On' ]] && FluentBitReadFromTail='Off'|| FluentBitReadFromTail='On'
[[ -z ${FluentBitHttpPort} ]] && FluentBitHttpServer='Off' || FluentBitHttpServer='On'
kubectl create configmap fluent-bit-cluster-info \
--from-literal=cluster.name=${ClusterName} \
--from-literal=http.server=${FluentBitHttpServer} \
--from-literal=http.port=${FluentBitHttpPort} \
--from-literal=read.head=${FluentBitReadFromHead} \
--from-literal=read.tail=${FluentBitReadFromTail} \
--from-literal=logs.region=${RegionName} -n amazon-cloudwatch
fluent-bitのDaemonSetを作成する。
$ kubectl apply -f https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/daemonset/container-insights-monitoring/fluent-bit/fluent-bit.yaml serviceaccount/fluent-bit created clusterrole.rbac.authorization.k8s.io/fluent-bit-role created clusterrolebinding.rbac.authorization.k8s.io/fluent-bit-role-binding created configmap/fluent-bit-config created daemonset.apps/fluent-bit created
CloudWatchAgentServerPolicyをIRSAでアタッチする。
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=containerd
NAMESPACE=amazon-cloudwatch
SERVICE_ACCOUNT=fluent-bit
POLICY_ARN="arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
eksctl create iamserviceaccount \
--name ${SERVICE_ACCOUNT} \
--namespace ${NAMESPACE} \
--cluster ${CLUSTER_NAME} \
--attach-policy-arn ${POLICY_ARN} \
--override-existing-serviceaccounts \
--approve
Podを再起動しておく。
k -n amazon-cloudwatch delete po --all
差異の確認
hey-yoをデプロイする。
k create deployment hey-yo --image=public.ecr.aws/toricls/everlasting-hey-yo:latest --replicas=4
確認する。
$ k get po -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES hey-yo-5749db6dc8-4rtv5 1/1 Running 0 29s 10.0.86.17 ip-10-0-88-133.ap-northeast-1.compute.internal <none> <none> hey-yo-5749db6dc8-kdkm9 1/1 Running 0 29s 10.0.114.125 ip-10-0-112-116.ap-northeast-1.compute.internal <none> <none> hey-yo-5749db6dc8-kzstv 1/1 Running 0 29s 10.0.101.41 ip-10-0-107-68.ap-northeast-1.compute.internal <none> <none> hey-yo-5749db6dc8-wchcb 1/1 Running 0 29s 10.0.69.158 ip-10-0-93-32.ap-northeast-1.compute.internal <none> <none>
CloudWatch Logs
CloudWatch Logsでhey-yoのログを確認する。
ログストリーム名には違いがない。
CloudWatch Logs上で確認できるログイベントはDockerの場合が以下。
{ "log": "Hey, Yo!\n", "stream": "stdout", "kubernetes": { "pod_name": "hey-yo-5749db6dc8-kzstv", "namespace_name": "default", "pod_id": "29b38e87-ee7d-4847-be73-2e281429c7bc", "host": "ip-10-0-107-68.ap-northeast-1.compute.internal", "container_name": "everlasting-hey-yo", "docker_id": "370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07", "container_hash": "public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c", "container_image": "public.ecr.aws/toricls/everlasting-hey-yo:latest" } }
containerdの場合が以下。
{ "log": "2021-09-13T21:30:13.223922107Z stdout F Hey, Yo!", "kubernetes": { "pod_name": "hey-yo-5749db6dc8-kdkm9", "namespace_name": "default", "pod_id": "62c0657f-1292-476a-a43e-6b971a7f8909", "host": "ip-10-0-112-116.ap-northeast-1.compute.internal", "container_name": "everlasting-hey-yo", "docker_id": "30aac73f69f88093e5c6191c1361336b021a4661adae375c42048ee5814ee28f", "container_hash": "public.ecr.aws/toricls/everlasting-hey-yo@sha256:00e3b7602c020a2a367dc0f75732c718a962898d80b02ca6ca3b30bce45c794c", "container_image": "public.ecr.aws/toricls/everlasting-hey-yo:latest" } }
logフィールドの内容と、streamフィールドの有無に違いがある。
ノード上のログ
各ノードにログインして直接確認する。
docker
ログローテーションの設定は/etc/docker/daemon.jsonにある。
[ec2-user@ip-10-0-107-68 ~]$ cat /etc/docker/daemon.json { "bridge": "none", "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "10" }, "live-restore": true, "max-concurrent-downloads": 10, "default-ulimits": { "memlock": { "Hard": -1, "Name": "memlock", "Soft": -1 } } }
/var/log/containers/は/var/log/pods/へのシンボリックリンクであり、/var/log/pods/も/var/lib/docker/containers/へのシンボリックリンクとなっている。
[ec2-user@ip-10-0-107-68 ~]$ ls -l /var/log/containers/hey-yo-5749db6dc8-kzstv_default_everlasting-hey-yo-370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07.log lrwxrwxrwx 1 root root 107 Sep 13 21:29 /var/log/containers/hey-yo-5749db6dc8-kzstv_default_everlasting-hey-yo-370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07.log -> /var/log/pods/default_hey-yo-5749db6dc8-kzstv_29b38e87-ee7d-4847-be73-2e281429c7bc/everlasting-hey-yo/0.log
[ec2-user@ip-10-0-107-68 ~]$ ls -l /var/log/pods/default_hey-yo-5749db6dc8-kzstv_29b38e87-ee7d-4847-be73-2e281429c7bc/everlasting-hey-yo/0.log lrwxrwxrwx 1 root root 165 Sep 13 21:29 /var/log/pods/default_hey-yo-5749db6dc8-kzstv_29b38e87-ee7d-4847-be73-2e281429c7bc/everlasting-hey-yo/0.log -> /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log
[ec2-user@ip-10-0-107-68 ~]$ sudo ls -l /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log -rw-r----- 1 root root 73117 Sep 13 21:44 /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log
/var/lib/docker/containers/以下のログが実体で、json形式になっている。
[ec2-user@ip-10-0-107-68 ~]$ sudo tail /var/lib/docker/containers/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07/370ac58a7dd90710b1a07505ddb73957f99071be3e3c297dfde1efa9e8502a07-json.log
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:15.410476113Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:16.410806875Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:17.411347021Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:18.411845789Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:19.412326515Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:20.412898298Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:21.413820362Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:22.413840922Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:23.422186054Z"}
{"log":"Hey, Yo!\n","stream":"stdout","time":"2021-09-13T21:45:24.414696293Z"}
containerd
こちらのノードも、docker自体はインストールされているが、起動していていない。
[ec2-user@ip-10-0-112-116 ~]$ systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: inactive (dead)
Docs: https://docs.docker.com
ローテーションはkubeletも役割となり、kubeletの設定の起動引数--container-log-max-filesと--container-log-max-sizeで設定できるが、kubeletの引数にはローテーションのオプションはない。デフォルトの5と10Miが使用される。
[ec2-user@ip-10-0-112-116 ~]$ ps -ef | grep kubelet | grep -v grep root 3342 1 0 Sep12 ? 00:13:02 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/kubelet-config.json --kubeconfig /var/lib/kubelet/kubeconfig --container-runtime remote --container-runtime-endpoint unix:///run/dockershim.sock --network-plugin cni --node-ip=10.0.112.116 --pod-infra-container-image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1 --v=2
kubeletの設定ファイルにもなし。
[ec2-user@ip-10-0-112-116 ~]$ cat /etc/kubernetes/kubelet/kubelet-config.json { "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1", "address": "0.0.0.0", "authentication": { "anonymous": { "enabled": false }, "webhook": { "cacheTTL": "2m0s", "enabled": true }, "x509": { "clientCAFile": "/etc/kubernetes/pki/ca.crt" } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "clusterDomain": "cluster.local", "hairpinMode": "hairpin-veth", "readOnlyPort": 0, "cgroupDriver": "cgroupfs", "cgroupRoot": "/", "featureGates": { "RotateKubeletServerCertificate": true }, "protectKernelDefaults": true, "serializeImagePulls": false, "serverTLSBootstrap": true, "tlsCipherSuites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_RSA_WITH_AES_256_GCM_SHA384", "TLS_RSA_WITH_AES_128_GCM_SHA256" ], "clusterDNS": [ "172.20.0.10" ], "evictionHard": { "memory.available": "100Mi", "nodefs.available": "10%", "nodefs.inodesFree": "5%" }, "kubeReserved": { "cpu": "70m", "ephemeral-storage": "1Gi", "memory": "574Mi" }, "maxPods": 29 }
/var/log/containers/は/var/log/pods/へのシンボリックリンクであり、/var/log/pods/以下のファイルが実体となっている。
[ec2-user@ip-10-0-112-116 ~]$ ls -l /var/log/containers/hey-yo-5749db6dc8-kdkm9_default_everlasting-hey-yo-30aac73f69f88093e5c6191c1361336b021a4661adae375c42048ee5814ee28f.log lrwxrwxrwx 1 root root 107 Sep 13 21:29 /var/log/containers/hey-yo-5749db6dc8-kdkm9_default_everlasting-hey-yo-30aac73f69f88093e5c6191c1361336b021a4661adae375c42048ee5814ee28f.log -> /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log
[ec2-user@ip-10-0-112-116 ~]$ ls -l /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log -rw-r----- 1 root root 89647 Sep 13 22:00 /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log
ログの形式がjsonではなくなっている。FはFullで、分割された場合はPartialのPになるらしい。分割されるサイズは16KiB。
[ec2-user@ip-10-0-112-116 ~]$ sudo tail /var/log/pods/default_hey-yo-5749db6dc8-kdkm9_62c0657f-1292-476a-a43e-6b971a7f8909/everlasting-hey-yo/0.log 2021-09-13T22:00:26.158635117Z stdout F Hey, Yo! 2021-09-13T22:00:27.159121471Z stdout F Hey, Yo! 2021-09-13T22:00:28.159559657Z stdout F Hey, Yo! 2021-09-13T22:00:29.160404057Z stdout F Hey, Yo! 2021-09-13T22:00:30.160645506Z stdout F Hey, Yo! 2021-09-13T22:00:31.16125444Z stdout F Hey, Yo! 2021-09-13T22:00:32.161720548Z stdout F Hey, Yo! 2021-09-13T22:00:33.162336488Z stdout F Hey, Yo! 2021-09-13T22:00:34.163012543Z stdout F Hey, Yo! 2021-09-13T22:00:35.163265301Z stdout F Hey, Yo!
fluent-bitの設定はこれ。
data: fluent-bit.conf: | [SERVICE] Flush 5 Log_Level info Daemon off Parsers_File parsers.conf HTTP_Server ${HTTP_SERVER} HTTP_Listen 0.0.0.0 HTTP_Port ${HTTP_PORT} storage.path /var/fluent-bit/state/flb-storage/ storage.sync normal storage.checksum off storage.backlog.mem_limit 5M @INCLUDE application-log.conf @INCLUDE dataplane-log.conf @INCLUDE host-log.conf application-log.conf: | [INPUT] Name tail Tag application.* Exclude_Path /var/log/containers/cloudwatch-agent*, /var/log/containers/fluent-bit*, /var/log/containers/aws-node*, /var/log/containers/kube-proxy* Path /var/log/containers/*.log Docker_Mode On Docker_Mode_Flush 5 Docker_Mode_Parser container_firstline Parser docker DB /var/fluent-bit/state/flb_container.db Mem_Buf_Limit 50MB Skip_Long_Lines On Refresh_Interval 10 Rotate_Wait 30 storage.type filesystem Read_from_Head ${READ_FROM_HEAD} [INPUT] Name tail Tag application.* Path /var/log/containers/fluent-bit* Parser docker DB /var/fluent-bit/state/flb_log.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 Read_from_Head ${READ_FROM_HEAD} [INPUT] Name tail Tag application.* Path /var/log/containers/cloudwatch-agent* Docker_Mode On Docker_Mode_Flush 5 Docker_Mode_Parser cwagent_firstline Parser docker DB /var/fluent-bit/state/flb_cwagent.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 Read_from_Head ${READ_FROM_HEAD} [FILTER] Name kubernetes Match application.* Kube_URL https://kubernetes.default.svc:443 Kube_Tag_Prefix application.var.log.containers. Merge_Log On Merge_Log_Key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off Labels Off Annotations Off [OUTPUT] Name cloudwatch_logs Match application.* region ${AWS_REGION} log_group_name /aws/containerinsights/${CLUSTER_NAME}/application log_stream_prefix ${HOST_NAME}- auto_create_group true extra_user_agent container-insights dataplane-log.conf: | [INPUT] Name systemd Tag dataplane.systemd.* Systemd_Filter _SYSTEMD_UNIT=docker.service Systemd_Filter _SYSTEMD_UNIT=kubelet.service DB /var/fluent-bit/state/systemd.db Path /var/log/journal Read_From_Tail ${READ_FROM_TAIL} [INPUT] Name tail Tag dataplane.tail.* Path /var/log/containers/aws-node*, /var/log/containers/kube-proxy* Docker_Mode On Docker_Mode_Flush 5 Docker_Mode_Parser container_firstline Parser docker DB /var/fluent-bit/state/flb_dataplane_tail.db Mem_Buf_Limit 50MB Skip_Long_Lines On Refresh_Interval 10 Rotate_Wait 30 storage.type filesystem Read_from_Head ${READ_FROM_HEAD} [FILTER] Name modify Match dataplane.systemd.* Rename _HOSTNAME hostname Rename _SYSTEMD_UNIT systemd_unit Rename MESSAGE message Remove_regex ^((?!hostname|systemd_unit|message).)*$ [FILTER] Name aws Match dataplane.* imds_version v1 [OUTPUT] Name cloudwatch_logs Match dataplane.* region ${AWS_REGION} log_group_name /aws/containerinsights/${CLUSTER_NAME}/dataplane log_stream_prefix ${HOST_NAME}- auto_create_group true extra_user_agent container-insights host-log.conf: | [INPUT] Name tail Tag host.dmesg Path /var/log/dmesg Parser syslog DB /var/fluent-bit/state/flb_dmesg.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 Read_from_Head ${READ_FROM_HEAD} [INPUT] Name tail Tag host.messages Path /var/log/messages Parser syslog DB /var/fluent-bit/state/flb_messages.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 Read_from_Head ${READ_FROM_HEAD} [INPUT] Name tail Tag host.secure Path /var/log/secure Parser syslog DB /var/fluent-bit/state/flb_secure.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 Read_from_Head ${READ_FROM_HEAD} [FILTER] Name aws Match host.* imds_version v1 [OUTPUT] Name cloudwatch_logs Match host.* region ${AWS_REGION} log_group_name /aws/containerinsights/${CLUSTER_NAME}/host log_stream_prefix ${HOST_NAME}. auto_create_group true extra_user_agent container-insights parsers.conf: | [PARSER] Name docker Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%LZ [PARSER] Name syslog Format regex Regex ^(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$ Time_Key time Time_Format %b %d %H:%M:%S [PARSER] Name container_firstline Format regex Regex (?<log>(?<="log":")\S(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=}) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%LZ [PARSER] Name cwagent_firstline Format regex Regex (?<log>(?<="log":")\d{4}[\/-]\d{1,2}[\/-]\d{1,2}[ T]\d{2}:\d{2}:\d{2}(?!\.).*?)(?<!\\)".*(?<stream>(?<="stream":").*?)".*(?<time>\d{4}-\d{1,2}-\d{1,2}T\d{2}:\d{2}:\d{2}\.\w*).*(?=}) Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%LZ
