以下のIssueにあるように、Falcoのプレビルドのカーネルモジュールがダウンロードできない場合について試したメモ。
コンポーネント | バージョン |
---|---|
EKS | 1.21 |
プラットフォームバージョン | eks.1 |
Falco | 0.29.1 |
Falcoチャート | 1.15.3 |
クラスターの準備
1.21でクラスターを作成する。
cat << EOF > cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: falco3 region: ap-northeast-1 version: "1.21" vpc: cidr: "10.0.0.0/16" availabilityZones: - ap-northeast-1a - ap-northeast-1c managedNodeGroups: - name: managed-ng-1 minSize: 2 maxSize: 2 desiredCapacity: 2 ssh: allow: true publicKeyName: default enableSsm: true privateNetworking: true cloudWatch: clusterLogging: enableTypes: ["*"] iam: withOIDC: true EOF
eksctl create cluster -f cluster.yaml
ノードのバージョンを確認する。カーネルバージョンが新しくまだサポートされていないはず。
$ k get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-101-224.ap-northeast-1.compute.internal Ready <none> 22m v1.21.2-13+d2965f0db10712 10.0.101.224 <none> Amazon Linux 2 5.4.129-62.227.amzn2.x86_64 docker://19.3.13 ip-10-0-91-102.ap-northeast-1.compute.internal Ready <none> 22m v1.21.2-13+d2965f0db10712 10.0.91.102 <none> Amazon Linux 2 5.4.129-62.227.amzn2.x86_64 docker://19.3.13
Falcoのデプロイ
Helm
FalcoのHelmチャートリポジトリを追加する。
helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update
チャートを確認する。
$ helm search repo falco NAME CHART VERSION APP VERSION DESCRIPTION falcosecurity/falco 1.15.3 0.29.1 Falco falcosecurity/falco-exporter 0.5.2 0.5.0 Prometheus Metrics Exporter for Falco output ev... falcosecurity/falcosidekick 0.3.11 2.23.1 A simple daemon to help you with falco's outputs stable/falco 1.1.8 0.0.1 DEPRECATED - incubator/falco
デフォルト設定で導入する。
helm upgrade --install falco falcosecurity/falco -n falco --create-namespace
エラーになることを確認。
$ k -n falco get po NAME READY STATUS RESTARTS AGE falco-r8lpf 0/1 Error 1 23s falco-rw95v 0/1 Error 1 23s
プレビルドモジュールのダウンロードに失敗している。
$ k -n falco logs falco-r8lpf * Setting up /usr/src links from host * Running falco-driver-loader for: falco version=0.29.1, driver version=17f5df52a7d9ed6bb12d3b1768460def8439936d * Running falco-driver-loader with: driver=module, compile=yes, download=yes * Unloading falco module, if present * Trying to load a system falco module, if present * Looking for a falco module locally (kernel 5.4.129-62.227.amzn2.x86_64) * Trying to download a prebuilt falco module from https://download.falco.org/driver/17f5df52a7d9ed6bb12d3b1768460def8439936d/falco_amazonlinux2_5.4.129-62.227.amzn2.x86_64_1.ko curl: (22) The requested URL returned error: 404 Unable to find a prebuilt falco module * Trying to dkms install falco module with GCC /usr/bin/gcc DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc) * Trying to dkms install falco module with GCC /usr/bin/gcc-8 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc-8) * Trying to dkms install falco module with GCC /usr/bin/gcc-6 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc-6) * Trying to dkms install falco module with GCC /usr/bin/gcc-5 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc-5) Consider compiling your own falco driver and loading it or getting in touch with the Falco community Tue Jul 20 22:29:24 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d) Tue Jul 20 22:29:24 2021: Falco initialized with configuration file /etc/falco/falco.yaml Tue Jul 20 22:29:24 2021: Loading rules from file /etc/falco/falco_rules.yaml: Tue Jul 20 22:29:24 2021: Loading rules from file /etc/falco/falco_rules.local.yaml: Tue Jul 20 22:29:25 2021: Unable to load the driver. Tue Jul 20 22:29:25 2021: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.
ホスト側にカーネルヘッダーを入れればよいのか確認する。
ノードグループを作成する。
cat << "EOF" > managed-ng-2.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: falco3 region: ap-northeast-1 managedNodeGroups: - name: managed-ng-2 minSize: 2 maxSize: 2 desiredCapacity: 2 privateNetworking: true iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore preBootstrapCommands: - | #!/bin/bash set -o errexit set -o pipefail set -o nounset yum -y install kernel-devel-$(uname -r) EOF
eksctl create nodegroup -f managed-ng-2.yaml
ノードを確認する。
$ k get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-10-0-67-203.ap-northeast-1.compute.internal Ready <none> 2m6s v1.21.2-13+d2965f0db10712 10.0.67.203 <none> Amazon Linux 2 5.4.129-62.227.amzn2.x86_64 docker://19.3.13 ip-10-0-72-194.ap-northeast-1.compute.internal Ready <none> 14m v1.21.2-13+d2965f0db10712 10.0.72.194 <none> Amazon Linux 2 5.4.129-62.227.amzn2.x86_64 docker://19.3.13 ip-10-0-98-194.ap-northeast-1.compute.internal Ready <none> 17m v1.21.2-13+d2965f0db10712 10.0.98.194 <none> Amazon Linux 2 5.4.129-62.227.amzn2.x86_64 docker://19.3.13 ip-10-0-99-60.ap-northeast-1.compute.internal Ready <none> 2m5s v1.21.2-13+d2965f0db10712 10.0.99.60 <none> Amazon Linux 2 5.4.129-62.227.amzn2.x86_64 docker://19.3.13
新しいカーネルヘッダーが入っているはずのノードでは成功する。
$ k -n falco get po NAME READY STATUS RESTARTS AGE falco-4xfzb 1/1 Running 0 113s falco-s5g6m 0/1 CrashLoopBackOff 7 14m falco-svvtj 0/1 CrashLoopBackOff 8 16m falco-xbf8h 1/1 Running 0 113s
カーネルモジュールがビルドされてインストールされている。
$ k -n falco logs falco-4xfzb * Setting up /usr/src links from host * Running falco-driver-loader for: falco version=0.29.1, driver version=17f5df52a7d9ed6bb12d3b1768460def8439936d * Running falco-driver-loader with: driver=module, compile=yes, download=yes * Unloading falco module, if present * Trying to load a system falco module, if present * Looking for a falco module locally (kernel 5.4.129-62.227.amzn2.x86_64) * Trying to download a prebuilt falco module from https://download.falco.org/driver/17f5df52a7d9ed6bb12d3b1768460def8439936d/falco_amazonlinux2_5.4.129-62.227.amzn2.x86_64_1.ko curl: (22) The requested URL returned error: 404 Unable to find a prebuilt falco module * Trying to dkms install falco module with GCC /usr/bin/gcc DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" Kernel preparation unnecessary for this kernel. Skipping... Building module: cleaning build area... '/tmp/falco-dkms-make'.... cleaning build area... DKMS: build completed. falco.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/5.4.129-62.227.amzn2.x86_64/kernel/extra/ depmod.... DKMS: install completed. * falco module installed in dkms, trying to insmod * Success: falco module found and loaded in dkms Tue Jul 20 23:21:29 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d) Tue Jul 20 23:21:29 2021: Falco initialized with configuration file /etc/falco/falco.yaml Tue Jul 20 23:21:29 2021: Loading rules from file /etc/falco/falco_rules.yaml: Tue Jul 20 23:21:30 2021: Loading rules from file /etc/falco/falco_rules.local.yaml: Tue Jul 20 23:21:30 2021: Starting internal webserver, listening on port 8765 23:21:30.276773000: Notice Privileged container started (user=root user_loginuid=0 command=container:a0a78572add8 k8s.ns=kube-system k8s.pod=kube-proxy-4tpvz container=a0a78572add8 image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1) k8s.ns=kube-system k8s.pod=kube-proxy-4tpvz container=a0a78572add8 23:21:30.278237000: Notice Privileged container started (user=root user_loginuid=0 command=container:8b6c4513d244 k8s.ns=kube-system k8s.pod=aws-node-wq455 container=8b6c4513d244 image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1) k8s.ns=kube-system k8s.pod=aws-node-wq455 container=8b6c4513d244 23:21:30.286672000: Notice Privileged container started (user=<NA> user_loginuid=0 command=container:bc14e1c8f69f k8s.ns=falco k8s.pod=falco-4xfzb container=bc14e1c8f69f image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1) k8s.ns=falco k8s.pod=falco-4xfzb container=bc14e1c8f69f
念のため、Init Container方式でも問題ないか確認する。
まずFalcoを削除する。
helm delete -n falco falco
既にカーネルモジュールがインストールされてしまったので、マネコンから全ノードを削除してASGに再作成させる。
ノードが新しくなったことを確認する。
$ k get node NAME STATUS ROLES AGE VERSION ip-10-0-109-245.ap-northeast-1.compute.internal Ready <none> 95s v1.21.2-13+d2965f0db10712 ip-10-0-69-39.ap-northeast-1.compute.internal Ready <none> 3m16s v1.21.2-13+d2965f0db10712 ip-10-0-75-251.ap-northeast-1.compute.internal Ready <none> 48s v1.21.2-13+d2965f0db10712 ip-10-0-99-198.ap-northeast-1.compute.internal Ready <none> 3m8s v1.21.2-13+d2965f0db10712
Init Containerを使う方法でFalcoをインストールする。
cat << EOF > values.yaml image: repository: falcosecurity/falco-no-driver extraInitContainers: - name: driver-loader image: docker.io/falcosecurity/falco-driver-loader:0.29.1 imagePullPolicy: Always securityContext: privileged: true volumeMounts: - mountPath: /host/proc name: proc-fs readOnly: true - mountPath: /host/boot name: boot-fs readOnly: true - mountPath: /host/lib/modules name: lib-modules - mountPath: /host/usr name: usr-fs readOnly: true - mountPath: /host/etc name: etc-fs readOnly: true EOF
helm upgrade --install falco falcosecurity/falco -n falco --create-namespace -f values.yaml
この方法でも問題ないことを確認。
$ k -n falco get po NAME READY STATUS RESTARTS AGE falco-9f6l5 1/1 Running 0 3m28s falco-9kpkm 0/1 Init:CrashLoopBackOff 4 3m28s falco-l4kgj 0/1 Init:Error 5 3m28s falco-s4vr8 1/1 Running 0 3m28s