Falcoでプレビルドのカーネルモジュールがダウンロードできない場合

以下のIssueにあるように、Falcoのプレビルドのカーネルモジュールがダウンロードできない場合について試したメモ。

コンポーネント バージョン
EKS 1.21
プラットフォームバージョン eks.1
Falco 0.29.1
Falcoチャート 1.15.3

クラスターの準備

1.21でクラスターを作成する。

cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: falco3
  region: ap-northeast-1
  version: "1.21"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    ssh:
      allow: true
      publicKeyName: default
      enableSsm: true
    privateNetworking: true

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true
EOF
eksctl create cluster -f cluster.yaml

ノードのバージョンを確認する。カーネルバージョンが新しくまだサポートされていないはず。

$ k get node -o wide
NAME                                              STATUS   ROLES    AGE   VERSION                     INTERNAL-IP    EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
ip-10-0-101-224.ap-northeast-1.compute.internal   Ready    <none>   22m   v1.21.2-13+d2965f0db10712   10.0.101.224   <none>        Amazon Linux 2   5.4.129-62.227.amzn2.x86_64   docker://19.3.13
ip-10-0-91-102.ap-northeast-1.compute.internal    Ready    <none>   22m   v1.21.2-13+d2965f0db10712   10.0.91.102    <none>        Amazon Linux 2   5.4.129-62.227.amzn2.x86_64   docker://19.3.13

Falcoのデプロイ

Helm

FalcoのHelmチャートリポジトリを追加する。

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

チャートを確認する。

$ helm search repo falco
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
falcosecurity/falco             1.15.3          0.29.1          Falco                                             
falcosecurity/falco-exporter    0.5.2           0.5.0           Prometheus Metrics Exporter for Falco output ev...
falcosecurity/falcosidekick     0.3.11          2.23.1          A simple daemon to help you with falco's outputs  
stable/falco                    1.1.8           0.0.1           DEPRECATED - incubator/falco

デフォルト設定で導入する。

helm upgrade --install falco falcosecurity/falco -n falco --create-namespace

エラーになることを確認。

$ k -n falco get po
NAME          READY   STATUS   RESTARTS   AGE
falco-r8lpf   0/1     Error    1          23s
falco-rw95v   0/1     Error    1          23s

プレビルドモジュールのダウンロードに失敗している。

$ k -n falco logs falco-r8lpf
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.29.1, driver version=17f5df52a7d9ed6bb12d3b1768460def8439936d
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to load a system falco module, if present
* Looking for a falco module locally (kernel 5.4.129-62.227.amzn2.x86_64)
* Trying to download a prebuilt falco module from https://download.falco.org/driver/17f5df52a7d9ed6bb12d3b1768460def8439936d/falco_amazonlinux2_5.4.129-62.227.amzn2.x86_64_1.ko
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco module
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc)
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc-8)
* Trying to dkms install falco module with GCC /usr/bin/gcc-6
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc-6)
* Trying to dkms install falco module with GCC /usr/bin/gcc-5
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/17f5df52a7d9ed6bb12d3b1768460def8439936d/build/make.log (with GCC /usr/bin/gcc-5)
Consider compiling your own falco driver and loading it or getting in touch with the Falco community
Tue Jul 20 22:29:24 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Tue Jul 20 22:29:24 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Jul 20 22:29:24 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Jul 20 22:29:24 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Jul 20 22:29:25 2021: Unable to load the driver.
Tue Jul 20 22:29:25 2021: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

ホスト側にカーネルヘッダーを入れればよいのか確認する。

ノードグループを作成する。

cat << "EOF" > managed-ng-2.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: falco3
  region: ap-northeast-1

managedNodeGroups:
  - name: managed-ng-2
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
    preBootstrapCommands:
      - |
        #!/bin/bash

        set -o errexit
        set -o pipefail
        set -o nounset

        yum -y install kernel-devel-$(uname -r)
EOF
eksctl create nodegroup -f managed-ng-2.yaml

ノードを確認する。

$ k get node -o wide
NAME                                             STATUS   ROLES    AGE    VERSION                     INTERNAL-IP   EXTERNAL-IP   OS-IMAGE         KERNEL-VERSION                CONTAINER-RUNTIME
ip-10-0-67-203.ap-northeast-1.compute.internal   Ready    <none>   2m6s   v1.21.2-13+d2965f0db10712   10.0.67.203   <none>        Amazon Linux 2   5.4.129-62.227.amzn2.x86_64   docker://19.3.13
ip-10-0-72-194.ap-northeast-1.compute.internal   Ready    <none>   14m    v1.21.2-13+d2965f0db10712   10.0.72.194   <none>        Amazon Linux 2   5.4.129-62.227.amzn2.x86_64   docker://19.3.13
ip-10-0-98-194.ap-northeast-1.compute.internal   Ready    <none>   17m    v1.21.2-13+d2965f0db10712   10.0.98.194   <none>        Amazon Linux 2   5.4.129-62.227.amzn2.x86_64   docker://19.3.13
ip-10-0-99-60.ap-northeast-1.compute.internal    Ready    <none>   2m5s   v1.21.2-13+d2965f0db10712   10.0.99.60    <none>        Amazon Linux 2   5.4.129-62.227.amzn2.x86_64   docker://19.3.13

新しいカーネルヘッダーが入っているはずのノードでは成功する。

$ k -n falco get po
NAME          READY   STATUS             RESTARTS   AGE
falco-4xfzb   1/1     Running            0          113s
falco-s5g6m   0/1     CrashLoopBackOff   7          14m
falco-svvtj   0/1     CrashLoopBackOff   8          16m
falco-xbf8h   1/1     Running            0          113s

カーネルモジュールがビルドされてインストールされている。

$ k -n falco logs falco-4xfzb
* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.29.1, driver version=17f5df52a7d9ed6bb12d3b1768460def8439936d
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to load a system falco module, if present
* Looking for a falco module locally (kernel 5.4.129-62.227.amzn2.x86_64)
* Trying to download a prebuilt falco module from https://download.falco.org/driver/17f5df52a7d9ed6bb12d3b1768460def8439936d/falco_amazonlinux2_5.4.129-62.227.amzn2.x86_64_1.ko
curl: (22) The requested URL returned error: 404 
Unable to find a prebuilt falco module
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Kernel preparation unnecessary for this kernel.  Skipping...

Building module:
cleaning build area...
'/tmp/falco-dkms-make'....
cleaning build area...

DKMS: build completed.

falco.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.129-62.227.amzn2.x86_64/kernel/extra/

depmod....

DKMS: install completed.
* falco module installed in dkms, trying to insmod
* Success: falco module found and loaded in dkms
Tue Jul 20 23:21:29 2021: Falco version 0.29.1 (driver version 17f5df52a7d9ed6bb12d3b1768460def8439936d)
Tue Jul 20 23:21:29 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Tue Jul 20 23:21:29 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Tue Jul 20 23:21:30 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Tue Jul 20 23:21:30 2021: Starting internal webserver, listening on port 8765
23:21:30.276773000: Notice Privileged container started (user=root user_loginuid=0 command=container:a0a78572add8 k8s.ns=kube-system k8s.pod=kube-proxy-4tpvz container=a0a78572add8 image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1) k8s.ns=kube-system k8s.pod=kube-proxy-4tpvz container=a0a78572add8
23:21:30.278237000: Notice Privileged container started (user=root user_loginuid=0 command=container:8b6c4513d244 k8s.ns=kube-system k8s.pod=aws-node-wq455 container=8b6c4513d244 image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1) k8s.ns=kube-system k8s.pod=aws-node-wq455 container=8b6c4513d244
23:21:30.286672000: Notice Privileged container started (user=<NA> user_loginuid=0 command=container:bc14e1c8f69f k8s.ns=falco k8s.pod=falco-4xfzb container=bc14e1c8f69f image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/pause:3.1-eksbuild.1) k8s.ns=falco k8s.pod=falco-4xfzb container=bc14e1c8f69f

念のため、Init Container方式でも問題ないか確認する。

まずFalcoを削除する。

helm delete -n falco falco

既にカーネルモジュールがインストールされてしまったので、マネコンから全ノードを削除してASGに再作成させる。

ノードが新しくなったことを確認する。

$ k get node
NAME                                              STATUS   ROLES    AGE     VERSION
ip-10-0-109-245.ap-northeast-1.compute.internal   Ready    <none>   95s     v1.21.2-13+d2965f0db10712
ip-10-0-69-39.ap-northeast-1.compute.internal     Ready    <none>   3m16s   v1.21.2-13+d2965f0db10712
ip-10-0-75-251.ap-northeast-1.compute.internal    Ready    <none>   48s     v1.21.2-13+d2965f0db10712
ip-10-0-99-198.ap-northeast-1.compute.internal    Ready    <none>   3m8s    v1.21.2-13+d2965f0db10712

Init Containerを使う方法でFalcoをインストールする。

cat << EOF > values.yaml
image:
  repository: falcosecurity/falco-no-driver

extraInitContainers:
  - name: driver-loader
    image: docker.io/falcosecurity/falco-driver-loader:0.29.1
    imagePullPolicy: Always
    securityContext:
      privileged: true
    volumeMounts:
      - mountPath: /host/proc
        name: proc-fs
        readOnly: true
      - mountPath: /host/boot
        name: boot-fs
        readOnly: true
      - mountPath: /host/lib/modules
        name: lib-modules
      - mountPath: /host/usr
        name: usr-fs
        readOnly: true
      - mountPath: /host/etc
        name: etc-fs
        readOnly: true
EOF
helm upgrade --install falco falcosecurity/falco -n falco --create-namespace -f values.yaml

この方法でも問題ないことを確認。

$ k -n falco get po
NAME          READY   STATUS                  RESTARTS   AGE
falco-9f6l5   1/1     Running                 0          3m28s
falco-9kpkm   0/1     Init:CrashLoopBackOff   4          3m28s
falco-l4kgj   0/1     Init:Error              5          3m28s
falco-s4vr8   1/1     Running                 0          3m28s