以下のAWSブログにしたがってEKSでFalcoを試したメモ。
もう一つFalcoに関するAWSブログがあるが、内容がちょっと古い。
| コンポーネント | バージョン |
|---|---|
| EKS | 1.18 |
| プラットフォームバージョン | eks.3 |
| Falco | 0.27.0 |
| Falcoチャート | 1.7.7 |
クラスターの準備
はじめは1.19で試したがFalcoのPodがドライバーのインストールに失敗し、CrashLoopBackOffとなり起動しなかった。
$ k logs -f falco-czwwm * Setting up /usr/src links from host * Running falco-driver-loader with: driver=module, compile=yes, download=yes * Unloading falco module, if present * Trying to dkms install falco module with GCC /usr/bin/gcc DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc) * Trying to dkms install falco module with GCC /usr/bin/gcc-8 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc-8) * Trying to dkms install falco module with GCC /usr/bin/gcc-6 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc-6) * Trying to dkms install falco module with GCC /usr/bin/gcc-5 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc-5) * Trying to load a system falco driver, if present * Trying to find locally a prebuilt falco module for kernel 5.4.95-42.163.amzn2.x86_64, if present * Trying to download prebuilt module from https://download.falco.org/driver/5c0b863ddade7a45568c0ac97d037422c9efb750/falco_amazonlinux2_5.4.95-42.163.amzn2.x86_64_1.ko curl: (22) The requested URL returned error: 404 Download failed, consider compiling your own falco module and loading it or getting in touch with the Falco community Tue Mar 9 02:23:42 2021: Falco version 0.27.0 (driver version 5c0b863ddade7a45568c0ac97d037422c9efb750) Tue Mar 9 02:23:42 2021: Falco initialized with configuration file /etc/falco/falco.yaml Tue Mar 9 02:23:42 2021: Loading rules from file /etc/falco/falco_rules.yaml: Tue Mar 9 02:23:43 2021: Loading rules from file /etc/falco/falco_rules.local.yaml: Tue Mar 9 02:23:44 2021: Unable to load the driver. Tue Mar 9 02:23:44 2021: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.
1.18でクラスターを作成する。
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: falco region: ap-northeast-1 version: "1.18" vpc: cidr: "10.1.0.0/16" availabilityZones: - ap-northeast-1a - ap-northeast-1c managedNodeGroups: - name: managed-ng-1 minSize: 2 maxSize: 2 desiredCapacity: 2 ssh: allow: true publicKeyName: default cloudWatch: clusterLogging: enableTypes: ["*"] iam: withOIDC: true
eksctl create cluster -f cluster.yaml
サンプルアプリのデプロイ
サンプルのNginx Deploymentを作成する。
apiVersion: apps/v1 kind: Deployment metadata: name: nginx2 labels: app: nginx2 spec: replicas: 3 selector: matchLabels: app: nginx2 template: metadata: labels: app: nginx2 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/arch operator: In values: - amd64 - arm64 containers: - name: nginx image: nginx:1.19.2 ports: - containerPort: 80
$ kubectl apply -f deployment.yaml deployment.apps/nginx2 created
Deploymentを確認する。
$ kubectl get deployments --all-namespaces NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default nginx2 3/3 3 3 10s kube-system coredns 2/2 2 2 26m
Fluent Bitのデプロイ
FalcoとFirelensとのインテグレーションのリポジトリをクローンする。
git clone http://github.com/sysdiglabs/falco-aws-firelens-integration
Fluent Bit用のIAMポリシーを作成する。
$ cd falco-aws-firelens-integration/eks/
$ aws iam create-policy --policy-name EKS-CloudWatchLogs --policy-document file://./fluent-bit/aws/iam_role_policy.json
{
"Policy": {
"PolicyName": "EKS-CloudWatchLogs",
"PolicyId": "ANPASYSBLVT2LCKTUML73",
"Arn": "arn:aws:iam::XXXXXXXXXXXX:policy/EKS-CloudWatchLogs",
"Path": "/",
"DefaultVersionId": "v1",
"AttachmentCount": 0,
"PermissionsBoundaryUsageCount": 0,
"IsAttachable": true,
"CreateDate": "2021-03-09T01:50:04+00:00",
"UpdateDate": "2021-03-09T01:50:04+00:00"
}
}
EKSのノードにこのポリシーをアタッチする。今ならIRSAを使うところだがこのままにする。
POLICY_ARN=$(aws iam list-policies | jq -r '.[][] | select(.PolicyName == "EKS-CloudWatchLogs") | .Arn')
ROLE_NAME=$(aws iam list-roles | jq -r '.[][] | select( .RoleName | contains("falco") and contains("NodeInstanceRole") ) | .RoleName')
aws iam attach-role-policy --role-name ${ROLE_NAME} --policy-arn ${POLICY_ARN}
Fluent Bitの設定ファイルは以下のようになっているので、リージョンだけap-northeast-1に直しておく。
apiVersion: v1 kind: ConfigMap metadata: name: fluent-bit-config labels: app.kubernetes.io/name: fluentbit data: fluent-bit.conf: | [SERVICE] Parsers_File parsers.conf [INPUT] Name tail Tag falco.* Path /var/log/containers/falco*.log Parser falco DB /var/log/flb_falco.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 [OUTPUT] Name cloudwatch Match falco.** region ap-northeast-1 log_group_name falco log_stream_name alerts auto_create_group true parsers.conf: | [PARSER] Name falco Format json Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L Time_Keep Off # Command | Decoder | Field | Optional Action # =============|==================|================= Decode_Field_As json log
Fluent Bitをデプロイする。
kubectl apply -f fluent-bit/kubernetes/
Podを確認する。
$ k get pod NAME READY STATUS RESTARTS AGE fluentbit-8s5tc 1/1 Running 0 37s fluentbit-mmqh9 1/1 Running 0 37s nginx2-5d48dd4879-k8rzr 1/1 Running 0 2m34s nginx2-5d48dd4879-klpx8 1/1 Running 0 2m34s nginx2-5d48dd4879-lc49k 1/1 Running 0 2m34s
Falcoのデプロイ
FalcoのHelmチャートリポジトリを追加する。
helm repo add falcosecurity https://falcosecurity.github.io/charts
リポジトリを確認する。
$ helm repo list NAME URL （省略） falcosecurity https://falcosecurity.github.io/charts
リポジトリをクローンする。
git clone https://github.com/falcosecurity/charts
falcoチャートのディレクトリのrulesフォルダにルールファイルが格納されている。
デフォルト設定でFalcoをインストールする。
$ helm install falco falcosecurity/falco NAME: falco LAST DEPLOYED: Tue Mar 9 13:09:41 2021 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: Falco agents are spinning up on each node in your cluster. After a few seconds, they are going to start monitoring your containers looking for security issues. No further action should be required. Tip: You can easily forward Falco events to Slack, Kafka, AWS Lambda and more with falcosidekick. Full list of outputs: https://github.com/falcosecurity/charts/falcosidekick. You can enable its deployment with `--set falcosidekick.enabled=true` or in your values.yaml. See: https://github.com/falcosecurity/charts/blob/master/falcosidekick/values.yaml for configuration values.
Tipに表示されているように以下を使うとイベントの連携が簡単にできそう。
Podを確認する。
$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE default falco-2h7sd 1/1 Running 0 32s default falco-pjlsw 1/1 Running 0 32s default fluentbit-8s5tc 1/1 Running 0 94s default fluentbit-mmqh9 1/1 Running 0 94s default nginx2-5d48dd4879-k8rzr 1/1 Running 0 3m31s default nginx2-5d48dd4879-klpx8 1/1 Running 0 3m31s default nginx2-5d48dd4879-lc49k 1/1 Running 0 3m31s kube-system aws-node-md79b 1/1 Running 0 7m41s kube-system aws-node-rtl2g 1/1 Running 0 7m44s kube-system coredns-86f7d88d77-7kggj 1/1 Running 0 30m kube-system coredns-86f7d88d77-mk88s 1/1 Running 0 30m kube-system kube-proxy-2rtww 1/1 Running 0 7m44s kube-system kube-proxy-wvfss 1/1 Running 0 7m41s
ログを確認する。ドライバーのダウンロードに成功している。
$ k logs falco-2h7sd * Setting up /usr/src links from host * Running falco-driver-loader with: driver=module, compile=yes, download=yes * Unloading falco module, if present * Trying to dkms install falco module with GCC /usr/bin/gcc DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc) * Trying to dkms install falco module with GCC /usr/bin/gcc-8 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc-8) * Trying to dkms install falco module with GCC /usr/bin/gcc-6 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc-6) * Trying to dkms install falco module with GCC /usr/bin/gcc-5 DIRECTIVE: MAKE="'/tmp/falco-dkms-make'" * Running dkms build failed, couldn't find /var/lib/dkms/falco/5c0b863ddade7a45568c0ac97d037422c9efb750/build/make.log (with GCC /usr/bin/gcc-5) * Trying to load a system falco driver, if present * Trying to find locally a prebuilt falco module for kernel 4.14.219-161.340.amzn2.x86_64, if present * Trying to download prebuilt module from https://download.falco.org/driver/5c0b863ddade7a45568c0ac97d037422c9efb750/falco_amazonlinux2_4.14.219-161.340.amzn2.x86_64_1.ko * Download succeeded * Success: falco module loaded Tue Mar 9 04:10:06 2021: Falco version 0.27.0 (driver version 5c0b863ddade7a45568c0ac97d037422c9efb750) Tue Mar 9 04:10:06 2021: Falco initialized with configuration file /etc/falco/falco.yaml Tue Mar 9 04:10:06 2021: Loading rules from file /etc/falco/falco_rules.yaml: Tue Mar 9 04:10:07 2021: Loading rules from file /etc/falco/falco_rules.local.yaml: Tue Mar 9 04:10:08 2021: Starting internal webserver, listening on port 8765 04:10:08.109765000: Notice Privileged container started (user=<NA> user_loginuid=0 command=container:e1ad82b7bf81 k8s.ns=kube-system k8s.pod=kube-proxy-wvfss container=e1ad82b7bf81 image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy:v1.18.8-eksbuild.1) k8s.ns=kube-system k8s.pod=kube-proxy-wvfss container=e1ad82b7bf81
CloudWatch Logsを確認すると、例えば上記の最後の1行が以下のように出力されているのが確認できる。Kubernetesのパーサーを通していないので、Namespaceとかのメタ情報は付与されていない。
{ "log": "04:10:08.109765000: Notice Privileged container started (user=<NA> user_loginuid=0 command=container:e1ad82b7bf81 k8s.ns=kube-system k8s.pod=kube-proxy-wvfss container=e1ad82b7bf81 image=602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy:v1.18.8-eksbuild.1) k8s.ns=kube-system k8s.pod=kube-proxy-wvfss container=e1ad82b7bf81\n", "stream": "stdout" }
テスト
例1
Nginxのコンテナに入り、/etc以下に対して操作を行う。
fluentbit-mmqh9 1/1 Running 0 11m nginx2-5d48dd4879-k8rzr 1/1 Running 0 13m nginx2-5d48dd4879-klpx8 1/1 Running 0 13m nginx2-5d48dd4879-lc49k 1/1 Running 0 13m $ k exec -it nginx2-5d48dd4879-k8rzr -- bash root@nginx2-5d48dd4879-k8rzr:/# touch /etc/2 root@nginx2-5d48dd4879-k8rzr:/# cat /etc/shadow > /dev/null 2>&1 root@nginx2-5d48dd4879-k8rzr:/#
このPodと同じノードのFalcoのPodのログに出力される。
04:20:57.661292144: Error File below /etc opened for writing (user=root user_loginuid=-1 command=touch /etc/2 parent=bash pcmdline=bash file=/etc/2 program=touch gparent=<NA> ggparent=<NA> gggparent=<NA> container_id=0dfdabe83a1e image=nginx) k8s.ns=default k8s.pod=nginx2-5d48dd4879-k8rzr container=0dfdabe83a1e k8s.ns=default k8s.pod=nginx2-5d48dd4879-k8rzr container=0dfdabe83a1e 04:21:26.844237768: Warning Sensitive file opened for reading by non-trusted program (user=root user_loginuid=-1 program=cat command=cat /etc/shadow file=/etc/shadow parent=bash gparent=<NA> ggparent=<NA> gggparent=<NA> container_id=0dfdabe83a1e image=nginx) k8s.ns=default k8s.pod=nginx2-5d48dd4879-k8rzr container=0dfdabe83a1e k8s.ns=default k8s.pod=nginx2-5d48dd4879-k8rzr container=0dfdabe83a1e
例2
Nginxのコンテナに入り、/binにディレクトリを作成する。
root@nginx2-5d48dd4879-k8rzr:/# cd /bin root@nginx2-5d48dd4879-k8rzr:/bin# mkdir hello
ログにアラートが出力される。
04:25:43.204565892: Error Directory below known binary directory created (user=root user_loginuid=-1 command=mkdir hello directory=/bin/hello container_id=0dfdabe83a1e image=nginx) k8s.ns=default k8s.pod=nginx2-5d48dd4879-k8rzr container=0dfdabe83a1e k8s.ns=default k8s.pod=nginx2-5d48dd4879-k8rzr container=0dfdabe83a1e
カスタムルールの作成
Helmチャートのデフォルト値を確認する。
$ helm inspect values falcosecurity/falco
custom_alerts.yamlを作成しカスタムルールを定義する。
customRules: rules-nginx.yaml: | - macro: nginx_consider_syscalls condition: (evt.num < 0) - macro: app_nginx condition: container and container.image contains "nginx" # Any outbound traffic raises a WARNING - rule: The program "whoami" is run in a container desc: An event will trigger every time you run "whoami" in a container condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = whoami output: "whoami command run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)" priority: NOTICE warn_evttypes: False - rule: The program "locate" is run in a container desc: An event will trigger every time you run "locate" in a container condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = locate output: "locate command run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)" priority: NOTICE warn_evttypes: False
リリースを更新する。
helm upgrade -i falco -f custom_alerts.yaml falcosecurity/falco
追加のルールは別のConfigMapになっている。
$ k get cm
NAME DATA AGE
falco 5 28m
falco-rules 1 10s
fluent-bit-config 2 29m
$ k get cm falco-rules -o yaml | k neat
apiVersion: v1
data:
rules-nginx.yaml: |-
- macro: nginx_consider_syscalls
condition: (evt.num < 0)
- macro: app_nginx
condition: container and container.image contains "nginx"
# Any outbound traffic raises a WARNING
- rule: The program "whoami" is run in a container
desc: An event will trigger every time you run "whoami" in a container
condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = whoami
output: "whoami command run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
priority: NOTICE
warn_evttypes: False
- rule: The program "locate" is run in a container
desc: An event will trigger every time you run "locate" in a container
condition: evt.type = execve and evt.dir=< and container.id != host and proc.name = locate
output: "locate command run in container (user=%user.name %container.info parent=%proc.pname cmdline=%proc.cmdline)"
priority: NOTICE
warn_evttypes: False
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: falco
meta.helm.sh/release-namespace: default
labels:
app: falco
app.kubernetes.io/managed-by: Helm
chart: falco-1.7.7
heritage: Helm
release: falco
name: falco-rules
namespace: default
Nginxのコンテナに入り、whoamiとfindコマンドを実行する。
Falcoのログを確認する。whoamiのログが確認できる。findは確認できない。そのようなルールは定義していないので、ブログの記載ミスと思われる。
04:41:08.686680090: Notice whoami command run in container (user=root k8s.ns=default k8s.pod=nginx2-5d48dd4879-k8rzr container=0dfdabe83a1e parent=bash cmdline=whoami)
CloudWatch Insights
CloudWatchで新しいダッシュボードを作成する。
ログテーブルのウィジットを2つ作成する。
fields @timestamp, @message | filter @message like 'Mkdir binary dirs' | sort @timestamp desc
fields @timestamp, @message | filter @message like 'Read sensitive file untrusted' | sort @timestamp desc
CloudWatch アラーム
falcoのロググループのIncommingLogEventsメトリクスに対してアラームを作成する。
これによってFalcoにログが追記されるとアラームが発報する。
なお、これだけだと起動時のログでも通知されるし、通知の中にログの中に内容が入っていないので、あまり実用的ではない。
Slack連携
Slack等への連携は、falcosecurity/falcosidekickが利用可能だが、Falco本体の機能でもOutputとしてプログラム実行ができ、SlackのIncoming Webhookを使った連携が可能。
チャートのvaluesに以下を追加する。
falco: jsonOutput: true programOutput: enabled: true keepAlive: false program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXXX"
リリースを更新する。
helm upgrade -i falco -f custom_alerts.yaml falcosecurity/falco
これでSlackに通知が飛ぶようになる。
