HelmでのFalco環境のセットアップをクラスターのデプロイからやり直し、Falcoルールをカスタマイズして実用に耐えそうな環境を目指すメモ。
コンポーネント | バージョン |
---|---|
EKS | 1.20 |
プラットフォームバージョン | eks.2 |
Falco | 0.29.1 |
Falcoチャート | 1.15.4 |
クラスターの作成
1.20でクラスターを作成する。ノードなしで作成する。
cat << EOF > cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: falco region: ap-northeast-1 version: "1.20" vpc: cidr: "10.0.0.0/16" availabilityZones: - ap-northeast-1a - ap-northeast-1c cloudWatch: clusterLogging: enableTypes: ["*"] iam: withOIDC: true EOF
eksctl create cluster -f cluster.yaml
ノードを作成する。
cat << "EOF" > managed-ng-1.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: falco region: ap-northeast-1 managedNodeGroups: - name: managed-ng-1 minSize: 2 maxSize: 2 desiredCapacity: 2 privateNetworking: true iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore preBootstrapCommands: - | #!/bin/bash set -o errexit set -o pipefail set -o nounset yum -y install kernel-devel-$(uname -r) EOF
eksctl create nodegroup -f managed-ng-1.yaml
Datadogエージェントを入れる(オプション)
helm repo add datadog https://helm.datadoghq.com helm repo update
APIキーを渡してインストールする。
helm upgrade --install datadog datadog/datadog \ -n datadog --create-namespace \ --set datadog.apiKey=hogehoge
Falcoのデプロイ
helm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update
チャートを確認する。
$ helm search repo falco NAME CHART VERSION APP VERSION DESCRIPTION falcosecurity/falco 1.15.4 0.29.1 Falco falcosecurity/falco-exporter 0.5.2 0.5.0 Prometheus Metrics Exporter for Falco output ev... falcosecurity/falcosidekick 0.3.11 2.23.1 A simple daemon to help you with falco's outputs stable/falco 1.1.8 0.0.1 DEPRECATED - incubator/falco
valuesファイルを作成する。初期化コンテナを使う方法でFalcoをインストールする。auditLog.enabled=true
とすることでServiceが作られる。falcosidekickとの連携のための設定を入れるが、一緒には入れずに別に入れる。
cat << EOF > falco-values.yaml image: repository: falcosecurity/falco-no-driver auditLog: enabled: true falco: jsonOutput: true jsonIncludeOutputProperty: true httpOutput: enabled: true url: "http://falcosidekick:2801/" extraInitContainers: - name: driver-loader image: docker.io/falcosecurity/falco-driver-loader:0.29.1 imagePullPolicy: Always securityContext: privileged: true volumeMounts: - mountPath: /host/proc name: proc-fs readOnly: true - mountPath: /host/boot name: boot-fs readOnly: true - mountPath: /host/lib/modules name: lib-modules - mountPath: /host/usr name: usr-fs readOnly: true - mountPath: /host/etc name: etc-fs readOnly: true EOF
インストールする。
helm upgrade --install falco falcosecurity/falco \ -n falco --create-namespace \ -f falco-values.yaml
falcosidekickの導入
valuesファイルを作成する。
cat << EOF > falcosidekick-values.yaml config: slack: webhookurl: "https://hooks.slack.com/hogehoge" datadog: apikey: "hogehoge" aws: sns: topicarn: "arn:aws:sns:ap-northeast-1:XXXXXXXXXXXX:Falco_Alarms_Topic" webui: enabled: true EOF
インストールする。
helm upgrade --install falcosidekick falcosecurity/falcosidekick \ -n falco --create-namespace \ -f falcosidekick-values.yaml
IRSAで権限を与えるためのポリシーを作成する。
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) AWS_REGION=$(aws configure get region) cat <<EOF >eks-falcosidekick-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "sns:Publish", "Resource": "arn:aws:sns:${AWS_REGION}:${ACCOUNT_ID}:Falco_Alarms_Topic" } ] } EOF aws iam create-policy \ --policy-name eks-falcosidekick-policy \ --policy-document file://eks-falcosidekick-policy.json
IRSAでロールをアタッチする。
eksctl create iamserviceaccount \ --name falcosidekick \ --namespace falco \ --cluster falco \ --attach-policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/eks-falcosidekick-policy \ --override-existing-serviceaccounts \ --approve
ポートフォワードでfalcosidekick-uiにアクセスする。
k -n falco port-forward svc/falcosidekick-ui 2802
ekscloudwatchの導入
ConfigMapを作成する。
cat << "EOF" > ekscloudwatch-config.yaml apiVersion: v1 kind: ConfigMap metadata: name: ekscloudwatch-config namespace: falco data: # Required: Endpoint to forward audit events to, such as Sysdig Secure agent # The agent must expose a k8s audit server (k8s_audit_server_port must be configured in the agent as well) endpoint: "http://falco:8765/k8s-audit" # Required: Cloudwatch polling interval cw_polling: "1m" # Required: CloudWatch query filter cw_filter: '{ $.sourceIPs[0] != "::1" && $.sourceIPs[0] != "127.0.0.1" }' # Optional: both the EKS cluster name and region must be set # This can be omitted if the EC2 instance can perform the ec2metadata and ec2:DescribeInstances action cluster_name: "falco" aws_region: "ap-northeast-1" EOF
ConfiMapを作成する。
k apply -f ekscloudwatch-config.yaml
CLUSTER_NAME="falco" ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) AWS_REGION=$(aws configure get region) cat <<EOF >eks-ekscloudwatch-policy.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:DescribeQueries", "logs:DescribeExportTasks", "logs:GetLogRecord", "logs:GetQueryResults", "logs:StopQuery", "logs:TestMetricFilter", "logs:DescribeQueryDefinitions", "logs:DescribeResourcePolicies", "logs:GetLogDelivery", "logs:DescribeDestinations", "logs:ListLogDeliveries" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "logs:Describe*", "logs:List*", "logs:StartQuery", "logs:FilterLogEvents", "logs:Get*" ], "Resource": [ "arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:log-stream:*", "arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster" ] } ] } EOF aws iam create-policy \ --policy-name eks-ekscloudwatch-policy \ --policy-document file://eks-ekscloudwatch-policy.json
IRSAで必要な権限を与える。
eksctl create iamserviceaccount \ --name eks-cloudwatch \ --namespace falco \ --cluster falco \ --attach-policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/eks-ekscloudwatch-policy \ --override-existing-serviceaccounts \ --approve
Deploymentを作成する。
cat << EOF > ekscloudwatch-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: eks-cloudwatch namespace: falco spec: minReadySeconds: 5 replicas: 1 selector: matchLabels: app: eks-cloudwatch template: metadata: labels: app: eks-cloudwatch spec: serviceAccountName: eks-cloudwatch containers: - image: sysdiglabs/k8sauditlogforwarder:ekscloudwatch-0.3 imagePullPolicy: Always name: eks-cloudwatch-container env: - name: ENDPOINT valueFrom: configMapKeyRef: name: ekscloudwatch-config key: endpoint - name: CLUSTER_NAME valueFrom: configMapKeyRef: name: ekscloudwatch-config key: cluster_name - name: AWS_REGION valueFrom: configMapKeyRef: name: ekscloudwatch-config key: aws_region - name: CW_POLLING valueFrom: configMapKeyRef: name: ekscloudwatch-config key: cw_polling - name: CW_FILTER valueFrom: configMapKeyRef: name: ekscloudwatch-config key: cw_filter EOF
デプロイする。
k apply -f ekscloudwatch-deployment.yaml
Podが動いていることを確認する。
$ k -n falco get po NAME READY STATUS RESTARTS AGE eks-cloudwatch-66f84688d9-q8nb2 1/1 Running 0 50s falco-ckpfx 1/1 Running 0 32m falco-sbfm8 1/1 Running 0 31m falcosidekick-96876945c-cftkw 1/1 Running 0 26m falcosidekick-96876945c-k5mtd 1/1 Running 0 26m falcosidekick-ui-db9fdf886-j2xrq 1/1 Running 0 26m
EKS 1.20だからか、この通知がたくさん出る。
このリストを上書きするカスタムルールを追加する必要がある。
1.20から追加されたEKS独自のユーザーはここに記載がある。以下を追加すれば出まくることは防げそうだったが、念のため全部追加する。
- "eks:certificate-controller"
- "eks:k8s-metrics"
- "eks:cluster-event-watcher"
- "eks:pod-identity-mutating-webhook"
cat << EOF > falco-values.yaml image: repository: falcosecurity/falco-no-driver auditLog: enabled: true falco: jsonOutput: true jsonIncludeOutputProperty: true httpOutput: enabled: true url: "http://falcosidekick:2801/" extraInitContainers: - name: driver-loader image: docker.io/falcosecurity/falco-driver-loader:0.29.1 imagePullPolicy: Always securityContext: privileged: true volumeMounts: - mountPath: /host/proc name: proc-fs readOnly: true - mountPath: /host/boot name: boot-fs readOnly: true - mountPath: /host/lib/modules name: lib-modules - mountPath: /host/usr name: usr-fs readOnly: true - mountPath: /host/etc name: etc-fs readOnly: true customRules: k8s_audit_rules.local.yaml: |- - list: allowed_k8s_users items: [ "minikube", "minikube-user", "kubelet", "kops", "admin", "kube", "kube-proxy", "kube-apiserver-healthcheck", "kubernetes-admin", vertical_pod_autoscaler_users, cluster-autoscaler, "system:addon-manager", "cloud-controller-manager", "eks:node-manager", "system:kube-controller-manager", "eks:authenticator", "eks:certificate-controller", "eks:cluster-event-watcher", "eks:fargate-scheduler", "eks:k8s-metrics", "eks:nodewatcher", "eks:pod-identity-mutating-webhook", "eks:cluster-bootstrap" ] EOF
アップデートする。
helm upgrade --install falco falcosecurity/falco \ -n falco --create-namespace \ -f falco-values.yaml
これである程度収まったが、K8s Serviceaccount Createdが繰り返し出る。
ルールのカスタマイズ
通知がいろいろ出るので、ホワイトリストへの追加などを行い、通知を抑制する。またEKS Best PracticeにLogs Insightsのサンプルクエリとして載っている以下の内容をFalcoルールとして実装してみる。
- Lists updates to the aws-auth ConfigMap
- Lists creation of new or changes to validation webhooks
- Lists create, update, delete operations to Roles
- Lists create, update, delete operations to RoleBindings
- Lists create, update, delete operations to ClusterRoles
- Lists create, update, delete operations to ClusterRoleBindings
- Plots unauthorized read operations against Secrets
- List of failed anonymous requests
こんな感じになった。
cat << EOF > falco-values.yaml image: repository: falcosecurity/falco-no-driver auditLog: enabled: true falco: jsonOutput: true jsonIncludeOutputProperty: true httpOutput: enabled: true url: "http://falcosidekick:2801/" extraInitContainers: - name: driver-loader image: docker.io/falcosecurity/falco-driver-loader:0.29.1 imagePullPolicy: Always securityContext: privileged: true volumeMounts: - mountPath: /host/proc name: proc-fs readOnly: true - mountPath: /host/boot name: boot-fs readOnly: true - mountPath: /host/lib/modules name: lib-modules - mountPath: /host/usr name: usr-fs readOnly: true - mountPath: /host/etc name: etc-fs readOnly: true customRules: falco_rules.local.yaml: |- - list: falco_privileged_images items: [ docker.io/calico/node, calico/node, docker.io/cloudnativelabs/kube-router, docker.io/docker/ucp-agent, docker.io/falcosecurity/falco, docker.io/mesosphere/mesos-slave, docker.io/rook/toolbox, docker.io/sysdig/sysdig, falcosecurity/falco, gcr.io/google_containers/kube-proxy, gcr.io/google-containers/startup-script, gcr.io/projectcalico-org/node, gke.gcr.io/kube-proxy, gke.gcr.io/gke-metadata-server, gke.gcr.io/netd-amd64, gcr.io/google-containers/prometheus-to-sd, k8s.gcr.io/ip-masq-agent-amd64, k8s.gcr.io/kube-proxy, k8s.gcr.io/prometheus-to-sd, quay.io/calico/node, sysdig/sysdig, sematext_images, docker.io/falcosecurity/falco-no-driver, falcosecurity/falco-no-driver, 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy, 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni-init ] - list: falco_sensitive_mount_images items: [ docker.io/sysdig/sysdig, sysdig/sysdig, docker.io/falcosecurity/falco, falcosecurity/falco, gcr.io/google_containers/hyperkube, gcr.io/google_containers/kube-proxy, docker.io/calico/node, docker.io/rook/toolbox, docker.io/cloudnativelabs/kube-router, docker.io/consul, docker.io/datadog/docker-dd-agent, docker.io/datadog/agent, docker.io/docker/ucp-agent, docker.io/gliderlabs/logspout, docker.io/netdata/netdata, docker.io/google/cadvisor, docker.io/prom/node-exporter, amazon/amazon-ecs-agent, prom/node-exporter, amazon/cloudwatch-agent, docker.io/falcosecurity/falco-no-driver, falcosecurity/falco-no-driver, gcr.io/datadoghq/agent ] - list: falco_hostnetwork_images items: [ gcr.io/google-containers/prometheus-to-sd, gcr.io/projectcalico-org/typha, gcr.io/projectcalico-org/node, gke.gcr.io/gke-metadata-server, gke.gcr.io/kube-proxy, gke.gcr.io/netd-amd64, k8s.gcr.io/ip-masq-agent-amd64 k8s.gcr.io/prometheus-to-sd, 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni, 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy ] k8s_audit_rules.local.yaml: |- - list: allowed_k8s_users items: [ "minikube", "minikube-user", "kubelet", "kops", "admin", "kube", "kube-proxy", "kube-apiserver-healthcheck", "kubernetes-admin", vertical_pod_autoscaler_users, cluster-autoscaler, "system:addon-manager", "cloud-controller-manager", "eks:node-manager", "system:kube-controller-manager", "eks:authenticator", "eks:certificate-controller", "eks:cluster-event-watcher", "eks:fargate-scheduler", "eks:k8s-metrics", "eks:nodewatcher", "eks:pod-identity-mutating-webhook", "eks:cluster-bootstrap" ] - rule: K8s Serviceaccount Created desc: Detect any attempt to create a service account condition: (kactivity and kcreate and serviceaccount and response_successful and non_system_user) output: K8s Serviceaccount Created (user=%ka.user.name user=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: INFO source: k8s_audit tags: [k8s] - list: user_allowed_kube_namespace_image_list items: [ 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni, 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy, 602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/coredns, user_trusted_image_list ] - rule: K8s aws-auth ConfigMap Edited desc: Detect any attempt to edit the aws-auth configmap. condition: (kactivity and kmodify and configmap and ka.target.namespace=kube-system and ka.target.name=aws-auth and response_successful) output: K8s aws-auth ConfigMap Edited (sourceip=%jevt.value[/sourceIPs/0] user=%ka.user.name verb=%ka.verb configmap=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit tags: [k8s] - macro: validatingwebhookconfiguration condition: ka.target.resource=validatingwebhookconfigurations - rule: K8s ValidatingWebhookConfiguration Created or Edited. desc: Detect any attempt to create or edit a validatingwebhookconfiguration. condition: (kactivity and (kcreate or kmodify) and validatingwebhookconfiguration and response_successful) output: K8s ValidatingWebhookConfiguration Created or Edited (user=%ka.user.name verb=%ka.verb validatingwebhookconfiguration=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit tags: [k8s] - rule: K8s Role/Clusterrole Created desc: Detect any attempt to create a cluster role/role enabled: false condition: (kactivity and kcreate and (clusterrole or role) and response_successful) output: K8s Cluster Role Created (user=%ka.user.name role=%ka.target.name rules=%ka.req.role.rules resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: INFO source: k8s_audit tags: [k8s] - rule: K8s Role/Clusterrole Deleted desc: Detect any attempt to delete a cluster role/role enabled: false condition: (kactivity and kdelete and (clusterrole or role) and response_successful) output: K8s Cluster Role Deleted (user=%ka.user.name role=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: INFO source: k8s_audit tags: [k8s] - rule: K8s Role/Clusterrolebinding Created desc: Detect any attempt to create a clusterrolebinding enabled: false condition: (kactivity and kcreate and clusterrolebinding and response_successful) output: K8s Cluster Role Binding Created (user=%ka.user.name binding=%ka.target.name subjects=%ka.req.binding.subjects role=%ka.req.binding.role resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: INFO source: k8s_audit tags: [k8s] - rule: K8s Role/Clusterrolebinding Deleted desc: Detect any attempt to delete a clusterrolebinding enabled: false condition: (kactivity and kdelete and clusterrolebinding and response_successful) output: K8s Cluster Role Binding Deleted (user=%ka.user.name binding=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: INFO source: k8s_audit tags: [k8s] - rule: K8s Role Created, Edited or Deleted desc: Detect any attempt to create, edit or delete a role. condition: (kactivity and (kcreate or kmodify or kdelete) and role and response_successful) output: K8s Role Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb role=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit tags: [k8s] - macro: rolebinding condition: ka.target.resource=rolebindings - rule: K8s RoleBinding Create, Edited or Deleted desc: Detect any attempt to create, edit or delete a rolebinding. condition: (kactivity and (kcreate or kmodify or kdelete) and rolebinding and response_successful) output: K8s RoleBinding Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb rolebinding=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit tags: [k8s] - rule: K8s ClusterRole Created, Edited or Deleted desc: Detect any attempt to create, edit or delete a clusterrole. condition: (kactivity and (kcreate or kmodify or kdelete) and clusterrole and response_successful) output: K8s ClusterRole Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb clusterrole=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit - rule: K8s ClusterRoleBinding Create, Edited or Deleted desc: Detect any attempt to create, edit or delete a clusterrolebinding. condition: (kactivity and (kcreate or kmodify or kdelete) and clusterrolebinding and response_successful) output: K8s ClusterRoleBinding Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb clusterrolebinding=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit tags: [k8s] - macro: kread condition: (ka.verb in (get,watch,list)) - rule: K8s Unauthorized read operations against Secret desc: Detect unauthorized read operations against secret. condition: (kactivity and kread and secret and not response_successful) output: K8s Unauthorized read operations against Secret (user=%ka.user.name secret=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit tags: [k8s] - macro: anonymous condition: ka.user.name=system:anonymous - rule: K8s Failed anonymous requests desc: Detect failed anonymous requests. enabled: false condition: (kactivity and anonymous and not response_successful) output: K8s Failed anonymous requests (user=%ka.user.name verb=%ka.verb uri=%ka.uri sourceip=%jevt.value[/sourceIPs/0] resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason) priority: ERROR source: k8s_audit tags: [k8s] EOF
アップデートする。
helm upgrade --install falco falcosecurity/falco \ -n falco --create-namespace \ -f falco-values.yaml
Falcoのルールは複数のルールがマッチする場合に複数のアラートが出るわけではなく、1回しかアラートが出ないようである。したがって、カスタムしたルールを入れていても、デフォルトのルールにマッチしてしまうとそちらの方でアラートが出てしまう。デフォルトのルールのほうのPriorityが低かったりする場合は注意が必要そう。