HelmでのFalco環境のセットアップをクラスターのデプロイからやり直し、Falcoルールをカスタマイズして実用に耐えそうな環境を目指すメモ。
| コンポーネント |
バージョン |
| EKS |
1.20 |
| プラットフォームバージョン |
eks.2 |
| Falco |
0.29.1 |
| Falcoチャート |
1.15.4 |
1.20でクラスターを作成する。ノードなしで作成する。
cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: falco
region: ap-northeast-1
version: "1.20"
vpc:
cidr: "10.0.0.0/16"
availabilityZones:
- ap-northeast-1a
- ap-northeast-1c
cloudWatch:
clusterLogging:
enableTypes: ["*"]
iam:
withOIDC: true
EOF
eksctl create cluster -f cluster.yaml
ノードを作成する。
cat << "EOF" > managed-ng-1.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: falco
region: ap-northeast-1
managedNodeGroups:
- name: managed-ng-1
minSize: 2
maxSize: 2
desiredCapacity: 2
privateNetworking: true
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
preBootstrapCommands:
- |
set -o errexit
set -o pipefail
set -o nounset
yum -y install kernel-devel-$(uname -r)
EOF
eksctl create nodegroup -f managed-ng-1.yaml
Datadogエージェントを入れる（オプション）
helm repo add datadog https://helm.datadoghq.com
helm repo update
APIキーを渡してインストールする。
helm upgrade --install datadog datadog/datadog \
-n datadog --create-namespace \
--set datadog.apiKey=hogehoge
Falcoのデプロイ
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
チャートを確認する。
$ helm search repo falco
NAME CHART VERSION APP VERSION DESCRIPTION
falcosecurity/falco 1.15.4 0.29.1 Falco
falcosecurity/falco-exporter 0.5.2 0.5.0 Prometheus Metrics Exporter for Falco output ev...
falcosecurity/falcosidekick 0.3.11 2.23.1 A simple daemon to help you with falco's outputs
stable/falco 1.1.8 0.0.1 DEPRECATED - incubator/falco
valuesファイルを作成する。初期化コンテナを使う方法でFalcoをインストールする。auditLog.enabled=trueとすることでServiceが作られる。falcosidekickとの連携のための設定を入れるが、一緒には入れずに別に入れる。
cat << EOF > falco-values.yaml
image:
repository: falcosecurity/falco-no-driver
auditLog:
enabled: true
falco:
jsonOutput: true
jsonIncludeOutputProperty: true
httpOutput:
enabled: true
url: "http://falcosidekick:2801/"
extraInitContainers:
- name: driver-loader
image: docker.io/falcosecurity/falco-driver-loader:0.29.1
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- mountPath: /host/proc
name: proc-fs
readOnly: true
- mountPath: /host/boot
name: boot-fs
readOnly: true
- mountPath: /host/lib/modules
name: lib-modules
- mountPath: /host/usr
name: usr-fs
readOnly: true
- mountPath: /host/etc
name: etc-fs
readOnly: true
EOF
インストールする。
helm upgrade --install falco falcosecurity/falco \
-n falco --create-namespace \
-f falco-values.yaml
falcosidekickの導入
valuesファイルを作成する。
cat << EOF > falcosidekick-values.yaml
config:
slack:
webhookurl: "https://hooks.slack.com/hogehoge"
datadog:
apikey: "hogehoge"
aws:
sns:
topicarn: "arn:aws:sns:ap-northeast-1:XXXXXXXXXXXX:Falco_Alarms_Topic"
webui:
enabled: true
EOF
インストールする。
helm upgrade --install falcosidekick falcosecurity/falcosidekick \
-n falco --create-namespace \
-f falcosidekick-values.yaml
IRSAで権限を与えるためのポリシーを作成する。
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
AWS_REGION=$(aws configure get region)
cat <<EOF >eks-falcosidekick-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sns:Publish",
"Resource": "arn:aws:sns:${AWS_REGION}:${ACCOUNT_ID}:Falco_Alarms_Topic"
}
]
}
EOF
aws iam create-policy \
--policy-name eks-falcosidekick-policy \
--policy-document file://eks-falcosidekick-policy.json
IRSAでロールをアタッチする。
eksctl create iamserviceaccount \
--name falcosidekick \
--namespace falco \
--cluster falco \
--attach-policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/eks-falcosidekick-policy \
--override-existing-serviceaccounts \
--approve
ポートフォワードでfalcosidekick-uiにアクセスする。
k -n falco port-forward svc/falcosidekick-ui 2802
ekscloudwatchの導入
ConfigMapを作成する。
cat << "EOF" > ekscloudwatch-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: ekscloudwatch-config
namespace: falco
data:
endpoint: "http://falco:8765/k8s-audit"
cw_polling: "1m"
cw_filter: '{ $.sourceIPs[0] != "::1" && $.sourceIPs[0] != "127.0.0.1" }'
cluster_name: "falco"
aws_region: "ap-northeast-1"
EOF
ConfiMapを作成する。
k apply -f ekscloudwatch-config.yaml
CLUSTER_NAME="falco"
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
AWS_REGION=$(aws configure get region)
cat <<EOF >eks-ekscloudwatch-policy.json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:DescribeQueries",
"logs:DescribeExportTasks",
"logs:GetLogRecord",
"logs:GetQueryResults",
"logs:StopQuery",
"logs:TestMetricFilter",
"logs:DescribeQueryDefinitions",
"logs:DescribeResourcePolicies",
"logs:GetLogDelivery",
"logs:DescribeDestinations",
"logs:ListLogDeliveries"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:Describe*",
"logs:List*",
"logs:StartQuery",
"logs:FilterLogEvents",
"logs:Get*"
],
"Resource": [
"arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:log-stream:*",
"arn:aws:logs:${AWS_REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster"
]
}
]
}
EOF
aws iam create-policy \
--policy-name eks-ekscloudwatch-policy \
--policy-document file://eks-ekscloudwatch-policy.json
IRSAで必要な権限を与える。
eksctl create iamserviceaccount \
--name eks-cloudwatch \
--namespace falco \
--cluster falco \
--attach-policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/eks-ekscloudwatch-policy \
--override-existing-serviceaccounts \
--approve
Deploymentを作成する。
cat << EOF > ekscloudwatch-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: eks-cloudwatch
namespace: falco
spec:
minReadySeconds: 5
replicas: 1
selector:
matchLabels:
app: eks-cloudwatch
template:
metadata:
labels:
app: eks-cloudwatch
spec:
serviceAccountName: eks-cloudwatch
containers:
- image: sysdiglabs/k8sauditlogforwarder:ekscloudwatch-0.3
imagePullPolicy: Always
name: eks-cloudwatch-container
env:
- name: ENDPOINT
valueFrom:
configMapKeyRef:
name: ekscloudwatch-config
key: endpoint
- name: CLUSTER_NAME
valueFrom:
configMapKeyRef:
name: ekscloudwatch-config
key: cluster_name
- name: AWS_REGION
valueFrom:
configMapKeyRef:
name: ekscloudwatch-config
key: aws_region
- name: CW_POLLING
valueFrom:
configMapKeyRef:
name: ekscloudwatch-config
key: cw_polling
- name: CW_FILTER
valueFrom:
configMapKeyRef:
name: ekscloudwatch-config
key: cw_filter
EOF
デプロイする。
k apply -f ekscloudwatch-deployment.yaml
Podが動いていることを確認する。
$ k -n falco get po
NAME READY STATUS RESTARTS AGE
eks-cloudwatch-66f84688d9-q8nb2 1/1 Running 0 50s
falco-ckpfx 1/1 Running 0 32m
falco-sbfm8 1/1 Running 0 31m
falcosidekick-96876945c-cftkw 1/1 Running 0 26m
falcosidekick-96876945c-k5mtd 1/1 Running 0 26m
falcosidekick-ui-db9fdf886-j2xrq 1/1 Running 0 26m
EKS 1.20だからか、この通知がたくさん出る。
このリストを上書きするカスタムルールを追加する必要がある。
1.20から追加されたEKS独自のユーザーはここに記載がある。以下を追加すれば出まくることは防げそうだったが、念のため全部追加する。
- "eks:certificate-controller"
- "eks:k8s-metrics"
- "eks:cluster-event-watcher"
- "eks:pod-identity-mutating-webhook"
cat << EOF > falco-values.yaml
image:
repository: falcosecurity/falco-no-driver
auditLog:
enabled: true
falco:
jsonOutput: true
jsonIncludeOutputProperty: true
httpOutput:
enabled: true
url: "http://falcosidekick:2801/"
extraInitContainers:
- name: driver-loader
image: docker.io/falcosecurity/falco-driver-loader:0.29.1
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- mountPath: /host/proc
name: proc-fs
readOnly: true
- mountPath: /host/boot
name: boot-fs
readOnly: true
- mountPath: /host/lib/modules
name: lib-modules
- mountPath: /host/usr
name: usr-fs
readOnly: true
- mountPath: /host/etc
name: etc-fs
readOnly: true
customRules:
k8s_audit_rules.local.yaml: |-
- list: allowed_k8s_users
items: [
"minikube", "minikube-user", "kubelet", "kops", "admin", "kube", "kube-proxy", "kube-apiserver-healthcheck",
"kubernetes-admin",
vertical_pod_autoscaler_users,
cluster-autoscaler,
"system:addon-manager",
"cloud-controller-manager",
"eks:node-manager",
"system:kube-controller-manager",
"eks:authenticator",
"eks:certificate-controller",
"eks:cluster-event-watcher",
"eks:fargate-scheduler",
"eks:k8s-metrics",
"eks:nodewatcher",
"eks:pod-identity-mutating-webhook",
"eks:cluster-bootstrap"
]
EOF
アップデートする。
helm upgrade --install falco falcosecurity/falco \
-n falco --create-namespace \
-f falco-values.yaml
これである程度収まったが、K8s Serviceaccount Createdが繰り返し出る。
ルールのカスタマイズ
通知がいろいろ出るので、ホワイトリストへの追加などを行い、通知を抑制する。またEKS Best PracticeにLogs Insightsのサンプルクエリとして載っている以下の内容をFalcoルールとして実装してみる。
- Lists updates to the aws-auth ConfigMap
- Lists creation of new or changes to validation webhooks
- Lists create, update, delete operations to Roles
- Lists create, update, delete operations to RoleBindings
- Lists create, update, delete operations to ClusterRoles
- Lists create, update, delete operations to ClusterRoleBindings
- Plots unauthorized read operations against Secrets
- List of failed anonymous requests
こんな感じになった。
cat << EOF > falco-values.yaml
image:
repository: falcosecurity/falco-no-driver
auditLog:
enabled: true
falco:
jsonOutput: true
jsonIncludeOutputProperty: true
httpOutput:
enabled: true
url: "http://falcosidekick:2801/"
extraInitContainers:
- name: driver-loader
image: docker.io/falcosecurity/falco-driver-loader:0.29.1
imagePullPolicy: Always
securityContext:
privileged: true
volumeMounts:
- mountPath: /host/proc
name: proc-fs
readOnly: true
- mountPath: /host/boot
name: boot-fs
readOnly: true
- mountPath: /host/lib/modules
name: lib-modules
- mountPath: /host/usr
name: usr-fs
readOnly: true
- mountPath: /host/etc
name: etc-fs
readOnly: true
customRules:
falco_rules.local.yaml: |-
- list: falco_privileged_images
items: [
docker.io/calico/node,
calico/node,
docker.io/cloudnativelabs/kube-router,
docker.io/docker/ucp-agent,
docker.io/falcosecurity/falco,
docker.io/mesosphere/mesos-slave,
docker.io/rook/toolbox,
docker.io/sysdig/sysdig,
falcosecurity/falco,
gcr.io/google_containers/kube-proxy,
gcr.io/google-containers/startup-script,
gcr.io/projectcalico-org/node,
gke.gcr.io/kube-proxy,
gke.gcr.io/gke-metadata-server,
gke.gcr.io/netd-amd64,
gcr.io/google-containers/prometheus-to-sd,
k8s.gcr.io/ip-masq-agent-amd64,
k8s.gcr.io/kube-proxy,
k8s.gcr.io/prometheus-to-sd,
quay.io/calico/node,
sysdig/sysdig,
sematext_images,
docker.io/falcosecurity/falco-no-driver,
falcosecurity/falco-no-driver,
602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy,
602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni-init
]
- list: falco_sensitive_mount_images
items: [
docker.io/sysdig/sysdig, sysdig/sysdig,
docker.io/falcosecurity/falco, falcosecurity/falco,
gcr.io/google_containers/hyperkube,
gcr.io/google_containers/kube-proxy, docker.io/calico/node,
docker.io/rook/toolbox, docker.io/cloudnativelabs/kube-router, docker.io/consul,
docker.io/datadog/docker-dd-agent, docker.io/datadog/agent, docker.io/docker/ucp-agent, docker.io/gliderlabs/logspout,
docker.io/netdata/netdata, docker.io/google/cadvisor, docker.io/prom/node-exporter,
amazon/amazon-ecs-agent, prom/node-exporter, amazon/cloudwatch-agent,
docker.io/falcosecurity/falco-no-driver, falcosecurity/falco-no-driver,
gcr.io/datadoghq/agent
]
- list: falco_hostnetwork_images
items: [
gcr.io/google-containers/prometheus-to-sd,
gcr.io/projectcalico-org/typha,
gcr.io/projectcalico-org/node,
gke.gcr.io/gke-metadata-server,
gke.gcr.io/kube-proxy,
gke.gcr.io/netd-amd64,
k8s.gcr.io/ip-masq-agent-amd64
k8s.gcr.io/prometheus-to-sd,
602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni,
602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy
]
k8s_audit_rules.local.yaml: |-
- list: allowed_k8s_users
items: [
"minikube", "minikube-user", "kubelet", "kops", "admin", "kube", "kube-proxy", "kube-apiserver-healthcheck",
"kubernetes-admin",
vertical_pod_autoscaler_users,
cluster-autoscaler,
"system:addon-manager",
"cloud-controller-manager",
"eks:node-manager",
"system:kube-controller-manager",
"eks:authenticator",
"eks:certificate-controller",
"eks:cluster-event-watcher",
"eks:fargate-scheduler",
"eks:k8s-metrics",
"eks:nodewatcher",
"eks:pod-identity-mutating-webhook",
"eks:cluster-bootstrap"
]
- rule: K8s Serviceaccount Created
desc: Detect any attempt to create a service account
condition: (kactivity and kcreate and serviceaccount and response_successful and non_system_user)
output: K8s Serviceaccount Created (user=%ka.user.name user=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: INFO
source: k8s_audit
tags: [k8s]
- list: user_allowed_kube_namespace_image_list
items: [
602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/amazon-k8s-cni,
602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/kube-proxy,
602401143452.dkr.ecr.ap-northeast-1.amazonaws.com/eks/coredns,
user_trusted_image_list
]
- rule: K8s aws-auth ConfigMap Edited
desc: Detect any attempt to edit the aws-auth configmap.
condition: (kactivity and kmodify and configmap and ka.target.namespace=kube-system and ka.target.name=aws-auth and response_successful)
output: K8s aws-auth ConfigMap Edited (sourceip=%jevt.value[/sourceIPs/0] user=%ka.user.name verb=%ka.verb configmap=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
tags: [k8s]
- macro: validatingwebhookconfiguration
condition: ka.target.resource=validatingwebhookconfigurations
- rule: K8s ValidatingWebhookConfiguration Created or Edited.
desc: Detect any attempt to create or edit a validatingwebhookconfiguration.
condition: (kactivity and (kcreate or kmodify) and validatingwebhookconfiguration and response_successful)
output: K8s ValidatingWebhookConfiguration Created or Edited (user=%ka.user.name verb=%ka.verb validatingwebhookconfiguration=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
tags: [k8s]
- rule: K8s Role/Clusterrole Created
desc: Detect any attempt to create a cluster role/role
enabled: false
condition: (kactivity and kcreate and (clusterrole or role) and response_successful)
output: K8s Cluster Role Created (user=%ka.user.name role=%ka.target.name rules=%ka.req.role.rules resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: INFO
source: k8s_audit
tags: [k8s]
- rule: K8s Role/Clusterrole Deleted
desc: Detect any attempt to delete a cluster role/role
enabled: false
condition: (kactivity and kdelete and (clusterrole or role) and response_successful)
output: K8s Cluster Role Deleted (user=%ka.user.name role=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: INFO
source: k8s_audit
tags: [k8s]
- rule: K8s Role/Clusterrolebinding Created
desc: Detect any attempt to create a clusterrolebinding
enabled: false
condition: (kactivity and kcreate and clusterrolebinding and response_successful)
output: K8s Cluster Role Binding Created (user=%ka.user.name binding=%ka.target.name subjects=%ka.req.binding.subjects role=%ka.req.binding.role resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: INFO
source: k8s_audit
tags: [k8s]
- rule: K8s Role/Clusterrolebinding Deleted
desc: Detect any attempt to delete a clusterrolebinding
enabled: false
condition: (kactivity and kdelete and clusterrolebinding and response_successful)
output: K8s Cluster Role Binding Deleted (user=%ka.user.name binding=%ka.target.name resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: INFO
source: k8s_audit
tags: [k8s]
- rule: K8s Role Created, Edited or Deleted
desc: Detect any attempt to create, edit or delete a role.
condition: (kactivity and (kcreate or kmodify or kdelete) and role and response_successful)
output: K8s Role Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb role=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
tags: [k8s]
- macro: rolebinding
condition: ka.target.resource=rolebindings
- rule: K8s RoleBinding Create, Edited or Deleted
desc: Detect any attempt to create, edit or delete a rolebinding.
condition: (kactivity and (kcreate or kmodify or kdelete) and rolebinding and response_successful)
output: K8s RoleBinding Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb rolebinding=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
tags: [k8s]
- rule: K8s ClusterRole Created, Edited or Deleted
desc: Detect any attempt to create, edit or delete a clusterrole.
condition: (kactivity and (kcreate or kmodify or kdelete) and clusterrole and response_successful)
output: K8s ClusterRole Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb clusterrole=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
- rule: K8s ClusterRoleBinding Create, Edited or Deleted
desc: Detect any attempt to create, edit or delete a clusterrolebinding.
condition: (kactivity and (kcreate or kmodify or kdelete) and clusterrolebinding and response_successful)
output: K8s ClusterRoleBinding Created, Edited or Deleted (user=%ka.user.name verb=%ka.verb clusterrolebinding=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
tags: [k8s]
- macro: kread
condition: (ka.verb in (get,watch,list))
- rule: K8s Unauthorized read operations against Secret
desc: Detect unauthorized read operations against secret.
condition: (kactivity and kread and secret and not response_successful)
output: K8s Unauthorized read operations against Secret (user=%ka.user.name secret=%ka.target.name ns=%ka.target.namespace resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
tags: [k8s]
- macro: anonymous
condition: ka.user.name=system:anonymous
- rule: K8s Failed anonymous requests
desc: Detect failed anonymous requests.
enabled: false
condition: (kactivity and anonymous and not response_successful)
output: K8s Failed anonymous requests (user=%ka.user.name verb=%ka.verb uri=%ka.uri sourceip=%jevt.value[/sourceIPs/0] resp=%ka.response.code decision=%ka.auth.decision reason=%ka.auth.reason)
priority: ERROR
source: k8s_audit
tags: [k8s]
EOF
アップデートする。
helm upgrade --install falco falcosecurity/falco \
-n falco --create-namespace \
-f falco-values.yaml
Falcoのルールは複数のルールがマッチする場合に複数のアラートが出るわけではなく、1回しかアラートが出ないようである。したがって、カスタムしたルールを入れていても、デフォルトのルールにマッチしてしまうとそちらの方でアラートが出てしまう。デフォルトのルールのほうのPriorityが低かったりする場合は注意が必要そう。
