ekscloudwatchを試す

EKSでFalcoを試す2の続きでekscloudwatchを試したメモ。

コンポーネント	バージョン
EKS	1.19
プラットフォームバージョン	eks.5
Falco	0.29.1
Falcoチャート	1.15.3
ekscloudwatch	ekscloudwatch-0.3

参考リンク

• https://github.com/sysdiglabs/ekscloudwatch
• https://faun.pub/analyze-aws-eks-audit-logs-with-falco-95202167f2e

クラスターの準備

1.19でクラスターを作成する。

cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: ekscloudwatch
  region: ap-northeast-1
  version: "1.19"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    privateNetworking: true

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true
EOF

eksctl create cluster -f cluster.yaml

Falcoのデプロイ

• https://github.com/falcosecurity/charts/tree/master/falco

初期化コンテナを使う方法でFalcoをインストールする。auditLog.enabled=trueとすることでServiceが作られる。

cat << EOF > values.yaml
image:
  repository: falcosecurity/falco-no-driver

auditLog:
  enabled: true

extraInitContainers:
  - name: driver-loader
    image: docker.io/falcosecurity/falco-driver-loader:0.29.1
    imagePullPolicy: Always
    securityContext:
      privileged: true
    volumeMounts:
      - mountPath: /host/proc
        name: proc-fs
        readOnly: true
      - mountPath: /host/boot
        name: boot-fs
        readOnly: true
      - mountPath: /host/lib/modules
        name: lib-modules
      - mountPath: /host/usr
        name: usr-fs
        readOnly: true
      - mountPath: /host/etc
        name: etc-fs
        readOnly: true

falcosidekick:
  enabled: true
  webui:
    enabled: true
EOF

$ helm upgrade --install falco falcosecurity/falco -n falco --create-namespace -f values.yaml
Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Wed Jul 14 03:23:33 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.


No further action should be required.

PodとServiceを確認する。

$ k -n falco get po
NAME                                      READY   STATUS    RESTARTS   AGE
falco-falcosidekick-5cbc97b7d9-887hg      1/1     Running   0          5m27s
falco-falcosidekick-5cbc97b7d9-rnbwn      1/1     Running   0          5m27s
falco-falcosidekick-ui-79c4d8b546-c9jcp   1/1     Running   0          5m27s
falco-nrffv                               1/1     Running   0          5m27s
falco-v8hh2                               1/1     Running   0          5m27s

$ k -n falco get svc
NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
falco                    ClusterIP   172.20.175.25    <none>        8765/TCP   5m31s
falco-falcosidekick      ClusterIP   172.20.61.171    <none>        2801/TCP   5m31s
falco-falcosidekick-ui   ClusterIP   172.20.179.147   <none>        2802/TCP   5m31s

ポートフォワードでfalcosidekick-uiにアクセスする。

k -n falco port-forward svc/falco-falcosidekick-ui 2802

• http://localhost:2802/ui

ekscloudwatchのデプロイの前提となるEKSのセットアップ

コントロールプレーンのロギングは有効化済み。

VPCエンドポイントを作り、クラスターセキュリティグループからのアクセスを許可するように説明があるが、インターネット経由でアクセスする場合は不要なはず。

CloudWatchReadOnlyAccessが必要なため、後ほどIRSAで付与する。

ekscloudwatchのデプロイ

リポジトリをクローンする。

git clone https://github.com/sysdiglabs/ekscloudwatch.git

クローンしたリポジトリのekscloudwatch-config.yamlをベースにfalco用にNamespaceとサービスの宛先をカスタマイズする。k8s_auditではなくk8s-auditなので注意。クラスター名とリージョンはオプションとあるが、取得できなかったので明示的に指定した。ポーリング間隔もテスト目的なので短くした。

cat << "EOF" > ekscloudwatch-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: ekscloudwatch-config
  namespace: falco
data:
  # Required: Endpoint to forward audit events to, such as Sysdig Secure agent
  # The agent must expose a k8s audit server (k8s_audit_server_port must be configured in the agent as well)
  endpoint: "http://falco:8765/k8s-audit"

  # Required: Cloudwatch polling interval
  cw_polling: "1m"

  # Required: CloudWatch query filter
  cw_filter: '{ $.sourceIPs[0] != "::1" && $.sourceIPs[0] != "127.0.0.1" }'

  # Optional: both the EKS cluster name and region must be set
  # This can be omitted if the EC2 instance can perform the ec2metadata and ec2:DescribeInstances action
  cluster_name: "ekscloudwatch"
  aws_region: "ap-northeast-1"
EOF

ConfiMapを作成する。

k apply -f ekscloudwatch-config.yaml

ServiceAccountを作成する。

k -n falco create sa eks-cloudwatch

IRSAで必要な権限を与える。

eksctl create iamserviceaccount \
    --name eks-cloudwatch \
    --namespace falco \
    --cluster ekscloudwatch \
    --attach-policy-arn arn:aws:iam::aws:policy/CloudWatchReadOnlyAccess \
    --override-existing-serviceaccounts \
    --approve

deployment.yamlもNamespaceを修正し、ServiceAccoutNameを指定する。

cat << EOF > ekscloudwatch-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eks-cloudwatch
  namespace: falco
spec:
  minReadySeconds: 5
  replicas: 1
  selector:
    matchLabels:
      app: eks-cloudwatch
  template:
    metadata:
      labels:
        app: eks-cloudwatch
    spec:
      serviceAccountName: eks-cloudwatch
      containers:
        - image: sysdiglabs/k8sauditlogforwarder:ekscloudwatch-0.3
          imagePullPolicy: Always
          name: eks-cloudwatch-container
          env:
            - name: ENDPOINT
              valueFrom:
                configMapKeyRef:
                  name: ekscloudwatch-config
                  key: endpoint
            - name: CLUSTER_NAME
              valueFrom:
                configMapKeyRef:
                  name: ekscloudwatch-config
                  key: cluster_name
            - name: AWS_REGION
              valueFrom:
                configMapKeyRef:
                  name: ekscloudwatch-config
                  key: aws_region
            - name: CW_POLLING
              valueFrom:
                configMapKeyRef:
                  name: ekscloudwatch-config
                  key: cw_polling
            - name: CW_FILTER
              valueFrom:
                configMapKeyRef:
                  name: ekscloudwatch-config
                  key: cw_filter
EOF

デプロイする。

k apply -f ekscloudwatch-deployment.yaml

Podが動いていることを確認する。

$ k -n falco get po
NAME                                      READY   STATUS    RESTARTS   AGE
eks-cloudwatch-66f84688d9-9rtjj           1/1     Running   0          13s
falco-falcosidekick-5cbc97b7d9-887hg      1/1     Running   0          16m
falco-falcosidekick-5cbc97b7d9-rnbwn      1/1     Running   0          16m
falco-falcosidekick-ui-79c4d8b546-c9jcp   1/1     Running   0          16m
falco-nrffv                               1/1     Running   0          16m
falco-v8hh2                               1/1     Running   0          16m

$ k -n falco logs eks-cloudwatch-66f84688d9-9rtjj
2021/07/13 18:39:35 Release 0.3
2021/07/13 18:39:35 Cloudwatch EKS log started
2021/07/13 18:39:38 386 logs sent to the agent (386 total)
2021/07/13 18:39:38 386 total logs

以下を参考にテストする。

• https://falco.org/docs/event-sources/kubernetes-audit/

テストのため、AWSアクセスキーを含むConfigMapを作成する。

cat << "EOF" > my-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
  namespace: default
data:
  ui.properties: |
    color.good=purple
    color.bad=yellow
    allow.textmode=true
  access.properties: |
    aws_access_key_id = MY-ID
    aws_secret_access_key = MY-KEY
EOF

k apply -f my-config.yaml

しばらくログを見ておく。

以下のログが出た。falcosidekickを有効にしたのでjson形式になっている。

{"output":"18:41:09.285203968: Notice K8s ConfigMap Created (user=kubernetes-admin configmap=my-config ns=default resp=201 decision=allow reason=)","priority":"Notice","rule":"K8s ConfigMap Created","time":"2021-07-13T18:41:09.285203968Z", "output_fields": {"jevt.time":"18:41:09.285203968","ka.auth.decision":"allow","ka.auth.reason":"","ka.response.code":"201","ka.target.name":"my-config","ka.target.namespace":"default","ka.user.name":"kubernetes-admin"}}

Falcoのドキュメントには以下のようにあったのでちょっと違う。

17:18:28.428398080: Warning K8s ConfigMap with private credential (user=minikube-user verb=create configmap=my-config config={"access.properties":"aws_access_key_id = MY-ID\naws_secret_access_key = MY-KEY\n","ui.properties":"color.good=purple\ncolor.bad=yellow\nallow.textmode=true\n"})

Logs Insightsでクエリする。

fields @timestamp, @message
| sort @timestamp desc
| filter @message like /my-config/
| limit 20

以下のようなイベントが記録されている。

{
    "kind": "Event",
    "apiVersion": "audit.k8s.io/v1",
    "level": "Metadata",
    "auditID": "3f90731e-1bd7-449d-bd0a-158d5e702ec4",
    "stage": "ResponseComplete",
    "requestURI": "/api/v1/namespaces/default/configmaps?fieldManager=kubectl-client-side-apply",
    "verb": "create",
    "user": {
        "username": "kubernetes-admin",
        "uid": "heptio-authenticator-aws:190189382900:AIDASYSBLVT2NYMBDBBGS",
        "groups": [
            "system:masters",
            "system:authenticated"
        ],
        "extra": {
            "accessKeyId": [
                "AKIASYSBLVT2CGD36GGA"
            ]
        }
    },
    "sourceIPs": [
        "27.0.3.145"
    ],
    "userAgent": "kubectl/v1.21.2 (darwin/amd64) kubernetes/092fbfb",
    "objectRef": {
        "resource": "configmaps",
        "namespace": "default",
        "name": "my-config",
        "apiVersion": "v1"
    },
    "responseStatus": {
        "metadata": {},
        "code": 201
    },
    "requestReceivedTimestamp": "2021-07-13T18:41:09.271404Z",
    "stageTimestamp": "2021-07-13T18:41:09.285204Z",
    "annotations": {
        "authorization.k8s.io/decision": "allow",
        "authorization.k8s.io/reason": ""
    }
}

FalcoのドキュメントではRequestResponseのレベルのイベントを前提としているが、EKSの場合Metadataレベルでしかとっていないためと思われる。

EKSではaudit-log policyはカスタマイズ不可。

• https://github.com/aws/containers-roadmap/issues/566

EKSベスプラによると、aws-authの編集はRequestResponceでとるようになっている。

• https://aws.github.io/aws-eks-best-practices/security/docs/detective.html

ベースのポリシーの参照先がずれていてわからないが、今だとたぶんここ。

• https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/gci/configure-helper.sh#L1116-L1241

他に以下のようなログも出ているがこれはよくわからない。

{"output":"18:39:28.992314112: Notice K8s Serviceaccount Created (user=system:node:ip-10-0-104-136.ap-northeast-1.compute.internal user=eks-cloudwatch ns=falco resp=201 decision=allow reason=)","priority":"Notice","rule":"K8s Serviceaccount Created","time":"2021-07-13T18:39:28.992314112Z", "output_fields": {"jevt.time":"18:39:28.992314112","ka.auth.decision":"allow","ka.auth.reason":"","ka.response.code":"201","ka.target.name":"eks-cloudwatch","ka.target.namespace":"falco","ka.user.name":"system:node:ip-10-0-104-136.ap-northeast-1.compute.internal"}}
{"output":"18:41:01.996801024: Notice K8s Serviceaccount Created (user=system:kube-controller-manager user=generic-garbage-collector ns=kube-system resp=201 decision=allow reason=RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\")","priority":"Notice","rule":"K8s Serviceaccount Created","time":"2021-07-13T18:41:01.996801024Z", "output_fields": {"jevt.time":"18:41:01.996801024","ka.auth.decision":"allow","ka.auth.reason":"RBAC: allowed by ClusterRoleBinding \"system:kube-controller-manager\" of ClusterRole \"system:kube-controller-manager\" to User \"system:kube-controller-manager\"","ka.response.code":"201","ka.target.name":"generic-garbage-collector","ka.target.namespace":"kube-system","ka.user.name":"system:kube-controller-manager"}}