AppArmorに入門したメモ。
AppArmorが使えるLinuxディストリビューションではDockerではデフォルトで有効で、Docker用の制限されたプロファイルがある。Kubernetesではデフォルトでは有効ではなく、指定する必要がある。ランタイムのデフォルトを参照することは可能。
参考リンク
- Restrict a Container's Access to Resources with AppArmor | Kubernetes
- AppArmor security profiles for Docker | Docker Documentation
- https://github.com/killer-sh/cks-course-environment/tree/master/course-content/system-hardening/kernel-hardening-tools/apparmor
curl
curlが動作することを確認する。
root@cks-worker:~# curl -v killer.sh * Rebuilt URL to: killer.sh/ * Trying 157.245.26.192... * TCP_NODELAY set * Connected to killer.sh (157.245.26.192) port 80 (#0) > GET / HTTP/1.1 > Host: killer.sh > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < location: https://killer.sh/ < date: Mon, 04 Jan 2021 13:59:04 GMT < server: istio-envoy < content-length: 0 < * Connection #0 to host killer.sh left intact
必要なツールを入れる。
apt-get install -y apparmor-utils
curl用のプロファイルを作る。ログを分析していないので、まだ何も許可しない。
root@cks-worker:~# aa-genprof curl Writing updated profile for /usr/bin/curl. Setting /usr/bin/curl to complain mode. Before you begin, you may wish to check if a profile already exists for the application you wish to confine. See the following wiki page for more information: http://wiki.apparmor.net/index.php/Profiles Profiling: /usr/bin/curl Please start the application to be profiled in another window and exercise its functionality now. Once completed, select the "Scan" option below in order to scan the system logs for AppArmor events. For each AppArmor event, you will be given the opportunity to choose whether the access should be allowed or denied. [(S)can system log for AppArmor events] / (F)inish Setting /usr/bin/curl to enforce mode. Reloaded AppArmor profiles in enforce mode. Please consider contributing your new profile! See the following wiki page for more information: http://wiki.apparmor.net/index.php/Profiles Finished generating profile for /usr/bin/curl.
プロファイルが作成されたことを確認する。
apparmor module is loaded. 26 profiles are loaded. 21 profiles are in enforce mode. /sbin/dhclient /snap/snapd/10492/usr/lib/snapd/snap-confine /snap/snapd/10492/usr/lib/snapd/snap-confine//mount-namespace-capture-helper /usr/bin/curl /usr/bin/lxc-start /usr/bin/man /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/NetworkManager/nm-dhcp-helper /usr/lib/connman/scripts/dhclient-script /usr/lib/snapd/snap-confine /usr/lib/snapd/snap-confine//mount-namespace-capture-helper /usr/sbin/chronyd /usr/sbin/tcpdump docker-default lxc-container-default lxc-container-default-cgns lxc-container-default-with-mounting lxc-container-default-with-nesting man_filter man_groff snap-update-ns.google-cloud-sdk 5 profiles are in complain mode. snap.google-cloud-sdk.anthoscli snap.google-cloud-sdk.bq snap.google-cloud-sdk.docker-credential-gcloud snap.google-cloud-sdk.gcloud snap.google-cloud-sdk.gsutil 6 processes have profiles defined. 6 processes are in enforce mode. /usr/sbin/chronyd (1395) docker-default (3566) docker-default (3581) docker-default (4844) docker-default (4989) docker-default (5048) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined.
プロファイルは/etc/apparmor.dディレクトリに作成されており、以下のような内容。
root@cks-worker:~# cd /etc/apparmor.d/
root@cks-worker:/etc/apparmor.d# cat usr.bin.curl
# Last Modified: Mon Jan 4 14:02:20 2021
#include <tunables/global>
/usr/bin/curl {
#include <abstractions/base>
/lib/x86_64-linux-gnu/ld-*.so mr,
/usr/bin/curl mr,
}
curlを実行すると失敗することを確認する。これによってログがでる。
root@cks-worker:~# curl -v killer.sh * Rebuilt URL to: killer.sh/ * Could not resolve host: killer.sh * Closing connection 0 curl: (6) Could not resolve host: killer.sh
ログを分析してプロファイルを更新する。
root@cks-worker:/etc/apparmor.d# aa-logprof Reading log entries from /var/log/syslog. Updating AppArmor profiles in /etc/apparmor.d. Enforce-mode changes: Profile: /usr/bin/curl Path: /etc/ssl/openssl.cnf New Mode: owner r Severity: 2 [1 - #include <abstractions/lxc/container-base>] 2 - #include <abstractions/lxc/start-container> 3 - #include <abstractions/openssl> 4 - #include <abstractions/ssl_keys> 5 - owner /etc/ssl/openssl.cnf r, (A)llow / [(D)eny] / (I)gnore / (G)lob / Glob with (E)xtension / (N)ew / Audi(t) / (O)wner permissions off / Abo(r)t / (F)inish Adding #include <abstractions/lxc/container-base> to profile. Deleted 2 previous matching profile entries. = Changed Local Profiles = The following local profiles were changed. Would you like to save them? [1 - /usr/bin/curl] (S)ave Changes / Save Selec(t)ed Profile / [(V)iew Changes] / View Changes b/w (C)lean profiles / Abo(r)t Writing updated profile for /usr/bin/curl.
プロファイルの内容が変わったことを確認する。
root@cks-worker:/etc/apparmor.d# cat usr.bin.curl
# Last Modified: Mon Jan 4 14:05:21 2021
#include <tunables/global>
/usr/bin/curl {
#include <abstractions/base>
#include <abstractions/lxc/container-base>
}
curlが実行できるようになった。
root@cks-worker:/etc/apparmor.d# curl -v killer.sh * Rebuilt URL to: killer.sh/ * Trying 157.245.26.192... * TCP_NODELAY set * Connected to killer.sh (157.245.26.192) port 80 (#0) > GET / HTTP/1.1 > Host: killer.sh > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 301 Moved Permanently < location: https://killer.sh/ < date: Mon, 04 Jan 2021 14:06:22 GMT < server: istio-envoy < content-length: 0 < * Connection #0 to host killer.sh left intact
Docker
プロファイルのファイルを作成する。以下のファイルを/etc/apparmor.dに作成する。
#include <tunables/global>
profile docker-nginx flags=(attach_disconnected,mediate_deleted) {
#include <abstractions/base>
network inet tcp,
network inet udp,
network inet icmp,
deny network raw,
deny network packet,
file,
umount,
deny /bin/** wl,
deny /boot/** wl,
deny /dev/** wl,
deny /etc/** wl,
deny /home/** wl,
deny /lib/** wl,
deny /lib64/** wl,
deny /media/** wl,
deny /mnt/** wl,
deny /opt/** wl,
deny /proc/** wl,
deny /root/** wl,
deny /sbin/** wl,
deny /srv/** wl,
deny /tmp/** wl,
deny /sys/** wl,
deny /usr/** wl,
audit /** w,
/var/run/nginx.pid w,
/usr/sbin/nginx ix,
deny /bin/dash mrwklx,
deny /bin/sh mrwklx,
deny /usr/bin/top mrwklx,
capability chown,
capability dac_override,
capability setuid,
capability setgid,
capability net_bind_service,
deny @{PROC}/* w, # deny write for all files directly in /proc (not in a subdir)
# deny write to files not in /proc/<number>/** or /proc/sys/**
deny @{PROC}/{[^1-9],[^1-9][^0-9],[^1-9s][^0-9y][^0-9s],[^1-9][^0-9][^0-9][^0-9]*}/** w,
deny @{PROC}/sys/[^k]** w, # deny /proc/sys except /proc/sys/k* (effectively /proc/sys/kernel)
deny @{PROC}/sys/kernel/{?,??,[^s][^h][^m]**} w, # deny everything except shm* in /proc/sys/kernel/
deny @{PROC}/sysrq-trigger rwklx,
deny @{PROC}/mem rwklx,
deny @{PROC}/kmem rwklx,
deny @{PROC}/kcore rwklx,
deny mount,
deny /sys/[^f]*/** wklx,
deny /sys/f[^s]*/** wklx,
deny /sys/fs/[^c]*/** wklx,
deny /sys/fs/c[^g]*/** wklx,
deny /sys/fs/cg[^r]*/** wklx,
deny /sys/firmware/** rwklx,
deny /sys/kernel/security/** rwklx,
}
ファイルを指定してロードする。
apparmor_parser /etc/apparmor.d/docker-nginx
ロードされたことを確認する。
root@cks-worker:/etc/apparmor.d# aa-status apparmor module is loaded. 27 profiles are loaded. 22 profiles are in enforce mode. /sbin/dhclient /snap/snapd/10492/usr/lib/snapd/snap-confine /snap/snapd/10492/usr/lib/snapd/snap-confine//mount-namespace-capture-helper /usr/bin/curl /usr/bin/lxc-start /usr/bin/man /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/NetworkManager/nm-dhcp-helper /usr/lib/connman/scripts/dhclient-script /usr/lib/snapd/snap-confine /usr/lib/snapd/snap-confine//mount-namespace-capture-helper /usr/sbin/chronyd /usr/sbin/tcpdump docker-default docker-nginx lxc-container-default lxc-container-default-cgns lxc-container-default-with-mounting lxc-container-default-with-nesting man_filter man_groff snap-update-ns.google-cloud-sdk 5 profiles are in complain mode. snap.google-cloud-sdk.anthoscli snap.google-cloud-sdk.bq snap.google-cloud-sdk.docker-credential-gcloud snap.google-cloud-sdk.gcloud snap.google-cloud-sdk.gsutil 6 processes have profiles defined. 6 processes are in enforce mode. /usr/sbin/chronyd (1395) docker-default (3566) docker-default (3581) docker-default (4844) docker-default (4989) docker-default (5048) 0 processes are in complain mode. 0 processes are unconfined but have a profile defined.
Dockerでnginxを普通に実行する。
root@cks-worker:/etc/apparmor.d# docker run --rm nginx /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh /docker-entrypoint.sh: Configuration complete; ready for start up
Dockerでnginxをデフォルトのプロファイルを指定して実行する。
root@cks-worker:/etc/apparmor.d# docker run --rm --security-opt apparmor=docker-default nginx /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh 10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf 10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh /docker-entrypoint.sh: Configuration complete; ready for start up
Dockerでnginxをデフォルトの先ほど作成したプロファイルを指定して実行する。起動はするが、何かが拒否されている。
root@cks-worker:/etc/apparmor.d# docker run --rm --security-opt apparmor=docker-nginx nginx /docker-entrypoint.sh: No files found in /docker-entrypoint.d/, skipping configuration /docker-entrypoint.sh: 13: /docker-entrypoint.sh: cannot create /dev/null: Permission denied
別のターミナルでコンテナにログインして、アクセス制御がされていることを確認する。
root@cks-worker:~# docker exec -it 6535dec3d7e5 sh # touch /root/test touch: cannot touch '/root/test': Permission denied #
Kubernetes
プロファイルはアノテーションで指定する。
存在しないプロファイルを指定してPodを作成するとブロックされる。
root@cks-master:~# k apply -f secure.yaml
pod/secure created
root@cks-master:~# k get pod
NAME READY STATUS RESTARTS AGE
secure 0/1 Blocked 0 20s
root@cks-master:~# k describe pod secure
Name: secure
Namespace: default
Priority: 0
Node: cks-worker/10.146.0.7
Start Time: Mon, 04 Jan 2021 14:38:05 +0000
Labels: run=secure
Annotations: container.apparmor.security.beta.kubernetes.io/secure: localhost/hello
Status: Pending
Reason: AppArmor
Message: Cannot enforce AppArmor: profile "hello" is not loaded
IP:
IPs: <none>
Containers:
secure:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: Blocked
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-r2nz7 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-r2nz7:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-r2nz7
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 37s default-scheduler Successfully assigned default/secure to cks-worker
root@cks-master:~# k delete pod secure
pod "secure" deleted
存在するプロファイルを指定して試す。アクセス拒否がされていることを確認する。
root@cks-master:~# k apply -f secure.yaml pod/secure created root@cks-master:~# k get pod NAME READY STATUS RESTARTS AGE secure 0/1 ContainerCreating 0 3s root@cks-master:~# k get pod NAME READY STATUS RESTARTS AGE secure 1/1 Running 0 7s root@cks-master:~# k exec -it secure -- sh # touch /root/test touch: cannot touch '/root/test': Permission denied #
