EKSでCiliumをネットワークポリシーエンジンとして使う方法を試してみる。CNIとしてCiliumを使う方法と、ネットワークポリシーエンジンとしてだけ使う方法があるが、後者。
- https://docs.cilium.io/en/v1.9/gettingstarted/k8s-install-eks/
- https://docs.cilium.io/en/v1.9/gettingstarted/cni-chaining-aws-cni/
| コンポーネント | バージョン | 備考 |
|---|---|---|
| eksctl | 0.36.2 | |
| Kubernetes バージョン | 1.18 | |
| プラットフォームのバージョン | eks.3 | |
| VPC CNI Plugin | 1.7.5 | |
| Cilium | 1.9.3 |
クラスターの作成
クラスターを作成する。
apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: cilium region: ap-northeast-1 vpc: cidr: "10.1.0.0/16" availabilityZones: - ap-northeast-1a - ap-northeast-1c managedNodeGroups: - name: managed-ng-1 minSize: 2 maxSize: 2 desiredCapacity: 2 ssh: allow: true publicKeyName: default privateNetworking: true cloudWatch: clusterLogging: enableTypes: ["*"]
eksctl create cluster -f cilium.yaml
Ciliumのインストール
Helmリポジトリを追加する。
helm repo add cilium https://helm.cilium.io/
HelmでCiliumをインストールする。
helm install cilium cilium/cilium --version 1.9.3 \ --namespace kube-system \ --set cni.chainingMode=aws-cni \ --set masquerade=false \ --set tunnel=disabled \ --set nodeinit.enabled=true
Ciliumがインストールされたことを確認する。
$ kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-p6k9m 1/1 Running 0 41h kube-system aws-node-qttwr 1/1 Running 0 41h kube-system cilium-dffdz 1/1 Running 0 4m46s kube-system cilium-node-init-f6snw 1/1 Running 0 4m46s kube-system cilium-node-init-f72cr 1/1 Running 0 4m46s kube-system cilium-operator-db487bc5b-2m86x 1/1 Running 0 4m46s kube-system cilium-operator-db487bc5b-8s4sm 1/1 Running 0 4m46s kube-system cilium-smm9f 1/1 Running 0 4m46s kube-system coredns-86f7d88d77-dl769 0/1 ContainerCreating 0 4m14s kube-system coredns-86f7d88d77-rt84l 0/1 ContainerCreating 0 3m44s kube-system kube-proxy-59krv 1/1 Running 0 41h kube-system kube-proxy-mh4dv 1/1 Running 0 41h
Ciliumは起動したが、CoreDNSが起動しなくなっている。Podをdescribeすると以下のようなWarningがでている。
Warning FailedCreatePodSandBox 3m2s (x4 over 3m5s) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "8bd40e1f66457a69ed99969214e248f4dbc97d499683ef612bbe30b7137a1ec0" network for pod "coredns-86f7d88d77-dl769": networkPlugin cni failed to set up pod "coredns-86f7d88d77-dl769_kube-system" network: netplugin failed but error parsing its diagnostic message "{\"level\":\"debug\",\"ts\":\"2021-01-29T02:41:14.743Z\",\"caller\":\"routed-eni-cni-plugin/cni.go:123\",\"msg\":\"MTU not set, defaulting to 9001\"}\n{\"level\":\"info\",\"ts\":\"2021-01-29T02:41:14.743Z\",\"caller\":\"routed-eni-cni-plugin/cni.go:117\",\"msg\":\"Received CNI add request: ContainerID(8bd40e1f66457a69ed99969214e248f4dbc97d499683ef612bbe30b7137a1ec0) Netns(/proc/32025/ns/net) IfName(eth0) Args(IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=coredns-86f7d88d77-dl769;K8S_POD_INFRA_CONTAINER_ID=8bd40e1f66457a69ed99969214e248f4dbc97d499683ef612bbe30b7137a1ec0) Path(/opt/cni/bin) argsStdinData({\\\"cniVersion\\\":\\\"0.3.1\\\",\\\"name\\\":\\\"aws-cni\\\",\\\"type\\\":\\\"aws-cni\\\",\\\"vethPrefix\\\":\\\"eni\\\"})\"}\n{\"level\":\"debug\",\"ts\":\"2021-01-29T02:41:14.744Z\",\"caller\":\"routed-eni-cni-plugin/cni.go:117\",\"msg\":\"MTU value set is 9001:\"}\n{\"level\":\"error\",\"ts\":\"2021-01-29T02:41:14.745Z\",\"caller\":\"routed-eni-cni-plugin/cni.go:117\",\"msg\":\"Failed to assign an IP address to container 8bd40e1f66457a69ed99969214e248f4dbc97d499683ef612bbe30b7137a1ec0\"}\n{\n \"code\": 100,\n \"msg\": \"add cmd: failed to assign an IP address to container\"\n}": invalid character '{' after top-level value
関連するかも知れないIssueがある。
- New pods failing to start with
FailedCreatePodSandBoxwarning for CNI versions 1.7.x with Cilium #1265 - Coredns stuck on ContainerCreating with
FailedCreatePodSandBoxwarning for CNI versions 1.7.6 with Cilium 1.9.1 #1314
VPC CNI PluginのバージョンとCiliumのバージョンを確認する。
$ kubectl describe daemonset aws-node --namespace kube-system | grep Image | cut -d "/" -f 2 amazon-k8s-cni-init:v1.7.5-eksbuild.1 amazon-k8s-cni:v1.7.5-eksbuild.1 $ helm ls -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION cilium kube-system 1 2021-01-29 11:40:28.449879 +0900 JST deployed cilium-1.9.3 1.9.3
Issueにあるように、手作業で編集してみる。
編集前はこのようになっている。
[root@ip-10-1-90-101 ~]# cat /etc/cni/net.d/10-aws.conflist
{
"cniVersion": "0.3.1",
"name": "aws-cni",
"plugins": [
{
"name": "aws-cni",
"type": "aws-cni",
"vethPrefix": "eni",
"mtu": "9001",
"pluginLogFile": "/var/log/aws-routed-eni/plugin.log",
"pluginLogLevel": "DEBUG"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"snat": true
}
]
}[root@ip-10-1-90-101 ~]# cat /etc/cni/net.d/05-cilium.conflist
{
"cniVersion": "0.3.1",
"name": "aws-cni",
"plugins": [
{
"name": "aws-cni",
"type": "aws-cni",
"vethPrefix": "eni"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"snat": true
},
{
"name": "cilium",
"type": "cilium-cni",
"enable-debug": false
}
]
}
[root@ip-10-1-90-101 ~]#
2つめのファイルのほうを編集する。
[root@ip-10-1-90-101 ~]# cat /etc/cni/net.d/05-cilium.conflist
{
"cniVersion": "0.3.1",
"name": "aws-cni",
"plugins": [
{
"name": "aws-cni",
"type": "aws-cni",
"vethPrefix": "eni",
"mtu": "9001",
"pluginLogFile": "/var/log/aws-routed-eni/plugin.log",
"pluginLogLevel": "Debug"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"snat": true
},
{
"name": "cilium",
"type": "cilium-cni",
"enable-debug": false
}
]
}
[root@ip-10-1-90-101 ~]#
CoreDNSが起動したことを確認する。
$ k get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-p6k9m 1/1 Running 0 41h kube-system aws-node-qttwr 1/1 Running 0 41h kube-system cilium-dffdz 1/1 Running 0 35m kube-system cilium-node-init-f6snw 1/1 Running 0 35m kube-system cilium-node-init-f72cr 1/1 Running 0 35m kube-system cilium-operator-db487bc5b-2m86x 1/1 Running 0 35m kube-system cilium-operator-db487bc5b-8s4sm 1/1 Running 0 35m kube-system cilium-smm9f 1/1 Running 0 35m kube-system coredns-86f7d88d77-dl769 1/1 Running 0 35m kube-system coredns-86f7d88d77-rt84l 1/1 Running 0 34m kube-system kube-proxy-59krv 1/1 Running 0 41h kube-system kube-proxy-mh4dv 1/1 Running 0 41h
Ciliumをインストールした後、Ciliumでポリシーを適用するためにはPodの再起動が必要であり、CoreDNSは自動で再起動されたようだ。
テスト
テストをデプロイする。
$ kubectl create ns cilium-test namespace/cilium-test created $ kubectl apply -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/v1.9/examples/kubernetes/connectivity-check/connectivity-check.yaml deployment.apps/echo-a created deployment.apps/echo-b created deployment.apps/echo-b-host created deployment.apps/pod-to-a created deployment.apps/pod-to-external-1111 created deployment.apps/pod-to-a-denied-cnp created deployment.apps/pod-to-a-allowed-cnp created deployment.apps/pod-to-external-fqdn-allow-google-cnp created deployment.apps/pod-to-b-multi-node-clusterip created deployment.apps/pod-to-b-multi-node-headless created deployment.apps/host-to-b-multi-node-clusterip created deployment.apps/host-to-b-multi-node-headless created deployment.apps/pod-to-b-multi-node-nodeport created deployment.apps/pod-to-b-intra-node-nodeport created service/echo-a created service/echo-b created service/echo-b-headless created service/echo-b-host-headless created ciliumnetworkpolicy.cilium.io/pod-to-a-denied-cnp created ciliumnetworkpolicy.cilium.io/pod-to-a-allowed-cnp created ciliumnetworkpolicy.cilium.io/pod-to-external-fqdn-allow-google-cnp created
Ciliumの管理下にあるPodはCiliumEndpoint(cep)リソースとして確認できる。
$ kubectl get cep -A NAMESPACE NAME ENDPOINT ID IDENTITY ID INGRESS ENFORCEMENT EGRESS ENFORCEMENT VISIBILITY POLICY ENDPOINT STATE IPV4 IPV6 cilium-test echo-a-57cbbd9b8b-mxrfs 3296 5791 ready 10.1.67.236 cilium-test echo-b-6db5fc8ff8-rkm9w 2571 1204 ready 10.1.96.204 cilium-test pod-to-a-648fd74787-4gxhr 3667 57208 ready 10.1.85.107 cilium-test pod-to-a-allowed-cnp-7776c879f-qltmn 3192 34348 ready 10.1.112.226 cilium-test pod-to-a-denied-cnp-b5ff897c7-kwz2m 919 45310 ready 10.1.76.211 cilium-test pod-to-b-intra-node-nodeport-6546644d59-6qfc7 1334 44803 ready 10.1.112.247 cilium-test pod-to-b-multi-node-clusterip-7d54c74c5f-7g72l 272 58159 ready 10.1.91.91 cilium-test pod-to-b-multi-node-headless-76db68d547-kkg7f 3158 55761 ready 10.1.65.159 cilium-test pod-to-b-multi-node-nodeport-7496df84d7-qtxl2 2704 20459 ready 10.1.94.155 cilium-test pod-to-external-1111-6d4f9d9645-pnlgv 1517 23485 ready 10.1.104.150 cilium-test pod-to-external-fqdn-allow-google-cnp-5bc496897c-cspwr 157 43510 ready 10.1.65.164 kube-system coredns-86f7d88d77-dl769 195 3617 ready 10.1.98.80 kube-system coredns-86f7d88d77-rt84l 395 3617 ready 10.1.87.50
CiliumのCRDを確認する。
$ k api-resources | grep -e NAME -e cilium NAME SHORTNAMES APIVERSION NAMESPACED KIND ciliumclusterwidenetworkpolicies ccnp cilium.io/v2 false CiliumClusterwideNetworkPolicy ciliumendpoints cep,ciliumep cilium.io/v2 true CiliumEndpoint ciliumexternalworkloads cew cilium.io/v2 false CiliumExternalWorkload ciliumidentities ciliumid cilium.io/v2 false CiliumIdentity ciliumlocalredirectpolicies clrp cilium.io/v2 true CiliumLocalRedirectPolicy ciliumnetworkpolicies cnp,ciliumnp cilium.io/v2 true CiliumNetworkPolicy ciliumnodes cn,ciliumn cilium.io/v2 false CiliumNode
テストで作成されたポリシーを確認する。
$ k -n cilium-test get cnp NAME AGE pod-to-a-allowed-cnp 7m43s pod-to-a-denied-cnp 7m43s pod-to-external-fqdn-allow-google-cnp 7m43s
1つめのポリシーを確認する。
apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: labels: component: policy-check name: pod-to-a-allowed-cnp quarantine: "false" topology: any traffic: internal type: autocheck name: pod-to-a-allowed-cnp namespace: cilium-test spec: egress: - toEndpoints: - matchLabels: name: echo-a toPorts: - ports: - port: "8080" protocol: TCP - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: "53" protocol: ANY - toEndpoints: - matchLabels: k8s:dns.operator.openshift.io/daemonset-dns: default k8s:io.kubernetes.pod.namespace: openshift-dns toPorts: - ports: - port: "5353" protocol: UDP endpointSelector: matchLabels: name: pod-to-a-allowed-cnp
- 対象は
name=pod-to-a-allowed-cnpのラベルを持つPod name=echo-aのラベルを持つEndpointへのTCP: 8080へのアウトバウンド接続を許可- 名前解決の問い合わせの許可
確認する。名前解決できる。
$ k exec -it pod-to-a-allowed-cnp-7776c879f-qltmn -- sh / # nslookup kubernetes.default nslookup: can't resolve '(null)': Name does not resolve Name: kubernetes.default Address 1: 172.20.0.1 kubernetes.default.svc.cluster.local
name=echo-aのラベルを持つPodの8080ポートに疎通する。
/ # curl 10.1.67.236:8080 <html> <head> （省略）
他の同じく8080をListenしているPodには繋がらない。
$ k exec -it pod-to-a-allowed-cnp-7776c879f-qltmn -- sh / # curl 10.1.96.204:8080
2つめのポリシーを確認する。
apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: labels: component: policy-check name: pod-to-a-denied-cnp quarantine: "false" topology: any traffic: internal type: autocheck name: pod-to-a-denied-cnp namespace: cilium-test spec: egress: - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: "53" protocol: ANY - toEndpoints: - matchLabels: k8s:dns.operator.openshift.io/daemonset-dns: default k8s:io.kubernetes.pod.namespace: openshift-dns toPorts: - ports: - port: "5353" protocol: UDP endpointSelector: matchLabels: name: pod-to-a-denied-cnp
- 対象は
name=pod-to-a-denied-cnpのラベルを持つPod - 名前解決の問い合わせの許可
確認する。名前解決できる。
$ k exec -it pod-to-a-denied-cnp-b5ff897c7-kwz2m -- sh / # nslookup kubernetes.default nslookup: can't resolve '(null)': Name does not resolve Name: kubernetes.default Address 1: 172.20.0.1 kubernetes.default.svc.cluster.local
name=echo-aのラベルを持つPodの8080ポートに疎通しないようになっている。
/ # curl 10.1.67.236:8080
3つめのポリシーを確認する。
apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: labels: component: policy-check name: pod-to-external-fqdn-allow-google-cnp quarantine: "false" topology: any traffic: external type: autocheck name: pod-to-external-fqdn-allow-google-cnp namespace: cilium-test spec: egress: - toFQDNs: - matchPattern: '*.google.com' - toEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: kube-system k8s:k8s-app: kube-dns toPorts: - ports: - port: "53" protocol: ANY rules: dns: - matchPattern: '*' - toEndpoints: - matchLabels: k8s:dns.operator.openshift.io/daemonset-dns: default k8s:io.kubernetes.pod.namespace: openshift-dns toPorts: - ports: - port: "5353" protocol: UDP rules: dns: - matchPattern: '*' endpointSelector: matchLabels: name: pod-to-external-fqdn-allow-google-cnp
- 対象は
pod-to-external-fqdn-allow-google-cnpのラベルを持つPod *.google.comへのアウトバウンド接続の許可- 名前解決の問い合わせの許可
確認する。google.comは繋がる。
$ k exec -it pod-to-external-fqdn-allow-google-cnp-5bc496897c-cspwr -- sh / # curl http://www.google.com/ <!doctype html><html itemscope="" itemtype="http://schema.org/WebPage" lang="ja"> （省略）
確認する。google.co.jpは繋がらない。
/ # curl http://www.google.co.jp/
Hubbleのインストール
可視化ツールのHubbleを導入する。以下のリンク先も参照すること。
export CILIUM_NAMESPACE=kube-system helm upgrade cilium cilium/cilium --version 1.9.3 \ --namespace $CILIUM_NAMESPACE \ --reuse-values \ --set hubble.listenAddress=":4244" \ --set hubble.relay.enabled=true \ --set hubble.ui.enabled=true
HubbleのPodが起動したことを確認する。
$ k get po -n kube-system NAME READY STATUS RESTARTS AGE aws-node-p6k9m 1/1 Running 0 4d20h aws-node-qttwr 1/1 Running 0 4d20h cilium-dffdz 1/1 Running 0 3d3h cilium-node-init-f6snw 1/1 Running 0 3d3h cilium-node-init-f72cr 1/1 Running 0 3d3h cilium-operator-db487bc5b-2m86x 1/1 Running 0 3d3h cilium-operator-db487bc5b-8s4sm 1/1 Running 0 3d3h cilium-smm9f 1/1 Running 0 3d3h coredns-86f7d88d77-dl769 1/1 Running 0 3d3h coredns-86f7d88d77-rt84l 1/1 Running 0 3d3h hubble-relay-f489fcbbb-8xg6b 1/1 Running 0 3m44s hubble-ui-769fb95577-g87xw 3/3 Running 0 3m44s kube-proxy-59krv 1/1 Running 0 4d20h kube-proxy-mh4dv 1/1 Running 0 4d20h
Hubble UIに対してポートフォワードでアクセスする。
kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-ui --address 0.0.0.0 --address :: 12000:80
トラフィックが可視化される。これはかっこいい！
Hubble CLIもある。
Hubble Relayに対してポートフォワードする。
kubectl port-forward -n $CILIUM_NAMESPACE svc/hubble-relay --address 0.0.0.0 --address :: 4245:80
エンドポイントに接続できることを確認する。
$ hubble --server localhost:4245 status Healthcheck (via localhost:4245): Ok Current/Max Flows: 8192/8192 (100.00%) Flows/s: 57.08 Connected Nodes: 2/2
$ hubble --server localhost:4245 observe TIMESTAMP SOURCE DESTINATION TYPE VERDICT SUMMARY Feb 1 06:22:34.295 kube-system/coredns-86f7d88d77-dl769:35746 10.1.0.2:53 to-stack FORWARDED UDP Feb 1 06:22:34.295 10.1.0.2:53 kube-system/coredns-86f7d88d77-dl769:35746 to-endpoint FORWARDED UDP Feb 1 06:22:34.295 kube-system/coredns-86f7d88d77-dl769:53 10.1.90.101:53224 to-stack FORWARDED UDP Feb 1 06:22:34.299 cilium-test/pod-to-external-fqdn-allow-google-cnp-5bc496897c-cspwr:57414 www.google.com:80 to-stack FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:34.378 www.google.com:80 cilium-test/pod-to-external-fqdn-allow-google-cnp-5bc496897c-cspwr:57414 to-endpoint FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:34.378 cilium-test/pod-to-external-fqdn-allow-google-cnp-5bc496897c-cspwr:57414 www.google.com:80 to-stack FORWARDED TCP Flags: ACK, FIN Feb 1 06:22:34.380 www.google.com:80 cilium-test/pod-to-external-fqdn-allow-google-cnp-5bc496897c-cspwr:57414 to-endpoint FORWARDED TCP Flags: ACK, FIN Feb 1 06:22:34.380 cilium-test/pod-to-external-fqdn-allow-google-cnp-5bc496897c-cspwr:57414 www.google.com:80 to-stack FORWARDED TCP Flags: ACK Feb 1 06:22:34.691 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 kube-system/coredns-86f7d88d77-dl769:53 L3-L4 FORWARDED UDP Feb 1 06:22:34.691 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 kube-system/coredns-86f7d88d77-dl769:53 to-stack FORWARDED UDP Feb 1 06:22:34.691 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 kube-system/coredns-86f7d88d77-dl769:53 to-endpoint FORWARDED UDP Feb 1 06:22:34.691 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 kube-system/coredns-86f7d88d77-dl769:53 to-stack FORWARDED UDP Feb 1 06:22:34.691 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 kube-system/coredns-86f7d88d77-dl769:53 to-endpoint FORWARDED UDP Feb 1 06:22:34.692 kube-system/coredns-86f7d88d77-dl769:53 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 to-stack FORWARDED UDP Feb 1 06:22:34.692 kube-system/coredns-86f7d88d77-dl769:53 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 to-endpoint FORWARDED UDP Feb 1 06:22:34.692 kube-system/coredns-86f7d88d77-dl769:53 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 to-stack FORWARDED UDP Feb 1 06:22:34.692 kube-system/coredns-86f7d88d77-dl769:53 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:33934 to-endpoint FORWARDED UDP Feb 1 06:22:34.692 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 L3-L4 FORWARDED TCP Flags: SYN Feb 1 06:22:34.692 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-stack FORWARDED TCP Flags: SYN Feb 1 06:22:34.693 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-endpoint FORWARDED TCP Flags: SYN Feb 1 06:22:34.693 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 to-stack FORWARDED TCP Flags: SYN, ACK Feb 1 06:22:34.694 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 to-endpoint FORWARDED TCP Flags: SYN, ACK Feb 1 06:22:34.694 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-stack FORWARDED TCP Flags: ACK Feb 1 06:22:34.694 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-stack FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:34.695 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-endpoint FORWARDED TCP Flags: ACK Feb 1 06:22:34.695 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-endpoint FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:34.698 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 to-stack FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:34.699 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 to-endpoint FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:34.699 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-stack FORWARDED TCP Flags: ACK, FIN Feb 1 06:22:34.700 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-endpoint FORWARDED TCP Flags: ACK, FIN Feb 1 06:22:34.700 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 to-stack FORWARDED TCP Flags: ACK, FIN Feb 1 06:22:34.701 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 to-endpoint FORWARDED TCP Flags: ACK, FIN Feb 1 06:22:34.702 cilium-test/pod-to-a-allowed-cnp-7776c879f-qltmn:40020 cilium-test/echo-a-57cbbd9b8b-mxrfs:8080 to-endpoint FORWARDED TCP Flags: ACK Feb 1 06:22:34.989 10.1.84.176:443 kube-system/coredns-86f7d88d77-rt84l:41758 to-endpoint FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:35.456 10.1.90.101:47900 kube-system/coredns-86f7d88d77-rt84l:8080 to-endpoint FORWARDED TCP Flags: SYN Feb 1 06:22:35.456 kube-system/coredns-86f7d88d77-rt84l:8080 10.1.90.101:47900 to-stack FORWARDED TCP Flags: SYN, ACK Feb 1 06:22:35.456 10.1.90.101:47900 kube-system/coredns-86f7d88d77-rt84l:8080 to-endpoint FORWARDED TCP Flags: ACK Feb 1 06:22:35.456 10.1.90.101:47900 kube-system/coredns-86f7d88d77-rt84l:8080 to-endpoint FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:35.456 kube-system/coredns-86f7d88d77-rt84l:8080 10.1.90.101:47900 to-stack FORWARDED TCP Flags: ACK, PSH Feb 1 06:22:35.457 kube-system/coredns-86f7d88d77-rt84l:8080 10.1.90.101:47900 to-stack FORWARDED TCP Flags: ACK, FIN
疑問点の調査
まずテスト用アプリは削除する。
kubectl delete -n cilium-test -f https://raw.githubusercontent.com/cilium/cilium/v1.9/examples/kubernetes/connectivity-check/connectivity-check.yaml kubectl delete ns cilium-test
普通のNetwork Policyは使えるのか
CiliumのマニュアルにはCiliumNetworkPolicyの例があるが、そもそもKubernetesのNetworkPolicyが使えるのかを確認する。
まずPodを2つ起動する。
$ k run pod1 --image=nginx pod/pod1 created $ k run pod2 --image=nginx pod/pod2 created $ k label pod pod1 app=pod1 pod/pod1 labeled $ k label pod pod2 app=pod2 pod/pod2 labeled
pod1からpod2に疎通できることを確認する。
$ k get po -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod1 1/1 Running 0 9m9s 10.1.92.166 ip-10-1-66-141.ap-northeast-1.compute.internal <none> <none>
pod2 1/1 Running 0 8m58s 10.1.85.107 ip-10-1-66-141.ap-northeast-1.compute.internal <none> <none>
$ k exec -it pod1 -- curl 10.1.85.107
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
全てブロックするNetworkPolicyを作成する。
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all spec: podSelector: {} policyTypes: - Ingress - Egress
$ k apply -f default-deny-all-np.yaml networkpolicy.networking.k8s.io/default-deny-all created
ブロックされることを確認する。
$ k exec -it pod1 -- curl 10.1.85.107
つまり、普通のNetworkPolicyも使える。
両方定義したらどうなるのか
Calicoの場合は、拒否ルールが使えたり優先度の定義ができたりする。Ciliumも拒否ポリシーはあるが、ベータ機能。
先ほどKubernetesのNetworkPolicyで全て拒否を定義したが、CiliumNetworkPolicyでpod1からpod2への通信の許可を定義する。
apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: pod1-egress spec: endpointSelector: matchLabels: app: pod1 egress: - toEndpoints: - matchLabels: app: pod2
apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: pod2-ingress spec: endpointSelector: matchLabels: app: pod2 ingress: - fromEndpoints: - matchLabels: app: pod1
$ k apply -f pod1-egress-cnp.yaml ciliumnetworkpolicy.cilium.io/pod1-egress created $ k apply -f pod2-ingress-cnp.yaml ciliumnetworkpolicy.cilium.io/pod2-ingress created
動作確認する。
$ k exec -it pod1 -- curl 10.1.85.107
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
両方を組み合わせても使える。
念のため、反対にしてみる。NetworkPolicyとCiliumNetworkPolicyを削除する。
k delete netpol default-deny-all k delete cnp pod1-egress k delete cnp pod2-ingress
全て禁止するCiliumNetworkPolicyを作成する。
apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: default-deny-all spec: endpointSelector: {} ingress: - {} egress: - {}
$ k apply -f default-deny-all-cnp.yaml ciliumnetworkpolicy.cilium.io/default-deny-all created
ブロックされることを確認する。
$ k exec -it pod1 -- curl 10.1.85.107
KubernetesのNetworkPolicyでpod1からpod2への通信の許可を定義する。
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: pod1-egress spec: podSelector: matchLabels: app: pod1 policyTypes: - Egress egress: - to: - podSelector: matchLabels: app: pod2
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: pod2-ingress spec: podSelector: matchLabels: app: pod2 policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: pod1
$ k apply -f pod1-egress-np.yaml networkpolicy.networking.k8s.io/pod1-egress created $ k apply -f pod2-ingress-np.yaml networkpolicy.networking.k8s.io/pod2-ingress created
疎通を確認する。
$ k exec -it pod1 -- curl 10.1.85.107
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
両方定義した和集合でよさそう。
