ExternalDNSを試すメモ。
コンポーネント | バージョン |
---|---|
EKS | 1.21 |
プラットフォームバージョン | eks.2 |
AWS Load Balancer Controller | v2.2.4 |
AWS Load Balancer Controllerチャート | 1.2.7 |
ExternalDNS | v0.7.6 |
クラスターの作成
1.21でクラスターを作成する。ノードなしで作成する。
cat << EOF > cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: externaldns region: ap-northeast-1 version: "1.21" vpc: cidr: "10.0.0.0/16" availabilityZones: - ap-northeast-1a - ap-northeast-1c cloudWatch: clusterLogging: enableTypes: ["*"] iam: withOIDC: true managedNodeGroups: - name: managed-ng-1 minSize: 2 maxSize: 2 desiredCapacity: 2 privateNetworking: true iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore EOF
eksctl create cluster -f cluster.yaml
AWS Load Balancer Controller
必須ではないがAWS Load Balancer Controllerを入れる。
curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json aws iam create-policy \ --policy-name AWSLoadBalancerControllerIAMPolicy \ --policy-document file://iam_policy.json
IAMロールとServiceAccountを作成する。
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) CLUSTER_NAME=externaldns eksctl create iamserviceaccount \ --cluster=${CLUSTER_NAME} \ --namespace=kube-system \ --name=aws-load-balancer-controller \ --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/AWSLoadBalancerControllerIAMPolicy \ --override-existing-serviceaccounts \ --approve
HelmでAWS Load Balancer Controllerをインストールする。
TargetGroupBindingのカスタムリソース定義をインストールする。
$ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master" customresourcedefinition.apiextensions.k8s.io/ingressclassparams.elbv2.k8s.aws created customresourcedefinition.apiextensions.k8s.io/targetgroupbindings.elbv2.k8s.aws created
リポジトリを最新化する。
helm repo add eks https://aws.github.io/eks-charts helm repo update
コントローラーをデプロイする。
helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \ --set clusterName=${CLUSTER_NAME} \ --set serviceAccount.create=false \ --set serviceAccount.name=aws-load-balancer-controller \ -n kube-system
Release "aws-load-balancer-controller" does not exist. Installing it now. NAME: aws-load-balancer-controller LAST DEPLOYED: Mon Sep 6 03:49:20 2021 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: AWS Load Balancer controller installed!
確認する。
$ kubectl get deployment -n kube-system aws-load-balancer-controller NAME READY UP-TO-DATE AVAILABLE AGE aws-load-balancer-controller 2/2 2 2 21s
ExternalDNS
パブリックなCloud Map名前空間を作成する。
- https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws-sd.md
- https://aws.amazon.com/jp/premiumsupport/knowledge-center/create-subdomain-route-53/
aws servicediscovery create-public-dns-namespace --name "external-dns-test.sotosugi.com"
確認する。
$ aws servicediscovery list-namespaces { "Namespaces": [ { "Id": "ns-3driiduhotfeurah", "Arn": "arn:aws:servicediscovery:ap-northeast-1:XXXXXXXXXXXX:namespace/ns-3driiduhotfeurah", "Name": "external-dns-test.sotosugi.com", "Type": "DNS_PUBLIC", "Properties": { "DnsProperties": { "HostedZoneId": "Z0678027REBIAZYHXZWX", "SOA": { "TTL": 60 } }, "HttpProperties": { "HttpName": "external-dns-test.sotosugi.com" } }, "CreateDate": "2021-09-06T04:39:50.799000+09:00" } ] }
ExternalDNSをデプロイする。なお、Helmチャートもある。ここで指定しているイメージのバージョンはちょっと古そう。
cat << EOF > external-dns.yaml apiVersion: v1 kind: ServiceAccount metadata: name: external-dns --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: external-dns rules: - apiGroups: [""] resources: ["services","endpoints","pods"] verbs: ["get","watch","list"] - apiGroups: ["extensions","networking.k8s.io"] resources: ["ingresses"] verbs: ["get","watch","list"] - apiGroups: [""] resources: ["nodes"] verbs: ["list","watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: external-dns-viewer roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: external-dns subjects: - kind: ServiceAccount name: external-dns namespace: kube-system --- apiVersion: apps/v1 kind: Deployment metadata: name: external-dns spec: strategy: type: Recreate selector: matchLabels: app: external-dns template: metadata: labels: app: external-dns spec: serviceAccountName: external-dns containers: - name: external-dns image: k8s.gcr.io/external-dns/external-dns:v0.7.6 env: - name: AWS_REGION value: ap-northeast-1 # put your CloudMap NameSpace region args: - --source=service - --source=ingress - --domain-filter=external-dns-test.sotosugi.com # Makes ExternalDNS see only the namespaces that match the specified domain. Omit the filter if you want to process all available namespaces. - --provider=aws-sd - --aws-zone-type=public # Only look at public namespaces. Valid values are public, private, or no value for both) - --txt-owner-id=my-identifier EOF
kubectl apply -f external-dns.yaml -n kube-system
IAMポリシーを作成する。
cat << EOF > iam-policy-external-dns.json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "route53:GetHostedZone", "route53:ListHostedZonesByName", "route53:CreateHostedZone", "route53:DeleteHostedZone", "route53:ChangeResourceRecordSets", "route53:CreateHealthCheck", "route53:GetHealthCheck", "route53:DeleteHealthCheck", "route53:UpdateHealthCheck", "ec2:DescribeVpcs", "ec2:DescribeRegions", "servicediscovery:*" ], "Resource": [ "*" ] } ] } EOF
aws iam create-policy \ --policy-name ExternalDNSIAMPolicy \ --policy-document file://iam-policy-external-dns.json
IAMロールとServiceAccountを作成する。
ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) CLUSTER_NAME=externaldns eksctl create iamserviceaccount \ --cluster=${CLUSTER_NAME} \ --namespace=kube-system \ --name=external-dns \ --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/ExternalDNSIAMPolicy \ --override-existing-serviceaccounts \ --approve
external-dnsのPodを再起動し、ログを確認する。
$ k logs -n kube-system external-dns-cc9498875-v7ptx time="2021-09-05T19:48:26Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws-sd GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[external-dns-test.sotosugi.com] ExcludeDomains:[] ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:my-identifier TXTPrefix: TXTSuffix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME]}" time="2021-09-05T19:48:26Z" level=info msg="Instantiating new Kubernetes client" time="2021-09-05T19:48:26Z" level=info msg="Using inCluster-config based on serviceaccount-token" time="2021-09-05T19:48:26Z" level=info msg="Created Kubernetes client https://172.20.0.1:443" time="2021-09-05T19:48:28Z" level=info msg="Registry \"txt\" cannot be used with AWS Cloud Map. Switching to \"aws-sd\"." time="2021-09-05T19:48:35Z" level=info msg="All records are already up to date"
Serviceの確認
LoadBalancerタイプのサービスを作成する。
cat << EOF > nginx-lb.yaml apiVersion: v1 kind: Service metadata: name: nginx annotations: external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com spec: type: LoadBalancer ports: - port: 80 name: http targetPort: 80 selector: app: nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx ports: - containerPort: 80 name: http EOF
k apply -f nginx-lb.yaml
確認する。
$ k get po,svc NAME READY STATUS RESTARTS AGE pod/nginx-bdc5c7d65-bclwg 1/1 Running 0 15m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 103m service/nginx LoadBalancer 172.20.192.78 ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com 80:31224/TCP 15m
ログを確認する。
time="2021-09-05T19:49:33Z" level=info msg="Creating a new service \"nginx\" in \"ns-3driiduhotfeurah\" namespace" time="2021-09-05T19:49:33Z" level=info msg="Registering a new instance \"ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
サービスとサービスインスタンスがCloud Mapに登録されている。
Route53にも登録されている。
名前解決できてブラウザでもアクセスできることを確認する。
削除する。
k delete -f nginx-lb.yaml
続いてNodePortサービスで試す。
cat << EOF > nginx-np.yaml apiVersion: v1 kind: Service metadata: name: nginx annotations: external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com spec: type: NodePort ports: - port: 80 name: http targetPort: 80 selector: app: nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx ports: - containerPort: 80 name: http EOF
k apply -f nginx-np.yaml
確認する。
$ k get po,svc NAME READY STATUS RESTARTS AGE pod/nginx-bdc5c7d65-25c5f 1/1 Running 0 91s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 110m service/nginx NodePort 172.20.74.220 <none> 80:31544/TCP 91s
ログに出力される。
time="2021-09-05T19:54:35Z" level=info msg="De-registering an instance \"ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)" time="2021-09-05T19:55:36Z" level=info msg="Registering a new instance \"10.0.119.198\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)" time="2021-09-05T19:55:36Z" level=info msg="Registering a new instance \"10.0.69.80\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
$ k get node NAME STATUS ROLES AGE VERSION ip-10-0-119-198.ap-northeast-1.compute.internal Ready <none> 96m v1.21.2-eks-55daa9d ip-10-0-69-80.ap-northeast-1.compute.internal Ready <none> 96m v1.21.2-eks-55daa9d
この場合はノードのIPが登録されている。
これだと外からアクセスできるわけではない。アクセスできる場所にいたとしても、NodePortがわからないとアクセスできない。
$ nslookup nginx.external-dns-test.sotosugi.com Server: 172.17.192.154 Address: 172.17.192.154#53 Non-authoritative answer: Name: nginx.external-dns-test.sotosugi.com Address: 10.0.69.80
削除する。
k delete -f nginx-np.yaml
続いてClusterIPで試す。
cat << EOF > nginx-clusterip.yaml apiVersion: v1 kind: Service metadata: name: nginx annotations: external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com spec: type: ClusterIP ports: - port: 80 name: http targetPort: 80 selector: app: nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx ports: - containerPort: 80 name: http EOF
k apply -f nginx-clusterip.yaml
確認する。
$ k get po,svc NAME READY STATUS RESTARTS AGE pod/nginx-bdc5c7d65-vd9d4 1/1 Running 0 22s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 116m service/nginx ClusterIP 172.20.0.51 <none> 80/TCP 22s
しばらく待っても登録される様子がない。LoadBalancerかNodePortでないとだめっぽい。
time="2021-09-05T20:00:40Z" level=info msg="De-registering an instance \"10.0.119.198\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)" time="2021-09-05T20:00:40Z" level=info msg="De-registering an instance \"10.0.69.80\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)" time="2021-09-05T20:01:41Z" level=info msg="All records are already up to date" time="2021-09-05T20:02:41Z" level=info msg="All records are already up to date" time="2021-09-05T20:03:42Z" level=info msg="All records are already up to date"
k delete -f nginx-clusterip.yaml
他にExternalNameはサポートしているようだが、使い道がよくわからない。
- https://github.com/kubernetes-sigs/external-dns/blob/master/docs/faq.md#which-kubernetes-objects-are-supported
- https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/externalname.md
Ingressの確認
Ingressを作成する。
cat << EOF > nginx-ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: nginx annotations: kubernetes.io/ingress.class: alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/target-type: ip spec: rules: - host: nginx.external-dns-test.sotosugi.com http: paths: - path: / pathType: Prefix backend: service: name: nginx port: number: 80 --- apiVersion: v1 kind: Service metadata: name: nginx annotations: external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com spec: type: ClusterIP ports: - port: 80 name: http targetPort: 80 selector: app: nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx ports: - containerPort: 80 name: http EOF
k apply -f nginx-ingress.yaml
確認する。
$ k get po,svc,ing NAME READY STATUS RESTARTS AGE pod/nginx-bdc5c7d65-rc7jh 1/1 Running 0 7m26s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 142m service/nginx ClusterIP 172.20.124.151 <none> 80/TCP 7m26s NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/nginx <none> nginx.external-dns-test.sotosugi.com k8s-default-nginx-8007b1159a-1140119829.ap-northeast-1.elb.amazonaws.com 80 6m58s
ログ出力される。
time="2021-09-05T20:28:54Z" level=info msg="Updating service \"nginx\"" time="2021-09-05T20:28:54Z" level=info msg="Registering a new instance \"k8s-default-nginx-8007b1159a-1140119829.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"