ExternelDNSを試す

ExternalDNSを試すメモ。

コンポーネント バージョン
EKS 1.21
プラットフォームバージョン eks.2
AWS Load Balancer Controller v2.2.4
AWS Load Balancer Controllerチャート 1.2.7
ExternalDNS v0.7.6

クラスターの作成

1.21でクラスターを作成する。ノードなしで作成する。

cat << EOF > cluster.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: externaldns
  region: ap-northeast-1
  version: "1.21"
vpc:
  cidr: "10.0.0.0/16"

availabilityZones:
  - ap-northeast-1a
  - ap-northeast-1c

cloudWatch:
  clusterLogging:
    enableTypes: ["*"]

iam:
  withOIDC: true

managedNodeGroups:
  - name: managed-ng-1
    minSize: 2
    maxSize: 2
    desiredCapacity: 2
    privateNetworking: true
    iam:
      attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EOF
eksctl create cluster -f cluster.yaml

AWS Load Balancer Controller

必須ではないがAWS Load Balancer Controllerを入れる。

curl -o iam_policy.json https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.2.0/docs/install/iam_policy.json
aws iam create-policy \
    --policy-name AWSLoadBalancerControllerIAMPolicy \
    --policy-document file://iam_policy.json

IAMロールとServiceAccountを作成する。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=externaldns
eksctl create iamserviceaccount \
  --cluster=${CLUSTER_NAME} \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/AWSLoadBalancerControllerIAMPolicy \
  --override-existing-serviceaccounts \
  --approve

HelmでAWS Load Balancer Controllerをインストールする。

TargetGroupBindingのカスタムリソース定義をインストールする。

$ kubectl apply -k "github.com/aws/eks-charts/stable/aws-load-balancer-controller/crds?ref=master"
customresourcedefinition.apiextensions.k8s.io/ingressclassparams.elbv2.k8s.aws created
customresourcedefinition.apiextensions.k8s.io/targetgroupbindings.elbv2.k8s.aws created

リポジトリを最新化する。

helm repo add eks https://aws.github.io/eks-charts
helm repo update

コントローラーをデプロイする。

helm upgrade -i aws-load-balancer-controller eks/aws-load-balancer-controller \
  --set clusterName=${CLUSTER_NAME} \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  -n kube-system
Release "aws-load-balancer-controller" does not exist. Installing it now.
NAME: aws-load-balancer-controller
LAST DEPLOYED: Mon Sep  6 03:49:20 2021
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS Load Balancer controller installed!

確認する。

$ kubectl get deployment -n kube-system aws-load-balancer-controller
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
aws-load-balancer-controller   2/2     2            2           21s

ExternalDNS

パブリックなCloud Map名前空間を作成する。

aws servicediscovery create-public-dns-namespace --name "external-dns-test.sotosugi.com"

確認する。

$ aws servicediscovery list-namespaces
{
    "Namespaces": [
        {
            "Id": "ns-3driiduhotfeurah",
            "Arn": "arn:aws:servicediscovery:ap-northeast-1:XXXXXXXXXXXX:namespace/ns-3driiduhotfeurah",
            "Name": "external-dns-test.sotosugi.com",
            "Type": "DNS_PUBLIC",
            "Properties": {
                "DnsProperties": {
                    "HostedZoneId": "Z0678027REBIAZYHXZWX",
                    "SOA": {
                        "TTL": 60
                    }
                },
                "HttpProperties": {
                    "HttpName": "external-dns-test.sotosugi.com"
                }
            },
            "CreateDate": "2021-09-06T04:39:50.799000+09:00"
        }
    ]
}

ExternalDNSをデプロイする。なお、Helmチャートもある。ここで指定しているイメージのバージョンはちょっと古そう。

cat << EOF > external-dns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: external-dns
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: kube-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: k8s.gcr.io/external-dns/external-dns:v0.7.6
        env:
          - name: AWS_REGION
            value: ap-northeast-1 # put your CloudMap NameSpace region
        args:
        - --source=service
        - --source=ingress
        - --domain-filter=external-dns-test.sotosugi.com # Makes ExternalDNS see only the namespaces that match the specified domain. Omit the filter if you want to process all available namespaces.
        - --provider=aws-sd
        - --aws-zone-type=public # Only look at public namespaces. Valid values are public, private, or no value for both)
        - --txt-owner-id=my-identifier
EOF
kubectl apply -f external-dns.yaml -n kube-system

IAMポリシーを作成する。

cat << EOF > iam-policy-external-dns.json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "route53:GetHostedZone",
        "route53:ListHostedZonesByName",
        "route53:CreateHostedZone",
        "route53:DeleteHostedZone",
        "route53:ChangeResourceRecordSets",
        "route53:CreateHealthCheck",
        "route53:GetHealthCheck",
        "route53:DeleteHealthCheck",
        "route53:UpdateHealthCheck",
        "ec2:DescribeVpcs",
        "ec2:DescribeRegions",
        "servicediscovery:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF
aws iam create-policy \
    --policy-name ExternalDNSIAMPolicy \
    --policy-document file://iam-policy-external-dns.json

IAMロールとServiceAccountを作成する。

ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
CLUSTER_NAME=externaldns
eksctl create iamserviceaccount \
  --cluster=${CLUSTER_NAME} \
  --namespace=kube-system \
  --name=external-dns \
  --attach-policy-arn=arn:aws:iam::${ACCOUNT_ID}:policy/ExternalDNSIAMPolicy \
  --override-existing-serviceaccounts \
  --approve

external-dnsのPodを再起動し、ログを確認する。

$ k logs -n kube-system external-dns-cc9498875-v7ptx
time="2021-09-05T19:48:26Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s ContourLoadBalancerService:heptio-contour/contour SkipperRouteGroupVersion:zalando.org/v1 Sources:[service ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws-sd GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[external-dns-test.sotosugi.com] ExcludeDomains:[] ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:my-identifier TXTPrefix: TXTSuffix: Interval:1m0s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME]}"
time="2021-09-05T19:48:26Z" level=info msg="Instantiating new Kubernetes client"
time="2021-09-05T19:48:26Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-09-05T19:48:26Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2021-09-05T19:48:28Z" level=info msg="Registry \"txt\" cannot be used with AWS Cloud Map. Switching to \"aws-sd\"."
time="2021-09-05T19:48:35Z" level=info msg="All records are already up to date"

Serviceの確認

LoadBalancerタイプのサービスを作成する。

cat << EOF > nginx-lb.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF
k apply -f nginx-lb.yaml

確認する。

$ k get po,svc
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-bclwg   1/1     Running   0          15m

NAME                 TYPE           CLUSTER-IP      EXTERNAL-IP                                                                    PORT(S)        AGE
service/kubernetes   ClusterIP      172.20.0.1      <none>                                                                         443/TCP        103m
service/nginx        LoadBalancer   172.20.192.78   ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com   80:31224/TCP   15m

ログを確認する。

time="2021-09-05T19:49:33Z" level=info msg="Creating a new service \"nginx\" in \"ns-3driiduhotfeurah\" namespace"
time="2021-09-05T19:49:33Z" level=info msg="Registering a new instance \"ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"

サービスとサービスインスタンスがCloud Mapに登録されている。

f:id:sotoiwa:20210906054050p:plain

f:id:sotoiwa:20210906054104p:plain

Route53にも登録されている。

f:id:sotoiwa:20210906054125p:plain

名前解決できてブラウザでもアクセスできることを確認する。

f:id:sotoiwa:20210906054141p:plain

削除する。

k delete -f nginx-lb.yaml

続いてNodePortサービスで試す。

cat << EOF > nginx-np.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: NodePort
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF
k apply -f nginx-np.yaml

確認する。

$ k get po,svc
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-25c5f   1/1     Running   0          91s

NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
service/kubernetes   ClusterIP   172.20.0.1      <none>        443/TCP        110m
service/nginx        NodePort    172.20.74.220   <none>        80:31544/TCP   91s

ログに出力される。

time="2021-09-05T19:54:35Z" level=info msg="De-registering an instance \"ae453e8e938554c71a894a4fcd02da71-1691206785.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T19:55:36Z" level=info msg="Registering a new instance \"10.0.119.198\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T19:55:36Z" level=info msg="Registering a new instance \"10.0.69.80\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
$ k get node
NAME                                              STATUS   ROLES    AGE   VERSION
ip-10-0-119-198.ap-northeast-1.compute.internal   Ready    <none>   96m   v1.21.2-eks-55daa9d
ip-10-0-69-80.ap-northeast-1.compute.internal     Ready    <none>   96m   v1.21.2-eks-55daa9d

この場合はノードのIPが登録されている。

これだと外からアクセスできるわけではない。アクセスできる場所にいたとしても、NodePortがわからないとアクセスできない。

$ nslookup nginx.external-dns-test.sotosugi.com
Server:         172.17.192.154
Address:        172.17.192.154#53

Non-authoritative answer:
Name:   nginx.external-dns-test.sotosugi.com
Address: 10.0.69.80

削除する。

k delete -f nginx-np.yaml

続いてClusterIPで試す。

cat << EOF > nginx-clusterip.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF
k apply -f nginx-clusterip.yaml

確認する。

$ k get po,svc
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-vd9d4   1/1     Running   0          22s

NAME                 TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   172.20.0.1    <none>        443/TCP   116m
service/nginx        ClusterIP   172.20.0.51   <none>        80/TCP    22s

しばらく待っても登録される様子がない。LoadBalancerかNodePortでないとだめっぽい。

time="2021-09-05T20:00:40Z" level=info msg="De-registering an instance \"10.0.119.198\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T20:00:40Z" level=info msg="De-registering an instance \"10.0.69.80\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"
time="2021-09-05T20:01:41Z" level=info msg="All records are already up to date"
time="2021-09-05T20:02:41Z" level=info msg="All records are already up to date"
time="2021-09-05T20:03:42Z" level=info msg="All records are already up to date"
k delete -f nginx-clusterip.yaml

他にExternalNameはサポートしているようだが、使い道がよくわからない。

Ingressの確認

Ingressを作成する。

cat << EOF > nginx-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
spec:
  rules:
  - host: nginx.external-dns-test.sotosugi.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.external-dns-test.sotosugi.com
spec:
  type: ClusterIP
  ports:
  - port: 80
    name: http
    targetPort: 80
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          name: http
EOF
k apply -f nginx-ingress.yaml

確認する。

$ k get po,svc,ing
NAME                        READY   STATUS    RESTARTS   AGE
pod/nginx-bdc5c7d65-rc7jh   1/1     Running   0          7m26s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   172.20.0.1       <none>        443/TCP   142m
service/nginx        ClusterIP   172.20.124.151   <none>        80/TCP    7m26s

NAME                              CLASS    HOSTS                                  ADDRESS                                                                    PORTS   AGE
ingress.networking.k8s.io/nginx   <none>   nginx.external-dns-test.sotosugi.com   k8s-default-nginx-8007b1159a-1140119829.ap-northeast-1.elb.amazonaws.com   80      6m58s

ログ出力される。

time="2021-09-05T20:28:54Z" level=info msg="Updating service \"nginx\""
time="2021-09-05T20:28:54Z" level=info msg="Registering a new instance \"k8s-default-nginx-8007b1159a-1140119829.ap-northeast-1.elb.amazonaws.com\" for service \"nginx\" (srv-7mzsw7ldbaixtq2c)"

f:id:sotoiwa:20210906054203p:plain

f:id:sotoiwa:20210906054219p:plain