EKS Distroをkopsで動かしてみたメモ。
参考リンク
手順
まずリポジトリをクローンする。
git clone git@github.com:aws/eks-distro.git cd eks-distro
クラスター作成スクリプトを実行するが、プロファイルがap-northeast-1を向いている状態でスクリプトを実行し、us-uast-1にkopsのバケットを作ろうとすると上手くいかなかったので、ちょっとだけスクリプトを直した。
# Create the bucket if it doesn't exist _bucket_name=$(aws s3api list-buckets --query "Buckets[?Name=='$S3_BUCKET'].Name | [0]" --out text) if [ $_bucket_name == "None" ]; then echo "Creating S3 bucket: $S3_BUCKET" if [ "$AWS_DEFAULT_REGION" == "us-east-1" ]; then - aws s3api create-bucket --bucket $S3_BUCKET + aws s3api create-bucket --bucket $S3_BUCKET --region us-east-1 else aws s3api create-bucket --bucket $S3_BUCKET --create-bucket-configuration LocationConstraint=$AWS_DEFAULT_REGION fi fi
スクリプトを実行する。
cd development/kops ./create_cluster.sh
対話に答える。ここで指定したリージョンにバケットだけでなくクラスターも作成されてしまうので、リージョンを選択したい場合は生成されたクラスター定義を編集してからクラスターを作成する必要がありそう。
$ ./create_cluster.sh
Cluster name must be an FQDN: <yourcluster>.yourdomain.com or <yourcluster>.sub.yourdomain.com
What is the name of your Cluster? myeksdcluster.sotoiwa.dev
Please set a default region for the kops config bucket (e.g. us-west-2)
What region would you like to store the config? us-east-1
Using S3 bucket kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw: to use with kops run
Creating S3 bucket: kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw
{
"Location": "/kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw"
}
W1215 16:20:47.763377 6086 root.go:251] no context set in kubecfg
Creating cluster
Created cluster/myeksdcluster.sotoiwa.dev
Created instancegroup/control-plane-us-east-1a
Created instancegroup/nodes
To deploy these resources, run: kops update cluster myeksdcluster.sotoiwa.dev --yes
Creating cluster ssh key
Do you want to create the cluster now? [Y/n] y
*********************************************************************************
A new kubernetes version is available: 1.18.12
Upgrading is recommended (try kops upgrade cluster)
More information: https://github.com/kubernetes/kops/blob/master/permalinks/upgrade_k8s.md#1.18.12
*********************************************************************************
*********************************************************************************
Kubelet anonymousAuth is currently turned on. This allows RBAC escalation and remote code execution possibilities.
It is highly recommended you turn it off by setting 'spec.kubelet.anonymousAuth' to 'false' via 'kops edit cluster'
See https://kops.sigs.k8s.io/security/#kubelet-api
*********************************************************************************
I1215 16:21:09.254257 6111 networking.go:62] Using CNI asset version "https://distro.eks.amazonaws.com/kubernetes-1-18/releases/1/artifacts/plugins/v0.8.7/cni-plugins-linux-amd64-v0.8.7.tar.gz", as set in CNI_VERSION_URL
I1215 16:21:09.254523 6111 networking.go:66] Using CNI asset hash "sha256:7426431524c2976f481105b80497238030e1c3eedbfcad00e2a9ccbaaf9eef9d", as set in CNI_ASSET_HASH_STRING
I1215 16:21:14.803035 6111 executor.go:103] Tasks: 0 done / 94 total; 45 can run
I1215 16:21:16.247255 6111 vfs_castore.go:590] Issuing new certificate: "etcd-manager-ca-main"
I1215 16:21:16.345858 6111 vfs_castore.go:590] Issuing new certificate: "etcd-peers-ca-main"
I1215 16:21:16.352367 6111 vfs_castore.go:590] Issuing new certificate: "ca"
I1215 16:21:16.358723 6111 vfs_castore.go:590] Issuing new certificate: "etcd-clients-ca"
I1215 16:21:16.382699 6111 vfs_castore.go:590] Issuing new certificate: "apiserver-aggregator-ca"
I1215 16:21:16.436799 6111 vfs_castore.go:590] Issuing new certificate: "etcd-manager-ca-events"
I1215 16:21:16.472615 6111 vfs_castore.go:590] Issuing new certificate: "etcd-peers-ca-events"
I1215 16:21:20.008759 6111 executor.go:103] Tasks: 45 done / 94 total; 29 can run
I1215 16:21:21.245835 6111 vfs_castore.go:590] Issuing new certificate: "master"
I1215 16:21:21.269008 6111 vfs_castore.go:590] Issuing new certificate: "kube-controller-manager"
I1215 16:21:21.280184 6111 vfs_castore.go:590] Issuing new certificate: "kube-scheduler"
I1215 16:21:21.306500 6111 vfs_castore.go:590] Issuing new certificate: "kubecfg"
I1215 16:21:21.370665 6111 vfs_castore.go:590] Issuing new certificate: "aws-iam-authenticator"
I1215 16:21:21.396568 6111 vfs_castore.go:590] Issuing new certificate: "apiserver-proxy-client"
I1215 16:21:21.442213 6111 vfs_castore.go:590] Issuing new certificate: "kops"
I1215 16:21:21.486057 6111 vfs_castore.go:590] Issuing new certificate: "kubelet-api"
I1215 16:21:21.609507 6111 vfs_castore.go:590] Issuing new certificate: "kube-proxy"
I1215 16:21:21.664793 6111 vfs_castore.go:590] Issuing new certificate: "kubelet"
I1215 16:21:22.569972 6111 vfs_castore.go:590] Issuing new certificate: "apiserver-aggregator"
I1215 16:21:24.557641 6111 executor.go:103] Tasks: 74 done / 94 total; 18 can run
I1215 16:21:29.878421 6111 executor.go:103] Tasks: 92 done / 94 total; 2 can run
I1215 16:21:31.246037 6111 executor.go:103] Tasks: 94 done / 94 total; 0 can run
I1215 16:21:31.246890 6111 dns.go:156] Pre-creating DNS records
I1215 16:21:33.902947 6111 update_cluster.go:308] Exporting kubecfg for cluster
kops has set your kubectl context to myeksdcluster.sotoiwa.dev
Cluster is starting. It should be ready in a few minutes.
Suggestions:
* validate cluster: kops validate cluster --wait 10m
* list nodes: kubectl get nodes --show-labels
* ssh to the master: ssh -i ~/.ssh/id_rsa ubuntu@api.myeksdcluster.sotoiwa.dev
* the ubuntu user is specific to Ubuntu. If not using Ubuntu please use the appropriate user based on your OS.
* read about installing addons at: https://kops.sigs.k8s.io/operations/addons.
You can validate your cluster with
KOPS_STATE_STORE=s3://kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw kops validate cluster --wait 10m
Kops state is stored in s3://kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw
You will need to manually deploy the aws authenticator config once the cluster is ready with
kubectl apply -f ./aws-iam-authenticator.yaml
作成されたクラスターの定義は以下のようになっている。
apiVersion: kops.k8s.io/v1alpha2 kind: Cluster metadata: creationTimestamp: null name: myeksdcluster.sotoiwa.dev spec: api: dns: {} authorization: rbac: {} channel: stable cloudProvider: aws configBase: s3://kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw containerRuntime: docker etcdClusters: - cpuRequest: 200m etcdMembers: - instanceGroup: control-plane-us-east-1a name: a memoryRequest: 100Mi name: main - cpuRequest: 100m etcdMembers: - instanceGroup: control-plane-us-east-1a name: a memoryRequest: 100Mi name: events iam: allowContainerRegistry: true legacy: false kubelet: anonymousAuth: false kubernetesApiAccess: - 0.0.0.0/0 kubernetesVersion: https://distro.eks.amazonaws.com/kubernetes-1-18/releases/1/artifacts/kubernetes/v1.18.9 masterPublicName: api.myeksdcluster.sotoiwa.dev networkCIDR: 172.20.0.0/16 networking: kubenet: {} nonMasqueradeCIDR: 100.64.0.0/10 sshAccess: - 0.0.0.0/0 subnets: - cidr: 172.20.32.0/19 name: us-east-1a type: Public zone: us-east-1a - cidr: 172.20.64.0/19 name: us-east-1b type: Public zone: us-east-1b - cidr: 172.20.96.0/19 name: us-east-1c type: Public zone: us-east-1c topology: dns: type: Public masters: public nodes: public kubeAPIServer: image: public.ecr.aws/eks-distro/kubernetes/kube-apiserver:v1.18.9-eks-1-18-1 kubeControllerManager: image: public.ecr.aws/eks-distro/kubernetes/kube-controller-manager:v1.18.9-eks-1-18-1 kubeScheduler: image: public.ecr.aws/eks-distro/kubernetes/kube-scheduler:v1.18.9-eks-1-18-1 kubeProxy: image: public.ecr.aws/eks-distro/kubernetes/kube-proxy:v1.18.9-eks-1-18-1 # Metrics Server will be supported with kops 1.19 metricsServer: enabled: true image: public.ecr.aws/eks-distro/kubernetes-sigs/metrics-server:v0.4.0-eks-1-18-1 authentication: aws: image: public.ecr.aws/eks-distro/kubernetes-sigs/aws-iam-authenticator:v0.5.2-eks-1-18-1 kubeDNS: provider: CoreDNS coreDNSImage: public.ecr.aws/eks-distro/coredns/coredns:v1.7.0-eks-1-18-1 externalCoreFile: | .:53 { errors health { lameduck 5s } kubernetes cluster.local. in-addr.arpa ip6.arpa { pods insecure #upstream fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf loop cache 30 loadbalance reload } masterKubelet: podInfraContainerImage: public.ecr.aws/eks-distro/kubernetes/pause:v1.18.9-eks-1-18-1 # kubelet might already be defined, append the following config kubelet: podInfraContainerImage: public.ecr.aws/eks-distro/kubernetes/pause:v1.18.9-eks-1-18-1 --- apiVersion: kops.k8s.io/v1alpha2 kind: InstanceGroup metadata: creationTimestamp: null labels: kops.k8s.io/cluster: myeksdcluster.sotoiwa.dev name: control-plane-us-east-1a spec: image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20201026 machineType: t3.medium maxSize: 1 minSize: 1 nodeLabels: kops.k8s.io/instancegroup: control-plane-us-east-1a role: Master subnets: - us-east-1a --- apiVersion: kops.k8s.io/v1alpha2 kind: InstanceGroup metadata: creationTimestamp: null labels: kops.k8s.io/cluster: myeksdcluster.sotoiwa.dev name: nodes spec: image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20201026 machineType: t3.medium maxSize: 3 minSize: 3 nodeLabels: kops.k8s.io/instancegroup: nodes role: Node subnets: - us-east-1a - us-east-1b - us-east-1c
クラスターの作成完了を待つ。
KOPS_STATE_STORE=s3://kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw kops validate cluster --wait 10m
バリデーションがいつまでも完了しないが、失敗しているのはaws-iam-authenticatorのPodなので、待っていても完了しなそう。
INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east-1a Master t3.medium 1 1 us-east-1a nodes Node t3.medium 3 3 us-east-1a,us-east-1b,us-east-1c NODE STATUS NAME ROLE READY ip-172-20-123-145.ec2.internal node True ip-172-20-44-162.ec2.internal node True ip-172-20-56-21.ec2.internal master True ip-172-20-92-213.ec2.internal node True VALIDATION ERRORS KIND NAME MESSAGE Pod kube-system/aws-iam-authenticator-snsng system-node-critical pod "aws-iam-authenticator-snsng" is pending Validation Failed W1215 16:29:55.652349 6878 validate_cluster.go:221] (will retry): cluster not yet healthy
止めて手動で適用する。
$ kubectl apply -f ./aws-iam-authenticator.yaml configmap/aws-iam-authenticator created
もう一度バリデーションする。
$ KOPS_STATE_STORE=s3://kops-state-store-jqxgvzcjuqkbzgpsxodjujmlcfafdzvw kops validate cluster --wait 10m Using cluster from kubectl context: myeksdcluster.sotoiwa.dev Validating cluster myeksdcluster.sotoiwa.dev INSTANCE GROUPS NAME ROLE MACHINETYPE MIN MAX SUBNETS control-plane-us-east-1a Master t3.medium 1 1 us-east-1a nodes Node t3.medium 3 3 us-east-1a,us-east-1b,us-east-1c NODE STATUS NAME ROLE READY ip-172-20-123-145.ec2.internal node True ip-172-20-44-162.ec2.internal node True ip-172-20-56-21.ec2.internal master True ip-172-20-92-213.ec2.internal node True Your cluster myeksdcluster.sotoiwa.dev is ready
確認する。
$ k get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-172-20-123-145.ec2.internal Ready node 5m43s v1.18.9-eks-1-18-1 172.20.123.145 34.229.194.96 Ubuntu 20.04.1 LTS 5.4.0-1029-aws docker://19.3.11 ip-172-20-44-162.ec2.internal Ready node 5m34s v1.18.9-eks-1-18-1 172.20.44.162 3.237.65.147 Ubuntu 20.04.1 LTS 5.4.0-1029-aws docker://19.3.11 ip-172-20-56-21.ec2.internal Ready master 7m4s v1.18.9-eks-1-18-1 172.20.56.21 3.239.123.140 Ubuntu 20.04.1 LTS 5.4.0-1029-aws docker://19.3.11 ip-172-20-92-213.ec2.internal Ready node 5m48s v1.18.9-eks-1-18-1 172.20.92.213 52.55.213.153 Ubuntu 20.04.1 LTS 5.4.0-1029-aws docker://19.3.11
$ k get po -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system aws-iam-authenticator-snsng 1/1 Running 0 6m24s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system coredns-6fc4c7f56d-9ldzr 1/1 Running 0 7m20s 100.96.1.3 ip-172-20-92-213.ec2.internal <none> <none> kube-system coredns-6fc4c7f56d-dmlr2 1/1 Running 0 5m29s 100.96.3.2 ip-172-20-44-162.ec2.internal <none> <none> kube-system coredns-autoscaler-78c9dc7f45-mkxcx 1/1 Running 0 7m20s 100.96.1.2 ip-172-20-92-213.ec2.internal <none> <none> kube-system dns-controller-85fb76bb5b-nnrml 1/1 Running 0 7m20s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system etcd-manager-events-ip-172-20-56-21.ec2.internal 1/1 Running 0 6m56s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system etcd-manager-main-ip-172-20-56-21.ec2.internal 1/1 Running 0 6m39s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system kops-controller-4vljx 1/1 Running 0 6m24s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system kube-apiserver-ip-172-20-56-21.ec2.internal 2/2 Running 0 6m29s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system kube-controller-manager-ip-172-20-56-21.ec2.internal 1/1 Running 0 6m51s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system kube-proxy-ip-172-20-123-145.ec2.internal 1/1 Running 0 4m50s 172.20.123.145 ip-172-20-123-145.ec2.internal <none> <none> kube-system kube-proxy-ip-172-20-44-162.ec2.internal 1/1 Running 0 5m24s 172.20.44.162 ip-172-20-44-162.ec2.internal <none> <none> kube-system kube-proxy-ip-172-20-56-21.ec2.internal 1/1 Running 0 6m24s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none> kube-system kube-proxy-ip-172-20-92-213.ec2.internal 1/1 Running 0 6m1s 172.20.92.213 ip-172-20-92-213.ec2.internal <none> <none> kube-system kube-scheduler-ip-172-20-56-21.ec2.internal 1/1 Running 0 6m42s 172.20.56.21 ip-172-20-56-21.ec2.internal <none> <none>
