App MeshのマルチEKSクラスターのチュートリアルをやってみたメモ。
コンポーネント | バージョン | 備考 |
---|---|---|
eksctl | 0.17.0 | |
Kubernetes バージョン | 1.15 | |
プラットフォームのバージョン | eks.9 | |
App Mesh Controller | 0.5.0 | CHART VERSION 0.6.0 |
App Mesh Inject | 0.5.0 | CHART VERSION 0.13.0 |
手順
この手順では、以下のような環境を構築する。
- 同じVPCの2つのEKSクラスターを起動する
- 2つのEKSクラスターにまたがるサービスメッシュを構築する
- frontコンテナを1つめのクラスター(eksc1)、colorappコンテナを2つめのクラスター(eksc2)にデプロイする
- colorappのデプロイメントはblueとredの2つ
クラスターの作成
IRSAがない時代のブログなので、WorkerノードにIAMロールをつけているが、このままやる。1つ目のクラスターを作成する。
eksctl create cluster --name=eksc2 --nodes=3 \ --region=ap-northeast-1 \ --vpc-cidr 172.16.0.0/16 \ --ssh-access --ssh-public-key=sotosugi \ --alb-ingress-access \ --asg-access \ --full-ecr-access \ --external-dns-access \ --appmesh-access \ --auto-kubeconfig
Publicサブネットを取得する。
aws ec2 describe-subnets | \ jq -r '.Subnets[] | select( .Tags ) | select( .MapPublicIpOnLaunch ) | select( [ select( .Tags[].Value | test("eksctl-eksc2-cluster") ) ] | length > 0 ) | .SubnetId'
Priavateサブネットを取得する。
aws ec2 describe-subnets | \ jq -r '.Subnets[] | select( .Tags ) | select( .MapPublicIpOnLaunch | not ) | select( [ select( .Tags[].Value | test("eksctl-eksc2-cluster") ) ] | length > 0 ) | .SubnetId'
作成されたサブネットを指定して2つめのクラスターを作成する。
eksctl create cluster --name=eksc1 --nodes=2 \ --region=ap-northeast-1 \ --vpc-private-subnets=subnet-0f1679713680741b3,subnet-04181c7e03c3ce839,subnet-05bd4759ab3dbf557 \ --vpc-public-subnets=subnet-0db4eced689df5abd,subnet-073a4226744579908,subnet-04b835e53840abd75 \ --ssh-access --ssh-public-key=sotosugi \ --alb-ingress-access \ --asg-access \ --full-ecr-access \ --external-dns-access \ --appmesh-access \ --auto-kubeconfig
各クラスターのWorkノードが通信できるように、マネジメントコンソールからセキュリティグループを修正しておく。
X-Rayを使う場合は、WorkerノードのインスタンスロールにAWSXRayDaemonWriteAccess
が必要なので、マネジメントコンソールから手動でポリシーをアタッチしておく。
App Meshコントローラーのセットアップ
チャートを確認する。
$ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "incubator" chart repository ...Successfully got an update from the "eks" chart repository ...Successfully got an update from the "stable" chart repository Update Complete. ⎈ Happy Helming!⎈ $ helm search repo appmesh NAME CHART VERSION APP VERSION DESCRIPTION eks/appmesh-controller 0.6.0 0.5.0 App Mesh controller Helm chart for Kubernetes eks/appmesh-grafana 0.1.0 6.4.3 App Mesh Grafana Helm chart for Kubernetes eks/appmesh-inject 0.13.0 0.5.0 App Mesh Inject Helm chart for Kubernetes eks/appmesh-jaeger 0.2.0 1.14.0 App Mesh Jaeger Helm chart for Kubernetes eks/appmesh-prometheus 0.3.0 2.13.1 App Mesh Prometheus Helm chart for Kubernetes
両方のクラスターにApp Mesh ControllerとApp Mesh Injectをセットアップする。
kubectl create ns appmesh-system kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/appmesh-controller/crds/crds.yaml helm upgrade -i appmesh-controller eks/appmesh-controller --namespace appmesh-system helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set mesh.create=true --set mesh.name=global # X-Rayを有効にする場合は以下も実施する helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set tracing.enabled=true --set tracing.provider=x-ray
$ kubectl create ns appmesh-system namespace/appmesh-system created $ kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/appmesh-controller/crds/crds.yaml customresourcedefinition.apiextensions.k8s.io/meshes.appmesh.k8s.aws created customresourcedefinition.apiextensions.k8s.io/virtualnodes.appmesh.k8s.aws created customresourcedefinition.apiextensions.k8s.io/virtualservices.appmesh.k8s.aws created $ helm upgrade -i appmesh-controller eks/appmesh-controller --namespace appmesh-system Release "appmesh-controller" does not exist. Installing it now. NAME: appmesh-controller LAST DEPLOYED: Thu May 7 14:34:28 2020 NAMESPACE: appmesh-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: AWS App Mesh controller installed! $ helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set mesh.create=true --set mesh.name=global Release "appmesh-inject" does not exist. Installing it now. NAME: appmesh-inject LAST DEPLOYED: Thu May 7 14:34:50 2020 NAMESPACE: appmesh-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: AWS App Mesh Inject installed! $ helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set tracing.enabled=true --set tracing.provider=x-ray Release "appmesh-inject" has been upgraded. Happy Helming! NAME: appmesh-inject LAST DEPLOYED: Thu May 7 15:28:20 2020 NAMESPACE: appmesh-system STATUS: deployed REVISION: 2 TEST SUITE: None NOTES: AWS App Mesh Inject installed!
アプリケーションとApp Mesh CRDの作成
アプリのリポジトリをクローンする。ここからの作業は1つのコンソールでやればOK。
git clone https://github.com/aws/aws-app-mesh-examples.git
cd aws-app-mesh-examples/walkthroughs/howto-k8s-cross-cluster
環境変数を設定する。Envoyイメージについてはここを参照。
export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) export AWS_DEFAULT_REGION=$(aws configure get region) export ENVOY_IMAGE=840364872350.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/aws-appmesh-envoy:v1.12.3.0-prod export VPC_ID=$(aws ec2 describe-vpcs | jq -r '.Vpcs[] | select( .Tags ) | select( [ select( .Tags[].Value | test("eksctl-eksc2-cluster") ) ] | length > 0 ) | .VpcId') export CLUSTER1=eksc1 export CLUSTER2=eksc2
ここからは、deploy.sh
の内容を手動で実行する。
変数をセットする。
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)" PROJECT_NAME="appmesh-demo" APP_NAMESPACE=${PROJECT_NAME} MESH_NAME=${PROJECT_NAME} CLOUDMAP_NAMESPACE="${PROJECT_NAME}.pvt.aws.local" ECR_IMAGE_PREFIX="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${PROJECT_NAME}" FRONT_APP_IMAGE="${ECR_IMAGE_PREFIX}/feapp" COLOR_APP_IMAGE="${ECR_IMAGE_PREFIX}/colorapp"
Cloud Mapの名前空間を作成する。
aws servicediscovery create-private-dns-namespace \ --name "${CLOUDMAP_NAMESPACE}" \ --vpc "${VPC_ID}"
colorappとfeappイメージをビルドしてECRにpushする。
for app in colorapp feapp; do aws ecr describe-repositories --repository-name $PROJECT_NAME/$app >/dev/null 2>&1 || aws ecr create-repository --repository-name $PROJECT_NAME/$app docker build -t ${ECR_IMAGE_PREFIX}/${app} ${DIR}/${app} aws ecr get-login-password | docker login --username AWS --password-stdin https://${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com docker push ${ECR_IMAGE_PREFIX}/${app} done
ekcs1クラスターにアプリとApp Meshリソースをデプロイする。
EXAMPLES_OUT_DIR="${DIR}/_output/" mkdir -p ${EXAMPLES_OUT_DIR} eval "cat <<EOF $(<${DIR}/cluster1.yaml.template) EOF " >${EXAMPLES_OUT_DIR}/cluster1.yaml KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER1}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster1.yaml
$ KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER1}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster1.yaml namespace/appmesh-demo created deployment.apps/front created service/front created
確認する。
$ kubectl get all -n appmesh-demo NAME READY STATUS RESTARTS AGE pod/front-578b54fc49-kzlhq 3/3 Running 0 98s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/front LoadBalancer 10.100.58.100 a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com 80:31145/TCP 98s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/front 1/1 1 1 98s NAME DESIRED CURRENT READY AGE replicaset.apps/front-578b54fc49 1 1 1 98s NAME AGE mesh.appmesh.k8s.aws/global 56m
$ kubectl get all -n appmesh-system NAME READY STATUS RESTARTS AGE pod/appmesh-controller-54dd6bdfd8-bfhzm 1/1 Running 0 58m pod/appmesh-inject-54dc557cd6-lkxgz 1/1 Running 0 4m26s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/appmesh-inject ClusterIP 10.100.61.44 <none> 443/TCP 58m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/appmesh-controller 1/1 1 1 58m deployment.apps/appmesh-inject 1/1 1 1 58m NAME DESIRED CURRENT READY AGE replicaset.apps/appmesh-controller-54dd6bdfd8 1 1 1 58m replicaset.apps/appmesh-inject-54dc557cd6 1 1 1 4m26s replicaset.apps/appmesh-inject-747db6b88 0 0 0 58m NAME AGE mesh.appmesh.k8s.aws/global 58m
適用しているのは次のようなYAML。frontコンテナは環境変数で指定されたホスト名とURLにリクエストするが、ここがFQDNなので、次に作成するVirtualServiceでもFQDNを指定する必要があるものと思われる。
App Meshのリソースはどっちのクラスターで作ってもよいものと思われる。
--- apiVersion: v1 kind: Namespace metadata: labels: appmesh.k8s.aws/sidecarInjectorWebhook: enabled name: appmesh-demo --- apiVersion: apps/v1 kind: Deployment metadata: name: front namespace: appmesh-demo spec: replicas: 1 selector: matchLabels: app: front version: v1 template: metadata: annotations: appmesh.k8s.aws/mesh: appmesh-demo labels: app: front version: v1 spec: containers: - name: front image: XXXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/appmesh-demo/feapp ports: - containerPort: 8080 env: - name: "PORT" value: "80" - name: "COLOR_HOST" value: "colorapp.appmesh-demo.pvt.aws.local:8080" --- apiVersion: v1 kind: Service metadata: name: front namespace: appmesh-demo spec: type: LoadBalancer ports: - port: 80 protocol: TCP name: http selector: app: front
ekcs2クラスターにアプリとApp Meshリソースをデプロイする。
eval "cat <<EOF $(<${DIR}/cluster2.yaml.template) EOF " >${EXAMPLES_OUT_DIR}/cluster2.yaml KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER2}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster2.yaml
$ KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER2}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster2.yaml namespace/appmesh-demo created mesh.appmesh.k8s.aws/appmesh-demo created virtualnode.appmesh.k8s.aws/front created virtualnode.appmesh.k8s.aws/colorapp-blue created virtualnode.appmesh.k8s.aws/colorapp-red created virtualservice.appmesh.k8s.aws/colorapp.appmesh-demo.pvt.aws.local created deployment.apps/colorapp-blue created deployment.apps/colorapp-red created
確認する。
$ kubectl get all -n appmesh-demo NAME READY STATUS RESTARTS AGE pod/colorapp-blue-66b44c5c77-q8prf 3/3 Running 0 2m53s pod/colorapp-red-54cdb4f6bf-jjjf9 3/3 Running 0 2m53s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/colorapp-blue 1/1 1 1 2m54s deployment.apps/colorapp-red 1/1 1 1 2m53s NAME DESIRED CURRENT READY AGE replicaset.apps/colorapp-blue-66b44c5c77 1 1 1 2m54s replicaset.apps/colorapp-red-54cdb4f6bf 1 1 1 2m53s NAME AGE mesh.appmesh.k8s.aws/appmesh-demo 2m54s mesh.appmesh.k8s.aws/global 58m NAME AGE virtualnode.appmesh.k8s.aws/colorapp-blue 2m54s virtualnode.appmesh.k8s.aws/colorapp-red 2m54s virtualnode.appmesh.k8s.aws/front 2m54s NAME AGE virtualservice.appmesh.k8s.aws/colorapp.appmesh-demo.pvt.aws.local 2m54s
$ kubectl get all -n appmesh-system NAME READY STATUS RESTARTS AGE pod/appmesh-controller-54dd6bdfd8-f2drn 1/1 Running 0 58m pod/appmesh-inject-54dc557cd6-8h5tr 1/1 Running 0 5m10s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/appmesh-inject ClusterIP 10.100.179.184 <none> 443/TCP 58m NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/appmesh-controller 1/1 1 1 58m deployment.apps/appmesh-inject 1/1 1 1 58m NAME DESIRED CURRENT READY AGE replicaset.apps/appmesh-controller-54dd6bdfd8 1 1 1 58m replicaset.apps/appmesh-inject-54dc557cd6 1 1 1 5m10s replicaset.apps/appmesh-inject-747db6b88 0 0 0 58m NAME AGE mesh.appmesh.k8s.aws/appmesh-demo 3m mesh.appmesh.k8s.aws/global 58m
適用しているのは次のようなYAML。仮想ノードのService DiscoveryはDNSではなくCloud Mapになっている。 K8s Serviceは作っていない。
--- apiVersion: v1 kind: Namespace metadata: labels: appmesh.k8s.aws/sidecarInjectorWebhook: enabled name: appmesh-demo --- apiVersion: appmesh.k8s.aws/v1beta1 kind: Mesh metadata: name: appmesh-demo --- apiVersion: appmesh.k8s.aws/v1beta1 kind: VirtualNode metadata: name: front namespace: appmesh-demo spec: meshName: appmesh-demo listeners: - portMapping: port: 8080 protocol: http serviceDiscovery: cloudMap: namespaceName: appmesh-demo.pvt.aws.local serviceName: front backends: - virtualService: virtualServiceName: colorapp.appmesh-demo.pvt.aws.local --- apiVersion: appmesh.k8s.aws/v1beta1 kind: VirtualNode metadata: name: colorapp-blue namespace: appmesh-demo spec: meshName: appmesh-demo listeners: - portMapping: port: 8080 protocol: http serviceDiscovery: cloudMap: namespaceName: appmesh-demo.pvt.aws.local serviceName: colorapp --- apiVersion: appmesh.k8s.aws/v1beta1 kind: VirtualNode metadata: name: colorapp-red namespace: appmesh-demo spec: meshName: appmesh-demo listeners: - portMapping: port: 8080 protocol: http serviceDiscovery: cloudMap: namespaceName: appmesh-demo.pvt.aws.local serviceName: colorapp --- apiVersion: appmesh.k8s.aws/v1beta1 kind: VirtualService metadata: name: colorapp.appmesh-demo.pvt.aws.local namespace: appmesh-demo spec: meshName: appmesh-demo virtualRouter: name: colorapp-router listeners: - portMapping: port: 8080 protocol: http routes: - name: color-route http: match: prefix: / action: weightedTargets: - virtualNodeName: colorapp-red weight: 1 - virtualNodeName: colorapp-blue weight: 1 --- apiVersion: apps/v1 kind: Deployment metadata: name: colorapp-blue namespace: appmesh-demo spec: replicas: 1 selector: matchLabels: app: colorapp version: blue template: metadata: annotations: appmesh.k8s.aws/mesh: appmesh-demo labels: app: colorapp version: blue spec: containers: - name: colorapp image: XXXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/appmesh-demo/colorapp ports: - containerPort: 8080 env: - name: "PORT" value: "8080" - name: "COLOR" value: "blue" --- apiVersion: apps/v1 kind: Deployment metadata: name: colorapp-red namespace: appmesh-demo spec: replicas: 1 selector: matchLabels: app: colorapp version: red template: metadata: annotations: appmesh.k8s.aws/mesh: appmesh-demo labels: app: colorapp version: red spec: containers: - name: colorapp image: XXXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/appmesh-demo/colorapp ports: - containerPort: 8080 env: - name: "PORT" value: "8080" - name: "COLOR" value: "red" ---
Cloud Mapの確認
Cloud Mapでcolorappのバックエンドのサービスインスタンスを確認する。
$ aws servicediscovery discover-instances --namespace appmesh-demo.pvt.aws.local --service-name colorapp { "Instances": [ { "InstanceId": "172.16.33.210", "NamespaceName": "appmesh-demo.pvt.aws.local", "ServiceName": "colorapp", "HealthStatus": "UNKNOWN", "Attributes": { "AWS_INSTANCE_IPV4": "172.16.33.210", "app": "colorapp", "appmesh.k8s.aws/mesh": "appmesh-demo", "appmesh.k8s.aws/virtualNode": "colorapp-blue-appmesh-demo", "k8s.io/namespace": "appmesh-demo", "k8s.io/pod": "colorapp-blue-66b44c5c77-q8prf", "pod-template-hash": "66b44c5c77", "version": "blue" } }, { "InstanceId": "172.16.57.230", "NamespaceName": "appmesh-demo.pvt.aws.local", "ServiceName": "colorapp", "HealthStatus": "UNKNOWN", "Attributes": { "AWS_INSTANCE_IPV4": "172.16.57.230", "app": "colorapp", "appmesh.k8s.aws/mesh": "appmesh-demo", "appmesh.k8s.aws/virtualNode": "colorapp-red-appmesh-demo", "k8s.io/namespace": "appmesh-demo", "k8s.io/pod": "colorapp-red-54cdb4f6bf-jjjf9", "pod-template-hash": "54cdb4f6bf", "version": "red" } } ] }
これを誰が登録しているのかの挙動がよくわからない。 VirtualNodeを消すとエントリが消えるので、VirtualNodeの属性に基づいて登録されているように思えるが、VirtualNodeを定義しただけでは追加されないのでVirtualServiceも必要そうに思える。
このFQDNがeksc1で名前解決できることを確認する。
$ kubectl run busybox --image=busybox:1.28 --rm -it --restart=Never --command -- nslookup colorapp.appmesh-demo.pvt.aws.local Server: 10.100.0.10 Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local Name: colorapp.appmesh-demo.pvt.aws.local Address 1: 172.16.88.89 ip-172-16-88-89.ap-northeast-1.compute.internal Address 2: 172.16.37.132 ip-172-16-37-132.ap-northeast-1.compute.internal pod "busybox" deleted
アプリケーションのテスト
eksc1で外向けServiceのホスト名を確認する。
$ kubectl get svc -n appmesh-demo NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE front LoadBalancer 10.100.58.100 a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com 80:31145/TCP 58m
このホストの/color
にアクセスする。
$ curl -w "\n" http://a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com/color red $ curl -w "\n" http://a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com/color red $ curl -w "\n" http://a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com/color blue
適当にリクエストを送ってX-Rayコンソールを見てみる。アプリはX-Ray SDKが入っていて、Traceもちゃんと見える。