App MeshのマルチEKSクラスターのチュートリアル

App MeshのマルチEKSクラスターのチュートリアルをやってみたメモ。

コンポーネント バージョン 備考
eksctl 0.17.0
Kubernetes バージョン 1.15
プラットフォームのバージョン eks.9
App Mesh Controller 0.5.0 CHART VERSION 0.6.0
App Mesh Inject 0.5.0 CHART VERSION 0.13.0

手順

この手順では、以下のような環境を構築する。

  • 同じVPCの2つのEKSクラスターを起動する
  • 2つのEKSクラスターにまたがるサービスメッシュを構築する
  • frontコンテナを1つめのクラスター(eksc1)、colorappコンテナを2つめのクラスター(eksc2)にデプロイする
  • colorappのデプロイメントはblueとredの2つ

クラスターの作成

IRSAがない時代のブログなので、WorkerノードにIAMロールをつけているが、このままやる。1つ目のクラスターを作成する。

eksctl create cluster --name=eksc2 --nodes=3 \
  --region=ap-northeast-1 \
  --vpc-cidr 172.16.0.0/16 \
  --ssh-access --ssh-public-key=sotosugi \
  --alb-ingress-access \
  --asg-access \
  --full-ecr-access \
  --external-dns-access \
  --appmesh-access \
  --auto-kubeconfig

Publicサブネットを取得する。

aws ec2 describe-subnets | \
  jq -r '.Subnets[] |
           select( .Tags ) |
           select( .MapPublicIpOnLaunch ) |
           select( [ select( .Tags[].Value | test("eksctl-eksc2-cluster") ) ] | length > 0 ) |
           .SubnetId'

Priavateサブネットを取得する。

aws ec2 describe-subnets | \
  jq -r '.Subnets[] |
           select( .Tags ) |
           select( .MapPublicIpOnLaunch | not ) |
           select( [ select( .Tags[].Value | test("eksctl-eksc2-cluster") ) ] | length > 0 ) |
           .SubnetId'

作成されたサブネットを指定して2つめのクラスターを作成する。

eksctl create cluster --name=eksc1 --nodes=2 \
  --region=ap-northeast-1 \
  --vpc-private-subnets=subnet-0f1679713680741b3,subnet-04181c7e03c3ce839,subnet-05bd4759ab3dbf557 \
  --vpc-public-subnets=subnet-0db4eced689df5abd,subnet-073a4226744579908,subnet-04b835e53840abd75 \
  --ssh-access --ssh-public-key=sotosugi \
  --alb-ingress-access \
  --asg-access \
  --full-ecr-access \
  --external-dns-access \
  --appmesh-access \
  --auto-kubeconfig

クラスターのWorkノードが通信できるように、マネジメントコンソールからセキュリティグループを修正しておく。

X-Rayを使う場合は、WorkerノードのインスタンスロールにAWSXRayDaemonWriteAccessが必要なので、マネジメントコンソールから手動でポリシーをアタッチしておく。

App Meshコントローラーのセットアップ

チャートを確認する。

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "incubator" chart repository
...Successfully got an update from the "eks" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈
$ helm search repo appmesh
NAME                     CHART VERSION   APP VERSION DESCRIPTION
eks/appmesh-controller   0.6.0          0.5.0          App Mesh controller Helm chart for Kubernetes
eks/appmesh-grafana     0.1.0            6.4.3        App Mesh Grafana Helm chart for Kubernetes
eks/appmesh-inject      0.13.0           0.5.0        App Mesh Inject Helm chart for Kubernetes
eks/appmesh-jaeger      0.2.0            1.14.0       App Mesh Jaeger Helm chart for Kubernetes
eks/appmesh-prometheus  0.3.0            2.13.1       App Mesh Prometheus Helm chart for Kubernetes

両方のクラスターにApp Mesh ControllerとApp Mesh Injectをセットアップする。

kubectl create ns appmesh-system
kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/appmesh-controller/crds/crds.yaml
helm upgrade -i appmesh-controller eks/appmesh-controller --namespace appmesh-system
helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set mesh.create=true --set mesh.name=global
# X-Rayを有効にする場合は以下も実施する
helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set tracing.enabled=true --set tracing.provider=x-ray
$ kubectl create ns appmesh-system
namespace/appmesh-system created
$ kubectl apply -f https://raw.githubusercontent.com/aws/eks-charts/master/stable/appmesh-controller/crds/crds.yaml
customresourcedefinition.apiextensions.k8s.io/meshes.appmesh.k8s.aws created
customresourcedefinition.apiextensions.k8s.io/virtualnodes.appmesh.k8s.aws created
customresourcedefinition.apiextensions.k8s.io/virtualservices.appmesh.k8s.aws created
$ helm upgrade -i appmesh-controller eks/appmesh-controller --namespace appmesh-system
Release "appmesh-controller" does not exist. Installing it now.
NAME: appmesh-controller
LAST DEPLOYED: Thu May  7 14:34:28 2020
NAMESPACE: appmesh-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS App Mesh controller installed!
$ helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set mesh.create=true --set mesh.name=global
Release "appmesh-inject" does not exist. Installing it now.
NAME: appmesh-inject
LAST DEPLOYED: Thu May  7 14:34:50 2020
NAMESPACE: appmesh-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWS App Mesh Inject installed!
$ helm upgrade -i appmesh-inject eks/appmesh-inject --namespace appmesh-system --set tracing.enabled=true --set tracing.provider=x-ray
Release "appmesh-inject" has been upgraded. Happy Helming!
NAME: appmesh-inject
LAST DEPLOYED: Thu May  7 15:28:20 2020
NAMESPACE: appmesh-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
AWS App Mesh Inject installed!

アプリケーションとApp Mesh CRDの作成

アプリのリポジトリをクローンする。ここからの作業は1つのコンソールでやればOK。

git clone https://github.com/aws/aws-app-mesh-examples.git
cd aws-app-mesh-examples/walkthroughs/howto-k8s-cross-cluster

環境変数を設定する。Envoyイメージについてはここを参照。

export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account)
export AWS_DEFAULT_REGION=$(aws configure get region)
export ENVOY_IMAGE=840364872350.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/aws-appmesh-envoy:v1.12.3.0-prod
export VPC_ID=$(aws ec2 describe-vpcs |
                  jq -r '.Vpcs[] |
                    select( .Tags ) |
                    select( [ select( .Tags[].Value | test("eksctl-eksc2-cluster") ) ] | length > 0 ) |
                    .VpcId')
export CLUSTER1=eksc1
export CLUSTER2=eksc2

ここからは、deploy.shの内容を手動で実行する。

変数をセットする。

DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null && pwd)"
PROJECT_NAME="appmesh-demo"
APP_NAMESPACE=${PROJECT_NAME}
MESH_NAME=${PROJECT_NAME}
CLOUDMAP_NAMESPACE="${PROJECT_NAME}.pvt.aws.local"

ECR_IMAGE_PREFIX="${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com/${PROJECT_NAME}"
FRONT_APP_IMAGE="${ECR_IMAGE_PREFIX}/feapp"
COLOR_APP_IMAGE="${ECR_IMAGE_PREFIX}/colorapp"

Cloud Mapの名前空間を作成する。

aws servicediscovery create-private-dns-namespace \
    --name "${CLOUDMAP_NAMESPACE}" \
    --vpc "${VPC_ID}"

colorappとfeappイメージをビルドしてECRにpushする。

for app in colorapp feapp; do
    aws ecr describe-repositories --repository-name $PROJECT_NAME/$app >/dev/null 2>&1 || aws ecr create-repository --repository-name $PROJECT_NAME/$app
    docker build -t ${ECR_IMAGE_PREFIX}/${app} ${DIR}/${app}
    aws ecr get-login-password | docker login --username AWS --password-stdin https://${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com
    docker push ${ECR_IMAGE_PREFIX}/${app}
done

ekcs1クラスターにアプリとApp Meshリソースをデプロイする。

EXAMPLES_OUT_DIR="${DIR}/_output/"
mkdir -p ${EXAMPLES_OUT_DIR}

eval "cat <<EOF
$(<${DIR}/cluster1.yaml.template)
EOF
" >${EXAMPLES_OUT_DIR}/cluster1.yaml

KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER1}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster1.yaml
$ KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER1}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster1.yaml
namespace/appmesh-demo created
deployment.apps/front created
service/front created

確認する。

$ kubectl get all -n appmesh-demo
NAME                         READY   STATUS    RESTARTS   AGE
pod/front-578b54fc49-kzlhq   3/3     Running   0          98s

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP                                                                    PORT(S)        AGE
service/front   LoadBalancer   10.100.58.100   a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com   80:31145/TCP   98s

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/front   1/1     1            1           98s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/front-578b54fc49   1         1         1       98s

NAME                          AGE
mesh.appmesh.k8s.aws/global   56m
$ kubectl get all -n appmesh-system
NAME                                      READY   STATUS    RESTARTS   AGE
pod/appmesh-controller-54dd6bdfd8-bfhzm   1/1     Running   0          58m
pod/appmesh-inject-54dc557cd6-lkxgz       1/1     Running   0          4m26s

NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/appmesh-inject   ClusterIP   10.100.61.44   <none>        443/TCP   58m

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/appmesh-controller   1/1     1            1           58m
deployment.apps/appmesh-inject       1/1     1            1           58m

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/appmesh-controller-54dd6bdfd8   1         1         1       58m
replicaset.apps/appmesh-inject-54dc557cd6       1         1         1       4m26s
replicaset.apps/appmesh-inject-747db6b88        0         0         0       58m

NAME                          AGE
mesh.appmesh.k8s.aws/global   58m

適用しているのは次のようなYAML。frontコンテナは環境変数で指定されたホスト名とURLにリクエストするが、ここがFQDNなので、次に作成するVirtualServiceでもFQDNを指定する必要があるものと思われる。

App Meshのリソースはどっちのクラスターで作ってもよいものと思われる。

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
  name: appmesh-demo

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: front
  namespace: appmesh-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: front
      version: v1
  template:
    metadata:
      annotations:
        appmesh.k8s.aws/mesh: appmesh-demo
      labels:
        app: front
        version: v1
    spec:
      containers:
        - name: front
          image: XXXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/appmesh-demo/feapp
          ports:
            - containerPort: 8080
          env:
            - name: "PORT"
              value: "80"
            - name: "COLOR_HOST"
              value: "colorapp.appmesh-demo.pvt.aws.local:8080"

---
apiVersion: v1
kind: Service
metadata:
  name: front
  namespace: appmesh-demo
spec:
  type: LoadBalancer
  ports:
  - port: 80
    protocol: TCP
    name: http
  selector:
    app: front

ekcs2クラスターにアプリとApp Meshリソースをデプロイする。

eval "cat <<EOF
$(<${DIR}/cluster2.yaml.template)
EOF
" >${EXAMPLES_OUT_DIR}/cluster2.yaml

KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER2}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster2.yaml
$ KUBECONFIG="$HOME/.kube/eksctl/clusters/${CLUSTER2}" kubectl apply -f ${EXAMPLES_OUT_DIR}/cluster2.yaml
namespace/appmesh-demo created
mesh.appmesh.k8s.aws/appmesh-demo created
virtualnode.appmesh.k8s.aws/front created
virtualnode.appmesh.k8s.aws/colorapp-blue created
virtualnode.appmesh.k8s.aws/colorapp-red created
virtualservice.appmesh.k8s.aws/colorapp.appmesh-demo.pvt.aws.local created
deployment.apps/colorapp-blue created
deployment.apps/colorapp-red created

確認する。

$ kubectl get all -n appmesh-demo
NAME                                 READY   STATUS    RESTARTS   AGE
pod/colorapp-blue-66b44c5c77-q8prf   3/3     Running   0          2m53s
pod/colorapp-red-54cdb4f6bf-jjjf9    3/3     Running   0          2m53s

NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/colorapp-blue   1/1     1            1           2m54s
deployment.apps/colorapp-red    1/1     1            1           2m53s

NAME                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/colorapp-blue-66b44c5c77   1         1         1       2m54s
replicaset.apps/colorapp-red-54cdb4f6bf    1         1         1       2m53s

NAME                                AGE
mesh.appmesh.k8s.aws/appmesh-demo   2m54s
mesh.appmesh.k8s.aws/global         58m

NAME                                        AGE
virtualnode.appmesh.k8s.aws/colorapp-blue   2m54s
virtualnode.appmesh.k8s.aws/colorapp-red    2m54s
virtualnode.appmesh.k8s.aws/front           2m54s

NAME                                                                 AGE
virtualservice.appmesh.k8s.aws/colorapp.appmesh-demo.pvt.aws.local   2m54s
$ kubectl get all -n appmesh-system
NAME                                      READY   STATUS    RESTARTS   AGE
pod/appmesh-controller-54dd6bdfd8-f2drn   1/1     Running   0          58m
pod/appmesh-inject-54dc557cd6-8h5tr       1/1     Running   0          5m10s

NAME                     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/appmesh-inject   ClusterIP   10.100.179.184   <none>        443/TCP   58m

NAME                                 READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/appmesh-controller   1/1     1            1           58m
deployment.apps/appmesh-inject       1/1     1            1           58m

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/appmesh-controller-54dd6bdfd8   1         1         1       58m
replicaset.apps/appmesh-inject-54dc557cd6       1         1         1       5m10s
replicaset.apps/appmesh-inject-747db6b88        0         0         0       58m

NAME                                AGE
mesh.appmesh.k8s.aws/appmesh-demo   3m
mesh.appmesh.k8s.aws/global         58m

適用しているのは次のようなYAML。仮想ノードのService DiscoveryはDNSではなくCloud Mapになっている。 K8s Serviceは作っていない。

---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    appmesh.k8s.aws/sidecarInjectorWebhook: enabled
  name: appmesh-demo

---
apiVersion: appmesh.k8s.aws/v1beta1
kind: Mesh
metadata:
  name: appmesh-demo
---
apiVersion: appmesh.k8s.aws/v1beta1
kind: VirtualNode
metadata:
  name: front
  namespace: appmesh-demo
spec:
  meshName: appmesh-demo
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  serviceDiscovery:
    cloudMap:
      namespaceName: appmesh-demo.pvt.aws.local
      serviceName: front
  backends:
    - virtualService:
        virtualServiceName: colorapp.appmesh-demo.pvt.aws.local
---
apiVersion: appmesh.k8s.aws/v1beta1
kind: VirtualNode
metadata:
  name: colorapp-blue
  namespace: appmesh-demo
spec:
  meshName: appmesh-demo
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  serviceDiscovery:
    cloudMap:
      namespaceName: appmesh-demo.pvt.aws.local
      serviceName: colorapp
---
apiVersion: appmesh.k8s.aws/v1beta1
kind: VirtualNode
metadata:
  name: colorapp-red
  namespace: appmesh-demo
spec:
  meshName: appmesh-demo
  listeners:
    - portMapping:
        port: 8080
        protocol: http
  serviceDiscovery:
    cloudMap:
      namespaceName: appmesh-demo.pvt.aws.local
      serviceName: colorapp
---
apiVersion: appmesh.k8s.aws/v1beta1
kind: VirtualService
metadata:
  name: colorapp.appmesh-demo.pvt.aws.local
  namespace: appmesh-demo
spec:
  meshName: appmesh-demo
  virtualRouter:
    name: colorapp-router
    listeners:
      - portMapping:
          port: 8080
          protocol: http
  routes:
    - name: color-route
      http:
        match:
          prefix: /
        action:
          weightedTargets:
            - virtualNodeName: colorapp-red
              weight: 1
            - virtualNodeName: colorapp-blue
              weight: 1

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: colorapp-blue
  namespace: appmesh-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: colorapp
      version: blue
  template:
    metadata:
      annotations:
        appmesh.k8s.aws/mesh: appmesh-demo
      labels:
        app: colorapp
        version: blue
    spec:
      containers:
        - name: colorapp
          image: XXXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/appmesh-demo/colorapp
          ports:
            - containerPort: 8080
          env:
            - name: "PORT"
              value: "8080"
            - name: "COLOR"
              value: "blue"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: colorapp-red
  namespace: appmesh-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: colorapp
      version: red
  template:
    metadata:
      annotations:
        appmesh.k8s.aws/mesh: appmesh-demo
      labels:
        app: colorapp
        version: red
    spec:
      containers:
        - name: colorapp
          image: XXXXXXXXXXXX.dkr.ecr.ap-northeast-1.amazonaws.com/appmesh-demo/colorapp
          ports:
            - containerPort: 8080
          env:
            - name: "PORT"
              value: "8080"
            - name: "COLOR"
              value: "red"
---

Cloud Mapの確認

Cloud Mapでcolorappのバックエンドのサービスインスタンスを確認する。

$ aws servicediscovery discover-instances --namespace appmesh-demo.pvt.aws.local --service-name colorapp
{
    "Instances": [
        {
            "InstanceId": "172.16.33.210",
            "NamespaceName": "appmesh-demo.pvt.aws.local",
            "ServiceName": "colorapp",
            "HealthStatus": "UNKNOWN",
            "Attributes": {
                "AWS_INSTANCE_IPV4": "172.16.33.210",
                "app": "colorapp",
                "appmesh.k8s.aws/mesh": "appmesh-demo",
                "appmesh.k8s.aws/virtualNode": "colorapp-blue-appmesh-demo",
                "k8s.io/namespace": "appmesh-demo",
                "k8s.io/pod": "colorapp-blue-66b44c5c77-q8prf",
                "pod-template-hash": "66b44c5c77",
                "version": "blue"
            }
        },
        {
            "InstanceId": "172.16.57.230",
            "NamespaceName": "appmesh-demo.pvt.aws.local",
            "ServiceName": "colorapp",
            "HealthStatus": "UNKNOWN",
            "Attributes": {
                "AWS_INSTANCE_IPV4": "172.16.57.230",
                "app": "colorapp",
                "appmesh.k8s.aws/mesh": "appmesh-demo",
                "appmesh.k8s.aws/virtualNode": "colorapp-red-appmesh-demo",
                "k8s.io/namespace": "appmesh-demo",
                "k8s.io/pod": "colorapp-red-54cdb4f6bf-jjjf9",
                "pod-template-hash": "54cdb4f6bf",
                "version": "red"
            }
        }
    ]
}

これを誰が登録しているのかの挙動がよくわからない。 VirtualNodeを消すとエントリが消えるので、VirtualNodeの属性に基づいて登録されているように思えるが、VirtualNodeを定義しただけでは追加されないのでVirtualServiceも必要そうに思える。

このFQDNがeksc1で名前解決できることを確認する。

$ kubectl run busybox --image=busybox:1.28 --rm -it --restart=Never --command -- nslookup colorapp.appmesh-demo.pvt.aws.local
Server:    10.100.0.10
Address 1: 10.100.0.10 kube-dns.kube-system.svc.cluster.local

Name:      colorapp.appmesh-demo.pvt.aws.local
Address 1: 172.16.88.89 ip-172-16-88-89.ap-northeast-1.compute.internal
Address 2: 172.16.37.132 ip-172-16-37-132.ap-northeast-1.compute.internal
pod "busybox" deleted

アプリケーションのテスト

eksc1で外向けServiceのホスト名を確認する。

$ kubectl get svc -n appmesh-demo
NAME    TYPE           CLUSTER-IP      EXTERNAL-IP                                                                    PORT(S)        AGE
front   LoadBalancer   10.100.58.100   a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com   80:31145/TCP   58m

このホストの/colorにアクセスする。

$ curl -w "\n" http://a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com/color
red
$ curl -w "\n" http://a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com/color
red
$ curl -w "\n" http://a2d9a6f3033d749018bad44c9570621a-1456827507.ap-northeast-1.elb.amazonaws.com/color
blue

適当にリクエストを送ってX-Rayコンソールを見てみる。アプリはX-Ray SDKが入っていて、Traceもちゃんと見える。

f:id:sotoiwa:20200507174519p:plain

f:id:sotoiwa:20200507174540p:plain