Knative Servingを試す

Knative ServingをEKSで動かしてみたメモ。

コンポーネント	バージョン	備考
eksctl	0.17.0
Kubernetes バージョン	1.15
プラットフォームのバージョン	eks.2
Istio	1.3.6
Knative Serving	v0.14.0

クラスターの作成

クラスターを作成する。Knative Serving v0.14はKubernetes v1.15が必要。

eksctl create cluster --name=knative --version 1.15 --nodes=3 --managed --ssh-access --ssh-public-key=sotosugi

Knative Servingのセットアップ

以下のドキュメントにしたがって実施。

• Installing Knative

Knative ServingのCRDをインストールする。

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.14.0/serving-crds.yaml

いくつかのCRDが作成される。

$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.14.0/serving-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev created
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev created
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev created
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev created
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev created
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev created

Knative Serviceのコアコンポーネントをインストールする。

kubectl apply --filename https://github.com/knative/serving/releases/download/v0.14.0/serving-core.yaml

HPAを作っているが、デフォルトのEKSでは使えないので使えるようにセットアップしてあげる必要があるかもしれない。

$ kubectl apply --filename https://github.com/knative/serving/releases/download/v0.14.0/serving-core.yaml
customresourcedefinition.apiextensions.k8s.io/images.caching.internal.knative.dev unchanged
namespace/knative-serving created
serviceaccount/controller created
clusterrole.rbac.authorization.k8s.io/knative-serving-admin created
clusterrolebinding.rbac.authorization.k8s.io/knative-serving-controller-admin created
image.caching.internal.knative.dev/queue-proxy created
configmap/config-autoscaler created
configmap/config-defaults created
configmap/config-deployment created
configmap/config-domain created
configmap/config-gc created
configmap/config-leader-election created
configmap/config-logging created
configmap/config-network created
configmap/config-observability created
configmap/config-tracing created
horizontalpodautoscaler.autoscaling/activator created
deployment.apps/activator created
service/activator-service created
deployment.apps/autoscaler created
service/autoscaler created
deployment.apps/controller created
service/controller created
deployment.apps/webhook created
service/webhook created
customresourcedefinition.apiextensions.k8s.io/certificates.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/configurations.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/ingresses.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/metrics.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/podautoscalers.autoscaling.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/revisions.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/routes.serving.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/serverlessservices.networking.internal.knative.dev unchanged
customresourcedefinition.apiextensions.k8s.io/services.serving.knative.dev unchanged
clusterrole.rbac.authorization.k8s.io/knative-serving-addressable-resolver created
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-admin created
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-edit created
clusterrole.rbac.authorization.k8s.io/knative-serving-namespaced-view created
clusterrole.rbac.authorization.k8s.io/knative-serving-core created
clusterrole.rbac.authorization.k8s.io/knative-serving-podspecable-binding created
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.serving.knative.dev created
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.serving.knative.dev created
validatingwebhookconfiguration.admissionregistration.k8s.io/validation.webhook.serving.knative.dev created
secret/webhook-certs created

Podは以下が動いている。

$ kubectl get pod -n knative-serving
NAME                          READY   STATUS    RESTARTS   AGE
activator-8cb6d456-49rg5      1/1     Running   0          2m6s
autoscaler-dd459ddbb-vcv9f    1/1     Running   0          2m6s
controller-8564567c4c-9xdwk   1/1     Running   0          2m6s
webhook-7fbf9c6d49-qvdj2      1/1     Running   0          2m5s

ネットワークレイヤーは複数の選択肢があるが今回はIstioをインストールする。

• Ambassador
• Contour
• Gloo
• Istio
• Kourier

Istioのの場合もIstio LeanというPilotだけを使う方法と、Istio with Service Meshとがあり、Istio Leanのほうが推奨となっている。

• Installing Istio for Knative

Istioをダウンロードする。

export ISTIO_VERSION=1.4.6
curl -L https://git.io/getLatestIstio | sh -
cd istio-${ISTIO_VERSION}

CRDをインストールする。

for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done

$ for i in install/kubernetes/helm/istio-init/files/crd*yaml; do kubectl apply -f $i; done
customresourcedefinition.apiextensions.k8s.io/attributemanifests.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/clusterrbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/httpapispecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/meshpolicies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/policies.authentication.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecbindings.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/quotaspecs.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/rbacconfigs.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/rules.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/servicerolebindings.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/serviceroles.rbac.istio.io created
customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/adapters.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/instances.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/templates.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/handlers.config.istio.io created
customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io created
customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.security.istio.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created

istio-system Namespacesを作成する。

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
  name: istio-system
  labels:
    istio-injection: disabled
EOF

Istio Leanでインストールする。helm templateを使う。コメントに記載の通り、pilotとgatewayのみをインストールしている。

# A lighter template, with just pilot/gateway.
# Based on install/kubernetes/helm/istio/values-istio-minimal.yaml
helm template --namespace=istio-system \
  --set prometheus.enabled=false \
  --set mixer.enabled=false \
  --set mixer.policy.enabled=false \
  --set mixer.telemetry.enabled=false \
  `# Pilot doesn't need a sidecar.` \
  --set pilot.sidecar=false \
  --set pilot.resources.requests.memory=128Mi \
  `# Disable galley (and things requiring galley).` \
  --set galley.enabled=false \
  --set global.useMCP=false \
  `# Disable security / policy.` \
  --set security.enabled=false \
  --set global.disablePolicyChecks=true \
  `# Disable sidecar injection.` \
  --set sidecarInjectorWebhook.enabled=false \
  --set global.proxy.autoInject=disabled \
  --set global.omitSidecarInjectorConfigMap=true \
  --set gateways.istio-ingressgateway.autoscaleMin=1 \
  --set gateways.istio-ingressgateway.autoscaleMax=2 \
  `# Set pilot trace sampling to 100%` \
  --set pilot.traceSampling=100 \
  --set global.mtls.auto=false \
  install/kubernetes/helm/istio \
  > ./istio-lean.yaml

kubectl apply -f istio-lean.yaml

$ kubectl apply -f istio-lean.yaml
poddisruptionbudget.policy/istio-ingressgateway created
poddisruptionbudget.policy/istio-pilot created
serviceaccount/istio-ingressgateway-service-account created
serviceaccount/istio-pilot-service-account created
serviceaccount/istio-multi created
configmap/istio created
clusterrole.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrole.rbac.authorization.k8s.io/istio-reader created
clusterrolebinding.rbac.authorization.k8s.io/istio-pilot-istio-system created
clusterrolebinding.rbac.authorization.k8s.io/istio-multi created
role.rbac.authorization.k8s.io/istio-ingressgateway-sds created
rolebinding.rbac.authorization.k8s.io/istio-ingressgateway-sds created
service/istio-ingressgateway created
service/istio-pilot created
deployment.apps/istio-ingressgateway created
deployment.apps/istio-pilot created
horizontalpodautoscaler.autoscaling/istio-ingressgateway created
horizontalpodautoscaler.autoscaling/istio-pilot created

クラスター内のみにServiceを公開できるようにするにはCluster Local Gatewayのセットアップも必要。

# Add the extra gateway.
helm template --namespace=istio-system \
  --set gateways.custom-gateway.autoscaleMin=1 \
  --set gateways.custom-gateway.autoscaleMax=2 \
  --set gateways.custom-gateway.cpu.targetAverageUtilization=60 \
  --set gateways.custom-gateway.labels.app='cluster-local-gateway' \
  --set gateways.custom-gateway.labels.istio='cluster-local-gateway' \
  --set gateways.custom-gateway.type='ClusterIP' \
  --set gateways.istio-ingressgateway.enabled=false \
  --set gateways.istio-egressgateway.enabled=false \
  --set gateways.istio-ilbgateway.enabled=false \
  --set global.mtls.auto=false \
  install/kubernetes/helm/istio \
  -f install/kubernetes/helm/istio/example-values/values-istio-gateways.yaml \
  | sed -e "s/custom-gateway/cluster-local-gateway/g" -e "s/customgateway/clusterlocalgateway/g" \
  > ./istio-local-gateway.yaml

kubectl apply -f istio-local-gateway.yaml

$ kubectl apply -f istio-local-gateway.yaml
poddisruptionbudget.policy/cluster-local-gateway created
serviceaccount/cluster-local-gateway-service-account created
serviceaccount/istio-multi unchanged
clusterrole.rbac.authorization.k8s.io/istio-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/istio-multi unchanged
service/cluster-local-gateway created
deployment.apps/cluster-local-gateway created

Podを確認する。

$ kubectl get pod --namespace istio-system
NAME                                     READY   STATUS    RESTARTS   AGE
cluster-local-gateway-6d6d8b7c89-r7br6   1/1     Running   0          20s
istio-ingressgateway-c6978c57b-44656     1/1     Running   0          3m15s
istio-pilot-5bdb6c9ddf-k7c5z             1/1     Running   0          3m15s

istio-ingressgatewayを確認する。

$ kubectl get svc istio-ingressgateway -n istio-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP                                                                   PORT(S)                                                                                                                                      AGE
istio-ingressgateway   LoadBalancer   10.100.194.51   a835103f01248419189973f3fa3c4230-396417625.ap-northeast-1.elb.amazonaws.com   15020:31137/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31767/TCP,15030:32323/TCP,15031:30873/TCP,15032:30897/TCP,15443:30243/TCP   3m30s

DNS設定をカスタマイズする。<ksvc名>.<namespace名>.<ここで設定したドメイン名>でサービスが公開される。Route53でワールドカードドメインを作成してCNAMEをELBに向けてあげる必要があるが今回は省略。

kubectl patch configmap/config-domain \
  --namespace knative-serving \
  --type merge \
  --patch '{"data":{"knative.example.com":""}}'

Knative Istio Controllerを作成する。

kubectl apply --filename https://github.com/knative/net-istio/releases/download/v0.14.0/release.yaml

Gatewayリソースなどを作っている。

$ kubectl apply --filename https://github.com/knative/net-istio/releases/download/v0.14.0/release.yaml
clusterrole.rbac.authorization.k8s.io/knative-serving-istio created
gateway.networking.istio.io/knative-ingress-gateway created
gateway.networking.istio.io/cluster-local-gateway created
mutatingwebhookconfiguration.admissionregistration.k8s.io/webhook.istio.networking.internal.knative.dev created
validatingwebhookconfiguration.admissionregistration.k8s.io/config.webhook.istio.networking.internal.knative.dev created
secret/istio-webhook-certs created
configmap/config-istio created
deployment.apps/networking-istio created
deployment.apps/istio-webhook created
service/istio-webhook created

Knative Serviceを試す

• Getting Started with App Deployment

helloworld

helloworldをデプロイする。

cat <<EOF > service.yaml
apiVersion: serving.knative.dev/v1 # Current version of Knative
kind: Service
metadata:
  name: helloworld-go # The name of the app
  namespace: default # The namespace the app will use
spec:
  template:
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go # The URL to the image of the app
          env:
            - name: TARGET # The environment variable printed out by the sample app
              value: "Go Sample v1"
EOF
kubectl apply --filename service.yaml

作成したKnaticve Service確認する。

$ kubectl get ksvc -n default
NAME            URL                                                LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.default.knative.example.com   helloworld-go-gdfx4   helloworld-go-gdfx4   True

Deploymentを確認する。

$ kubectl get deploy -n default
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
helloworld-go-gdfx4-deployment   1/1     1            1           46s

しばらく待つとゼロにスケールダウンする。

$ kubectl get deploy -n default
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
helloworld-go-gdfx4-deployment   0/0     0            0           99s

curlでアクセスする。

$ curl -H "Host: helloworld-go.default.knative.example.com" a835103f01248419189973f3fa3c4230-396417625.ap-northeast-1.elb.amazonaws.com
Hello Go Sample v1!

ゼロからスケールしていることを確認。

$ kubectl get deploy -n default
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
helloworld-go-gdfx4-deployment   1/1     1            1           3m5s

Knative Servingのリソースの関連性

Knative Servingのリソースは4つ。

リソース	説明
Service	RouteとConfigurationを管理する。
Route	トラフィックを流すConfiguraionとそのRevisionが指定される。
Configuration	Revisionを管理する。
Revision	Configurationで定義されたTemplateをもとに作成される。

Knative Serviceを作るとRouteとConfigurationが作られ、Configurationが作られると、Revisionが作られる。

$ kubectl get routes -n default
NAME            URL                                                READY   REASON
helloworld-go   http://helloworld-go.default.knative.example.com   True
$ kubectl get configuration -n default
NAME            LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   helloworld-go-gdfx4   helloworld-go-gdfx4   True
$ kubectl get revision -n default
NAME                  CONFIG NAME     K8S SERVICE NAME      GENERATION   READY   REASON
helloworld-go-gdfx4   helloworld-go   helloworld-go-gdfx4   1            True

Routeの内容の抜粋。

apiVersion: serving.knative.dev/v1
kind: Route
metadata:
  name: helloworld-go
  namespace: default
spec:
  traffic:
  - configurationName: helloworld-go
    latestRevision: true
    percent: 100

Configurationの内容の抜粋。

apiVersion: serving.knative.dev/v1
kind: Configuration
metadata:
  name: helloworld-go
  namespace: default
spec:
  template:
    spec:
      containers:
      - env:
        - name: TARGET
          value: Go Sample v1
        image: gcr.io/knative-samples/helloworld-go

Revisionの内容の抜粋。

apiVersion: serving.knative.dev/v1
kind: Revision
metadata:
  name: helloworld-go-gdfx4
  namespace: default
spec:
  containers:
  - env:
    - name: TARGET
      value: Go Sample v1
    image: gcr.io/knative-samples/helloworld-go
    name: user-container

Knative Servingリソースの設定を更新する。

cat <<EOF > service-v2.yaml
apiVersion: serving.knative.dev/v1 # Current version of Knative
kind: Service
metadata:
  name: helloworld-go # The name of the app
  namespace: default # The namespace the app will use
spec:
  template:
    spec:
      containers:
        - image: gcr.io/knative-samples/helloworld-go # The URL to the image of the app
          env:
            - name: TARGET # The environment variable printed out by the sample app
              value: "Go Sample v2"
EOF
kubectl apply --filename service-v2.yaml

確認する。

$ kubectl get ksvc -n default
NAME            URL                                                LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   http://helloworld-go.default.knative.example.com   helloworld-go-n42vj   helloworld-go-n42vj   True
$ kubectl get routes -n default
NAME            URL                                                READY   REASON
helloworld-go   http://helloworld-go.default.knative.example.com   True
$ kubectl get configuration -n default
NAME            LATESTCREATED         LATESTREADY           READY   REASON
helloworld-go   helloworld-go-n42vj   helloworld-go-n42vj   True
$ kubectl get revision -n default
NAME                  CONFIG NAME     K8S SERVICE NAME      GENERATION   READY   REASON
helloworld-go-n42vj   helloworld-go   helloworld-go-n42vj   2            True
helloworld-go-p689m   helloworld-go   helloworld-go-p689m   1            True

DeploymentとK8s Service名はRevision毎に作成される。

$ kubectl get deploy -n default
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
helloworld-go-n42vj-deployment   0/0     0            0           5m12s
helloworld-go-p689m-deployment   0/0     0            0           7m41s
$ kubectl get svc -n default
NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP                                            PORT(S)                             AGE
helloworld-go                 ExternalName   <none>           cluster-local-gateway.istio-system.svc.cluster.local   <none>                              2d5h
helloworld-go-n42vj           ClusterIP      10.100.120.49    <none>                                                 80/TCP                              5m24s
helloworld-go-n42vj-private   ClusterIP      10.100.248.152   <none>                                                 80/TCP,9090/TCP,9091/TCP,8022/TCP   5m24s
helloworld-go-p689m           ClusterIP      10.100.221.120   <none>                                                 80/TCP                              7m53s
helloworld-go-p689m-private   ClusterIP      10.100.33.82     <none>                                                 80/TCP,9090/TCP,9091/TCP,8022/TCP   7m53s
kubernetes                    ClusterIP      10.100.0.1       <none>                                                 443/TCP                             2d6h

メモ

以下によればIstio 1.5.1でも動かせそう。

• Where to check the supported versions of Kubernetes and Istio? #6024
• https://github.com/knative/serving/tree/master/third_party

参考資料

• Knative Serving in Production
• Zero Scale Abstraction in Knative Serving