MinIO を試す。EKS 用の手順もあるようだが、Upstream と書かれている手順を試す。
クラスターの作成
CLUSTER_NAME="minio" MY_ARN=$(aws sts get-caller-identity --output text --query Arn) AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) cat << EOF > cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: ${CLUSTER_NAME} region: ap-northeast-1 version: "1.29" vpc: cidr: "10.0.0.0/16" availabilityZones: - ap-northeast-1a - ap-northeast-1c cloudWatch: clusterLogging: enableTypes: ["*"] iam: withOIDC: true accessConfig: bootstrapClusterCreatorAdminPermissions: false authenticationMode: API accessEntries: - principalARN: arn:aws:iam::${AWS_ACCOUNT_ID}:role/Admin accessPolicies: - policyARN: arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy accessScope: type: cluster EOF
eksctl create cluster -f cluster.yaml
ノードを作成する。
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) cat << EOF > m1.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: ${CLUSTER_NAME} region: ap-northeast-1 managedNodeGroups: - name: m1 minSize: 3 maxSize: 3 desiredCapacity: 3 privateNetworking: true iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore EOF
eksctl create nodegroup -f m1.yaml
ノードを確認する。
$ k get node NAME STATUS ROLES AGE VERSION ip-10-0-105-238.ap-northeast-1.compute.internal Ready <none> 24m v1.29.0-eks-5e0fdde ip-10-0-117-206.ap-northeast-1.compute.internal Ready <none> 24m v1.29.0-eks-5e0fdde ip-10-0-86-177.ap-northeast-1.compute.internal Ready <none> 24m v1.29.0-eks-5e0fdde
Pod を確認する。
$ k get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-cxt7p 2/2 Running 0 25m kube-system aws-node-dp2p4 2/2 Running 0 25m kube-system aws-node-vmj54 2/2 Running 0 25m kube-system coredns-676bf68468-bqbnp 1/1 Running 0 37m kube-system coredns-676bf68468-g846n 1/1 Running 0 37m kube-system kube-proxy-hkj5h 1/1 Running 0 25m kube-system kube-proxy-lms44 1/1 Running 0 25m kube-system kube-proxy-t87sp 1/1 Running 0 25m
MinIO のインストール
まずはデフォルト設定で入れてみる。MinIO Operator をデプロイする。
$ kubectl minio init # Warning: 'patchesJson6902' is deprecated. Please use 'patches' instead. Run 'kustomize edit fix' to update your Kustomization automatically. namespace/minio-operator created serviceaccount/minio-operator created clusterrole.rbac.authorization.k8s.io/minio-operator-role created clusterrolebinding.rbac.authorization.k8s.io/minio-operator-binding created customresourcedefinition.apiextensions.k8s.io/tenants.minio.min.io created customresourcedefinition.apiextensions.k8s.io/policybindings.sts.min.io created customresourcedefinition.apiextensions.k8s.io/miniojobs.job.min.io created service/operator created service/sts created deployment.apps/minio-operator created serviceaccount/console-sa created secret/console-sa-secret created clusterrole.rbac.authorization.k8s.io/console-sa-role created clusterrolebinding.rbac.authorization.k8s.io/console-sa-binding created configmap/console-env created service/console created deployment.apps/console created ----------------- To open Operator UI, start a port forward using this command: kubectl minio proxy -n minio-operator -----------------
Pod を確認する。
$ k get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-cxt7p 2/2 Running 0 28m kube-system aws-node-dp2p4 2/2 Running 0 28m kube-system aws-node-vmj54 2/2 Running 0 28m kube-system coredns-676bf68468-bqbnp 1/1 Running 0 40m kube-system coredns-676bf68468-g846n 1/1 Running 0 40m kube-system kube-proxy-hkj5h 1/1 Running 0 28m kube-system kube-proxy-lms44 1/1 Running 0 28m kube-system kube-proxy-t87sp 1/1 Running 0 28m minio-operator console-86878b559f-tkzts 1/1 Running 0 22s minio-operator minio-operator-54bf877d58-7rbx9 1/1 Running 0 22s minio-operator minio-operator-54bf877d58-mx64t 1/1 Running 0 22s
Operator コンソールにアクセスする。
$ kubectl minio proxy Starting port forward of the Console UI. To connect open a browser and go to http://localhost:9090 Current JWT to login: eyJhbGciOiJSUzI1NiIsImtpZCI6IlljTERycFVtNS1FdTBpMXBiYkZ0RDUyUUZIT1Fwdlk5MmtTTFI3bzlSY00ifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtaW5pby1vcGVyYXRvciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjb25zb2xlLXNhLXNlY3JldCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjb25zb2xlLXNhIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNWMwODM0NjEtNzk3Yy00ZDc4LTlkZDgtNjUxNmYzYjJmNzU5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Om1pbmlvLW9wZXJhdG9yOmNvbnNvbGUtc2EifQ.esXqD_rdGqn8cnca2_Rr83anD1TWQiUY8J0o4_JEh7Kk-vLaTFo83l1wisrBhooNWxqFCo-5Ypc0MRG7lMHoRLo8Zq4mcPCQN1uElnlNLalwZtgkmcu2khaV6SmonNyr0i7tw7mXXfx6VOiM6fSFQZMoK0YwXZx1Dso_TWnZo1eWPmuxGfzpSkUCvIdqgtAq0N7a2YYxmf9yvgCySTreNQEN1xVt6G2lo__KVN2F0E0PBJJDofnLeRNz3u2hBtHYFgFgSbnWCjRMscxXT8s83JtoqTFbPck-ZVXKh8nWqXaRqY7rIoZq0lrTYz5piALpW-xlgdFLtjaw-dHDhycbcw Forwarding from 0.0.0.0:9090 -> 9090
EBS CSI Driver の導入
AWS_ACCOUNT_ID=$(aws sts get-caller-identity --output text --query Account) cat << EOF > addon.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: ${CLUSTER_NAME} region: ap-northeast-1 addons: - name: vpc-cni version: latest attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy # serviceAccountRoleARN: arn:aws:iam::XXXXXXXXXXXX:role/eksctl-fully-private-addon-iamserviceaccount-Role1-LRQ0AZXOE60K configurationValues: |- env: WARM_IP_TARGET: "2" MINIMUM_IP_TARGET: "10" resolveConflicts: overwrite - name: coredns version: latest - name: kube-proxy version: latest - name: aws-ebs-csi-driver version: latest wellKnownPolicies: ebsCSIController: true EOF
eksctl create addon -f addon.yaml
$ k get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-8n8mq 2/2 Running 0 4m39s kube-system aws-node-cf5b5 2/2 Running 0 5m25s kube-system aws-node-dtvlm 2/2 Running 0 5m2s kube-system coredns-5877997cb7-4hxql 1/1 Running 0 2m30s kube-system coredns-5877997cb7-8nf5z 1/1 Running 0 2m29s kube-system ebs-csi-controller-7cddb57f8d-9xrn2 5/6 Running 0 12s kube-system ebs-csi-controller-7cddb57f8d-hjk2w 5/6 Running 0 12s kube-system ebs-csi-node-btcwj 3/3 Running 0 12s kube-system ebs-csi-node-cjnl7 3/3 Running 0 12s kube-system ebs-csi-node-qd4tm 3/3 Running 0 12s kube-system kube-proxy-2xm44 1/1 Running 0 2m29s kube-system kube-proxy-k9fhs 1/1 Running 0 2m26s kube-system kube-proxy-x4j2c 1/1 Running 0 2m23s minio-operator console-86878b559f-tkzts 1/1 Running 0 20m minio-operator minio-operator-54bf877d58-7rbx9 1/1 Running 0 20m minio-operator minio-operator-54bf877d58-mx64t 1/1 Running 0 20m
$ k get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 (default) kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 61m
テナントの作成
Operator コンソールでもできそうだが、ここでは kubectl で実施する。
Tenant のマニフェストを生成してみる。
$ kubectl minio tenant create minio1 \ --capacity 16Gi \ --servers 4 \ --volumes 8 \ --namespace minio-tenant-1 \ --storage-class gp2 \ --output apiVersion: minio.min.io/v2 kind: Tenant metadata: creationTimestamp: null name: minio1 namespace: minio-tenant-1 scheduler: name: "" spec: certConfig: commonName: '*.minio1-hl.minio-tenant-1.svc.cluster.local' dnsNames: - minio1-ss-0-{0...3}.minio1-hl.minio-tenant-1.svc.cluster.local organizationName: - system:nodes configuration: name: minio1-env-configuration exposeServices: {} features: enableSFTP: false image: minio/minio:RELEASE.2024-02-09T21-25-16Z imagePullPolicy: IfNotPresent imagePullSecret: {} mountPath: /export podManagementPolicy: Parallel pools: - affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: v1.min.io/tenant operator: In values: - minio1 - key: v1.min.io/pool operator: In values: - "" topologyKey: kubernetes.io/hostname name: ss-0 resources: {} servers: 4 volumeClaimTemplate: apiVersion: v1 kind: persistentvolumeclaims metadata: creationTimestamp: null spec: accessModes: - ReadWriteOnce resources: requests: storage: 2Gi storageClassName: gp2 status: {} volumesPerServer: 2 requestAutoCert: true serviceAccountName: minio1-sa users: - name: minio1-user-1 status: availableReplicas: 0 certificates: {} currentState: "" pools: null revision: 0 syncVersion: "" usage: {} --- apiVersion: v1 data: config.env: ZXhwb3J0IE1JTklPX1JPT1RfUEFTU1dPUkQ9IklCU0hiN1JLQ1ZOYWpzOHo2VEt0bWNlZmppdzg4Y3JseEVhZm44anAiCmV4cG9ydCBNSU5JT19ST09UX1VTRVI9IjJDQk9aRVZRWlkyOFdBU0tTOVdMIgo= kind: Secret metadata: creationTimestamp: null name: minio1-env-configuration namespace: minio-tenant-1 --- apiVersion: v1 data: CONSOLE_ACCESS_KEY: NVBOMjQxUTNFTUU0WFNBUTNZWFE= CONSOLE_SECRET_KEY: bkVqa3RsWEFJYlJyQXZvT3dlMFVMQ003eHVWNEliRFowRVI3QjFObg== kind: Secret metadata: creationTimestamp: null name: minio1-user-1 namespace: minio-tenant-1
apply する。
$ k create ns minio-tenant-1 namespace/minio-tenant-1 created $ kubectl minio tenant create minio1 \ --capacity 16Gi \ --servers 4 \ --volumes 8 \ --namespace minio-tenant-1 \ --storage-class gp2 W0308 18:58:11.797099 36063 warnings.go:70] unknown field "spec.pools[0].volumeClaimTemplate.metadata.creationTimestamp" Tenant 'minio1' created in 'minio-tenant-1' Namespace Username: ET3U6W5UKQXFG7FDY1Q0 Password: AAnut9ff6x6jwABk4qSsF0nHJpvChKo8TH99ZUaE Note: Copy the credentials to a secure location. MinIO will not display these again. APPLICATION SERVICE NAME NAMESPACE SERVICE TYPE SERVICE PORT MinIO minio minio-tenant-1 ClusterIP 443 Console minio1-console minio-tenant-1 ClusterIP 9443
Pod を確認する。
$ k get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-8n8mq 2/2 Running 2 (5d7h ago) 8d kube-system aws-node-cf5b5 2/2 Running 2 (5d7h ago) 8d kube-system aws-node-dtvlm 2/2 Running 2 (5d7h ago) 8d kube-system coredns-5877997cb7-4hxql 1/1 Running 1 (5d7h ago) 8d kube-system coredns-5877997cb7-8nf5z 1/1 Running 1 (5d7h ago) 8d kube-system ebs-csi-controller-7cddb57f8d-9xrn2 6/6 Running 6 (5d7h ago) 8d kube-system ebs-csi-controller-7cddb57f8d-hjk2w 6/6 Running 6 (5d7h ago) 8d kube-system ebs-csi-node-btcwj 3/3 Running 3 (5d7h ago) 8d kube-system ebs-csi-node-cjnl7 3/3 Running 3 (5d7h ago) 8d kube-system ebs-csi-node-qd4tm 3/3 Running 3 (5d7h ago) 8d kube-system kube-proxy-2xm44 1/1 Running 1 (5d7h ago) 8d kube-system kube-proxy-k9fhs 1/1 Running 1 (5d7h ago) 8d kube-system kube-proxy-x4j2c 1/1 Running 1 (5d7h ago) 8d minio-operator console-86878b559f-jxxvg 1/1 Running 0 5m1s minio-operator minio-operator-54bf877d58-8jrvl 1/1 Running 0 5m1s minio-operator minio-operator-54bf877d58-fm8t7 1/1 Running 0 5m1s minio-tenant-1 minio1-ss-0-0 2/2 Running 0 2m3s minio-tenant-1 minio1-ss-0-1 2/2 Running 0 2m2s minio-tenant-1 minio1-ss-0-2 2/2 Running 0 2m2s minio-tenant-1 minio1-ss-0-3 2/2 Running 0 2m2s
ポートフォワードしてコンソールにアクセスする。
$ k -n minio-tenant-1 get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE minio ClusterIP 172.20.1.251 <none> 443/TCP 2m34s minio1-console ClusterIP 172.20.15.213 <none> 9443/TCP 2m34s minio1-hl ClusterIP None <none> 9000/TCP 2m33s
$ k -n minio-tenant-1 port-forward svc/minio1-console 9443:9443 Forwarding from 127.0.0.1:9443 -> 9443 Forwarding from [::1]:9443 -> 9443
PV/PVC を確認する。
$ k get pv,pvc -A NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-1ebcb2c9-051c-4800-8974-e0664ac25fa3 2Gi RWO Delete Bound minio-tenant-1/1-minio1-ss-0-3 gp2 <unset> 4m30s persistentvolume/pvc-2c60e4b0-3473-44ef-8bc7-a30043f5efcf 2Gi RWO Delete Bound minio-tenant-1/1-minio1-ss-0-0 gp2 <unset> 4m30s persistentvolume/pvc-2d7e0cd1-d9e6-4ea5-8ab3-00272dc54350 2Gi RWO Delete Bound minio-tenant-1/0-minio1-ss-0-0 gp2 <unset> 4m30s persistentvolume/pvc-535a3c5c-f7a8-4fe6-b7b2-ec2256e312d2 2Gi RWO Delete Bound minio-tenant-1/0-minio1-ss-0-2 gp2 <unset> 4m30s persistentvolume/pvc-5aa0edf5-423f-431e-8547-7bc01a00ca25 2Gi RWO Delete Bound minio-tenant-1/1-minio1-ss-0-2 gp2 <unset> 4m30s persistentvolume/pvc-84e16280-bb09-4cb3-a64e-1de7f4f8b469 2Gi RWO Delete Bound minio-tenant-1/0-minio1-ss-0-1 gp2 <unset> 4m30s persistentvolume/pvc-b19e0010-0f93-4dbe-bdc3-3a22be6795e8 2Gi RWO Delete Bound minio-tenant-1/0-minio1-ss-0-3 gp2 <unset> 4m30s persistentvolume/pvc-bc73c448-5ed2-4740-bc85-be7481126b0e 2Gi RWO Delete Bound minio-tenant-1/1-minio1-ss-0-1 gp2 <unset> 4m30s NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE minio-tenant-1 persistentvolumeclaim/0-minio1-ss-0-0 Bound pvc-2d7e0cd1-d9e6-4ea5-8ab3-00272dc54350 2Gi RWO gp2 <unset> 4m35s minio-tenant-1 persistentvolumeclaim/0-minio1-ss-0-1 Bound pvc-84e16280-bb09-4cb3-a64e-1de7f4f8b469 2Gi RWO gp2 <unset> 4m35s minio-tenant-1 persistentvolumeclaim/0-minio1-ss-0-2 Bound pvc-535a3c5c-f7a8-4fe6-b7b2-ec2256e312d2 2Gi RWO gp2 <unset> 4m35s minio-tenant-1 persistentvolumeclaim/0-minio1-ss-0-3 Bound pvc-b19e0010-0f93-4dbe-bdc3-3a22be6795e8 2Gi RWO gp2 <unset> 4m34s minio-tenant-1 persistentvolumeclaim/1-minio1-ss-0-0 Bound pvc-2c60e4b0-3473-44ef-8bc7-a30043f5efcf 2Gi RWO gp2 <unset> 4m35s minio-tenant-1 persistentvolumeclaim/1-minio1-ss-0-1 Bound pvc-bc73c448-5ed2-4740-bc85-be7481126b0e 2Gi RWO gp2 <unset> 4m35s minio-tenant-1 persistentvolumeclaim/1-minio1-ss-0-2 Bound pvc-5aa0edf5-423f-431e-8547-7bc01a00ca25 2Gi RWO gp2 <unset> 4m35s minio-tenant-1 persistentvolumeclaim/1-minio1-ss-0-3 Bound pvc-1ebcb2c9-051c-4800-8974-e0664ac25fa3 2Gi RWO gp2 <unset> 4m34s
AWS CLI を使ったアクセス
AWS CLI にクレデンシャルを設定する。ここでは、上述のコンソールユーザーのクレデンシャルを使用する。
$ aws configure --profile minio AWS Access Key ID [None]: ET3U6W5UKQXFG7FDY1Q0 AWS Secret Access Key [None]: AAnut9ff6x6jwABk4qSsF0nHJpvChKo8TH99ZUaE Default region name [None]: ap-northeast-1 Default output format [None]:
署名バージョンを指定する。
aws configure set s3.signature_version s3v4 --profile minio
.aws/config は以下のようになる。
[profile minio]
region = ap-northeast-1
s3 =
signature_version = s3v4
別のターミナルでポートフォワードしておく。
$ k -n minio-tenant-1 port-forward svc/minio1-hl 9000:9000 Forwarding from 127.0.0.1:9000 -> 9000 Forwarding from [::1]:9000 -> 9000
プロファイルを指定する。
export AWS_PROFILE=minio
$ aws --no-verify-ssl --endpoint-url https://localhost:9000 s3 ls /opt/homebrew/Cellar/awscli/2.15.28/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( $ aws --no-verify-ssl --endpoint-url https://localhost:9000 s3 mb s3://hoge-bucket /opt/homebrew/Cellar/awscli/2.15.28/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( make_bucket: hoge-bucket $ aws --no-verify-ssl --endpoint-url https://localhost:9000 s3 ls /opt/homebrew/Cellar/awscli/2.15.28/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( 2024-03-21 18:38:51 hoge-bucket $ echo hello > hello.txt $ aws --no-verify-ssl --endpoint-url https://localhost:9000 s3 cp hello.txt s3://hoge-bucket/ /opt/homebrew/Cellar/awscli/2.15.28/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( upload: ./hello.txt to s3://hoge-bucket/hello.txt $ aws --no-verify-ssl --endpoint-url https://localhost:9000 s3 ls s3://hoge-bucket/ /opt/homebrew/Cellar/awscli/2.15.28/libexec/lib/python3.11/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'localhost'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings warnings.warn( 2024-03-21 19:08:13 6 hello.txt
エラーがでているが、バケットの作成と表示、オブジェクトのコピーができた。
SFTP でのアクセス
テナントの設定を変更する。
k -n minio-tenant-1 edit tenant minio1
spec: ... features: enableSFTP: true # false から true に変更
Service のポートに 8022 が追加される。
$ k -n minio-tenant-1 get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE minio ClusterIP 172.20.1.251 <none> 443/TCP 13d minio1-console ClusterIP 172.20.15.213 <none> 9443/TCP 13d minio1-hl ClusterIP None <none> 9000/TCP,8022/TCP 13d
これもポートフォワードする。
$ k -n minio-tenant-1 port-forward svc/minio1-hl 8022:8022 Forwarding from 127.0.0.1:8022 -> 8022 Forwarding from [::1]:8022 -> 8022
sftp でアクセスする。
$ sftp -P 8022 ET3U6W5UKQXFG7FDY1Q0@localhost The authenticity of host '[localhost]:8022 ([::1]:8022)' can't be established. ECDSA key fingerprint is SHA256:fzvIFWM20Ay8Nj4zo/K+gvu4blDaoHSf2p9fdcQA5JI. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '[localhost]:8022' (ECDSA) to the list of known hosts. ET3U6W5UKQXFG7FDY1Q0@localhost's password: Connected to localhost. sftp> ls hoge-bucket sftp> ls hoge-bucket hoge-bucket/hello.txt sftp>
