- Introduction
- Design Consideration
- Deployment
- Summary
Introduction
Alibaba Cloud VMware Service (ACVS) is a cloud-based service offered by Alibaba Cloud in collaboration with VMware. AVS allows customers to run their VMware workloads on Alibaba Cloud’s infrastructure, providing a seamless and compatible environment for existing VMware-based applications. With ACVS, customers can deploy and manage their vSphere VM and related resources on Alibaba Cloud’s highly available and scalable infrastructure. This service enables organizations to migrate their on-premises VMware environments to the cloud without the need for significant changes to their existing VMware configurations or toolsets.
VMware Tanzu Kubernetes Grid (TKG) is a Kubernetes distribution and management platform provided by VMware. It is designed to simplify the deployment and operations of Kubernetes clusters across various infrastructures, including on-premises, public clouds, and edge environments. TKG builds upon open-source Kubernetes and adds additional capabilities to enhance its deployment and management experience. It provides a consistent and secure Kubernetes runtime across multiple environments, making it easier for organizations to adopt and manage Kubernetes at scale.
TKG includes integrated open source technologies for monitoring, logging, ingress, and more. This makes it easier to install and deploy these technologies locally as well as across clusters in multiple environments.
Design Consideration
- Components version
- Network design
- Load balancer design
- Firewall
As the architecture shows the internet access for TKG service is configured in Alicloud NAT Gateway that will be routed via T0 t0 vRotuer in the VPC. We created three segments, TKG management is for management appliances like NSX Advanced Load Balancers controller, Harbor, Bootstrap VM etc. TKG cluster is for TKG management cluster and workload cluster, the VIP network is for load balancer service of applications.
Besides, the security group on Alicloud should be properly configured to allow mandatory TCP or UDP port that you may refer to here.
Below are version information of SDDC and Tanzu that this environment is build on,
SDDC version
ACVS SDDC components | Version |
vSphere | 7.0.3 (Build: 22513421) |
vCenter | 7.0.3 (Build: 20870699) |
NSX Manager | 3.1.5 (Build: 21341580) |
NSX Advanced Load Balancer | 22.1.2 |
TKG components and related version
TKG Components | Downloaded file | Version | Download URL |
TKG | NA | 2.2.0 | NA |
Kubectl for VMware Tanzu Kubernetes Grid 2.2.0 | kubectl-linux-v1.25.7+vmware.2.gz | v1.25.7 | click |
Base OS | ubuntu-2004-kube-v1.25.7+vmware.2-tkg.1-8a74b9f12e488c54605b3537acb683bc.ova | v1.25.7 | click |
Tanzu CLI for Linux | tanzu-cli-bundle-linux-amd64.tar.gz | v0.29.1 | click |
Harbor | harbor-offline-installer-v2.7.1.tgz | v2.7.1 | click |
Pre-req
Before we can use the Tanzu CLI or UI to deploy a management cluster, we must prepare your vSphere environment.
Items | Requirement |
Bootstrap machine | with the Tanzu CLI, Docker, and kubectl installed |
vSphere environment | vSphere 8 or vSphere 7 or vSphere 6.7u3 |
Base image | upload in vCenter and covert to template matches the management cluster’s Kubernetes version which is 1.25.7 under TKG 2.2 |
NSX Advanced Load Balancer | deployed |
static virtual IP addresses in NSX Advanced Load Balancer | for all of the clusters that created for both management and workload clusters |
DHCP server | to allocate IP addresses to the created Kubernetes nodes VM in TKG |
DNS nameserver | use tkgm220.acvs in this lab |
ACVS NAT Gateway | enable SNAT |
Firewall port | allow port 443 between vCenter and TKG cluster, port 6443 between bootstrap machine and all VMs in the clusters created |
Besides, here is some resource usage in ACVS Account
Resource | Tanzu Kubernetes Grid Uses |
VPC | 1 (shared the VPC ACVS uses) |
Elastic IP Addresses | 1 |
vSwitch | NA, as we use NSX logical segment |
NAT Gateway | 1 |
Deployment
We need to use bootstrap machine to initialize a Tanzu Kubernetes Grid instance by bootstrapping a management cluster. I used CentOS 7 to create this machine, after setting up we need to add this machine to domain which is necessary to install Harbor in later stage.
I have uploaded kubectl, tanzu cli and harbor installer into the bootstrap machine with folder ‘tkgm’ as below shows,
Step 1: install Kubectl tool, and make it available to the system
gunzip kubectl-linux-v1.25.7+vmware.2.gz
install kubectl-linux-v1.25.7+vmware.2 /usr/local/bin/kubectl
Step 2: install Tanzu-CLI tool, and make it available to the system
tar xvf tanzu-cli-bundle-linux-amd64.tar.gz lo
cli/core/v0.29.0/tanzu-core-linux_amd64 /usr/local/bin/tanzu
Step 3: install ytt
gunzip ytt-linux-amd64-v0.43.1+vmware.1.gz
chmod +x ytt-linux-amd64-v0.43.1+vmware.1
mv ytt-linux-amd64-v0.43.1+vmware.1 /usr/local/bin/ytt
ytt version
Step 4: install kapp
gunzip kapp-linux-amd64-v0.53.2+vmware.1.gz
chmod +x kapp-linux-amd64-v0.53.2+vmware.1
mv kapp-linux-amd64-v0.53.2+vmware.1 /usr/local/bin/kapp
kapp version
Step 5: install kbld
gunzip kbld-linux-amd64-v0.35.1+vmware.1.gz
chmod +x kbld-linux-amd64-v0.35.1+vmware.1
mv kbld-linux-amd64-v0.35.1+vmware.1 /usr/local/bin/kbld
kbld version
Step 6: install imgpkg
gunzip imgpkg-linux-amd64-v0.31.1+vmware.1.gz
chmod +x imgpkg-linux-amd64-v0.31.1+vmware.1
mv imgpkg-linux-amd64-v0.31.1+vmware.1 /usr/local/bin/imgpkg
imgpkg version
Step 7: install docker
sudo yum install -y yum-utils
sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
sudo yum install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
sudo systemctl start docker
sudo systemctl enable docker
systemctl stop firewalld
systemctl disable firewalld
Step 8: Initialize the Tanzu CLI
tanzu init
Step 9 : generate certificate for Harbor
Generate CA private key called ‘ca.crt’ and ‘ca.key’,
openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout ca.key \
-x509 -days 365 -out ca.crt \
-subj "/C=CN/ST=Guangdong/L=Shenzhen/O=test_company/OU=IT/CN=harbor103.tkgm220.acvs"
Generate certification request with name ‘harbor-registry.csr’ and key with ‘harbor-registry.key’,
openssl req \
-newkey rsa:4096 -nodes -sha256 -keyout harbor-registry.key \
-out harbor-registry.csr \
-subj "/C=CN/ST=Guangdong/L=Shenzhen/O=test_company/OU=IT/CN=harbor103.tkgm220.acvs"
Generate X509 Ext file called ‘v3.ext’,
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1=harbor103.tkgm220.acvs
DNS.2=tkgm220.acvs
DNS.3=harbor103
EOF
Create the certification,
mkdir -p /etc/docker/certs.d/harbor103.tkgm220.acvs/
cp harbor-registry.cert /etc/docker/certs.d/harbor103.tkgm220.acvs/
cp harbor-registry.key /etc/docker/certs.d/harbor103.tkgm220.acvs/
cp ca.crt /etc/docker/certs.d/harbor103.tkgm220.acvs/
change hostname,
vi /etc/hostname
harbor75.vmc.lab
reboot
Restart docker daemon,
systemctl daemon-reload
systemctl restart docker
systemctl enable docker
Step 10: install Harbor
tar zxvf harbor-offline-installer-v2.8.2.tgz
#ll
-rw-r--r--. 1 root root 3639 Jun 2 07:43 common.sh
-rw-r--r--. 1 root root 610962984 Jun 2 07:44 harbor.v2.8.2.tar.gz
-rw-r--r--. 1 root root 11736 Jun 2 07:43 harbor.yml.tmpl
-rwxr-xr-x. 1 root root 2725 Jun 2 07:43 install.sh
-rw-r--r--. 1 root root 11347 Jun 2 07:43 LICENSE
-rwxr-xr-x. 1 root root 1881 Jun 2 07:43 prepare
cd harbor
cp harbor.yml.tmpl harbor.yml
vi harbor.yml
hostname: reg.mydomain.com harbor103.tkgm220.acvs
certificate: /your/certificate/path/etc/docker/certs.d/harbor103.tkgm220.acvs/harbor-registry.cert
private_key: /your/private/key/path/etc/docker/certs.d/harbor103.tkgm220.acvs/harbor-registry.key
harbor_admin_password: Harbor12345 VMware1!
./prepare
./install.sh
docker login harbor103.tkgm220.acvs
Username: admin
Password: VMware1!
In case need to disable and uninstall Harbor, here is the command
cd harbor
docker compose down -v #stop service
ls /data/*
rm -rf /data/database
rm -rf /data/registry
rm -rf /data/*
rm -rf /var/log/harbor*
docker compose up -d #start service
Step 11: Install Tanzu Management Cluster
tanzu management-cluster create --ui --bind 10.100.0.103:8099 --browser none --timeout 5h0m0s
Validating the pre-requisites...
Serving kickstart UI at http://10.100.0.103:8099
ℹ All required plugins are already installed and up-to-date
Open 10.100.0.103 in browser which is IP address of bootstrap machine, you will see the installation UI, and click Deploy,
Select vCenter and provide username and password,
Choose cluster type that I use Development to save space and give a name, then choose NSX Advanced Load Balancer as management cluster endpoint provider,
Validate NSX ALB credential and certificate,
Go to NSX ALB to copy the certificate,
Choose NSX ALB cloud and select network segments of control node and data node for management cluster and workload cluster separately,
Choose the resource pool and VM folder,
Use the default Kubernetes network,
Skip Identify management,
Select the VM template for OS image,
The deployment takes a while about 15 min to 45 min up to size of bootstrap machine and vSphere.
Go to the folder assigned in vCenter we can see the management cluster and workload cluster were created, each with one control node and one worker node,
The VIP service of management cluster and workload cluster was also configured and the associated server pool,
Checking node status in console,
Access workload cluster and switch the context, we may can see that there are two nodes which is mater and worker respectively.
Summary
We explored the concept of Tanzu Kubernetes Grid design and its step-by-step deployment on the Alibaba Cloud VMware Solution (ACVS). Tanzu Kubernetes Grid design is a modern approach to building and managing applications that leverages the power of Kubernetes. By deploying TKG on ACVS, organizations can unlock a range of key benefits to simplified application management and enable hybrid cloud capability.