以前写过使用keepalived来创建kubernetes高可用集群，现在有了kube-vip方案，因此将安装方法更新下：
所使用的环境如下：
Ubuntu Server 20.04 LTS (自从CentOS变成CentOS Stream后就转用Debian/Ubuntu了)
containerd 1.5.5 (k8s 1.24之后就不再支持docker了，因此改用containerd)
Kubernetes v1.23.5
kube-vip v0.4.3 (这里为了简单部署使用L2 ARP方式)
+-------------+---------------+--------+ | Hostname | IP Address | Role | +-------------+---------------+--------+ | k8s | 192.168.1.210 | VIP | +-------------+---------------+--------+ | k8s-master1 | 192.168.1.211 | Master | +-------------+---------------+--------+ | k8s-master2 | 192.168.1.212 | Master | +-------------+---------------+--------+ | k8s-master3 | 192.168.1.213 | Master | +-------------+---------------+--------+ | k8s-worker1 | 192.168.1.214 | Worker | +-------------+---------------+--------+ | k8s-worker2 | 192.168.1.215 | Worker | +-------------+---------------+--------+ | k8s-worker3 | 192.168.1.216 | Worker | +-------------+---------------+--------+
使用kube-vip来构建master节点的vip，这样就不需要外部负载均衡器或者keepalived+haproxy之类的设置了。架构图如下：
1. 环境准备
配置相关内核模块
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter EOF cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward = 1 EOF sudo sysctl --system
关闭swap
sudo swapoff -a sudo sed -i '/swap/s/^/#/' /etc/fstab
安装所需软件包及containerd
sudo apt-get update sudo apt-get install -y apt-transport-https ca-certificates curl chrony jq containerd
配置containerd
sudo mkdir -p /etc/containerd containerd config default | \ sed -e 's,SystemdCgroup = .*,SystemdCgroup = true,' | \ sudo tee /etc/containerd/config.toml sudo systemctl restart containerd
2. 安装K8S所需工具
安装kubectl, kubeadm和kubectl，并配置containerd为k8s默认CRI
sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo crictl config runtime-endpoint /run/containerd/containerd.sock
确认安装的版本，本文使用的是v1.23.5版本
kubectl version --client --short
添加命令补全
crictl completion > /etc/bash_completion.d/crictl kubectl completion bash > /etc/bash_completion.d/kubectl kubeadm completion bash > /etc/bash_completion.d/kubeadm
从国内站点下载容器镜像（所有节点）
sudo cat << EOF > ~/pull_k8s_images.sh
#!/bin/bash
#Made by www.ebanban.com
registry=registry.cn-hangzhou.aliyuncs.com/google_containers
images=\`kubeadm config images list |awk -F '/' '{print \$NF}'\`
for image in \$images
do
if [[ \$image =~ "coredns" ]]; then
ctr -n k8s.io image pull \${registry}/\$image
if [ \$? -eq 0 ]; then
ctr -n k8s.io image tag \${registry}/\$image k8s.gcr.io/coredns/\$image
ctr -n k8s.io image rm \${registry}/\$image
else
echo "ERROR: download failed，\$image"
fi
else
ctr -n k8s.io image pull \${registry}/\$image
if [ \$? -eq 0 ]; then
ctr -n k8s.io image tag \${registry}/\$image k8s.gcr.io/\$image
ctr -n k8s.io image rm \${registry}/\$image
else
echo "ERROR: download failed，\$image"
fi
fi
done
EOF
sudo bash ~/pull_k8s_images.sh
sudo rm ~/pull_k8s_images.sh
3. 创建Kube-vip的静态pod
kube-vip的pod需要先于k8s集群创建，因此我们采用static pod方式来创建kube-vip的pod。在三台master服务器上执行以下命令（VIP和网卡名根据实际情况修改）
export VIP=192.168.1.210
export INTERFACE=ens32
KVVERSION=$(curl -sL https://api.github.com/repos/kube-vip/kube-vip/releases | jq -r ".[0].name")
alias kube-vip="ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:$KVVERSION vip /kube-vip"
kube-vip manifest pod \
--interface $INTERFACE \
--address $VIP \
--controlplane \
--services \
--arp \
--leaderElection | tee /etc/kubernetes/manifests/kube-vip.yaml
4. 部署第一台k8s master节点
使用kubeadm init来初始化第一个节点
export VIP=192.168.1.210 sudo kubeadm init --control-plane-endpoint "$VIP:6443" --upload-certs
执行完成后会出现以下结果。记录下其中kubeadm join开头的命令用于后续节点加入
Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.1.211:6443 --token vwgly8.gkbh84snlmffq1k6 \ --discovery-token-ca-cert-hash sha256:19cfbebb57eb86b75df3759929d255d2ee661d367f7b53328b9d6f209951ce9e
执行以下命令来开始操作k8s集群，使用kubectl get pod来查看容器运行情况
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config kubectl get pod -n kube-system -w
5. 部署第二、三台k8s master节点
使用第一台master节点安装后生成的命令来加入第二、三台master。使用以下命令生成join命令。
echo $(kubeadm token create --print-join-command) --control-plane --certificate-key $(kubeadm init phase upload-certs --upload-certs | sed -n '3p')
然后在第二、三台master节点执行生成的命令
(以下为示例，请根据实际进行调整) sudo kubeadm join 192.168.1.210:6443 --token 0r35b6.pe7st8nz7hfzs5j7 \ --discovery-token-ca-cert-hash sha256:9353b8a8b5ea759e3ffdddb805b076e087588c5ea8a2b40ddfd3a19d0e5e600f \ --control-plane --certificate-key 77f3d44ed8b2e0b5521cb878e49bec627e2619b7ba4a2ea63b47f4c911f7e058
第二、三台加入后使用kubectl get nodes或者node状态为NotReady，这是由于还没有部署网络CNI
NAME STATUS ROLES AGE VERSION k8s-master1 NotReady control-plane,master 5m v1.23.5 k8s-master2 NotReady control-plane,master 7m v1.23.5 k8s-master3 NotReady control-plane,master 7m v1.23.5
6. 部署CNI
CNI组件本文使用Cilium
首先要安装Cilium CLI
curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz{,.sha256sum}
sha256sum --check cilium-linux-amd64.tar.gz.sha256sum
sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin
rm cilium-linux-amd64.tar.gz{,.sha256sum}
使用以下命令进行安装并确认状态及验证连通结果（测试部分可能由于无法连到国外网络而报错）
cilium install cilium status --wait cilium connectivity test
7. 加入worker节点
加入worker节点的方法与master节点类似，命令更简单点。
join命令可用kubeadm token create –print-join-command命令生成。
(以下为示例，请根据实际进行调整) export VIP=192.168.1.210 sudo kubeadm join $VIP:6443 --token 2edvrs.b2a86ndl64kctmlu --discovery-token-ca-cert-hash sha256:9353b8a8b5ea759e3ffdddb805b076e087588c5ea8a2b40ddfd3a19d0e5e600f
本文加入了三台worker节点，完成后确认状态如下：
# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master1 Ready control-plane,master 2h v1.23.5 k8s-master2 Ready control-plane,master 2h v1.23.5 k8s-master3 Ready control-plane,master 2h v1.23.5 k8s-worker1 Ready <none> 2h v1.23.5 k8s-worker2 Ready <none> 2h v1.23.5 k8s-worker3 Ready <none> 2h v1.23.5 # # kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-4n7lv 1/1 Running 0 2h kube-system cilium-bvjmn 1/1 Running 0 2h kube-system cilium-dxwv9 1/1 Running 0 2h kube-system cilium-operator-75d6565577-vrm67 1/1 Running 0 2h kube-system cilium-wkv86 1/1 Running 0 2h kube-system cilium-wvdhc 1/1 Running 0 2h kube-system cilium-xsq7c 1/1 Running 0 2h kube-system coredns-64897985d-6n8mg 1/1 Running 0 2h kube-system coredns-64897985d-l5wj9 1/1 Running 0 2h kube-system etcd-k8s-master1 1/1 Running 0 2h kube-system etcd-k8s-master2 1/1 Running 0 2h kube-system etcd-k8s-master3 1/1 Running 0 2h kube-system kube-apiserver-k8s-master1 1/1 Running 0 2h kube-system kube-apiserver-k8s-master2 1/1 Running 0 2h kube-system kube-apiserver-k8s-master3 1/1 Running 0 2h kube-system kube-controller-manager-k8s-master1 1/1 Running 0 2h kube-system kube-controller-manager-k8s-master2 1/1 Running 0 2h kube-system kube-controller-manager-k8s-master3 1/1 Running 0 2h kube-system kube-proxy-2cqfw 1/1 Running 0 2h kube-system kube-proxy-fww5l 1/1 Running 0 2h kube-system kube-proxy-hspp7 1/1 Running 0 2h kube-system kube-proxy-lzvtv 1/1 Running 0 2h kube-system kube-proxy-w9d56 1/1 Running 0 2h kube-system kube-proxy-wm9bn 1/1 Running 0 2h kube-system kube-scheduler-k8s-master1 1/1 Running 0 2h kube-system kube-scheduler-k8s-master2 1/1 Running 0 2h kube-system kube-scheduler-k8s-master3 1/1 Running 0 2h kube-system kube-vip-k8s-master1 1/1 Running 0 2h kube-system kube-vip-k8s-master2 1/1 Running 0 2h kube-system kube-vip-k8s-master3 1/1 Running 0 2h # # kubectl get svc -A NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2h kube-system kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 2h
8. 部署MetalLB负载均衡
对于本地部署的k8s是需要额外部署负载均衡器的，这样保证我们在expose类型为LoadBalancer类型的服务时，external ip不会一直处于Pending状态。虽然Kube-vip也可以用来实现服务的负载均衡，但我们这里部署的MetalLB使用L2的方式，BGP方式请自行研究。
使用以下命令完成MetalLB的部署
METALLBVER=$(curl -sL https://api.github.com/repos/metallb/metallb/releases | jq -r ".[0].name") kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/$METALLBVER/manifests/namespace.yaml kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/$METALLBVER/manifests/metallb.yaml
然后需要为MetalLB设置外部IP地址段，使用ConfigMap，本文使用192.168.1.221-230
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
namespace: metallb-system
name: config
data:
config: |
address-pools:
- name: default
protocol: layer2
addresses:
- 192.168.1.221-192.168.1.230
EOF
9. 部署Ingress Controller
使用以下命令来部署Ingress Controller
INGRESSVER=$(curl -sL https://api.github.com/repos/kubernetes/ingress-nginx/releases | jq -r '.[] | select(.target_commitish=="main" and .prerelease==false and .draft==false) | .tag_name' | sed -n '1p') kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/$INGRESSVER/deploy/static/provider/baremetal/deploy.yaml
修改ingress-nginx-controller服务类型为LoadBalancer
kubectl get svc -n ingress-nginx ingress-nginx-controller -o yaml | \ sed -e 's/type: NodePort/type: LoadBalancer/' | \ kubectl apply -f -
如果要对ingress service设置指定IP，则在ingress-nginx-controller配置的spec中加入loadBalancerIP: X.X.X.X，这里我们使用metallb自动分配的ip
10. 部署Dashboard
使用以下命令来部署dashboard
DASHBOARDVER=$(curl -sL https://api.github.com/repos/kubernetes/dashboard/releases | jq -r ".[0].name") kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/$DASHBOARDVER/aio/deploy/recommended.yaml
创建Role和用户
cat <<EOF | kubectl create -f - apiVersion: v1 kind: ServiceAccount metadata: name: admin-user namespace: kubernetes-dashboard --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: admin-user namespace: kubernetes-dashboard EOF
生成证书和secret并创建dashboard的ingress
export INGRESSEIP=192.168.1.221
openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:2048 -keyout kubernetes-dashboard-certs.key -out kubernetes-dashboard-certs.crt -subj "/CN=$INGRESSEIP.nip.io/O=$INGRESSEIP.nip.io"
kubectl create secret tls kubernetes-dashboard-certs --key kubernetes-dashboard-certs.key --cert kubernetes-dashboard-certs.crt -n kubernetes-dashboard
cat <<EOF | sudo kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-kubernetes-dashboard
namespace: kubernetes-dashboard
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
nginx.ingress.kubernetes.io/rewrite-target: /\$2
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/use-regex: "true"
spec:
rules:
- host: \$INGRESSEIP.nip.io
http:
paths:
- path: /dashboard(/|\$)(.*)
pathType: Prefix
backend:
service:
name: kubernetes-dashboard
port:
number: 443
tls:
- hosts:
- \$VIP.nip.io
secretName: kubernetes-dashboard-certs
ingressClassName: nginx
EOF
获取登录的token
kubectl -n kubernetes-dashboard describe secret $(kubectl -n kubernetes-dashboard get secret | grep kubernetes-dashboard-token | awk '{print $1}')
然后打开https://192.168.1.221.nip.io/dashboard，输入上面获得token就能登录啦