使用kubeadm部署高可用Kubernetes v1.11.1集群

1. 环境配置
生产环境部署高可用Kubernetes环境时至少配置三台Master节点,如有更高要求可以再增加,但Master节点数量应为奇数

+-------------+---------------+---------------------------+
|   Hostname  |   IP Address  |           Role            |
+-------------+---------------+---------------------------+
| k8s         | 192.168.1.100 | VIP                       |
+-------------+---------------+---------------------------+
| k8s-master1 | 192.168.1.101 | Master,Keepalived,HAProxy |
+-------------+---------------+---------------------------+
| k8s-master2 | 192.168.1.102 | Master,Keepalived,HAProxy |
+-------------+---------------+---------------------------+
| k8s-master3 | 192.168.1.103 | Master,Keepalived,HAProxy |
+-------------+---------------+---------------------------+
| k8s-worker1 | 192.168.1.104 | Worker                    |
+-------------+---------------+---------------------------+
| k8s-worker2 | 192.168.1.105 | Worker                    |
+-------------+---------------+---------------------------+
| k8s-worker3 | 192.168.1.106 | Worker                    |
+-------------+---------------+---------------------------+

在所有节点加入hosts信息,如有DNS记录则不用

cat <<EOF >> /etc/hosts
192.168.1.100 k8s k8s.test.local
192.168.1.101 k8s-master1 k8s-master1.test.local
192.168.1.102 k8s-master2 k8s-master2.test.local
192.168.1.103 k8s-master3 k8s-master3.test.local
192.168.1.104 k8s-worker1 k8s-worker1.test.local
192.168.1.105 k8s-worker2 k8s-worker2.test.local
192.168.1.106 k8s-worker3 k8s-worker3.test.local
EOF

安装Docker,步骤略可参考http://www.ebanban.com/?p=496

在所有Master节点上输入以下环境变量,主机名和IP信息根据自己的实际的情况进行修改

export KUBECONFIG=/etc/kubernetes/admin.conf
export LOAD_BALANCER_DNS=k8s.test.local
export LOAD_BALANCER_PORT=8443
export CP1_HOSTNAME=k8s-master1.test.local
export CP2_HOSTNAME=k8s-master2.test.local
export CP3_HOSTNAME=k8s-master3.test.local
export VIP_IP=192.168.1.100
export CP1_IP=192.168.1.101
export CP2_IP=192.168.1.102
export CP3_IP=192.168.1.103

关闭防火墙、关闭swap、关闭SELinux、调整内核参数

sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo setenforce 0
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

从mirrorgooglecontainers源下载kubernetes镜像

docker pull mirrorgooglecontainers/kube-apiserver-amd64:v1.11.1
docker pull mirrorgooglecontainers/kube-scheduler-amd64:v1.11.1
docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.11.1
docker pull mirrorgooglecontainers/kube-controller-manager-amd64:v1.11.1
docker pull mirrorgooglecontainers/pause-amd64:3.1
docker pull mirrorgooglecontainers/etcd-amd64:3.2.18
docker pull coredns/coredns:1.1.3
docker pull quay.io/coreos/flannel:v0.10.0-amd64

将镜像标记为k8s.gcr.io的名称

docker tag mirrorgooglecontainers/kube-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1
docker tag mirrorgooglecontainers/kube-scheduler-amd64:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1
docker tag mirrorgooglecontainers/kube-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker tag mirrorgooglecontainers/kube-controller-manager-amd64:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
docker tag mirrorgooglecontainers/pause-amd64:3.1 k8s.gcr.io/pause:3.1
docker tag mirrorgooglecontainers/etcd-amd64:3.2.18 k8s.gcr.io/etcd-amd64:3.2.18
docker tag coredns/coredns:1.1.3 k8s.gcr.io/coredns:1.1.3

删除无用镜像名

docker images | grep mirrorgooglecontainers | awk '{print "docker rmi "$1":"$2}' | sh
docker rmi coredns/coredns:1.1.3

安装、配置kubelet

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet-1.11.1 kubeadm-1.11.1 kubectl-1.11.1
systemctl enable kubelet

2. 准备SSH Keys
生成SSH Key(通常在第一台Master上操作,可以在终端上操作)

ssh-keygen -t rsa -b 2048 -f /root/.ssh/id_rsa -N ""

将SSH Key复制给其他主机

for host in {$CP1_HOSTNAME,$CP2_HOSTNAME,$CP3_HOSTNAME}; do ssh-copy-id $host; done

3. 部署keepalived(Master节点)
keepalived用于生产浮动的虚拟IP,并将浮动IP分配给优先级最高且haproxy正常运行的节点
在第一台Master上配置和启动keepalived,若网卡名称不为示例中的eth0则改为对应名称

yum install -y keepalived curl psmisc && systemctl enable keepalived
cat << EOF > /etc/keepalived/keepalived.conf
vrrp_script haproxy-check {
    script "killall -0 haproxy"
    interval 2
    weight 20
} 
vrrp_instance haproxy-vip {
    state BACKUP
    priority 102
    interface eth0
    virtual_router_id 51
    advert_int 3
    unicast_src_ip $CP1_IP
    unicast_peer {
        $CP2_IP
        $CP3_IP
    }
    virtual_ipaddress {
        $VIP_IP
    }
    track_script {
        haproxy-check weight 20
    }
}
EOF
systemctl start keepalived

在第二台Master上配置和启动keepalived

yum install -y keepalived curl psmisc && systemctl enable keepalived
cat << EOF > /etc/keepalived/keepalived.conf
vrrp_script haproxy-check {
    script "killall -0 haproxy"
    interval 2
    weight 20
}
vrrp_instance haproxy-vip {
    state BACKUP
    priority 101
    interface eth0
    virtual_router_id 51
    advert_int 3
    unicast_src_ip $CP2_IP
    unicast_peer {
        $CP1_IP
        $CP3_IP
    }
    virtual_ipaddress {
        $VIP_IP
    }
    track_script {
        haproxy-check weight 20
    }
}
EOF
systemctl start keepalived

在第三台主机上配置和启动keepalived

yum install -y keepalived curl psmisc && systemctl enable keepalived
cat << EOF > /etc/keepalived/keepalived.conf
vrrp_script haproxy-check {
    script "killall -0 haproxy"
    interval 2
    weight 20
}
vrrp_instance haproxy-vip {
    state BACKUP
    priority 100
    interface eth0
    virtual_router_id 51
    advert_int 3
    unicast_src_ip $CP3_IP
    unicast_peer {
        $CP1_IP
        $CP2_IP
    }
    virtual_ipaddress {
        $VIP_IP
    }
    track_script {
        haproxy-check weight 20
    }
}
EOF
systemctl start keepalived

4. 部署HAProxy(Master节点)
HAProxy用于检测集群内api-server的健康状况并进行负载均衡
在三台Master节点上执行以下命令安装和启用HAProxy

yum install -y haproxy && systemctl enable haproxy
cat << EOF > /etc/haproxy/haproxy.cfg
global
  log 127.0.0.1 local0
  log 127.0.0.1 local1 notice
  tune.ssl.default-dh-param 2048

defaults
  log global
  mode http
  option dontlognull
  timeout connect 5000ms
  timeout client  600000ms
  timeout server  600000ms

listen stats
    bind :9090
    mode http
    balance
    stats uri /haproxy_stats
    stats auth admin:admin
    stats admin if TRUE

frontend kube-apiserver-https
   mode tcp
   bind :8443
   default_backend kube-apiserver-backend

backend kube-apiserver-backend
    mode tcp
    balance roundrobin
    stick-table type ip size 200k expire 30m
    stick on src
    server k8s-master1 192.168.1.101:6443 check
    server k8s-master2 192.168.1.102:6443 check
    server k8s-master3 192.168.1.103:6443 check
EOF
systemctl start haproxy

5. 初始化k8s集群(第一台)
在第一台Master上执行以下命令,10.244.0.0/16是flannel的CIDR地址,如果用其他CNI需要改成对应的CIDR。

cat << EOF > ~/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
apiServerCertSANs:
- "$LOAD_BALANCER_DNS"
api:
  controlPlaneEndpoint: "$LOAD_BALANCER_DNS:$LOAD_BALANCER_PORT"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://$CP1_IP:2379"
      advertise-client-urls: "https://$CP1_IP:2379"
      listen-peer-urls: "https://$CP1_IP:2380"
      initial-advertise-peer-urls: "https://$CP1_IP:2380"
      initial-cluster: "$CP1_HOSTNAME=https://$CP1_IP:2380"
      name: $CP1_HOSTNAME
    serverCertSANs:
      - $CP1_HOSTNAME
      - $CP1_IP
    peerCertSANs:
      - $CP1_HOSTNAME
      - $CP1_IP
networking:
    podSubnet: "10.244.0.0/16"
EOF

初始化第一台master,初始化完成后记录生成的kubeadm join命令(包含token),用于后面worker节点加入时使用

kubeadm init --config ~/kubeadm-config.yaml

将相关证书文件复制到其他master节点

CONTROL_PLANE_HOSTS="$CP2_HOSTNAME $CP3_HOSTNAME"
for host in $CONTROL_PLANE_HOSTS; do
    scp /etc/kubernetes/pki/ca.crt $host:
    scp /etc/kubernetes/pki/ca.key $host:
    scp /etc/kubernetes/pki/sa.key $host:
    scp /etc/kubernetes/pki/sa.pub $host:
    scp /etc/kubernetes/pki/front-proxy-ca.crt $host:
    scp /etc/kubernetes/pki/front-proxy-ca.key $host:
    scp /etc/kubernetes/pki/etcd/ca.crt $host:etcd-ca.crt
    scp /etc/kubernetes/pki/etcd/ca.key $host:etcd-ca.key
    scp /etc/kubernetes/admin.conf $host:
done

6. 加入k8s集群(第二台)
在第二台Master上执行以下命令

cat << EOF > ~/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
apiServerCertSANs:
- "$LOAD_BALANCER_DNS"
api:
    controlPlaneEndpoint: "$LOAD_BALANCER_DNS:$LOAD_BALANCER_PORT"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://$CP2_IP:2379"
      advertise-client-urls: "https://$CP2_IP:2379"
      listen-peer-urls: "https://$CP2_IP:2380"
      initial-advertise-peer-urls: "https://$CP2_IP:2380"
      initial-cluster: "$CP1_HOSTNAME=https://$CP1_IP:2380,$CP2_HOSTNAME=https://$CP2_IP:2380"
      initial-cluster-state: existing
      name: $CP2_HOSTNAME
    serverCertSANs:
      - $CP2_HOSTNAME
      - $CP2_IP
    peerCertSANs:
      - $CP2_HOSTNAME
      - $CP2_IP
networking:
    podSubnet: "10.244.0.0/16"
EOF

将证书移到相应目录

mkdir -p /etc/kubernetes/pki/etcd
mv ~/ca.crt /etc/kubernetes/pki/
mv ~/ca.key /etc/kubernetes/pki/
mv ~/sa.pub /etc/kubernetes/pki/
mv ~/sa.key /etc/kubernetes/pki/
mv ~/front-proxy-ca.crt /etc/kubernetes/pki/
mv ~/front-proxy-ca.key /etc/kubernetes/pki/
mv ~/etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt
mv ~/etcd-ca.key /etc/kubernetes/pki/etcd/ca.key
mv ~/admin.conf /etc/kubernetes/admin.conf

配置并启动kubelet

kubeadm alpha phase certs all --config ~/kubeadm-config.yaml
kubeadm alpha phase kubelet config write-to-disk --config ~/kubeadm-config.yaml
kubeadm alpha phase kubelet write-env-file --config ~/kubeadm-config.yaml
kubeadm alpha phase kubeconfig kubelet --config ~/kubeadm-config.yaml
systemctl start kubelet

加入etcd集群

kubectl exec -n kube-system etcd-${CP1_HOSTNAME} -- etcdctl \
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
--key-file /etc/kubernetes/pki/etcd/peer.key \
--endpoints=https://${CP1_IP}:2379 \
member add ${CP2_HOSTNAME} https://${CP2_IP}:2380
kubeadm alpha phase etcd local --config ~/kubeadm-config.yaml

将节点配置为master

kubeadm alpha phase kubeconfig all --config ~/kubeadm-config.yaml
kubeadm alpha phase controlplane all --config ~/kubeadm-config.yaml
kubeadm alpha phase mark-master --config ~/kubeadm-config.yaml

7. 加入k8s集群(第三台)
在第三台Master上执行以下命令

cat << EOF > ~/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
apiServerCertSANs:
- "$LOAD_BALANCER_DNS"
api:
    controlPlaneEndpoint: "$LOAD_BALANCER_DNS:$LOAD_BALANCER_PORT"
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://$CP3_IP:2379"
      advertise-client-urls: "https://$CP3_IP:2379"
      listen-peer-urls: "https://$CP3_IP:2380"
      initial-advertise-peer-urls: "https://$CP3_IP:2380"
      initial-cluster: "$CP1_HOSTNAME=https://$CP1_IP:2380,$CP2_HOSTNAME=https://$CP2_IP:2380,$CP3_HOSTNAME=https://$CP3_IP:2380"
      initial-cluster-state: existing
      name: $CP3_HOSTNAME
    serverCertSANs:
      - $CP3_HOSTNAME
      - $CP3_IP
    peerCertSANs:
      - $CP3_HOSTNAME
      - $CP3_IP
networking:
    podSubnet: "10.244.0.0/16"
EOF

将证书移到相应目录

mkdir -p /etc/kubernetes/pki/etcd
mv ~/ca.crt /etc/kubernetes/pki/
mv ~/ca.key /etc/kubernetes/pki/
mv ~/sa.pub /etc/kubernetes/pki/
mv ~/sa.key /etc/kubernetes/pki/
mv ~/front-proxy-ca.crt /etc/kubernetes/pki/
mv ~/front-proxy-ca.key /etc/kubernetes/pki/
mv ~/etcd-ca.crt /etc/kubernetes/pki/etcd/ca.crt
mv ~/etcd-ca.key /etc/kubernetes/pki/etcd/ca.key
mv ~/admin.conf /etc/kubernetes/admin.conf

配置并启动kubelet

kubeadm alpha phase certs all --config ~/kubeadm-config.yaml
kubeadm alpha phase kubelet config write-to-disk --config ~/kubeadm-config.yaml
kubeadm alpha phase kubelet write-env-file --config ~/kubeadm-config.yaml
kubeadm alpha phase kubeconfig kubelet --config ~/kubeadm-config.yaml
systemctl start kubelet

加入etcd集群

kubectl exec -n kube-system etcd-${CP1_HOSTNAME} -- etcdctl \
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
--key-file /etc/kubernetes/pki/etcd/peer.key \
--endpoints=https://${CP1_IP}:2379 \
member add ${CP3_HOSTNAME} https://${CP3_IP}:2380
kubeadm alpha phase etcd local --config ~/kubeadm-config.yaml

将节点配置为master

kubeadm alpha phase kubeconfig all --config ~/kubeadm-config.yaml
kubeadm alpha phase controlplane all --config ~/kubeadm-config.yaml
kubeadm alpha phase mark-master --config ~/kubeadm-config.yaml

8. 配置网络插件(以flannel为例)

curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

9. 将worker节点加入集群
下载kubernetes镜像

docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.11.1
docker pull mirrorgooglecontainers/pause-amd64:3.1
docker pull quay.io/coreos/flannel:v0.10.0-amd64
docker tag mirrorgooglecontainers/kube-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker tag mirrorgooglecontainers/pause-amd64:3.1 k8s.gcr.io/pause:3.1
docker images | grep mirrorgooglecontainers | awk '{print "docker rmi "$1":"$2}' | sh

安装kubelet、kubeadm

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet-1.11.1 kubeadm-1.11.1
systemctl enable kubelet

将worker加入k8s集群,使用在初始化第一台Master时生成的命令,如

kubeadm join k8s.test.local:8443 --token bqnani.kwxe3y34vy22xnhm --discovery-token-ca-cert-hash sha256:b6146fea7a63d3a66e406c12f55f8d99537db99880409939e4aba206300e06cc

10. 确认集群运行状态
确认etcd运行状态

docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd-amd64:3.2.18 etcdctl \
--cert-file /etc/kubernetes/pki/etcd/peer.crt \
--key-file /etc/kubernetes/pki/etcd/peer.key \
--ca-file /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://$CP1_IP:2379 cluster-health

若正常应返回类似如下结果

member 4fea45cc5063c213 is healthy: got healthy result from https://192.168.1.101:2379
member 963074f50ce23d9a is healthy: got healthy result from https://192.168.1.102:2379
member 9a186be7d1ea4bbe is healthy: got healthy result from https://192.168.1.103:2379

确认k8s集群nodes运行状态

kubectl get nodes

若正常应返回类似如下结果

NAME                     STATUS    ROLES     AGE       VERSION
k8s-master1.test.local   Ready     master    1d        v1.11.1
k8s-master2.test.local   Ready     master    1d        v1.11.1
k8s-master3.test.local   Ready     master    1d        v1.11.1
k8s-worker1.test.local   Ready         1d        v1.11.1
k8s-worker2.test.local   Ready         1d        v1.11.1
k8s-worker3.test.local   Ready         1d        v1.11.1

确认k8s集群pods运行状态

kubectl get pods -n kube-system

若正常应返回类似如下结果,其中etcd、kube-apiserver、kube-controller-manager和kube-scheduler应该各有三个,coredns默认有两个,kube-proxy和kube-flannel的数量应和node数量一致,本示例中为6个

NAMESPACE       NAME                                            READY     STATUS    RESTARTS   AGE
kube-system     coredns-78fcdf6894-j6cpl                        1/1       Running   0          1d
kube-system     coredns-78fcdf6894-kgqp7                        1/1       Running   0          1d
kube-system     etcd-k8s-master1.test.local                     1/1       Running   0          1d
kube-system     etcd-k8s-master2.test.local                     1/1       Running   0          1d
kube-system     etcd-k8s-master3.test.local                     1/1       Running   0          1d
kube-system     kube-apiserver-k8s-master1.test.local           1/1       Running   0          1d
kube-system     kube-apiserver-k8s-master2.test.local           1/1       Running   0          1d
kube-system     kube-apiserver-k8s-master3.test.local           1/1       Running   0          1d
kube-system     kube-controller-manager-k8s-master1.test.local  1/1       Running   0          1d
kube-system     kube-controller-manager-k8s-master2.test.local  1/1       Running   0          1d
kube-system     kube-controller-manager-k8s-master3.test.local  1/1       Running   0          1d
kube-system     kube-flannel-ds-amd64-2r7jp                     1/1       Running   0          1d
kube-system     kube-flannel-ds-amd64-d5vlw                     1/1       Running   0          1d
kube-system     kube-flannel-ds-amd64-qd5x6                     1/1       Running   0          1d
kube-system     kube-flannel-ds-amd64-wzl26                     1/1       Running   0          1d
kube-system     kube-flannel-ds-amd64-xklr6                     1/1       Running   0          1d
kube-system     kube-flannel-ds-amd64-4jr5v                     1/1       Running   0          1d
kube-system     kube-proxy-8gmdd                                1/1       Running   0          1d
kube-system     kube-proxy-8rs8m                                1/1       Running   0          1d
kube-system     kube-proxy-pm6tq                                1/1       Running   0          1d
kube-system     kube-proxy-shsjv                                1/1       Running   0          1d
kube-system     kube-proxy-vj5gk                                1/1       Running   0          1d
kube-system     kube-proxy-wd8xj                                1/1       Running   0          1d
kube-system     kube-scheduler-k8s-master1.test.local           1/1       Running   0          1d
kube-system     kube-scheduler-k8s-master2.test.local           1/1       Running   0          1d
kube-system     kube-scheduler-k8s-master3.test.local           1/1       Running   0          1d

使用kubeadm部署高可用Kubernetes v1.11.1集群》上有5条评论

  1. Pingback引用通告: Kubernetes Dashboard安装 | eBanBan Studio

  2. ZHIHUI DUAN

    apiServerCertSANs:
    – “$LOAD_BALANCER_DNS”

    $LOAD_BALANCER_DNS可以是域名吗 自定义域名

    回复
    1. daban 文章作者

      在前面的命令里有这个变量的信息“export LOAD_BALANCER_DNS=k8s.test.local”,k8s.test.local是这个浮动IP对应的域名,可以自定义,只要DNS或hosts能解析到就行

      回复

daban进行回复 取消回复

电子邮件地址不会被公开。 必填项已用*标注