月度归档:2018年07月

使用docker部署3节点etcd集群

在每个节点输入以下变量,根据实际情况修改主机名和IP
为了将etcd API暴露给docker主机外的客户端,必须使用docker inspect命令获得容器的IP来配置,这里使用了–net=host主机网络的方式来简化这一步骤。

ETCD_VERSION=latest
TOKEN=my-etcd-token
CLUSTER_STATE=new
NAME_1=etcd-node1
NAME_2=etcd-node2
NAME_3=etcd-node3
HOST_1=192.168.1.101
HOST_2=192.168.1.102
HOST_3=192.168.1.103
CLUSTER=${NAME_1}=http://${HOST_1}:2380,${NAME_2}=http://${HOST_2}:2380,${NAME_3}=http://${HOST_3}:2380
DATA_DIR=/var/lib/etcd

在节点1输入以下命令

THIS_NAME=${NAME_1}
THIS_IP=${HOST_1}
docker run -d \
  --net=host \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd quay.io/coreos/etcd:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

在节点2输入以下命令

THIS_NAME=${NAME_2}
THIS_IP=${HOST_2}
docker run -d \
  --net=host \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd quay.io/coreos/etcd:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

在节点3输入以下命令

THIS_NAME=${NAME_3}
THIS_IP=${HOST_3}
docker run -d \
  --net=host \
  --volume=${DATA_DIR}:/etcd-data \
  --name etcd quay.io/coreos/etcd:${ETCD_VERSION} \
  /usr/local/bin/etcd \
  --data-dir=/etcd-data --name ${THIS_NAME} \
  --initial-advertise-peer-urls http://${THIS_IP}:2380 --listen-peer-urls http://${THIS_IP}:2380 \
  --advertise-client-urls http://${THIS_IP}:2379 --listen-client-urls http://${THIS_IP}:2379 \
  --initial-cluster ${CLUSTER} \
  --initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

全部执行完后使用以下命令查看集群节点信息

docker exec etcd /usr/local/bin/etcdctl --endpoints=http://${HOST_1}:2379 member list
20c4dbd9ca01c9fc: etcd-node-2 peerURLs=http://192.168.1.103:2380 clientURLs=http://192.168.1.103:2379 isLeader=false
52b6c5eaedead574: etcd-node-1 peerURLs=http://192.168.1.102:2380 clientURLs=http://192.168.1.102:2379 isLeader=false
7623946005cf410f: etcd-node-0 peerURLs=http://192.168.1.101:2380 clientURLs=http://192.168.1.101:2379 isLeader=true

更改Docker的lvm挂载方式从loop-vm至direct-lvm

Docker安装后默认在/var/lib/docker/devicemapper/devicemapper目录下生成data和metadata两个文件用于存放docker的数据,然而这种默认的loop-lvm挂载方式不适合生产环境使用,并且你也会收到docker的如下提示,生产环境应改为direct-lvm方式。
WARNING: devicemapper: usage of loopback devices is strongly discouraged for production use.
Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.

以下是更改docker device mapper的方法

方法一: 允许Docker配置direct-lvm模式
编辑/etc/docker/daemon.json文件(如果该文件不存在则进行创建),参数表如下
dm.directlvm_device:块设备路径(必须)
dm.thinp_percent:块设备使用比率,默认95
dm.thinp_metapercent:元数据使用比率,默认1
dm.thinp_autoextend_threshold:自动扩容阈值,默认80
dm.thinp_autoextend_percent:自动扩容比率,20
dm.directlvm_device_force:强制格式化设备,默认false
以下为示例

{
  "storage-driver": "devicemapper",
  "storage-opts": [
    "dm.directlvm_device=/dev/xdf",
    "dm.thinp_percent=95",
    "dm.thinp_metapercent=1",
    "dm.thinp_autoextend_threshold=80",
    "dm.thinp_autoextend_percent=20",
    "dm.directlvm_device_force=false"
  ]
}

重启docker以生效

sudo systemctl restart docker

方法二: 手工配置direct-lvm模式
停止docker服务

sudo systemctl stop docker

安装必要的软件包

sudo yum install -y device-mapper-persistent-data lvm2

创建pv,示例中/dev/sdb需改为对应物理卷

sudo pvcreate /dev/sdb

创建用于docker的vg

sudo vgcreate vgdocker /dev/sdb

创建两个lv,用于data和metadata,最后两个参数指定了该thin pool允许自动扩容能到达的VG百分比

sudo lvcreate --wipesignatures y -n thinpool vgdocker -l 95%VG
sudo lvcreate --wipesignatures y -n thinpoolmeta vgdocker -l 1%VG

将lv转换为thin pool

sudo lvconvert -y --zero n -c 512K \
--thinpool vgdocker/thinpool \
--poolmetadata vgdocker/thinpoolmeta

修改自动扩容配置,其中thin_pool_autoextend_threshold为自动扩容阈值,thin_pool_autoextend_percent为每次扩容的比率

sudo cat <<EOF > /etc/lvm/profile/vgdocker-thinpool.profile
activation {
  thin_pool_autoextend_threshold=80
  thin_pool_autoextend_percent=20
}
EOF

应用LVM profile

sudo lvchange --metadataprofile vgdocker-thinpool vgdocker/thinpool

启动lv监控来实现自动扩容

sudo lvs -o+seg_monitor

移动旧docker数据以便恢复

mkdir /var/lib/docker.bak
mv /var/lib/docker/* /var/lib/docker.bak

修改/etc/docker/daemon.json(如果该文件不存在则进行创建)

{
    "storage-driver": "devicemapper",
    "storage-opts": [
    "dm.thinpooldev=/dev/mapper/vgdocker-thinpool",
    "dm.use_deferred_removal=true",
    "dm.use_deferred_deletion=true"
    ]
}

启动docker服务

sudo systemctl start docker

确认已使用pool

docker info | grep Pool

确认无问题后删除旧docker数据

rm -rf /var/lib/docker.bak

逻辑卷的扩容
扩展vg,/dev/sdc为新加的物理卷

sudo vgextend docker /dev/sdc

扩展lv,并通过docker info命令确认扩容情况

sudo lvextend -l+100%FREE -n vgdocker/thinpool
docker info

重启后的激活
若系统重启后发现docker无法启动,能需要使用以下命令重新激活lv

sudo lvchange -ay vgdocker/thinpool

重新安装kubernetes时需要执行的命令

执行以下命令来清除kubernetes的配置以及桥接网卡等

kubeadm reset -f
systemctl stop kubelet
systemctl stop docker
rm -rf /var/lib/cni/
rm -rf /var/lib/kubelet/*
rm -rf /etc/cni/
ifconfig cni0 down
ifconfig flannel.1 down
ifconfig docker0 down
ip link delete cni0
ip link delete flannel.1
systemctl start docker

使用kubeadm部署单master节点Kubernetes v1.11.1

上次安装Kubernetes v1.11.0遇到问题,kubeadm init总过不去,现在v1.11.1版本发布了,试了一下,可以正常安装了,安装过程与v1.10.5基本相同,可以参考原文章Kubernetes v1.10.5安装
1. 主机配置
简单写在一起了,包含关闭firewall、swap、selinux,开启内核参数,启用bash补全

sudo systemctl stop firewalld
sudo systemctl disable firewalld
sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo setenforce 0
sed -i s/SELINUX=enforcing/SELINUX=disabled/g /etc/selinux/config
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system
yum install bash-completion -y
echo "source <(kubectl completion bash)" >> ~/.bashrc

2. 下载Kubernetes镜像
从mirrorgooglecontainers源下载镜像(CoreDNS已正式上线,代替原来kube-dns组件)

docker pull mirrorgooglecontainers/kube-apiserver-amd64:v1.11.1
docker pull mirrorgooglecontainers/kube-scheduler-amd64:v1.11.1
docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.11.1
docker pull mirrorgooglecontainers/kube-controller-manager-amd64:v1.11.1
docker pull mirrorgooglecontainers/pause-amd64:3.1
docker pull mirrorgooglecontainers/etcd-amd64:3.2.18
docker pull coredns/coredns:1.1.3
docker pull quay.io/coreos/flannel:v0.10.0-amd64

将镜像标记为k8s.gcr.io的名称

docker tag mirrorgooglecontainers/kube-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1
docker tag mirrorgooglecontainers/kube-scheduler-amd64:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1
docker tag mirrorgooglecontainers/kube-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker tag mirrorgooglecontainers/kube-controller-manager-amd64:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
docker tag mirrorgooglecontainers/pause-amd64:3.1 k8s.gcr.io/pause:3.1
docker tag mirrorgooglecontainers/etcd-amd64:3.2.18 k8s.gcr.io/etcd-amd64:3.2.18
docker tag coredns/coredns:1.1.3 k8s.gcr.io/coredns:1.1.3

删除无用镜像名

docker images | grep mirrorgooglecontainers | awk '{print "docker rmi "$1":"$2}' | sh
docker rmi coredns/coredns:1.1.3

安装、配置kubelet

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet-1.11.1 kubeadm-1.11.1 kubectl-1.11.1
systemctl enable kubelet
systemctl start kubelet

3. 初始化k8s集群
指定kubernetes-version版本,由于有墙。token-ttl默认有效为24小时,改为0为永久有效。设置pod-network-cidr为flannel做准备。

kubeadm init --kubernetes-version v1.11.1 --token-ttl 0 \
--pod-network-cidr 10.244.0.0/16

复制admin配置文件

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

配置flannel网络插件

curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

4. 确认kubernetes运行状态

kubectl get pods --all-namespaces

所有容器都运行时即部署完成

NAMESPACE     NAME                                      READY     STATUS    RESTARTS   AGE
kube-system   coredns-78fcdf6894-j2xwq                  1/1       Running   0          1m
kube-system   coredns-78fcdf6894-sn28d                  1/1       Running   0          2m
kube-system   etcd-k8s1.test.local                      1/1       Running   0          1m
kube-system   kube-apiserver-k8s1.test.local            1/1       Running   0          1m
kube-system   kube-controller-manager-k8s1.test.local   1/1       Running   0          1m
kube-system   kube-flannel-ds-amd64-zkgkb               1/1       Running   0          1m
kube-system   kube-proxy-7r8zc                          1/1       Running   0          2m
kube-system   kube-scheduler-k8s1.test.local            1/1       Running   0          1m

使用kubeadm部署单master节点Kubernetes v1.10.5

安装环境: CentOS 7.5,Docker CE 17.03
(本想写1.11.0版本的安装,由于遇到问题,退到1.10.5版本了)

1. 主机配置
关闭防火墙

sudo systemctl stop firewalld
sudo systemctl disable firewalld

关闭swap

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

关闭SELinux

sudo setenforce 0

设置内核参数

cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

添加host记录,如有dns记录则不需要

cat >> /etc/hosts << EOF
192.168.1.101	k8s1 k8s1.test.local
192.168.1.102	k8s2 k8s2.test.local
192.168.1.103	k8s3 k8s3.test.local
EOF

2. 下载Kubernetes镜像
从mirrorgooglecontainers源下载镜像

docker pull mirrorgooglecontainers/kube-apiserver-amd64:v1.10.5
docker pull mirrorgooglecontainers/kube-scheduler-amd64:v1.10.5
docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.10.5
docker pull mirrorgooglecontainers/kube-controller-manager-amd64:v1.10.5
docker pull mirrorgooglecontainers/pause-amd64:3.1
docker pull mirrorgooglecontainers/k8s-dns-kube-dns-amd64:1.14.8
docker pull mirrorgooglecontainers/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker pull mirrorgooglecontainers/k8s-dns-sidecar-amd64:1.14.8
docker pull quay.io/coreos/etcd:v3.1.12
docker pull quay.io/coreos/flannel:v0.10.0-amd64
docker pull coredns/coredns:1.0.6

将镜像标记为k8s.gcr.io的名称

docker tag mirrorgooglecontainers/kube-apiserver-amd64:v1.10.5 k8s.gcr.io/kube-apiserver-amd64:v1.10.5
docker tag mirrorgooglecontainers/kube-scheduler-amd64:v1.10.5 k8s.gcr.io/kube-scheduler-amd64:v1.10.5
docker tag mirrorgooglecontainers/kube-proxy-amd64:v1.10.5 k8s.gcr.io/kube-proxy-amd64:v1.10.5
docker tag mirrorgooglecontainers/kube-controller-manager-amd64:v1.10.5 k8s.gcr.io/kube-controller-manager-amd64:v1.10.5
docker tag mirrorgooglecontainers/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
docker tag mirrorgooglecontainers/k8s-dns-kube-dns-amd64:1.14.8 k8s.gcr.io/k8s-dns-kube-dns-amd64:1.14.8
docker tag mirrorgooglecontainers/k8s-dns-dnsmasq-nanny-amd64:1.14.8 k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.14.8
docker tag mirrorgooglecontainers/k8s-dns-sidecar-amd64:1.14.8 k8s.gcr.io/k8s-dns-sidecar-amd64:1.14.8
docker tag quay.io/coreos/etcd:v3.1.12 k8s.gcr.io/etcd-amd64:3.1.12

删除无用镜像名

docker images | grep mirrorgooglecontainers | awk '{print "docker rmi "$1":"$2}' | sh

安装、配置kubelet

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum install -y kubelet-1.10.5 kubeadm-1.10.5 kubectl-1.10.5
systemctl enable kubelet
sed -i "s/cgroup-driver=systemd/cgroup-driver=cgroupfs/g" /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
systemctl daemon-reload
systemctl start kubelet

安装bash自动补全

yum install bash-completion -y
echo "source <(kubectl completion bash)" >> ~/.bashrc

3. 初始化k8s集群

kubeadm init --kubernetes-version v1.10.5 --token-ttl 0 --pod-network-cidr 10.244.0.0/16

指定kubernetes-version版本,由于有墙
token-ttl默认有效为24小时,改为0为永久有效
设置pod-network-cidr为flannel做准备

可以在初始化集群时使用CoreDNS代替kube-dns

kubeadm init --kubernetes-version v1.10.5 --token-ttl 0 \
--pod-network-cidr 10.244.0.0/16 --feature-gates CoreDNS=true
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果是root用户也可以使用以下命令

export KUBECONFIG=/etc/kubernetes/admin.conf

配置flannel网络插件

curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
kubectl apply -f kube-flannel.yml

4. 确认kubernetes运行状态

kubectl get pods --all-namespaces

所有容器都运行时即部署完成

NAMESPACE     NAME                                    READY     STATUS    RESTARTS   AGE
kube-system   etcd-k8s1                               1/1       Running   1          6m
kube-system   kube-apiserver-k8s1                     1/1       Running   1          6m
kube-system   kube-controller-manager-k8s1            1/1       Running   1          6m
kube-system   kube-dns-86f4d74b45-lmcqv               3/3       Running   3          6m
kube-system   kube-flannel-ds-amd64-g6g66             1/1       Running   1          6m
kube-system   kube-proxy-rqnhh                        1/1       Running   1          6m
kube-system   kube-scheduler-k8s1                     1/1       Running   1          6m

5. 加入Node节点
Node节点的配置与Master基本相同(参见1、2节),只是所需的docker image少一些,只需要kube-proxy-amd64:v1.10.5、pause-amd64:3.1和flannel:v0.10.0-amd64(如果需要)

docker pull mirrorgooglecontainers/kube-proxy-amd64:v1.10.5
docker pull mirrorgooglecontainers/pause-amd64:3.1
docker pull quay.io/coreos/flannel:v0.10.0-amd64
docker tag mirrorgooglecontainers/kube-proxy-amd64:v1.10.5 k8s.gcr.io/kube-proxy-amd64:v1.10.5
docker tag mirrorgooglecontainers/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
docker images | grep mirrorgooglecontainers | awk '{print "docker rmi "$1":"$2}' | sh

然后使用kubeadm join加入k8s集群,该命令会在Master执行kubeadm init最后生成,如果没有记录下来可用以下命令生成

kubeadm token create --print-join-command

再使用kubectl get node命令确认节点加入情况

NAME              STATUS    ROLES     AGE       VERSION
k8s1.test.local   Ready     master    5m        v1.10.5
k8s2.test.local   Ready     <none>    3m        v1.10.5
k8s3.test.local   Ready     <none>    3m        v1.10.5