背景

使用 kubeadm 安装的证书 除了 CA 证书是10年之外，其他证书的默认有效期为1年，当使用 kubeadm upgrade 进行升级时，证书会重新颁发。但有些场景比如内网环境，又没有升级，也没有使用 kubeadm alpha certs renew 重新续签导致。

错误信息

kubectl 执行报错

1kubectl get pods -o wide
2The connection to the server <apiserver_advertise_ip>:6443 was refused - did you specify the right host or port?

kubelet 状态

1systemctl status kubelet
2● kubelet.service - kubelet: The Kubernetes Node Agent
3   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
4  Drop-In: /usr/lib/systemd/system/kubelet.service.d
5           └─10-kubeadm.conf
6   Active: activating (auto-restart) (Result: exit-code) since Mon 2020-06-01 08:51:47 +0530; 3s ago
7     Docs: https://kubernetes.io/docs/
8  Process: 14027 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=255)
9 Main PID: 14027 (code=exited, status=255)

kubelet 错误日志

 1journalctl | grep kubelet
 2Jun 01 08:42:53 <node_name> systemd[1]: Started kubelet: The Kubernetes Node Agent.
 3Jun 01 08:42:54 <node_name> kubelet[3653]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
 4Jun 01 08:42:54 <node_name> kubelet[3653]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
 5Jun 01 08:42:54 <node_name> kubelet[3653]: I0601 08:42:54.224801    3653 server.go:417] Version: v1.14.1
 6Jun 01 08:42:54 <node_name> kubelet[3653]: I0601 08:42:54.226118    3653 plugins.go:103] No cloud provider specified.
 7Jun 01 08:42:54 <node_name> kubelet[3653]: I0601 08:42:54.226152    3653 server.go:754] Client rotation is on, will bootstrap in background
 8Jun 01 08:42:54 <node_name> kubelet[3653]: E0601 08:42:54.232397    3653 bootstrap.go:264] Part of the existing bootstrap client certificate is expired: 2020-04-11 02:01:22 +0000 UTC
 9Jun 01 08:42:54 <node_name> kubelet[3653]: F0601 08:42:54.234118    3653 server.go:265] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
10Jun 01 08:42:54 <node_name> systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
11Jun 01 08:42:54 <node_name> systemd[1]: Unit kubelet.service entered failed state.
12Jun 01 08:42:54 <node_name> systemd[1]: kubelet.service failed.
13Jun 01 08:43:04 <node_name> systemd[1]: kubelet.service holdoff time over, scheduling restart.
14Jun 01 08:43:04 <node_name> systemd[1]: Stopped kubelet: The Kubernetes Node Agent.

查看证书有效期

1.15-

1openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text
2openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -text

1.15+

1kubeadm alpha certs check-expiration

手动更新证书

1.15-

该方法在1.15+上面也可以用

获取 kubeadm 配置

1kubeadm config view > <k8_specs_directory>/kubeadm_config.yaml

备份证书和私钥

一定要移走，文件存在的话，不会重新生成

1 cd /etc/kubernetes/pki/
2mkdir -p ~/tmp/BACKUP_etc_kubernetes_pki/etcd/
3sudo mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/tmp/BACKUP_etc_kubernetes_pki/
4sudo mv {etcd/healthcheck-client.crt,etcd/healthcheck-client.key,etcd/peer.crt,etcd/peer.key,etcd/server.crt,etcd/server.key} ~/tmp/BACKUP_etc_kubernetes_pki/etcd/

生成新的证书和私钥

 1sudo kubeadm init phase certs all --config <k8_specs_directory>/kubeadm_config.yaml
 2[certs] Using certificateDir folder "/etc/kubernetes/pki"
 3[certs] Generating "front-proxy-ca" certificate and key
 4[certs] Generating "front-proxy-client" certificate and key
 5[certs] Using existing etcd/ca certificate authority
 6[certs] Generating "etcd/peer" certificate and key
 7[certs] etcd/peer serving cert is signed for DNS names [<apiserver_advertise_host> localhost] and IPs [<apiserver_advertise_ip> 127.0.0.1 ::1]
 8[certs] Generating "etcd/server" certificate and key
 9[certs] etcd/server serving cert is signed for DNS names [<apiserver_advertise_host> localhost] and IPs [<apiserver_advertise_ip> 127.0.0.1 ::1]
10[certs] Generating "etcd/healthcheck-client" certificate and key
11[certs] Generating "apiserver-etcd-client" certificate and key
12[certs] Using existing ca certificate authority
13[certs] Generating "apiserver-kubelet-client" certificate and key
14[certs] Generating "apiserver" certificate and key
15[certs] apiserver serving cert is signed for DNS names [<apiserver_advertise_host> kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [<k8_subnet_and_host_ips>]
16[certs] Using the existing "sa" key

备份当前配置

1cd /etc/kubernetes/
2mkdir -p ~/tmp/BACKUP_etc_kubernetes
3sudo mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/tmp/BACKUP_etc_kubernetes/

生成新的配置

1sudo kubeadm init phase kubeconfig all --config <k8_specs_directory>/kubeadm_config.yaml
2[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
3[kubeconfig] Writing "admin.conf" kubeconfig file
4[kubeconfig] Writing "kubelet.conf" kubeconfig file
5[kubeconfig] Writing "controller-manager.conf" kubeconfig file
6[kubeconfig] Writing "scheduler.conf" kubeconfig file

重启服务器

1sudo systemctl reboot

拷贝 kubeconfig 文件

1mkdir -p ~/tmp/BACKUP_home_.kube/
2cp -r ~/.kube/* ~/tmp/BACKUP_home_.kube/.
3sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

1.15+

1kubeadm alpha certs renew