Kubernetes 集群-更新证书

2020-06-02 by dongnan

问题描述

使用kubeadm创建的K8S集群默认证书有效期一年，一旦证书过期使用kubectl出现如下提示：

kubectl get pod

Unable to connect to the server: x509: certificate has expired or is not yet valid

检查证书：

kubeadm alpha certs check-expiration

[check-expiration] Reading configuration from the cluster...
#...省略
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 03, 2020 04:59 UTC   <invalid>                               no      
apiserver                  May 03, 2020 04:59 UTC   <invalid>       ca                      no      
apiserver-etcd-client      May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   May 03, 2020 04:59 UTC   <invalid>       ca                      no      
controller-manager.conf    May 03, 2020 04:59 UTC   <invalid>                               no      
etcd-healthcheck-client    May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
etcd-server                May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         May 03, 2020 04:59 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             May 03, 2020 04:59 UTC   <invalid>                               no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 01, 2030 04:59 UTC   8y              no      
etcd-ca                 Apr 01, 2030 04:59 UTC   8y              no

可以看到证书是5月3号过期。

环境描述

在前面的文章中，使用 Kubeadm 部署了 K8S集群环境：

测试的K8S集群由一个Master管理节点、两个Worker计算节点组成。

解决方法

使用 kubeadm alpha certs renew 命令更新证书。注意certs 命令选项仅支持 kubeadm v1.15 及其以上的版本。

更新证书

kubeadm alpha certs renew all

检查证书

kubeadm alpha certs check-expiration

[check-expiration] Reading configuration from the cluster...
#...省略
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 03, 2021 02:29 UTC   364d                                    no      
apiserver                  May 03, 2021 02:29 UTC   364d            ca                      no      
apiserver-etcd-client      May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   May 03, 2021 02:29 UTC   364d            ca                      no      
controller-manager.conf    May 03, 2021 02:29 UTC   364d                                    no      
etcd-healthcheck-client    May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
etcd-peer                  May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
etcd-server                May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
front-proxy-client         May 03, 2021 02:29 UTC   364d            front-proxy-ca          no      
scheduler.conf             May 03, 2021 02:29 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 01, 2030 04:59 UTC   8y              no      
etcd-ca                 Apr 01, 2030 04:59 UTC   8y              no      
front-proxy-ca          Apr 01, 2030 04:59 UTC   8y              no

证书的有效期变为：May 03, 2021。

重启kubelet服务

systemctl restart kubelet.service

重启docker服务

systemctl restart docker.service

更新 Rancher 证书(可选)

如果K8S集群证书已过期，那么Rancher服务器也会运行报错： x509: certificate has expired or is not yet valid。我们需要重建 Rancher 容器，它将自动获取K8S集群新证书。

操作步骤

停止容器：

docker stop rancher2

删除容器证书目录：

cd /var/lib/docker/volumes/rancher-data/_data/k3s/server/
# 重命名
mv tls/ tls2020

删除并重新创建容器：

# 删除
docker rm rancher2 
# 创建
docker run -d --restart=unless-stopped \
    --name rancher2 \
    -p 80:80 -p 443:443 \
    -v rancher-data:/var/lib/rancher/ \
    -v rancher-log:/var/log/auditlog \
    -e TZ=Asia/Shanghai \
    -e LANG=en_US.UTF-8 \
    -e CATTLE_SYSTEM_CATALOG=bundled \
    -e AUDIT_LEVEL=3 \
    rancher/rancher:stable --no-cacerts

注意事项：

采用这种方式是为了删除过期的证书目录，重新创建rancher容器会自动拷贝k8s的新证书。
这里创建容器时已经设置了数据持久化的目录 -v rancher-data:/var/lib/rancher/。

K8s使用的证书

证书目录

tree /etc/kubernetes/pki/
/etc/kubernetes/pki/
├── apiserver.crt
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
├── apiserver.key
├── apiserver-kubelet-client.crt
├── apiserver-kubelet-client.key
├── ca.crt
├── ca.key
├── etcd
│   ├── ca.crt
│   ├── ca.key
│   ├── healthcheck-client.crt
│   ├── healthcheck-client.key
│   ├── peer.crt
│   ├── peer.key
│   ├── server.crt
│   └── server.key
├── front-proxy-ca.crt
├── front-proxy-ca.key
├── front-proxy-client.crt
├── front-proxy-client.key
├── sa.key
└── sa.pub

1 directory, 22 files

Kubernetes 集群根证书

/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key

由此根证书签发的证书有:

kube-apiserver 组件持有的服务端证书
/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver.key

kubelet 组件持有的客户端证书
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.key

汇聚层(aggregator)证书

/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key

由此根证书签发的证书只有:

代理端使用的客户端证书, 用作代用户与 kube-apiserver 认证
/etc/kubernetes/pki/front-proxy-client.crt
/etc/kubernetes/pki/front-proxy-client.key

etcd 集群根证书

/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key

由此根证书签发机构签发的证书有:

etcd server 持有的服务端证书
/etc/kubernetes/pki/etcd/server.crt
/etc/kubernetes/pki/etcd/server.key

peer 集群中节点互相通信使用的客户端证书
/etc/kubernetes/pki/etcd/peer.crt
/etc/kubernetes/pki/etcd/peer.key

pod 中定义 Liveness 探针使用的客户端证书
/etc/kubernetes/pki/etcd/healthcheck-client.crt
/etc/kubernetes/pki/etcd/healthcheck-client.key

配置在 kube-apiserver 中用来与 etcd server 做双向认证的客户端证书
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key

Serveice Account秘钥

这组的密钥对儿仅提供给kube-controller-manager 使用， kube-controller-manager 通过 sa.key 对 token 进行签名, Master 节点通过公钥 sa.pub 进行签名的验证：

/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub

kubeadm 创建的集群, kube-proxy、flannel、coreDNS是以 pod形式运行的，在pod中，直接使用 service account 与 kube-apiserver 进行认证，此时就不需要再单独为 kube-proxy 创建证书。