跳转至

Kubernetes 集群-更新证书


2020-06-02 by dongnan

问题描述

使用kubeadm创建的K8S集群默认证书有效期一年,一旦证书过期使用kubectl出现如下提示:

kubectl get pod

Unable to connect to the server: x509: certificate has expired or is not yet valid

检查证书:

kubeadm alpha certs check-expiration

[check-expiration] Reading configuration from the cluster...
#...省略
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 03, 2020 04:59 UTC   <invalid>                               no      
apiserver                  May 03, 2020 04:59 UTC   <invalid>       ca                      no      
apiserver-etcd-client      May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
apiserver-kubelet-client   May 03, 2020 04:59 UTC   <invalid>       ca                      no      
controller-manager.conf    May 03, 2020 04:59 UTC   <invalid>                               no      
etcd-healthcheck-client    May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
etcd-peer                  May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
etcd-server                May 03, 2020 04:59 UTC   <invalid>       etcd-ca                 no      
front-proxy-client         May 03, 2020 04:59 UTC   <invalid>       front-proxy-ca          no      
scheduler.conf             May 03, 2020 04:59 UTC   <invalid>                               no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 01, 2030 04:59 UTC   8y              no      
etcd-ca                 Apr 01, 2030 04:59 UTC   8y              no

可以看到证书是5月3号过期。

环境描述

在前面的文章中,使用 Kubeadm 部署了 K8S集群环境:

测试的K8S集群由一个Master管理节点、两个Worker计算节点组成。

解决方法

使用 kubeadm alpha certs renew 命令更新证书。注意certs 命令选项仅支持 kubeadm v1.15 及其以上的版本。

更新证书

kubeadm alpha certs renew all

检查证书

kubeadm alpha certs check-expiration

[check-expiration] Reading configuration from the cluster...
#...省略
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 May 03, 2021 02:29 UTC   364d                                    no      
apiserver                  May 03, 2021 02:29 UTC   364d            ca                      no      
apiserver-etcd-client      May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
apiserver-kubelet-client   May 03, 2021 02:29 UTC   364d            ca                      no      
controller-manager.conf    May 03, 2021 02:29 UTC   364d                                    no      
etcd-healthcheck-client    May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
etcd-peer                  May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
etcd-server                May 03, 2021 02:29 UTC   364d            etcd-ca                 no      
front-proxy-client         May 03, 2021 02:29 UTC   364d            front-proxy-ca          no      
scheduler.conf             May 03, 2021 02:29 UTC   364d                                    no

CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Apr 01, 2030 04:59 UTC   8y              no      
etcd-ca                 Apr 01, 2030 04:59 UTC   8y              no      
front-proxy-ca          Apr 01, 2030 04:59 UTC   8y              no

证书的有效期变为:May 03, 2021

重启kubelet服务

systemctl restart kubelet.service

重启docker服务

systemctl restart docker.service

更新 Rancher 证书(可选)

如果K8S集群证书已过期,那么Rancher服务器也会运行报错: x509: certificate has expired or is not yet valid。 我们需要重建 Rancher 容器,它将自动获取K8S集群新证书。

操作步骤

停止容器:

docker stop rancher2

删除容器证书目录:

cd /var/lib/docker/volumes/rancher-data/_data/k3s/server/
# 重命名
mv tls/ tls2020

删除并重新创建容器:

# 删除
docker rm rancher2 
# 创建
docker run -d --restart=unless-stopped \
    --name rancher2 \
    -p 80:80 -p 443:443 \
    -v rancher-data:/var/lib/rancher/ \
    -v rancher-log:/var/log/auditlog \
    -e TZ=Asia/Shanghai \
    -e LANG=en_US.UTF-8 \
    -e CATTLE_SYSTEM_CATALOG=bundled \
    -e AUDIT_LEVEL=3 \
    rancher/rancher:stable --no-cacerts

注意事项:

  • 采用这种方式是为了删除过期的证书目录,重新创建rancher容器会自动拷贝k8s的新证书。
  • 这里创建容器时已经设置了数据持久化的目录 -v rancher-data:/var/lib/rancher/

K8s使用的证书

证书目录

tree /etc/kubernetes/pki/
/etc/kubernetes/pki/
├── apiserver.crt
├── apiserver-etcd-client.crt
├── apiserver-etcd-client.key
├── apiserver.key
├── apiserver-kubelet-client.crt
├── apiserver-kubelet-client.key
├── ca.crt
├── ca.key
├── etcd
│   ├── ca.crt
│   ├── ca.key
│   ├── healthcheck-client.crt
│   ├── healthcheck-client.key
│   ├── peer.crt
│   ├── peer.key
│   ├── server.crt
│   └── server.key
├── front-proxy-ca.crt
├── front-proxy-ca.key
├── front-proxy-client.crt
├── front-proxy-client.key
├── sa.key
└── sa.pub

1 directory, 22 files

Kubernetes 集群根证书

/etc/kubernetes/pki/ca.crt
/etc/kubernetes/pki/ca.key

由此根证书签发的证书有:

kube-apiserver 组件持有的服务端证书
/etc/kubernetes/pki/apiserver.crt
/etc/kubernetes/pki/apiserver.key

kubelet 组件持有的客户端证书
/etc/kubernetes/pki/apiserver-kubelet-client.crt
/etc/kubernetes/pki/apiserver-kubelet-client.key

汇聚层(aggregator)证书

/etc/kubernetes/pki/front-proxy-ca.crt
/etc/kubernetes/pki/front-proxy-ca.key

由此根证书签发的证书只有:

代理端使用的客户端证书, 用作代用户与 kube-apiserver 认证
/etc/kubernetes/pki/front-proxy-client.crt
/etc/kubernetes/pki/front-proxy-client.key

etcd 集群根证书

/etc/kubernetes/pki/etcd/ca.crt
/etc/kubernetes/pki/etcd/ca.key

由此根证书签发机构签发的证书有:

etcd server 持有的服务端证书
/etc/kubernetes/pki/etcd/server.crt
/etc/kubernetes/pki/etcd/server.key

peer 集群中节点互相通信使用的客户端证书
/etc/kubernetes/pki/etcd/peer.crt
/etc/kubernetes/pki/etcd/peer.key

pod 中定义 Liveness 探针使用的客户端证书
/etc/kubernetes/pki/etcd/healthcheck-client.crt
/etc/kubernetes/pki/etcd/healthcheck-client.key

配置在 kube-apiserver 中用来与 etcd server 做双向认证的客户端证书
/etc/kubernetes/pki/apiserver-etcd-client.crt
/etc/kubernetes/pki/apiserver-etcd-client.key

Serveice Account秘钥

这组的密钥对儿仅提供给kube-controller-manager 使用, kube-controller-manager 通过 sa.keytoken 进行签名, Master 节点通过公钥 sa.pub 进行签名的验证:

/etc/kubernetes/pki/sa.key
/etc/kubernetes/pki/sa.pub

kubeadm 创建的集群, kube-proxyflannelcoreDNS是以 pod形式运行的,在pod中, 直接使用 service accountkube-apiserver 进行认证,此时就不需要再单独为 kube-proxy 创建证书 。

参考

欢迎关注微信公众号: 运维录

Back to top