2.9 安装 `ceph-rbd` 存储类 `StorageClass`

前面系统已经预留了 50G 的磁盘空间用于 ceph 安装的，ceph 的安装采用官方的 ceph-ansible 进行安装。

ceph-ansible

部署文档

2.9.1 版本说明

stable-3.0 分支支持的 Ceph 版本为 jewel 和 luminous，该分支需要 Ansible 版本为 2.4。
stable-3.1 分支支持的 Ceph 版本为 luminous 和 mimic，该分支需要 Ansible 版本为 2.4。
stable-3.2 分支支持的 Ceph 版本为 luminous 和 mimic，该分支需要 Ansible 版本为 2.6。
stable-4.0 分支支持的 Ceph 版本为 nautilus，该分支需要 Ansible 版本为 2.8。
stable-5.0 分支支持的 Ceph 版本为 octopus，该分支需要 Ansible 版本为 2.9。
master 分支支持的 Ceph 主分支版本，该分支需要 Ansible 版本为 2.9。

[!Note] stable-3.0 stable-3.1 分支已经废弃并不在维护。

这里安装常用的 nautilus 版本

2.9.2 安装 `ceph` 集群

下载安装脚本

wget https://github.com/ceph/ceph-ansible/archive/refs/tags/v4.0.71.tar.gz
tar xf v4.0.71.tar.gz
cd ceph-ansible-4.0.71

安装 `ansible`

yum -y install python-pip
pip install pip==20.3.4
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

[!Note] python2 版本的 pip 最大支持 20.3.4

'ImportError: No module named setuptools_rust': 升级 pip 版本

配置 `ceph`

创建 hosts 文件

cat > ceph-hosts <<EOF
[mons]
k8s-m1
k8s-m2
k8s-m3

[osds]
k8s-m1
k8s-m2
k8s-m3

[mgrs]
k8s-m1
k8s-m2
k8s-m3

[clients]
k8s-m1
k8s-m2
k8s-m3
EOF

修改配置

cd group_vars
cp all.yml.sample all.yml
grep -Ev '^#|^$' all.yml
---
dummy:
configure_firewall: False # 由于跟kubernetes是同一节点，防火墙需要关闭
ceph_repository_type: repository # <== repository表示使用新的库，不使用官方仓库
ceph_origin: repository # <== 安装方式，repository值表示指定使用仓库安装，
ceph_repository: community # <== 选择使用库的来源类型,community为免费社区版
ceph_mirror: http://mirrors.tuna.tsinghua.edu.cn/ceph/ # <== ceph仓库的路径
ceph_stable_key: http://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc # <== ceph key
ceph_stable_release: nautilus # <== 安装的ceph版本 nautilus为14，最新的stable版本
ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}" # <== yum仓库地址
monitor_interface: eth0 # <== 网卡名
public_network: 192.168.100.0/24 # <== 公共网段
cluster_network: 192.168.100.0/24 # <== 集群网段
osd_objectstore: bluestore # <== ceph存储引擎
dashboard_enabled: False # <== 是否启用dashboard

cp osds.yml.sample osds.yml
grep -Ev '^#|^$' osds.yml
---
dummy:
devices:
- /dev/vdb # <== osd用的磁盘，这里只有一块，可以通过fdisk -l查看，我这里使用 kvm 
           # 创建的虚拟机，显示是 vdb，如果用 virtualbox，vmware 部署可能是 sdb

cp clients.yml.sample clients.yml
cp mons.yml.sample mons.yml
cp mgrs.yml.sample mgrs.yml

cd ..
cp site.yml.sample site.yml

部署 `ceph` 集群

ansible-playbook -i ceph-hosts site.yml

输出如下

...
INSTALLER STATUS ****************************************************************************************************************************************************************************
Install Ceph Monitor           : Complete (0:02:28)
Install Ceph Manager           : Complete (0:03:59)
Install Ceph OSD               : Complete (0:04:00)
Install Ceph Client            : Complete (0:01:45)
Install Ceph Crash             : Complete (0:00:36)

Thursday 27 January 2022  23:43:57 +0800 (0:00:00.072)       0:19:13.053 ******
===============================================================================
ceph-common : install redhat ceph packages ----------------------------------------------------------------------------------------------------------------------------------------- 204.82s
ceph-osd : use ceph-volume lvm batch to create bluestore osds ----------------------------------------------------------------------------------------------------------------------- 74.48s
install ceph-mgr packages on RedHat or SUSE ----------------------------------------------------------------------------------------------------------------------------------------- 70.07s
gather facts ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 51.19s
ceph-config : look up for ceph-volume rejected devices ------------------------------------------------------------------------------------------------------------------------------ 47.89s
ceph-config : look up for ceph-volume rejected devices ------------------------------------------------------------------------------------------------------------------------------ 44.33s
ceph-config : look up for ceph-volume rejected devices ------------------------------------------------------------------------------------------------------------------------------ 44.10s
ceph-config : look up for ceph-volume rejected devices ------------------------------------------------------------------------------------------------------------------------------ 42.91s
ceph-mon : fetch ceph initial keys -------------------------------------------------------------------------------------------------------------------------------------------------- 19.54s
ceph-osd : apply operating system tuning -------------------------------------------------------------------------------------------------------------------------------------------- 18.87s
ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created ------------------------------------------------------------------------------------------- 16.92s
ceph-common : install yum plugin priorities ----------------------------------------------------------------------------------------------------------------------------------------- 16.38s
ceph-infra : install chrony --------------------------------------------------------------------------------------------------------------------------------------------------------- 16.05s
ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created ------------------------------------------------------------------------------------------- 14.94s
ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created ------------------------------------------------------------------------------------------- 14.15s
ceph-config : run 'ceph-volume lvm batch --report' to see how many osds are to be created ------------------------------------------------------------------------------------------- 13.23s
ceph-common : configure red hat ceph community repository stable key ---------------------------------------------------------------------------------------------------------------- 11.08s
ceph-mgr : create ceph mgr keyring(s) on a mon node --------------------------------------------------------------------------------------------------------------------------------- 10.55s
ceph-mgr : wait for all mgr to be up ------------------------------------------------------------------------------------------------------------------------------------------------- 9.65s
ceph-mon : start the monitor service ------------------------------------------------------------------------------------------------------------------------------------------------- 8.68s

检查集群状态

ceph -s

输出如下

  cluster:
    id:     5b83f693-3714-4e00-b0e6-d241dd1c4033
    health: HEALTH_WARN
            mons are allowing insecure global_id reclaim

  services:
    mon: 3 daemons, quorum k8s-m1,k8s-m2,k8s-m3 (age 11m)
    mgr: k8s-m3(active, since 6m), standbys: k8s-m1, k8s-m2
    osd: 3 osds: 3 up (since 2m), 3 in (since 2m)

  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 147 GiB / 150 GiB avail
    pgs:

ceph 集群已经安装好了，接下来床将存储类。

2.9.3 使用 `ceph-rbd` 作为存储类

创建存储池

ceph osd pool create kube 128 128
pool 'kube' created

# 获取认证信息
ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=kube' -o ceph.client.kube.keyring

创建 `rbd-provisioner`

cat > rbd-provisioner.yaml <<EOF
kind: ClusterRole 
apiVersion: rbac.authorization.k8s.io/v1 
metadata: 
  name: rbd-provisioner 
rules: 
  - apiGroups: [""] 
    resources: ["persistentvolumes"] 
    verbs: ["get", "list", "watch", "create", "delete"] 
  - apiGroups: [""] 
    resources: ["persistentvolumeclaims"] 
    verbs: ["get", "list", "watch", "update"] 
  - apiGroups: ["storage.k8s.io"] 
    resources: ["storageclasses"] 
    verbs: ["get", "list", "watch"] 
  - apiGroups: [""] 
    resources: ["events"] 
    verbs: ["create", "update", "patch"] 
  - apiGroups: [""] 
    resources: ["services"] 
    resourceNames: ["kube-dns","coredns"] 
    verbs: ["list", "get"] 
--- 
kind: ClusterRoleBinding 
apiVersion: rbac.authorization.k8s.io/v1 
metadata: 
  name: rbd-provisioner 
subjects: 
  - kind: ServiceAccount 
    name: rbd-provisioner 
    namespace: default 
roleRef: 
  kind: ClusterRole 
  name: rbd-provisioner 
  apiGroup: rbac.authorization.k8s.io 
--- 
apiVersion: rbac.authorization.k8s.io/v1 
kind: Role 
metadata: 
  name: rbd-provisioner 
rules: 
- apiGroups: [""] 
  resources: ["secrets"] 
  verbs: ["get"] 
- apiGroups: [""] 
  resources: ["endpoints"] 
  verbs: ["get", "list", "watch", "create", "update", "patch"] 
--- 
apiVersion: rbac.authorization.k8s.io/v1 
kind: RoleBinding 
metadata: 
  name: rbd-provisioner 
roleRef: 
  apiGroup: rbac.authorization.k8s.io 
  kind: Role 
  name: rbd-provisioner 
subjects: 
  - kind: ServiceAccount 
    name: rbd-provisioner 
    namespace: default 
--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: rbd-provisioner 
spec: 
  selector:
    matchLabels:
      app: rbd-provisioner 
  replicas: 2
  strategy: 
    type: Recreate 
  template: 
    metadata: 
      labels: 
        app: rbd-provisioner 
    spec: 
      containers: 
      - name: rbd-provisioner 
        # 该镜像3年没更新了
        image: quay.io/external_storage/rbd-provisioner:v2.1.1-k8s1.11
        imagePullPolicy: IfNotPresent
        env: 
        - name: PROVISIONER_NAME 
          value: ceph.com/rbd 
      serviceAccount: rbd-provisioner 
--- 
apiVersion: v1 
kind: ServiceAccount 
metadata: 
  name: rbd-provisioner
EOF

kubectl apply -f rbd-provisioner.yaml
clusterrole.rbac.authorization.k8s.io/rbd-provisioner created
clusterrolebinding.rbac.authorization.k8s.io/rbd-provisioner created
role.rbac.authorization.k8s.io/rbd-provisioner created
rolebinding.rbac.authorization.k8s.io/rbd-provisioner created
deployment.apps/rbd-provisioner created
serviceaccount/rbd-provisioner created

为 `kubelet` 提供 `rbd` 命令

创建 secret

CEPH_ADMIN_SECRET=$(ceph auth get-key client.admin | base64)

将获取的 key 作为 secret 的 key

cat > ceph-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
  namespace: default
data:
  key: ${CEPH_ADMIN_SECRET}
type: kubernetes.io/rbd
EOF

创建 secret

kubectl apply -f ceph-secret.yaml
secret/ceph-secret created

创建 `user-secret`

CEPH_USER_SECRET=$(ceph auth get-key client.kube | base64)

cat > ceph-user-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ceph-user-secret
  namespace: default
data:
  key: ${CEPH_USER_SECRET}
type: kubernetes.io/rbd
EOF

kubectl apply -f ceph-user-secret.yaml
secret/ceph-user-secret created

创建 `storageclass`

cat > ceph-storageclass.yaml <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-rbd
  namespace: default
  annotations:
     storageclass.beta.kubernetes.io/is-default-class: "true" # <= 设置为默认存储类
provisioner: ceph.com/rbd
reclaimPolicy: Retain
parameters:
  monitors: 192.168.122.10:6789,192.168.122.20:6789,192.168.122.30:6789 # <== monitor节点，','分隔
  adminId: admin
  adminSecretName: ceph-secret   # <== 前面创建的secret，type必须为`kubernetes.io/rbd`
  adminSecretNamespace: default # <== secret所在名称空间
  pool: kube         # <== 在mon节点创建的存储池
  fsType: xfs        # <== 文件系统类型
  userId: kube       # <== 前面使用`ceph auth get-or-create`创建的用户
  userSecretName: ceph-user-secret
  imageFormat: "2"
  imageFeatures: "layering"
EOF

kubectl apply -f ceph-storageclass.yaml
storageclass.storage.k8s.io/ceph-rbd created

查看 `storageclass`

kubectl get sc

输出如下

NAME                 PROVISIONER    RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
ceph-rbd (default)   ceph.com/rbd   Retain          Immediate           false                  14s

2.9.4 测试 `ceph-rbd` 存储类

测试 pvc 动态申请

kubectl apply -f - <<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test-claim
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: ceph-busybox
    image: busybox
    command: ["sleep", "60000"]
    volumeMounts:
    - name: ceph-vol1
      mountPath: /usr/share/busybox
      readOnly: false
  volumes:
  - name: ceph-vol1
    persistentVolumeClaim:
      claimName: test-claim
EOF
persistentvolumeclaim/ceph-claim created
pod/ceph-pod1 created

检查 pod 和 pvc 创建情况

kubectl get po test-pod
NAME       READY   STATUS    RESTARTS   AGE
test-pod   1/1     Running   0          38s

kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-claim   Bound    pvc-80db245e-a30e-444d-b9ad-67557e207003   1Gi        RWO            ceph-rbd       4m27s

[!Note] provision "default/test-claim" class "ceph-rbd": unexpected error getting claim reference: selfLink was empty, can't make reference 修改apiserver，添加- --feature-gates=RemoveSelfLink=false

1.20 默认就移除了 SelfLink #94397

测试完正常后删除

kubectl delete pod test-pod
kubectl delete pvc test-claim

2.9 安装 ceph-rbd 存储类 StorageClass

2.9 安装 `ceph-rbd` 存储类 `StorageClass`

2.9.1 版本说明

2.9.2 安装 `ceph` 集群

下载安装脚本

安装 `ansible`

配置 `ceph`

部署 `ceph` 集群

检查集群状态

2.9.3 使用 `ceph-rbd` 作为存储类

创建存储池

创建 `rbd-provisioner`

为 `kubelet` 提供 `rbd` 命令

创建 `user-secret`

创建 `storageclass`

查看 `storageclass`

2.9.4 测试 `ceph-rbd` 存储类

测试 pvc 动态申请

测试完正常后删除

results matching ""

No results matching ""

2.9 安装 ceph-rbd 存储类 StorageClass

2.9.1 版本说明

2.9.2 安装 ceph 集群

下载安装脚本

安装 ansible

配置 ceph

部署 ceph 集群

检查集群状态

2.9.3 使用 ceph-rbd 作为存储类

创建存储池

创建 rbd-provisioner

为 kubelet 提供 rbd 命令

创建 user-secret

创建 storageclass

查看 storageclass

2.9.4 测试 ceph-rbd 存储类

测试 pvc 动态申请

测试完正常后删除

results matching ""

No results matching ""

2.9 安装 `ceph-rbd` 存储类 `StorageClass`

2.9.2 安装 `ceph` 集群

安装 `ansible`

配置 `ceph`

部署 `ceph` 集群

2.9.3 使用 `ceph-rbd` 作为存储类

创建 `rbd-provisioner`

为 `kubelet` 提供 `rbd` 命令

创建 `user-secret`

创建 `storageclass`

查看 `storageclass`

2.9.4 测试 `ceph-rbd` 存储类