```sh Cephadm通过manager daemon SSH连接到主机部署和管理Ceph群集，以添加，删除或更新Ceph daemon containers。它不依赖于诸如Ansible，Rook或Salt的外部配置或编排工具。 Cephadm管理Ceph集群的整个生命周期。它首先在一个节点上引导一个微小的Ceph集群(one monitor and one manager)，然后自动将集群扩展到多个主机节点，并提供所有Ceph守护程序和服务。这可以通过Ceph命令行界面（CLI）或仪表板（GUI）执行。Cephadm是Octopus v15.2.0版本中的新增功能，并且不支持旧版本的Ceph。 ``` 节点规划: ```sh +--------+---------------+------+---------------------------------+ | Host | ip | Disk | Role | +--------+---------------+------+---------------------------------+ | test01 | 172.16.100.1 | sdc | cephadm,mon,mgr,mds,osd | | test02 | 172.16.100.2 | sdc | mon,mgr,osd | | test11 | 172.16.100.11 | sdb | mon,mds,osd | +--------+---------------+------+---------------------------------+ ``` ##### 安装说明 ```sh ceph版本：15.2.14 octopus(stable) python: 3+ ``` #### 安装docker [参考CentOS7安装Docker-CE](https://ynotes.cn/blog/article_detail/118) #### 安装cephadm(`test01`) ```sh curl --silent --remote-name --location https://github.com/ceph/ceph/raw/octopus/src/cephadm/cephadm chmod +x cephadm ./cephadm add-repo --release octopus ./cephadm install ``` cephadm引导ceph集群 ```sh cephadm bootstrap --mon-ip 172.16.100.1 ``` ```sh Creating directory /etc/ceph for ceph.conf Verifying podman|docker is present... Verifying lvm2 is present... Verifying time synchronization is in place... Unit chronyd.service is enabled and running Repeating the final host check... podman|docker (/usr/bin/docker) is present systemctl is present lvcreate is present Unit chronyd.service is enabled and running Host looks OK Cluster fsid: 797b1aa4-16cf-11ec-a787-00505693fd22 Verifying IP 172.16.100.1 port 3300 ... Verifying IP 172.16.100.1 port 6789 ... Mon IP 172.16.100.1 is in CIDR network 172.16.100.0/24 Pulling container image docker.io/ceph/ceph:v15... Extracting ceph user uid/gid from container image... Creating initial keys... Creating initial monmap... Creating mon... Waiting for mon to start... Waiting for mon... mon is available Assimilating anything we can from ceph.conf... Generating new minimal ceph.conf... Restarting the monitor... Setting mon public_network... Creating mgr... Verifying port 9283 ... Wrote keyring to /etc/ceph/ceph.client.admin.keyring Wrote config to /etc/ceph/ceph.conf Waiting for mgr to start... Waiting for mgr... mgr not available, waiting (1/10)... mgr not available, waiting (2/10)... mgr not available, waiting (3/10)... mgr not available, waiting (4/10)... mgr not available, waiting (5/10)... mgr not available, waiting (6/10)... mgr is available Enabling cephadm module... Waiting for the mgr to restart... Waiting for Mgr epoch 5... Mgr epoch 5 is available Setting orchestrator backend to cephadm... Generating ssh key... Wrote public SSH key to to /etc/ceph/ceph.pub Adding key to root@localhost's authorized_keys... Adding host test01... Deploying mon service with default placement... Deploying mgr service with default placement... Deploying crash service with default placement... Enabling mgr prometheus module... Deploying prometheus service with default placement... Deploying grafana service with default placement... Deploying node-exporter service with default placement... Deploying alertmanager service with default placement... Enabling the dashboard module... Waiting for the mgr to restart... Waiting for Mgr epoch 13... Mgr epoch 13 is available Generating a dashboard self-signed certificate... Creating initial admin user... Fetching dashboard port number... Ceph Dashboard is now available at: URL: https://ceph1:8443/ User: admin Password: hj49zqnfm2 You can access the Ceph CLI with: sudo /usr/sbin/cephadm shell --fsid 797b1aa4-16cf-11ec-a787-00505693fd22 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring Please consider enabling telemetry to help improve Ceph: ceph telemetry on For more information see: https://docs.ceph.com/docs/master/mgr/telemetry/ Bootstrap complete. ``` ```sh 在本地主机上为新集群创建monitor 和 manager daemon守护程序。 为Ceph集群生成一个新的SSH密钥，并将其添加到root用户的/root/.ssh/authorized_keys文件中。 将与新群集进行通信所需的最小配置文件保存到/etc/ceph/ceph.conf。 向/etc/ceph/ceph.client.admin.keyring写入client.admin管理（特权！）secret key的副本。 将public key的副本写入/etc/ceph/ceph.pub ``` 查看节点运行的docker容器 ```sh $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 3a707133d00e ceph/ceph:v15 "/usr/bin/ceph-mds -…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-mds.cephfs.test01.kqizcm 36ada96953ea ceph/ceph:v15 "/usr/bin/ceph-osd -…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-osd.0 08aa352c61f1 ceph/ceph:v15 "/usr/bin/ceph-mgr -…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-mgr.test01.soadiy 47be185fc946 prom/node-exporter:v0.18.1 "/bin/node_exporter …" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-node-exporter.test01 dad697dd6669 ceph/ceph-grafana:6.7.4 "/bin/sh -c 'grafana…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-grafana.test01 754ba4d9e132 prom/alertmanager:v0.20.0 "/bin/alertmanager -…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-alertmanager.test01 2b7e3664edbf ceph/ceph:v15 "/usr/bin/ceph-crash…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-crash.test01 c7cf9f5502de ceph/ceph:v15 "/usr/bin/ceph-mon -…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-mon.test01 ba3958a9dec8 prom/prometheus:v2.18.1 "/bin/prometheus --c…" 5 days ago Up 5 days ceph-797b1aa4-16cf-11ec-a787-00505693fd22-prometheus.test01 c737fb65080c elasticsearch:7.8.0 "/tini -- /usr/local…" 7 months ago Up 5 days 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp es7.6 ``` 添加节点 ```sh ceph orch host add test02 172.16.100.2 ceph orch host add test11 172.16.100.11 ``` 添加硬盘 ```sh ceph orch daemon add osd test01:/dev/sdc ceph orch daemon add osd test02:/dev/sdc ceph orch daemon add osd test11:/dev/sdb ``` ceph osd pool create k8s 64