因为要在 k8s 内使用存储，考虑删除容器后文件还能持久保存，所以要装个分布式文件存储系统，当前比较火的莫过于 ceph 了。

本文记了个部署 ceph 的流水账。

工具选择

参考：https://docs.ceph.com/en/latest/install/

网页上的推荐方法

  • cephadm 使用容器
    自动部署需要 pull docker io 的包，即使你先离线导入了，还是会去请求，获取最新的 commit id，离线不可用

  • rook 使用容器

  • ceph-ansible 使用 Ansible.

    • 意味着服务端需要安装 ansible
    • 没有集成 Nautlius 和 Octopus 版本新加入的 orchestrator API，不能使用新的管理功能，面板集成也不能用

  • ceph-deploy

    • 快速部署 ceph 的工具
    • 没有在活跃的维护，官方没有在 Nautilus 以后的版本测试过，不支持 RHEL8, CentOS 8, 或者更新的操作系统.

我们的机器环境是 CentOS 7，犹豫再三，相比手动安装，我选择了 ceph-deploy ，最新支持到 nautilus 版本

Host

10.101.235.84 ceph-1

  • sda 500G
  • data 3T raid0 x 12 ( sdb - sdm)
  • mem 128G
  • cpu 24c

10.101.235.217 ceph-2

  • sda 446G
  • data 6T raid0 x 12 ( sdb - sdm)
  • mem 256G
  • cpu 32c

10.101.235.252 ceph-3

  • sda 414G
  • sdb 60T 12*6T raid5 (还没有重做 raid)
  • mem 256G
  • cpu 32c

准备工作

  1. host

hostnamectl set-hostname ceph-1
hostnamectl set-hostname ceph-2
hostnamectl set-hostname ceph-3

写入 /etc/hosts

1
2
3
10.101.235.84 ceph-1
10.101.235.217 ceph-2
10.101.235.38 ceph-3

  1. ssh 免密

ssh-keygen

ssh-copy-id ceph-1

ssh-copy-id ceph-2

ssh-copy-id ceph-3

  1. 安全设置

关闭 selinux 和防火墙

for i in {1..3};do echo $i;ssh ceph-$i “systemctl disable –now firewalld”;done

for i in {1..3};do echo $i;ssh ceph-$i “setenforce 0”;done

for i in {1..3};do echo $i;ssh ceph-$i “sed -i ‘s/^SELINUX=.*/SELINUX=disabled/‘ /etc/selinux/config”;done

  1. ntp

ceph-1 上
vi /etc/ntp.conf

1
server 10.110.38.240 minpoll 4 maxpoll 5

ntpq -pn 查看同步状态

1
2
3
4
[[email protected] ceph-cluster]# ntpq -pn
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*10.110.38.240   10.108.84.45     3 u   10   16  377    0.901  -283.35   0.407

ceph-2 ceph-3 上 写 ceph-1 的地址 10.101.235.84
vi /etc/ntp.conf

1
server 10.101.235.84 iburst

ntpq -pn

查看同步状态

同步完成，ip 前会显示*号

生产环境最好配置多个 ntp server

for i in {1..3};do echo $i;ssh ceph-$i “date”;done

  1. yum 源

ceph-deploy version 2.0.0+

http://download.ceph.com/rpm-nautilus/el7/noarch/

ceph.repo

1
2
3
4
5
[ceph]
name=ceph
baseurl=http://10.110.38.20/ceph-nautilus/
enable=1
gpgcheck=0

yum makecache

  1. 安装 ceph-deploy

在 ceph-1

yum install python-setuptools ceph-deploy

ceph-deploy –version

deploy

在 ceph-1

mkdir my-cluster
cd my-cluster

安装会生成 ceph.conf keyring

ceph-deploy new –cluster-network=10.101.235.1/24 ceph-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[[email protected] ceph]# ceph-deploy new --cluster-network=10.101.235.1/24 ceph-1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy new --cluster-network=10.101.235.1/24 ceph-1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  func                          : <function new at 0x1bb8c80>
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1c19f80>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  ssh_copykey                   : True
[ceph_deploy.cli][INFO  ]  mon                           : ['ceph-1']
[ceph_deploy.cli][INFO  ]  public_network                : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  cluster_network               : 10.101.235.1/24
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  fsid                          : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO  ] making sure passwordless SSH succeeds
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph-1][DEBUG ] find the location of an executable
[ceph-1][INFO  ] Running command: /usr/sbin/ip link show
[ceph-1][INFO  ] Running command: /usr/sbin/ip addr show
[ceph-1][DEBUG ] IP addresses found: [u'10.101.235.84']
[ceph_deploy.new][DEBUG ] Resolving host ceph-1
[ceph_deploy.new][DEBUG ] Monitor ceph-1 at 10.101.235.84
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph-1']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.101.235.84']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

ceph-deploy install {ceph-node} […]
会自动配置覆盖 yum 源，这里就手动安装

yum install -y ceph ceph-mon ceph-mgr ceph-radosgw ceph-mds

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
Error: Package: librdkafka-0.11.5-1.el7.x86_64 (ceph)
           Requires: liblz4.so.1()(64bit)
Error: Package: 2:ceph-base-14.2.22-0.el7.x86_64 (ceph)
           Requires: liblz4.so.1()(64bit)
Error: Package: policycoreutils-2.5-34.el7.x86_64 (ceph)
           Requires: libsemanage >= 2.5-14
           Installed: libsemanage-2.5-8.el7.x86_64 (@anaconda)
               libsemanage = 2.5-8.el7
Error: Package: policycoreutils-2.5-34.el7.x86_64 (ceph)
           Requires: libsepol >= 2.5-10
           Installed: libsepol-2.5-6.el7.x86_64 (@anaconda)
               libsepol = 2.5-6.el7
Error: Package: selinux-policy-3.13.1-268.el7_9.2.noarch (ceph)
           Requires: libsemanage >= 2.5-13
           Installed: libsemanage-2.5-8.el7.x86_64 (@anaconda)
               libsemanage = 2.5-8.el7
Error: Package: 2:ceph-mon-14.2.22-0.el7.x86_64 (ceph)
           Requires: liblz4.so.1()(64bit)
Error: Package: policycoreutils-2.5-34.el7.x86_64 (ceph)
           Requires: libselinux-utils >= 2.5-14
           Installed: libselinux-utils-2.5-11.el7.x86_64 (@anaconda)
               libselinux-utils = 2.5-11.el7
Error: Package: 2:ceph-common-14.2.22-0.el7.x86_64 (ceph)
           Requires: liblz4.so.1()(64bit)
Error: Package: 2:ceph-osd-14.2.22-0.el7.x86_64 (ceph)
           Requires: liblz4.so.1()(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

在公网的机器上

1
2
3
4
5
yumdownloader --resolve libsemanage.x86_64
yumdownloader --resolve libsepol.x86_64
yumdownloader --resolve libselinux-utils
yumdownloader --resolve libselinux
yumdownloader --resolve lz4

liblz4 https://serverfault.com/questions/917688/unable-to-update-centos-7-yum-update-broken

把 rpm 放在一个目录下

1
yum install *

如果单个 yum install 会提示你这样依赖不对，那样依赖不对，没那么智能，要一起安装才行

node 1 2 3 分别安装

1
yum install -y ceph ceph-mon ceph-mgr ceph-radosgw ceph-mds

在 node 1 上

cd my-cluster

ceph-deploy mon create-initial # node 1 初始化 mon

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
[[email protected] ceph]# ceph-deploy mon create-initial
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy mon create-initial
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create-initial
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1784248>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mon at 0x177b2a8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  keyrings                      : None
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts ceph-1
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-1 ...
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph-1][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO  ] distro info: CentOS Linux 7.4.1708 Core
[ceph-1][DEBUG ] determining if provided host has same hostname in remote
[ceph-1][DEBUG ] get remote short hostname
[ceph-1][DEBUG ] deploying mon to ceph-1
[ceph-1][DEBUG ] get remote short hostname
[ceph-1][DEBUG ] remote hostname: ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][DEBUG ] create the mon path if it does not exist
[ceph-1][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-1/done
[ceph-1][DEBUG ] done path does not exist: /var/lib/ceph/mon/ceph-ceph-1/done
[ceph-1][INFO  ] creating keyring file: /var/lib/ceph/tmp/ceph-ceph-1.mon.keyring
[ceph-1][DEBUG ] create the monitor keyring file
[ceph-1][INFO  ] Running command: ceph-mon --cluster ceph --mkfs -i ceph-1 --keyring /var/lib/ceph/tmp/ceph-ceph-1.mon.keyring --setuser 167 --setgroup 167
[ceph-1][INFO  ] unlinking keyring file /var/lib/ceph/tmp/ceph-ceph-1.mon.keyring
[ceph-1][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-1][DEBUG ] create the init path if it does not exist
[ceph-1][INFO  ] Running command: systemctl enable ceph.target
[ceph-1][INFO  ] Running command: systemctl enable [email protected]
[ceph-1][WARNIN] Created symlink from /etc/systemd/system/ceph-mon.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-1][INFO  ] Running command: systemctl start [email protected]
[ceph-1][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-1.asok mon_status
[ceph-1][DEBUG ] ********************************************************************************
[ceph-1][DEBUG ] status for monitor: mon.ceph-1
[ceph-1][DEBUG ] {
[ceph-1][DEBUG ]   "election_epoch": 3, 
[ceph-1][DEBUG ]   "extra_probe_peers": [], 
[ceph-1][DEBUG ]   "feature_map": {
[ceph-1][DEBUG ]     "mon": [
[ceph-1][DEBUG ]       {
[ceph-1][DEBUG ]         "features": "0x3ffddff8ffecffff", 
[ceph-1][DEBUG ]         "num": 1, 
[ceph-1][DEBUG ]         "release": "luminous"
[ceph-1][DEBUG ]       }
[ceph-1][DEBUG ]     ]
[ceph-1][DEBUG ]   }, 
[ceph-1][DEBUG ]   "features": {
[ceph-1][DEBUG ]     "quorum_con": "4611087854035861503", 
[ceph-1][DEBUG ]     "quorum_mon": [
[ceph-1][DEBUG ]       "kraken", 
[ceph-1][DEBUG ]       "luminous", 
[ceph-1][DEBUG ]       "mimic", 
[ceph-1][DEBUG ]       "osdmap-prune", 
[ceph-1][DEBUG ]       "nautilus"
[ceph-1][DEBUG ]     ], 
[ceph-1][DEBUG ]     "required_con": "2449958747315912708", 
[ceph-1][DEBUG ]     "required_mon": [
[ceph-1][DEBUG ]       "kraken", 
[ceph-1][DEBUG ]       "luminous", 
[ceph-1][DEBUG ]       "mimic", 
[ceph-1][DEBUG ]       "osdmap-prune", 
[ceph-1][DEBUG ]       "nautilus"
[ceph-1][DEBUG ]     ]
[ceph-1][DEBUG ]   }, 
[ceph-1][DEBUG ]   "monmap": {
[ceph-1][DEBUG ]     "created": "2021-07-21 17:31:24.761362", 
[ceph-1][DEBUG ]     "epoch": 1, 
[ceph-1][DEBUG ]     "features": {
[ceph-1][DEBUG ]       "optional": [], 
[ceph-1][DEBUG ]       "persistent": [
[ceph-1][DEBUG ]         "kraken", 
[ceph-1][DEBUG ]         "luminous", 
[ceph-1][DEBUG ]         "mimic", 
[ceph-1][DEBUG ]         "osdmap-prune", 
[ceph-1][DEBUG ]         "nautilus"
[ceph-1][DEBUG ]       ]
[ceph-1][DEBUG ]     }, 
[ceph-1][DEBUG ]     "fsid": "d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb", 
[ceph-1][DEBUG ]     "min_mon_release": 14, 
[ceph-1][DEBUG ]     "min_mon_release_name": "nautilus", 
[ceph-1][DEBUG ]     "modified": "2021-07-21 17:31:24.761362", 
[ceph-1][DEBUG ]     "mons": [
[ceph-1][DEBUG ]       {
[ceph-1][DEBUG ]         "addr": "10.101.235.84:6789/0", 
[ceph-1][DEBUG ]         "name": "ceph-1", 
[ceph-1][DEBUG ]         "public_addr": "10.101.235.84:6789/0", 
[ceph-1][DEBUG ]         "public_addrs": {
[ceph-1][DEBUG ]           "addrvec": [
[ceph-1][DEBUG ]             {
[ceph-1][DEBUG ]               "addr": "10.101.235.84:3300", 
[ceph-1][DEBUG ]               "nonce": 0, 
[ceph-1][DEBUG ]               "type": "v2"
[ceph-1][DEBUG ]             }, 
[ceph-1][DEBUG ]             {
[ceph-1][DEBUG ]               "addr": "10.101.235.84:6789", 
[ceph-1][DEBUG ]               "nonce": 0, 
[ceph-1][DEBUG ]               "type": "v1"
[ceph-1][DEBUG ]             }
[ceph-1][DEBUG ]           ]
[ceph-1][DEBUG ]         }, 
[ceph-1][DEBUG ]         "rank": 0
[ceph-1][DEBUG ]       }
[ceph-1][DEBUG ]     ]
[ceph-1][DEBUG ]   }, 
[ceph-1][DEBUG ]   "name": "ceph-1", 
[ceph-1][DEBUG ]   "outside_quorum": [], 
[ceph-1][DEBUG ]   "quorum": [
[ceph-1][DEBUG ]     0
[ceph-1][DEBUG ]   ], 
[ceph-1][DEBUG ]   "quorum_age": 2, 
[ceph-1][DEBUG ]   "rank": 0, 
[ceph-1][DEBUG ]   "state": "leader", 
[ceph-1][DEBUG ]   "sync_provider": []
[ceph-1][DEBUG ] }
[ceph-1][DEBUG ] ********************************************************************************
[ceph-1][INFO  ] monitor: mon.ceph-1 is running
[ceph-1][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-1.asok mon_status
[ceph_deploy.mon][INFO  ] processing monitor mon.ceph-1
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph-1][DEBUG ] find the location of an executable
[ceph-1][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-1.asok mon_status
[ceph_deploy.mon][INFO  ] mon.ceph-1 monitor has reached quorum!
[ceph_deploy.mon][INFO  ] all initial monitors are running and have formed quorum
[ceph_deploy.mon][INFO  ] Running gatherkeys...
[ceph_deploy.gatherkeys][INFO  ] Storing keys in temp directory /tmp/tmplLWg_z
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph-1][DEBUG ] get remote short hostname
[ceph-1][DEBUG ] fetch remote file
[ceph-1][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.ceph-1.asok mon_status
[ceph-1][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-1/keyring auth get client.admin
[ceph-1][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-1/keyring auth get client.bootstrap-mds
[ceph-1][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-1/keyring auth get client.bootstrap-mgr
[ceph-1][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-1/keyring auth get client.bootstrap-osd
[ceph-1][INFO  ] Running command: /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-ceph-1/keyring auth get client.bootstrap-rgw
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.client.admin.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mds.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-mgr.keyring
[ceph_deploy.gatherkeys][INFO  ] keyring 'ceph.mon.keyring' already exists
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-osd.keyring
[ceph_deploy.gatherkeys][INFO  ] Storing ceph.bootstrap-rgw.keyring
[ceph_deploy.gatherkeys][INFO  ] Destroy temp directory /tmp/tmplLWg_z

生成了很多 keyring

将 keyring 推送到所有的 node 上

ceph-deploy admin ceph-1 ceph-2 ceph-3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
[[email protected] ceph-cluster]# ceph-deploy admin ceph-1 ceph-2 ceph-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy admin ceph-1 ceph-2 ceph-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1ed6878>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['ceph-1', 'ceph-2', 'ceph-3']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x1e3e0c8>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-1
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-2
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
[ceph-2][DEBUG ] detect machine type
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-3
[ceph-3][DEBUG ] connected to host: ceph-3 
[ceph-3][DEBUG ] detect platform information from remote host
[ceph-3][DEBUG ] detect machine type
[ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

输入 ceph -s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[[email protected] ceph-cluster]# ceph -s
  cluster:
    id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
    health: HEALTH_WARN
            mon is allowing insecure global_id reclaim
 
  services:
    mon: 1 daemons, quorum ceph-1 (age 3h)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

提示 mon is allowing insecure global_id reclaim

解决：

1
ceph config set mon auth_allow_insecure_global_id_reclaim false

ceph -s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[[email protected] ceph-cluster]# ceph -s
  cluster:
    id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum ceph-1 (age 3h)
    mgr: no daemons active
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

monitor 1 个 是 ceph-1
manager 还没有

当前 cluster 目录下

1
2
3
4
5
6
7
8
9
10
[[email protected] ceph-cluster]# ll
total 48
-rw------- 1 root root   113 Jul 21 17:31 ceph.bootstrap-mds.keyring
-rw------- 1 root root   113 Jul 21 17:31 ceph.bootstrap-mgr.keyring
-rw------- 1 root root   113 Jul 21 17:31 ceph.bootstrap-osd.keyring
-rw------- 1 root root   113 Jul 21 17:31 ceph.bootstrap-rgw.keyring
-rw------- 1 root root   151 Jul 21 17:31 ceph.client.admin.keyring
-rw-r--r-- 1 root root   231 Jul 21 15:34 ceph.conf
-rw-r--r-- 1 root root 17956 Jul 21 21:09 ceph-deploy-ceph.log
-rw------- 1 root root    73 Jul 21 15:34 ceph.mon.keyring

将 ceph-1 作为 mgr

1
ceph-deploy mgr create ceph-1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
[[email protected] ceph-cluster]# ceph-deploy mgr create ceph-1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy mgr create ceph-1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  mgr                           : [('ceph-1', 'ceph-1')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x27b3ab8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0x2745f50>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts ceph-1:ceph-1
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][WARNIN] mgr keyring does not exist yet, creating one
[ceph-1][DEBUG ] create a keyring file
[ceph-1][DEBUG ] create path recursively if it doesn't exist
[ceph-1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-1 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-1/keyring
[ceph-1][INFO  ] Running command: systemctl enable [email protected]
[ceph-1][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-1][INFO  ] Running command: systemctl start [email protected]
[ceph-1][INFO  ] Running command: systemctl enable ceph.target

ceph -s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[[email protected] ceph-cluster]# ceph -s
  cluster:
    id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum ceph-1 (age 4h)
    mgr: ceph-1(active, since 2s)
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

准备 osd

删除分区

umount

清除 /etc/fstab

fdisk /dev/sdb
d
w

ceph-deploy osd create ceph-1 –data /dev/sdb
ceph-deploy osd create ceph-2 –data /dev/sdb
ceph-deploy osd create ceph-3 –data /dev/sdb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
[[email protected] ceph-cluster]# ceph-deploy osd create ceph-1 --data /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy osd create ceph-1 --data /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1d86830>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  journal                       : None
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  host                          : ceph-1
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x1d76758>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.cli][INFO  ]  data                          : /dev/sdb
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device /dev/sdb
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph-1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][WARNIN] osd keyring does not exist yet, creating one
[ceph-1][DEBUG ] create a keyring file
[ceph-1][DEBUG ] find the location of an executable
[ceph-1][INFO  ] Running command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[ceph-1][WARNIN] usage: ceph-volume lvm create [-h] [--crush-device-class CRUSH_DEVICE_CLASS]
[ceph-1][WARNIN]                               [--data-slots DATA_SLOTS]
[ceph-1][WARNIN]                               [--data-size DATA_SIZE]
[ceph-1][WARNIN]                               [--cluster-fsid CLUSTER_FSID] [--dmcrypt]
[ceph-1][WARNIN]                               [--osd-fsid OSD_FSID] --data DATA
[ceph-1][WARNIN]                               [--osd-id OSD_ID] [--no-systemd]
[ceph-1][WARNIN]                               [--block.wal-size BLOCK_WAL_SIZE]
[ceph-1][WARNIN]                               [--block.wal-slots BLOCK_WAL_SLOTS]
[ceph-1][WARNIN]                               [--block.db-size BLOCK_DB_SIZE]
[ceph-1][WARNIN]                               [--block.wal BLOCK_WAL]
[ceph-1][WARNIN]                               [--block.db-slots BLOCK_DB_SLOTS] [--bluestore]
[ceph-1][WARNIN]                               [--block.db BLOCK_DB] [--filestore]
[ceph-1][WARNIN]                               [--journal-size JOURNAL_SIZE]
[ceph-1][WARNIN]                               [--journal JOURNAL]
[ceph-1][WARNIN]                               [--journal-slots JOURNAL_SLOTS]
[ceph-1][WARNIN] ceph-volume lvm create: error: GPT headers found, they must be removed on: /dev/sdb
[ceph-1][ERROR ] RuntimeError: command returned non-zero exit status: 2
[ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs

[ceph-1][WARNIN] ceph-volume lvm create: error: GPT headers found, they must be removed on: /dev/sdb

https://www.jianshu.com/p/d7fcf1cb5a48

该错误关键点就是 GPT headers found, they must be removed，发生原因应该是之前磁盘被分区过，虽然删掉了分区，但是还存在 GPT 数据结构，使用 sgdisk 命令进行清除。

1
#sgdisk --zap-all /dev/sdX

sgdisk 我们的 yum 源没有

https://zhuanlan.zhihu.com/p/73479251

可以看到, 新增的硬盘中发现了 GPT 分区表, 所以添加失败了, 我们要手动清理掉硬盘的分区表 (当然如果硬盘是全新的, 这里应该就成功了).

这里我们直接暴力干掉分区表, 不用费事的操作 PV 和 VG 了.

注意, 一定要再三检查目标硬盘是否是期望的硬盘, 如果操作错了硬盘, 分区表直接就没了.

[[email protected] ceph-9]# dd if=/dev/zero of=/dev/sde bs=512K count=1
1+0 records in
1+0 records out
524288 bytes (524 kB) copied, 0.00109677 s, 478 MB/s

利用 dd 命令把硬盘的前 512K 填充为 0, 直接干掉分区信息.

看评论，似乎还可以

可以用 wipefs -a /dev/sde，这样更快些可以用 partprobe 让内核重新读分区表，不用重启

没有尝试过

dd if=/dev/zero of=/dev/sdb bs=512K count=1

继续

ceph-deploy osd create ceph-1 –data /dev/sdb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
[[email protected] ceph-cluster]# ceph-deploy osd create ceph-1 --data /dev/sdb
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy osd create ceph-1 --data /dev/sdb
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2425830>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  block_wal                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  journal                       : None
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  host                          : ceph-1
[ceph_deploy.cli][INFO  ]  filestore                     : None
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x2415758>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.cli][INFO  ]  data                          : /dev/sdb
[ceph_deploy.cli][INFO  ]  block_db                      : None
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.osd][DEBUG ] Creating OSD on cluster ceph with data device /dev/sdb
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph-1][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][DEBUG ] find the location of an executable
[ceph-1][INFO  ] Running command: /usr/sbin/ceph-volume --cluster ceph lvm create --bluestore --data /dev/sdb
[ceph-1][WARNIN] Running command: /bin/ceph-authtool --gen-print-key
[ceph-1][WARNIN] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new 749563a0-bcb0-4246-a5b7-9a9681b2bb68
[ceph-1][WARNIN] Running command: /sbin/vgcreate --force --yes ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37 /dev/sdb
[ceph-1][WARNIN]  stdout: Physical volume "/dev/sdb" successfully created.
[ceph-1][WARNIN]  stdout: Volume group "ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37" successfully created
[ceph-1][WARNIN] Running command: /sbin/lvcreate --yes -l 7867810 -n osd-block-749563a0-bcb0-4246-a5b7-9a9681b2bb68 ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37
[ceph-1][WARNIN]  stdout: Wiping xfs signature on /dev/ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37/osd-block-749563a0-bcb0-4246-a5b7-9a9681b2bb68.
[ceph-1][WARNIN]  stdout: Logical volume "osd-block-749563a0-bcb0-4246-a5b7-9a9681b2bb68" created.
[ceph-1][WARNIN] Running command: /bin/ceph-authtool --gen-print-key
[ceph-1][WARNIN] Running command: /bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
[ceph-1][WARNIN] Running command: /bin/chown -h ceph:ceph /dev/ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37/osd-block-749563a0-bcb0-4246-a5b7-9a9681b2bb68
[ceph-1][WARNIN] Running command: /bin/chown -R ceph:ceph /dev/dm-3
[ceph-1][WARNIN] Running command: /bin/ln -s /dev/ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37/osd-block-749563a0-bcb0-4246-a5b7-9a9681b2bb68 /var/lib/ceph/osd/ceph-0/block
[ceph-1][WARNIN] Running command: /bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
[ceph-1][WARNIN]  stderr: 2021-07-21 22:40:12.775 7fb864843700 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,: (2) No such file or directory
[ceph-1][WARNIN] 2021-07-21 22:40:12.775 7fb864843700 -1 AuthRegistry(0x7fb860066aa8) no keyring found at /etc/ceph/ceph.client.bootstrap-osd.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin,, disabling cephx
[ceph-1][WARNIN]  stderr: got monmap epoch 1
[ceph-1][WARNIN] Running command: /bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQDLMfhg4mvlDxAAanSsTFDDOf03gDL2/rJxTA==
[ceph-1][WARNIN]  stdout: creating /var/lib/ceph/osd/ceph-0/keyring
[ceph-1][WARNIN] added entity osd.0 auth(key=AQDLMfhg4mvlDxAAanSsTFDDOf03gDL2/rJxTA==)
[ceph-1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
[ceph-1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
[ceph-1][WARNIN] Running command: /bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid 749563a0-bcb0-4246-a5b7-9a9681b2bb68 --setuser ceph --setgroup ceph
[ceph-1][WARNIN]  stderr: 2021-07-21 22:40:13.390 7f7493915a80 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
[ceph-1][WARNIN] --> ceph-volume lvm prepare successful for: /dev/sdb
[ceph-1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[ceph-1][WARNIN] Running command: /bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37/osd-block-749563a0-bcb0-4246-a5b7-9a9681b2bb68 --path /var/lib/ceph/osd/ceph-0 --no-mon-config
[ceph-1][WARNIN] Running command: /bin/ln -snf /dev/ceph-2adefc7a-f41f-48d4-92eb-6427b5a50e37/osd-block-749563a0-bcb0-4246-a5b7-9a9681b2bb68 /var/lib/ceph/osd/ceph-0/block
[ceph-1][WARNIN] Running command: /bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-0/block
[ceph-1][WARNIN] Running command: /bin/chown -R ceph:ceph /dev/dm-3
[ceph-1][WARNIN] Running command: /bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0
[ceph-1][WARNIN] Running command: /bin/systemctl enable [email protected]
[ceph-1][WARNIN]  stderr: Created symlink from /etc/systemd/system/multi-user.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-1][WARNIN] Running command: /bin/systemctl enable --runtime [email protected]
[ceph-1][WARNIN]  stderr: Created symlink from /run/systemd/system/ceph-osd.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-1][WARNIN] Running command: /bin/systemctl start [email protected]
[ceph-1][WARNIN] --> ceph-volume lvm activate successful for osd ID: 0
[ceph-1][WARNIN] --> ceph-volume lvm create successful for: /dev/sdb
[ceph-1][INFO  ] checking OSD status...
[ceph-1][DEBUG ] find the location of an executable
[ceph-1][INFO  ] Running command: /bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host ceph-1 is now ready for osd use.

这时候 ceph -s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[[email protected] ceph-cluster]# ceph -s
  cluster:
    id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
    health: HEALTH_WARN
            OSD count 1 < osd_pool_default_size 3
 
  services:
    mon: 1 daemons, quorum ceph-1 (age 5h)
    mgr: ceph-1(active, since 55m)
    osd: 1 osds: 1 up (since 100s), 1 in (since 100s)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   1.0 GiB used, 30 TiB / 30 TiB avail
    pgs:

ceph-deploy osd create ceph-2 –data /dev/sdb
ceph-deploy osd create ceph-3 –data /dev/sdb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
[[email protected] ceph-cluster]# ceph -s
  cluster:
    id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum ceph-1 (age 5h)
    mgr: ceph-1(active, since 59m)
    osd: 3 osds: 3 up (since 7s), 3 in (since 7s)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 94 TiB / 94 TiB avail
    pgs:

此时不再显示 warn 了

1
2
HEALTH_WARN
            OSD count 1 < osd_pool_default_size 3

查看 osd 状态

1
2
3
4
5
6
7
8
[[email protected] ceph-cluster]# ceph osd status 
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| id |  host  |  used | avail | wr ops | wr data | rd ops | rd data |   state   |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+
| 0  | ceph-1 | 1025M | 30.0T |    0   |     0   |    0   |     0   | exists,up |
| 1  | ceph-2 | 1025M | 60.0T |    0   |     0   |    0   |     0   | exists,up |
| 2  | ceph-3 | 1025M | 3904G |    0   |     0   |    0   |     0   | exists,up |
+----+--------+-------+-------+--------+---------+--------+---------+-----------+

拓展 mon 和 mgr

参考 https://docs.ceph.com/en/octopus/install/ceph-deploy/quick-ceph-deploy/#expanding-your-cluster

mon 需要高可用
mon 挂了 整个集群挂了
Paxos 算法 奇数个

ceph-deploy mon add ceph-2 ceph-3

报错

1
2
3
4
5
6
7
8
[ceph-2][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph-2][WARNIN] ceph-2 is not defined in `mon initial members`
[ceph-2][WARNIN] monitor ceph-2 does not exist in monmap
[ceph-2][WARNIN] neither `public_addr` nor `public_network` keys are defined for monitors
[ceph-2][WARNIN] monitors may not be able to form quorum
[ceph-2][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-2.asok mon_status
[ceph-2][ERROR ] admin_socket: exception getting command descriptions: [Errno 2] No such file or directory
[ceph-2][WARNIN] monitor: mon.ceph-2, might not be running yet

原因 ceph.conf 配置文件中缺少 public network 的配置

添加了后，把配置推送到每个节点

ceph-deploy –overwrite-conf config push ceph-1 ceph-2 ceph-3

ceph-deploy mon add ceph-2

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
[[email protected] ceph-cluster]# ceph-deploy mon add ceph-2 
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy mon add ceph-2
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : add
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x22553b0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  mon                           : ['ceph-2']
[ceph_deploy.cli][INFO  ]  func                          : <function mon at 0x224c2a8>
[ceph_deploy.cli][INFO  ]  address                       : None
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mon][INFO  ] ensuring configuration of new mon host: ceph-2
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph-2
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
[ceph-2][DEBUG ] detect machine type
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.mon][DEBUG ] Adding mon to cluster ceph, host ceph-2
[ceph_deploy.mon][DEBUG ] using mon address by resolving host: 10.101.235.217
[ceph_deploy.mon][DEBUG ] detecting platform for host ceph-2 ...
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
[ceph-2][DEBUG ] detect machine type
[ceph-2][DEBUG ] find the location of an executable
[ceph_deploy.mon][INFO  ] distro info: CentOS Linux 7.4.1708 Core
[ceph-2][DEBUG ] determining if provided host has same hostname in remote
[ceph-2][DEBUG ] get remote short hostname
[ceph-2][DEBUG ] adding mon to ceph-2
[ceph-2][DEBUG ] get remote short hostname
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-2][DEBUG ] create the mon path if it does not exist
[ceph-2][DEBUG ] checking for done path: /var/lib/ceph/mon/ceph-ceph-2/done
[ceph-2][DEBUG ] create a done file to avoid re-doing the mon deployment
[ceph-2][DEBUG ] create the init path if it does not exist
[ceph-2][INFO  ] Running command: systemctl enable ceph.target
[ceph-2][INFO  ] Running command: systemctl enable [email protected]
[ceph-2][INFO  ] Running command: systemctl start [email protected]
[ceph-2][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-2.asok mon_status
[ceph-2][WARNIN] ceph-2 is not defined in `mon initial members`
[ceph-2][WARNIN] monitor ceph-2 does not exist in monmap
[ceph-2][INFO  ] Running command: ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph-2.asok mon_status
[ceph-2][DEBUG ] ********************************************************************************
[ceph-2][DEBUG ] status for monitor: mon.ceph-2
[ceph-2][DEBUG ] {
[ceph-2][DEBUG ]   "election_epoch": 0, 
[ceph-2][DEBUG ]   "extra_probe_peers": [], 
[ceph-2][DEBUG ]   "feature_map": {
[ceph-2][DEBUG ]     "mon": [
[ceph-2][DEBUG ]       {
[ceph-2][DEBUG ]         "features": "0x3ffddff8ffecffff", 
[ceph-2][DEBUG ]         "num": 1, 
[ceph-2][DEBUG ]         "release": "luminous"
[ceph-2][DEBUG ]       }
[ceph-2][DEBUG ]     ]
[ceph-2][DEBUG ]   }, 
[ceph-2][DEBUG ]   "features": {
[ceph-2][DEBUG ]     "quorum_con": "0", 
[ceph-2][DEBUG ]     "quorum_mon": [], 
[ceph-2][DEBUG ]     "required_con": "2449958197560098820", 
[ceph-2][DEBUG ]     "required_mon": [
[ceph-2][DEBUG ]       "kraken", 
[ceph-2][DEBUG ]       "luminous", 
[ceph-2][DEBUG ]       "mimic", 
[ceph-2][DEBUG ]       "osdmap-prune", 
[ceph-2][DEBUG ]       "nautilus"
[ceph-2][DEBUG ]     ]
[ceph-2][DEBUG ]   }, 
[ceph-2][DEBUG ]   "monmap": {
[ceph-2][DEBUG ]     "created": "2021-07-21 17:31:24.761362", 
[ceph-2][DEBUG ]     "epoch": 1, 
[ceph-2][DEBUG ]     "features": {
[ceph-2][DEBUG ]       "optional": [], 
[ceph-2][DEBUG ]       "persistent": [
[ceph-2][DEBUG ]         "kraken", 
[ceph-2][DEBUG ]         "luminous", 
[ceph-2][DEBUG ]         "mimic", 
[ceph-2][DEBUG ]         "osdmap-prune", 
[ceph-2][DEBUG ]         "nautilus"
[ceph-2][DEBUG ]       ]
[ceph-2][DEBUG ]     }, 
[ceph-2][DEBUG ]     "fsid": "d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb", 
[ceph-2][DEBUG ]     "min_mon_release": 14, 
[ceph-2][DEBUG ]     "min_mon_release_name": "nautilus", 
[ceph-2][DEBUG ]     "modified": "2021-07-21 17:31:24.761362", 
[ceph-2][DEBUG ]     "mons": [
[ceph-2][DEBUG ]       {
[ceph-2][DEBUG ]         "addr": "10.101.235.84:6789/0", 
[ceph-2][DEBUG ]         "name": "ceph-1", 
[ceph-2][DEBUG ]         "public_addr": "10.101.235.84:6789/0", 
[ceph-2][DEBUG ]         "public_addrs": {
[ceph-2][DEBUG ]           "addrvec": [
[ceph-2][DEBUG ]             {
[ceph-2][DEBUG ]               "addr": "10.101.235.84:3300", 
[ceph-2][DEBUG ]               "nonce": 0, 
[ceph-2][DEBUG ]               "type": "v2"
[ceph-2][DEBUG ]             }, 
[ceph-2][DEBUG ]             {
[ceph-2][DEBUG ]               "addr": "10.101.235.84:6789", 
[ceph-2][DEBUG ]               "nonce": 0, 
[ceph-2][DEBUG ]               "type": "v1"
[ceph-2][DEBUG ]             }
[ceph-2][DEBUG ]           ]
[ceph-2][DEBUG ]         }, 
[ceph-2][DEBUG ]         "rank": 0
[ceph-2][DEBUG ]       }
[ceph-2][DEBUG ]     ]
[ceph-2][DEBUG ]   }, 
[ceph-2][DEBUG ]   "name": "ceph-2", 
[ceph-2][DEBUG ]   "outside_quorum": [], 
[ceph-2][DEBUG ]   "quorum": [], 
[ceph-2][DEBUG ]   "rank": -1, 
[ceph-2][DEBUG ]   "state": "probing", 
[ceph-2][DEBUG ]   "sync_provider": []
[ceph-2][DEBUG ] }
[ceph-2][DEBUG ] ********************************************************************************
[ceph-2][INFO  ] monitor: mon.ceph-2 is currently at the state of probing

ceph-deploy mon add ceph-3

ceph -s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cluster:
  id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
  health: HEALTH_WARN
          clock skew detected on mon.ceph-2, mon.ceph-3
 
services:
  mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 14s)
  mgr: ceph-1(active, since 9m)
  osd: 3 osds: 3 up (since 27m), 3 in (since 27m)
 
data:
  pools:   0 pools, 0 pgs
  objects: 0 objects, 0 B
  usage:   3.0 GiB used, 94 TiB / 94 TiB avail
  pgs:

ceph quorum_status –format json-pretty

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
[[email protected] ceph-cluster]# ceph quorum_status --format json-pretty

{
    "election_epoch": 12,
    "quorum": [
        0,
        1,
        2
    ],
    "quorum_names": [
        "ceph-1",
        "ceph-2",
        "ceph-3"
    ],
    "quorum_leader_name": "ceph-1",
    "quorum_age": 63,
    "monmap": {
        "epoch": 3,
        "fsid": "d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb",
        "modified": "2021-07-21 23:12:49.336700",
        "created": "2021-07-21 17:31:24.761362",
        "min_mon_release": 14,
        "min_mon_release_name": "nautilus",
        "features": {
            "persistent": [
                "kraken",
                "luminous",
                "mimic",
                "osdmap-prune",
                "nautilus"
            ],
            "optional": []
        },
        "mons": [
            {
                "rank": 0,
                "name": "ceph-1",
                "public_addrs": {
                    "addrvec": [
                        {
                            "type": "v2",
                            "addr": "10.101.235.84:3300",
                            "nonce": 0
                        },
                        {
                            "type": "v1",
                            "addr": "10.101.235.84:6789",
                            "nonce": 0
                        }
                    ]
                },
                "addr": "10.101.235.84:6789/0",
                "public_addr": "10.101.235.84:6789/0"
            },
            {
                "rank": 1,
                "name": "ceph-2",
                "public_addrs": {
                    "addrvec": [
                        {
                            "type": "v2",
                            "addr": "10.101.235.217:3300",
                            "nonce": 0
                        },
                        {
                            "type": "v1",
                            "addr": "10.101.235.217:6789",
                            "nonce": 0
                        }
                    ]
                },
                "addr": "10.101.235.217:6789/0",
                "public_addr": "10.101.235.217:6789/0"
            },
            {
                "rank": 2,
                "name": "ceph-3",
                "public_addrs": {
                    "addrvec": [
                        {
                            "type": "v2",
                            "addr": "10.101.235.38:3300",
                            "nonce": 0
                        },
                        {
                            "type": "v1",
                            "addr": "10.101.235.38:6789",
                            "nonce": 0
                        }
                    ]
                },
                "addr": "10.101.235.38:6789/0",
                "public_addr": "10.101.235.38:6789/0"
            }
        ]
    }
}

ceph mon stat

1
2
[[email protected] ceph-cluster]# ceph mon stat
e3: 3 mons at {ceph-1=[v2:10.101.235.84:3300/0,v1:10.101.235.84:6789/0],ceph-2=[v2:10.101.235.217:3300/0,v1:10.101.235.217:6789/0],ceph-3=[v2:10.101.235.38:3300/0,v1:10.101.235.38:6789/0]}, election epoch 12, leader 0 ceph-1, quorum 0,1,2 ceph-1,ceph-2,ceph-3

ceph mon dump

1
2
3
4
5
6
7
8
9
10
[[email protected] ceph-cluster]# ceph mon dump 
epoch 3
fsid d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
last_changed 2021-07-21 23:12:49.336700
created 2021-07-21 17:31:24.761362
min_mon_release 14 (nautilus)
0: [v2:10.101.235.84:3300/0,v1:10.101.235.84:6789/0] mon.ceph-1
1: [v2:10.101.235.217:3300/0,v1:10.101.235.217:6789/0] mon.ceph-2
2: [v2:10.101.235.38:3300/0,v1:10.101.235.38:6789/0] mon.ceph-3
dumped monmap epoch 3

创建 mgr

ceph-deploy mgr create ceph-2 ceph-3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
[[email protected] ceph-cluster]# ceph-deploy mgr create ceph-2 ceph-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy mgr create ceph-2 ceph-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  mgr                           : [('ceph-2', 'ceph-2'), ('ceph-3', 'ceph-3')]
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0xef1c20>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mgr at 0xe83f50>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mgr][DEBUG ] Deploying mgr, cluster ceph hosts ceph-2:ceph-2 ceph-3:ceph-3
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
[ceph-2][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-2
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-2][WARNIN] mgr keyring does not exist yet, creating one
[ceph-2][DEBUG ] create a keyring file
[ceph-2][DEBUG ] create path recursively if it doesn't exist
[ceph-2][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-2 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-2/keyring
[ceph-2][INFO  ] Running command: systemctl enable [email protected]
[ceph-2][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-2][INFO  ] Running command: systemctl start [email protected]
[ceph-2][INFO  ] Running command: systemctl enable ceph.target
[ceph-3][DEBUG ] connected to host: ceph-3 
[ceph-3][DEBUG ] detect platform information from remote host
[ceph-3][DEBUG ] detect machine type
[ceph_deploy.mgr][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mgr][DEBUG ] remote host will use systemd
[ceph_deploy.mgr][DEBUG ] deploying mgr bootstrap to ceph-3
[ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-3][WARNIN] mgr keyring does not exist yet, creating one
[ceph-3][DEBUG ] create a keyring file
[ceph-3][DEBUG ] create path recursively if it doesn't exist
[ceph-3][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mgr --keyring /var/lib/ceph/bootstrap-mgr/ceph.keyring auth get-or-create mgr.ceph-3 mon allow profile mgr osd allow * mds allow * -o /var/lib/ceph/mgr/ceph-ceph-3/keyring
[ceph-3][INFO  ] Running command: systemctl enable [email protected]
[ceph-3][WARNIN] Created symlink from /etc/systemd/system/ceph-mgr.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-3][INFO  ] Running command: systemctl start [email protected]
[ceph-3][INFO  ] Running command: systemctl enable ceph.target

ceph -s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[[email protected] ceph-cluster]# ceph -s
  cluster:
    id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
    health: HEALTH_WARN
            clock skew detected on mon.ceph-2, mon.ceph-3
 
  services:
    mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 7m)
    mgr: ceph-1(active, since 16m), standbys: ceph-2, ceph-3
    osd: 3 osds: 3 up (since 34m), 3 in (since 34m)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 94 TiB / 94 TiB avail
    pgs:

dashboard 面板

12.x luminous 以上版本提供

https://docs.ceph.com/en/nautilus/mgr/dashboard/

WebUI based on Angular/TypeScript

特性

https://docs.ceph.com/en/nautilus/mgr/dashboard/#feature-overview

  • 多用户 多权限
  • 支持 SSO （单一登陆）
  • SSL/TLS 支持：自签证书，CA 发行的证书
  • 审计：命令审计
  • i18n 国际化

管理和监控功能

  • 整个集群的健康
  • 内嵌 grafana 面板
  • 集群日志
  • 主机管理
  • 性能监控
  • 监控
  • 配置

yum install ceph-mgr-dashboard

自动安装 ceph-grafana-dashboards

ceph-grafana-dashboards 这个包里带有 grafana dashboard json

for i in {1..3};do echo $i;ssh ceph-$i “yum install -y ceph-mgr-dashboard”;done

开启 dashboard 记得要在所有 mgr 节点上安装 dashboard 包，否则会报错

就算 –force 忽略错误，如果当前 mgr 挂了，mgr 转移到别的 standby mgr 节点，你的 dashboard 将无法使用

ceph mgr module enable dashboard

SSL

ceph dashboard create-self-signed-cert

1
2
[[email protected] ceph-cluster]# ceph dashboard create-self-signed-cert
Self-signed certificate created

openssl req -new -nodes -x509 -subj “/O=IT/CN=ceph-mgr-dashboard” -days 3650 -keyout dashboard.key -out dashboard.crt -extensions v3_ca

1
2
3
4
5
6
[[email protected] ceph-cluster]# openssl req -new -nodes -x509   -subj "/O=IT/CN=ceph-mgr-dashboard" -days 3650   -keyout dashboard.key -out dashboard.crt -extensions v3_ca
Generating a 2048 bit RSA private key
.....................................+++
........+++
writing new private key to 'dashboard.key'
-----

ceph config-key set mgr mgr/dashboard/crt -i dashboard.crt
ceph config-key set mgr mgr/dashboard/key -i dashboard.key
ceph config-key set mgr/dashboard/ceph1/crt -i dashboard.crt

1
2
3
4
5
6
7
[[email protected] ceph-cluster]# ceph config-key set mgr mgr/dashboard/crt -i dashboard.crt
set mgr
[[email protected] ceph-cluster]# ceph config-key set mgr mgr/dashboard/key -i dashboard.key
set mgr
[[email protected] ceph-cluster]# ceph config-key set mgr/dashboard/ceph1/crt -i dashboard.crt
WARNING: it looks like you might be trying to set a ceph-mgr module configuration key.  Since Ceph 13.0.0 (Mimic), mgr module configuration is done with `config set`, and new values set using `config-key set` will be ignored.
set mgr/dashboard/ceph1/crt

ceph dashboard set-ssl-certificate -i dashboard.crt
ceph dashboard set-ssl-certificate-key -i dashboard.key
ceph dashboard set-ssl-certificate ceph-1 -i dashboard.crt
ceph dashboard set-ssl-certificate-key ceph-1 -i dashboard.key

1
2
3
4
5
6
7
8
[[email protected] ceph-cluster]# ceph dashboard set-ssl-certificate -i dashboard.crt
SSL certificate updated
[[email protected] ceph-cluster]# ceph dashboard set-ssl-certificate-key -i dashboard.key
SSL certificate key updated
[[email protected] ceph-cluster]# ceph dashboard set-ssl-certificate  ceph-1 -i dashboard.crt
SSL certificate updated
[[email protected] ceph-cluster]# ceph dashboard set-ssl-certificate-key ceph-1 -i dashboard.key
SSL certificate key updated

ceph mgr services

1
2
3
4
[[email protected] ceph-cluster]# ceph mgr services
{
    "dashboard": "https://ceph-1:8443/"
}

创建用户
n版本，需要把密码写在文本中传进去
模版
ceph dashboard ac-user-create admin -i password.txt administrator

不太懂这个操作，防止在 history 里面泄露么

部署 mds

ceph-deploy mds create ceph-1 ceph-2 ceph-3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
[[email protected] ceph-cluster]# ceph-deploy mds create ceph-1 ceph-2 ceph-3
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy mds create ceph-1 ceph-2 ceph-3
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1b0d7e8>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x1af3d70>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('ceph-1', 'ceph-1'), ('ceph-2', 'ceph-2'), ('ceph-3', 'ceph-3')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts ceph-1:ceph-1 ceph-2:ceph-2 ceph-3:ceph-3
[ceph-1][DEBUG ] connected to host: ceph-1 
[ceph-1][DEBUG ] detect platform information from remote host
[ceph-1][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-1
[ceph-1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-1][WARNIN] mds keyring does not exist yet, creating one
[ceph-1][DEBUG ] create a keyring file
[ceph-1][DEBUG ] create path if it doesn't exist
[ceph-1][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-1 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-1/keyring
[ceph-1][INFO  ] Running command: systemctl enable [email protected]
[ceph-1][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-1][INFO  ] Running command: systemctl start [email protected]
[ceph-1][INFO  ] Running command: systemctl enable ceph.target
[ceph-2][DEBUG ] connected to host: ceph-2 
[ceph-2][DEBUG ] detect platform information from remote host
[ceph-2][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-2
[ceph-2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-2][WARNIN] mds keyring does not exist yet, creating one
[ceph-2][DEBUG ] create a keyring file
[ceph-2][DEBUG ] create path if it doesn't exist
[ceph-2][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-2 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-2/keyring
[ceph-2][INFO  ] Running command: systemctl enable [email protected]
[ceph-2][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-2][INFO  ] Running command: systemctl start [email protected]
[ceph-2][INFO  ] Running command: systemctl enable ceph.target
[ceph-3][DEBUG ] connected to host: ceph-3 
[ceph-3][DEBUG ] detect platform information from remote host
[ceph-3][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: CentOS Linux 7.4.1708 Core
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to ceph-3
[ceph-3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph-3][WARNIN] mds keyring does not exist yet, creating one
[ceph-3][DEBUG ] create a keyring file
[ceph-3][DEBUG ] create path if it doesn't exist
[ceph-3][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.ceph-3 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-ceph-3/keyring
[ceph-3][INFO  ] Running command: systemctl enable [email protected]
[ceph-3][WARNIN] Created symlink from /etc/systemd/system/ceph-mds.target.wants/[email protected] to /usr/lib/systemd/system/[email protected]
[ceph-3][INFO  ] Running command: systemctl start [email protected]
[ceph-3][INFO  ] Running command: systemctl enable ceph.target
[[email protected] ceph-cluster]#

ceph -s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[[email protected] ceph-cluster]# ceph -s
  cluster:
    id:     d26fd4cc-7ba1-4744-91d5-f5ccf291c5eb
    health: HEALTH_WARN
            clock skew detected on mon.ceph-2, mon.ceph-3
 
  services:
    mon: 3 daemons, quorum ceph-1,ceph-2,ceph-3 (age 68m)
    mgr: ceph-1(active, since 84m), standbys: ceph-2, ceph-3
    mds:  3 up:standby
    osd: 3 osds: 3 up (since 2h), 3 in (since 2h)
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0 objects, 0 B
    usage:   3.0 GiB used, 94 TiB / 94 TiB avail
    pgs:

3 个 up:standy

因为当前没有文件系统

重做 单盘 raid0

https://medium.com/@george.shuklin/how-to-remove-osd-from-ceph-cluster-b4c37cc0ec87

osd invalid

  1. ceph osd out osd.11
  2. If you see “osd.11 is already out” — it’s ok.
  3. ceph osd down osd.11
  4. Remove it: ceph osd rm osd.11. If it says ‘Error EBUSY: osd.11 is still up; must be down before removal.’ that means OSD is not dead yet. Go to the host it resides on and kill it (systemctl stop [email protected]), and repeat rm operation.
  5. Now it would list in ceph osd tree with ‘DNE’ status (DNE = do not exists). To clean up this status, remove it from CRUSH map: ceph osd crush rm osd.11
  6. Last step: remove it authorization (it should prevent problems with ‘couldn’t add new osd with same number’): ceph auth del osd.