### 准备工作 所有节点: - 系统: `CentOS7.6` - 数据库: `MariaDB 5.5.60` - VIP: `172.16.0.200` node1节点： - IP/主机：`172.16.0.100` node2节点: - IP/主机：`172.16.0.101` 网络配置如下图    ### 安装Pacemaker和Corosync #### 安装Pacemaker,Corosync,pcs `node1和node2执行` ```bash yum -y install corosync pacemaker pcs ``` #### 设置集群用户密码 `node1和node2执行` ```bash echo "passwd" | passwd hacluster --stdin ``` 启动和开启服务 `node1和node2执行` ```bash systemctl start pcsd systemctl enable pcsd pcs cluster enable --all #配置集群服务开机启动 ``` #### 配置Corosync `node1执行` 认证用户hacluster,将授权tokens存储在文件/var/lib/pcsd/tokens中. ```bash pcs cluster auth node1 node2 -u hacluster -p passwd ``` ``` node1: Authorized node2: Authorized ``` #### 生成和同步Corosync配置 `node1执行` ```bash pcs cluster setup --name mysql_cluster node1 node2 ``` #### 在所有节点启动集群 `node1执行` ```bash pcs cluster start --all ``` ### 安装DRBD `node1和node2执行` ```bash rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm yum install -y kmod-drbd84 drbd84-utils ``` #### 配置DRBD `node1和node2执行` ```bash vim /etc/drbd.d/mysql01.res ``` ```ini resource mysql01 { protocol C; meta-disk internal; device /dev/drbd0; disk /dev/vdb; #/dev/vdb为空闲的块设备，可以LVM创建一个逻辑卷 handlers { split-brain "/usr/lib/drbd/notify-split-brain.sh root"; } net { allow-two-primaries no; after-sb-0pri discard-zero-changes; after-sb-1pri discard-secondary; after-sb-2pri disconnect; rr-conflict disconnect; } disk { on-io-error detach; } syncer { verify-alg sha1; } on node1 { address 172.16.0.100:7789; } on node2 { address 172.16.0.101:7789; } } ``` #### 初始化DRBD`(创建DRBD metadata)` `node1和node2执行` ```bash drbdadm create-md mysql01 ``` #### 启动mysql01 `node1和node2执行` ```bash drbdadm up mysql01 ``` #### 指定主节点 `node1执行` ```bash drbdadm primary --force mysql01 ``` #### 查看drbd状态 `node1执行` ```bash cat /proc/drbd ``` ``` version: 8.4.11-1 (api:1/proto:86-101) GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r----- ns:136 nr:288 dw:428 dr:13125 al:5 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0 ``` #### 快速同步 ```bash drbdadm new-current-uuid --clear-bitmap mysql01/0 ``` #### 格式化drbd设备 等待上面主从的块设备同步(UpToDate/UpToDate)之后执行 格式化drbd成ext4格式 `node1执行` ```bash mkfs.ext4 -m 0 -L drbd /dev/drbd0 tune2fs -c 30 -i 180d /dev/drbd0 ``` #### 挂载drbd设备 `node1执行` ```bash mount /dev/drbd0 /mnt ```   ### 安装MariaDB `node1和node2执行` ```bash yum install -y mariadb-server mariadb systemctl disable mariadb.service #设置开启不启动，通过pacemaker去管理 ``` `node1执行` ```bash systemctl start mariadb ``` #### 安装数据库 `node1执行` ```bash mysql_install_db --datadir=/mnt --user=mysql ``` #### 执行安全安装 `node1执行` ```bash mysql_secure_installation ``` #### 卸载drbd和停止数据库 `node1执行` ```bash umount /mnt #卸载目录 systemctl stop mariadb #停止数据库 ``` #### 配置mysql `node1和node2执行` ```bash vim /etc/my.cnf ``` ```ini [mysqld] symbolic-links=0 bind_address = 0.0.0.0 datadir = /var/lib/mysql pid_file = /var/run/mariadb/mysqld.pid socket = /var/run/mariadb/mysqld.sock [mysqld_safe] bind_address = 0.0.0.0 datadir = /var/lib/mysql pid_file = /var/run/mariadb/mysqld.pid socket = /var/run/mariadb/mysqld.sock !includedir /etc/my.cnf.d ```   ### 配置Pacemaker集群 #### 配置逻辑和顺序如下 ```bash Start: mysql_fs01 -> mysql_service01 -> mysql_VIP01, Stop: mysql_VIP01 -> mysql_service01 -> mysql_fs01. ``` mysql_fs01是文件系统资源,mysql_service01是服务资源，mysql_VIP01是浮动虚拟IP `172.16.0.200` pcs具有的一个方便功能是能够将多个更改排入文件并以原子方式提交这些更改。为此，我们首先使用CIB中的当前原始XML配置填充文件 `node1执行` ```bash pcs cluster cib clust_cfg ``` 关闭STONITH(`注意:依赖具体的环境视情况操作`) `node1执行` ```bash pcs -f clust_cfg property set stonith-enabled=false ``` 设置quorum策略为ignore `node1执行` ```bash pcs -f clust_cfg property set no-quorum-policy=ignore ``` 防止资源在恢复后移动，因为它通常会增加停机时间 `node1执行` ```bash pcs -f clust_cfg resource defaults resource-stickiness=200 ``` 为了达到这个效果，Pacemaker 有一个叫做“资源粘性值”的概念，它能够控制一个服务(资源)有多想呆在它正在运行的节点上。 Pacemaker为了达到最优分布各个资源的目的，默认设置这个值为0。我们可以为每个资源定义不同的粘性值，但一般来说，更改默认粘性值就够了。资源粘性表示资源是否倾向于留在当前节点，如果为正整数，表示倾向，负数则会离开，-inf表示负无穷，inf表示正无穷。   为drbd设备创建名为mysql_data01的集群资源和一个额外的克隆资源MySQLClone01，允许资源同时在两个集群节点上运行 `node1执行` ```bash pcs -f clust_cfg resource create mysql_data01 ocf:linbit:drbd \ drbd_resource=mysql01 \ op monitor interval=30s ``` ```bash pcs -f clust_cfg resource master MySQLClone01 mysql_data01 \ master-max=1 master-node-max=1 \ clone-max=2 clone-node-max=1 \ notify=true ``` master-max: 可以将多少资源副本提升为主状态 master-node-max: 可以在单个节点上将多少个资源副本提升为主状态 clone-max: 要启动多少个资源副本。默认为群集中的节点数 clone-node-max: 可以在单个节点上启动多少个资源副本 notify: 停止或启动克隆副本时，请事先告知所有其他副本以及操作何时成功   为文件系统创建名为mysql_fs01的集群资源,告诉群集克隆资源MySQLClone01必须在与文件系统资源相同的节点上运行，并且必须在文件系统资源之前启动克隆资源。 `node1执行` ```bash pcs -f clust_cfg resource create mysql_fs01 Filesystem \ device="/dev/drbd0" \ directory="/var/lib/mysql" \ fstype="ext4" ``` ```bash pcs -f clust_cfg constraint colocation add mysql_fs01 with MySQLClone01 \ INFINITY with-rsc-role=Master ``` ```bash pcs -f clust_cfg constraint order promote MySQLClone01 then start mysql_fs01 ``` 为MariaDB服务创建名为mysql_service01的集群资源。告诉群集MariaDB服务必须在与mysql_fs01文件系统资源相同的节点上运行，并且必须首先启动文件系统资源。 `node1执行` ```bash pcs -f clust_cfg resource create mysql_service01 ocf:heartbeat:mysql \ binary="/usr/bin/mysqld_safe" \ config="/etc/my.cnf" \ datadir="/var/lib/mysql" \ pid="/var/lib/mysql/mysql.pid" \ socket="/var/lib/mysql/mysql.sock" \ additional_parameters="--bind-address=0.0.0.0" \ op start timeout=60s \ op stop timeout=60s \ op monitor interval=20s timeout=30s ``` ```bash pcs -f clust_cfg constraint colocation add mysql_service01 with mysql_fs01 INFINITY ``` ```bash pcs -f clust_cfg constraint order mysql_fs01 then mysql_service01 ``` 为虚拟IP 172.16.0.200创建名为mysql_VIP01的集群资源 `node1执行` ```bash pcs -f clust_cfg resource create mysql_VIP01 ocf:heartbeat:IPaddr2 \ ip=172.16.0.200 cidr_netmask=32 \ op monitor interval=30s ``` 当然，虚拟IP mysql_VIP01资源必须与MariaDB资源在同一节点上运行，并且必须在最后一个时启动。这是为了确保在连接到虚拟IP之前已经启动了所有其他资源。 `node1执行` ```bash pcs -f clust_cfg constraint colocation add mysql_VIP01 with mysql_service01 INFINITY ``` ```bash pcs -f clust_cfg constraint order mysql_service01 then mysql_VIP01 ``` 检查配置 `node1执行` ```bash pcs -f clust_cfg constraint ``` ``` Location Constraints: Ordering Constraints: promote MySQLClone01 then start mysql_fs01 (kind:Mandatory) start mysql_fs01 then start mysql_service01 (kind:Mandatory) start mysql_service01 then start mysql_VIP01 (kind:Mandatory) Colocation Constraints: mysql_fs01 with MySQLClone01 (score:INFINITY) (with-rsc-role:Master) mysql_service01 with mysql_fs01 (score:INFINITY) mysql_VIP01 with mysql_service01 (score:INFINITY) ``` ```bash pcs -f clust_cfg resource show ``` ``` Master/Slave Set: MySQLClone01 [mysql_data01] Stopped: [ node1 node2 ] mysql_fs01 (ocf::heartbeat:Filesystem): Stopped mysql_service01 (ocf::heartbeat:mysql): Stopped mysql_VIP01 (ocf::heartbeat:IPaddr2): Stopped ``` 提交修改并查看集群状态 `node1执行` ```bash pcs cluster cib-push clust_cfg ``` ```bash pcs status ``` ``` Cluster name: mysql_cluster Stack: corosync Current DC: node1 (version 1.1.19-8.el7_6.4-c3c624ea3d) - partition with quorum Last updated: Mon Jul 29 06:51:22 2019 Last change: Mon Jul 29 02:49:38 2019 by root via cibadmin on node1 2 nodes configured 5 resources configured Online: [ node1 node2 ] Full list of resources: Master/Slave Set: MySQLClone01 [mysql_data01] Masters: [ node1 ] Slaves: [ node2 ] mysql_fs01 (ocf::heartbeat:Filesystem): Started node1 mysql_service01 (ocf::heartbeat:mysql): Started node1 mysql_VIP01 (ocf::heartbeat:IPaddr2): Started node1 Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled ``` 一旦配置提交，Pacemaker将会执行以下操作 - 在集群节点启动DRBD - 选择一个节点提升为主节点 - 在同一个节点挂载文件系统，配置集群IP地址，启动MariaDB - 开始监控资源 通过telenet虚拟IP地址和3306端口，测试MariaDB服务 `client执行` ```bash telnet 172.16.0.200 3306 ``` ``` Trying 172.16.0.200... Connected to 172.16.0.200. Escape character is '^]'. GHost '172.16.0.200' is not allowed to connect to this MariaDB serverConnection closed by foreign host. ``` ### 常用命令汇总 查看集群状态： ```bash pcs status ``` 查看集群当前配置： ```bash pcs config ``` 开机后集群自启动： ```bash pcs cluster enable --all ``` 启动集群： ```bash pcs cluster start --all ``` 查看集群资源状态： ```bash pcs resource show ``` 验证集群配置情况： ```bash crm_verify -L -V ``` 测试资源配置： ```bash pcs resource debug-start resource ``` 设置节点为备用状态： ```bash pcs cluster standby node1 ``` 列出集群属性 ```bash pcs property list ``` 测试corosync成员 ```bash corosync-cmapctl | grep members ``` 查看corosync成员 ```bash pcs status corosync ``` 将集群配置保存到文件 ```bash pcs cluster cib filename ``` 创建资源不应用到集群，写入到文件 ```bash pcs -f testfile1 resource create VirtualIP ocf:heartbeat:IPaddr2 ip=192.168.0.120 cidr_netmask=24 op monitor interval=30s ``` 将文件配置应用到集群 ```bash pcs cluster cib-push filename ``` 备份集群配置 ```bash pcs config backup filename ``` 使用恢复集群配置 ```bash pcs config restore [--local] [filename] #--local只还原当前节点，没有执行filename则读取标准输入 ``` 添加集群节点 ```bash pcs cluster node add node ``` 删除集群节点 ```bash pcs cluster node remove node ``` 显示资源的参数 ```bash pcs resource describe standard:provider:type|type #例如:pcs resource describe ocf:heartbeat:IPaddr2 ``` 设置节点进入待机状态 ```bash pcs cluster standby node | --all ``` 设置节点从待机状态移除 ```bash pcs cluster unstandby node | --all ``` 删除集群配置(`警告:这个命令可永久移除已创建的集群配置`) ```bash pcs cluster stop pcs cluster destroy ``` 参考: https://www.lisenet.com/2016/activepassive-mysql-high-availability-pacemaker-cluster-with-drbd-on-centos-7/ https://linux.cn/article-3963-1.html https://www.howtoforge.com/tutorial/how-to-set-up-nginx-high-availability-with-pacemaker-corosync-on-centos-7/ http://www.alexlinux.com/pacemaker-corosync-nginx-cluster/ https://access.redhat.com/documentation/zh-cn/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/ch-clusteradmin-haar