事件2RAC 投票盘Voting Disk故障恢复1.1 事件概述1.1.1 事件描述RAC集群中的投票盘Voting Disk是Clusterware维护集群一致性的核心组件用于脑裂检测和仲裁。在2节点RAC环境中通常配置3个投票盘以提供冗余。当投票盘发生故障时根据损坏数量的不同1个、2个或3个会导致不同程度的后果——从集群继续运行到完全宕机。本文档详细说明这三种故障场景的具体表现、影响分析以及标准化的恢复步骤。1.1.2 适用范围Oracle 19c Grid Infrastructure2节点RAC环境3个投票盘的标准配置投票盘存放于ASM磁盘组如OCR_VOTE操作系统Linux/Unix平台1.1.3 投票盘工作原理简介仲裁机制Oracle Clusterware采用多数可用算法进行集群脑裂仲裁。一个集群要正常运作必须有超过半数的投票盘在线。2节点集群配置3个投票盘时多数要求为2个1.5因此损坏1个2个在线满足多数要求集群继续运行损坏2个1个在线不满足多数要求集群立即宕机损坏3个0个在线不满足多数要求集群立即宕机1.1.4 故障影响等级损坏数量影响等级业务影响集群状态恢复难度响应时限1个低无影响业务持续集群存活低在线修复24小时内修复2个高服务完全中断集群宕机中需停机恢复立即响应3个严重服务完全中断集群宕机中高停机重建立即响应1.2 故障诊断流程1.2.1 快速诊断步骤步骤1检查集群整体状态# 执行命令 [gridrac1 ~]$ crsctl check cluster -all # 正常情况下的输出示例集群运行正常 ************************************************************************* rac1 ************************************************************************* CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************************* rac2 ************************************************************************* CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online # 损坏2个或3个投票盘后的输出示例集群已宕机 [gridrac1 ~]$ crsctl check cluster CRS-4639: Could not contact Cluster Ready Services CRS-4000: Command Check failed, or completed with errors.步骤2查看投票盘状态# 执行命令 [gridrac1 ~]$ crsctl query css votedisk # 场景1正常情况下的输出示例3个ONLINE ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE a1b2c3d4e5f6g7h8i9j0 (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE b2c3d4e5f6g7h8i9j0k1 (/dev/asm-ocr2) [OCR_VOTE] 3. ONLINE c3d4e5f6g7h8i9j0k1l2 (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s). # 场景2损坏1个投票盘的输出示例1个OFFLINE ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE a1b2c3d4e5f6g7h8i9j0 (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE b2c3d4e5f6g7h8i9j0k1 (/dev/asm-ocr2) [OCR_VOTE] 3. OFFLINE c3d4e5f6g7h8i9j0k1l2 (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s). # 场景3损坏2个投票盘的输出示例2个OFFLINE ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE a1b2c3d4e5f6g7h8i9j0 (/dev/asm-ocr1) [OCR_VOTE] 2. OFFLINE b2c3d4e5f6g7h8i9j0k1 (/dev/asm-ocr2) [OCR_VOTE] 3. OFFLINE c3d4e5f6g7h8i9j0k1l2 (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s). # 场景4损坏3个投票盘的输出示例3个OFFLINE或命令执行失败 ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. OFFLINE a1b2c3d4e5f6g7h8i9j0 (/dev/asm-ocr1) [OCR_VOTE] 2. OFFLINE b2c3d4e5f6g7h8i9j0k1 (/dev/asm-ocr2) [OCR_VOTE] 3. OFFLINE c3d4e5f6g7h8i9j0k1l2 (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s). # 或当集群完全宕机时的输出 [gridrac1 ~]$ crsctl query css votedisk CRS-4678: Successful get but no voting disk present - may be because Oracle Clusterware is down步骤3查看CSS日志定位故障原因# 查看CSS日志了解投票盘故障详情 [gridrac1 ~]$ tail -100 $GRID_HOME/log/rac1/cssd/ocssd.log # 损坏1个投票盘的日志示例 2024-01-20 10:23:45.567: [CSSD][123456] clssnmvDiskVerify: voting disk 3 (/dev/asm-ocr3) open failed with error 4 2024-01-20 10:23:45.568: [CSSD][123456] clssnmvDiskCheck: voting disk /dev/asm-ocr3 verification failed, removing 2024-01-20 10:23:45.569: [CSSD][123456] clssnmvDiskVoteDiskState: voting disk 3 is now OFFLINE 2024-01-20 10:23:45.570: [CSSD][123456] clssnmvDiskKillCheck: voting disk count3, majority2, now voting offline count1 2024-01-20 10:23:45.571: [CSSD][123456] clssnmvDiskKillCheck: cluster can continue, voting disk count sufficient # 损坏2个投票盘的日志示例 2024-01-20 14:20:15.123: [CSSD][234567] clssnmvDiskVerify: voting disk 1 (/dev/asm-ocr1) I/O error 2024-01-20 14:20:15.124: [CSSD][234567] clssnmvDiskVerify: voting disk 2 (/dev/asm-ocr2) I/O error 2024-01-20 14:20:15.125: [CSSD][234567] clssnmvDiskVoteDiskState: voting disk 1 is now OFFLINE 2024-01-20 14:20:15.126: [CSSD][234567] clssnmvDiskVoteDiskState: voting disk 2 is now OFFLINE 2024-01-20 14:20:15.127: [CSSD][234567] clssnmvDiskVoteDiskState: voting disk 3 is now ONLINE 2024-01-20 14:20:15.128: [CSSD][234567] clssnmvDiskKillCheck: voting disk count3, majority2, now voting offline count2 2024-01-20 14:20:15.129: [CSSD][234567] clssnmvDiskKillCheck: insufficient voting disks, terminating cluster 2024-01-20 14:20:15.130: [CSSD][234567] clssnmCompleteNode: Node rac1 will be removed from cluster configuration1.2.2 故障场景分类表场景损坏数量集群状态诊断命令输出特征处理策略场景A1个存活crsctl query css votedisk显示1个OFFLINE在线修复无需停机场景B2个宕机集群不可用crsctl check cluster失败停机恢复利用OCR备份场景C3个宕机集群不可用crsctl query css votedisk显示0个盘停机恢复可能需重新配置1.3 故障恢复流程1.3.1 恢复前的准备工作无论哪种故障场景在开始恢复前需完成以下准备工作# 1. 确认OCR备份可用性 [rootrac1 ~]# $GRID_HOME/bin/ocrconfig -showbackup rac1 2024/01/20 03:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/backup_20240120_030000.ocr rac1 2024/01/19 15:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/backup_20240119_150000.ocr rac1 2024/01/19 03:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/backup_20240119_030000.ocr rac2 2024/01/20 01:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/day.ocr # 2. 确认投票盘所在ASM磁盘组名称 [rootrac1 ~]# $GRID_HOME/bin/asmcmd lsdg | grep -i ocr OCR_VOTE NORMAL MOUNTED 3 2 ... /dev/asm-ocr* # 3. 准备新磁盘如果需要替换损坏的磁盘 # 确认新磁盘已配置共享访问且所有节点都能看到相同路径 [rootrac1 ~]# ls -l /dev/asm-ocr_new* brw-rw---- 1 grid asmadmin 8, 64 Jan 20 11:00 /dev/asm-ocr_new1.3.2 场景A损坏1个投票盘在线修复适用条件集群仍在运行crsctl query css votedisk显示1个盘为OFFLINE。恢复步骤步骤A1确认故障盘信息[gridrac1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE a1b2c3d4e5f6g7h8i9j0 (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE b2c3d4e5f6g7h8i9j0k1 (/dev/asm-ocr2) [OCR_VOTE] 3. OFFLINE c3d4e5f6g7h8i9j0k1l2 (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s). # 记录故障盘的FUID: c3d4e5f6g7h8i9j0k1l2步骤A2准备新磁盘# 如果原盘已修复使用原盘路径 # 如果使用新盘确认新盘已正确配置 [rootrac1 ~]# ls -l /dev/asm-ocr3_new brw-rw---- 1 grid asmadmin 8, 64 Jan 20 11:05 /dev/asm-ocr3_new # 确认新磁盘在所有节点权限一致 [rootrac2 ~]# ls -l /dev/asm-ocr3_new brw-rw---- 1 grid asmadmin 8, 64 Jan 20 11:05 /dev/asm-ocr3_new # 确认新磁盘已在ASM磁盘组中如果使用ASM管理 [gridrac1 ~]$ asmcmd lsdsk --diskgroup OCR_VOTE Path /dev/asm-ocr1 /dev/asm-ocr2 /dev/asm-ocr3_new步骤A3删除故障盘[gridrac1 ~]$ crsctl delete css votedisk c3d4e5f6g7h8i9j0k1l2 # 成功输出 CRS-4706: Voting disk deleted successfully.步骤A4验证删除后状态[gridrac1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE a1b2c3d4e5f6g7h8i9j0 (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE b2c3d4e5f6g7h8i9j0k1 (/dev/asm-ocr2) [OCR_VOTE] Located 2 voting disk(s).步骤A5添加新投票盘# # 方法1直接指定磁盘路径推荐最直接 # [gridrac1 ~]$ crsctl add css votedisk /dev/asm-ocr3_new -force # 成功输出 CRS-4606: Successful addition of voting disk /dev/asm-ocr3_new. # # 方法2使用ASM磁盘别名如果配置了ASM磁盘别名 # [gridrac1 ~]$ crsctl add css votedisk /dev/asm/ocr3_new -force # 成功输出 CRS-4606: Successful addition of voting disk /dev/asm/ocr3_new. # # ⚠️ 注意以下命令是错误的请勿使用 # # crsctl add css votedisk OCR_VOTE -force ← 这个命令会执行失败 # 错误信息CRS-4611: Voting file location OCR_VOTE is not a valid voting disk path.步骤A6验证最终状态[gridrac1 ~]$ crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE a1b2c3d4e5f6g7h8i9j0 (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE b2c3d4e5f6g7h8i9j0k1 (/dev/asm-ocr2) [OCR_VOTE] 3. ONLINE d4e5f6g7h8i9j0k1l2m3 (/dev/asm-ocr3_new) [OCR_VOTE] Located 3 voting disk(s). # 检查CSS日志确认 [gridrac1 ~]$ tail -5 $GRID_HOME/log/rac1/cssd/ocssd.log 2024-01-20 11:15:12.345: [CSSD][123456] clssnmvDiskAdd: voting disk 3 added successfully 2024-01-20 11:15:12.346: [CSSD][123456] clssnmvDiskVoteDiskState: voting disk 3 is now ONLINE 2024-01-20 11:15:12.347: [CSSD][123456] clssnmvDiskKillCheck: all voting disks ONLINE, cluster healthy1.3.3 场景B损坏2个投票盘停机恢复适用条件集群已宕机超过半数的投票盘不可用2个或3个OFFLINE但OCR备份可用存储已修复。恢复步骤步骤B1强制停止所有节点的集群# 在所有节点以root用户执行 [rootrac1 ~]# crsctl stop crs -f CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on rac1 CRS-2673: Attempting to stop ora.crsd on rac1 CRS-2673: Attempting to stop ora.asm on rac1 CRS-2677: Stop of ora.asm on rac1 succeeded CRS-2677: Stop of ora.crsd on rac1 succeeded CRS-2673: Attempting to stop ora.cssd on rac1 CRS-2677: Stop of ora.cssd on rac1 succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on rac1 complete [rootrac2 ~]# crsctl stop crs -f ...类似输出 # 确认进程已全部停止 [rootrac1 ~]# ps -ef | grep d.bin | grep -v grep [rootrac1 ~]#步骤B2修复存储并确认盘符# 确保所有投票盘对应的物理设备已修复 # 检查磁盘是否存在 [rootrac1 ~]# ls -l /dev/asm-ocr* brw-rw---- 1 grid asmadmin 8, 16 Jan 20 14:35 /dev/asm-ocr1 brw-rw---- 1 grid asmadmin 8, 32 Jan 20 14:35 /dev/asm-ocr2 brw-rw---- 1 grid asmadmin 8, 48 Jan 20 14:35 /dev/asm-ocr3 # 在节点2同样检查确保路径一致 [rootrac2 ~]# ls -l /dev/asm-ocr* brw-rw---- 1 grid asmadmin 8, 16 Jan 20 14:35 /dev/asm-ocr1 brw-rw---- 1 grid asmadmin 8, 32 Jan 20 14:35 /dev/asm-ocr2 brw-rw---- 1 grid asmadmin 8, 48 Jan 20 14:35 /dev/asm-ocr3步骤B3在节点1以排他模式启动集群# 在节点1执行 [rootrac1 ~]# crsctl start crs -excl -nocrs CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start ora.evmd on rac1 CRS-2672: Attempting to start ora.cssd on rac1 CRS-2676: Start of ora.evmd on rac1 succeeded CRS-2676: Start of ora.cssd on rac1 succeeded CRS-2672: Attempting to start ora.cluster_interconnect.haip on rac1 CRS-2676: Start of ora.cluster_interconnect.haip on rac1 succeeded # 等待30秒后检查状态 [rootrac1 ~]# crsctl check crs CRS-4638: Oracle High Availability Services is online CRS-4535: Cannot communicate with Cluster Ready Services CRS-4529: Cluster Synchronization Services is online CRS-4534: Cannot communicate with Event Manager # 注意CRS-4535和CRS-4534是正常的因为是以-excl -nocrs模式启动步骤B4查看OCR备份[rootrac1 ~]# ocrconfig -showbackup rac1 2024/01/20 03:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/backup_20240120_030000.ocr rac1 2024/01/19 15:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/backup_20240119_150000.ocr rac1 2024/01/19 03:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/backup_20240119_030000.ocr rac2 2024/01/20 01:00:00 /u01/app/19.0.0/grid/cdata/rac-cluster/day.ocr步骤B5恢复OCR# 选择最新的自动备份文件恢复 [rootrac1 ~]# ocrconfig -restore /u01/app/19.0.0/grid/cdata/rac-cluster/backup_20240120_030000.ocr # 验证OCR恢复 [rootrac1 ~]# ocrcheck Status of Oracle Cluster Registry is as follows : Version : 4 Total space (kbytes) : 409568 Used space (kbytes) : 4852 Available space (kbytes) : 404716 ID : 1234567890 Device/File Name : OCR_VOTE Device/File integrity check succeeded Cluster registry integrity check succeeded Logical corruption check succeeded步骤B6查看当前投票盘状态[rootrac1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- Located 0 voting disk(s). # 说明投票盘需要重建步骤B7重建投票盘# 使用ASM磁盘组重建所有投票盘 # ⚠️ 注意这个命令只能在排他模式下执行 [rootrac1 ~]# crsctl replace votedisk OCR_VOTE # 成功输出 CRS-4602: Starting asynchronous replacement of voting disk group OCR_VOTE. CRS-4603: Successful addition of voting disk 293f7e8a4f1f4f22bfa53f1e8d4a8f1c. CRS-4603: Successful addition of voting disk 3b4c5d6e7f8g9h0i1j2k3l4m5n6o7p8q. CRS-4603: Successful addition of voting disk 4c5d6e7f8g9h0i1j2k3l4m5n6o7p8q9r. CRS-4604: Successful deletion of 0 voting disk(s). CRS-4601: Successful replacement of voting disk group OCR_VOTE with 3 voting disks.步骤B8验证重建后的投票盘[rootrac1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE 293f7e8a4f1f4f22bfa53f1e8d4a8f1c (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE 3b4c5d6e7f8g9h0i1j2k3l4m5n6o7p8q (/dev/asm-ocr2) [OCR_VOTE] 3. ONLINE 4c5d6e7f8g9h0i1j2k3l4m5n6o7p8q9r (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s).步骤B9停止排他模式并正常启动# 在节点1停止集群 [rootrac1 ~]# crsctl stop crs CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on rac1 ...省略详细输出 CRS-2793: Shutdown of Oracle High Availability Services-managed resources on rac1 complete # 在节点1正常启动集群 [rootrac1 ~]# crsctl start crs CRS-4123: Oracle High Availability Services has been started. ...等待约2分钟 # 在节点2启动集群 [rootrac2 ~]# crsctl start crs CRS-4123: Oracle High Availability Services has been started.步骤B10最终验证# 在节点1执行完整验证 [rootrac1 ~]# crsctl check cluster -all ************************************************************************* rac1 ************************************************************************* CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************************* rac2 ************************************************************************* CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [rootrac1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE 293f7e8a4f1f4f22bfa53f1e8d4a8f1c (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE 3b4c5d6e7f8g9h0i1j2k3l4m5n6o7p8q (/dev/asm-ocr2) [OCR_VOTE] 3. ONLINE 4c5d6e7f8g9h0i1j2k3l4m5n6o7p8q9r (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s).1.3.4 场景C损坏3个投票盘全部丢失适用条件所有投票盘完全损坏集群已宕机。此场景有两种恢复方法方法C1利用现有OCR备份恢复同场景B方法C2无可用备份需重新配置集群Oracle 19c标准流程方法C1利用现有OCR备份恢复如果OCR备份可用且存储已修复按照场景B的步骤B1-B10操作即可。方法C2无可用备份重新配置集群Oracle 19c标准流程步骤C1强制停止所有节点的集群# 所有节点执行 [rootrac1 ~]# crsctl stop crs -f ...省略输出 [rootrac2 ~]# crsctl stop crs -f ...省略输出步骤C2在所有节点上执行集群去配置使用rootcrs.sh# 在每个节点上以root用户执行 [rootrac1 ~]# cd $GRID_HOME/crs/install [rootrac1 install]# ./rootcrs.sh -deconfig -force # 输出示例 2024-01-20 17:00:01: Running deconfig on cluster Stopping Clusterware resources CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources... ... Deconfiguring Oracle Cluster Registry Deconfiguring Oracle High Availability Services Deconfig operation completed successfully. # 节点2执行相同命令 [rootrac2 ~]# cd $GRID_HOME/crs/install [rootrac2 install]# ./rootcrs.sh -deconfig -force步骤C3清理残留文件和磁盘头# 在所有节点执行残留文件清理 [rootrac1 ~]# rm -rf /etc/oracle [rootrac1 ~]# rm -rf /var/tmp/.oracle [rootrac1 ~]# rm -rf /tmp/.oracle [rootrac1 ~]# rm -f /etc/oraInst.loc [rootrac1 ~]# rm -f /etc/oratab # 检查并清除inittab中的CRS条目如果有 [rootrac1 ~]# cat /etc/inittab | grep -i crs # 如果有输出编辑/etc/inittab删除相关行 [rootrac1 ~]# vi /etc/inittab # 清除OCR和投票盘磁盘头仅在节点1执行即可因为磁盘是共享的 [rootrac1 ~]# dd if/dev/zero of/dev/asm-ocr1 bs1M count100 1000 records in 1000 records out 104857600 bytes (105 MB) copied, 1.23456 s, 85.0 MB/s [rootrac1 ~]# dd if/dev/zero of/dev/asm-ocr2 bs1M count100 [rootrac1 ~]# dd if/dev/zero of/dev/asm-ocr3 bs1M count100步骤C4在节点1重新配置集群[rootrac1 ~]# cd $GRID_HOME/crs/install [rootrac1 install]# ./rootcrs.sh -config # 输出示例关键信息 2024-01-20 17:05:01: Running configuration on cluster Configuring Oracle Cluster Registry (OCR) Initializing OCR with voting disk group OCR_VOTE OCR configured successfully Configuring Oracle High Availability Services CRS-4123: Oracle High Availability Services has been started. CRS-2672: Attempting to start ora.evmd on rac1 CRS-2672: Attempting to start ora.cssd on rac1 ... Configuration completed successfully.步骤C5验证节点1集群启动[rootrac1 ~]# crsctl check cluster CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online [rootrac1 ~]# crsctl query css votedisk ## STATE File Universal Id File Name Disk group -- ----- --------------------- --------- ---------- 1. ONLINE 5a6b7c8d9e0f1a2b3c4d5e6f (/dev/asm-ocr1) [OCR_VOTE] 2. ONLINE 6b7c8d9e0f1a2b3c4d5e6f7g (/dev/asm-ocr2) [OCR_VOTE] 3. ONLINE 7c8d9e0f1a2b3c4d5e6f7g8h (/dev/asm-ocr3) [OCR_VOTE] Located 3 voting disk(s).步骤C6在节点2加入集群[rootrac2 ~]# cd $GRID_HOME/crs/install [rootrac2 install]# ./rootcrs.sh -config ...等待加入完成步骤C7验证两节点集群状态[rootrac1 ~]# crsctl check cluster -all ************************************************************************* rac1 ************************************************************************* CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online ************************************************************************* rac2 ************************************************************************* CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online步骤C8重建ASM磁盘组如果丢失# 以grid用户执行 [gridrac1 ~]$ sqlplus / as sysasm SQL*Plus: Release 19.0.0.0.0 - Production on Mon Jan 20 17:30:00 2024 Version 19.3.0.0.0 Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0 SQL CREATE DISKGROUP OCR_VOTE NORMAL REDUNDANCY 2 DISK /dev/asm-ocr1 NAME ocr1, 3 /dev/asm-ocr2 NAME ocr2, 4 /dev/asm-ocr3 NAME ocr3 5 ATTRIBUTE compatible.asm 19.0.0; Diskgroup created. SQL select name, state, type from v$asm_diskgroup; NAME STATE TYPE ------------------------------ ----------- --------------- OCR_VOTE MOUNTED NORMAL DATA MOUNTED EXTERN SQL exit步骤C9重建监听器和数据库资源根据实际情况执行# 重建监听器 [gridrac1 ~]$ netca -silent -responsefile $GRID_HOME/assistants/netca/netca.rsp # 重建数据库资源 [gridrac1 ~]$ srvctl add database -db orcl -oraclehome /u01/app/oracle/product/19.0.0/dbhome_1 -dbtype RAC -spfile DATA/ORCL/spfileorcl.ora -startoption open [gridrac1 ~]$ srvctl add instance -db orcl -instance orcl1 -node rac1 [gridrac1 ~]$ srvctl add instance -db orcl -instance orcl2 -node rac2 [gridrac1 ~]$ srvctl start database -db orcl [gridrac1 ~]$ srvctl status database -db orcl Instance orcl1 is running on node rac1 Instance orcl2 is running on node rac21.4 命令使用场景总结表命令适用场景集群状态示例说明crsctl delete css votedisk FUID场景A正常运行crsctl delete css votedisk c3d4e5f6...删除单个故障盘crsctl add css votedisk 磁盘路径场景A正常运行crsctl add css votedisk /dev/asm-ocr3_new添加单个新盘crsctl replace votedisk DGNAME场景B/C排他模式crsctl replace votedisk OCR_VOTE重建所有投票盘ocrconfig -restore 备份文件场景B/C排他模式ocrconfig -restore /backup/ocr.ocr恢复OCR./rootcrs.sh -deconfig -force场景C2所有节点停止在$GRID_HOME/crs/install下执行彻底清除集群配置./rootcrs.sh -config场景C2节点停止在$GRID_HOME/crs/install下执行重新配置集群⚠️ 重要提醒❌不要在场景A中使用crsctl replace votedisk OCR_VOTE——这个命令需要排他模式❌不要在场景A中使用crsctl add css votedisk OCR_VOTE——语法错误✅场景A正确命令crsctl add css votedisk /dev/asm-ocr3_new -force✅场景B/C正确命令crsctl replace votedisk OCR_VOTE排他模式下✅场景C2正确去配置命令./rootcrs.sh -deconfig -forceOracle 19c标准1.5 故障恢复后的验证清单验证项命令预期结果实际结果集群整体状态crsctl check cluster -all所有节点ONLINE✓投票盘状态crsctl query css votedisk3个盘均为ONLINE✓OCR完整性ocrcheckStatus 3无错误✓ASM实例srvctl status asm所有节点ASM running✓ASM磁盘组asmcmd lsdgOCR_VOTE MOUNTED✓监听器srvctl status listener所有节点LISTENER running✓数据库实例srvctl status database -db dbname所有实例OPEN✓CSS日志tail -20 $GRID_HOME/log/rac1/cssd/ocssd.log无错误所有盘ONLINE✓文档信息项目内容事件编号EVENT-002事件名称RAC投票盘Voting Disk故障恢复文档版本4.0最后更新2024-01-21编写人DBA团队审核人DBA经理维护记录版本日期修改人修改说明1.02024-01-01DBA_A初始版本创建2.02024-01-15DBA_B增加详细命令输出示例3.02024-01-20DBA_C重构为投票盘专项文档3.12024-01-21DBA_D修正场景A中错误命令增加命令对比表4.02024-01-21DBA_E修正场景C中过时脚本采用rootcrs.sh标准流程