【ASM】udev简介及配置、多路径（multipath）等
【ASM】udev简介及配置、多路径等
（一）udev是什么？如何配置udev？
udev是Linux 2.6内核里的一个功能，它替代了原来的devfs，成为当前Linux默认的设备管理工具。udev以守护进程的形式运行，通过侦听内核发出来的uevent来管理/dev目录下的设备文件。
在Linux中，所有的设备都是以设备文件的形式存在的。在早期的Linux版本中，“/dev”目录包含了所有可能出现的设备的设备文件，所以，Linux用户很难在这些大量的设备文件中找到匹配条件的设备文件。现在，udev只为那些连接到Linux操作系统的设备产生设备文件，并且udev能通过定义一个udev规则（rule）来产生匹配设备属性的设备文件，这些设备属性可以是内核设备名称、总线路径、厂商名称、型号、***或者磁盘大小等等。当设备添加/删除时，udev的守护进程侦听来自内核的uevent，以此添加或者删除/dev下的设备文件，所以udev只为已经连接的设备产生设备文件，而不会在/dev下产生大量虚无的设备文件。
配置udev有如下几个步骤：
1.确认在所有RAC节点上已经安装了必要的udev包
[[email protected] ~]# rpm -qa|grep udev
udev-095-14.21.el5
2.通过scsi_id获取设备的块设备的唯一标识名，假设系统上已有LUN sdc-sde
for i in c d e ;
do
echo "KERNEL==\"sd*\", BUS==\"scsi\", PROGRAM==\"/sbin/scsi_id --whitelisted --device=/dev/\$name\",RESULT==\"`scsi_id --whitelisted --device=/dev/sd$i`\",NAME=\"asm-disk$i\",OWNER=\"grid\",GROUP=\"asmadmin\",MODE=\"0660\""
done
3.创建必要的udev配置文件
首先切换到配置文件目录：
[[email protected] ~]# cd /etc/udev/rules.d
定义必要的规则配置文件：
[[email protected] rules.d]# cat 99-oracle-asmdevices.rules
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --device=/dev/$name",RESULT=="14f504e46494c455232326c6c76442d4361634f2d4d4f4d41",NAME="asm-diskc",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --device=/dev/$name",RESULT=="14f504e46494c455232326c6c76442d4361634f2d4d4f4d41",NAME="asm-diskd",OWNER="grid",GROUP="asmadmin",MODE="0660"
KERNEL=="sd*", BUS=="scsi", PROGRAM=="/sbin/scsi_id --whitelisted --device=/dev/$name",RESULT=="14f504e46494c455242674c7079392d753750482d63734443",NAME="asm-diske",OWNER="grid",GROUP="asmadmin",MODE="0660"
需要注意的是，一个KERNEL就是一行，不能换行。
4.将该规则文件拷贝到其它节点上
[[email protected] rules.d]# scp 99-oracle-asmdevices.rules Other_node:/etc/udev/rules.d
5.在所有节点上启动udev服务，或者重启服务器即可
[[email protected] rules.d]# /sbin/udevcontrol reload_rules
[[email protected] rules.d]# /sbin/start_udev
Starting udev: [ OK ]
6.检查设备是否到位
[[email protected] rules.d]# cd /dev
[[email protected] dev]# ls -l asm-disk*
brw-rw---- 1 grid asmadmin 8, 64 Jul 10 17:31 asm-diskc
brw-rw---- 1 grid asmadmin 8, 208 Jul 10 17:31 asm-diskd
brw-rw---- 1 grid asmadmin 8, 224 Jul 10 17:31 asm-diske
配置完成后也可以使用udevadm进行测试，这里不再详述。
UDEV简介及配置过程
原文地址：UDEV简介及配置过程 作者：kuqlan
在Linux环境下安装Oracle11g RAC时，OS层面配置好多路径软件后（multipath），下一步就需要配置udev或asmlib来处理共享分区（Lun），以便Orace ASM能够看到这些分区（Lun）。Linux6后，UDEV配置比asmlib配置起来相对简单。关于Multipath的配置说明，在上一文章(Linux MultiPath多路径软件实施说明 )已经进行了说明。接下来将要说明UDEV的配置操作。
udev 简介
简单的讲，通过UDEV配置，能够让Oracle对磁盘名进行持久化并改变磁盘访问权限为grid:asmadmin。这样在ASM的配置过程中能够看到磁盘。今后增加磁盘也不会改变原有的磁盘名称。
在存储行业中，用户总是有这样的要求：linux系统中原来有一块SCSI硬盘，系统分配的设备文件是/dev/sda。现在新增加了一个外置的磁盘阵列，通过SCSI卡连接。在接上这个磁盘阵列后，/dev/sda变成了磁盘阵列的硬盘了，原来内置的SCSI硬盘变成了 /dev/sdb，如何将设备文件名固定下来呢？
在linux kernel 2.6内核之前，我们只能和用户说，这个比较麻烦，因为/dev/sda等文件都是linux内核自动分配的。很难固定下来，除非你更改加载SCSI卡驱动程序的顺序，而且对于其他的即插即用设备，如USB设备等都不适用。
升级到linux kernel 2.6内核后，这个问题已经可以通过linux kernel 2.6内核新的sysfs文件系统和udev程序来解决了。udev是一种工具，它能够根据系统中的硬件设备的状态动态更新设备文件，包括完成对设备文件的创建、删除和刷新。
udev原理和结构
1. 当udev从kernel收到设备add/remove 事件后, udev 将解析:
§ /etc/udev/rules.d目录中的用户自定义规则文件
§ 使用自定义规则输出（可选的）
§ 从/sys中查询相应信息
2. 根据解析的信息，udev汇集了处理设备命名：
§ 确定将创建什么设备文件(device file)或符号链接（symlink)
§ 确定如何设置设备文件属性
§ 确定后续操作
配置udev
所有的udev配置文件被放置在/etc/udev目录下，所有的配置文件均采用文本格式方便用户配置修改，在udev读取过程中将忽略配置文件中以#号开头的行。
udev的全局控制配置文件是/etc/udev/udev.conf，它定义了udev的全局配置参数。/etc/udev/udev.conf文件配置项目前只有3个：
- udev_root ── 定义在那个目录中创建所需的设备文件，默认位置为/dev目录。
- udev_rules ── 定义从那个配置文件或配置目录中读取udev规则，默认位置为 /etc/udev/rules.d目录
- udev_log ── 定义在传递log到syslog系统日志时的优先级，在调试udev的时候也可以run-time的使用udevcontrol命令修改这个优先级。 默认优先级为err，其它可选优先级为info和debug。
udev规则
1. 文件名位置及命名方法
- 默认放置在/etc/udev/rules.d目录下
- 命名为.rules,例如
- 40multipath.rules
- 60net.rules
- bluetooth.rules
2. 规则结构
- value[,....]value[,....]
例如:
ACTION=="add",SUBSYSTEM=="mmc", UN+="modprobe mmc_block"
ACTION=="add", KERNEL=="sda", RUN+="/bin/raw /dev/raw/raw1 %N"
3. 规则文件将在第一次预读后缓冲
- 当规则文件被修改后时间戳的更新会强制udev从读规则文件，如果想强制重读可以使用touch命令更新时间戳。
udev通过读取规则文件来决定如何命名或重命名设备和执行附加操作。一般情况下，规则文件放置在/etc/udev/rules.d目录下以 .rules 结尾的方式命名。如果配置文件不止一个，udev将按其文件名的词汇顺序读取每个规则配置文件。通常，我们以两位的数字开头定义udev规则文件，比如:
40multipath.rules和60net.rules。40multipath.rules 规则文件会在60net.rules规则文件之前被读取，因为按照词汇顺序4在6之前。缺省的udev规则存放在/etc/udev/rules.d/50udev.rules文件中。它包含一些例子，和一些默认的规则以提供一种devfs风格的/dev层。通常，我们希望自定义的规则先于默认的规则被运行，所以我建议你新增。
关于/etc/udev/rules.d/下新增配置文件的具体内容，与多路径配置时形成磁盘分区名称、类型、uuid等有关联，在此引用Deploying Oracle RAC 11g R2 Database on Red Hat EnterpriseLinux 6 的相关介绍，最后以生产系统为例进行说明：
99-oracle-asmdevices.rules文件的配置过程
1. On the first node of the Oracle RAC cluster, as the root user, identify the Device
Mapper Universally Unique IDentifier (DM_UUID) for each device mapper volume. The
example below shows the DM_UID for the partitions of the volumes labeled
ocrvote1,ocrvote2,ocrvote3,db1,db2,fra, and redo.
# for i in ocrvote1p1 ocrvote2p1 ocrvote3p1 db1p1 db2p1 frap1 redop1; do
printf "%s %s\n" "$i" "$(udevadm info --query=all --name=/dev/mapper/$i |grep -i dm_uuid)"; done
其中ocrvote1p1 ocrvote2p1 ocrvote3p1 db1p1 db2p1 frap1 redop1为multipath形成的裸设备分区名。在OS下执行以上文件后，会生成如下内容（根据实际环境有所不同）：
ocrvote1p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe596a0f65101000000
ocrvote2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5a2a0f65101000000
ocrvote3p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe5b4a0f65101000000
db1p1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a854a0f65101000000
db2p1 E: DM_UUID=part1-mpath-3600c0ff000dabfe562a0f65101000000
frap1 E: DM_UUID=part1-mpath-3600c0ff000d7e7a874a0f65101000000
redop1 E: DM_UUID=part1-mpath-3600c0ff000dabfe585a0f65101000000
2. Create a file labeled 99-oracle-asmdevices.rules within /etc/udev/rules.d/
3. Within 99-oracle-asmdevices.rules file, create rules for each device similar to the
example below:
/etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="dm-*",ENV{DM_UUID}=="part1-mpath-
3600c0ff000dabfe5f4d8515101000000",OWNER="grid",GROUP="asmadmin",MODE="06
60"
To understand the rule above, it can be read as follows:
If any dm- device matches the DM_UUID of part1-mpath-
3600c0ff000dabfe5f4d8515101000000, assign to that dm- device to be owned by the
grid user and part of the asmadmin group with the permission mode set to 0660.
The 0660 value provides read and write permissions to the user grid and owner
asmadmin.
4. Save the file labeled 99-oracle-asmdevices.rules
5. Copy the 99-oracle-asmdevices.rules file to each node within the Oracle RAC
Database 11g Release 2 cluster using the scp command and enter the appropriate
password credentials for the other nodes. The example below shows how to copy the
file to node two of the Oracle RAC Database 11g Release 2 cluster.
# scp /etc/udev/rules.d/99-oracle-asmdevices.rules db-oraclenode2:/
etc/udev/rules.d/
Warning: Permanently added 'db-oracle-node2,10.16.142.52' (RSA) to the
list of known hosts.
[email protected]'s password:
99-oracleasmdevices.rules 100% 833 0.8KB/s 00:00
6. On each node within the Oracle RAC Database cluster, locate the dm- device for each
Oracle related partition. An example of how to find the dm- device for each partition is
to run the following command:
# for i in db1p1 db2p1 frap1 redop1 ocrvote1p1 ocrvote2p1 ocrvote3p1; do
printf "%s %s\n" "$i" "$(ls -ll /dev/mapper/$i)"; done
db1p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:21 /dev/mapper/db1p1 -> ../dm-14
db2p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/db2p1 -> ../dm-15
frap1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/frap1 -> ../dm-17
redop1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/redop1 -> ../dm-16
ocrvote1p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/ocrvote1p1-> ../dm-18
ocrvote2p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/ocrvote2p1-> ../dm-19
ocrvote3p1 lrwxrwxrwx. 1 root root 8 Aug 1 15:22 /dev/mapper/ocrvote3p1-> ../dm-20
7. On each node within the Oracle RAC Database cluster, apply and test the rules for
each dm- device created within the 99-oracle-asmdevices.rules by running a udevadm
test on each device. The example below demonstrates a udevadm test on dm-11.
# udevadm test /sys/block/dm-11
[ ... Output Abbreviated ... ]
udevadm_test: DM_NAME=db1p1
udevadm_test: DM_UUID=part1-mpath-3600c0ff000d7e7a86485ac5101000000
udevadm_test: DM_SUSPENDED=0
udevadm_test: DEVLINKS=/dev/mapper/db1p1 /dev/disk/by-id/dm-name-db1p1
/dev/disk/by-id/dm-uuid-part1-mpath-3600c0ff000d7e7a86485ac5101000000
/dev/block/253:11
udevadm_test: ID_FS_TYPE=oracleasm
8. Confirm each device has the desired permissions on each node within the cluster.
# ls -lh /dev/dm-*
brw-rw----. 1 grid asmadmin 253, 14 Aug 1 16:02 /dev/dm-14
brw-rw----. 1 grid asmadmin 253, 15 Aug 1 16:02 /dev/dm-15
brw-rw----. 1 grid asmadmin 253, 16 Aug 1 16:02 /dev/dm-16
brw-rw----. 1 grid asmadmin 253, 17 Aug 1 16:02 /dev/dm-17
brw-rw----. 1 grid asmadmin 253, 18 Aug 1 16:02 /dev/dm-18
brw-rw----. 1 grid asmadmin 253, 19 Aug 1 16:03 /dev/dm-19
brw-rw----. 1 grid asmadmin 253, 20 Aug 1 16:02 /dev/dm-20
NOTE: If the desired permissions are not visibile, please reboot the particular node
from the Oracle RAC Database cluster.
理解以上内容的基础上，将99-oracle-asmdevices.rules文件内容的配置可以简化为如下：
在第一节点通过fdisk格式化磁盘，做成分区形成：
fdisk /dev/mapper/mpathb
fdisk /dev/mapper/mpathc
fdisk /dev/mapper/mpathd
fdisk /dev/mapper/mpathe
.....
格式化fdisk /dev/mapper/mpathb，fdisk /dev/mapper/mpathc，fdisk /dev/mapper/mpathd，fdisk /dev/mapper/mpathe
等设备，形成/dev/mapper/mpathcp1，fdisk /dev/mapper/mpathdp1，fdisk /dev/mapper/mpathep1 ...
作为专业一点儿的DBA，我们也可以在multipath配置中设置alias，如下：
multipath {
wwid 3600c0ff000d7e7a854a0f65101000000
alias data01p1
}
二节点扫描设备：
# partprobe
编写UDEV文件（双节点都要配置）
编写UDEV权限文件：vi /etc/udev/rules.d/99-oracle-asmdevices.rules
PROGRAM="/bin/chown grid:oinstall /dev/mapper/orcvotep1"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/data01p1"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/data02p1"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/data03p1"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/data04p1"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/data05p1"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra01"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra02"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra03"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra04"
PROGRAM="/bin/chown grid:oinstall /dev/mapper/fra05"
PROGRAM="/bin/chmod 0660 /dev/mapper/orcvotep1"
PROGRAM="/bin/chmod 0660 /dev/mapper/data01p1"
PROGRAM="/bin/chmod 0660 /dev/mapper/data02p1"
PROGRAM="/bin/chmod 0660 /dev/mapper/data03p1"
PROGRAM="/bin/chmod 0660 /dev/mapper/data04p1"
PROGRAM="/bin/chmod 0660 /dev/mapper/data05p1"
PROGRAM="/bin/chmod 0660 /dev/mapper/fra01"
PROGRAM="/bin/chmod 0660 /dev/mapper/fra02"
PROGRAM="/bin/chmod 0660 /dev/mapper/fra03"
PROGRAM="/bin/chmod 0660 /dev/mapper/fra04"
PROGRAM="/bin/chmod 0660 /dev/mapper/fra05"
双节点重新启动UDEV：
#start_udev
Oracle ASM on RHEL 6 with udev and multipath
As the root user, install the device-mapper-multipath package using the yum package manager
[[email protected] ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.4 (Santiago)
[[email protected] ~]# uname -p
x86_64
[[email protected] ~]# yum install device-mapper-multipath
Copy the multipath.conf file found within /usr/share/doc/device-mapper-multipath-0.4.9/ to /etc/
[[email protected] ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf
Capture the scsi id of the local disks on the system.
[[email protected] ~]# for i in b c d e f g h i j
> do
> scsi_id –whitelisted –replace-whitespace –device=/dev/sd$i
> done
1ATA_VBOX_HARDDISK_VBda7290c1-228b4bfb
1ATA_VBOX_HARDDISK_VBcb76adf7-43d27c31
1ATA_VBOX_HARDDISK_VB2077c81c-7cfe5922
1ATA_VBOX_HARDDISK_VB7282bfc8-69601726
1ATA_VBOX_HARDDISK_VBa0072176-9e9ab0a0
1ATA_VBOX_HARDDISK_VB2a7b18e2-04053806
1ATA_VBOX_HARDDISK_VB40939671-89967d5e
1ATA_VBOX_HARDDISK_VB664cb850-2065525c
1ATA_VBOX_HARDDISK_VBa498ea69-53cdb110
[[email protected] ~]#
Enter the above values into /etc/multipath and give friendly aliases to uuid
[[email protected] ~]# cat /etc/multipath.conf
blacklist {
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
wwid 1ATA_VBOX_HARDDISK_VB82d3887f-a892e124 #exclude sda wwid
}
defaults {
user_friendly_names no
getuid_callout "/sbin/scsi_id –whitelisted –replace-whitespace –device=/dev/%n"
}
multipaths {
multipath {
wwid 1ATA_VBOX_HARDDISK_VBda7290c1-228b4bfb
alias ocrvotedisk1
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VBcb76adf7-43d27c31
alias ocrvotedisk2
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VB2077c81c-7cfe5922
alias ocrvotedisk3
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VB7282bfc8-69601726
alias ocrvotedisk4
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VBa0072176-9e9ab0a0
alias ocrvotedisk5
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VB2a7b18e2-04053806
alias data1
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VB40939671-89967d5e
alias data2
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VB664cb850-2065525c
alias fra1
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
multipath {
wwid 1ATA_VBOX_HARDDISK_VBa498ea69-53cdb110
alias fra2
path_grouping_policy multibus
path_selector "round-robin 0"
failback immediate
rr_weight priorities
no_path_retry 5
}
}
Restart multipathd service
[[email protected] ~]# service multipathd restart
ok
Stopping multipathd daemon: [ OK ]
Starting multipathd daemon: [ OK ]
[[email protected] ~]# chkconfig multipathd on
Check if multipath is setup properly
[[email protected] ~]# multipath -ll
fra2 (1ATA_VBOX_HARDDISK_VBa498ea69-53cdb110) dm-8 ATA,VBOX HARDDISK
size=12G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 11:0:0:0 sdj 8:144 active ready running
ocrvotedisk5 (1ATA_VBOX_HARDDISK_VBa0072176-9e9ab0a0) dm-4 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 7:0:0:0 sdf 8:80 active ready running
fra1 (1ATA_VBOX_HARDDISK_VB664cb850-2065525c) dm-7 ATA,VBOX HARDDISK
size=12G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 10:0:0:0 sdi 8:128 active ready running
ocrvotedisk4 (1ATA_VBOX_HARDDISK_VB7282bfc8-69601726) dm-3 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 6:0:0:0 sde 8:64 active ready running
ocrvotedisk3 (1ATA_VBOX_HARDDISK_VB2077c81c-7cfe5922) dm-2 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 5:0:0:0 sdd 8:48 active ready running
ocrvotedisk2 (1ATA_VBOX_HARDDISK_VBcb76adf7-43d27c31) dm-1 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 4:0:0:0 sdc 8:32 active ready running
ocrvotedisk1 (1ATA_VBOX_HARDDISK_VBda7290c1-228b4bfb) dm-0 ATA,VBOX HARDDISK
size=2.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 3:0:0:0 sdb 8:16 active ready running
data2 (1ATA_VBOX_HARDDISK_VB40939671-89967d5e) dm-6 ATA,VBOX HARDDISK
size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 9:0:0:0 sdh 8:112 active ready running
data1 (1ATA_VBOX_HARDDISK_VB2a7b18e2-04053806) dm-5 ATA,VBOX HARDDISK
size=20G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 8:0:0:0 sdg 8:96 active ready running
[[email protected] ~]# ls -l /dev/mapper/
total 0
crw-rw—- 1 root root 10, 236 Sep 24 13:56 control
lrwxrwxrwx 1 root root 7 Sep 24 13:57 data1 -> ../dm-5
lrwxrwxrwx 1 root root 7 Sep 24 13:57 data2 -> ../dm-6
lrwxrwxrwx 1 root root 7 Sep 24 13:56 fra1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Sep 24 13:56 fra2 -> ../dm-8
lrwxrwxrwx 1 root root 7 Sep 24 13:57 ocrvotedisk1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Sep 24 13:56 ocrvotedisk2 -> ../dm-1
lrwxrwxrwx 1 root root 7 Sep 24 13:56 ocrvotedisk3 -> ../dm-2
lrwxrwxrwx 1 root root 7 Sep 24 13:56 ocrvotedisk4 -> ../dm-3
lrwxrwxrwx 1 root root 7 Sep 24 13:56 ocrvotedisk5 -> ../dm-4
[[email protected] ~]# for i in ocrvotedisk1 ocrvotedisk2 ocrvotedisk3 ocrvotedisk4 ocrvotedisk5 data1 data2 fra1 fra2
> do
> printf "%s %s\n" "$i" "$(ls -ll /dev/mapper/$i)"
> done
ocrvotedisk1 lrwxrwxrwx 1 root root 7 Sep 24 13:57 /dev/mapper/ocrvotedisk1 -> ../dm-0
ocrvotedisk2 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk2 -> ../dm-1
ocrvotedisk3 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk3 -> ../dm-2
ocrvotedisk4 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk4 -> ../dm-3
ocrvotedisk5 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/ocrvotedisk5 -> ../dm-4
data1 lrwxrwxrwx 1 root root 7 Sep 24 13:57 /dev/mapper/data1 -> ../dm-5
data2 lrwxrwxrwx 1 root root 7 Sep 24 13:57 /dev/mapper/data2 -> ../dm-6
fra1 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/fra1 -> ../dm-7
fra2 lrwxrwxrwx 1 root root 7 Sep 24 13:56 /dev/mapper/fra2 -> ../dm-8
Determine partitioned alias for target device
[[email protected] ~]# dmsetup ls|sort
data1 (252:5)
data2 (252:6)
fra1 (252:7)
fra2 (252:8)
ocrvotedisk1 (252:0)
ocrvotedisk2 (252:1)
ocrvotedisk3 (252:2)
ocrvotedisk4 (252:3)
ocrvotedisk5 (252:4)
Edit /etc/udev/rules.d/12-dm-permissions.rules file to set the above device ownership as grid:asmadmin
[[email protected] ~]# cat /etc/udev/rules.d/12-dm-permissions.rules
ENV{DM_NAME}=="ocrvotedisk1", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk2", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk3", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk4", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="ocrvotedisk5", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="data1", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="data2", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="fra1", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
ENV{DM_NAME}=="fra2", OWNER:="grid", GROUP:="asmadmin", MODE:="660", SYMLINK+="iscsi/oraasm-$env{DM_NAME}"
Restart multipathd service
[[email protected] ~]# /etc/init.d/multipathd restart
ok
Stopping multipathd daemon: [ OK ]
Starting multipathd daemon: [ OK ]
Check if permissions are correct
[[email protected] ~]# ls -l /dev/dm-*
brw-rw—- 1 grid asmadmin 252, 0 Sep 24 16:35 /dev/dm-0
brw-rw—- 1 grid asmadmin 252, 1 Sep 24 16:35 /dev/dm-1
brw-rw—- 1 grid asmadmin 252, 2 Sep 24 16:35 /dev/dm-2
brw-rw—- 1 grid asmadmin 252, 3 Sep 24 16:35 /dev/dm-3
brw-rw—- 1 grid asmadmin 252, 4 Sep 24 16:35 /dev/dm-4
brw-rw—- 1 grid asmadmin 252, 5 Sep 24 16:35 /dev/dm-5
brw-rw—- 1 grid asmadmin 252, 6 Sep 24 16:35 /dev/dm-6
brw-rw—- 1 grid asmadmin 252, 7 Sep 24 16:35 /dev/dm-7
brw-rw—- 1 grid asmadmin 252, 8 Sep 24 16:35 /dev/dm-8
Check if oracle ASM symbolic links exists for multipath devices
[[email protected] ~]# ls -l /dev/iscsi/*
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-data1 -> ../dm-5
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-data2 -> ../dm-6
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-fra1 -> ../dm-7
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-fra2 -> ../dm-8
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk1 -> ../dm-0
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk2 -> ../dm-1
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk3 -> ../dm-2
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk4 -> ../dm-3
lrwxrwxrwx 1 root root 7 Sep 24 16:38 /dev/iscsi/oraasm-ocrvotedisk5 -> ../dm-4
Configure ASM disk change discovery path
[[email protected] ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
– —– —————– ——— ———
1. ONLINE 1374f266ca874f15bfde7ca23d073e5f (/dev/mapper/ocrvotedisk1) [CRSDG]
2. ONLINE 4145a926ff0d4f6dbf63e2c3695c580e (/dev/mapper/ocrvotedisk2) [CRSDG]
3. ONLINE 6f3ab4bbe0c14f97bfb20c91820d898b (/dev/mapper/ocrvotedisk3) [CRSDG]
4. ONLINE 8c973dd3f1da4f0dbf2080e04fcf9bd7 (/dev/mapper/ocrvotedisk4) [CRSDG]
5. ONLINE 1bed4a3c6f624fabbfbe2564e81df250 (/dev/mapper/ocrvotedisk5) [CRSDG]
Located 5 voting disk(s).
[[email protected] ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.4.0 Production on Wed Sep 24 17:17:22 2014
Copyright (c) 1982, 2013, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> select * from v$active_instances;
INST_NUMBER INST_NAME
———– ——————–
1 vzwc1:+ASM1
2 vzwc2:+ASM2
SQL> select path from v$asm_disk;
PATH
——————————————————————————–
/dev/mapper/data2
/dev/mapper/fra2
/dev/mapper/fra1
/dev/mapper/data1
/dev/mapper/ocrvotedisk2
/dev/mapper/ocrvotedisk3
/dev/mapper/ocrvotedisk1
/dev/mapper/ocrvotedisk4
/dev/mapper/ocrvotedisk5
9 rows selected.
Oracle ASM on RHEL 5 with multipath
As the root user, install the device-mapper-multipath package using the yum package manager
[[email protected] ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.9 (Tikanga)
[[email protected] ~]# yum install device-mapper*
Loaded plugins: rhnplugin, security
This system is not registered with ULN.
You can use up2date –register to register.
ULN support will be disabled.
Setting up Install Process
Package device-mapper-multipath-0.4.9-56.0.3.el5.x86_64 already installed and latest version
Package device-mapper-multipath-libs-0.4.9-56.0.3.el5.x86_64 already installed and latest version
Package device-mapper-event-1.02.67-2.el5.x86_64 already installed and latest version
Package device-mapper-1.02.67-2.el5.x86_64 already installed and latest version
Package device-mapper-1.02.67-2.el5.i386 already installed and latest version
Nothing to do
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]#
[[email protected] ~]# rpm -qa|grep device-mapper
device-mapper-multipath-libs-0.4.9-56.0.3.el5
device-mapper-event-1.02.67-2.el5
device-mapper-1.02.67-2.el5
device-mapper-1.02.67-2.el5
device-mapper-multipath-0.4.9-56.0.3.el5
Capture the scsi id of the local disks on the system
[[email protected] ~]# /sbin/scsi_id -g -u -s /block/sdb
SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_
[[email protected] ~]# /sbin/scsi_id -g -u -s /block/sdc
SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_
Get to the grid user's uid and gid, here the gid is asmadmin group
[[email protected] ~]# cat /etc/passwd|grep grid
grid:x:1100:1000:Grid Infrastructure Owner:/home/grid:/bin/bash
[[email protected] ~]# cat /etc/group|grep asmadmin
asmadmin:x:1200:grid,oracle
[[email protected] ~]# id -a grid
uid=1100(grid) gid=1000(oinstall) groups=1000(oinstall),1200(asmadmin),1201(asmdba),1202(asmoper),1300(dba)
Enter the above values into /etc/multipath.conf
[[email protected] ~]# cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/
[[email protected] ~]# cat /etc/multipath.conf
defaults {
user_friendly_names no
queue_without_daemon no
flush_on_last_del yes
max_fds max
}
blacklist {
devnode "^hd[a-z]"
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^cciss.*"
}
devices {
device {
vendor "OPNFILER "
product "LUN"
path_grouping_policy group_by_prio
features "3 queue_if_no_path pg_init_retries 50"
getuid_callout "/sbin/scsi_id -g -u -s /block/%n"
path_checker tur
path_selector "round-robin 0"
hardware_handler "1 alua"
failback immediate
rr_weight uniform
rr_min_io 128
}
}
multipaths {
multipath {
wwid SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_
alias datadisk
uid 1100
gid 1200
}
multipath {
wwid SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_
alias fradisk
uid 1100
gid 1200
}
}
Start multipathd service
[[email protected] ~]# modprobe dm-multipath
[[email protected] ~]# modprobe dm-round-robin
[[email protected] ~]# chkconfig multipathd start
[[email protected] ~]# chkconfig multipathd on
[[email protected] ~]# /etc/init.d/multipathd start
Starting multipathd daemon: [ OK ]
[[email protected] ~]# ps -ef|grep multipathd
root 3084 1 0 13:16 ? 00:00:00 /sbin/multipathd
root 3186 2877 0 13:16 pts/0 00:00:00 grep multipathd
[[email protected] ~]# multipath -F
[[email protected] ~]# multipath -v2
Jan 15 13:17:40 | SATA_VBOX_HARDDISK_VBa64c2685-cc2f3dd0_: ignoring map
create: datadisk (SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_) undef ATA,VBOX HARDDISK
size=16G features='0' hwhandler='0' wp=undef
`-+- policy='round-robin 0' prio=1 status=undef
`- 1:0:0:0 sdb 8:16 undef ready running
create: fradisk (SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_) undef ATA,VBOX HARDDISK
size=10G features='0' hwhandler='0' wp=undef
`-+- policy='round-robin 0' prio=1 status=undef
`- 2:0:0:0 sdc 8:32 undef ready running
[[email protected] ~]#
[[email protected] ~]# multipath -ll
fradisk (SATA_VBOX_HARDDISK_VB3edd7dc1-fd831bf4_) dm-3 ATA,VBOX HARDDISK
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 2:0:0:0 sdc 8:32 active ready running
datadisk (SATA_VBOX_HARDDISK_VB3c0bb909-10aab3a0_) dm-2 ATA,VBOX HARDDISK
size=16G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
`- 1:0:0:0 sdb 8:16 active ready running
[[email protected] ~]# ll /dev/mapper/*disk
brw-rw—- 1 grid asmadmin 252, 2 Jan 15 13:17 /dev/mapper/datadisk
brw-rw—- 1 grid asmadmin 252, 3 Jan 15 13:17 /dev/mapper/fradisk
About Me
...............................................................................................................................
● 本文整理自网络
● 本文在itpub（http://blog.itpub.net/26736162）、博客园（http://www.cnblogs.com/lhrbest）和个人微信公众号（xiaomaimiaolhr）上有同步更新
● 本文itpub地址：http://blog.itpub.net/26736162/abstract/1/
● 本文博客园地址：http://www.cnblogs.com/lhrbest
● 本文pdf版及小麦苗云盘地址：http://blog.itpub.net/26736162/viewspace-1624453/
● 数据库笔试面试题库及解答：http://blog.itpub.net/26736162/viewspace-2134706/
● QQ群：230161599 微信群：私聊
● 联系我请加QQ好友(646634621)，注明添加缘由
● 于 2017-07-01 09:00 ~ 2017-07-31 22:00 在魔都完成
● 文章内容来源于小麦苗的学习笔记，部分整理自网络，若有侵权或不当之处还请谅解
● 版权所有，欢迎分享本文，转载请保留出处
...............................................................................................................................
拿起手机使用微信客户端扫描下边的左边图片来关注小麦苗的微信公众号：xiaomaimiaolhr，扫描右边的二维码加入小麦苗的QQ群，学习最实用的数据库技术。
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/26736162/viewspace-2135271/，如需转载，请注明出处，否则将追究法律责任。