Adding more than 512 LUNs to Oracle Enterprise Linux

When dealing with large databases it might be neccessary to have more than 512 LUNs attached to the server. Most Linux distributions default to 512 luns max. This gets even worse with multipathing where each path is represented as a single lun.

Error messages like this indicate the limit was hit:

Dec 02 22:24:46 server kernel: scsi: host 2 channel 0 id 2 lun258 has a LUN larger than allowed by the host adapter
 Dec 02 22:24:46 server kernel: scsi: host 2 channel 0 id 2 lun259 has a LUN larger than allowed by the host adapter

So what is needed to change that limit?

First you need to modify the maximum numbers of LUNs allowed by the SCSI stack by adding the following line to /etc/modprobe.conf:

options scsi_mod max_luns=2048

Depending on the SCSI HBA used the driver module itself might need some adjustments. For QLOgic HBAs the following line ist required in /etc/modprobe.conf:

options lpfc lpfc_max_luns=2048

After rebooting or removing and re-inserting the module you should be able to attach 2048 LUNs.

By the way the linux device interface (including the multipath driver) supprts more than 254 LUNs sind Kernel 2.6:

[root@ora1 ~]# vgs 
 VG     #PV #LV #SN Attr   VSize   VFree 
 vgroot   1 388   0 wz--n- 179,88G 94,50G

As you can see the VG has 388 LVs. More than 254 … so we actually should run out of minor numbers in /dev, right? Actually major- and minor numbers were increaed to 10 bits in kernel 2.6:

[root@ora1 mapper]# ls -la /dev/mapper/vgroot-disk00*
brw-rw---- 1 root disk 253,  6 24. Dez 05:27 /dev/mapper/vgroot-disk001
brw-rw---- 1 root disk 253,  7 24. Dez 05:27 /dev/mapper/vgroot-disk002
brw-rw---- 1 root disk 253,  8 24. Dez 05:27 /dev/mapper/vgroot-disk003

brw-rw---- 1 root disk 253, 308 24. Dez 05:27 /dev/mapper/vgroot-disk306
brw-rw---- 1 root disk 253, 309 24. Dez 05:27 /dev/mapper/vgroot-disk307
brw-rw---- 1 root disk 253, 310 24. Dez 05:27 /dev/mapper/vgroot-disk308
brw-rw---- 1 root disk 253, 311 24. Dez 05:27 /dev/mapper/vgroot-disk309