Ronny Egners Blog » Oracle ASM

ASM resilvering – or – how to recover your crashed cluster

Ronny Egner — Tue, 27 Oct 2009 06:00:20 +0000

In this and the following posts i will perform some crash and recover scenarios and show how to recover the cluster successfully.

At the moment the following tests are planned and will be published during the next days:

suddenly turning off the power and restarting the node
terminating private network connect between the cluster nodes
recovering an ACFS file system which will not mount automatically
(was not able to reproduce more than once; sorry guys)
overwriting the ASM disk header with the disk group being offline
corrupting an online and active ASM disk by writing chunks of random data to the disk randomly
simulating disk errors by removing the device from the operating system
corrupting the OCR
corrupting the Voting Disk

The environment used for the posts are explained in detail here.

Useful scripts can be found here.

ASM resilvering – or – how to recover your crashed cluster – Test no 4

Ronny Egner — Tue, 13 Oct 2009 11:50:35 +0000

Test #4: Corrupting the ASM disk with ASM disk group being online and active

After overwriting the ASM disk header while the disk group was offline we will now put some load on the full running cluster and corrupt the asm disk slightly.

Testcase

Put load on the database

Prior starting this test i created a larger table with one column and approx 1.2 GB size and an empty table with the same structure. So for our simple test we will just copy one table into another table. This forces oracle to read the database blocks and do some writes as well.

create table test2_empty as select * from test where rownum<1;

insert into test2_empty select * from test;

Corrupting the disk

According to fdisk the asm lun has a size of 21474836480 bytes:

Disk /dev/sde: 21.4 GB, 21474836480 bytes
64 heads, 32 sectors/track, 20480 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes

 Device Boot      Start         End      Blocks     Id  System
/dev/sde1               1       20480    20971504   83  Linux

And the disk is indeed the mirror partner of the disk we destroyed and re-added in the earlier part.

[root@rac1 ~]# oracleasm querydisk /dev/sde1
Device "/dev/sde1" is marked an ASM disk with the label "DISK003B"

We will corrupt the disk by writing 512 byte chunks of random data from /dev/urandom to 1000 different locations all over the lun. Given the lun size this makes 41943040 blocks of 512 bytes each.

Our command to corrupt the lun will be:

dd if=/dev/random bs=512 count=1 of=/dev/sde1 seek=nnnn

Where “seek=nnn” is a number from 0 to 41943040-1.

Note that this time we manipulate the second mirror disk of disk group DATA2 – DISK003B. In test no 3 we overwrote the asm disk header of DISK003A and re-added the disk to the data group. In this example we manipulate DISK003B. If there were undetected errors rebalancing the disks we will discover it now.

For this we first of all needed a small little random number generator within the limits of 0 to 41943040; here it is:

#!/bin/bash

function random()
{
 od -d /dev/urandom | sed -e 's/^[0-9]* //' -e 's/ //g' |\
 while read L ; do echo -n $L ; done |\
 dd bs=1 count=${1:-10} 2>/dev/null
}

A=$(random 10) #return random number with 10 numbers
B=$(expr $C % 41943040)  # keep number within range
echo $B

With this random number generator we created a small shell script for corrupting the asm lun which looks like this:

dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=986024
dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=1594423
dd if=/dev/urandom of=/dev/sde1 bs=512 count=1 seek=2236024
[..]

Expected test results

So what are we expecting?

According to oracle documents (note 416046.1) there are two different scenarios:

If the primary extent is detected to be corrupt (asm only reads from primary extent unless a read preference set) ASM retries the read. If re-read data is also corrupt ASM fails over to the secondary extent. If data in secondary extent is valid data in primary extent will be overwritten with the good data from the secondary extent.
If data in secondary extent is corrupt it will remain undetected. If new data is written the new data will be written both to the primary and secondary disk. This will most likely overwrite the corrupted data. But if the primary extent fails oracle will read the secondary extent and will discover the corruption as well - but this time the corruption is not fixable. You will end up with restoring/recovering the corrupt database blocks and probably much more.

In addition to that i expect ASM and/or the database to automatically fix detected corruptions.

Testing

Message no. 1: Database detected and fixed corruption

During our Insert statement running the database alert.log showed:

Fri Oct 02 10:47:21 2009
Hex dump of (file 6, block 156029) in trace file /u01/app/oracle/diag/rdbms
  /ora11p/ora11p1/trace/ora11p1_ora_30023.trc
Corrupt block relative dba: 0x0182617d (file 6, block 156029)
Bad header found during multiblock buffer read
Data in bad block:
 type: 180 format: 3 rdba: 0xb19a95b2
 last change scn: 0x6f92.6c357e5d seq: 0x1 flg: 0x5a
 spare1: 0xfd spare2: 0x80 spare3: 0x4f3c
 consistency value in tail: 0x1591148c
 check value in block header: 0x2abf
 block checksum disabled
Reading datafile '+DATA2/ora11p/users02.dbf' for corruption at rdba:
  0x0182617d (file 6, block 156029)
Read datafile mirror 'DISK003B' (file 6, block 156029) found same corrupt data
Read datafile mirror 'DISK003A' (file 6, block 156029) found valid data
Repaired corruption at (file 6, block 156029)

Message no. 2: Database detected corruption

Corrupt block relative dba: 0x01c12d35 (file 7, block 77109)
Bad check value found during validation
Data in bad block:
 type: 2 format: 2 rdba: 0x01c12d35
 last change scn: 0x0000.00253bd5 seq: 0x73 flg: 0x04
 spare1: 0x0 spare2: 0x0 spare3: 0x0
 consistency value in tail: 0x3bd50273
 check value in block header: 0x94cb
 computed block checksum: 0xf0eb
Trying mirror side DISK003B.
Reread of blocknum=77109, file=+DATA2/ora11p/datafile/undotbs01_02.dbf. found
 same corrupt data
Reread of blocknum=77109, file=+DATA2/ora11p/datafile/undotbs01_02.dbf. found
 valid data

Note: This time the database does not say anything about the corruption being fixed.

Message no. 3: ASM detected and fixed corruption

Mon Oct 05 09:32:15 2009
WARNNING: cache read a corrupted block group=DATA2 fn=1
 blk=893 from disk 0
NOTE: a corrupted block from group DATA2 was dumped to /u01/app/oracle/diag/asm
 /+asm/+ASM1/trace/+ASM1_ora_4739.trc
WARNNING: cache read(retry) a corrupted block group=DATA2 fn=1
 blk=893 from disk 0
NOTE: cache repaired a corrupted block group=DATA2 fn=1 blk=893
 from disk 0

Conclusion

First of all: regardless of the caveats described below the insert statement most of the time completed successfully:

SQL> insert into test2 select * from sys.test;
134217728 rows created.

Second: During our tests ASM performed well and stable. We were consistently able to work on the disk groups even if the disk group missed their mirrors.

However i discovered some problems as well:

ASM has no ability to check data on disk groups completely (i.e. compare data on all mirror partners and check for corruptions); so corruptions on secondary extents will remain undetected until primary extent fails and secondary extent is read
ASM nor the database does always fix corruptions found in primary extents (see message 2 above… corruption was detected but NOT corrected and remained on disk) as indicated by note 416046
Checking database with RMAN and “backup validate database” finds and reports errors but does not fix ANY error found. The error message no. 2 originated from a “select * from test….” but rman produces this type of error messages in alert.log as well. From the Metalink document which describes handling of block corruptions in ASM i would have expected ASM or the database to fix these errors automatically.
There are scenarios where corrupted primary blocks prevent the database from starting at all. During our tests we have seen this two times with different error messages. One sample error message stack is reproduced in error message no. 3 coming from the asm instance and in the following lines below coming from the database instance:

Exception [type: SIGILL, Illegal operand] [ADDR:0x105DC01]
[PC:0x105DC01, kkestRCSBase()+221] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
trace/ora11p1_ora_5206.trc  (incident=48254):
ORA-07445: exception encountered: core dump [kkestRCSBase()+221]
[SIGILL] [ADDR:0x105DC01] [PC:0x105DC01] [Illegal operand] []
Incident details in: /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
incident/incdir_48254/ora11p1_ora_5206_i48254.trc
Mon Oct 05 09:33:47 2009
Trace dumping is performing id=[cdmp_20091005093347]
Mon Oct 05 09:33:49 2009
Exception [type: SIGILL, Illegal operand] [ADDR:0x105DC01]
[PC:0x105DC01, kkestRCSBase()+221] [flags: 0x0, count: 1]
Errors in file /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
trace/ora11p1_ora_5296.trc  (incident=48310):
ORA-07445: exception encountered: core dump [kkestRCSBase()+221]
[SIGILL] [ADDR:0x105DC01] [PC:0x105DC01] [Illegal operand] []
Incident details in: /u01/app/oracle/diag/rdbms/ora11p/ora11p1/
incident/incdir_48310/ora11p1_ora_5296_i48310.trc

Although we met this errors we were able to fix them: In one case dropping the disk containing the corrupted blocks was enough to start the instance again. The second incident reproduced above required the corrupted disk to be dropped, re-added and ASM instance to be restarted before we were able to successfully start the database instance.

Dropping a corrupted disk and re-adding the disk requires the disk group the disk belonged to be taken offline. Otherwise labeling the former member disk failed with “device or resource busy”. I will investigate this further.

Summarizing my experiences until now i would go with ASM and normal or even high redundancy. This enables far more options for fixing errors without having to rebuild the disk group and restoring everything from tape/disk.

Oracle 11 Release 2 Install Guide – ACFS and ADVM

Ronny Egner — Wed, 23 Sep 2009 08:16:22 +0000

ACFS and ADVM

System configuration

We will use the system configured and installed in part 2

General information

ADVM = ASM dynamic volume manager
ACFS = ASM cluster file system
Basic layout

(Source: Oracle Database Storage Administrator’s Guide 11g Release 2)

ADVM – Advantages

Integrated in ASM; this can be an disadvantage as well :-)
Inhertits storage from ASM hence enables host-based mirroring (either 2- or 3-way-mirroring)
multiple volumes within a disk group can be created with an file system such as ext3, ext4, reiserfs, … on top of it and will support storage of any file type as the file system normally woud – EXCEPT files which can be place in ASM directly
ADVM volume dynamically resizeable

ADVM – Disadvantages

ADVM volumes may be resized online; but the used file system must support it as well (ext3 on OEL 5 does support online resizing but does not support online shrinking)
Storing files which can be stored in ASM directly in ADVM + file system is not supported
NFS on top of ADVM is also not supported
ASM configuration assistant (asmca) only supports creation of volumes / file system… delete a volume / file system requires command line

ACFS – Advantages

cluster file system on top of ASM and ADVM
as available as ASM is (inherits storage from ASM disk group and ADVM volume)
Supports storage of files which cannot be directly stored in ASM, i.e.
- executables
- trace files
- log files
- …
- Supports even oracle database binary installations
On ACFS read-only Snapshots can be created
dynamically resizeableUseable accross plattforms
Thoughts
- Do i need licenses for grid infrastructrue?
- If not: What if grid infrastructure + ASM used to provide host-based mirroring and cluster file system for non-oracle applications, for instance web servers ACFS Mount registry: used for mouting ACFS and ADVM file system across reboots

ACFS – Disadvantages

for example storing database files in ACFS is not supported, according to the documentation
„Oracle Support Services will not take calls and development will not fix bugs associated with storing unsupported file types in Oracle ACFS“
Only available with RedHat Server 5 or Oracle Enterprise Linux 5 !
Disk group compatible parameter COPATBILE.ASM and COMPATIBLE.ADVM must be set so 11.2
ASM configuration assistant (asmca) only supports creation of volumes / file system… delete a volume / file system requires command line

First steps with ADVM and ACFS

Create a disk group for use with ADVM and ACFS

Lets first create an additional disk group called „DATA2“ which consists for two iSCSI LUNs with 30 GB each

Preparation:

LUNs visible with „fdisk -l“
Partition created (one on each LUN)
disk labeled with „oracleasm createdisk “
Create disk group in ASM (remember to connect as „sys as sysASM“!)

Notes on disk groups
- AUSIZE of 4 MB recommended by Oracle documentation due to:
  - Increased I/O through the I/O subsystem if the I/O size is increased to the AU size.
  - Reduced SGA size to manage the extent maps in the database instance.
  - Faster datafile initialization if the I/O size is increased to the AU size.
  - Increased file size limits.
  - Reduced database open time.
- Large AUSIZE requires as well
  - Increasing size of maximum IO request to at least 4 MB in operating system, drive, HBA, storage system
  - Larger stripe size in storage system (pre 11g R2: 1 MB stripe size, with 11g R2: 4 MB? → to be tested)

Read the documentation on COMPATIBLE parameters;most „cool“ features are only available with 11.2 COMPATIBLE parameter hence require 11g R2 database

Creating an ADVM and afterwards an ACFS

Create an ADVM

ACFS requires ADVM in which ACFS can be created
volcreate creates ADVM volume

The command above shows minimal command creating an ADVM volume; redundancy is derived from disk group, our data group was created with „normal“ redundancy so the volume inherits this as well)
Creation with SQL also possible: „ALTER DISKGROUP data2 ADD VOLUME volume1 SIZE 10G;“

Create ACFS on top of ADVM

Requires ADVM in which ACFS can be created
volinfo shows detailed information
Especially device path is important for creating the file system

create ACFS from operating system (only on one node)

register acfs in registry to be mounted across reboots with „acfsutil“
ATTENTION: DO NOT register shared oracle home directories with acfsutil; this will be done later by the clusterware itself!
test of everything works by issueing „mount.acfs -o all“ on all nodes; the file system should be mounted and accessible

Simple Performance tests

dd if=/dev/zero bs=1024k of= count=1000 oflag=direct

→ direct I/O used; no caching, performed 10 times

write to ACFS
- ACFS 2-way mirror: ~ 6 MB/s average
- ACFS unprotected: ~ 12 MB/s averga
- → expected… one disk, double I/O halfed throughput
direct write to iSCSI LUN: ~ 14.3 MB/s average

Attention: Tests were performed within a VMWare… so results are most likely not accurate… but we get an impression.. we will check this on real hardware later!

Building and using the kfed utility

Ronny Egner — Mon, 21 Sep 2009 13:52:41 +0000

When using ASM sometimes it it extremely helpful to get more information on the asm disk header. The “kfed” utility from oracle enables to dump the asm disk header and many more.

With this tool corruptions to the asm can be easily checked (and repaired).

Building kfed

This method works from 10g onwards to and including 11g R2:

-bash-3.2$ cd $ORACLE_HOME/rdbms/lib
-bash-3.2$ make -f ins_rdbms.mk ikfed

Linking KFED utility (kfed)
rm -f /u01/app/oracle/product/ora11r2p/rdbms/lib/kfed
gcc -o /u01/app/oracle/product/ora11r2p/rdbms/lib/kfed -m64 -L/u01/app/oracle/product/ora11r2p/rdbms/lib/
-L/u01/app/oracle/product/ora11r2p/lib/ -L/u01/app/oracle/product/ora11r2p/lib/stubs/ 
/u01/app/oracle/product/ora11r2p/lib/s0main.o /u01/app/oracle/product/ora11r2p/rdbms/lib/sskfeded.o

/u01/app/oracle/product/ora11r2p/rdbms/lib/skfedpt.o -ldbtools11 -lasmclnt11 -lcommon11 -lcell11 -lskgxp11
-lhasgen11 -lskgxn2 -lnnz11 -lzt11 -lxml11 -locr11 -locrb11 -locrutl11 -lhasgen11 -lskgxn2 -lnnz11 -lzt11
-lxml11 -lasmclnt11 -lcommon11 -lcell11 -lskgxp11 -lgeneric11  -lcommon11 -lgeneric11  -lclntsh 
`cat /u01/app/oracle/product/ora11r2p/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11
-lnro11 `cat /u01/app/oracle/product/ora11r2p/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11
-lnnz11 -lzt11 -lztkg11 -lztkg11 -lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11 -lmm -lsnls11
-lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11
-lcore11 -lnls11 `cat /u01/app/oracle/product/ora11r2p/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11
-lnl11 -lnro11 `cat /u01/app/oracle/product/ora11r2p/lib/ldflags`    -lncrypt11 -lnsgr11 -lnzjs11 -ln11 -lnl11
-lclient11 -lnnetd11  -lvsn11 -lcommon11 -lgeneric11   -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11
-lsnls11 -lnls11 -lxml11 -lcore11 -lunls11 -lsnls11 -lnls11 -lcore11 -lnls11 -lclient11 -lnnetd11  -lvsn11
-lcommon11 -lgeneric11 -lsnls11 -lnls11  -lcore11 -lsnls11 -lnls11 -lcore11 -lsnls11 -lnls11 -lxml11 -lcore11
-lunls11 -lsnls11 -lnls11 -lcore11 -lnls11   `cat /u01/app/oracle/product/ora11r2p/lib/sysliblist`
-Wl,-rpath,/u01/app/oracle/product/ora11r2p/lib -lm    `cat /u01/app/oracle/product/ora11r2p/lib/sysliblist`
-ldl -lm   -L/u01/app/oracle/product/ora11r2p/lib
test ! -f /u01/app/oracle/product/ora11r2p/bin/kfed ||\
           mv -f /u01/app/oracle/product/ora11r2p/bin/kfed /u01/app/oracle/product/ora11r2p/bin/kfedO
mv /u01/app/oracle/product/ora11r2p/rdbms/lib/kfed /u01/app/oracle/product/ora11r2p/bin/kfed
chmod 751 /u01/app/oracle/product/ora11r2p/bin/kfed

Using kfed to dump disk header

In the following example we use kfed to dump the asm disk header of an asm disk represented by device “/dev/sdg1″:

/u01/app/oracle/product/11.2.0/ora11p/bin/kfed read /dev/sdg1
kfbh.endian:                          1 ; 0x000: 0x01
kfbh.hard:                          130 ; 0x001: 0x82
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD
kfbh.datfmt:                          1 ; 0x003: 0x01
kfbh.block.blk:                       0 ; 0x004: T=0 NUMB=0x0
kfbh.block.obj:              2147483648 ; 0x008: TYPE=0x8 NUMB=0x0
kfbh.check:                  2733723458 ; 0x00c: 0xa2f14f42
kfbh.fcn.base:                        0 ; 0x010: 0x00000000
kfbh.fcn.wrap:                        0 ; 0x014: 0x00000000
kfbh.spare1:                          0 ; 0x018: 0x00000000
kfbh.spare2:                          0 ; 0x01c: 0x00000000
kfdhdb.driver.provstr: ORCLDISKDISK003A ; 0x000: length=16
kfdhdb.driver.reserved[0]:   1263749444 ; 0x008: 0x4b534944
kfdhdb.driver.reserved[1]:   1093873712 ; 0x00c: 0x41333030
kfdhdb.driver.reserved[2]:            0 ; 0x010: 0x00000000
kfdhdb.driver.reserved[3]:            0 ; 0x014: 0x00000000
kfdhdb.driver.reserved[4]:            0 ; 0x018: 0x00000000
kfdhdb.driver.reserved[5]:            0 ; 0x01c: 0x00000000
kfdhdb.compat:                186646528 ; 0x020: 0x0b200000
kfdhdb.dsknum:                        0 ; 0x024: 0x0000
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER
kfdhdb.dskname:                DISK003A ; 0x028: length=8
kfdhdb.grpname:                   DATA2 ; 0x048: length=5
kfdhdb.fgname:              CONTROLLER1 ; 0x068: length=11
kfdhdb.capname:                         ; 0x088: length=0
kfdhdb.crestmp.hi:             32925079 ; 0x0a8: HOUR=0x17 DAYS=0xc MNTH=0x9 YEAR=0x7d9
kfdhdb.crestmp.lo:           2888263680 ; 0x0ac: USEC=0x0 MSEC=0x1da SECS=0x2 MINS=0x2b
kfdhdb.mntstmp.hi:             32925206 ; 0x0b0: HOUR=0x16 DAYS=0x10 MNTH=0x9 YEAR=0x7d9
kfdhdb.mntstmp.lo:           1862809600 ; 0x0b4: USEC=0x0 MSEC=0x20e SECS=0x30 MINS=0x1b
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000
kfdhdb.ausize:                  4194304 ; 0x0bc: 0x00400000
kfdhdb.mfact:                    454272 ; 0x0c0: 0x0006ee80
kfdhdb.dsksize:                    5119 ; 0x0c4: 0x000013ff
kfdhdb.pmcnt:                         2 ; 0x0c8: 0x00000002
kfdhdb.fstlocn:                       1 ; 0x0cc: 0x00000001
kfdhdb.altlocn:                       2 ; 0x0d0: 0x00000002
kfdhdb.f1b1locn:                      2 ; 0x0d4: 0x00000002
kfdhdb.redomirrors[0]:                0 ; 0x0d8: 0x0000
kfdhdb.redomirrors[1]:                0 ; 0x0da: 0x0000
kfdhdb.redomirrors[2]:                0 ; 0x0dc: 0x0000
kfdhdb.redomirrors[3]:                0 ; 0x0de: 0x0000
kfdhdb.dbcompat:              186646528 ; 0x0e0: 0x0b200000
kfdhdb.grpstmp.hi:             32925079 ; 0x0e4: HOUR=0x17 DAYS=0xc MNTH=0x9 YEAR=0x7d9
kfdhdb.grpstmp.lo:           2887388160 ; 0x0e8: USEC=0x0 MSEC=0x283 SECS=0x1 MINS=0x2b
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000
kfdhdb.spfile:                        0 ; 0x0f4: 0x00000000
kfdhdb.spfflg:                        0 ; 0x0f8: 0x00000000
kfdhdb.ub4spare[0]:                   0 ; 0x0fc: 0x00000000
kfdhdb.ub4spare[1]:                   0 ; 0x100: 0x00000000
kfdhdb.ub4spare[2]:                   0 ; 0x104: 0x00000000
kfdhdb.ub4spare[3]:                   0 ; 0x108: 0x00000000
kfdhdb.ub4spare[4]:                   0 ; 0x10c: 0x00000000
kfdhdb.ub4spare[5]:                   0 ; 0x110: 0x00000000
kfdhdb.ub4spare[6]:                   0 ; 0x114: 0x00000000
kfdhdb.ub4spare[7]:                   0 ; 0x118: 0x00000000
kfdhdb.ub4spare[8]:                   0 ; 0x11c: 0x00000000
kfdhdb.ub4spare[9]:                   0 ; 0x120: 0x00000000
kfdhdb.ub4spare[10]:                  0 ; 0x124: 0x00000000
kfdhdb.ub4spare[11]:                  0 ; 0x128: 0x00000000
kfdhdb.ub4spare[12]:                  0 ; 0x12c: 0x00000000
kfdhdb.ub4spare[13]:                  0 ; 0x130: 0x00000000
kfdhdb.ub4spare[14]:                  0 ; 0x134: 0x00000000
kfdhdb.ub4spare[15]:                  0 ; 0x138: 0x00000000
kfdhdb.ub4spare[16]:                  0 ; 0x13c: 0x00000000
kfdhdb.ub4spare[17]:                  0 ; 0x140: 0x00000000
kfdhdb.ub4spare[18]:                  0 ; 0x144: 0x00000000
kfdhdb.ub4spare[19]:                  0 ; 0x148: 0x00000000
kfdhdb.ub4spare[20]:                  0 ; 0x14c: 0x00000000
kfdhdb.ub4spare[21]:                  0 ; 0x150: 0x00000000
kfdhdb.ub4spare[22]:                  0 ; 0x154: 0x00000000
kfdhdb.ub4spare[23]:                  0 ; 0x158: 0x00000000
kfdhdb.ub4spare[24]:                  0 ; 0x15c: 0x00000000
kfdhdb.ub4spare[25]:                  0 ; 0x160: 0x00000000
kfdhdb.ub4spare[26]:                  0 ; 0x164: 0x00000000
kfdhdb.ub4spare[27]:                  0 ; 0x168: 0x00000000
kfdhdb.ub4spare[28]:                  0 ; 0x16c: 0x00000000
kfdhdb.ub4spare[29]:                  0 ; 0x170: 0x00000000
kfdhdb.ub4spare[30]:                  0 ; 0x174: 0x00000000
kfdhdb.ub4spare[31]:                  0 ; 0x178: 0x00000000
kfdhdb.ub4spare[32]:                  0 ; 0x17c: 0x00000000
kfdhdb.ub4spare[33]:                  0 ; 0x180: 0x00000000
kfdhdb.ub4spare[34]:                  0 ; 0x184: 0x00000000
kfdhdb.ub4spare[35]:                  0 ; 0x188: 0x00000000
kfdhdb.ub4spare[36]:                  0 ; 0x18c: 0x00000000
kfdhdb.ub4spare[37]:                  0 ; 0x190: 0x00000000
kfdhdb.ub4spare[38]:                  0 ; 0x194: 0x00000000
kfdhdb.ub4spare[39]:                  0 ; 0x198: 0x00000000
kfdhdb.ub4spare[40]:                  0 ; 0x19c: 0x00000000
kfdhdb.ub4spare[41]:                  0 ; 0x1a0: 0x00000000
kfdhdb.ub4spare[42]:                  0 ; 0x1a4: 0x00000000
kfdhdb.ub4spare[43]:                  0 ; 0x1a8: 0x00000000
kfdhdb.ub4spare[44]:                  0 ; 0x1ac: 0x00000000
kfdhdb.ub4spare[45]:                  0 ; 0x1b0: 0x00000000
kfdhdb.ub4spare[46]:                  0 ; 0x1b4: 0x00000000
kfdhdb.ub4spare[47]:                  0 ; 0x1b8: 0x00000000
kfdhdb.ub4spare[48]:                  0 ; 0x1bc: 0x00000000
kfdhdb.ub4spare[49]:                  0 ; 0x1c0: 0x00000000
kfdhdb.ub4spare[50]:                  0 ; 0x1c4: 0x00000000
kfdhdb.ub4spare[51]:                  0 ; 0x1c8: 0x00000000
kfdhdb.ub4spare[52]:                  0 ; 0x1cc: 0x00000000
kfdhdb.ub4spare[53]:                  0 ; 0x1d0: 0x00000000
kfdhdb.acdb.aba.seq:                  0 ; 0x1d4: 0x00000000
kfdhdb.acdb.aba.blk:                  0 ; 0x1d8: 0x00000000
kfdhdb.acdb.ents:                     0 ; 0x1dc: 0x0000
kfdhdb.acdb.ub2spare:                 0 ; 0x1de: 0x0000

What does it all mean?

For most usages the following rows are most important:

Status of disk:

kfdhdb.hdrsts: 3 ; 0×027: KFDHDR_MEMBER

Name of the disk:

kfdhdb.dskname: DISK003A ; 0×028: length=8

Name of disk group the disk belongs to:

kfdhdb.grpname: DATA2 ; 0×048: length=5

Name of failure group the disk belongs to

kfdhdb.fgname: CONTROLLER1 ; 0×068: length=11

Sector size of disk:

kfdhdb.secsize: 512 ; 0x0b8: 0×0200

Blocksize of disk:

kfdhdb.blksize: 4096 ; 0x0ba: 0×1000

Provision string for use with asm:

kfdhdb.driver.provstr: ORCLDISKDISK003A

–> which finally means: “ORCL:DISK003A”

AU size:

kfdhdb.ausize: 4194304

In a followup i will dig all little bit deeper in ASM and provide samples of different headers in different combinations (external / normal / high redundancy, ….).

Using ASM with files (either locally or via NFS)

Ronny Egner — Mon, 21 Sep 2009 13:17:31 +0000

From time to time i need to configure additional and temporary storage for data migration or just testing purposes. Raw Devices are not recommended to use from 11g onwards.

So why not use the loopback device for that? With this scenario you can even place these disks on NFS and use it across nodes in a cluster:

1. Create the file

dd if=/dev/zero bs=1024k count=1000 of=/disk1

2. Setup loopback driver

losetup /dev/loop1 /disk1

3. Label device

oracleasm createdisk LOOP1 /dev/loop1

4. Use it with ASM