PRVF-5300: Failed to retrieve active version for CRS on this node

The error stack in the installation log is:

ID: oracle.install.commons.util.exception.DefaultErrorAdvisor:745
oracle.cluster.verification.VerificationException: An internal error occurred within cluster
verification framework

ERRORMSG(linux): PRVF-5300 : Failed to retrieve active version for CRS on this node
        at oracle.cluster.verification.ClusterVerification.getPreReqTasksForSIDBInst(ClusterVerification.java:615)
        at oracle.install.ivw.db.action.PrereqAction.getProductVerificationTasks(PrereqAction.java:111)
        at oracle.install.commons.base.interview.common.action.AbstractPrereqAction.execute
        (AbstractPrereqAction.java:86)
        at oracle.install.commons.flow.AbstractFlowExecutor.startAction(AbstractFlowExecutor.java:358)
        at oracle.install.commons.flow.AbstractFlowExecutor.enterVertex(AbstractFlowExecutor.java:571)
        at oracle.install.commons.flow.AbstractFlowExecutor.transition(AbstractFlowExecutor.java:333)
        at oracle.install.commons.flow.AbstractFlowExecutor.nextState(AbstractFlowExecutor.java:268)
        at oracle.install.commons.flow.AbstractFlowExecutor.nextViewState(AbstractFlowExecutor.java:227)
        at oracle.install.commons.flow.DefaultFlowNavigator.goForward(DefaultFlowNavigator.java:58)
        at oracle.install.commons.flow.jewt.FlowWizard$1.run(FlowWizard.java:125)
        at oracle.install.commons.flow.jewt.FlowWizard$TransitionManager$1.run(FlowWizard.java:101)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
        at java.lang.Thread.run(Thread.java:595)

The problem

I started the installer with debug enabled just add “-debug -logLevel finest >inst1.out 2>inst2.out”). The log files gave some insight:

[Version.getVersion:497]  version String is 11.2.0.3.0
[Version.getVersion:498]  new Version().toString is 11.2.0.2.0
[VerificationUtil.getSIHAReleaseVersionObj:4986]  Configuration Exception:
PRKC-1137 : Unable to find Version object with string value 11.2.0.3.0
[VerificationUtil.getCRSUser:1362]  Active Version = null

The related query command is

"GI_HOME/bin/crsctl query has releaseversion"

Obviously 11.2.0.2.0 installer has problems with the string “11.2.0.3.0″.

Solution

One simple approach was to created a wrapper around crsctl to report a version of 11.2.0.2.0 when querying releaseversion:

cd $GRID_HOME /bin

mv crsctl crsctl.orig

Now create a script “crsctl” with the following contents:

EXEC=/u01/app/oragrid/product/11.2.0.3.0/bin/crsctl.orig
case $1 in
query)
 echo "Oracle High Availability Services release version on the local node is [11.2.0.2.0]"
;;
*)
        $EXEC $*
;;
esac

You can start the database installation. During the verification steps the installer might report the Oracle Restart Registry as invalid. Just ignore it. The installation should now run fine.

Note that this bug is NOT related to OEL 5.6. It is the installer which cannot deal with the version string of the newer grid infrastructure. So you will face this error on OEL 6, RedHat and SuSE as well.

Dont forget to revert the changes after the installation!

After installation finished i was able to create a database using ASM without any problems. Registering the database into Oracle Restart also worked fine.

INFO: task bonnie++:31785 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
bonnie++      D ffff810009004420     0 31785  11051               11096 (NOTLB)
ffff81021c771aa8 0000000000000082 ffff81103e62ccc0 ffffffff88031cb3
ffff810ac94cd6c0 0000000000000007 ffff810220347820 ffffffff80310b60
00016803dfd77991 00000000001312ee ffff810220347a08 0000000000000001
Call Trace:
[<ffffffff88031cb3>] :jbd:do_get_write_access+0x4f9/0x530
[<ffffffff800ce675>] zone_statistics+0x3e/0x6d
[<ffffffff88032002>] :jbd:start_this_handle+0x2e5/0x36c
[<ffffffff800a28b4>] autoremove_wake_function+0x0/0x2e
[<ffffffff88032152>] :jbd:journal_start+0xc9/0x100
[<ffffffff88050362>] :ext3:ext3_write_begin+0x9a/0x1cc
[<ffffffff8000fda3>] generic_file_buffered_write+0x14b/0x675
[<ffffffff80016679>] __generic_file_aio_write_nolock+0x369/0x3b6
[<ffffffff80021850>] generic_file_aio_write+0x65/0xc1
[<ffffffff8804c1b6>] :ext3:ext3_file_write+0x16/0x91
[<ffffffff800182df>] do_sync_write+0xc7/0x104
[<ffffffff800a28b4>] autoremove_wake_function+0x0/0x2e
[<ffffffff80062ff0>] thread_return+0x62/0xfe
[<ffffffff80016a81>] vfs_write+0xce/0x174
[<ffffffff80017339>] sys_write+0x45/0x6e
[<ffffffff8005d28d>] tracesys+0xd5/0xe0

This is a know bug. By default Linux uses up to 40% of the available memory for file system caching. After this mark has been reached the file system flushes all outstanding data to disk causing all following IOs going synchronous. For flushing out this data to disk this there is a time limit of 120 seconds by default. In the case here the IO subsystem is not fast enough to flush the data withing 120 seconds. This especially happens on systems with a lof of memory.

The problem is solved in later kernels and there is not “fix” from Oracle. I fixed this by lowering the mark for flushing the cache from 40% to 10% by setting “vm.dirty_ratio=10″ in /etc/sysctl.conf. This setting does not influence overall database performance since you hopefully use Direct IO and bypass the file system cache completely.

I am a little bit late with this news. Martin Bach already  posted a lot of stuff about this so let me link to his guides:

• - Release Accouncement
• - Installation 11.2.0.3.0 on Oracle Linux 6.1 with UEK Kernel

At the moment 11.2.0.3.0 is available for Linux (32/64 bit) and Solaris SPARC.

ext4 was designed to allow far more larger file systems than ext3. According to wikipedia ext4 has a maximum file system size of 1 EiB (approx. one exabyte or 1024 TB).

Now if you´d try to create one single large file system with ext4 on every linux distribution out there (including OEL 6.1; as of 18th August 2011) you will end up with:

[root@localhost ~]# mkfs.ext4 /dev/iscsi/test mke4fs 1.41.9 (22-Aug-2009)
mkfs.ext4: Size of device /dev/iscsi/test too big to be expressed in 32 bit susing a blocksize of 4096.

This post is about how to solve the issue.

The demo system

My demo system consists of one large LUNof 18 TB encapsulated in LVM with a logical volume of 17 TB on a Oracle Enterprise Linux (OEL 5.5):

[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.18-194.el5 #1 SMP Mon Mar 29 22:10:29 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

[root@localhost ~]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 5.5 (Tikanga)

[root@localhost ~]# fdisk -l /dev/sdb
Disk /dev/sdb: 19791.2 GB, 19791209299968 bytes
255 heads, 63 sectors/track, 2406144 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table 

[root@localhost ~]# vgdisplay iscsi
--- Volume group ---
VG Name               iscsi
System ID
Format                lvm2
Metadata Areas        1
Metadata Sequence No  2
VG Access             read/write
VG Status             resizable
MAX LV                0
Cur LV                1
Open LV               0
Max PV                0
Cur PV                1
Act PV                1
VG Size               18.00 TB
PE Size               4.00 MB
Total PE              4718591
Alloc PE / Size       4456448 / 17.00 TB
Free  PE / Size       262143 / 1024.00 GB
VG UUID               tdi4f2-3ZYr-c1P0-NuSl-i3w2-5qQl-K75guj

[root@localhost ~]# lvdisplay iscsi
--- Logical volume ---
LV Name                /dev/iscsi/test
VG Name                iscsi
LV UUID                8q1UrT-ludC-FEkT-NExO-4Gzd-cn5H-FYJcB1
LV Write Access        read/write
LV Status              available
# open                 0
LV Size                17.00 TB
Current LE             4456448
Segments               1
Allocation             inherit
Read ahead sectors     auto
- currently set to     256
Block device           253:2

Creating file systems  larger than 16TB with ext4:

If you try to create a ext4 file system on the 17 TB logical volume:

[root@localhost ~]# mkfs.ext4 /dev/iscsi/test mke4fs 1.41.9 (22-Aug-2009)
mkfs.ext4: Size of device /dev/iscsi/test too big to be expressed in 32 bit susing a blocksize of 4096.

OK. Maybe with ext4dev:

[root@localhost ~]# mkfs.ext4dev /dev/iscsi/test mke4fs 1.41.9 (22-Aug-2009)
mkfs.ext4dev: Size of device /dev/iscsi/test too big to be expressed in 32 bits using a blocksize of 4096.

Nope – no success. The reason behind that are the e2fsprogs (or how they are called on OEL: e4fsprogs) are not able to deal with file systems larger than ~ 16 TB.

To be specific: Even with the most recent e2fsprogs 1.41.14 there is no way to create file systems larger than 16 TB.

But: According to this post it should work since June:

It’s taken way too long, but I’ve finally finished integrating the 64-bit patches into e2fsprogs’s mainline repository. All of the necessary patches should now be in the master branch for e2fsprogs. The big change from before is that I replaced Val’s changes for fixing up how mke2fs picked the correct fs-type profile from mke2fs.conf with something that I think works much better and leaves the code much cleaner. With this change you need to add the following to your /etc/mke2fs.conf file if you want to enable the 64-bit feature flag automatically for a big disk:

[fs_types] ext4 = {
features = has_journal,extent,huge_file,flex_bg,uninit_bg,dir_nlink,extra_isize
auto_64-bit_support = 1 # <—- add this line
inode_size = 256
}

Alternatively you can change the features line to include the feature “64bit”; this will force the use of the 64-bit fields, and double the size of the block group descriptors, even for smaller file systems that don’t require the 64-bit support. (This was one of my problems with Val’s implementation; it forced the mke2fs.conf file to always enable the 64-bit feature flag, which would cause backwards compatibility issues.) This might be a good thing to do for debugging purposes, though, so this is an option which I left open, but the better way of doing things is to use the auto_64-bit-support flag.

So the change must be there. A short look at the ‘WIP’ (work-in-progress) branch of the e2fsprogrs confirmed the integration.

So i tried to build the most recent e2fsprogs (Remeber: This are *development* tools – use at your OWN RISK):

[root@vm-mkmoel ~] git clone git://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git
[root@vm-mkmoel ~]# cd e2fsprogs
[root@vm-mkmoel e2fsprogs]# mkdir build ; cd build/
[root@vm-mkmoel build]# ../configure
[root@vm-mkmoel build]# make
[root@vm-mkmoel build]# make install

So let´s try to create a file system:

[root@vm-mkmoel misc]# ./mke2fs -O 64bit,has_journal,extents,huge_file,flex_bg, \
uninit_bg,dir_nlink,extra_isize -i 4194304 /dev/iscsi/test 

mke2fs 1.42-WIP (02-Jul-2011)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
4456448 inodes, 4563402752 blocks
228170137 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=6710886400
139264 block groups
32768 blocks per group, 32768 fragments per group
32 inodes per group

Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544, 1934917632,
2560000000, 3855122432
Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 0 mounts or 0 days,
whichever comes first.  Use tune2fs -c or -i to override.

OK. Seems to have worked. Lets check it:

[root@vm-mkmoel misc]# mount /dev/iscsi/test /mnt

[root@vm-mkmoel misc]# df -h
Filesystem                          Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00     18G  2.6G   14G  16% /
/dev/sda1                           99M  13M  82M    14% /boot
tmpfs                               502M 0    502M   0% /dev/shm
/dev/mapper/iscsi-test              17T  229M   17T   1% /mnt

[root@vm-mkmoel misc]# mount | grep mnt
/dev/mapper/iscsi-test on /mnt type ext4 (rw)

As you can see: With the most recent development e2fsprogrs it is possible to create ext4 file systems larger than 16 TB.

I even tried it with a 50 TB file system (because thats what i needed i my use case):

[root@vm-mkmoel misc]# df -h
Filesystem                          Size Used Avail Use% Mounted on
/dev/mapper/iscsi-test              50T  237M   48T   1% /mnt

Update:

Today i tested some more user space tools.

fsck

Maybe the most important tool in case the journaling fails. I copied some data to the file system (roughly about 2 TB) and had 73% of my 6.5 million inodes (one inode per 8 MB) allocated. Running fsck on my demo system with 1 GB memory yields:

[root@vm-mkmoel ~]# fsck.ext4 -f /dev/iscsi/test
e2fsck 1.42-WIP (02-Jul-2011)
Pass 1: Checking inodes, blocks, and sizes
Error allocating block bitmap (4): Memory allocation failed

fsck is some kind of messy with memory. Increasing the memory to 8 GB did it. While running fsck i noticed a memory consumption of up to 3.4 GB! So large file systems require a lot of memory for fscking. It requires even more memory with more inodes!

resize2fs

After fscking my file system i tried to resize it:

[root@localhost sbin]# lvresize -l +7199 /dev/iscsi/test
  Extending logical volume test to 50.00 TB
  Logical volume test successfully resized

[root@localhost sbin]# resize2fs /dev/iscsi/test
resize2fs 1.42-WIP (02-Jul-2011)
resize2fs: New size too large to be expressed in 32 bits

As you can see resizing the file system is not yet supported/implemented. So it would be wise to create the file system with the final size from start since growing is NOT possible!

tune2fs

tune2fs seems to work – at least it dumps the suberblock contents:

[root@localhost sbin]# tune2fs -l /dev/iscsi/test
tune2fs 1.42-WIP (02-Jul-2011)
Filesystem volume name:   <none>
Last mounted on:          /mnt/mnt
Filesystem UUID:          a754e947-8b89-415d-909d-000e6c95c44a
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags:         signed_directory_hash
Default mount options:    user_xattr acl
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              6550000
Block count:              13414400000
Reserved block count:     670720000
Free blocks:              13394134177
Free inodes:              1484526
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1024
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         16
Inode blocks per group:   1
Flex block group size:    16
Filesystem created:       Wed Oct 19 17:09:06 2011
Last mount time:          Wed Oct 19 18:45:47 2011
Last write time:          Wed Oct 19 18:45:47 2011
Mount count:              1
Maximum mount count:      20
Last checked:             Wed Oct 19 18:35:36 2011
Check interval:           0 (<none>)
Lifetime writes:          2511 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      ea117174-a04a-412e-a067-7972804f83d7
Journal backup:           inode blocks

Setting properties works as well:

[root@localhost sbin]# tune2fs -L test /dev/iscsi/test
tune2fs 1.42-WIP (02-Jul-2011)

[root@localhost sbin]# tune2fs -l /dev/iscsi/test | head -10
tune2fs 1.42-WIP (02-Jul-2011)
Filesystem volume name:   test
Last mounted on:          /mnt/mnt
[...]

e4defrag

e4defrag is a new tool to defragment the ext4 file system. According to the man page:

e4defrag  reduces  fragmentation of extent based file. The file targeted by e4defrag is created on ext4 filesystem made with “-O extent” option (see  mke2fs(8)).   The  targeted  file gets more contiguous blocks and improves the file access speed.

I am not yet sure how this affects file systems used for oracle datafiles. All i can say is that e4defrag seems to work with >16 TB file systems:

[root@localhost sbin]# e4defrag /mnt/
ext4 defragmentation for directory(/mnt/)
[....]
        Success:                        [ 4772040/5065465 ]
        Failure:                        [ 293425/5065465 ]

The failures are from directories which cannot be defragmented.

Conclusion

With the most recent e2fstools (1.42-WIP) it is possible to create ext4 file system larger than 16 TB.

If you do so remember the following:

• the tool is still in development – use at your own risk!
• tune the values for autocheck (after x mounts / after y days)
• adjust the “-i” switch which defnes the bytes/inode ratio; in the example above one inode is created for every 8 MB
• the more inodes you create the longer fsck takes and the more memory it needs
• Resizing the file system (growing / shrinking) is NOT possible at the moment

Applying a kernel patch or a Solaris patch bundle is not the equivalent
to  installing the specific Solaris 10 "update 6" image. 11gR2 RDBMS software
is  only certified for a base install image of Solaris 10 update 6 or greater.

There is a FAQ (ID 971464.1) on this problem. Here it states:

Oracle/Sun has specifically started that "installing patches will not bring it  to Update 6".

and

It is only certified for a base install image of Solaris 10 Update 6  or greater, or an
upgraded image of an earlier Solaris 10 update to at least  Update 6 or greater. There are
only two methods to accomplish this " image".  Please see Question #9 for more details.

So keep this in mind when installing 11g R2 on Solaris SPARC.

The task i tried to accomplish was to install Oracle Enterprise Linux in a para-virtualized guest via PXE and Kickstart. Sounds not too complicated, does it? If you are familiar with VMWARE and know how easy it is to accomplish this task be advised: With Oracle VM it is complicated.

First of all: Oracle VM para-virtualized guests are not able to boot from ISO files (fully virtualized ones are). So for booting you need a boot image containing the kernel which is accessible either via NFS or HTTP:

Task #1: Configure a NFS Server

(Note: A HTTP Server will also do the trick; in my environment configuring a NFS server was faster)

1. Copy the first CD or the whole DVD to a directory (or mount the ISO image); e.g. /export/oel
2. Export this directory via NFS by putting the following lines in /etx/exports/export/oel   *(ro)
3. Start the NFS server:/etc/init.d/nfs start

You should now be able to mount the NFS directory on the OVM physical hosts.

Task #2: Create a Kickstart-File

This task is usually done by the Systems Administrators. The Kickstart file itself can be placed either locally, on an NFS share  or via on a HTTP server. In this example the kickstart file is downloaded from a HTTP server.

Task #3: Install the guest

The following assumptions apply here:

• Hostname: GUEST
• Disk File Location: /var/ovs/mount/D5Elongid/running_pool/GUEST/system.img
  (Note: “D5Elongid” is just a placeholder for the full path pointing to the storage)
• Size of the disk file: 90 GB

Use the following command line to install the guest:

virt-install -n GUEST -r 4096 --vcpus=1 \
-f /var/ovs/mount/D5Elongid/running_pool/GUEST/system.img \
-s 90 -l nfs:servername.domain:/export/oel \
-x "ks=http://192.168.1.1:80/ks.cfg" -p

The following table explains the switches used:

Task #4: Import the guest into OVM Manager

1. Locate the XEN config file in /etc/xen and move it to the guests directory (in this example: /var/ovs/mount/D5Elongid/running_pool/GUEST) AND rename it to ‘vm.cfg’
2. In OVM Manager navigate to: RESOURCES –> VIRTUAL MACHINE IMAGES
3. Click Import
4. Choose ‘Select from Server Pool’ and fill out the following form
5. After youre done the machine will be registered
6. Click approve to finally add the machine

Conclusion

My current conclusion at the moment: Oracle VM 2.2.1 compared to e.g. VMWARE is in terms of usability worse. Simple tasks can be easily achieved GUI-wise; more complex tasks (e.g. booting via PXE or supplying boot-arguments) can only be done at command level.

In addition to that the documentation is complete crap. Basic information on setting everything up and creating virtual machines in the GUI are documented. Detailed information on how to do the more complexer tasks are not.

Bug #1 (10205230) [Note ID 1318986.1]: ORA-00600 or DATA CORRUPTION in RAC Environments
when using shutdown mode "normal", "transactional" or "immediate" on 11.2.0.2.1 and 11.2.0.2.0.

This Bug is fixed in 11.2.0.2 PSU 2.

Now if you install 11.2.0.2 PSU 2 you might find the next major bug:

Bug #2 (12431716): Mutex waits may cause higher CPU usage in 11.2.0.2.2 PSU / GI PSU [ID 12431716.8]

• Oracle running on Linux / Solaris
• Oracle data files and control file are stored on NFS
• NFS server used Opensolaris or Nexenta

Symptom

ORA-00205: error in identifying controlfile, check alert log for more info
ORA-00210: cannot open the specified  control file
ORA-00202: control file:: '/u02/oradata/ORA11P/control02.ctl'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8

Cause

Due to a database crash or unclean shutdown NFS locks were not properly release on storage side. You have to clean them manually in order to be able to startup the database again.

Solution

Step 1 – Remove NFS locks on Opensolaris / Nexenta side

root@nex2:/volumes# clear_locks oracle11
Clearing locks held for NFS client oracle11 on server nex2
clear of locks held for oracle11 on nex2 returned success

Note: It is NOT sufficient to enter the IP adress. You have to use the HOSTNAME here.

Step 2 – Mount & Open the database

oracle@oracle11:/u02/oradata/LIMSTEST> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Apr 28 20:53:19 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Release 11.2.0.2.0 - 64bit Production

SQL> alter database mount;
Database altered.
SQL> alter database open;
Database altered.

ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper

It turned out the kernel settings were insufficent. The semaphore settings caused problems.

In /etc/sysctl.conf:

kernel.sem = 250 32000 100 128

This line needs to be changed to:

kernel.sem = 250 32000 100 256

Note the change of the last number from 128 to 256. After that applying the settings is done as root with:

sysctl -p

After that all instances came up just fine.

For installation refer to these posts here and here.

Patch number is: 10387939

If youre worried about the fact the patch says it is for Exadata read this post from Oracle.

Money Quote:

Officially this Bundle Patch for Oracle Database 11.2.0.2 is titled “Exadata Database recommended patch” and got released yesterday. But I would recommend this one to all customers using 11.2.0.2 Grid Infrastructure, RAC and ASM.