Ronny Egners Blog » Oracle in general

PRVF-5300: Failed to retrieve active version for CRS on this node when installing 11.2.0.2 DB on 11.2.0.3.0 Grid Infrastructure

Ronny Egner — Thu, 13 Oct 2011 12:31:12 +0000

I just played with 11.2.0.3.0 patchset on Linux x86_64 (in my testcase Oracle Enterprise Linux 5.6) and tried to install a 11.2.0.2.0 database on it. It fails with:

PRVF-5300: Failed to retrieve active version for CRS on this node

The error stack in the installation log is:

ID: oracle.install.commons.util.exception.DefaultErrorAdvisor:745
oracle.cluster.verification.VerificationException: An internal error occurred within cluster
verification framework

ERRORMSG(linux): PRVF-5300 : Failed to retrieve active version for CRS on this node
        at oracle.cluster.verification.ClusterVerification.getPreReqTasksForSIDBInst(ClusterVerification.java:615)
        at oracle.install.ivw.db.action.PrereqAction.getProductVerificationTasks(PrereqAction.java:111)
        at oracle.install.commons.base.interview.common.action.AbstractPrereqAction.execute
        (AbstractPrereqAction.java:86)
        at oracle.install.commons.flow.AbstractFlowExecutor.startAction(AbstractFlowExecutor.java:358)
        at oracle.install.commons.flow.AbstractFlowExecutor.enterVertex(AbstractFlowExecutor.java:571)
        at oracle.install.commons.flow.AbstractFlowExecutor.transition(AbstractFlowExecutor.java:333)
        at oracle.install.commons.flow.AbstractFlowExecutor.nextState(AbstractFlowExecutor.java:268)
        at oracle.install.commons.flow.AbstractFlowExecutor.nextViewState(AbstractFlowExecutor.java:227)
        at oracle.install.commons.flow.DefaultFlowNavigator.goForward(DefaultFlowNavigator.java:58)
        at oracle.install.commons.flow.jewt.FlowWizard$1.run(FlowWizard.java:125)
        at oracle.install.commons.flow.jewt.FlowWizard$TransitionManager$1.run(FlowWizard.java:101)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
        at java.lang.Thread.run(Thread.java:595)

The problem

I started the installer with debug enabled just add “-debug -logLevel finest >inst1.out 2>inst2.out”). The log files gave some insight:

[Version.getVersion:497]  version String is 11.2.0.3.0
[Version.getVersion:498]  new Version().toString is 11.2.0.2.0
[VerificationUtil.getSIHAReleaseVersionObj:4986]  Configuration Exception:
PRKC-1137 : Unable to find Version object with string value 11.2.0.3.0
[VerificationUtil.getCRSUser:1362]  Active Version = null

The related query command is

"GI_HOME/bin/crsctl query has releaseversion"

Obviously 11.2.0.2.0 installer has problems with the string “11.2.0.3.0″.

Solution

One simple approach was to created a wrapper around crsctl to report a version of 11.2.0.2.0 when querying releaseversion:

cd $GRID_HOME /bin

mv crsctl crsctl.orig

Now create a script “crsctl” with the following contents:

EXEC=/u01/app/oragrid/product/11.2.0.3.0/bin/crsctl.orig
case $1 in
query)
 echo "Oracle High Availability Services release version on the local node is [11.2.0.2.0]"
;;
*)
        $EXEC $*
;;
esac

You can start the database installation. During the verification steps the installer might report the Oracle Restart Registry as invalid. Just ignore it. The installation should now run fine.

Note that this bug is NOT related to OEL 5.6. It is the installer which cannot deal with the version string of the newer grid infrastructure. So you will face this error on OEL 6, RedHat and SuSE as well.

Dont forget to revert the changes after the installation!

After installation finished i was able to create a database using ASM without any problems. Registering the database into Oracle Restart also worked fine.

INFO: task blocked for more than 120 seconds.

Ronny Egner — Thu, 13 Oct 2011 05:53:06 +0000

When running some high workloads on UEK kernels on systems with a lot of memory you might see the following errors in /var/log/messages:

INFO: task bonnie++:31785 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
bonnie++      D ffff810009004420     0 31785  11051               11096 (NOTLB)
ffff81021c771aa8 0000000000000082 ffff81103e62ccc0 ffffffff88031cb3
ffff810ac94cd6c0 0000000000000007 ffff810220347820 ffffffff80310b60
00016803dfd77991 00000000001312ee ffff810220347a08 0000000000000001
Call Trace:
[] :jbd:do_get_write_access+0x4f9/0x530
[] zone_statistics+0x3e/0x6d
[] :jbd:start_this_handle+0x2e5/0x36c
[] autoremove_wake_function+0x0/0x2e
[] :jbd:journal_start+0xc9/0x100
[] :ext3:ext3_write_begin+0x9a/0x1cc
[] generic_file_buffered_write+0x14b/0x675
[] __generic_file_aio_write_nolock+0x369/0x3b6
[] generic_file_aio_write+0x65/0xc1
[] :ext3:ext3_file_write+0x16/0x91
[] do_sync_write+0xc7/0x104
[] autoremove_wake_function+0x0/0x2e
[] thread_return+0x62/0xfe
[] vfs_write+0xce/0x174
[] sys_write+0x45/0x6e
[] tracesys+0xd5/0xe0

This is a know bug. By default Linux uses up to 40% of the available memory for file system caching. After this mark has been reached the file system flushes all outstanding data to disk causing all following IOs going synchronous. For flushing out this data to disk this there is a time limit of 120 seconds by default. In the case here the IO subsystem is not fast enough to flush the data withing 120 seconds. This especially happens on systems with a lof of memory.

The problem is solved in later kernels and there is not “fix” from Oracle. I fixed this by lowering the mark for flushing the cache from 40% to 10% by setting “vm.dirty_ratio=10″ in /etc/sysctl.conf. This setting does not influence overall database performance since you hopefully use Direct IO and bypass the file system cache completely.

11.2.0.3.0 is out

Ronny Egner — Thu, 13 Oct 2011 05:46:32 +0000

Since almost two week Oracle released Oracle database patchset 11.2.0.3.0.

I am a little bit late with this news. Martin Bach already posted a lot of stuff about this so let me link to his guides:

At the moment 11.2.0.3.0 is available for Linux (32/64 bit) and Solaris SPARC.

When patching is not enough: Oracle 11g R2 on Solaris SPARC requires fresh base installation of Solaris 10 U6

Ronny Egner — Wed, 15 Jun 2011 14:18:44 +0000

While checking MOS i found an interesting note (ID 964976.1) which states:

Applying a kernel patch or a Solaris patch bundle is not the equivalent
to  installing the specific Solaris 10 "update 6" image. 11gR2 RDBMS software
is  only certified for a base install image of Solaris 10 update 6 or greater.

There is a FAQ (ID 971464.1) on this problem. Here it states:

Oracle/Sun has specifically started that "installing patches will not bring it  to Update 6".

and

It is only certified for a base install image of Solaris 10 Update 6  or greater, or an
upgraded image of an earlier Solaris 10 update to at least  Update 6 or greater. There are
only two methods to accomplish this " image".  Please see Question #9 for more details.

So keep this in mind when installing 11g R2 on Solaris SPARC.

11.2.0.2: two critical bugs

Ronny Egner — Fri, 27 May 2011 14:09:03 +0000

Just found two nice bugs in Metalink for 11.2.0.2.x:

Bug #1 (10205230) [Note ID 1318986.1]: ORA-00600 or DATA CORRUPTION in RAC Environments
when using shutdown mode "normal", "transactional" or "immediate" on 11.2.0.2.1 and 11.2.0.2.0.

This Bug is fixed in 11.2.0.2 PSU 2.

Now if you install 11.2.0.2 PSU 2 you might find the next major bug:

Bug #2 (12431716): Mutex waits may cause higher CPU usage in 11.2.0.2.2 PSU / GI PSU [ID 12431716.8]

Removing NFS Locks on Opensolaris / Nexenta (or handling ORA-27086: unable to lock file – already in use when using Oracle over NFS)

Ronny Egner — Fri, 29 Apr 2011 06:46:48 +0000

Facts

Oracle running on Linux / Solaris
Oracle data files and control file are stored on NFS
NFS server used Opensolaris or Nexenta

Symptom

ORA-00205: error in identifying controlfile, check alert log for more info
ORA-00210: cannot open the specified  control file
ORA-00202: control file:: '/u02/oradata/ORA11P/control02.ctl'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8

Cause

Due to a database crash or unclean shutdown NFS locks were not properly release on storage side. You have to clean them manually in order to be able to startup the database again.

Solution

Step 1 – Remove NFS locks on Opensolaris / Nexenta side

root@nex2:/volumes# clear_locks oracle11
Clearing locks held for NFS client oracle11 on server nex2
clear of locks held for oracle11 on nex2 returned success

Note: It is NOT sufficient to enter the IP adress. You have to use the HOSTNAME here.

Step 2 – Mount & Open the database

oracle@oracle11:/u02/oradata/LIMSTEST> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Apr 28 20:53:19 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Release 11.2.0.2.0 - 64bit Production

SQL> alter database mount;
Database altered.
SQL> alter database open;
Database altered.

ORA-27154: post/wait create failed / ORA-27301: OS failure message: No space left on device when starting ASM or database instance

Ronny Egner — Mon, 11 Apr 2011 09:02:12 +0000

Today i came accross a very strange error. After rebooting one cluster node (which ran flawlessly before!) ASM instance came up fine but database instance failed with:

ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper

It turned out the kernel settings were insufficent. The semaphore settings caused problems.

In /etc/sysctl.conf:

kernel.sem = 250 32000 100 128

This line needs to be changed to:

kernel.sem = 250 32000 100 256

Note the change of the last number from 128 to 256. After that applying the settings is done as root with:

sysctl -p

After that all instances came up just fine.

PSU Bundle Patch 3 for 11.2.0.2.0

Ronny Egner — Wed, 30 Mar 2011 09:49:39 +0000

Bundle Patch 3 for Oracle 11.2.0.2.0 is available.

For installation refer to these posts here and here.

Patch number is: 10387939

If youre worried about the fact the patch says it is for Exadata read this post from Oracle.

Money Quote:

Officially this Bundle Patch for Oracle Database 11.2.0.2 is titled “Exadata Database recommended patch” and got released yesterday. But I would recommend this one to all customers using 11.2.0.2 Grid Infrastructure, RAC and ASM.

RMAN backup validate database on databases in noarchivelog mode

Ronny Egner — Thu, 24 Mar 2011 09:00:36 +0000

Starting with 11g RMAN seems to allow a “backup validate database” even if the database is not in archive log mode:

SQL*Plus: Release 11.2.0.2.0 Production on Wed Mar 23 11:18:01 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
SQL> archive log list;
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            /u01/oradata/ORA11/archive
Oldest online log sequence     140
Current log sequence           143

As you can see from the listing above the database is not in archive log mode. Nevertheless doing a “backup validate database” works:

:/tmp# rman target /
Recovery Manager: Release 11.2.0.2.0 - Production on Wed Mar 23 11:18:25 2011
Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.
connected to target database: ORA11(DBID=12345678)
RMAN> backup validate datafile 1;
Starting backup at 23-MAR-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=124 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001 name=/u01/oradata/ORA11/system01.dbf
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:07
List of Datafiles
=================
File Status Marked Corrupt Empty Blocks Blocks Examined High SCN
---- ------ -------------- ------------ --------------- ----------
1    OK     0              47774        307200          404393656
File Name: 
Block Type Blocks Failing Blocks Processed
---------- -------------- ----------------
Data       0              237701
Index      0              19286
Other      0              2439

Doing the same with a 10g database yields:

:/tmp# rman target /
Recovery Manager: Release 10.2.0.5.0 - Production on Wed Mar 23 11:19:20 2011
Copyright (c) 1982, 2007, Oracle.  All rights reserved.
connected to target database: ORA10 (DBID=12343456)
RMAN> backup validate datafile 1;
Starting backup at 23-MAR-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: sid=201 devtype=DISK
channel ORA_DISK_1: starting full datafile backupset
channel ORA_DISK_1: specifying datafile(s) in backupset
RMAN-03009: failure of backup command on ORA_DISK_1 channel at 03/23/2011 11:19:27
ORA-19602: cannot backup or copy active file in NOARCHIVELOG mode

Networker cumulative hotfixes

Ronny Egner — Wed, 29 Dec 2010 07:15:15 +0000

EMC offers so called cumulative hotfixes which include all available and tested hotfixes for a specific version. These hotfixes can be found at the following URL:

ftp://ftp.legato.com/pub/NetWorker/Cumulative_Hotfixes