KB867473: Fix 34352668 in 19.29 and 19.30 DBRU May Cause Corruption in Primary Database and Redo Corruption Impacting Standby Recovery

Oracle has just released a warning about possible block and/or redo corruptions for 19.29 and 19.30 DBRU.

Please check out the following note for more details: KB867473

So far it seems only RAC databases are affected. This has been confirmed by a SR:

This bug is affecting database with multiple RAC instance.
Since oracle Restart uses ASM and its has only one db instance, oracle restart env and standalone db's are not affected.
This is not an ASM level bug and to highlight that it was described in such a way.

Posted in Oracle in general | Leave a comment

AHF-00026: Perl is not owned by root. AHF Requires Perl to be root owned for AHF Installations as the root user.

Trying to install AHF in version 25.11 on Oracle Linux 9 fails with the following error:

[root@XXX ahf]# ./ahf_setup
AHF Installer for Platform Linux Architecture x86_64
AHF Installation Log : /tmp/ahf_install_2511000_14614_2026_01_07-14_52_12.log
Starting Autonomous Health Framework (AHF) Installation
AHF Version: 25.11.0 Build Date: 202512150704
PERL : /u01/app/grid/product/19.0.0/grid/perl/bin/perl
[ERROR] : AHF-00026: Perl is not owned by root. AHF Requires Perl to be root owned for AHF Installations as the root user.

This is caused by missing perl modules in the default perl installation.

Checking the AHF install logs shows for example:

[2026-01-07 14:50:11.625 CET] Successfully validated Perl Owner of : /bin/perl
[2026-01-07 14:50:11.632 CET] Perl in checkPerlModules : /bin/perl
[2026-01-07 14:50:11.639 CET] Skip Perl Module List :
[2026-01-07 14:50:11.671 CET] Perl Module : Data::Dumper, Version : 2.174, Exit Status : 0
[2026-01-07 14:50:11.741 CET] Perl Module : File::Find, Version : , Exit Status : 2
[2026-01-07 14:50:11.749 CET] Perl Module not found : File::Find
[2026-01-07 14:50:11.781 CET] Perl Module : File::Path, Version : 2.18, Exit Status : 0
[2026-01-07 14:50:11.810 CET] Perl Module : File::Spec::Functions, Version : 3.78, Exit Status : 0
[2026-01-07 14:50:11.837 CET] Perl Module : File::Spec, Version : 3.78, Exit Status : 0
[2026-01-07 14:50:11.882 CET] Perl Module : File::Temp, Version : 0.2311, Exit Status : 0
[2026-01-07 14:50:11.897 CET] Perl Module : FindBin, Version : , Exit Status : 2
[2026-01-07 14:50:11.903 CET] Perl Module not found : FindBin
[2026-01-07 14:50:12.128 CET] Perl Module : List::Util, Version : 1.56, Exit Status : 0
[2026-01-07 14:50:12.142 CET] Perl Module : Math::BigInt, Version : , Exit Status : 2
[2026-01-07 14:50:12.149 CET] Perl Module not found : Math::BigInt
[2026-01-07 14:50:12.183 CET] Perl Module : Net::Domain, Version : 3.13, Exit Status : 0
[2026-01-07 14:50:12.198 CET] Perl Module : Net::Ping, Version : , Exit Status : 2
[2026-01-07 14:50:12.205 CET] Perl Module not found : Net::Ping
[2026-01-07 14:50:12.235 CET] Perl Module : Storable, Version : 3.21, Exit Status : 0
[2026-01-07 14:50:12.262 CET] Perl Module : Term::ANSIColor, Version : 5.01, Exit Status : 0
[2026-01-07 14:50:12.285 CET] Perl Module : Text::ParseWords, Version : 3.30, Exit Status : 0
[2026-01-07 14:50:12.312 CET] Perl Module : Text::Wrap, Version : 2013.0523, Exit Status : 0
[2026-01-07 14:50:12.339 CET] Perl Module : Time::Local, Version : 1.30, Exit Status : 0
[2026-01-07 14:50:12.354 CET] Perl Module : lib, Version : , Exit Status : 2
[2026-01-07 14:50:12.361 CET] Perl Module not found : lib
[2026-01-07 14:50:12.367 CET] PERL_ERROR in checkPerlModules : AHF-00074: Required Perl Modules not found : File::Find FindBin Math::BigInt Net::Ping lib
[2026-01-07 14:50:12.374 CET] Successfully validated Perl Modules in : /bin/perl

The solution is pretty simple: Install the missing perl modules. In my case with AHF 25.11 on Oracle Linux 9:

dnf install perl-File-Find perl-FindBin perl-Math-BigInt perl-Net-Ping perl-lib perl-File-Copy perl-English perl-Sys-Hostname perl-Term-ReadLine perl-locale

Posted in Oracle in general | Leave a comment

Changing the Timezone for a particular Database *only* in a RAC

Sometimes there are requests that multiple databases running on the same RAC return different time values — so they must run in different time zones.

This goal cannot achieved by changing the server time zone because that would affect ALL databases.

The solution is fairly simple:

bash-4.2$ srvctl setenv database -db ORCL -env "TZ=CET-1"
bash-4.2$ srvctl stop database -d ORCL
bash-4.2$ srvctl start  database -d ORCL

This effectively changes the environment variables in the clusterware for this particular database:

 

bash-4.2$ sqlplus c##ronny/ronny@ORCL

SQL*Plus: Release 18.0.0.0.0 - Production on Wed Feb 6 09:29:03 2019

Version 18.3.0.0.0

Copyright (c) 1982, 2018, Oracle.  All rights reserved.

Connected to:

Oracle Database 18c Enterprise Edition Release 18.0.0.0.0 - Production

Version 18.3.0.0.0

SQL> select to_char(sysdate, 'HH24:MI') from dual;

TO_CH
-----
08:29

SQL> !date

Wed Feb  6 09:29:16 CET 2019

Note that this only works if you connect via SQLNet. Local connetions are unaffected!

Posted in Oracle in general | Leave a comment

ORA-01422: exact fetch returns more than requested number of rows ORA-06512: at “SYS.CDBVIEW_INTERN during upgrade

During a database upgrade vom 11.2.0.4 to 18.4.1 i encountered the following problem:

REASON:
      ERRORS FOUND: During Upgrade
         FILENAME: /data/oracle/product/cfgtoollogs/dbua/upgrade2019-02-22_10-18-20AM/DB/catupgrd1R0.log AT LINE NUMBER: 666432
------------------------------------------------------
Identifier CATALOG 19-02-22 11:00:37
SCRIPT    = [/data/oracle/product/18.4/rdbms/admin/cdcore_cols.sql]
ERROR     = [ORA-01422: exact fetch returns more than requested 
             number of rows ORA-06512: at "SYS.CDBVIEW_INTERNAL", line 199
ORA-06512: at "SYS.CDBVIEW_INTERNAL", line 64
ORA-06512: at "SYS.CDBVIEW", line 6
ORA-06512: at line 1
]
STATEMENT = [BEGIN CDBView.create_cdbview(false,'SYS','DBA_TAB_COLS','CDB_TAB_COLS'); END;
]
------------------------------------------------------

Note that the view name (here: DBA_TAB_COLS) can change!

 

The reason for this to fail is in the file $ORACLE_HOME/rdbms/admin/catcdbviews.sql:

 

  -- Check if oldview already exist, qualify with owner# to avoid ORA-1422
  -- error,if two dictionary view with the same name exists across different
  -- schemas like DBA_DV_STATUS which exists in both SYS and DVSYS schemas.

    begin
      execute immediate 'SELECT 1 FROM OBJ$ WHERE NAME = :1' ||
                       ' AND TYPE# in (2, 4) and owner# = :2'
               into cCheck using oldview, ownerId;

 

The developer already identified one issue with his code, but overlooked another: Whenever a database link gets referenced in a piece of pl/sql code (user code or internal code – does not matter), a object with the same name as referenced by the database link is created in OBJ$.

 

SQL> select obj#, name, remoteowner,linkname from obj$ where TYPE# in (2, 4) and name=’DBA_TAB_COLS’;

 

      OBJ# NAME           REMOTEOWNER   LINKNAME
---------- -------------- ------------- --------------------------
  14000219 DBA_TAB_COLS   SYS           STAGING_SYSTEM
      3334 DBA_TAB_COLS

As you can see we have a second object in OBJ$ named DBA_TAB_COLS for the database link. And this causes the upgrade to fail.

 

The fix is relatively simple: Either get a patch from Oracle or fix the code in catcdbviews.sql yourself:

begin
      execute immediate 'SELECT 1 FROM OBJ$ WHERE NAME = :1' ||
          ' AND TYPE# in (2, 4) and remoteowner is null and owner# = :2'
               into cCheck using oldview, ownerId;

After editing the scirpt, install it by executing catcdbview.sql to apply the changes and restart the upgrade. The upgrade will start from the failed step onwards.

Note that the package you edit here is only used during the database upgrade to transfer comment from non-CDB to CDB views. It is deleted after the upgrade and is not needed or used during normal database operation.

Posted in Oracle in general | Leave a comment

RMAN shows: could not locate the LNM save file ‘fqnacfte_1_1’ on server ‘nsrserverhost’. (2:9:2)

During an RMAN restore of a backup made to a networker server the restore fails retrieving a backup piece:

restore datafile 2;

channel ORA_SBT_TAPE_1: starting datafile backup set restore
channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
channel ORA_SBT_TAPE_1: restoring datafile 00002 to +DATA/dwh/datafile/sysaux.257.764502905
channel ORA_SBT_TAPE_1: reading from backup piece DWH_fqnacfte_1_1
channel ORA_SBT_TAPE_1: ORA-19870: error while restoring backup piece DWH_fqnacfte_1_1
ORA-19507: failed to retrieve sequential file, handle="DWH_fqnacfte_1_1", parms=""
ORA-27029: skgfrtrv: sbtrestore returned error
ORA-19511: Error received from media manager layer, error text:
   Could not locate the LNM save file 'DWH_fqnacfte_1_1' on server 'nsrserverhost'. (2:9:2)

Lets´s check if the backup set exists on the server:

root@nsrserverhost # mminfo -a -v -r name,ssid,cloneid,savetime,client,state,location,ssretent,ssbrowse,ssflags -q client=clientname | grep DWH_fqnacfte_1_1
RMAN:DWH_fqnacfte_1_1     3349640857  1336374016 05/07/12 clientname  E      JUKEBOX         05/14/12 05/14/12 vrEF
RMAN:DWH_fqnacfte_1_1     3349640857  1336374016 05/07/12 clientname         JUKEBOX         05/14/12 05/14/12 vrEF

So we can see the save set actually exists but is marked “Expired” (E). So let´s try to change that back:

nsrmm -S 3349640857 -w 07/28/12 -e 07/28/12
Cannot change browse/retention time of recyclable save set 3349640857

root@nsrserverhost # nsrmm -o notrecyclable -v -S 3349640857
Save set 3349640857 cannot be marked as notrecyclable.
Please specify the ssid/cloneid of the particular clone instance.
root@nsrserverhost # nsrmm -o notrecyclable -v -S 3349640857/1336374016
Mark save set copy 3349640857/1336374016 as not recyclable? y
root@nsrserverhost # nsrmm -S 3349640857 -w 07/28/12 -e 07/28/12
Cannot exceed the browse time of a recoverable save set 3349640857

Does not work as the information is already purged from the client database. In that case we need to restore the client index from a date before the expiry and change the browse and retention:

root@nsrserverhost # nsrck -L7 -t 05/07/12 clientname
nsrck: checking index for 'clientname'
9343:nsrck: The file index for client 'clientname' will be recovered.
Requesting 1 recover session(s) from server.
Recover completion time: Tue May 15 19:22:37 2012
nsrck: /var/nsr/index/clientname contains 69 records occupying 37 KB
nsrck: Completed checking 1 client(s)
root@scrkva16 # mminfo -a -v -r name,ssid,cloneid,savetime,client,state,location,ssretent,ssbrowse,ssflags -q client=scrdwh02 | grep DWHKVASY_fqnacfte_1_1
RMAN:DWH_fqnacfte_1_1     3349640857  1336374016 05/07/12 clientname         JUKEBOX         05/23/12 05/23/12 vF
RMAN:DWH_fqnacfte_1_1     3349640857  1336374016 05/07/12 clientname         JUKEBOX         05/23/12 05/23/12 vF
root@nsrserverhost # nsrmm -S 3349640857 -w 07/28/12 -e 07/28/12

And now the restore works as expected:

restore datafile 2;

channel ORA_SBT_TAPE_1: starting datafile backup set restore
channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
channel ORA_SBT_TAPE_1: restoring datafile 00002 to +DATA/dwh/datafile/sysaux.257.764502905
channel ORA_SBT_TAPE_1: reading from backup piece DWH_fqnacfte_1_1

Posted in Networker, Oracle in general | Leave a comment

Direct NFS: Failed to set socket buffer size.wtmax=[1048576] rtmax=[1048576], errno=-1 on Solaris

This error occured on Solaris (SPARC) using Direct NFS (dNFS). The MOS Note is incomplete and only recommends to increase the send and receive buffers, but the solution is below:

 

/usr/sbin/ndd -set /dev/tcp tcp_xmit_hiwat 1056768
/usr/sbin/ndd -set /dev/tcp tcp_recv_hiwat 1056768
/usr/sbin/ndd -set /dev/tcp tcp_max_buf 2097152

To check the values, use:

/usr/sbin/ndd /dev/tcp tcp_max_buf
/usr/sbin/ndd /dev/tcp tcp_xmit_hiwat
/usr/sbin/ndd /dev/tcp tcp_recv_hiwat
Posted in Oracle in general | Leave a comment

Exadata outage due to ASM disks missing

On some older Exadatas it can be that Grid Disks belonging to a cell are not found anymore. And not even FORCEing the moutn helps:

FAILGROUP HEADER_STATU COUNT(*)
------------------------------ ------------ ----------
EXA01CEL02 MEMBER 34
EXA01CEL03 MEMBER 34
EXA01CEL04 MEMBER 34
EXA01CEL05 MEMBER 34
EXA01CEL06 MEMBER 34
EXA01CEL07 MEMBER 34
EXA01CEL08 UNKNOWN 34
EXA01CEL09 UNKNOWN 34
EXA01CEL10 UNKNOWN 32
EXA01CEL11 UNKNOWN 34
EXA01CEL12 MEMBER 34
EXA01CEL13 MEMBER 34
EXA01CEL14 MEMBER 34



SQL> alter diskgroup DBFS_DG mount force;
alter diskgroup DBFS_DG mount force
*
ERROR at line 1:
ORA-15042: ASM disk "71" is missing from group number "1"
ORA-15042: ASM disk "70" is missing from group number "1"
ORA-15042: ASM disk "69" is missing from group number "1"
ORA-15042: ASM disk "68" is missing from group number "1"
ORA-15042: ASM disk "67" is missing from group number "1"
ORA-15042: ASM disk "66" is missing from group number "1"
ORA-15042: ASM disk "65" is missing from group number "1"
ORA-15042: ASM disk "64" is missing from group number "1"
ORA-15042: ASM disk "63" is missing from group number "1"
ORA-15042: ASM disk "62" is missing from group number "1"
ORA-15042: ASM disk "61" is missing from group number "1"
ORA-15042: ASM disk "60" is missing from group number "1"
ORA-15042: ASM disk "59" is missing from group number "1"
ORA-15042: ASM disk "58" is missing from group number "1"
ORA-15042: ASM disk "57" is missing from group number "1"
ORA-15042: ASM disk "56" is missing from group number "1"
ORA-15042: ASM disk "55" is missing from group number "1"
ORA-15042: ASM disk "54" is missing from group number "1"
ORA-15042: ASM disk "53" is missing from group number "1"
ORA-15042: ASM disk "52" is missing from group number "1"
ORA-15042: ASM disk "51" is missing from group number "1"
ORA-15042: ASM disk "50" is missing from group number "1"
ORA-15042: ASM disk "39" is missing from group number "1"
ORA-15042: ASM disk "38" is missing from group number "1"
ORA-15042: ASM disk "37" is missing from group number "1"
ORA-15042: ASM disk "36" is missing from group number "1"
ORA-15042: ASM disk "35" is missing from group number "1"
ORA-15042: ASM disk "34" is missing from group number "1"
ORA-15042: ASM disk "33" is missing from group number "1"
ORA-15042: ASM disk "32" is missing from group number "1"
ORA-15042: ASM disk "31" is missing from group number "1"
ORA-15042: ASM disk "30" is missing from group number "1"

This is fixed with Bug:

16769943 EXADATA: SYSTEM OUTAGE - FOUR CELL NODES IN UNKNOWN STATUS TO ASM
Posted in Oracle in general | Leave a comment

ORA-29760: instance_number parameter not specified — without INSTANCE_NUMBER set

Oracle is complaining about:

ORA-29760: instance_number parameter not specified

And you have triple checked that the INSTANCE_NUMBER parameter is specified, but you still get the error.

The reason for that is, that your ORACLE_SID contains a underscore, e.g. ORA_SB. Starting with Oracle 12c the *underscore* parameter is a reserved character to indicate a server pool in a RAC.
Blase do not use the underscore parameter in the SID and the problem is gone.

Posted in Oracle in general | Leave a comment

ORA-00902: invalid datatype on MDSYS.SDO_GEORASTER during CREATE TABLE in 12c

Since i haven´t found this on the web:

When you try to create a table with MDSYS.SGO_GEORASTER and you´re receiving an ORA-00902: invalid datapye try the following:

SQL> @create_tab.sql
        "GEORASTER" "MDSYS"."SDO_GEORASTER" ,
                            *
ERROR at line 12:
ORA-00902: invalid datatype

Try to grant EXECUTE permissions on the data type to the user creating the table.

Posted in Oracle in general | Leave a comment

RAC refuses to start – or how installing TSM caused a cluster outage

Some days back i setup a new server for a customer of mine. Installation and patching ran fine, cluster was stable – no issues.

Then suddenly the cluster started to act strange. Commands did not work anymore; after a reboot the closure refused to start. The log files showed:

[root@node1 trace]# cat crsctl_60236.trc
Trace file /u01/app/oracle/diag/crs/node1/crs/trace/crsctl_60236.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
2014-11-18 18:55:52.765853 :  OCRMSG:2014938688: prom_waitconnect: CONN NOT ESTABLISHED (0,9,1,2)
2014-11-18 18:55:52.765870 :  OCRMSG:2014938688: GIPC error [9] msg [gipcretPermissions]
   CLWAL:2014938688: clsw_Initialize: Error [32] from procr_init_ext
  CLWAL:2014938688: clsw_Initialize: Error [PROCL-32: Oracle High Availability Services on the local node is not running Messaging error [gipcretPermissions] [9]] from procr_init_ext
2014-11-18 18:55:52.766123 :    GPNP:2014938688: clsgpnpkww_initclswcx: [at clsgpnpkww.c:351] Result: (56) CLSGPNP_OCR_INIT. (:GPNP01201:)Failed to init CLSW-OLR context. CLSW Error (3): CLSW-3: Error in the cluster registry (OCR) layer. [32] [PROCL-32: Oracle High Availability Services on the local node is not running Messaging error [gipcretPermissions] [9]]
2014-11-18 18:55:52.781979 : GIPCNET:2014938688:  gipcmodNetworkProcessConnect: [network]  failed connect attempt endp 0x139a810 [000000000000003e] { gipcEndpoint : localAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=)(GIPCID=00000000-00000000-0))', remoteAddr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_node1_)(GIPCID=00000000-00000000-0))', numPend 0, numReady 1, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 0, wobj 0x136c6e0, sendp 0x136c4a0 status 13flags 0xa02c0712, flags-2 0x1, usrFlags 0x14000 }, req 0x13684c0 [0000000000000048] { gipcConnectRequest : addr 'clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_node1_)(GIPCID=00000000-00000000-0))', parentEndp 0x139a810, ret gipcretPermissions (9), objFlags 0x0, reqFlags 0x2 }
2014-11-18 18:55:52.781998 : GIPCNET:2014938688:  gipcmodNetworkProcessConnect: slos op  :  sgipcnDSConnectHelper
2014-11-18 18:55:52.782001 : GIPCNET:2014938688:  gipcmodNetworkProcessConnect: slos dep :  Permission denied (13)
2014-11-18 18:55:52.782003 : GIPCNET:2014938688:  gipcmodNetworkProcessConnect: slos loc :  connect
2014-11-18 18:55:52.782009 : GIPCNET:2014938688:  gipcmodNetworkProcessConnect: slos info:  failed to /var/tmp/.oracle/sOCSSD_LL_node1_
2014-11-18 18:55:52.782215 : CSSCLNT:2014938688: clsssConnect: gipc request failed with 9 (0x31)
2014-11-18 18:55:52.782274 : CSSCLNT:2014938688: clsssInitNative: connect to (ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_node1_)) failed, rc 9
2014-11-18 18:55:52.782442 :  CRSCTL:2014938688: GetParser::getClustermode: crsctl_cssctx failed with 6

The interesting bit is the “permission denied” thing. It most cases it points to a file system permission issue. As you can see from the code the affected directory is “/var/tmp/.oracle” which (at least on an Exacta) is a link to “/tmp”.

After looking carefully i saw that the directory permission for “/tmp” got changed:

drwxr-xr--   12 root root      4096 Oct  8 23:56 tmp

Which is not right! Luckily the fix is easy:

chmod 1777 /tmp

But what caused the issue? It turned out one of the things i´ve done was to install TSM. I´ve extracted the TAR archive in /tmp and installed it from there. I tried to reproduce the issue and succeeded indeed. By extracting the TAR file the directory permissions of the parent directory was changed:

BEFORE:

drwxrwxrwt   12 root root      4096 Nov 19 12:24 tmp

AFTER:

drwxr-xr--   12 root root      4096 Oct  8 23:56 tmp

Great work, IBM!

Posted in Oracle in general | Leave a comment