Archive

Archive for April, 2011

Removing NFS Locks on Opensolaris / Nexenta (or handling ORA-27086: unable to lock file – already in use when using Oracle over NFS)

April 29th, 2011 No comments

Facts

  • Oracle running on Linux / Solaris
  • Oracle data files and control file are stored on NFS
  • NFS server used Opensolaris or Nexenta

Symptom

ORA-00205: error in identifying controlfile, check alert log for more info
ORA-00210: cannot open the specified  control file
ORA-00202: control file:: '/u02/oradata/ORA11P/control02.ctl'
ORA-27086: unable to lock file - already in use
Linux-x86_64 Error: 11: Resource temporarily unavailable
Additional information: 8

Cause

Due to a database crash or unclean shutdown NFS locks were not properly release on storage side. You have to clean them manually in order to be able to startup the database again.

Solution

Step 1 – Remove NFS locks on Opensolaris / Nexenta side

root@nex2:/volumes# clear_locks oracle11
Clearing locks held for NFS client oracle11 on server nex2
clear of locks held for oracle11 on nex2 returned success

Note: It is NOT sufficient to enter the IP adress. You have to use the HOSTNAME here.

Step 2 – Mount & Open the database

oracle@oracle11:/u02/oradata/LIMSTEST> sqlplus / as sysdba

SQL*Plus: Release 11.2.0.2.0 Production on Thu Apr 28 20:53:19 2011
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Release 11.2.0.2.0 - 64bit Production

SQL> alter database mount;
Database altered.
SQL> alter database open;
Database altered.
Categories: Oracle in general Tags:

ORA-27154: post/wait create failed / ORA-27301: OS failure message: No space left on device when starting ASM or database instance

April 11th, 2011 15 comments

Today i came accross a very strange error. After rebooting one cluster node (which ran flawlessly before!) ASM instance came up fine but database instance failed with:

ORA-27154: post/wait create failed
ORA-27300: OS system dependent operation:semget failed with status: 28
ORA-27301: OS failure message: No space left on device
ORA-27302: failure occurred at: sskgpsemsper

It turned out the kernel settings were insufficent. The semaphore settings caused problems.

In /etc/sysctl.conf:

kernel.sem = 250 32000 100 128

This line needs to be changed to:

kernel.sem = 250 32000 100 256

Note the change of the last number from 128 to 256. After that applying the settings is done as root with:

sysctl -p

After that all instances came up just fine.

Categories: Oracle in general Tags: