Oracle 11g Release 2 – SSH equivalence gone after reboot

Today i came across a strange phenomenon:

I set up ssh equivalence during database installation but after rebooting the system some tool complained about misconfigured ssh equivalence. So i digged a little bit. I found out SSH is complaining about wrong directory permissions:

Sep 28 13:57:03 rac1 sshd[31620]: Authentication refused: bad ownership or modes for directory /u01/app/oracle/product/11.2.0/ora11p

The directory mentioned here is an ACFS shared oracle home and the home directory for the oracle user. After changing the mode for the directory to 755 ssh equivalence was restored. But after stopping and starting acfs resource the equivalence was gone again.

So i worked a little bit on that:

Lets first check directory permission with ACFS resource not started and not mounted:

[root@rac1 ~]# ll /u01/app/oracle/product/11.2.0/
total 4
drwxr-xr-x 2 ora11p dba 4096 Sep 13 00:02 ora11p
[root@rac1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              26G   13G   12G  52% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 1.9G  164M  1.7G   9% /dev/shm

Directory Mode is 775.

So now lets mount ACFS volume manually and check the permission again:

[root@rac1 ~]# mount /dev/asm/ora11p_home-132 /u01/app/oracle/product/11.2.0/ora11p/ -t acfs
[root@rac1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3              26G   13G   12G  52% /
/dev/sda1              99M   12M   83M  13% /boot
tmpfs                 1.9G  164M  1.7G   9% /dev/shm
/dev/asm/ora11p_home-132
 10G  4.7G  5.4G  47% /u01/app/oracle/product/11.2.0/ora11p
[root@rac1 ~]# ll /u01/app/oracle/product/11.2.0/
total 16
drwxrwx--- 83 ora11p dba 12288 Sep 25 10:37 ora11p

775 – still no surprise. So we change them to 775 as required by SSH:

[root@rac1 ~]# chmod 755 /u01/app/oracle/product/11.2.0/ora11p
[root@rac1 ~]# ll /u01/app/oracle/product/11.2.0/
total 16
drwxr-xr-x 83 ora11p dba 12288 Sep 25 10:37 ora11p

The directory show required permissions. We will now unmount and mount the file system again:

[root@rac1 ~]# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/sda3             26306612  12828996  12119724  52% /
/dev/sda1               101086     11683     84184  13% /boot
tmpfs                  1933304    167016   1766288   9% /dev/shm
/dev/asm/ora11p_home-132
 10485760   4864328   5621432  47% /u01/app/oracle/product/11.2.0/ora11p
[root@rac1 ~]# umount /u01/app/oracle/product/11.2.0/ora11p
[root@rac1 ~]#
[root@rac1 ~]# umount /u01/app/oracle/product/11.2.0/ora11p
[root@rac1 ~]# mount /dev/asm/ora11p_home-132 /u01/app/oracle/product/11.2.0/ora11p/ -t acfs           
[root@rac1 ~]# ll /u01/app/oracle/product/11.2.0/
total 16
drwxr-xr-x 83 ora11p dba 12288 Sep 25 10:37 ora11p

There we are: Permission mode 775 is consistent accross mounts. So lets unmount everything and start acfs resource again:

[root@rac1 ~]# umount /u01/app/oracle/product/11.2.0/ora11p
[root@rac1 ~]#
[root@rac1 ~]# /u01/app/11.2.0/grid/bin/crsctl start resource  ora.data2.ora11p_home.acfs
CRS-2672: Attempting to start 'ora.data2.ora11p_home.acfs' on 'rac1'
CRS-2672: Attempting to start 'ora.data2.ora11p_home.acfs' on 'rac2'
CRS-2676: Start of 'ora.data2.ora11p_home.acfs' on 'rac2' succeeded
CRS-2676: Start of 'ora.data2.ora11p_home.acfs' on 'rac1' succeeded
[root@rac1 ~]# ll /u01/app/oracle/product/11.2.0/
total 16
drwxrwx--- 83 ora11p dba 12288 Sep 25 10:37 ora11p

The directory mode 770 was changed to 770…. but by which component and configuration?

Until now i did not find a solution for this behavior yet. I will open a SR and see what Oracle thinks about this “feature”.

In the meantime to get rid of this behavior you can set “StrictModes no” in sshd_config to prevent ssh from checking directory ownerships.

Oracle 11 Release 2 Install Guide – Install Oracle RAC

System configuration

Hardware configuration

  • two virtual machines (VMWARE)
    • 1 vCPU
    • 4 GB RAM
    • 40 GB local Disk
  • Storage exported via ISCSI
    • 4 LUNs with 10 GB each
    • 2 LUNs with 30 GB each

Operating system configuration

  • Oracle Enterprise Linux 5.3 x86_64 (Kernel 2.6.18-128.el5)
  • Installed packages: default system + development packages

Cluster configuration

  • Cluster Name: „RAC“
  • Grid Binary installation on local disk
  • OCR, Voting and datafiles stored in ASM
  • Oracle HOME stored on ACFS file system (this menas we will have a SHARED home directory among all nodes!)
  • Oracle HOME will be installed unser user „ora11p“

Installation – Requirements and Preparations

Installation Steps outlined

  1. Installation of Oracle 11g Release 2 Grid Infrastructure → done here
  2. Installation of Oracle 11g Release 2 Database (rac installation)
    1. Review system requirements
    2. Install database software (aka „binaries“)
    3. Create database

general Requirements

All requirments from grid infrastructure installation apply here as well, for instance:

  • Kernel parameter
  • Limits (esp. configure limits for new user „ora11p“ which will hold the binary database installation)
  • Synchronous time
  • dns-resolvable SCAN name
  • working public and private interconnect
  • shared and accessible storage
  • at least ONE better two more disk groups created:
    • one for database files and binary installation
    • one for flashback recovery area
  • Note: SSH equivalence will be set up by installer

Preparations

Create ACFS file system for storing oracle binary installation

  • Create further ACFS mount directory
    mkdir -p /u01/app/oracle/product/11.2.0/ora11p
  • Create ADVM volume

step4_010

  • Create ACFS file system

step4_011

  • DO NOT register ACFS file system mountpoint for the oracle home directory! We will do this later (when creating the database) as clusterware resource
  • Mount ACFS file system on both nodes ( “mount /dev/asm/ora11p_home-132 /u01/app/oracle/product/11.2.0/ora11p”)

step4_014

 

Create User on both nodes:

useradd -u 501 -g dba -d /u01/app/oracle/product/11.2.0/ora11p ora11p
passwd ora11p
chown -R root:dba /u01/app/oracle/product
chmod -R 775 /u01/app/oracle/product
chown -R ora11p:dba /u01/app/oracle/product/11.2.0/ora11p

Create .bash_profile for user ora11p (note: changes on node A will be visible on node B cause were using ACFS; so changing profile file is needed only once)

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/ora11p
export ORACLE_SID=ORA11P
if [ -t 0 ]; then
stty intr ^C
fi

One word on Cluster Verification utility (cluvfy)

  • should be used prior installation
  • At this point the user „ora11p“ does not have SSH equivalence configured
  • So there are two options:
    • start cluvfy after installer set up ssh equivanlence
    • set up SSH equivalence manually
  • I personally prefer to run cluvfy after installer set up ssh equivalence

However the database installer seems to run a more or less complete check automatically

If you want to start cluvfy here is the syntax:

cluvfy stage -pre dbinst -fixup -n nodeA,nodeB -r <release> -osdba <name of sysdba group on unix> -verbose
<release> is: 10gR1, 10gR2, 11gR1, 11gR2

Installation

Start installer als user „ora11p“ on any node

step4_015

For testing purposes i deselected update notifications… in productive environments this is highly recommended

step4_016

we will install database software only and create database later

step4_017

both nodes were discovered correctly….

step4_018

Because were installing as user „ora11p“ which is different from the user holding our infrastructure installation we need to create passwordless ssh connectitivty

!! you also need to select „user home is shared“ cause were using ACFS !!

step4_019

step4_019

SSH connectivity successfully established…

step4_020

Select language support

step4_021

Select Edition

step4_022

Select Options

step4_023

Select ORACLE_BASE and ORACLE_HOME; we set it in profile file so it is entered by the installer automatically

step4_024

step4_025

Installation Summary

step4_026

Install process

step4_027

step4_028

Installation nearly finished.. just start „root.sh“ as root on all nodes

step4_029

root.sh sample output

step4_030

Finished

step4_031

Where to go now?

  • We just installed all neccessary components for creating our first RAC database → this will be the next post
  • backup current configuration

Oracle 11 Release 2 Install Guide – ACFS and ADVM

ACFS and ADVM

System configuration

We will use the system configured and installed in part 2

General information

  • ADVM = ASM dynamic volume manager
  • ACFS = ASM cluster file system
  • Basic layout

step3_010

(Source: Oracle Database Storage Administrator’s Guide 11g Release 2)

ADVM – Advantages

  • Integrated in ASM; this can be an disadvantage as well :-)
  • Inhertits storage from ASM hence enables host-based mirroring (either 2- or 3-way-mirroring)
  • multiple volumes within a disk group can be created with an file system such as ext3, ext4, reiserfs, … on top of it and will support storage of any file type as the file system normally woud – EXCEPT files which can be place in ASM directly
  • ADVM volume dynamically resizeable

ADVM – Disadvantages

  • ADVM volumes may be resized online; but the used file system must support it as well (ext3 on OEL 5 does support online resizing but does not support online shrinking)
  • Storing files which can be stored in ASM directly in ADVM + file system is not supported
  • NFS on top of ADVM is also not supported
  • ASM configuration assistant (asmca) only supports creation of volumes / file system… delete a volume / file system requires command line

ACFS – Advantages

  • cluster file system on top of ASM and ADVM
  • as available as ASM is (inherits storage from ASM disk group and ADVM volume)
  • Supports storage of files which cannot be directly stored in ASM, i.e.
    • executables
    • trace files
    • log files
    • Supports even oracle database binary installations
  • On ACFS read-only Snapshots can be created
  • dynamically resizeableUseable accross plattforms
  • Thoughts
    • Do i need licenses for grid infrastructrue?
    • If not: What if grid infrastructure + ASM used to provide host-based mirroring and cluster file system for non-oracle applications, for instance web servers ACFS Mount registry: used for mouting ACFS and ADVM file system across reboots

ACFS – Disadvantages

  • for example storing database files in ACFS is not supported, according to the documentation
    „Oracle Support Services will not take calls and development will not fix bugs associated with storing unsupported file types in Oracle ACFS“
  • Only available with RedHat Server 5 or Oracle Enterprise Linux 5 !
  • Disk group compatible parameter COPATBILE.ASM and COMPATIBLE.ADVM must be set so 11.2
  • ASM configuration assistant (asmca) only supports creation of volumes / file system… delete a volume / file system requires command line

First steps with ADVM and ACFS

Create a disk group for use with ADVM and ACFS

Lets first create an additional disk group called „DATA2“ which consists for two iSCSI LUNs with 30 GB each

Preparation:

  • LUNs visible with „fdisk -l“
  • Partition created (one on each LUN)
  • disk labeled with „oracleasm createdisk <name> <devpath>“
  • Create disk group in ASM (remember to connect as „sys as sysASM“!)

step3_011

  • Notes on disk groups
    • AUSIZE of 4 MB recommended by Oracle documentation due to:
      • Increased I/O through the I/O subsystem if the I/O size is increased to the AU size.
      • Reduced SGA size to manage the extent maps in the database instance.
      • Faster datafile initialization if the I/O size is increased to the AU size.
      • Increased file size limits.
      • Reduced database open time.
    • Large AUSIZE requires as well
      • Increasing size of maximum IO request to at least 4 MB in operating system, drive, HBA, storage system
      • Larger stripe size in storage system (pre 11g R2: 1 MB stripe size, with 11g R2: 4 MB? → to be tested)

Read the documentation on COMPATIBLE parameters;most „cool“ features are only available with 11.2 COMPATIBLE parameter hence require 11g R2 database

Creating an ADVM and afterwards an ACFS

Create an ADVM

  • ACFS requires ADVM in which ACFS can be created
  • volcreate creates ADVM volume

step3_012

  • The command above shows minimal command creating an ADVM volume; redundancy is derived from disk group, our data group was created with „normal“ redundancy so the volume inherits this as well)
  • Creation with SQL also possible: „ALTER DISKGROUP data2 ADD VOLUME volume1 SIZE 10G;“

Create ACFS on top of ADVM

  • Requires ADVM in which ACFS can be created
  • volinfo shows detailed information
  • Especially device path is important for creating the file system

step3_013

  • create ACFS from operating system (only on one node)

step3_014

  • register acfs in registry to be mounted across reboots with „acfsutil“
  • ATTENTION: DO NOT register shared oracle home directories with acfsutil; this will be done later by the clusterware itself!
  • test of everything works by issueing „mount.acfs -o all“ on all nodes; the file system should be mounted and accessible

step3_015

Simple Performance tests

dd if=/dev/zero bs=1024k of=<path> count=1000 oflag=direct

→ direct I/O used; no caching, performed 10 times

  • write to ACFS
    • ACFS 2-way mirror: ~ 6 MB/s average
    • ACFS unprotected: ~ 12 MB/s averga
    • → expected… one disk, double I/O halfed throughput
  • direct write to iSCSI LUN: ~ 14.3 MB/s average

Attention: Tests were performed within a VMWare… so results are most likely not accurate… but we get an impression.. we will check this on real hardware later!

Oracle 11g Release 2 Install Guide – Grid Infrastructure installation

System configuration:

Hardware configuration

  • two virtual machines (VMWARE)
  • 1 vCPU
  • 2 GB RAM –> bare minimum possible
  • 40 GB Disk
  • Storage exported via ISCSI
    • 4 LUNs with 2 GB each
    • 2 LUNs with 30 GB each

Operating system configuration

  • Oracle Enterprise Linux 5.3 x86_64 (Kernel 2.6.18-128.el5)
  • Installed packages: default system + development packages

Grid Infrastructure configuration:

  • Cluster Name: “RAC”
  • Binary installation on local disk
  • OCR, Voting and datafiles stored in ASM

Steps to install Oracle 11g Release 2 Grid Infrastructure

  • Configure Linux and pre-requirements
  • Configure Storage
  • Binary installation of grid infrastructure
  • Installation of Oracle 11g Release 2 Database (either single or rac installation)

Configure Linux and pre-requirements

SWAP

  • Between 1 and 2 GB RAM –>SWAP 1.5 times the size of RAM
  • Between 2 and 16 GB RAM –> equal to size of RA
  • > 16 GB RAM –> 16 GB SWAP

Memory

  • according to grid infrastructure documentation “>= 1 GB Memory”
  • bare minumum from authors experience:
    • 1 GB for grid infrastructure components
    • 500 MB for operating system
    • 1 GB for cluster database SGA/PGA/UGA
    • = 2,5 GB bare minimum!

See below for memory consumption with grid infrastructure installed: > 800 MB for infrastructure processes

step2_010

Automatic Memory Management

Required /dev/shm with appropriate size (i.e. SGA of 16 GB required /dev/shm to be 16 GB+)

Huge Pages and autom. Memory Management are INCOMPATIBLE

Checking required packages

(see required packages for single database installation; this applies here as well cause we will end up install a database in the end)

According to the documentation the following packages are needed:

  • binutils-2.17.50.0.6
  • compat-libstdc++-33-3.2.3
  • compat-libstdc++-33-3.2.3 (32 bit)
  • elfutils-libelf-0.125
  • elfutils-libelf-devel-0.125
  • gcc-4.1.2, gcc-c++-4.1.2
  • glibc-2.5-24, glibc-2.5-24 (32 bit)
  • glibc-common-2.5
  • glibc-devel-2.5
  • glibc-devel-2.5 (32 bit)
  • glibc-headers-2.5
  • ksh-20060214
  • libaio-0.3.106
  • libaio-0.3.106 (32 bit)
  • libaio-devel-0.3.106
  • libaio-devel-0.3.106 (32 bit)
  • libgcc-4.1.2, libgcc-4.1.2 (32 bit)
  • libstdc++-4.1.2
  • libstdc++-4.1.2 (32 bit)
  • libstdc++-devel 4.1.2
  • make-3.81
  • sysstat-7.0.2
  • unixODBC-2.2.11
  • unixODBC-2.2.11 (32 bit)
  • unixODBC-devel-2.2.11
  • unixODBC-devel-2.2.11 (32 bit)

On sample system with OEL 5.3 and default + development packages installed only the following rpms were missing:

rpm -ihv libaio-devel-0.3.106-3.2.* libstdc++43-devel-4.3.2-7.el5.* sysstat-7.0.2-3.el5.x86_64.rpm unixODBC-2.2.11-7.1.* unixODBC-devel-2.2.11-7.1.*

Shell Limits

/etc/security/limits.conf

grid                  soft     nproc    16384
grid                  hard    nproc    16384
grid                  soft     nofile   65536
grid                  hard    nofile   65536
grid                  soft     stack      10240
grid                 hard    stack      10240

In /etc/pam.d/login add if not exists

session    required     pam_limits.so

Kernel Limits (MINIMUM values) in /etc/sysctl.conf

kernel.sem=250 32000 100 128
kernel.shmall=2097152
kernel.shmmax=536870912
kernel.shmmni=4096
fs.file-max=6815744
fs.aio-max-nr=1048576
net.ipv4.ip_local_port_range=9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048576

– SuSE only –
vm.hugetlb_shm_group=<gid of osdba group>

The values in /etc/sysctl.conf should be tuned (i.e. according to the number of instance, available memory, number of connections,…)

Kernel Limits on Linux (Calculate them)

kernel.sem

semmns = Total number of semaphores systemwide =

2 * sum (process parameters of all database instances on the system)
+ overhead for background processes
+ system and other application requirements

semmsl = total semaphoren for each set

semmni = total semaphore sets =  semmns divided by semmsl, rounded UP to nearest multiple to 1024

kernel.sem = <semmsl semmns semopm semmni>

semmsl    =  set to 256
semmns    = set total number of semaphoren (see above!)
semopm    = 100; in documentation not explicitly described
semmni    = see calculcation above

kernel.shmall

kernel.shmall = This parameter sets the total amount of shared memory pages that  can be used system wide. Hence, SHMALL should always be at least   ceil(shmmax/PAGE_SIZE). PAGE_SIZE is usually 4096 bytes unless you use Big Pages or Huge  Pages which supports the configuration of larger memory pages.  (quoted from: www.puschitz.com/TuningLinuxForOracle.shtml)

kernel.shmmax

kernel.shmmax = the maximum size of a single shared memory segment in bytes that
a linux process can allocate
If not set properly database startup can fail with:

ORA-27123: unable to attach to shared memory segment

kernel.shmmni

kernel.shmmni = system wide number of shared memory segments; Oracle recommendation for 11g Release 1 “at least to 4096”; i did not found anything for Release 2….

fs.file-max

fs.file-max = maximum number of open files system-wide; must be at least %G„%@6815744$(B!H(B

fs.aio-max-nr

fs.aio-max-nr = concurrent outstanding i/o requests; must be set to %G„%@1048576$(B!H(B

net.ipv4.ip_local_port_range

net.ipv4.ip_local_port_range = mimimum and maximum ports for use; must be set to  minimal “9000” and “65500” as maximum

net.core.rmem_default

net.core.rmem_default = the default size in bytes of the receive buffer; must  be set at least to “262144”

net.core.rmem_max

net.core.rmem_max = the maximum size in bytes of the receive buffer; must be set at least to “@4194304”

net.core.wmem_default

net.core.wmem_default = the default size in bytes of the send buffer; must be set at least to “262144”

net.core.wmem_max

net.core.wmem_max = the maximum size in bytes of the send buffer; must be set at least to “1048576”

Networking

Basic facts

  • Works completely different than 10g or 11g R1!
  • At least two separated networks (public and private) and therefore two network interfaces required
  • ATTENTION: Interface names must be equal on ALL nodes! (i.e. If private network interface on node A is eth2 the private network interface name on all other nodes must be eth2 as well…. )
  • Recommendation: Use bonding for:
    • Static naming (even if you use only one interface per bond)
    • Failover / Load Sharing
    • –> we will use network bonding with only one interface in the following
  • IP adresses can be given by two schemes:
    • GNS (grid naming service) –> automatic ip numbering
    • Manual Mode
    • –> we will use manual ip adressing mode in the following
  • GNS mode requires:
    • one fixed public IP for each node
    • one dhcp virtual IP for each node
    • one hdcp for fixed private IP for each node
    • three dhcp IP for the SCAN
    • Thougths by the author:
      • new
      • more complex
      • if working quite easy adding of an node; at least from the ip numbering point of view %G–%@ but how often do you add a node?
  • Manual Mode ip adressing requires:
    • one public IP for each node
    • one virtual IP for each node
    • one private IP for each node
    • one to three (recommended) IPs for providing the SCAN name

Naming schema used in the following (remember: 2-node-cluster)

step2_011

Configure Network Bonding

In /etc/modprobe.conf add line:

alias bond0 bonding
alias bond1 bonding
options bonding miimon=100 mode=1 max-bonds=2
(“mode=1” means active/passive failover… see “bonding.txt” in kernel sources for more options)
/etc/sysconfig/network-scripts/ifcfg-bond0 looks like:

DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.180.0
NETMASK=255.255.255.0
IPADDR=192.168.180.10
USERCTL=no
/etc/sysconfig/network-scripts/ifcfg-eth0 looks like:

DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=yes

(Note: Add a second interface to achive real fault tolerance…. for our testing environment we use bonding to provide a consistent name schema)

The configuration for bond1 is not shown… just alter interface names and IPs.

Configure NTP

Grid Infrastructure provides ntp-like time synchronization with “ctss” (cluster time synchronization service) ctssd is provided in case connections to ntp servers are not possible

If no running (“chkconfig ntpd off” and configured “rm /etc/ntp.conf” ntpd is found ctssd will be used; if ntpd is found  ctssd will start in observer mode.

ATTENTION: Set the “-x” flag if you use ntp to prevent ntp from stepping the clock in /etc/sysconfig/ntpd!

Check if NTP is working

  • start “ntpq”
  • enter “opeer” to see list of all peers
    In our example two peers: host “nb-next-egner” and the local clock

step2_012

enter “as” to see associations
“sys.peer” means the clock is synchronized against this; the order in which the entries apper is like “opeer” – so first entry means host “nb-next-egner” – fine!
reject means not synchronized against due to various reasons
enter “rv” for detailed information

step2_013

SCAN

  • SCAN = Single Client Access Name; new concept in 11g R2
  • DNS-based
  • nameing notation: <name of cluster>-scan.<domain>
  • for our cluster named  “rac” with domain “regner.de” this is    rac-scan.regner.de
  • You need at least ONE – better three IPs for the new database access schema called SCAN
  • IPs are configured in DNS (forward and reverse lookup);
  • !! using local hosts file failed verification after grid installation !!
  • forward- and reverse lookup needs to be configured
  • excerpt from zone file:
    rac-scan          IN A          192.168.180.6
    rac-scan          IN A          192.168.180.7
    rac-scan          IN A          192.168.180.8

After installation we will find three listeners running from grid infrastructure home:

bash# srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node rac1
SCAN Listener LISTENER_SCAN2 is enabled
SCAN listener LISTENER_SCAN2 is running on node rac2
SCAN Listener LISTENER_SCAN3 is enabled
SCAN listener LISTENER_SCAN3 is running on node rac2

Connection to database “RAC11P” using SCAN would use this tnsnames entry:

RAC11P =
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=rac-scan.regner.de)(PORT=1521))
(CONNECT_DATA=(SERVICE_NAME=RAC11P))
)

The “old fashioned” way still works:

RAC11P_old =
(DESCRIPTION=
(ADDRESS_LIST=
(ADDRESS=(PROTOCOL=tcp)(HOST=rac1-vip.regner.de)(PORT=1521))
(ADDRESS=(PROTOCOL=tcp)(HOST=rac2-vip.regner.de)(PORT=1521))
)
(CONNECT_DATA=(SERVICE_NAME=RAC11P))
)

Connecting to a named instance:

RAC11P =
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=rac-scan.regner.de)(PORT=1521))
(CONNECT_DATA=(SERVICE_NAME=RAC11P)
(INSTANCE_NAME=RAC11P1))
)

Check DNS for SCAN

Update [16th October 2009]: If you do not have a working DNS server available refer here to set up your own,

Forward lookup

Use “dig” to check: “dig rac-scan.regner.de”

step2_014

Reverse lookup

Use “dig -x” to check

dig -x 192.168.180.6
dig -x 192.168.180.7
dig -x 192.168.180.8

step2_015

Create User and Group

Create Group

groupadd -g 500 dba

Note: For educational purposes we use only one group. In productive enviroments there should be more groups to separate administrative duties.

Create User

mkdir -p /u01/app/11.2.0/grid
chown -R root:dba /u01
chmod -R 775 /u01
chown -R grid:dba /u01/app/11.2.0/grid
useradd -g dba -u 500 -d /u01/app/11.2.0/grid grid
passwd grid

Note: Oracle recommends different users for grid and database installation!

Make sure groupid and userid are the same on ALL nodes!

Create profile file (~/.bash_profile or ~/.profile on SuSE) for user “grid”

umask 022
if [ -t 0 ]; then
stty intr ^C
fi

Prepare and Configure Storage

  • Requirements
    • must be visible on all nodes
    • as always – recommendation: SAME (stripe and mirror everything)
  • What to store where:
    • OCR and Voting disk
      • ASM
      • NFS
      • RAW disks (deprecated; read doucmentation!)
    • Oracle Clusterware binaries
      • NFS
      • Local disk
    • Oracle RAC binaries
      • ACFS
      • NFS
      • local disk
    • Oracle database files
      • ASM
      • NFS
      • RAW disks
    • Oracle recovery files
      • ASM
      • NFS

Install RPMs

  • oracleasmsupport
  • oracleasmlib
  • oracleasm-<kernel-version>
    (see “Sources” for download locatio)

Configure ASM

/usr/sbin/oracleasm configure -i

step2_016

init ASM

/usr/sbin/oracleasm init

Create Partitions on disk with fdisk

In the following example disk /dev/sde (this is our iSCSI storage) does not contain a partition at all  – we will create one

step2_017

Create one whole disk partition on /dev/sde

step2_018

Label all disks with asm label

step2_019

Query disks on all nodes – Node “rac1”

step2_020

–> all disks visible with correct label

Query disks on all nodes – Node “rac2” (the other node)

–> also all four LUNs visible

step2_021

OCR and Voting disks

  • Will be placed in ASM (new in 11g R2)
  • three different redundancy levels:
    • External – 1 disk minimum needed
    • Normal – 3 disks minumum needed
    • High  – 5 disks minimum needed
  • Storage Requirments
    • External – 280 MB OCR + 280 MB Voting Disk
    • Normal – 560 MB OCR + 840 MB Voting Disk
    • High  – 840 MB OCR + 1,4 GB Voting Disk
    • plus Overhead for ASM Metadata

Overhead for ASM metadata

total =
[2 * ausize * disks]
+ [redundancy * (ausize * (nodes * (clients + 1) + 30) + (64 * nodes) + 533)]

redundancy = Number of mirrors: external = 1, normal = 2, high = 3.
ausize = Metadata AU size in megabytes.
nodes = Number of nodes in cluster.
clients – Number of database instances for each node.
disks – Number of disks in disk group.
For example, for a four-node Oracle RAC installation, using three disks in a normal redundancy disk group, you require 1684 MB of space for ASM metadata

[2 * 1 * 3]
+ [2 * (1 * (4 * (4 + 1)+ 30)+ (64 * 4)+ 533)]
= 1684 MB

OCR and Voting disks – recommendations

  • use high redundancy for OCR and Voting disks –  the correct function of your cluster depends on it!
  • use 5 disks with 10 GB each – enough space for all files plus asm metadata plus space for futher growth

Checklist

  • Storage visible
  • user and groups created
  • Kernel parameters configured
  • RPM Packages checked / installed
  • NTP working
  • DNS working
  • Connection (ping, ssh) between nodes working?
  • Backup available for rollback?

–> Alright! Lets start binary installation

Installing

Start installation as user “grid” (on one node (here on node “rac1”))

step2_022
step2_023
step2_024
step2_025

Remember: We choose not to use GNS; so it is deselected

step2_026

The node the installer was started is already added by default; add here all other nodes (in our case we added “rac2”)
step2_027

Click on “SSH Connectivity”, enter username and password and click on “Setup”
step2_028

If everything worked the following message appears
step2_029

If there are problems check:

  • Group ID and User ID on both nodes
  • Connectivity between both nodes
  • Passwords

Select which interface is the public and which the private one
step2_030

Where to place OCR and Voting disk… in our case we use ASM for everything
step2_031

For storing OCR and Voting disk we need to create a data group; our first data group is called “DATA1” and consists of the four LUNs we prepared and labeled before… here we see the disk names we labeled the disks with again. We choose “normal” redundancy which will create a mirror.
step2_032

Specify passwords for ASM and ASMSNMP.. choose strong passwords if possible (i was lazy and chose not that strong ones – acceptable for educational purposes but not in real productive scenarios)
step2_033

Grid Infrastructure can use IPMI for fencing… VMWARE does not have IPMI
step2_034

group mapping…for role separation… we have only “dba” ? change accordingly to your needs
step2_035

Set Oracle Base and software (install) location… software location must not be under oracle base location… else the installer throws an error saying so
step2_036

Inventoriy location…
step2_037

Make sure you fix every issue reported here (memory and swap size are limited on virtual machine so this is not fixable…but should anyway)

step2_038

Ready..
step2_039

Installing…
step2_040

Post-Installation scripts to be started in the following order:

  • orainstRoot.sh on node rac1
  • orainstRoot.sh on node rac2
  • root.sh on node rac1
  • root.sh on node rac2

step2_041

Sample of root.sh output

step2_042

The full output can be found rac-install-node1 and rac-install-node2.

If everything works as expected the result should look like this:
step2_043

FINISHED
step2_044

If there are problems:

  • Look at the log files located on /u01/app/oraInventory/logs
  • Fix the issues noted here
  • If this does not work out: Search Metalink / Open SR

Where to go now?

  • We just installed the Infrastructure needed for RAC, i.e. ASM and Clusterware
  • Install diagnostic utilities (strongly recommended)
    • OSWatcher
    • and RACDDT
  • Tune Kernel parameters (if not done before)
  • Create at least TWO more disk groups:
    • one for holding database files (i.e. datafiles and binary installation files)
    • one to be used as flashback recovery area
  • Backup current configuration
  • some ADVM (ASM Dynamic Volume Manager) and ACFS (ASM Cluster File system) foundations can be found here
  • now we need to install a RAC database – this is covered here and here

Oracle 11g Release 2 Install Guide – Single database binary installation

Oracle 11g Release 2 – Single database binary installation guide

 

Local System configuration

Installation took place on laptop with CentOS 5 x86_64 (Kernel 2.6.18-128.2.1.el5)

  • 4 GB Memory
  • 500 GB local disk

 

Steps required to install Oracle 11g Release 2

  • Configure Storage
  • Check and fulfill pre-requirements
  • Binary installation of database
  • Listener configuration
  • Creation of database

 

Check and fullfill requirements

Storage Requirements

  • As always – recommendation: SAME (stripe and mirror everything)
  • Valid storage options for single database instance:
    • file system (ext3, reiser, xfs, etc al)
    • ASM
    • ACFS (!! only for non-database files (i.e. Binary files, trace files and so on!!)
    • NFS
    • ISCSI
    • RAW Disks (read documentation; deprecated option!)

 

SWAP

  • Between 1 and 2 GB RAM –> SWAP 1.5 times the size of RAM
  • Between 2 and 16 GB RAM –> SWAP equal to size of RAM
  • > 16 GB RAM –> 16 GB SWAP

Automatic Memory Management

  • Required /dev/shm with appropriate size (i.e. SGA of 16 GB required /dev/shm to be 16 GB+)
  • Huge Pages and autom. Memory Management are INCOMPATIBLE

 

Supported Operating Systems

  • on 32-bit Linux
    • Asianux 2 Update 7 (Kernel 2.6.9 or later)
    • Asianux 3 (Kernel 2.6.18 or later)
    • Oracle Enterprise Linux 4 Update 7  (Kernel 2.6.9 or later)
    • Oracle Enterprise Linux 5 Update 2 (Kernel 2.6.18 or later)
    • Red Hat Enterprise Linux 4 Update 7 (Kernel 2.6.9 or later)
    • Red Hat Enterprise Linux 5 Update 2 (Kernel 2.6.18 or later)
    • SUSE Linux Enterprise Server 10 SP2 (Kernel 2.6.16.21 or later)
    • SUSE Linux Enterprise Server 11 (2.6.27.19 or later)

!! ACFS and ADVM are ONLY supported on RHEL 5  and OEL 5 !!

 

  • on 64-bit Linux
    • Asianux 2 (Kernel 2.6.9 or later)
    • Asianux 3 (Kernel 2.6.18 or later)
    • Oracle Enterprise Linux 4 Update 7 (Kernel 2.6.9 or later)
    • Oracle Enterprise Linux 5 Update 2 (Kernel 2.6.18 or later)
    • Red Hat Enterprise Linux 4 Update 3 (Kernel 2.6.9 or later)
    • Red Hat Enterprise Linux 5 Update 2 (Kernel 2.6.18 or later)
    • SUSE Linux Enterprise Server 10 SP2 (Kernel 2.6.16.21 or later)
    • SUSE Linux Enterprise Server 11 (2.6.27.19 or later)

!! ACFS and ADVM are ONLY supported on RHEL 5 and OEL 5 !!

 

Required Packages

refer to: http://download.oracle.com/docs/cd/E11882_01/install.112/e10840/pre_install.htm

Users and Groups

For separation of rights (i.e. Manage ASM storage, manage database instance)

Available groups:

  • OSDBA (typical: “dba”; have SYSDBA privileges on database)
  • SYSOPER (typical: “sysoper”; optional, limited set of administrative priv)
  • OSDBA for ASM (typical: “asmdba”; full administrative access to ASM instance)
  • OSASM for ASM (typical: “asmadmin”; administrative access to ASM instance via SQL*Net)
  • OSOPER for ASM (typical: “asmoper”; optional, like SYSOPER group for limited access)

 

  • Minimal group needed: OSDBA group
    (in this document the osdba group is named “dba”)

 

Shell Limits

  • In /etc/security/limits.conf

oracle              soft     nproc    16384
oracle              hard    nproc    16384
oracle              soft     nofile   65536
oracle              hard    nofile   65536

(replace “oracle” with user holding the installation)

  • In /etc/pam.d/login add if not existssession    required     pam_limits.so

 

Kernel Limits (MINIMUM values) in /etc/sysctl.conf

kernel.sem=250 32000 100 128
kernel.shmall=2097152
kernel.shmmax=536870912
kernel.shmmni=4096
fs.file-max=6815744
fs.aio-max-nr=1048576
net.ipv4.ip_local_port_range=9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048576

– SuSE only –
vm.hugetlb_shm_group=<gid of osdba group>

The values in /etc/sysctl.conf should be tuned (i.e. according to the number of instance, available memory, number of connections,…)

 

Kernel Limits – how to calculate them

The values in /etc/sysctl.conf should be tuned (i.e. according to the number of instance, available memory, number of connections,…)

see Part 2 for guides how to calculate the kernel parameters

 

User Profile file (minimum file) ~/.bash_profile (RHEL, OEL) or ~/.profile (SuSE)

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/11.2/ora11r2p
export ORACLE_SID=ORA11R2P
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH
umask 022

ATTENTION: This profile file is for use with a dedicated user for each database binary installation. If you wish to install several binary installation under one single user make sure ORACLE_HOME and TNS_ADMIN is unset

 

Installing the database binaries 

We will install an Oracle 11g Release 2 database (stand-alone without grid infrastructure, without asm), operating system user is named “ora11”, osdba group named “dba” with home directory “/u01/app/oracle/product/11.2.0/ora11”.

Note: According to OFA the directory shall be named “…11.2.0/db_1” For easy management we install a dedicated binary installation under a dedicated user for every database which runs on the system and name them accordingly.

For instance binary database installation for database with SID “ORA11” is held by operating system user named “ora11” with home directory “…11.2.0/ora11”.
The binary installation for the database with SID “ORA11T” is held by the user named “ora11t” with home directory “@…11.2.0/ora11t”.

You can of course use one single user on operating system level for having one or more binary installations. It´s up to you.
 

Create User

 mkdir -p /u01/app/oracle/product/11.2.0/ora11
 useradd -g dba -d /u01/app/oracle/product/11.2.0/ora11 ora11
 passwd ora11

Create profile file

 export ORACLE_BASE=/u01/app/oracle
 export ORACLE_HOME=$ORACLE_BASE/product/11.2.0/ora11
 export ORACLE_SID=ORA11
 export NLS_LANG=AMERICAN_AMERICA.WE8MSWIN1252
 export TMP=$ORACLE_HOME/tmp
 export TNS_ADMIN=$ORACLE_HOME/network/admin
 export TEMP=$TMP
 export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

 

Check, if X11 works

export DISPLAY=localhost:0.0
xterm

Check Limits

ulimit -a         (as user “ora11”)

 step1_001

Note: “open files” and “max user processes” should show values greater or equial the  requisites.

 

Start installer and perform binary installation

 
step1_002



step1_003
step1_004
step1_005
step1_006
step1_007
step1_008
step1_009
step1_010
step1_011
step1_012
step1_013
step1_014
step1_015
step1_016
step1_017
step1_018
step1_019

 

Thats it!

 

The next step would be creating the database itself. A guide how to do this can be found here.

Oracle 11g Release 2 Install Guide – Overview

Oracle 11g Release 2 – Overview

New Features

  • New Features – a non-complete overview:
    • Automatic Block repair in data guard scenarios (i.e. Replace defective block on standby with valid block from primary database or vice versa)
    • Automatic determance of parallelism degree based on object size, query complexity and hardware resouces
    • RAC installation complexity dramatically reduced – we will evaluate this :-)
    • Storage of Oracle Cluster Registry and Voting Disk in ASM
    • Oracle Restart (automatically (re)starts database instance, ASM, listener and so on)
    • Utility for complete uninstallation of Oracle RAC
    • Reduced (“zero downtime”) for patching RAC clusters
    • Instance Caging: Limit number of CPUs used
    • Scheduler improvements (emailing, file watch, run procedures on remote db)
    • ACFS (ASM Cluster File System): Cluster file system based on ASM for non-database datafiles, i.e. (Oracle) Binary installations, trace files, alert logs and so on
    • ASM Dynamic Volume Manager: create volumes out of disk groups and use it to create a file system (ext3, reiserfs, etc pp) on top of it
    • ASM FS Snapshots
    • Intelligent data placement: frequently accessed blocks are placed on the edge of the disk where I/O performance is higher by ASM
    • ASM File Access Control
    • New compression algorithm (LZO) offers fast compression and de-compression
    • most “cool” ASM features require 11g release 2 database and asm compatible level (= everything 11.2)

Oracle 11g R 2 – Components

  • Oracle Database
    • traditional database (rdbms)
    • ASM module
    • Listener et al
  • Oracle Infrastructure (=foundation for Oracle RAC), includes:
    • Clusterware
    • ASM Module
    • Oracle Restart

?? Is 11g Release 2 ready for production yet ??

  • From the Authors point of view: NO!
  • Release of 11g Release 2 was driven by marketing and promise “release in September 2009”
  • Latest beta of 11g R2 (released two weeks before “productive release” still contained critical bugs
  • 11.2.0.1.0 good for testing and educational purpose
  • For productive environments:
    • Wait at least until 11.2.0.2.0! (i.e. The first patchset)

Oracle 11g Release Installation HowTo

Oracle 11g Release 2 was released on 1st of September 2009. From my point of view this version will be the next widely used version after 10 Release 2. In myopinion 11g Release 1 is a quite good and stable version – but it is a “Release 1” and many ISVs waited for 11g R2 to apperas to certify their products against.

 

I am investigating Oracle 11g Release 2 over the next few weeks starting with a single database installation, installing a 2-node-rac with ASM, testing ADVM and ACFS. Finally i will investigate database features, for instance advanced compression.

I will continuously publish my experiences on this blog.

At this point (14th September) the document covers:

  • oracle 11g release overview
  • new features
  • single database installation
  • grid infrastructure installation (formerly known as “clusterware”)
  • Installing Oracle 11g Release 2 database binaries on ACFS
  • How to size ASM disk groups of OCR and Voting disks correctly

 

The most recent version can be found here:

oracle-11g-release2-installguide-2009-09-14

 

Hopefully today i will add the creation of a rac database as well.

 

Please leave me a note or send me an email if you have any feedback.